My family hates my boyfriend?

Im 19 and he is 21. Hes done a few hurtful things, but is a complete sweetheart and absolutely LOVES ME. He hasnt done or said anything to my family, yet they feel the needto tell me (and my grandma even told HIM) that hes unwelcome. They have no reason to feel this way. He moved just to be with me. Yes, hes taken up a lot of my time but I still make plenty of time for my family. What can I do? Hes literally too afraid to come visit because of this.