What is the most respectful way to reject someone?

I dont usually reject people because i dont get asked out that much, but this guy in my class asked me out yesterday and i said no. The guy i told no to was from a small town in the south. Some info about him is that he has a sister with down syndrome. Glancing back on the conversation i had with him, he literally told me about his sister and wanted to know if i was okay with dating someone who had a sibling that is disabled. I told him it's literally okay with me, i date the person not their family. He told me that the first girl he asked out in high school, she rejected him for that reason, she literally told him "you must carry that gene because your sister has it." I found this really fucked up and i asked him if he had a hard time asking people out after that. He told me that he usually only asks people out that he's really sure about. He said that he's been wanting to ask me out for a while but he was too nervous. I just told him that he's a really nice guy but im not really feeling it and that im really flattered he thought of me, and he literally thanked me for being considerate. I just feel bad for him because i feel like i shattered his confidence. I have to see him everyday so literally seeing him right now hurts me. I just dont want him to feel bad about himself is all.