Who thought hot air balloons were a viable option for manned flight?

Title basically says it all. I know that hot air balloons were the first manmade method of flight, but to me that makes no sense. Sure, if you knew nothing about air pressure and flight mechanics, I can understand how someone might think that a balloon full of hot air could just magic itself into the sky, but how did it take off so well? How was what was basically the scientific equivilant of a magical broomstick able to become a norm for the first few hundred years of man's flight?