ELI5: Why did no one go to the moon for 46 years

My dad claimed it was because people were scared to go but isn't the reason why we went to the moon the first time was because we were scared the Russians would be the first to go. I mean, even if the Russians go I don't see the point of us not going.