AITA for refusing to babysit my grandkids because I dont want to

This has been an ongoing issue for a while now. My son and his wife has a 2 boys aged 3 and 5. They usually use my wife and I as their baby sitters whenever they needed to go out. We stayed with them for the first year of the older boy but we moved out afterwards because we didnt want to intrude. But ever since then, they rely heavily on us to watch the kids. My son and his wife are very career conscious. Both of them make good money but they work long hours. My son works 12 hours a day (he's a nurse anesthetist) while his wife is lawyer. But thats their careers. We cant blame them for that. But we have needs too. I have been diagnosed with advanced parkinsons and my wife has bad arthritis. We just want to enjoy the remaining years of our lives in peace. But because we live nearby, my son and his wife always expect us to watch the kids without asking us first. They would sometimes drop the kids off without asking us first. But I just recently snapped. They asked us to watch the kids overnight because they have a wedding to go to. We said no. My son yelled at us that we're terrible grandparents. He went storming off and Im worried he will make a scene at the party. Aita here.