Prior to 2018, I had "only" one rare disease, hypermobile Ehlers-Danlos syndrome. Since the usual issues were nothing special or life-threatening (just painful and frustrating!), I didn't worry about it much. I was a bit careful too because I'd always get sick (travelling or just going anywhere, really... I had an immune dysfunction I only learned about in 2021).
I did have to plan A LOT, bring a lot of braces and medication, but that aside it was ok. I went to China, the Caribbean, travelled in the US (I'm from Canada), and was planning to travel as much as I could.
I got my adrenal insufficiency diagnosis in 2018, and it would mean paying extra for special travel insurance, being extra careful and I definitely would've worried more. Not for like a weekend trip by car, but for anything bigger. I still planned on travelling, but it worried me a lot more, as just the stress of travelling could cause an adrenal crisis!
Since 2020, I definitely wouldn't travel except to rent a cabin or go camping by car, COVID is not something I want, and it's very hard not to catch it travelling. I must say I also would think twice about all the pollution linked to travel.
Last time we rented a cabin wasn't easy, I didn't have much fun, as I was in a lot of pain from having to use stairs and a non-raised toilet and sofa... In fact it's how I realized just how much adaptations we made in our house which we forget but make my life easier!