You may not be old enough to remember, but Osama Bin Laden and Al-Qaeda operated out of Afghanistan. They actually considered all of the other countries you mentioned to be legitimate targets, and did conduct attacks in them. The Taliban refused to just hand them over, and Bush invaded. There was some talk about whether or not Bush jumped the gun and the Taliban would have been allies with a smidgen of diplomacy, but that’s the basics. We didn’t declare war on Saudi Arabia because the royal family of Saudi Arabia hated AQ as much as we did and was already fighting them.