You will suffer power loss at elevation, just more if you don't rejet (3%/1000' or so). I ride my SS at 6000' here in Wyoming but when I go to Tucson or Las Vegas, I go up 2 sizes on the main jet (and the pilot also). You can also cut the airbox top and install a K&N air filter, which should allow more air in. You don't absolutely have to jet leaner at altitude, but you will be pretty rich gaining nearly 5000' of elevation if you don't! In other words, yes, rejet.
Joe