As a shotgun shooter, I fail to see where you could come in physical contact with lead with the exception of MAYBE a very TINY amount of atomized/aerosolized lead that is released into the air when you pour it from the bag to the bottle (and even that is going to be almost completely graphite dust).
Unless you are running your hands through the bag of shot, it should be a non-issue. There is no contact when you reload (shot goes from bag to bottle to shell) and no contact when you shoot.
I doubt washing your hands is even really that necessary, since the cause of your hands getting dirty is from the carbon of the burnt powder, not from the lead (you never touch the lead, and nothing you ever touch touches the lead).
The only time I could see it being an issue would be if you were making your own lead shot, working with cast lead bullets, or shooting rifle/pistol @ an indoor range. Other than that, there is no way you can physically come into contact with lead (other than what I outlined before) unless you are purposefully doing it.