My opinion: Always carry with one in the chamber. It's there, ready to use, without having to think about it.
I've carried a handgun since Jan of 1972. Back then I don't remember anyone using a holster to carry our weapons off duty with the exception of winter in the north. Then we could put away our peashooter 38's and carry our magnums in shoulder holsters. We all would stick whatever we were carrying in our waistbands. Some would wind tape or rubberbands around the grip to keep the gun from sliding down inside our pants. Later on when we went to semi-auto's; Glocks, Sig's, S&W's, etc. same thing, slide them in the waistband and go. I think the main reason is that the guns were easier to hide without the bulge of a holster.
My son just bought an XDM and he was asking me about a carry holster saying that he remembered I never carried in one. He asked "weren't you afraid of the gun going off"? I told him if I was worried, I never would have done it. He's getting a holster for his peace of mind.
Now I have known several people who have shot themselves, one while practicing his draws from a holster in front of a mirror (in the ass), and one at the range (down the right side of her calf) putting her gun in the holster with her finger still on the trigger. Both were police officers with several years on the job who should have known better.
I have also heard years ago, of older, very cheaply made guns fire after being dropped to the floor or ground. Manufacturers have taken great strides to make better weapons, mostly because of the potential lawsuits.
If you are careless, bad things happen.
Always carry your weapon in the same place and manner, get comfortable with it.
Practice and practice again drawing and replacing the weapon only when its unloaded.
Never put your finger on the trigger until you are ready to fire that weapon and then only when the weapon is pointed in the direction you want to shoot.
Follow the rules of safety and respect your weapon(s) and you should be okay.
My 2 cents......