This is the most ask question on shooting forums. Not knowing you or your level of shooting knowledge it would be difficult to answer. It would be best for you to enroll in a NRA Home Defense Class. This will allow you to make your own determination.
What would be the best caliber for a self/home defense handgun
Since we are talking only "handguns" here, I would say the largest caliber that you can shoot well without any undue problems. Revolvers are handguns as well and 357 Mag takes care of almost any problem that you will run into, however the recoil on that round is stout and most revolvers are limited to 5, 6 or 7 rounds. In most cases that would not be a problem, since I believe that one or two rounds would be all that you would use against a single attacker. If however, you feel that the recoil is too stout or that your situation would require more shots against many attackers then the 9mm will give you the most number of rounds per magazine. While I am not a fan of the 9mm for home defense (I do own 2 and use them), I would say the 45 ACP would be more approprate at stopping an attacker, but then again you are limited to the number of rounds that the pistol will hold (7+1). The comprimise of a 40 S&W that holds 12, 13, or 14 rounds might give you the edge, but from my experience with it the recoil is substantualy less controlable than either the 9mm or the 45 ACP.
Each caliber has it's benefits and draw-backs and must be weighed on the basis of the owner's needs. Only you can determine what will suite YOU BEST.
10mm and 357 sig are great rounds but uncommon and expensive. I keep 15 horniday critical defense .40 s&w rounds in my night stand and 6 horniday critical defense .357 mag rounds between the mattress and box spring. In guns of course lol
It is my opinion that you should use what is comfortable for you to use based on knowing you're surroundings.I speak only for me when I say I am most comfortable with my p226 with win Ranger 40. bonded jhp.I live in a block home and its just me and my gf so I don't worry about over penetration .However the entrance to my bedroom is about a 15' hallway x about 4' making maneuvering my shotgun a little difficult.There are two openings within that 15' one to the kitchen and one to the garage so I would have to be able to sweep left and right to clear both connecting rooms to move on.
So figure out something that is good for you're specific environment.Something that works for you and train and train and train and when you think you've trained enough train some more.Every ones environment is a little different and everyone is a little different when it comes to being comfortably familiar with specific tools and ammo.