Depends entirely on what you want to DO with the handgun.
Hunting BIG animals with teeth and claws- the biggest caliber that does not require wheels to move it.
Social pupose concealed carry? I do well with a .357 revolver.
Putzing around on the target range? .22 LR
Just having fun? .44 ball and cap.
As much as I like steak, I do not have it 21 times a week. Sometimes lasagna, a salad, and some garlic bread fills the bill. I have .22 Short, LR, .25, 32, .380, pmm, 38 Special, 357, 44s, and 45s. Have never felt the attaction of the larger than .45 calibers, but under some circumstances, I would.
What we have here is... failure- to communicate.