Since you say you have shot both & like shooting both, I'd suggest you go with the .40S&W, although I personally prefer the 9mm.
Wound channel tests in gel, that I have seen, show the wound channel of the .40 being real close to that of the .45 with the 9mm being a distant 3rd. place. Of course, that will depend on the ammo selection & it's expansion, etc. so next question would be availability of ammo.
I prefer the 9mm because I am a firm believer in practicing with my defense firearm. I have both the 9mm & .40S&W in my carry rotation. But, I don't like practicing with my .40 as much as I do with the 9mm, so I find myself leaving the .40 home more than I should. If this isn't an issue for you, get the .40.