You need to try a holster from N82Tactical.com. I know you said you like your holster but these are great. Very comfortable, good price, no part of the gun touches the skin, which is good when your gun gets hot or cold in the car too.
If you really want grips, checkered wood grips are fairly comfortable, and there are many less aggressive textures you can buy in wood or G10. Check out VZ Grips, Larry Davidson, Esmeralda, Hogue. But none o these are as comfortable as the suede backing on the N82 holsters.
FYI: I usually do buy grips for my guns but it's more for looks than comfort.