I would also check for severe lead fouling, and address that first. If for some reason you are using heavy for caliber bullets, that may be a significant cause as well.
Not to sound confrontational, but devastating or not, hitting close to POA and keyholes at 20 feet are somwhat concerning to me. I wouldn't lose sleep, but I would either fix, or get rid of, the gun. Either case, keeping it for defensive use is unsat.