At 30 yds, either one will work. There is no dead, deader, deadest- only dead. The .17HMR does carry more energy than the .22 LR (@ 30 yds, 17 HMR 208 ft lbs, .22 LR 114 ft lbs) it shoots flatter, and it costs more (about $13/50 rds, vs $2.50/ 50 rds. The 17 really comes into its own for very tight groups, and at about 100 yds.
Personally I've seen nothing that a 17HMR can do that a 22 mag does not do better. I bought a couple 17's back when and never did really get into them. THey where pretty fun to shoot, I just liked my 22 more so the 17's left.
I am really wanting to try one of the 17m2's for some reason. Maybe I"ll convert one of my 10-22's and see what's up. I have no real reason to want one. It's just my being curious. But truth be told that was what got me to wan tot try the HMR too so....I guess we'll see.
I had a Marlin 17 HMR for a little while, didn't shoot it much but the few rounds I put through it were kind of fun. That being said I've also read that the little .17 cal bore fouls quickly with much shooting so IMHO the .22 Mag would be a better way to go. Either way I think even the Ruger Air Magnum(yep, I'm working it into conversation here,LOL) in .22cal would kill a coon at 30 yards with proper shot placement. Get what you can afford and find then shoot it enough to hit with it and you'll be good to go regardless of what caliber you get.
depending on the size of the coon, a 22lr will get most small to mediums. I've seen a med-large coon take one or two 22LR's to the body and then needed a head shot to finish it. But what caliper weapon do you have? a 22lr, 22mag, a 17HMR?
I do agree with most of the above recommendations as well. so Like I stated it depends on what you shot with and what you shoot at. If you have no weapon right now then I'd consider a 22lr or a 22mag over the 17hmr just because of pricing of ammo.