I had this problem for a bit and I've seen this complaint before.
You can send it in to Springfield and I'm sure they'll make it a bit easier.
But for a quick and dirty solution: Simply push your magazine in further a bit, towards the barrel. The magazine tends to have some force pushing it out a bit, which must misalign with the release. It should be easy to release if you do this.
I'm sure this is old news to many, but my friend figured it out on my XDm9 and I found it useful.