The FN 5.7 is a very small, very fast moving round with very little recoil. While this application is ideal for small hands, small stature types in a pistol configuration, it's tragic in the world of an AR usage log.
It's too small and provides too little wound cavity to be truly useful against anything but varmints, small birds and perhaps cats. The small, fast moving round would be ideal for punching through tough hide, but wouldn't do much once it got inside.
My initial thought is that the round would punch through the animal completely, leaving a moderate wound channel ( at best ) and would lead you on a track and stalk for a ways before it died a slow and painful death from fluid in the lungs or something like that. Either way, the adrenaline would be coursing through the meat, which isn't exactly leaving you with a Michelin 5 Star meat for the cooking.
Now, for a pistol to carry, I don't think the round can be discounted because it will definitely mess up a male sized attacker and the ability to put multiple rounds on target in a hurry because of the very low recoil would be a huge bonus.
Ammo is expensive. You are looking at $0.38/$0.45 a round and the AR platform is really built around the plan to shoot it quite a bit. Add to that the fun factor of shooting one and your budget just went up again.
.223/5.56mm can be had cheaper, it is WAY more readily available and it's got a much better application for up to "mid sized" game.