You stated that you have removed the hex screw and that the castellated nut is loose so you are almost there.
Clamp the barrel in the padded jaws of a vice and using a small block of hardwood, tap lightly from behind the bayonet lug if it has one or from behind the front sight base forward with a small hammer. You might have to loosen the castellated nut more to get it completely off the threads when the brake starts to move.
The splines could have been peened or there might be red loctite there as well. Just be careful. If you can't move it, take it to a competent smith.