Because to put a new finish on it it needs to be very smooth. Sandpaper will leave deep scratches that need to buffed out to a glass smooth finish. Removing the sandpaper scratches will remove more metal than needed.
About 95% of the polishing I do is done with paper but I chemically strip the bluing first. Unless I have to do some draw filing, the lowest grit I start with is 320 then I go up from there. Something that helps cut done on the gouging is to use good quality paper with a lubricant. My favorite is a 1:1 mix of kerosene and dark cutting oil, both can be found at a hardware store. Also, when I get close to the desired finish I switch to micron graded paper. The grit is kept at a very uniform height to it doesn't gouge and leaves a very even polish. A piece of micron 320 grit cuts like worn 400 grit in standard paper. Over the years I've buffers to do a lot of work, mostly because of the speed, but I prefer to polish by hand.