Nothing wrong with a drill press even if you had a mill you still have to do the sides with a drill press
There's plenty wrong doing it with a drill press, and no, if using a milling machine, that's all you need. You wouldn't even need the jigs.
A mill, (AKA Bridgeport, knee mill, vertical mill), is basically a drill press on steroids. I will do everything a drill press will do, and also holds the work very securely in a vise, bolted to an X/Y table. Some also have power feed on the table, as well as on the quill. A mill also has the ability to very securely hold, (usually using R-8 collets, but sometimes a drill chuck), various cutting tools. End mills, ball end mills, fly cutters, taps, drill bits, etc. Also, the mill may have, if you're lucky, a DRO. (Digital Read Out), that tells you exactly where you are, in relation to a zero point that you can set, both in X and Y. If no DRO, there are graduations on the hand wheels that do the same thing, but not as easily.
A drill press, on the other hand, is going to force you to use a jig, (more $$), does not hold tools or the work as securely, doesn't tell you where you are, give a zero point, have power feed, or a DRO, and is going to probably, from what I've seen, force you to use an end mill to finish off the side walls, using your hands to guide the jig. Not conducive to getting a good finished project. Not saying it can't be done, but the chances of success are much less. It's at best a half-a$$ed solution to the problem.
LOL, doing it with a hand drill alone, you might as well take that 80% lower and throw it in a tree shredder.