First, you need to figure out the twist in the rifling. Some muzzle loaders won't stabilize sabots, conicals. Yours sounds like one of them. Shoot balls.
Accuracy is a matter loading consistently.
I swab the bore after every shot. Patch on a jag. Figure what works and do it the same way every time.
Flash hole, nipple, ignition path needs to be clean, clear. Use a copper wire/pin. DON'T clear this with a charge in the breech!
You need to match patch/ball, and bore. I have a 54 cal. I found a patch that works best with a 0.530" dia. ball. You'll find that ball is sold in varying diameters. Figuring ball/patch is part of the "art." Patches vary in thickness. I found some cotton demim in "June's Fabric" . . . waltz in with a verneer caliper and look at the "cotton twill" . . . 100%, no polyester.
Find your "grease" . . . I like the butter flavored generic stuff in the shortning aisle. 100% veggie, but that's me. Some shortening has animal fat in it. I don't think animal fat is a good idea. I HATE the smell of Bore Butter!
Load . . . takes some 'justment . . . Get a consistent charge, and then pack it the same in the barrel each time. The hot shots have a mark on their ramrod where the powder sits. Charge affects accuracy, just like in reloading.
Grease the patch, same each time. Patch over the muzzle, start the ball with a "starter." If the ball has a sprue spot, place the spot facing up, or down -- same way each time. Press the ball flush with the muzzle and trim the patch with a patch knife. My "patch knife" is a carbon bladed "Old Hickory" paring knife . . . that's sharp as I can get it.
Run the ball to the end of your ball starter. DON'T hammer the starter! PUSH the ball down the bore with the ramrod, gently, NO HAMMERING, smooth push. Hammering distorts the ball.
Hot shots have a second mark on the rod for where the ball sets.
Half cock the hammer, seat cap on the nipple, flush, and even -- same way each time.
A good muzzle loader, shooting a good muzzle loader can shoot "cloverleafs" at 100 yds. The gun will do it if you work up your load, consistently each time.
If it were "simple" there'd be no mystic and everyone would be shootin' smoke poles. It's complicated, arcane, and dirty. We like it that way!