I'm a firm believer in breaking in a barrel. I can feel a difference in the bore after breaking in. When you slide a patch down the barrel it is noticeably smoother. A smooth barrel with the right type of patch will literally whistle when you slide the patch down the barrel. That results in less copper fouling which does affect accuracy. All the cleaning removes the fouling from the barrel so that the next round contacts the steel of the barrel rather than copper fouling deposited from the previous round. That lets the bullets smooth out the tool marks and burrs in the barrel, which is what causes a barrel to be rough. It also lets the bullet even out the high spots on the lands where the bullet first contacts them.
Premium custom barrels do not need breaking in because they are all hand lapped. The reason for the hand lapping is to smooth out the barrel. Production barrels are not hand lapped.
Oil in the barrel is a different issue. That causes hydraulic pressure to build up and can do all sorts of damage and crazy stuff with accuracy. Run a patch with oil down the barrel before storing, but always run a dry patch down a barrel before shooting.