Are you a smoker? I ask because when I was young and stupid I used to. I competed regularly in a lot of bullseye and 2,700 matches back in the 70's then, and I had similar issues. I found out it was smoking related. I quit, and within 6 months my shooting improved remarkably. Breath control is everything. Especially in rifle shooting. If you deplete the oxygen supply to your lungs by smoking, or even lack of exercise, this can cause the problems you're experiencing.
So if you do smoke, quit. If you don't try walking every day, or else do what I did and buy a treadmill. Here in Phoenix it gets too hot to exercise outdoors in the Summer months. So I put a treadmill in my master bedroom, and before I take my morning shower I spend time on it watching the morning new and business reports. How long depends on how I feel.
I usually alternate between a longer, slower paced workout, to a faster, shorter one from day to day. That way it doesn't get so monotonous. If you do this you'll be surprised at not only how good you feel, but also how well your shooting will improve.