First off, Welcome to the Forum -
You are experiencing what Snipers call the Cold Bore Shot. It absolutely has to do with the temperature of the steel in the barrel with your first shot, or shots in the case of the small round like the .22
That is what makes a Tactical Shooter / Sniper for a SWAT team, for the military or for the Secret Service so special as opposed to normal shooters.
They only get one shot, and on top of everything else they have to contend with, they have to contend with accuracy differences that a Cold Bore shot involves.
Some "sharpshooters" who take professional shots for less than 100 yards, which is your average police "sniper" - will clean their rifle, then fire a fouling round and leave the bore dirty when they store the weapon to help assist in this compensation.
For longer range shots, there is something called a Cold Zero - which is usually several clicks of the scope off what their normal zero is.
In this day and age, with the ability to put together tack driving rifles right from the manufacturer, the two great things the average shooter has to deal with is the cold bore first shot, and the environmental conditions that the shot is being taken in....
I would suggest reading "The Ultimate Sniper" by Maj. John Plaster as he has a TON of information not only about this issue, but about accurate shooting in general and tricks / tips the average shooter can utilize to improve their skills....