Are you shooting cold bore with a cold zero?
Or a cold bore with a hot zero?
Or a warm bore and a warm zero, but your first group is off because you are shooting cold??
Basically, if we assume that NOTHING is wrong with the rifle ( outside influences due to heat, directional force of a sling incorrectly applied, pressure on the barrel from bracing, etc. ), we assume that the ammo is consistent, the shooter is consistent, then we have optics and zero.
If the scope is not mounted correctly and is susceptible from heat caused by shooting, that could contribute.
However, without knowing when you zero, and when you shoot, that would be the first thing that I would question.