Sounds like you're getting away from the intended purpose of an ultra light rifle. It's rare that a pencil-thin barrel will hold a tight group for more than two or three rounds.
Your rifle is intended for carrying across country, with only an occasional shot. If the first shot is always to the same point of aim, it's doing a good job.
You might find some improvement by working over the bedding of the fore-end of the stock. If the wood is pressing on the barrel, there will be changes in pressure against the barrel as it heats up.
Free-float the barrel, but with not much clearance at the end of the fore-arm. Make a shim of kitchen waxpaper, where a five-pound pull will allow it to be inserted out at the end of the fore-arm. I have found that acts as a damper, as a "shock absorber", and invariably have been able to improve my groups.
Hope this helps, Art