swedes were the M96, not swiss.
now my nitpicking aside. I'm a purist when it comes to any old gun not russian or chinese so I would personally not use the M96 but if it was already drilled and tapped or otherwise bubbad then there is no point in preserving it and it would make a very good base to build off of. at the last gun show there was a swedish mauser that was converted to a scout rifle and though I'm in the same boat as you... I was drooling over it until someone else bought it first... the jerk.
it can be done and I bet it was a very good scout rifle. though the concept of the 308 was more for readily available ammo, 6.5x55 has a lot in the way of reloading components so in my opinion it would make a very suitable gun. it take any game in america just as well as a 308 could so I say go for it.
however... with my purist nature getting the best of me, if that m96 action is still returnable to collectors grade then I say buy a tika, sako or CZ and build off one of them... they wont be collectable until I'm too old to enjoy them anyway so what do I care
