First of all, begin the legal challenge about your denial. No matter what, you will need to get that straightened out.
The cannot legally keep the entire price of the gun. They can charge a rediculous "restocking fee" but they can't keep all your money, if you don't get the gun. The line about "they can keep it because you paid cash" is pure BS!
Some places will "sell" guns before the background check is done, because the law allows it. Delivery (not purchase) is what is being denied with the failed check. Its not ethical, but it seems to be legal. That's why they can charge a restocking feel, because you "bought" the gun. Its a trap, but under current law, there doesn't seem to be anything that can be done about it.
Keeping all the money you paid is bogus. Clearly they are hoping you will just eat your loss and go away.
There is a legal method for finding out why you were denied. ITs a hassle, takes time, is frustrating, and might even cost a bit, but DO IT! For one thing, if you want to buy a gun from another store (and I don't think I would do any more business with the one who is trying to rip you off), you will need to pass the background check. So get it cleared up. IT might be a clerical error. IT might be something you don't remember (or aren't aware of) that is snagging you. Also, find out if you whether you are DELAYED or DENIED! There is a huge difference! The store might be lying to you. Any place that commits one ethical violation is likely to commit others too!
If your record is squeaky clean, go get 'em tiger! If not, find out what needs to happen to fix it (a lawyer might be needed). There are things that can put you on the denied list that you might not be aware of (and it might be state denial, not federal). You need to find these things out, then decide the best course of action.
Please let us know how things proceed for you. A gun store that does that should not be allowed to stay in business.
One other thought, some stores are owned by people who don't run them. This may be a case of the clerk ripping you off, and the owner knowing nothing about it. I suggest you contact the store owner/ guy who holds the FFL, and see if they know, and approve of what's going on.
All else being equal (and it almost never is) bigger bullets tend to work better.