Don't forget that brasshopper had asked about primers for use in 9mm cartridges on another thread. So the answer for using rifle primers in any cartridge of this type is definitely - no. As for using rifle primers in handgun cartridges, I agree with bk40, it certainly can be done with good results. But, it is imperative that you know what you are doing and don't just substitute blindly. I have used rifle primers in handgun cartridges many times, but only in large capacity cartridges and only with revolvers. Revolvers, in general, use positive action firing pins as opposed to pistols which generally use inertia firing pins where ignition might be a little more iffy. Of course, you have to know your gun and load to know whether a rifle primer is in order in any gun.
A small demo I like to give to people interested in reloading is to load some rounds of most any type, leave the old primer in, and the bullet out of one round. We go to the range where I pour the powder from that round in a small heap and light it. Depending on the powder, of course, it usually burns up in about 3 seconds. Then I shoot one round. I know everyone has seen a gun go off, but this seems to be a real eye opener as to the chemistry of nitro cellulose. It emphasizes the effect of pressure on the burn rate. That is, seeing the same amount of powder that took 3 seconds to burn under atmospheric pressure now take roughly .001 seconds. And that burn rate is a direct result of the primer. It is obvious that the main purpose of a primer is to supply a flame to the powder. But an equally important function is to place the powder under high pressure along with the flame. I'm sure the chemists could come up with primers that would supply the flame without a large volume of gas, but the powder burn rate curve would initially be more shallow. Once the powder starts burning it increases the burn rate rapidly as the pressure builds to extreme values.
So my reason for using pistol primers in pistols and rifle primers in rifles is very simple, if you don't, you are playing with the formidable chemistry of nitro cellulose under conditions that haven't been tested. If you do the testing yourself and know when to STOP testing - and this point is not always obvious - then you should be safe.