Did you check the guide rod? Pull the stock off, and look where the end of the guide rod rests against the cross pin. The "tang" end of the guide rod needs to be on the upper side of the pin (the barrel side). If it isn't, you will experience exactly the problem you are describing. Don't start filing any parts, and for heaven's sake, don't break out the Dremmel tool!
Here, study the end of the guide rod in this photo. The 1911 buffer is optional, but the photo also shows how the end of the rod must engage the cross pin. It really sounds like yours is upside down...