A RootsMagic GEDCOM is created in version 5.5.1. and Ancestry and Family Tree Maker do not recognize any version above 5.5.
You will need to manually edit the GEDCOM. Open the GEDCOM file with notepad, near the top you will see the lines below (file will be different). Change VERS 5.5.1 to VERS 5.5, save and then upload the GEDCOM
1 DEST RootsMagic
1 DATE 19 APR 2011
1 FILE merge Smith.ged
2 VERS 5.5.1
2 FORM LINEAGE-LINKED
1 CHAR UTF-8
It is best when you create the GEDCOM to exclude the "Extra details (RM specific)" from the export.
Some users have reported issues uploading to Ancestry.com using the Firefox browser. It is recommended to use Internet Explorer instead.