RootsMagic 8 has a different file format and extension for database (.rmtree) and backup (.rmbackup) files. The RootsMagic 7 database (.rmgc) cannot be opened inside RootsMagic 8. The RootsMagic 7 database (.rmgc) file needs to be imported into RootsMagic 8 by creating a new file. The new database will then have the correct (.rmtree) file extension.
The RootsMagic 7 backup (.rmgb) needs to be restored inside of RootsMagic 8 by going to Files, Restore from backup. During the restore process it will create a new (.rmtree) file and leave the RootsMagic 7 backup file intact.
(See Creating a New Database for additional steps)