Failed to import a mongodb database with duplicate fields

Question

I'm trying to import a mongodb database and I've been using the mongorestore command, the import starts without problems until it gives the error: Failed: dbprod.mys_account: error creating collection dbprod.mys_account: error running create command: BSON field 'OperationSessionInfo.create' is a duplicate field

I already tried the option --drop but without success, I had never worked with mongodb ... I must clarify that I saved it in a mongodb 2.4 with mongodump and I try to import for a mongodb 3.6

Answer 1

I have successfully upgraded a database from 2.4 to 4.0 using the following 3 step method:

1. use 2.4 mongobackup on the 2.4 database
2. delete all .json files from the outputted dump directory
3. use 4.0 mongorestore on the remaining .bson files

Answer 2

As you said you are trying to import a MongoDB database, And you are using the MongoDB command such as mongorestore. And also as you said you have taken mongodump backup of MongoDB database , which MongoDB version is 2.4 & you are going to import that mongodump backup in MongoDB , which version is 3.6.

As I would like to say here as per MongoDB BOL documentation here Starting in 3.2, MongoDB uses the WiredTiger as the default storage engine. Previous versions used the MMAPv1 as the default storage engine. As the Compatibility Changes in MongoDB 3.2 here and here.

I am not sure what commands you have used to import the mongodump data. As mongoimport and mongorestore both commands have a different behaviour in MongoDB.

The mongoimport tool imports content from an Extended JSON, CSV, or TSV export created by mongoexport, or potentially, another third-party export tool.

Note : mongoimport only supports data files that are UTF-8 encoded. Using other encodings will produce errors.

For example mongoimport the syntax should be like as mention below

>mongoimport -d test -c zips C:\MongoDBDBA\zips.json

As here test is database name & zips is collection name and zips.json is (.json) file which is going to import in MongoDB database. For more details about mongoimport, you can find the Blog of Ken W. Alger here.

As the mongorestore program loads data from either a binary database dump created by mongodump or the standard input (starting in version 3.0.0) into a mongod or mongos instance.

If you are doing mongorestore stop here and must look upon the Version Compatibility.

Version Compatibility

The data format used by mongodump from version 2.2 or later is incompatible with earlier versions of mongod. Do not use recent versions of mongodump to back up older data stores.

Exclude system.profile Collection

mongorestore does not restore the system.profile collection data.

Required Access

To restore data to a MongoDB deployment that has access control enabled, the restore role provides access to restore any database if the backup data does not include system.profile collection data.

the mongorestore syntax should be as mention below

>mongorestore --collection people --db accounts dump/accounts/people.bson

Here accounts is a database name and people is collection name. And people.bson is mongodump (.bson) file, which is going to restore in MongoDB database.

For further your ref here and here

Answer 3

Problem is in metadata.json files. You need to remove 'options' and it will work just fine. I even created small python tool to do this:

from glob import glob
import json

for filename in sorted(glob('dump/*/*.metadata.json')):
    with open(filename, 'r') as f:
        data = json.loads(f.read())
        if 'options' in data:
            del data['options']

    with open(filename, 'w') as f:
        print(json.dumps(data), file=f)