Upload all text file content to mongodb (using pymongo) - file

Question

I'm starting with mongodb.
I have a text file with 3400000 lines of data and I want to upload that data to a mongodb database.
The text file looks like this:
360715;157.55.34.97;Mozilla/5.0;/pub/index.asp;NULL
3360714;157.55.32.233;Mozilla/5.0;/pub/index.asp;NULL
....
and I want to put it on a mongodb database with the following structure :
{'log' : '360715;157.55.34.97;Mozilla/5.0;/pub/index.asp;NULL'}
{'log' : '3360714;157.55.32.233;Mozilla/5.0;/pub/index.asp;NULL'}
....
Actually I am uploading line by line like this :
for data_line in records:
parsed_line = re.sub(r'[^a-zA-Z0-9,.\;():/]', '', data_line)
to_insert = unicode(parsed_line, "utf-8")
db.data_source.insert({'log':to_insert})
Is there a way to upload all lines at one time to the format I want to ?
The "line by line" approach is to much slower.
Previously , i had coded a script in python to parse the text file, and it takes one second to 100000 lines and takes 65 seconds to 3400000.
I considered to use mongodb to improve the process but , actually , with mongodb it takes the same time just to upload 100000 lines to the database that my script to performe all the data parse. So is not to much hard to say that I'm doing something badly wrong with mongodb.
I'm grateful if someone can help me.
Greetings , João

Answer 1

I would try bulk inserting the data into mongodb. Right now you are calling insert() once for every single line in your text file which is expensive.
If you are running mongoDB version 2.6.x you can use bulk inserts http://api.mongodb.org/python/current/examples/bulk.html . Example code below from the tutorial:
>>> import pymongo
>>> db = pymongo.MongoClient().bulk_example
>>> db.test.insert(({'i': i} for i in xrange(10000)))
[...]
>>> db.test.count()
10000
In this case a single insert() call inserted 10000 documents into mongodb.
Another example below:
>>> new_posts = [{"author": "Mike",
... "text": "Another post!",
... "tags": ["bulk", "insert"],
... "date": datetime.datetime(2009, 11, 12, 11, 14)},
... {"author": "Eliot",
... "title": "MongoDB is fun",
... "text": "and pretty easy too!",
... "date": datetime.datetime(2009, 11, 10, 10, 45)}]
>>> posts.insert(new_posts)
[ObjectId('...'), ObjectId('...')]
http://api.mongodb.org/python/current/tutorial.html