How can I export mongoDB collection into .csv which is readable/compatible in SQL Server - sql-server

Question

I used mongoexport to export the collection in to csv file along with fields. However when I tried to import the .csv in sql server using SSIS. I got errors and the data in preview section before executing the package was wrong. Can any one please guide me how can I export the data properly which can be easily imported into sql server. I am ready for minor tuning like adding id column or changing data types etc.

Answer 1

So it looks like you are getting a json formatted object for each row in the file. It may be different than that, and if that's the case, how you iterate through each might change a little. But here is something to get you started:
1 - There is no out of the box JSON parser in .NET, but this seems to be a popular utility: http://www.newtonsoft.com/json
2 - If you download the parser above, you'll have to go through a little pain to get it in a useable state with SSIS:
Go to the source folder and open the solution for net40
Sign the assembly
comment out the lines that cause build errors
//[assembly: InternalsVisibleTo("Newtonsoft.Json.Schema")]
//[assembly: InternalsVisibleTo("Newtonsoft.Json.Tests")]
install the assembly to the gac
3 - Once all that is out of the way, add a file connection manager to your package and point it to the mongdb file
4 - Add a dataflow and then add a script component source to the dataflow
5 - In the script component, configure the connection manager that you created in step three, I gave mine the friendly name "MongoDbOutput"
6 - In the Inputs and Outputs section, go to Output0 and add a column for each field in the JSON object, setting datatypes to string as they will be int by default
7 - Open the script and add a reference to Newtonsoft.Json and System.IO
8 - The script below shows how to access the connection string to the file in the connection manager, read the file with a streamreader one line at a time and then parse each JSON object for each address. One line is added for every address and the name and ssn are repeated for each line.
Note also, that I added a person and address class. The json.net is pretty cool, it will take that json and push it into the object - provided all the fields match up. Goog luck!
using System.IO;
using Newtonsoft.Json;
public override void CreateNewOutputRows()
{
object MongoDbPath = Connections.MongoDbOutput.AcquireConnection(null);
string filePath = MongoDbPath.ToString();
using (StreamReader fileContents = new StreamReader(filePath))
{
while (fileContents.Peek() >= 0)
{
var contents = fileContents.ReadLine();
Person person = JsonConvert.DeserializeObject<Person>(contents);
foreach (address address in person.addresses)
{
Output0Buffer.AddRow();
Output0Buffer.name = person.name;
Output0Buffer.ssn = person.ssn;
Output0Buffer.City = address.city;
Output0Buffer.Street = address.street;
Output0Buffer.country = address.cc;
}
}
}
}
public class Person
{
public string name { get; set; }
public string ssn { get; set; }
public address[] addresses
{ get; set; }
}
public class address
{
public string street { get; set; }
public string city { get; set; }
public string cc { get; set; }
}