I need to export the data from 36 SQL tables containing 24GB of data into flat files, copy them to the client and import them there into the existing tables in his SQL database.
And I will need this for several customers (same tables, though).
How do I mass export and import data?
Is there a command line tool for this so I can write a script for repeated use?
The basic knowledge you will find here Importing and Exporting Bulk Data
What is bcp ?
bcp.exe is the standard bulk import/export tool for MSSQL. Using SSIS packages is an alternative, but brings a lot of overhead with it: it's a full ETL tool. In TSQL there's also a BULK INSERT statement that you can use as an alternative to "bcp in", but I personally haven't played around to see which one is faster or more useful etc.
See "bulk exporting" and "bulk importing" in Books Online for all the details.
Related
I have more than 100 million records data in file txt. I would like to import them to SQL server. So, which one I can choose between data loader and SSIS. Thank you so much!
Assuming you are referring to the Import Data wizard when you say "The data loader", it is just a wizard that creates you an SSIS package to import your data. You even get the option to save your import as an SSIS package at the end of the process.
If you care more about the speed of the import, for 100 million records within a text file you would probably (but not definitely) be better off using the Bulk Copy Program (BCP) Utility provided by Mircosoft.
Edit following comments
From what I can see, DataLoader.io is a Salesforce only tool. It seems you cannot use it to load data into SQL Server. In this case, out of the two options you have suggested SSIS is the only viable option. Whether or not SSIS is suitable for your current and on-going situation however, is a larger discussion not really suited to the Stack Overflow format.
Which one is better option among following options in-terms of speed and performance for the purpose of exporting data from hive/hdfs to sql server.
1) Using Sqoop Export facility to connect to RDBMS (SQL server) and export data directly.
2) Dump CSV file using HIVE using INSERT OVERWRITE LOCAL DIRECTORY command and then perform BCP ( or Bulk Insert Query) on those CSV files to put the data into SQL server database.
Or,
Is there any other better option?
In my experience, I use bcp whenever I can. It's from what I can tell the fastest way to shotgun data into a database and is configurable on a (somewhat) fine grain level.
Couple things to consider:
Use a staging table. No primary key, no indexes, just raw data.
Have a "consolidation" proc to move data around after loading.
Use a row size of about 5000 to start, but if performance is of utmost concern, then test.
Make sure you increase your timeout.
I have a unique query regarding Apache Sqoop. I have imported data using apache Sqoop import facility into my HDFS files.
Next ,. I need to put the data back into another database (basically I am performing data transfer from one database vendor to another database vendor) using Hadoop (Sqoop).
To Put data into Sql Server , there are 2 options.
1) Using Sqoop Export facility to connect to my RDBMS,(SQL server) and export data directly.
2) Copy the HDFS data files (which are in CSV format) into my local machine using copyToLocal command and then perform BCP ( or Bulk Insert Query) on those CSV files to put the data into SQL server database.
I would like to understand which is the perfect(or rather correct) approach to do so and which one of them is more Faster out of the two - The Bulk Insert or Apache Sqoop Export from HDFS into RDBMS. ??
Are there any other ways apart from these 2 ways mentioned above which can transfer faster from one database vendor to another.?
I am using 6-7 mappers (records to be transferred is around 20-25 millions)
Please suggest and Kindly let me know if my Question is unclear.
Thanks in Advance.
If all you do is ETL from one vendor to another, then going through Sqoop/HDFS is a poor choice. Sqoop makes perfect sense if the data originates in HDFS or is meant to stay in HDFS. I would also consider sqoop if the set is so large as to warrant a large cluster for the transformation stage. But a mere 25 million records is not worth it.
With SQL Server import it is imperative, on large imports, to achieve minimally logging, which require bulk insert. Although 25 mil is not so large as to make the bulk option imperative, still AFAIK sqoop, nor sqoop2, do not support bulk insert for SQL Server yet.
I recommend SSIS instead. Is much more mature than sqoop, it has bulk insert task and has a rich transformation featureset. Your small import is well within the size SSIS can handle.
What is the best method to import data from an Excel worksheet? As of now I am use SSMS Express so I don't have access to SQL Import Wizard. I also don't have permissions to execute the BULK INSERT command.
My current workflow is as follow: Clean up the excel file, save as CSV, and import it into a SQLite database. Use an IDE like RazorSQL to generate SQL INSERT statements.
This worked nicely until I hit an Excel file about 75000 rows. SSMS just gives an error saying "query finished with errors" or something like that. No error message is shown. I tried adding GO at the end of each line but I got out of memory error.
What are my options?
To answer your question, the best method to import data from excel, in my past experience has been to read excel data into c#, do any clean up and formatting as necessary since excel likes to mess with the data, then use SqlBulkCopy (you only need select/insert permissions) to insert into SQL Server. See this SO answer if you need help reading excel from C#
Update: Given you're not a dev, try using the bcp utility (you should only need select/insert permission)you may need to save the excel file as CSV first, then import it directly into sql server, see this SO answer
You can use following:
bcp utility (between file system data dump and database),
OPENQUERY (can be used from SSMS, works between external datasource like Excel/csv and database),
BULK INSERT (can be used from SSMS, works between external file with user-defined structure and database),
SSIS (usually as dtsx package, has its own GUI, works with various souces and destinations)
Set of INSERT statements (all of them one after another, eventually sliced with GO or packed with UNION ALL)
Set of records serialized in XML variable (can be used from SSMS only; you have to serialize/deserialize it by your self using FOR XML and XML functions)
There are surely other possibilities, but these are maybe most used ones.
EDIT: It seems to me that you could try with GO after every 5-10K lines in your script.
If that doesn't work, XML serialization/deserialization could be the way to go.
Could you use a linked server to connect to the Excel Document? How to use Excel with SQL Server linked servers and distributed queries
A quick and dirty workaround: pull the rows in batches of 50k.
select * from employee limit 50000
select * from employee limit 50000, 100000
From
http://www.razorsql.com/articles/mysql_limit_query.html
I'd like to save data from an SQL Server database table to a file, then load it into another database that has the same table already created in it
How can I do this?
I know there should be some simple way of doing it, but Stack Overflow search and Google aren't yielding good answers (or I'm not asking a good question).
You could use the Import/Export Wizard. Right click on your database in Management Studio and look for import data and export data under the tasks item.
Check bcp utility and BULK INSERT statement.
the bcp utility and BULK INSERT as mentioned by #Lukasz Lysik are good for moving lots of data. However, sp_generate_inserts by Narayana Vyas Kondreddi allows for many filtering options of the data, and good when there isn't that much data.