What is the best method to import data from an Excel worksheet? As of now I am use SSMS Express so I don't have access to SQL Import Wizard. I also don't have permissions to execute the BULK INSERT command.
My current workflow is as follow: Clean up the excel file, save as CSV, and import it into a SQLite database. Use an IDE like RazorSQL to generate SQL INSERT statements.
This worked nicely until I hit an Excel file about 75000 rows. SSMS just gives an error saying "query finished with errors" or something like that. No error message is shown. I tried adding GO at the end of each line but I got out of memory error.
What are my options?
To answer your question, the best method to import data from excel, in my past experience has been to read excel data into c#, do any clean up and formatting as necessary since excel likes to mess with the data, then use SqlBulkCopy (you only need select/insert permissions) to insert into SQL Server. See this SO answer if you need help reading excel from C#
Update: Given you're not a dev, try using the bcp utility (you should only need select/insert permission)you may need to save the excel file as CSV first, then import it directly into sql server, see this SO answer
You can use following:
bcp utility (between file system data dump and database),
OPENQUERY (can be used from SSMS, works between external datasource like Excel/csv and database),
BULK INSERT (can be used from SSMS, works between external file with user-defined structure and database),
SSIS (usually as dtsx package, has its own GUI, works with various souces and destinations)
Set of INSERT statements (all of them one after another, eventually sliced with GO or packed with UNION ALL)
Set of records serialized in XML variable (can be used from SSMS only; you have to serialize/deserialize it by your self using FOR XML and XML functions)
There are surely other possibilities, but these are maybe most used ones.
EDIT: It seems to me that you could try with GO after every 5-10K lines in your script.
If that doesn't work, XML serialization/deserialization could be the way to go.
Could you use a linked server to connect to the Excel Document? How to use Excel with SQL Server linked servers and distributed queries
A quick and dirty workaround: pull the rows in batches of 50k.
select * from employee limit 50000
select * from employee limit 50000, 100000
From
http://www.razorsql.com/articles/mysql_limit_query.html
Related
I need a bit advice how to solve the following task:
I got a source system based on IBM DB2 (IBMDA400) which has a lot of tables that changes rapidly and daily in structure. I must load specified tables from the DB2 into a MSSQL 2008 R2 Server. Therefore i thought using SSIS is the best choice.
My first attempt was just to add both datasources, drop all tables in MSSQL and recreate them with a "Select * Into #Table From #Table". But I was not able to get this working because I could not connect both OLEDB Connections. I also tried this with an Openrowset statement but the SQL Server does not allow that for security reasons and I am not allowed to change that.
My second try was to manually read the tables from the source and drop and recreate the tables with a for each loop and then load the data via the Data Flow Task. But I got stuck on getting the meta data from the Execute SQL Task... so i dont got the column names and types.
I can not believe that this is too hard to archieve. Why is there no "create table if not exist" checkbox on the Data Flow Task?
Of course i searched for the problem here before but could not find a solution.
Thanks in advance,
Pad
This is the solution i got at the end:
Create a File/Table which is used for selection of the source tables.
Important: Create a linked Server on your SQL Instance or a working Connectionstring for the OPENROWSET (i was not able to do so - i choosed the linked server)
Query source File/Table
Build a loop through the resultset
Use Variables and Script Task to build your query
Drop the destination table
Build another Querystring with INSERT INTO TABLE FROM OPENROWSET (or if you used linked Server OPENQUERY)
Execute this Statement
Done.
As i said above i am not quite happy with this but for now it should be ok. I will update this if i got another solution.
I have a stored procedure that I am going to run every weekend, it produces a result set that I need to export into an Excel file.
For the above problem I want to automate this process, so I am going to create a SQL Job and I am going to run this stored procedure every weekend so that that generated Excel file is sent to my reporter.
For this I need steps to export the result set data to an Excel file.
And also is it possible to send that Excel file to the specific mail while running the job itself?
So, you might try your luck on https://dba.stackexchange.com/, but in my experience a SQL Agent job running a stored procedure could be coaxed to return CSV or XML - and those could end up in Excel, but there are missing links. I think the missing links would involve programming and potentially 3rd party tools to avoid using Excel's COM API.
I'd strongly recommend your pursuing SQL Server Reporting Services. It is included free with your edition of SQL and includes the ability to
run reports on a schedule (subscriptions),
format the results as an Excel file
distribute the results via email
You'd take your query and use it as the data source for a "report" and use the report wizard to create a very simple table with the results.
Avoid page headers (or footers) that span columns - this will keep the excel output cleaner.
References
Stack Overflow: reporting-services-export-to-excel-with-multiple-worksheets
Technet: Reporting Services
I've created the structure of my database first in PhpMyAdmin and exported it to a .sql file.
Now I'm looking everywhere in SQL Server Management Studio where I can import/add the data in a new database.
Does anybody where to look or what to click?
I'm using the 2014 version (CTP2)
If you have a .sql file which contains SQL statements, you can just copy and paste the contents (or open the file in a query window) and run it. This assumes it has all of the create table etc. statements to create the schema/structure and not just insert statements for the data.
Check the top of the file to make sure that it is first selecting the correct database, if not add a USE statement to select the correct database.
You didn't say how big the file was, but if it is quite large and has the insert statements (data as well as schema), then you'll probably want to run by CLI using sqlcmd command. Much faster and SSMS won't freak out.
Another alternative option to running the .sql file/code is to set up a data source for mysql and just use odbc to access the database itself.
Bear in mind that there are real and very annoying differences between mysql and t-sql that can make migration a pain. If you're just creating a few tables, it may not be an issue, but if there are a ton of tables with lots of fields of different data types, you may run into issues.
If you are looking to import table structure, you can copy-paste the content and run inside SSMS in a query window. Beware of syntax differences with MySQL and SQL Server. You will most likely get errors. You need to convert your SQL script from MySQL dialect to SQL Server dialect (or just add them manually if they are not too many). If you set the databases to a SQL standard-compatibility mode at the very beginning, you will have much less trouble.
If you are ONLY looking just to import the data into existing tables inside the SQL Server only, you can do the same (i.e. copy-paste and run in query window). You will have less trouble with that.
Open the server, open "Databases" and right click the database, go to "Tasks" and then Import Data...
I have had the most 'trouble free' success importing to SQL via a flat file method (comma delimited .txt file), the only stipulation when creating a flat file (i.e from Access) make sure the text identifier is set to {none} and not "".
To import the file: in the SQL Server Management Studio right click on Databases and create a new database. Then right click on the new database -> Tasks -> Import Data... The import window opens: in the DATA SOURCE option select Flat File Source and select the .txt file...click NEXT. In the DESTINATION field select SQL Server Native Client 11.0 and go through the import process. This worked very well for me.
I want to export data from a query to excel file. I know that there is a lot of questions like these one here, but no one is acceptable in my situation.
For example, like this topic using OPENROWSET: T-SQL: Export to new Excel file
insert into OPENROWSET('Microsoft.Jet.OLEDB.4.0',
'Excel 8.0;Database=D:\testing.xls;',
'SELECT * FROM [SheetName$]') select * from SQLServerTable
It only execute successfully when I create testing.xls myself, and also define exactly number of columns that will be export from my query in this excel file. Otherwise, an error occur: Column name or number of supplied values does not match table definition.
I also try another solution here: http://weblogs.sqlteam.com/mladenp/archive/2006/07/25/10771.aspx using Exec master..xp_cmdshell
It is really work, but, I heard that xp_cmdshell is a big security threat for SQL Server. So, may be I shouldn't use it.
Is there any other way I can try?
If you're looking for a quick and dirty solution, you can use Management Studio itself.
Here are the steps:
write your query and run it
right click the Results pane and select "Save Results As..."
select your folder/filename and ensure CSV type is selected below
now open your CSV file using Excel; if appropriate save it in Excel as a native Excel format
That won't help you if you need programmatic solution, in which case you have to use Microsoft (Microsoft.Office.Interop.Excel) or 3rd party solutions; you can even build one using Office Open XML.
If you need TSQL solution, OPENROWSET which you are mentioning in the question should be fine.
You can use Excel-only solution and import data using Data pane in the Excel itself (Import from SQL server).
Another possibility taht doesn't have to be coupled to SQL Server or Excel itself is Powershell. However, I'm not into Powershell so if you prefer this method you will have to investigate a bit more.
Also, this is a bit outdated but interesting read.
Try using a newer version of Excel for this. From your query it looks like you’re using Excel 97 – there are probably a ton of bugs and incompatibilities that exist in this version.
If this doesn’t help other options are:
- Do this manually from SSSM like Ozren suggested
- Try creating SSIS package for this
Here are couple other threads to get you started
http://www.connectionstrings.com/excel/
How do you transfer or export SQL Server 2005 data to Excel
Export SQL query data to Excel
is there any method to get data from excel to sql ? i think that we can do without any writing C#codes. For example : select * from MyExcellFile.xls.Sheet1. Or may be any wizard in sql?
See if this page with example of using OPENDATASOURCE helps.
EDIT: Towards the bottom of the page, you will see an example of a query that uses excel as its source.
You can save your EXCEL file as CSV file, and then you can use the following site in order to TRANSFORM the CSV file into a succession of INSERT statements:
http://csv2sql.evandavey.com/
(FREE Online CSV to SQL Converter)
Please be careful to avoid submitting sensitive data to the site, for obvious privacy reasons.
Easiest way is via the SSIS/DTS wizard. Right click on the database in SQL Server, choose Tasks then Import Data. One of the source data options is an Excel spreadsheet. You can them import it into it's own table in SQL Server or map the columns into existing tables. This is the easiest way for a one time upload.