I am trying to export data from SQL Server to Excel and then upload that data into another SQL Server instance.
While the export looks like it's working - no errors or warnings - I get an Excel file where all the data is "general" format.
When uploading the Excel to the new server, all columns are of type nvarchar, no dates floats int etc.
Does anyone have an idea why this is happening?
Related
Environment:
Microsoft® Excel® for Microsoft 365 MSO (Version 2112 Build 16.0.14729.20254) 64-bit
Microsoft SQL Server 2019 (RTM-CU14) (KB5007182) - 15.0.4188.2 (X64)
Microsoft Visual Studio 2019
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\ClickToRun\REGISTRY\MACHINE\Software\Microsoft\Office\16.0\Access Connectivity Engine\Engines\Excel\TypeGuessRows set to 0
In Excel I formated the cell as "Text" and I also filled up the 2nd and 3rd rows with some super-long dummy text... (1st row is title row)
When I go to Excel Source/Advanced editor, I can set the Output to Unicode text stream [DT_NTEXT] or anything else but the External column cannot be changed to something else than Unicode string [DT_WSTR] (255 characters) despite the setting in registry that normally should allow it and despite having super-long strings in first two rows (other than the 1st row which holds the field names)
Then of course when I try to execute the SSIS task it throws a normal truncation error.
Question: What I am doing wrong or what else should be done here to actually be able to import the data? By the way, this is supposed to be automated at some point.
Since an Excel workbook is not a database, the OLE DB provider tries to detect the most relevant metadata from the Excel worksheet and read it as tabular data, which is mostly inaccurate when handling medium and large Excel files. After spending years on creating SSIS packages, I will convert the Excel file to a CSV file and import it using a Flat File Connection Manager instead. Or I will use a C# script to import the data.
I. Converting Excel to CSV
You can automate the process of converting Excel to CSV using a C# script:
Converting XLSX file using to a CSV file
Convert .xlsx & .xls to .csv
How to Convert Excel to CSV using Interop
After converting the Excel file to a CSV file, you can dynamically import it using a Flat File Connection Manager:
Dynamic Flat File Connections in SQL Server Integration Services
II. Using a C# script
It is good to check the following class, which is a part of the SchemaMapper project:
SchemaMapper - MsExcelImport.cs
Besides, a step-by-step guide on how to use this library can be found in the following link:
Import data from multiple files into one SQL table step by step guide
III. Editing the Excel connection string
If you don't have the choice to convert Excel to flat files, then you can force the Excel connection manager to ignore headers from the first row by adding IMEX=1 to tell the OLEDB provider to specify data types from the first row (which is the header - all string most of the time).
To edit the connectionstring property, click on the Excel Connection Manager and press on the F4 key. In the Properties Tab, you can edit the connectionstring property.
SSIS Excel Import Columns with More or Less than 255 Characters
IV. Changing columns length from advanced editor
Try changing the Excel Source column metadata from the advanced editor:
In SSIS excel datasource not taking more characters than 255
Importing Excel using SSIS may cause a headache! You can check the following question:
Workaround for exporting data to Excel with more than 255 columns
Dynamically Creating Excel table through SSIS
SQL Server Import Wizard doesn't support importing from excel sheet with more than 255 columns
Importing Excel Data Seems to Randomly Give Null Values
Failing to read String value from an excel column
Importing Excel Data Seems to Randomly Give Null Values
SSIS - Excel data shows as scientific notations and Null Values
I'm having an issue exporting a large dataset (500k+) to Excel via SSIS, where the output file ends up with 0 rows exported. Before saying that I shouldn't be exporting that many records to Excel, let me state that I know and normally wouldn't. Accounting does not want a CSV and is unwilling to open a CSV in Excel.
Using Visual Studio 2012 SSDT, here are the components involved.
Execute SQL Task -> Creates the empty file with headers
Data Flow Task ->
OLE DB Source -> SQL Query
Excel Destination
While the package is running, you can see records flowing from the source to the destination. The package completes without error, but when you open the file, it's empty. The only thing in there is the header.
If I select the Top 1000 records and export to Excel, it works as intended.
Some things I've tried:
Export to Excel on the network
Export to Excel locally
Export to CSV to Excel on both network and locally
Export to Ole DB Destination using Office Access Database Engine 12.0 with "Excel 12.0" extended properties.
Tried running as different users
All with the same outcome.
Can anyone provide any insight into why this may be happening and how to proceed?
We experienced a similar behaviour, when runnig the ETL in a SQL Server Agent job. Debugging it in Visual Studio, worked, however. So I do not know, whether this solution applies to you.
The reason was that the user, under which the package ran, did not have access to C:\users\Default.
I found this out by using sysinternals process monitor.
I was inspired by that post: Empty Excel File permissions issue: SSIS Excel Destination buffers large record sets through C:\Users\Default
[I explained my search for the bug in my blog: https://www.csopro.de/biblog/2018/04/ssis-fehlerbehebung-bei-excel-destination-schreibt-keine-zeilen/ Unfortunately it is in German]
I'm trying to import a table from a txt file comma separated, the file was generated by the same SQL Server import/export assistant from my development database server.
Many tables has been imported by this way successfully.
The problem is when I have a column of the type date.
The message is: conversion unknown!
And then I cannot exec the package.
How can I solve this problem?
I have a CSV file with 10 columns. I want to import to SQL Server table with only 5 columns using powershell. Can anyone help in this ?
I suggest that Change your excel file to 5 sight column, and then using SQL Server Import\Export wizard or SSIS (if your convert is each day periodic) or PowerShell to convert your data.
You can you following reference in order to use PowerShell to convert your file:
SQL SERVER – Powershell – Importing CSV File Into Database
So I exported an sql database out as an excel file so that I could send it off to be edited by hand. I used the Microsoft SQL Server Management Studio then the SQL Server Import and Export Wizard to do so. Now the edits are finished and when I go to reimport the file I get
Found 7 unknown column type conversion(s)
You are only allowed to save the package
And it wont let me run the import. It seems to me as though the export to excel converted all int and bit datatypes into Double datatypes and It doesn't know how to convert back. I did a bit of googling and I found the SSIS10ToMSSQL.XML file (which appears to control how the conversion takes place), but I don't know how I should edit it to tell the wizard how to convert correctly.
Any input on this problem would be appreciated.
After WAYYYYYY to many hours spent trying other formats and settings, I just bit the bullet and wrote a console app to read in a csv (exported from excel) and run the necessary SQL on the database.