I have multiple Excel files that have the same format. I need to import them into SQL Server.
The issue I currently have is that there are two text columns that I need to ignore completely as they are free text and the character length for some rows exceeds what the server allows me to import which results in a truncation error.
Because I don't need these columns for my analysis, the table I'm importing to doesn't include these columns but for some reason the SSIS packages still picks up those columns and cuts the import job halfway through.
I tried using max character length for those columns which still results in the truncation error.
I need to create an SSIS package that ignores the two columns completely without deleting the columns from Excel.
You can specify which columns you need to ignore from the Edit Mappings dialog.
I have added the image for your reference:
If you just create the SSIS package in SSDT the Excel file can be queried to return only the required columns. In the package, create an Excel Connection Manager using the Excel file. Then on the Control Flow of the package add a Data Flow Task that has an Excel Source component in it. On this source, change the data access mode to SQL command and the file can then be queried similar to SQL. In the following example TabName is the name of the Excel tab containing the data that will be returned. If either the tab or any column names contain spaces they will need to be enclosed in square brackets, i.e. TabName would be [Tab Name].
Import/Export Wizard
Since you mentioned in the comments that you are using SQL Server Import/Export Wizard. You can solve that if you have a fixed columns (range) that you are looking to import (example: first 10 columns).
In Import/Export wizard, after selecting destination options you will be asked if you want to read from tables or query:
Select the query option, then use a simple select query and specify the columns range after the sheet name. As example:
SELECT * FROM [Sheet1$A:C]
The query above will read from the first 3 columns in Sheet1 since A:C represent the range between first column A and third column C.
Now, you can check the columns from the Edit Mappings dialog:
SSIS
You can use the same logic within SSIS package, just write the same SQL command in the Excel Source after changing the Access Mode to SQL Command.
The solution is simple. I needed to write a query that will exclude the columns. So instead of selecting "Copy data from one or more tables" you select "write a query" and exclude the columns you don't need. This one worked 100%
Related
I have created a Pivot table using columns passed in from a variable. This variable is set based on a flag in another table, so the name and number of columns can change month-to-month. I need to be able to export this pivot to a delimited text file. The pivot table is stored in a temp table ##PivotTable
I have tried to do this via SSMS, but the only way I know how to do this is to define a file connection manager, which needs set columns. I also tried via sqlcmd, but that one exports extra information and spaces.
I am open to pretty much any process that would allow me to run this via a SQL Agent job on a monthly basis.
Any assistance would be much appreciated.
You can use SQL server import and export wizard
Right click on the database-->Tasks-->ExportData..
map your ##PivotTable.
Click next and set your source database (Select OLEDB provider for SQL Server)
Set a file name for your file (e.g. d:\monthlyData\202006\MyFile.csv
Select option copy data from one to more tables or views -- do not use this option due the fact that your columns getting changed.
Select option write a quarry
You can run this immediately or save as SSIS package and reuse it using SQL job
I think you'll have to use dynamic SQL like found here.
I have an SSIS package with an Excel Source reads an Excel table. I currently am using the Table or View Data Access Mode and it is literally reading every row in the worksheet, 1,048,576 which is the maximum.
The source worksheet has an Excel table on it named PSA_DATA. Why isn't this table in the Table or View drop down? There is an option for the worksheet followed by _FilterDatabase but this fails when I run the package even though it pulls the correct data when I press Preview. Wouldn't this make more sense than using the SQL Command and SELECT * FROM [fact_PSA$Ax:Bx]? The whole reason we use Named Ranges and Tables in Excel is because they are dynamic! Now I have to hard code the range in every time with rows numbers?
What am I missing here? Is there an easier way I am missing? I just want to move an Excel table into a SQL table! Why don't doesn't the most ubiquitous piece of software in the world easily talk to the second most ubiquitous piece of software in the world!?!?!
If the sheet name is not shown in Table or view combobox, it is not a bad idea to use a Sql Command.
But When using SQL Comand to read from excel it is not necessary to specify a range, OLEDB will take used range by default just use the following command
SELECT * FROM [fact_PSA$]
Workaround
you can try reading your excel file from a script task or a script component, you can follow one of the following links to achieve this:
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/2d45f180-9fd0-4224-a298-cb99e2b2100a/how-to-read-the-contents-of-excel-file-through-ssis-script-task-without-the-headers?forum=sqlintegrationservices
https://msdn.microsoft.com/en-us/library/ms403358.aspx
http://billfellows.blogspot.com/2013/04/ssis-excel-source-via-script.html
Side Note: there are many links you can follow to import data from excel to SQL using SSIS:
http://www.sqlshack.com/using-ssis-packages-import-ms-excel-data-database/
https://www.mssqltips.com/sqlservertip/2770/importing-data-from-excel-using-ssis--part-1/
https://www.simple-talk.com/sql/ssis/moving-data-from-excel-to-sql-server-10-steps-to-follow/
https://www.simple-talk.com/sql/ssis/importing-excel-data-into-sql-server-via-ssis-questions-you-were-too-shy-to-ask/
I appreciate the links to work-arounds, but I didn't really get an answer to my question. Why can't we reference an EXCEL TABLE (not a worksheet) from the SSIS Excel Source???
I ended up using the SQL Command data access mode with this query:
SELECT * FROM [fact_PSA$A:W]
WHERE fact_PSA_ID IS NOT NULL
Somehow, using SQL stopped it from reading every possible row in the worksheet even though the range provided is set for "A:W" which is every row. I guess the "WHERE fact_PSA_ID" limits the rows read before it hits the SSIS source.
I am trying to import an excel sheet to SQL database. This sheet has 700+ columns, but i understand there is a limit of 255 columns. Is there a work around to include all the columns while uploading to database. I selected Excel 2007 while selecting excel version.
Sadly, of the resources I saw online, there wasn’t an easy way to do this. I tried the above solution of choosing “Excel 2007” but that didn’t work for me. One would expect Excel and SQL Server to have tighter integration.
However, by converting to a .txt file and then working through some of the truncation errors, I managed to load my data set. Below is some detail of my use case and process.
I had a fairly small dataset row-wise (~200 rows) but with a large set of columns (+500 columns). I first converted the Excel file to a text, 'tab-delimited' file (i.e. *.txt). In loading through the SQL Server Import and Export Wizard, I faced two truncation errors: one, a few of the column names were greater than 128 characters and two, the length of values in some of the rows where the default datatype was DT_STR was greater than 50 (the default output column width). For the first error, I just renamed the column(s) to something shorter. For the second error, I manually ran a count of lengths and found the maximum length of values for each column, which allowed me to isolate which columns would throw up an error. I followed the steps below:
1) In the SQL Server Import & Export Wizard 'General' Tab, I selected 'Flat File source' and accepted all the normal defaults.
2) In the second tab ('Columns'), ensure column delimiter is selected as Tab. I preferred this over a CSV format since there was some heavy text with commas in my dataset.
3) If the mapping works out, in the third tab ('Advanced'), you should get a laundry list of all your columns with their natural defaults. As detailed above, I isolated which columns had values that exceeded the default of DT_STR (50) and changed that to DT_TEXT.
4) The remaining steps just specify the destination, and whether you want to save the SSIS steps.
A more simple, straightforward solution: on excel, sort your rows so the one with the longest value is moved to the top. It looks like the import wizard only looks into the first 8 rows of your data file to determine the width of each column (weird!).
source: How does one change the default varchar 255 of a column when importing data from Excel to Sql Server using Import Export Wizard?
I have a few experience in using SQL Server 2012.
All I know to import a excel to database is like the following:
open SQL Server Management Studio
right click on the "table" folder -> Tasks -> Import Data
set data source to MS Excel.
It seems that only one Excel is entertained.
But I want to concatenate 6 Excel files (all with same column layouts) to form a single table in SQL Server.
P.S. No need to tell me to concatenate the Excel file manually by copy and paste because each individual Excel file has about ~50,000 records.
Any ideas / solutions by using sql scripts or any other programming methods?
Thanks a lot.
There's a range of ways to do this, but I'll give you the simplest that comes to mind without requiring any deep technical knowledge on your part.
Given that you're using the wizard, firstly on the 'Select Table Sources and Views' page, change the 'Destination' to be the name of the table you've previously created.
Then, under the 'Edit Mappings' menu when selecting your sheets, ensure you have 'Append rows to the destination table' selected, rather than Create/Delete. Within reason, this will achieve your goal.
There is a risk in flat file loading like this that SQL Server will create your table with unsuitable types (i.e. a Column is a text column, but only contained numbers on the first file - so the column was created as an INT and wont accept any other files). You'll need to create the tables from scratch again with the right structure or work with the mappings page to do this work.
Another way for the semi-technical type is as long as the data is equivelent between files, you can simply do your imports into a series of seperate tables:
Table1
Table2
Table3
...
Then do a
INSERT INTO Table1
SELECT * FROM Table2
UNION ALL
SELECT * FROM Table3
... Add tables here
You can then use DROP TABLE to remove the extras.
I am trying to export data from SQL server 2008 to Excel file using BIDS.
One of the fields 'DESCRIPTION' coming from SQL database is VARCHAR(4000).
I can export everything to excel but the 'DESCRIPTION' field size in excel is restricted to unicode 255 and no mater what I try it does not allow me to export the data over 255 characters (exports it as blank). I tried to change SQL field as varchar(max) or ntext but none of attempts worked. I used advanced editor in BIDS on excel destination to change 'DESCRIPTION' character length manually but as soon as I hit 'OK', it resets to unicode 255.
Could anybody please help me to resolve this issue?
Thanks,
Vishal
So, I did some testing. Excel data transformation is funky but I came up with a solution. I created an excel spreadsheet with fields as needed. I then created fake, dummy data in excel with character length far greater than 255 and hid the row. I then did the SSIS data transformation to the excel spreadsheet which worked. It's a weird and not preferable option but it works.
Problem: Excel only accepts 255 chars per cell when I attempt to use Excel Destination in SSIS (2008 R2) from a sql server table. SalesForce data loader would not accept CSV (with “” text qualifiers) created by
ssis flat file connection manager. SalesForce will only accept CSV (with “” text qualifiers). SalesForce will accept CSV as exported by Excel (2010).
Solution:
1. Create your excel connection manager, set name/path of the destination EXCEL file in your “Excel Destination Data Flow Competent” and map meta-data.
2. Open a new Excel file, remove all extra “sheets”, rename “sheet1” to that was created in step#1, above, select all cells and format to “text”, add all the column header names to the first row of your template sheet. In the columns that need to hold more data than 255 limit, paste in any characters that exceed your limit by 50% (just in case). These columns are now configured to hold your large data. Save the file, naming it something like TEMPLATE_Excel_forLargeCellValues.xlsx
3. Copy this template into your DESTINATION connection: Before your “Excel Destination Data Flow Competent” in the SSIS Control Flow, create a new “File System Task”. Create an ssis pkg level variable to hold the path/filename of your template excel file. In your “File System Task” set “IsSourcePathVariable” = TRUE, set “SourceVariable” to User::Template_Excel. Set “IsDestinationPathVariable” = FALSE, and set “DestinationConnection” = from step #1 above. Set “Operation” = Copy file. “OverwriteDestination”=TRUE. This will now copy your formatted Excel workbook/sheet into your destination folder with the file name you designated in step #1 above and because you put a larger amount of sample data in the columns that require more than 255 chars, all your data will fit.
Note: It is not necessary to delay validation on any components.
You're saying that the excel field is set to 255 right? Changing the SQL field won't have an effect on excel, you'd have to modify the excel file.
I don't believe you can modify the Excel output column to write more than 255 characters. Why not simply write your output to a csv, it can be opened and later modified in Excel anyway.
SSIS excel engine recognizes datatype of first 8 rows and assigns it to excel source or destination automatically. Even defining the excel column as memo wont work. I tried to resolve the error by changing registry value TypeGuessRows of excel engine but it did not work either. So I was not left with any other option but to create a dummy row(2nd row) with more than 255 characters and hide it.Excel source then identify the column with unicode text stream. You have to write some logic in SSIS package to exclude this row if you are trying to import the data from excel. I heard that this issue is resolved in excel versions on and after 2010. But BIDS 2008 does not have option to choose any version after 2007 so this is the only solution if you are working with BIDS 2008 and excel.
You have to select Microsoft Excel 97-2003 and use the xls as file extension in your file name for destination.
I got the same issue of the excel destination not allowing more than 255 characters. After spending almost a day, I tried adding more characters (to simplify, I added spaces more than 255) in the header of the column that has the issue with more than 255 characters. And it magically worked!
You can insert dummy data (260 characters) to under head column you want in your excel (Execute SQL Task)
Script Create and insert
CREATE TABLE `YourSheet` (`myColumn260char` LongText)
GO
INSERT INTO YourSheet(myColumn260char) Values('....................................................................................................................................................................................................................................................................')
And you can delete dummy row after imported.