I am able to execute my SSIS via a batch file but I am unable to execute it in Visual Studio 2010. We are using this for our scheduled jobs so it is fine.
I am extracting records from SQL Server and generate to an Excel document. I am using a DTS Configuration File when I execute the batch file thus there are no errors.
However, I would like to run this in Visual Studio as well.
This is because the Excel Connection String in Visual Studio is as follows:
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=xxx.xls;Extended Properties=HDR=YES;EXCEL 8.0;HDR=YES;
May I know why does HDR=YES appears twice and how do I to solve the issue above?
Source
Connecting to Excel
The Microsoft Jet provider is used to connect to an Excel workbook. In the following connection string, the Extended Properties keyword sets properties that are specific to Excel. "HDR=Yes;" indicates that the first row contains column names, not data, and "IMEX=1;" tells the driver to always read "intermixed" data columns as text.
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=D:\MyExcel.xls;Extended Properties=""Excel 8.0;HDR=Yes;IMEX=1""
Note that the double quotation character required for the Extended Properties must also be enclosed in double quotation marks.
Related
Environment:
Microsoft® Excel® for Microsoft 365 MSO (Version 2112 Build 16.0.14729.20254) 64-bit
Microsoft SQL Server 2019 (RTM-CU14) (KB5007182) - 15.0.4188.2 (X64)
Microsoft Visual Studio 2019
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\ClickToRun\REGISTRY\MACHINE\Software\Microsoft\Office\16.0\Access Connectivity Engine\Engines\Excel\TypeGuessRows set to 0
In Excel I formated the cell as "Text" and I also filled up the 2nd and 3rd rows with some super-long dummy text... (1st row is title row)
When I go to Excel Source/Advanced editor, I can set the Output to Unicode text stream [DT_NTEXT] or anything else but the External column cannot be changed to something else than Unicode string [DT_WSTR] (255 characters) despite the setting in registry that normally should allow it and despite having super-long strings in first two rows (other than the 1st row which holds the field names)
Then of course when I try to execute the SSIS task it throws a normal truncation error.
Question: What I am doing wrong or what else should be done here to actually be able to import the data? By the way, this is supposed to be automated at some point.
Since an Excel workbook is not a database, the OLE DB provider tries to detect the most relevant metadata from the Excel worksheet and read it as tabular data, which is mostly inaccurate when handling medium and large Excel files. After spending years on creating SSIS packages, I will convert the Excel file to a CSV file and import it using a Flat File Connection Manager instead. Or I will use a C# script to import the data.
I. Converting Excel to CSV
You can automate the process of converting Excel to CSV using a C# script:
Converting XLSX file using to a CSV file
Convert .xlsx & .xls to .csv
How to Convert Excel to CSV using Interop
After converting the Excel file to a CSV file, you can dynamically import it using a Flat File Connection Manager:
Dynamic Flat File Connections in SQL Server Integration Services
II. Using a C# script
It is good to check the following class, which is a part of the SchemaMapper project:
SchemaMapper - MsExcelImport.cs
Besides, a step-by-step guide on how to use this library can be found in the following link:
Import data from multiple files into one SQL table step by step guide
III. Editing the Excel connection string
If you don't have the choice to convert Excel to flat files, then you can force the Excel connection manager to ignore headers from the first row by adding IMEX=1 to tell the OLEDB provider to specify data types from the first row (which is the header - all string most of the time).
To edit the connectionstring property, click on the Excel Connection Manager and press on the F4 key. In the Properties Tab, you can edit the connectionstring property.
SSIS Excel Import Columns with More or Less than 255 Characters
IV. Changing columns length from advanced editor
Try changing the Excel Source column metadata from the advanced editor:
In SSIS excel datasource not taking more characters than 255
Importing Excel using SSIS may cause a headache! You can check the following question:
Workaround for exporting data to Excel with more than 255 columns
Dynamically Creating Excel table through SSIS
SQL Server Import Wizard doesn't support importing from excel sheet with more than 255 columns
Importing Excel Data Seems to Randomly Give Null Values
Failing to read String value from an excel column
Importing Excel Data Seems to Randomly Give Null Values
SSIS - Excel data shows as scientific notations and Null Values
I'm having an issue exporting a large dataset (500k+) to Excel via SSIS, where the output file ends up with 0 rows exported. Before saying that I shouldn't be exporting that many records to Excel, let me state that I know and normally wouldn't. Accounting does not want a CSV and is unwilling to open a CSV in Excel.
Using Visual Studio 2012 SSDT, here are the components involved.
Execute SQL Task -> Creates the empty file with headers
Data Flow Task ->
OLE DB Source -> SQL Query
Excel Destination
While the package is running, you can see records flowing from the source to the destination. The package completes without error, but when you open the file, it's empty. The only thing in there is the header.
If I select the Top 1000 records and export to Excel, it works as intended.
Some things I've tried:
Export to Excel on the network
Export to Excel locally
Export to CSV to Excel on both network and locally
Export to Ole DB Destination using Office Access Database Engine 12.0 with "Excel 12.0" extended properties.
Tried running as different users
All with the same outcome.
Can anyone provide any insight into why this may be happening and how to proceed?
We experienced a similar behaviour, when runnig the ETL in a SQL Server Agent job. Debugging it in Visual Studio, worked, however. So I do not know, whether this solution applies to you.
The reason was that the user, under which the package ran, did not have access to C:\users\Default.
I found this out by using sysinternals process monitor.
I was inspired by that post: Empty Excel File permissions issue: SSIS Excel Destination buffers large record sets through C:\Users\Default
[I explained my search for the bug in my blog: https://www.csopro.de/biblog/2018/04/ssis-fehlerbehebung-bei-excel-destination-schreibt-keine-zeilen/ Unfortunately it is in German]
I have an simple SSIS package and I'm trying to export same set of data from a table to both flat file and excel destination. The package works fine when I run locally and it creates both text file and excel file with data.
But when deployed to a different server the sql agent job runs fine and the log inside integrations services catalog for the package says it wrote like 9000 rows to excel, and a new excel file is also created but it doesn't write any data to it(blank with just headers). text file works fine and it has all data I need.
SSIS package flow:
I'm working with Sql server 2014, Visual studio 2013 with SSDT and used Excel 2007 in excel destination.
We had the same issue.
The solution is that the user, which runs the SSIS package, must have full access to c:\users\default.
You can check this by running sysinternals' process monitor on the machine that executes the SSIS job.
You can find more information here:
Empty Excel File permissions issue: SSIS Excel Destination buffers large record sets through C:\Users\Default - This post made me find this solution
https://www.csopro.de/biblog/2018/04/ssis-fehlerbehebung-bei-excel-destination-schreibt-keine-zeilen/ - my blog. Here I describe the issue - unfortunately in German]
I had the same problem writing to several worksheets in an Excel file from a scheduled SQL Agent job. It worked fine for about 4 months. Then suddenly with no changes to the package, one of the 5 worksheets was no longer populated with data. No error message generated and it worked fine on every test from Visual Studio and Data Tools (the old "BIDS" tools as we used to call it.)
I never did find a solution and it continues to not write any data to that single worksheet of the 5 in the Excel file. (So answers above about the Account that the job runs under from SQL Agent does not have the appropriate permissions is NOT a correct answer for this issue.)
Plus, a new package I built today is having the same issue, only this one has only a single worksheet. Again, works fine in the development environment, but no data appears in the destination file and no errors. Not only that, but the timestamp on the file is the same as the template file -- it seems that it never even TRIES to write to the file.
Checking each run log for the package in the Integration Services Catalog has an entry in each log that shows 9K+ records "written" for the dataflow task.
Lastly, if I change the destination file name, the SQL Agent job generates the expected error, so that rules out answers that guess that the path is wrong.
This is bizarre. And exasperating.
I have encountered odd behaviour when using scheduled SSIS packages which use the Excel object.
The fix for me, was to edit the Agent Job properties. On the Execution Properties tab, try enabling the "use 32-bit runtime" option and force the SSIS to run in 32-bit mode instead of 64-bit mode.
I have searched everywhere for an answer to this so I am hoping someone out there can help.
I am trying to follow the steps laid out for importing data from multiple excel files in SSIS. I have several excel 2010 files in a directory and am trying to move them into a SQL Server 2008 r2 Database. I have followed all of the directions for doing this with a For Each (File) Loop. I have set the collections information and am using the User::Filename variable to pull back the Fully Qualified filename. The problem I have is that when I check the properties of my excel connection manager-->click on expressions-->modify excelfilepath by setting it to #[User::Filename], SSIS immediately overwrites the property with the new value which is blank (the first time). This then causes my package to fail during execution because there is no filename to go to in the connection.
I have set the delayvalidation property to True on the package, the dataflow and the excel connection and this does not fix the problem. I have also tried to put a fully qualified filename into the User::Filename variable during initialization and this does allow me to process a few files but then gives me a locking error when SSIS tries to re-read the file that I put in during initialization.
I have not found anything like this on the net yet...Hopefully, someone out there has seen this.
I would split this SSIS package into two. The master package would go as deep as the For Each File Loop, but would not include the Excel Connection Manager. The sub-package would open and process each Excel file. It would receive the Filename variable value via Configuration for a Parent Package Variable or similar.
This seems to have the effect of "closing" the excel connection after each file is finished.
I also prefer the OLE DB driver from the Access Database Engine over the native SSIS Excel connection.
I have a folder with a .sql files; 1 file per query. I want to execute all queries/ sql files and save them as csv.
Is there a way to do that automated without using the windows cli (disabled in my environment). I do have the SQL Server Management Studio.
I would approach this task using SSIS, providing you have Business Intelligence Development Studio (BIDS) installed.
First create a 'Foreach Loop Container' pointed to the folder with the SQL files, then use a variable to retreive each file name.
Next, create a flat file connection and set the 'Connection String' property to the variable that contains the file location.
Next, using the 'Execute SQL Task' component set the 'SQLSourceType' to 'File Connection' and the 'FileConnection' to the one created in the previous step.
Finally, depending on how the data is returned you have a couple of options, if the result set is small, only a row or a single column, then you can save the results to a variable and using a 'Dataflow' task create a 'Derived Column' component and export the contents of that variable to a CSV file. Or, if the dataset is larger you could dump the results to a temp table and then using an 'OLE DB Source' and 'OLE DB Destination' you could push the full result set straight into a CSV.
Hopefully this isn't too convoluted of a solution, this approach has the advantage of being able be run from either a remote machine or from the server itself, plus you can automate its execution with a SQL Agent Job.
Create a VB.NET console application.
Generate a list of files that end in .SQL from the folder in question.
Load the contents of each file into individual SQL Commands
Execute the SQL Command for each, storing the results in DataSets.
For each table in each dataset, create a new .csv file
For each .csv file, you will need to iterate over each cell in the datatable, and utilize proper escaping for .csv files.
Use 'for' in combination with either sqlcmd or bcp command for each file in the script folder.