I have an SSIS package that exports data to a couple of Excel files for transfer to a third party. To get this to run as a scheduled job on a 64-bit server I understand that I need to set the step as a CmdExec type and call the 32-bit version of DTExec. But I don't seem to be able to get the command right to pass in the connection string for the Excel files.
So far I have this:
DTExec.exe /SQL \PackageName /SERVER OUR2005SQLSERVER /CONNECTION
LETTER_Excel_File;\""Provider=Microsoft.Jet.OLEDB.4.0";"Data
Source=""C:\Temp\BaseFiles\LETTER.xls";"Extended Properties=
""Excel 8.0;HDR=Yes"" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING E
This gives me the error: Option "Properties=Excel 8.0;HDR=Yes" is not valid.
I've tried a few variations with the Quotation marks but have not been able to get it right yet.
Does anyone know how to fix this?
UPDATE:
Thanks for your help but I've decided to go with CSV files for now, as they seem to just work on the 64-bit version.
This step-by-step example is for others who might stumble upon this question. This example uses SSIS 2005 and uses SQL Server 2005 64-bit edition server to run the job.
The answer here concentrates only on fixing the error message mentioned in the question. The example will demonstrate the steps to recreate the issue and also the cause of the issue followed by how to fix it.
NOTE: I would recommend using the option of storing the package configuration values in database or using indirect XML configuration with the help of Environment Variables. Also, the steps to create Excel file would be done using a template which would then archived by moving to a different folder. These steps are not discussed in this post. As mentioned earlier, the purpose of this post is to address the error.
Let’s proceed with the example. I have also blogged about this answer, which can be found in this link. It is the same answer.
Create an SSIS package (Steps to create an SSIS package). This example uses BIDS 2005. I have named the package in the format YYYYMMDD_hhmm in the beginning followed by SO stands for Stack Overflow, followed by the SO question id, and finally a description. I am not saying that you should name your package like this. This is for me to easily refer this back later. Note that I also have a Data Sources named Adventure Works. I will be using Adventure Works data source, which points to AdventureWorks database downloaded from this link. The example uses SQL Server 2008 R2 database. Refer screenshot #1.
In the AdventureWorks database, create a stored procedure named dbo.GetCurrency using the below given script.
CREATE PROCEDURE [dbo].[GetCurrency]
AS
BEGIN
SET NOCOUNT ON;
SELECT
TOP 10 CurrencyCode
, Name
, ModifiedDate
FROM Sales.Currency
ORDER BY CurrencyCode
END
GO
On the package’s Connection Manager section, right-click and select New Connection From Data Source. On the Select Data Source dialog, select Adventure Works and click OK. You should now see the Adventure Works data source under the Connection Managers section.
On the package’s Connection Managers section, right-click again but this time select New Connection…. This is to create the Excel connection. On the Add SSIS Connection Manager, select EXCEL. On the Excel Connection Manager, enter the path C:\Temp\Template.xls. When we deploy it to the server, we will change this path. I have selected Excel version Microsoft Excel 97-2005 and chose to leave the checkbox First row has column names checked so that the create the Excel file is created column headers. Click OK. Rename the Excel connection to Excel, just to keep it simple. Refer screenshots #2 - #7.
On the package, create the following variable. Refer screenshot #8.
SQLGetData: This variable is of type String. This will contain the Stored Procedure execution statement. This example uses the value EXEC dbo.GetCurrency
Screenshot #9 shows the output of the stored procedure execution statement EXEC dbo.GetCurrency
On the package’s Control Flow tab, place a Data Flow task and name it as Export to Excel. Refer screenshot #10.
Double-click on the Data Flow Task to switch to the Data Flow tab.
On the Data Flow tab, place an OLE DB Source to connect to the SQL Server data to fetch the data from the stored procedure and name it as SQL. Double-click on the OLE DB Source to bring up the OLE DB Source Editor. On the Connection Manager section, select Adventure Works from the OLE DB connection manager, select SQL command from variable from Data access mode and select the variable User::SQLGetData from the Variable name drop down. On the Columns section, make sure the column names are mapped correctly. Click OK to close the OLE DB Source Editor. Refer screenshots #11 and #12.
On the Data Flow tab, place an Excel Destination to insert the data into the Excel file and name it as Excel. Double-click on the Excel Destination to open the Excel Destination Editor. On the Connection Manager section, select Excel from the OLE DB connection manager and select Table or view from Data access mode. At this point, we don’t have an Excel because while creating the Excel connection manager, we simply specified the path but never created the file. Hence, there won’t be any values in the drop down Name of the Excel sheet. So, click the New… button (the second New one) to create a new Excel sheet. On the Create Table window, BIDS automatically provide a create sheet based on the incoming data source. You can change the values according to your preferences. I will simply click OK by retaining the default value. The name of the sheet will be populated in the drop down Name of the Excel sheet. The name of the sheet is taken from the task name, here in this case the Excel Destination, which we have named it as Excel. On the Mappings section, make sure the column names are mapped correctly. Click OK to close the Excel Destination Editor. Refer screenshots #13 - #16.
Once the data flow task is configured, it should look like as shown in screenshot #17.
Execute the package by pressing F5. Screenshots #18 - #21 show the successful execution of the package in both Control Flow and Data Flow Task. Also, the file is generated in the path C:\Temp\Template.xls provided in the Excel connection and the data shown in the stored procedure execution output matches with the data written to the file.
The package developed on my local machine in the folder path C:\Learn\Learn.VS2005\Learn.SSIS. Now, we need to deploy the files on to the Server that hosts the 64-bit version of the SQL Server to schedule a job. So, the folder on the server would be D:\SSIS\Practice. Copy the package file (.dtsx) from the local machine and paste it in the server folder. Also, in order for the package to run correctly, we need to have the Excel spreadsheet present on the server. Otherwise, the validation will fail. Usually, I create a Template folder that will contain the empty Excel spreadsheet file that matches the output. Later, during run time I will change the Excel output path to a different location using package configuration. For this example, I am going to keep it simple. So, let’s copy the Excel file generated in the local machine in the path C:\Temp\Template.xls to the server location D:\SSIS\Practice. I want the SQL job to generate the file in the name Currencies.xls. So, rename the file Template.xls to Currencies.xls. Refer screenshot #22.
To show that I am indeed going to run the job on the server in a 64-bit edition of SQL Server, I executed the command SELECT ##version on the SQL Server and screenshot #23 shows the results.
We will use Execute Package Utility (dtexec.exe) to generate the command line parameters. Log into the server which will run the SSIS package in an SQL job. Double-click on the package file, this will bring the Execute Package Utility. On the General section, select File system from Package source. Click on the Ellipsis and browse to the package path. On the Connection Managers section, select Excel and change the path inside the Excel file from C:\Temp\Template.xls to D:\SSIS\Practice\Currencies.xls. The changes made in the Utility will generate a command line accordingly on the Command Line section. On the Command Line section, copy the Command line that contains all the necessary parameters. We are not going to execute the package from here. Click Close. Refer screenshots #24 - #26.
Next, we need to set up a job to run the SSIS package. We cannot choose SQL Server Integration Services Package type because that will run under 64-bit and won’t find the Excel connection provider. So, we have to run it as Operating System (CmdExec) job type. Go to SQL Server Management Studio and connect to the Database Engine. Expand SQL Server Agent and right-click on Jobs node. Select New Job…. On the General section of the Job Properties window, provide the job name as 01_SSIS_Export_To_Excel, Owner will be the user creating the job. I have a Category named SSIS, so I will select that but the default category is [Uncategorized (Local)] and provide a brief description. On the Steps section, click New… button. This will bring Job Step properties. On the General section of the Job Step properties, provide Step name as Export to Excel, Select type Operating system (CmdExec), leave the default Run as account as SQL Server Agent Service Account and provide the following Command. Click OK. On the New Job window, Click OK. Refer screenshots #27 - #31.
C:\Program Files (x86)\Microsoft SQL Server\90\DTS\Binn\DTExec.exe /FILE
"D:\SSIS\Practice\20110723_1015_SO_21448_Excel_64_bit_Error.dtsx"
/CONNECTION Excel;"\"Provider=Microsoft.Jet.OLEDB.4.0;Data
Source=D:\SSIS\Practice\Currencies.xls;Extended Properties=""EXCEL 8.0;HDR=YES"";\""
/MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EWCDI
The new job should appear under SQL Server Agent –> Jobs node. Right-click on the newly created job 01_SSIS_Export_To_Excel and select Start Job at Step…, this will commence the job execution. The job will fail as expected because that is the context of this issue. Click Close to close the Start Jobs dialog. Refer screenshots #32 and #33.
Let’s take a look at what happened. Go to SQL Server Agent and Jobs node. Right-click on the job 01_SSIS_Export_To_Excel and select View History. This will bring the Log File Viewer window. You can notice that the job failed. Expand the node near the red cross and click on the line that Step ID value of 1. At the bottom section, you can see the error message Option “8.0;HDR=YES’;” is not valid. Click Close to close the Log File Viewer window. Refer screenshots #34 and #35.
Now, right-click on the job and select Properties to open the Job Properties. You can also double-click on the job to bring the Job Properties window. Click on the Steps on the left section. and click Edit. Replace the command with the following command and click OK. Click OK on the Job Properties to close the window. Right-click on the job 01_SSIS_Export_To_Excel and select Start Job at Step…, this will commence the job execution. The job will fail execute successfully. Click Close to close the Start Jobs dialog. Let’s take a look at the history. Right-click on the job 01_SSIS_Export_To_Excel and select View History. This will bring the Log File Viewer window. You can notice that the job succeeded during the second run. Expand the node near the green tick cross and click on the line that Step ID value of 1. At the bottom section, you can see the message Option The step succeeded. Click Close to close the Log File Viewer window. The file D:\SSIS\Practice\Currencies.xls will be successfully populated with the data. If you execute the job successfully multiple times, the data will get appended to the file and you will find more data. As I mentioned earlier, this is not the right-way to generate the files. This example was created to demonstrate a fix for this issue. Refer screenshots #36 - #38.
Screenshot #39 shows the differences between the working and the non-working command line arguments. The one on the right is the working command line and the left one is incorrect. It required another double quotes with backslash escape sequence to fix the error. There could be other ways to fix this well but this option seems to work.
Thus, the example demonstrated a way to fix the command line argument issue while accessing Excel data source from an SSIS package that is deployed on a 64-bit server.
Hope that helps someone.
Screenshots:
#1: Solution_Explorer
#2: New_Connection_Data_Source
#3: Select_Data_Source
#4: New_Connection
#5: Add_SSIS_Connection_Manager
#6: Excel_Connection_Manager
#7: Connection_Managers
#8: Variables
#9: Stored_Procedure_Output
#10: Control_Flow
#11: OLE_DB_Source_Connections_Manager
#12: OLE_DB_Source_Columns
#13: Excel_Destination_Editor_New
#14: Excel_Destination_Create_Table
#15: Excel_Destination_Edito
#16: Excel_Destination_Mappings
#17: Data_Flow
#18: Successful_Package_Execution_Control
#19: Successful_Package_Execution_Data_Flow
#20: C_Temp_File_Created
#21: Data_Populated
#22: File_On_Server
#23: SQL_Server_Version
#24: Execute_Package_Utility_General
#25: Execute_Package_Utility_Connection_Managers
#26: Execute_Package_Utility_Command_Line
#27: Job_New_Job
#28: New_Job_General
#29: New_Job_Step
#30: New_Job_Step_General
#31: New_Job_Steps_Added
#32: Job_Start_Job_at_Step
#33: SQL_Job_Execution_Failure
#34: View_History
#35: SQL_Job_Error_Message
#36: SQL_Job_Execution_Success
#37: SQL_Job_Success_Message
#38: Excel_File_Generated
#39: Command_Comparison
You can use an Excel connection in 64bit environment.
Go to the package configuration properties.
Debugging -> Debugging Options -> Run64BtRuntime -> change to False
In addition if you use SQL Agent go to the job step properties and then check the 32 bit runtime.
note: this only applies to debugging within Visual Studio...
There is no 64-bit Jet OLEDB provider, so you can't access Excel files from 64-bit SSIS.
However, you can use 32-bit SSIS even on 64-bit server. It is already installed when you installed 64-bit version, and all you need to do is run the 32-bit DTEXEC.EXE - the one installed Program Files (x86)\Microsoft Sql Server\90\Dts\Binn (replace 90 with 100 if you are using SSIS 2008).
Unless it's a business requirement, I suggest you move the connection string from the command line to the package and use a package configuration to define the path to the Excel file (in order not to hard-code it). This will make it easier to maintain.
Define a variable #ExcelPath.
Use connection's Expression property to construct a connection string - an example: "Data Source=" + #[User::FilePath] + ";Provider=Microsoft.Jet.OLEDB.4.0;Extended Properties=dBASE IV;"
Assign a value to #ExcelPath in the package configuration.
Take a closer look at the connection string above. It's taken from a working package. I'm not sure about this, but maybe you don't need any quotes at all (the ones above are only there because the expression editor requires them).
I have also had some problems with SSIS on 64-bit SQL Server 2005. That post from my blog does not answer your question, but it is somewhat related so I am posting the link.
I kinda did what Dr Zim did but I copied the DTExec file C:\Program Files (x86)\Microsoft SQL Server\90\DTS\Binn\DTExec.exe to C:\Program Files\Microsoft SQL Server\90\DTS\Binn\ folder but named the 32 bit one to DTExec32.exe
then I was able to run my SSIS script through a stored proc:
set #params = '/set \package.variables[ImportFilename].Value;"\"' + #FileName + '\"" '
set #cmd = 'dtexec32 /SQ "' + #packagename + ' ' + #params + '"'
--DECLARE #returncode int
exec master..xp_cmdshell #cmd
--exec #returncode = master..xp_cmdshell #cmd
--select #returncode
Related
I have a simple query that I want to run and save the results in excel file on daily basis.
I am trying to create a SSIS(SQL Server Integration Services) package to automate this thing.
My query is
select * from Customers c , Orders o where c.Company = 'Company A'
I was able to create a package and run it using Integration Service, but the issue is that when I make changes in the database and re run the package, instead of overwriting the existing excel, it simply appends the new rows. Is there any way that I can overwrite the existing excel?
The second issue is, I can't see the option to schedule the package under integration services. When I right-click on the package name, I see the following
There is no option to schedule the package.
Is there another way, I can achieve the same purpose? I tried creating a job also -
I choose the target file as an excel file which was already not present in the system -
After the job executes successfully, it creates the excel but I can't open it, it says the file is corrupted.
EDIT : My SSISDB doesn't show any packages. When I created the package, it showed me two options, either store it on sql server or file system, so I chose file system -
Steps how I created the package -
I right clicked the database -> Export Data -> specified the data source-> specified the above query-> saved the ssis on file system .
Initially, my package was not automatically present under the file system.I had to import it manually.
I followed the steps from here -
https://www.mssqltips.com/sqlservertutorial/202/simple-way-to-export-data-from-sql-server/
I'm trying to load multiple files from a location into DB using Foreach Loop Container & DataFlow task in SSIS.
It's getting crashed while I try to execute the package. It's not giving any error message, whenever I execute the package it crashes and closes the visual studio app immediately. I have to kill the debug task in the task manager for the next execution of the package.
So I tried the below steps:
I used a FileSystem task instead of DataFlow task to just
move all the files from the source to the archive directory, which ran
fine without any issues.
Ran the DataFlow task individually to load a single file into DB,
which was also executed successfully.
I couldn't figure out what was going wrong here. Any help would be appreciated! Thanks!
Screenshots
All screenshots look fine to me. I will give some tips to try to figure out the issue.
Since the File System Task is executed without any problem, there is no problem with the ForEach Loop Container. You can try to remove the OLE DB Destination and replace it with a dummy task to check if it causing the issue. If the issue remains, it means that the Flat File Source could be the cause.
Things to try
Make sure that the TargetServerVersion is accurate. You can learn more about this property in the following article: How to change TargetServerVersion of my SSIS Project
Try running the package in 32-bit mode. You can do this by changing the Run64bitRuntime property to False. You can learn more about this property in the following article: Run64bitRunTime debugging property
Running Visual Studio in safe mode. You can use the following command devenv.exe /safemode.
Workaround - Using Bulk Insert
Since you are inserting flat files into the SQL database without performing any transformation. Why not use the SSIS Bulk Insert Task. You can refer to the following step-by-step guide for more information:
SSIS Basics: Bulk-Import various text files into a table
As mentioned in the official documentation, make sure that the following requirements are met:
The server must have permission to access both the file and the destination database.
The server runs the Bulk Insert task. Therefore, any format file that the task uses must be located on the server.
The source file that the Bulk Insert task loads can be on the same server as the SQL Server database into which data is inserted, or on a remote server. If the file is on a remote server, you must specify the file name using the Universal Naming Convention (UNC) name in the path.
I have an simple SSIS package and I'm trying to export same set of data from a table to both flat file and excel destination. The package works fine when I run locally and it creates both text file and excel file with data.
But when deployed to a different server the sql agent job runs fine and the log inside integrations services catalog for the package says it wrote like 9000 rows to excel, and a new excel file is also created but it doesn't write any data to it(blank with just headers). text file works fine and it has all data I need.
SSIS package flow:
I'm working with Sql server 2014, Visual studio 2013 with SSDT and used Excel 2007 in excel destination.
We had the same issue.
The solution is that the user, which runs the SSIS package, must have full access to c:\users\default.
You can check this by running sysinternals' process monitor on the machine that executes the SSIS job.
You can find more information here:
Empty Excel File permissions issue: SSIS Excel Destination buffers large record sets through C:\Users\Default - This post made me find this solution
https://www.csopro.de/biblog/2018/04/ssis-fehlerbehebung-bei-excel-destination-schreibt-keine-zeilen/ - my blog. Here I describe the issue - unfortunately in German]
I had the same problem writing to several worksheets in an Excel file from a scheduled SQL Agent job. It worked fine for about 4 months. Then suddenly with no changes to the package, one of the 5 worksheets was no longer populated with data. No error message generated and it worked fine on every test from Visual Studio and Data Tools (the old "BIDS" tools as we used to call it.)
I never did find a solution and it continues to not write any data to that single worksheet of the 5 in the Excel file. (So answers above about the Account that the job runs under from SQL Agent does not have the appropriate permissions is NOT a correct answer for this issue.)
Plus, a new package I built today is having the same issue, only this one has only a single worksheet. Again, works fine in the development environment, but no data appears in the destination file and no errors. Not only that, but the timestamp on the file is the same as the template file -- it seems that it never even TRIES to write to the file.
Checking each run log for the package in the Integration Services Catalog has an entry in each log that shows 9K+ records "written" for the dataflow task.
Lastly, if I change the destination file name, the SQL Agent job generates the expected error, so that rules out answers that guess that the path is wrong.
This is bizarre. And exasperating.
I have encountered odd behaviour when using scheduled SSIS packages which use the Excel object.
The fix for me, was to edit the Agent Job properties. On the Execution Properties tab, try enabling the "use 32-bit runtime" option and force the SSIS to run in 32-bit mode instead of 64-bit mode.
I am using the SQL Server Import/Export Wizard to import data from an Excel file into a table. I will need to do this exact import many times so I selected the option "Save SSIS Package."
I tried saving to SQL Server and I tried saving to File System.
Either way, I cannot figure out how to use the package in SQL Server. I read that I need to use Agent Job Task to do it but I cannot find that anywhere in the SQL Server program.
I also looked for Integrated Services in SQL Server and I found an icon that doesn't seem to do anything.
How do I go from having a .dtsx somewhere on my computer to executing the import (in a query preferably) without having to go through the whole wizard process again and again?
Thank you
The Import/Export Wizard creates a file called .DTSX. This is your SSIS package. An SSIS package is a text file filled with a bunch of XML. Contrast this with something that ends in .EXE/.COM which is an executable that any windows machine will be able to run.
We need something to consume that .dtsx file and perform the ETL operations described therein. There are three options open to you: dtexec.exe, dtexecui.exe, and custom .NET code. dtexecui is just a graphical wrapper to the command line so really, your choices for running packages are the prebuilt command line utility or custom code. Running a package via dtexec can be as simple as dtexec.exe /file C:\MyPackage.dtsx
Since you want to run this from the context of SSMS, then you'll need to either Create a SQL Agent Job or create a stored procedure that calls dtexec. Since you're new to the whole business, the Job will provide the lowest barrier of entry.
In your SSMS, connect to an instance and navigate to the bottom. You should see "SQL Server Agent" Expand that node and Right-Click on "Jobs" and choose New Job. In the Steps tab, click New.... Change the Type to "SQL Server Integration Services Package"
Pick your source (file system or SQL Server) Fill out the server name or where the package lives and you should be good to go.
One very important thing you will need to know is that Excel drivers are only going to work in 32 bit mode. This means you will need to use the dtexec.exe that exists at C:\Program Files (x86)\Microsoft SQL Server\XXX\DTS\Binn\dtexec.exe To get that behaviour from the SQL Agent job, then you will need to add the /X86 option to the command line like
The x86 flag only works for SQL Agent job calls. From the command line, you must use the correct version of dtexec.exe
I have a folder with a .sql files; 1 file per query. I want to execute all queries/ sql files and save them as csv.
Is there a way to do that automated without using the windows cli (disabled in my environment). I do have the SQL Server Management Studio.
I would approach this task using SSIS, providing you have Business Intelligence Development Studio (BIDS) installed.
First create a 'Foreach Loop Container' pointed to the folder with the SQL files, then use a variable to retreive each file name.
Next, create a flat file connection and set the 'Connection String' property to the variable that contains the file location.
Next, using the 'Execute SQL Task' component set the 'SQLSourceType' to 'File Connection' and the 'FileConnection' to the one created in the previous step.
Finally, depending on how the data is returned you have a couple of options, if the result set is small, only a row or a single column, then you can save the results to a variable and using a 'Dataflow' task create a 'Derived Column' component and export the contents of that variable to a CSV file. Or, if the dataset is larger you could dump the results to a temp table and then using an 'OLE DB Source' and 'OLE DB Destination' you could push the full result set straight into a CSV.
Hopefully this isn't too convoluted of a solution, this approach has the advantage of being able be run from either a remote machine or from the server itself, plus you can automate its execution with a SQL Agent Job.
Create a VB.NET console application.
Generate a list of files that end in .SQL from the folder in question.
Load the contents of each file into individual SQL Commands
Execute the SQL Command for each, storing the results in DataSets.
For each table in each dataset, create a new .csv file
For each .csv file, you will need to iterate over each cell in the datatable, and utilize proper escaping for .csv files.
Use 'for' in combination with either sqlcmd or bcp command for each file in the script folder.