I'm having an issue exporting a large dataset (500k+) to Excel via SSIS, where the output file ends up with 0 rows exported. Before saying that I shouldn't be exporting that many records to Excel, let me state that I know and normally wouldn't. Accounting does not want a CSV and is unwilling to open a CSV in Excel.
Using Visual Studio 2012 SSDT, here are the components involved.
Execute SQL Task -> Creates the empty file with headers
Data Flow Task ->
OLE DB Source -> SQL Query
Excel Destination
While the package is running, you can see records flowing from the source to the destination. The package completes without error, but when you open the file, it's empty. The only thing in there is the header.
If I select the Top 1000 records and export to Excel, it works as intended.
Some things I've tried:
Export to Excel on the network
Export to Excel locally
Export to CSV to Excel on both network and locally
Export to Ole DB Destination using Office Access Database Engine 12.0 with "Excel 12.0" extended properties.
Tried running as different users
All with the same outcome.
Can anyone provide any insight into why this may be happening and how to proceed?
We experienced a similar behaviour, when runnig the ETL in a SQL Server Agent job. Debugging it in Visual Studio, worked, however. So I do not know, whether this solution applies to you.
The reason was that the user, under which the package ran, did not have access to C:\users\Default.
I found this out by using sysinternals process monitor.
I was inspired by that post: Empty Excel File permissions issue: SSIS Excel Destination buffers large record sets through C:\Users\Default
[I explained my search for the bug in my blog: https://www.csopro.de/biblog/2018/04/ssis-fehlerbehebung-bei-excel-destination-schreibt-keine-zeilen/ Unfortunately it is in German]
Related
On our servers (windows 2016, SQL reporting server 2016, Microsoft Access Database Engine 2016) we run 2 SSIS packages. 1 imports data from an excel file to the database and 1 exports data from the database to an excel file. Both are xlsx files.
We run this exact package on TST, ACC, RES and PRD (same server and access setup). We didn't have any issues until a week ago the packages on PRD just kept on getting stuck in the "beginning validation phase" of the Dataflow Task. The other environments are fine.
We've determined that it is not a problem in the application since a simple read package that we created for this issue, gave the same problem. It doesn't seem to be an access issue either. The account that runs the script is sysadmin in SQL and local admin on the fileserver.
We also tried
• Only using one import flows instead of two in Data Flow task: no change https://social.msdn.microsoft.com/Forums/sqlserver/en-US/781c855f-833e-4578-a43a-1729482bbabd/dtspipeline-validation-phase-is-beginning-but-never-stop?forum=sqlintegrationservices
• Set connection managers for OLE DB sources are all set toDelayValidation to True: no change SSIS pre-evaluation phase taking long
• Set ValidateExternalMetadata is set to false for Excel Sources: no change SSIS pre-evaluation phase taking long
• Reinstall Microsoft Access Database Engine on server: no change
• Tested reading a flat file (txt) which worked without issue.
We're fresh out of ideas so any help would be greatly appreciated.
UPDATE:
When manually trying to run the import/export wizard (and selecting excel file) I get "The operating system is not presently configured to run this application". Investigating this message as well.
If you had no problem reading a text file then that would point me to the excel driver (32 or 64), however I would think if that was the issue you would see a connection error. Do this as test.
Go to the console and open excel on the server. This will let you know if you have some licensing issue or something preventing excel from opening on the server.
Import a small amount of data into SQL server from excel using the import data into a test database (or just make a test table). Be sure to use the same driver you are using in the SSIS package.
Currently I am building a database in SQL Server Management Studio (SSMS). Several data sources are imported in this database, via the import/export wizard. Upon completion of the import, not only am I running the import, I also save an SSIS package. This SSIS package is to schedule a weekly refresh of the data source.
On my Windows Server 2012 R2 the Express edition is installed, therefore I have manually created several batch files that run every week (as scheduled via Task Scheduler). This works fine for most tables, however for some tables I encounter some strange (?) behaviour.
This is as follows: when creating the SSIS package via import/export wizard, and directly running the import, the table shows up correctly in the database. That is, with all column names and the thousands of rows it contains.
The strange thing is that, when executing the SSIS package (via the batch file), the table is empty (column names are correct though). For some tables, I do not encounter this behaviour. For others, this behaviour is encountered all the time.
The batch script is as follows (quite straightforward):
"C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\DTExec.exe" /F "C:\Users\username\ssispackage.dtsx"
The batch file seems to run correctly at all times, as the table 'creation_date' changes when I run the batch file. Moreover, for all the tables that do correctly 'refresh', these same batch files do the job.
Some settings of the SSIS package:
Data source: Oracle OLE DB provider
Destination: SQL Server Native Client / MS OLE DB Provider for SQL Server (tried both)
Data via query (as I am querying several tables from Oracle); query is parsing correctly
Mappings: Create destination table & Drop and re-create destination table
Dropping and re-creating is done, because the data source is rather small, and has some changes weekly/monthly (to some rows).
For most data sources imported (and refreshed weekly) via this method, the tables are correctly showing each week (simply dropping the previous table, and re-creating the source).
I hope someone can explain to me why this issue occurs. If some more information is needed from my side, please ask.
Thanks in advance!
UPDATE:
When looking at the log of the batch file, this is (part) of the output:
Source: ..... "SourceConnectionOLEDB"
Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "OraOLEDB" Hresult: 0x80004005 Description: **"ORA-01005: null password given; logon denied".**
End Error
.....next error.... "SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.
Thus, it seems that the password is not remembered/saved correctly in the SSIS package?
This is strange though, as for most tables it does correctly store the password (as those do refresh correctly).
When setting the properties of the data source, namely Oracle Provider for OLE DB, I select the option "Allow saving password". So it should store the password correctly?
Found the answer after all.. The .dtsx file that is saved (the SSIS package) contains the variables for the connection string, it shows that the Password (Sensitive="1") is there. But in the wizard, I did not select 'Save sensitive data with user key'. When selecting this option, an encryption string was added. Now the SSIS packages run well!
I have an simple SSIS package and I'm trying to export same set of data from a table to both flat file and excel destination. The package works fine when I run locally and it creates both text file and excel file with data.
But when deployed to a different server the sql agent job runs fine and the log inside integrations services catalog for the package says it wrote like 9000 rows to excel, and a new excel file is also created but it doesn't write any data to it(blank with just headers). text file works fine and it has all data I need.
SSIS package flow:
I'm working with Sql server 2014, Visual studio 2013 with SSDT and used Excel 2007 in excel destination.
We had the same issue.
The solution is that the user, which runs the SSIS package, must have full access to c:\users\default.
You can check this by running sysinternals' process monitor on the machine that executes the SSIS job.
You can find more information here:
Empty Excel File permissions issue: SSIS Excel Destination buffers large record sets through C:\Users\Default - This post made me find this solution
https://www.csopro.de/biblog/2018/04/ssis-fehlerbehebung-bei-excel-destination-schreibt-keine-zeilen/ - my blog. Here I describe the issue - unfortunately in German]
I had the same problem writing to several worksheets in an Excel file from a scheduled SQL Agent job. It worked fine for about 4 months. Then suddenly with no changes to the package, one of the 5 worksheets was no longer populated with data. No error message generated and it worked fine on every test from Visual Studio and Data Tools (the old "BIDS" tools as we used to call it.)
I never did find a solution and it continues to not write any data to that single worksheet of the 5 in the Excel file. (So answers above about the Account that the job runs under from SQL Agent does not have the appropriate permissions is NOT a correct answer for this issue.)
Plus, a new package I built today is having the same issue, only this one has only a single worksheet. Again, works fine in the development environment, but no data appears in the destination file and no errors. Not only that, but the timestamp on the file is the same as the template file -- it seems that it never even TRIES to write to the file.
Checking each run log for the package in the Integration Services Catalog has an entry in each log that shows 9K+ records "written" for the dataflow task.
Lastly, if I change the destination file name, the SQL Agent job generates the expected error, so that rules out answers that guess that the path is wrong.
This is bizarre. And exasperating.
I have encountered odd behaviour when using scheduled SSIS packages which use the Excel object.
The fix for me, was to edit the Agent Job properties. On the Execution Properties tab, try enabling the "use 32-bit runtime" option and force the SSIS to run in 32-bit mode instead of 64-bit mode.
I have a problem with SQL JOB.
I am using SSIS to import data into database. While I am running SSIS through Visual studio it works as expected. Also when I am running the SSIS from Integration Services Catalogues in SQL Server, it works fine. But my issue is that when I am trying to import data into my DB using SQL JOB (using the same SSIS as in previous cases), although there is no error message and Sql job runs with success, the destination table in my DB is empty (no data are imported into my DB).
When I checked the execution report for SSIS, run by SQL I see this warning:”Foreach Loop Container: Warning: The for each file enumerator is empty. The for each file enumerator did not find any files that matched the file pattern, or the specified directory was empty.”
Once again, everything works OK when I am running SSIS in other way than by using SQL JOB.
Am I doing something wrong? How can I solve this problem? Please help me, because I am not able to solve this issue.
Thank you.
I have a server that is capable of creating and running an Excel Import task using the Import Wizard. I am trying to automate that process by using a visual Studio 2010 Integration Services package, that I am developing on that server.
The problem happens when trying to design the package. I have added an excel connection and pointed it at the Excel file on a local disk (the same file I have already successfully imported using the import wizard). When I add an Excel Source to the DataFlow and specify the excel connection, when I go to the Name Of the Excel Sheet Drop down I just see "No tables or views can be loaded" and get the following error.
"Could not retrieve the table information for the connection manager.
Failed to connect to the source using the connection manager ..."
I can't find this error logged anywhere and i don't know why it is failing. The directory is shared to Authenticated users and the file is not in use.
Any ideas how to debug this error? I understand there can be issues running this in 64 bit mode, but does that apply to development?
I should add that it is an excel 2007 file .XLSX and the connection is set to Excel 2007.
2019-11-08 The answer by GavB841 below looks promising, if anyone tries it and it works please let me know. (I am no longer working in this area.)
It seems like the 32-bit version of Excel was not installed. Remember
that SSDT is a 32-bit IDE. Therefore, when data is access from SSDT
the 32-bit data providers are used. When running the package outside
of SSDT it runs in 64-bit mode (not always, but mostly) and uses the
64-bit data providers.
Always keep in mind that if you want to run your package in 64-bit
(which you should aim for) you will need both the 32-bit data
providers (for development in SSDT) as well as the 64-bit data
providers (for executing the package in production).
I downloaded the 32-bit access drivers from:
Microsoft Access Database Engine 2010 Redistributable.
After installation, I could see the worksheets
Source:
Extracting Data From Excel with SSIS
The workaround is, I save the excel file as excel 97-2003 then it works fine
I also ran into this problem today, but found a different solution from using Excel 97-2003. According to Maderia, the problem is SSDT (SQL Server Data Tools) is a 32bit application and can only use 32bit providers; but you likely have the 64bit ACE OLE DB provider installed. You could play around with trying to install the 32bit provider, but you can't have both the 64 & 32 version installed at the same time. The solution Maderia suggested (and I found worked for me) was to set the DelayValidation = TRUE on the tasks where I'm importing/exporting the Excel 2007 file.
The recommendations from this article Extracting Data From Excel with SSIS resolved the issue for me.
I downloaded MS Access Database Engine 2010 32 bit driver
from the link in that article.
Also set Project Configuration Properties for Debugging Run64BitRuntime = False
In SQL Server 2014 SSMS (Integration Service Catalog -> SSISDB -> Environments -> Projects for all Packages in Validate checked box 32 bit Runtime.
My SSIS packages are working now in both VS 2013 and SQL Server 2014 environments.
Simple workaround is to open the file and simply press save button in Excel (no need to change the format). once saved in excel it will start to work and you should be able to see its sheets in the DFT.
You need to use an older version of the data connectivity driver (2007 Office System Driver: Data Connectivity Components) and select Excel version 2007-2010 in the connection manager configuration window.
I assume the newest data connectivity driver for Office 2016 is corrupt
Here's the solution that works fine for me.
I just Saved the Excel file as an Excel 97-2003 Version.
As discussed in the below:
Solution:
Go to https://marketplace.visualstudio.com/items?itemName=ProBITools.MicrosoftAnalysisServicesModelingProjects and install the latest version, it has a fix in there to resolve this issue.
Reference: https://developercommunity.visualstudio.com/content/problem/745991/could-not-load-file-or-assembly-microsoftdatawareh.html
A fix for this issue has been internally implemented and is being prepared for release. We’ll update you once it becomes available for download. For now, please install latest SSAS from https://marketplace.visualstudio.com/items?itemName=ProBITools.MicrosoftAnalysisServicesModelingProjects to work around the issue. Sorry for any inconvenience.
I faced the same issue. I think #Rishit answer helped me.
This issue is related to 32 bit/ 64 bit version of driver. I was trying to read .xlsx files to SQL Server tables using SSIS
My machine was pre-installed with Office 2016 64 bit on Win 10 machine along
with MS Access
I was able to read excel 97-2003 (.xls) files using ssis, but unable
to connect .xlsx files
My requirement was to read .xlsx files
Installed AccessDatabaseEngine_X64 to read xlsx, that given me
the following error:
I uninstalled the AccessDatabaseEngine_X64 and installed
AccessDatabaseEngine 32 bit, that resolved the issue
My answer is very similar to the one from #biscoop, but I am going to elaborate a bit as it may apply to the question or to other people.
I had a .xls that was an extraction from one of our webapps. The Excel connection would not work (error message: "no tables or views could be loaded"). As a side note, when opening the file, there would be a warning stating that the file was from an online source and that the content needed activation.
I tried to save the same file as an .xlsx and it worked.
I tried to save the same file with another name as an .xls and it worked too.
So as a last test I only opened the source .xls file, clicking save and the connection worked.
Short answer: just try and see if opening the file and saving does the trick.
Solution
Actually, if you are using SSIS in VS2019 then a problem could occur so do the below things:
Go to the excel sheet and save it again in .xls format.
Once you will use it in .xls format, automatically type will be coming as Microsoft Excel 97-2003 worksheet.
Like in the attached picture.
You will be able to load your sheet data.
I found that my excel file that was created in Excel 365 was incompatible with any of the versions available. I re-saved the excel file in 97-2003 version and of course chose that version in the dropdown list and it read the file OK.
you can try this:
Uninstall office365
then install only Access Database Engine 2016 Redistributable 64 bit
Also set Project Configuration Properties for Debugging Run64BitRuntime = False
It should work.
After researching everywhere finally i have found out temporary solution. Because i have try all the solution installing access drivers but still i am facing same issues.
For excel source,
Before this step you need to change the setting. Save excel file as 2010 format.xlsx
Also set Project Configuration Properties for Debugging Run64BitRuntime = False
Drag and drop the excel source
Double click on the excel source and connect excel. Any way you will get an same error no table or view cannot load....
Click ok
Right click on excel source, click on show advanced edit.
In that click on component properties.
You can see openrowset. In that right side you need to enter you excel sheet name example: if in excel sheet1 then you need to enter sheet1$. I.e end with dollar symbol. And click ok.
Now you can do other works connecting to destination.
I am using visual studio 2017, sql server 2017, office 2016, and Microsoft access database 2010 engine 32bit. Os windows 10 64 bit.
This is temporary solution. Because many peoples are searching for this type of question. Finally I figured out and this solution is not available in any of the website.