The user case is that we have a file in our SharePoint online that I need to load into our on prem SQL server, using SSIS 2019 (64 bit).
This question is about how to use SSIS to copy the file from the online SharePoint Document Library to an on-prem server file location, from the on-prem server I indeed to use SSIS to load the file data required to the on-prem SQL server
I have read a number of question
Opening Excel file stored on SharePoint as data source using ADODB Connection
Import Excel file located in SharePoint Server into SSIS
However, in the main I am struggling to use the "Show in File Explorer" option even when I try to use IE11 Can any one suggest an easier way to convert the web file path to the correctly formatted UNC
I am aware that I could use a Power Automate to move the file to achieve the same thing, but I am finding it strange that SSIS does not do this
Can anyone help please
Related
I have an Access DB which I connected to a SQL Server 2012 (v11.0.7507) as a linked server using Microsoft Office 12.0 Access Database Engine OLE DB Provider.
The connection is working fine usually but there are 2 automated jobs that push/pull data and update some other data in the Access DB either of which sometimes (not quite often but sometimes) gets "stuck" while connected to the Access DB. The session in SQL Server is visible but cannot be killed - or actually when I try to kill it it gets stuck in KILLED/ROLLBACK status.
No MSACCESS.EXE is open in tasklist, however the Administrative Tools' Open Files section does show the accdb (and the laccdb) files open by the SQL Agent. Unfortunately closing them there does not solve the problem, the accdb and laccdb files are still locked.
The only solution is restarting the SQL Server and then deleting the laccdb file but that is not a very comfortable solution.
I tried searching for similar issue but did not find a solution so far that would work. The main problem is the (seemingly) random nature.
Have you guys experienced something similar? Do you know what could be wrong with this setup?
Many thanks in advance.
The Access OleDb provider has very limited support in unattended applications, and should never be loaded in a long-lived application. See the note on the download site:
The Office System Drivers are only supported under certain scenarios,
including:
Desktop applications which read from and write to various files
formats including Microsoft Office Access, Microsoft Office Excel and
text files.
To transfer data between supported file formats and a
database repository, such as SQL Server. For example, to transfer data
from an Excel workbook into SQL Server using the SQL Server Import and
Export Wizard or SQL Server Integration Services (provided the SSIS
jobs run in the context of a logged-on user with a valid
HKEY_CURRENT_USER registry hive).
Try using a separate short-lived process instead of Linked Server. An SSIS package is a common choice.
I have one .XLSB file (MS excel binary worksheet created using Office 2016) having more than million records. How can I load this using SSIS?
I have both SQL server 2014 and SSDT 2012 & SQL Server 2017 SSDT 2017 on two different PC.
The problem is : In SSIS , I'm using excel source , connection manager as excel, the name of the excel sheet is not appearing. Saying, no table or view could be loaded. However, my excel has sheet in it.
Could not retrieve the table information for the connection manager 'Excel Connection Manager'.
Failed to connect to the source using the connection manager 'Excel Connection Manager'
EDIT more on error
As per this : Accessing Excel 2007 Binary (.xlsb) via OleDb ACE installed the access (2016) engine. But, still having problem.
What could be the problem ? Any link would help? Thanks.
Install Microsoft.ACE.OLEDB.12.0 32 bit Provider. Then go to project property -> configuration properties -> Debugging -> set Run64BitRuntime = False.
Click on the image to see details about connection property and file extension
Based on your question you have two problems:
Access Database Engine is not registered error
Reading Xlsb file format using SSIS
Access Database Engine is not registered error
You can refer to the following links for more information on how to solve similar issue:
Access Database engine OLEDB in windows server 2016
Unable to Connect to Excel via SSIS
Reading Xlsb file format using SSIS
You have first to select a normal Excel file from the Excel connection manager (create it manually) then from the ExcelFilePath property change it to the .xlsb path;
Import Excel binary files into SQL Db using SSIS
Alternatives
Using Interop librairies to read from Excel:
Read Excel File in C#
Using Third party librairies to read from Excel:
Aspose.Cells for .NET
Closed XML To Read Excel Files In ASP.NET C#
Using OpenXml librairies:
Parse and read a large spreadsheet document (Open XML SDK)
Convert to csv file
I have installed Sql Server Data Tools for Visual Studio 2017 and noticed that there are no SharePoint connections in the SSIS toolbox. I know I can read from a SharePoint list using the ADO connecter and save it off somewhere, but what I'm looking for is reading a file in, a csv file, and then being able to write that information into a SharePoint list. This is a work machine and so I can't use the CodePlex archive options or the Kingswaysoft adapters. I was wondering is the SharePoint adapters only a third party option or is there an out of the box solution within the SSIS options that I'm missing. I've got Visual Studio 2017 Professional as well as Sql Server Management Studio 2014. The Sql Server Data Tools are for Visual Studio 2017.
Sharepoint Data Flow Components
I don't think there is an official Destination Component for Sharepoint in SSIS. Since you are using SQL Server 2014, you can benefit from ODATA Source component to connect to Sharepoint. These components are found in the SQL Server 2014 feature pack.
Unfortunately, if you need to write to a Sharepoint list, you have to use a third party components such as :
SSIS Data Flow Source & Destination for OData
OData SSIS Components
Using SSIS to Automatically Populate a SharePoint List
COZYROC - SharePoint Destination
SSIS Integration Toolkit for Microsoft SharePoint
Workarounds
You can use a Flat File Destination to Store the result inside a flat file on local folder. Then you can add a Script Task that will be executed after the Data Flow Task. Then write a C# / VB.NET script to upload the file to Sharepoint using Microsoft.Sharepoint.dll assembly. You can refer to the following links for more information:
How to: Upload a File to a SharePoint Site from a Local Folder
Another thing to try, if you are not familiar with Script Task and you have a knowledge in Web Services, try to create a WebService to upload files to SHarepoint List and use a Web Service Task instead of Script Task:
Upload a Document to the SharePoint Document Library using a Webservice
Upload document from Local Machine to SharePoint Library using WebService
can i use microsoft 1 drive to store the flat file and can i load the files from one drive to sql server database suing SSIS.Can any one share a thought on an alternative.Keeping the master data flat file on cloud and making it available on sql server.We do not want to download those files stored in cloud.The thing is would like to create ssis package to read the files on the cloud
I pretty new SQL Server Integration Server (SSIS) user. Is SSIS able to query data from text files located in another Windows Server? I mean that when SSIS is installed on Windwos Server A, is SSIS able to query data from e.g. one folder containing text files in Windows Server B (under same domain)? I have used only SAP BO Data Integrator ETL tool and it cannot query flat files from another Server: during execution, all files must be located on the Job Server machine that executes the job.
Yes, you can access text files on another server using SSIS by using a fileshare and accessing the files using the share name i.e. \\ServerB\MySSISFiles\.
You need to make sure that the account the SSIS is running under has access to the file share.