I'm not new to SQL Server but am new to Integration Services, so I want to understand if the following requirement is fully achievable in SSIS, or if I will need to consider some level of C# development to supplement:
We have 25 Azure VM's running Windows Server 2016 Datacenter - on each VM we have thousands of log files in different folders. We need to have all these folders monitored and upon creation of any new file with a certain string present in the name, we want the contents of the file exported to a table in our Azure SQL Server 2017 database.
Is this kind of custom logic configurable in an SSIS project, or is SSIS more geared toward static definitions of file folders/filenames?
I don't think that you can implement a folder watcher within SSIS package. But you can achieve this in different ways:
(1) - using a folder watcher
You have to develop the SSIS packages you need to import data, then you must develop a folder watcher using C# to read the changes in folder and if the file name meet the requirementes you should run dtexec utility (using shell) to run the relevant SSIS package.
(2) - using SQL agent job
You can configure an SQL server agent job to check the file changes periodically and run the SSIS package when required.
Related
I have a DB project in Visual Studio and I need to deploy it to a remote system as a pre-packaged script (or similar) as a new database.
The Publish option will generate a .publish SQL script but only if you supply a connection first, and the script includes machine-local information e.g. logging paths.
I want something I can copy onto the DB-server machine running SSMS and it will create the DB. What is the proper way to do this, I see I also have DAC and DLL files emitted during build.
What is the proper way to do this?
Connect to a server that doesn't have any version of your database, and publish. Publish calculates an incremental deployment, so it needs to see the current state of the target SQL Server.
Having now added a like for like DB to do UAT on to our server I have run into a problem with our SSIS Packages. They are stored as files and as such we use File System in SQL Server Agent to run them on a schedule. I now need to run these packages on both databases (different data sources).
I have checked over documentation and forums and when paramterizing over environments every time it relates to the deployment of packages to the catalog. Which I think is because with files they are saved with package data sources not project data sources.
This left me wondering...
Is it possible to switch data sources when calling a package using File System in SQL Server Agent?
Unfortunately the answer on this is no.
Moving to the SSIS catalog allows you to switch between environments which you can set in SSMS.
This is a great link: https://blog.exsilio.com/all/configuring-environment-variables-ssis-package/
I did get an error when running my agent job after this but this was a need to set the runtime to 32bit.
In SSIS - Right click your solution and head to debugging to find the option.
In SQL Server Agent - Head to Config and advanced where you would set the environment variable in the SSIS step you have created.
Company is setting up new instances of SQL Server and because I have many SSIS jobs, the idea was to setup a dedicated SSIS server. It seems to be a relatively common practice and makes a lot of sense. You're removing the resources needed for SSIS off the main database server versus how we've done it historically which is to store the packages in a directory on the main database server and having SSIS compete with the same resources as the actual database. Here's where I am confused.
I was told by our DBA to place a copy of the SSIS package on a file share directory on the SSIS server. Then using the SQL Agent on the database server, create a job to run the package like normal where Package Source = "File System" and Package location being the file share directory and file name on the SSIS server. Problem is I don't see how that kind of setup accomplishes the goal of removing the execution of the package off the main database server. Won't the main database server still be executing the package thus impacting performance and all we're accomplishing is using the SSIS server as an external drive or place to store the actual .dtsx files?
I'm trying to rebuild an SSIS environment on a new server. The existing environment uses a local drive for some data input, using SSIS to bulk load the data from csv files.
I'd like the new environment to use out relatively new NAS for this task. The nas is mapped to a drive letter on Windows Server, both using my domain name and the generic dwarehouse domain name used to install the software.
When I execute the new package manually using Visual Studio the bulk load process cannot see the file. If I copy the file from the NAS to a local drive, Visual Studio can see it.
I'm sure it's a matter of giving the right permissions to the correct user, but I can't figure out what user that should be.
FWIW, we are using SQL Server 2012 Data Warehouse Edition and Visual Studio 2012.
It should use the credentials that started the package, unless the connection has different explicit credentials provided. If it was started by a sql agent job, then it will run with the credentials the SQL Agent service runs with. If it was run from a scheduled Windows task, then check the Task Scheduler credentials.
Follow it back to how the package was started and you should be able to identify what credentials it uses.
I pretty new SQL Server Integration Server (SSIS) user. Is SSIS able to query data from text files located in another Windows Server? I mean that when SSIS is installed on Windwos Server A, is SSIS able to query data from e.g. one folder containing text files in Windows Server B (under same domain)? I have used only SAP BO Data Integrator ETL tool and it cannot query flat files from another Server: during execution, all files must be located on the Job Server machine that executes the job.
Yes, you can access text files on another server using SSIS by using a fileshare and accessing the files using the share name i.e. \\ServerB\MySSISFiles\.
You need to make sure that the account the SSIS is running under has access to the file share.