I've inherited an SSIS package which loads CSV files into a SQL database.
The first thing the package does is call a .BAT file which maps a network drive. The .BAT file contains the username and password used to map the drive in plain text, so urgently needs to be replaced.
I've written a solution which uses New-PSDrive in a Powershell script and creates a credentials XML file with the password encrypted. If I execute the ps1 script it works Ok.
If I call the ps1 script from within SSIS and run it through VS then it also works fine.
When I call the SSIS package that calls the script task through a SQL Agent job (executed as the SQL Server Agent user account) the drive mapping doesn't seem to work, and the package complains it cannot access the file from the file share.
I presume this is no working because the SQL Server Agent user account can't run Powershell queries? The package only errors because it cannot access the share, not during the execution of the Powershell script.
So, two questions:
Is there an obvious issue with the SQL Agent deployment idea
or
Is there a different way of mapping a network drive from within SSIS without storing the password in plain text anywhere?
This is the pattern I suggest using for accessing files on a remote server:
Access the files via unc path: \\MyServer\Files
This means that the package needs to run under a context that has access to the path. There are a couple of options:
Run the package under a proxy, i.e. the account credentials that were stored in the bat file. This means that you have to also grant access to the account on the sql instance as well
Use a group managed service account (gMSA). In this case, the sql agent service account is replaced with the gMSA and the job runs under sql agent. The gMSA needs to be granted access to the remote share as well as the sql instance. This is a much more secure way of addressing the whole process because there is no password to manage (AD takes care of that) and the account cannot be used to log in anywhere. But there is set up work to do to get it created so it's not the fastest option.
Related
I posted a similar question before, but I have now came back to the point where I need to deal with this and after some tuning I have managed to get rid of all the errors, except one warning that basically tells me that the path provided to the package not finding any files in the directory (which is false). This almost makes me want to believe this is again a permissions issue. As when I run the package locally it all works fine. Here is the warning:
Same message appears when I right click SSIS package under the catalogue and directly try to execute the job.
I made sure that the caller has full permissions to the folder in question
SQL Server Agent Job calls the job like so
With a single step in it to execute the SSIS package from the catalogue:
The history of the executions is all successful and the history of SQL Agent job is all green
One of the suggestions I got was to open SQL Server Configuration manager but I don't appear to have access to that. I am new to this whole process so I am not sure if authentication has something to do with it or the sa account's permissions.
Note: I am working on this on a remote dev server and not directly on my pc.
Any help would be greatly appreciated.
Since you're running the package as the Agent Service Account:
The permissions need to be applied to the SQL Server Agent Service Account or its per-service SID.
You can see the service account with PowerShell like this:
PS C:\Users\david> (Get-WmiObject win32_service | where Name -eq "SQLSERVERAGENT").StartName
NT Service\SQLSERVERAGENT
Turns out the database instance was on a whole different server that I wasn't even given access to. That server obviously didn't have the path specified in the SSIS package, so what I had to do is create a folder for files in the correct server, reroute my files there, and change the SSIS package path after obtaining the access to the server where DB instance lives. Me being new to all this, it was absolute frustration especially since our team is small and I am new in it I can't just ask someone questions about this all the time. Hopefully this will save someone a lot of time.
I have created SSIS packages which creates new folder using file system task at output location and then creates data files in that folder using data flow task.
I have created one network drive which is mapped to azure blob. So I pass the network drive path like Z:\ to package and the folder and files are created as expected which are also reflected on the azure blob.
Now when I schedule this package through SQL agent job I get error that the cannot find part of the path Z:\folderName from file system task. So I though it is because the sql server agent service was not running through my user id . So I started sql server agent with my credentials but it still gives me same error.
Note: My Id doesnt have direct access to azure blob and the network drive is only accessible by my id.
We are currently using azure blob for dev but we may use separate server to store files due to which I cannot use flexible file system task available in SSIS azure service pack
We had reasons we could not use UNC paths (\server\share\file.txt) and struggled with mapped drives as well. The approach we ended up using was to explicitly map and unmap drives as part of our ETL Process.
Execute Process Task - Mount drive
We used a batch file to mount the drive as it made things easier on us but the command would take the form of something like
net use Z: \\server\share mypassword user#azure
Execute process task - Unmount Drive
Again, execute process task
net use /delete Z:
We ended up with a precursor step that I don't remember how we accomplished it, but we would test whether the drive had been unmounted before we attempted to mount it. I think it was File System Task to see if the drive existed and if so, then we had the Unmount task duplicated.
I was able to resolve this using credential manager. I logged in using a domain user and then added the azure path like \\server, userid and password to windows credentials . Then started the sql server agent service using the same domain user id . Set up the file path as \\server\share similar to what I have provided in windows credentials into the ssis package configurations.After doing this I was able to successfully execute the package through sql agent job.
I was able to solve this issue using script task (C# Code) to move data. I've generated a project parameter for Target ShareFolder and used it as ReadOnlyVariable for script task. The target share folder has been already mapped as a network drive on my local PC as well as on Application Server.
I'm new to SSIS and stuck with a problem and hope some of them would have already gone through any of this.
Task:
Copying files from a remote server to a local machine folder using File System task and For each loop container.
Problem:
The job executes i.e. files are getting copied successfully when I execute from the SSIS designer but when deployed the project on the SQL server instance it isn't copying any files in fact the target folder is totally empty.
I'm not understanding this strange behavior. Any inputs would be of great help!
Regards-
Santosh G.
The For each loop will not error out if it doesn't find any files.
The SQL Agent account may not have access to read the directory contents.
Check is your path a variable - is it being set by a config or /SET statement?
Can you log the path before starting the for loop?
Can you copy a dummy file and see can SSIS see this file?
How are you running the job - cmd_exec() can give spurious results with file I/O tasks
The issue was related to the user authorizarions of the SQL Server agent service.
When I execute the job from SQL Server it uses agent service and for that agent service you need to assign a service user who has access rights to the desired file path.
I have a SSIS package which creates a folder in an UNC share and then creates a file there (using script task).
The domain account which is used by SSIS and Agent has all the possible permissions in the DB computer and the share computer.
It always fails there.
I created a test SQL Agent job which creates a backup of the database in the same location and it fails too (Operating system error 5 - access denied).
EDIT: The above test example may be irrelevant since the backup operation is executed by SQL Server Database Engine and not the SQL Agent itself (Agent still schedules the task).
I cannot debug the script task in SSIS and therefore Im not sure what is the problem.
I have managed to fix this problem. The first problem was lack of sufficient task activation/execution permissions in the DCOM config node in Component Services. The permissions had to be set for SQL Server Integration Services.
The second problem was the fact that the UNC path looked like this:
\\192.168.250.51\C$\Folder\
I needed to create another share (visible) like that:
\\COMPUTER-NAME\Folder\
Also don't try to map any drives to the folders. It won't work.
I have to say that I hate myself for such general question as "What I am doing wrong?" but I simply have no idea what can be the problem:
I've created SSIS package that takes the data from flat files (CSV), counts the average on one of the columns, groups by date and writes it to the database and deletes the original file. All works fine when executed within SSIS, but when I am scheduling it within Server Agent it simply doesn't work - log reports success but there is no new data in the database and the .csv file exists in its original location.
I know the problem with protection level set up in SSIS, so I've changed it to "EncryptAllWithPassword" and I use the same password with Server Agent.
Here is a link to the Server Agent Job script (created as "script job as DROP and CREATE")
Edit: Just to make things weirder, using
dtexcec /f {filepath} /de {password}
executes program without problem. I know I can shedule such command in the Windows itself, but i'd like to keep all scheduled jobs in one place - in the Server Agent
EDIT: Solved by changing the path to UNC
There are two important things to remember when setting up packages to run via a SQL Server Agent job.
Use UNC paths for all file locations, no matter how simple. There is a high probability that the server will have a different view of the file structure than your development machine, so UNC paths ensure that both machines are referencing the same paths.
Use a proxy account to execute that package, as described here http://www.mssqltips.com/sqlservertip/2163/running-a-ssis-package-from-sql-server-agent-using-a-proxy-account/.
The proxy account must have access to the physical paths and the server objects.
This also allows for security stratification on your various packages (not all packages need access to everything).