I am trying to sftp(SSH) transfer files from a remote server to a local directory in SSIS. I was recommended to use PSFTP.exe to conduct the transfer via a batch file. In debug mode, this works flawlessly, but when deployed to SSIS Catalog, the job stalls at the point to connect and transfer the files. I have no errors from SSIS. The job just freezes. The user set in the job is in the administrator's group. I am missing something. How do I troubleshoot?
Thank you.
Thanks to #Nick.McDermaid, I found out that adding the flag -batch and the key -hostkey aa:bb:cc:........ fixed my problem. for some reason, even though i have the RSA key approved via CMD line, the ssis package did not see the approval, forcing the acceptance. adding the hostkey flag solved the key issue, and the batch flag turned off the interactive setting.
Related
this is one that has me stumped and Ive been doing this a long while.
Migrating to SQL server 2016, large number of ETL. Easy enough.
One of the ETL packages has a simple script task to take a table of files, run a file exists foreach loop.
it uses a project parameter to create the unc ( \servername\share) and then binds that to the file name in the script task.
use an environment config setup in SSISDB
execute in SSDT works fine, deploy to catalog and it cant see the file. i know youll say permissions, but ive permissioned everyone group to share and drive in case its that. SSISDB execution means it should be running under my security context and im domain admin, local admin and creator owner of the share.
even strangeR, i have created simple package to grab the contents of one of the files and import into a dump table in case permissions or pathway were duff ( even though they work in SSDT might be the enviroNment config in SSISDB). THIS WORKS FINE, therefore it cant be the envrionment setup of SSISDB being referenced.
please note this is not running from an agent job yet so wont be due to agent server account issue. need to get it running from ssisdb first then ill create an agent job
So -- script task cant see unc share, built from two variables, that works in ssdt and its running under same credentials...
Go
For what its work the script task code is
Dts.Variables("BolFileExists").Value = File.Exists(Dts.Variables("StrLoadFileLocation").Value.ToString & Dts.Variables("StrCurrentFile").Value.ToString)
This is a slightly different answer as it shows a different approach and removes the script task. I use a foreach to check if the file exists using GUI tools provided by SSIS:
Well I found the answer and I deserve to punch myself in the face.
Tried everything, it was a file variable and path variable being pulled together in the script task so tried concatenation that before the script task, pumped this into a table to ensure it was going to write table.
Literally everything was fine and still didn’t work.
The issue....
Building it as a 2017 package onto a 2016 Sql server.
I’ve not found what was missing dll wise but it must have been one of those that meant the script task couldn’t find the files but weird it didn’t break and just said the files weren’t there!
Thanks all for input, I’m going to go put my head in the door and slam it
Each time I run the Microsoft Windows Backup and Restore App that is left over since the Microsoft Windows 7 Operating System, I get an error that some Files are missing and the Backup Process fails.
The Files are actually Folders. I have uninstalled some Apps in the meantime and now there is only one missing Folder that the Backup App does not find.
I have tried to run a Batch File within the CMD.EXE Command-Line Processor App with System Administrator Rights:
#ECHO OFF
SET DIR1="C:\Windows\System32\config\systemprofile\OneDrive\Pictures\Saved Pictures"
MKDIR %DIR1%
PAUSE
The Folder does get created well and nice, but the Backup App is still failing.
Could it be a Rights Dead-Lock?
I am creating the Folder using System Administrator Privileges because it is not possible otherwise.
I suspect that the Backup App is run with Normal Rights. However, the User Account that I am using is also part of the Administrators Group.
Please advise.
I could not reproduce this Issue.
The reason why I guess that this is happening is the following one:
The Microsoft Windows Insider Program is constantly rewriting the whole C:\Windows Folder on each Update, therefore the Folders that are missing have to be constantly recreated.
Earlier, I might have manually started the Microsoft Windows Backup and Restore Application and forgot to run the Batch File. The Application might have started to work on the Files and Folders to back up. Then, I might have manually run the Batch File that correctly created the Files and Folders, but that might have been too late - that is, after the Application already considered them as missing. Therefore, the error was happening.
I do not know for sure whether this is the cause for this error since I have encountered it a number of times, not only once, and I do not feel that it was possible to have manually run the Batch File later than needed each time.
Anyway, a possible workaround for this Issue might be the following one:
Create a Scheduled Task that first runs the Batch File and then runs the Microsoft Windows Backup and Restore Application. I do not know yet how to tell the Scheduler to automatically run the Application, but I can imagine that it might not be difficult to achieve this goal.
Then, whenever the manual Backup is needed to be performed, one can simply manually run the Scheduled Task. This way, this Issue might not reoccur, at least because the previously suspected behavior should be avoided.
I need to perform the backup manually because I am using Removable Disks as a Third Backup Solution. The First One is the ASUS Web Storage Cloud Provider and Synchronizer Application and the Second One is the File History Application run on an External Winchester Hard Disk Drive.
If anybody has a better solution for this Issue, then please let me know.
I'm learning how to use SSIS to create auto scheduled tasks. Everything went smoothly until the security policy has changed. Before whenever I wanted to connect to database, I just type in IP address, username and password in SSIS connection manager. Now I need to open up OpenVPN first which is still a manual process. So I really would like to put this bit into an SSIS task flow.
I had a search on Stackoverflow and over some other websites. Here is a link in stackoverflow. This example gives the idea that we can use VBA to setup the link. However that example shows we need a VPN connection name and VPN username which I don't really have and don't know where to find. Everytime I just right click on a .ovpn file and choose start openvpn on this config file. Also when I open that .ovpn file there is no command indicating which is my connection name or user name. Here is what it looks like when I open it up (there is no -auth-user-pass thing in this file and I can use this file without having to specify it):
client
dev tun
proto udp
remote a website here 1111
resolv-retry infinite
nobind
persist-key
persist-tun
comp-lzo
verb 3
ca ca.crt
cert office.crt
key office.key
askpass login.txt
Is it possible to find my connection name and user name? If I don't need connection name and user name, can I still write up VBA to build up an auto connection process?
I also notice there is an execute process task in SSIS. Is it possible that I can make use of it (because in my understanding obviously I use openvpn.exe to execute a .ovpn file)?
Please correct me if I'm saying something wrong since I'm new to both VPN and SSIS. And much appreciate for whoever can help me with this.
I don't know anything about OpenVPN, but I ran into a similar problem a while back with a server that insisted on an SSH tunnel using PuTTY as the only allowed connection method.
People who put these policies in place never think of the impact on SSIS, especially for unattended processing.
The solution I found was:
Find the command-line equivalent to the PuTTy client (which is called plink)
Have the detailed documentation to hand, because this is really difficult.
Have the SSIS package start plink.exe, passing various parameters (Note: don't use the Execute Process task, as that waits for the process to exit: launch the process in a Script task, and get the handle of plink.exe's ProcessID).
After a wait, have another Script task in SSIS connect to the plink process' InputBuffer, "typing in" a password. Yes, insecure and clunky, though you can store SSIS project parameters encrypted (this obviously requires SSIS 2012 or later)
The SSIS package can now connect to the server and do data stuff, as if it was a normal server without stupid access requirements.
At the end of the package, kill the plink.exe process.
Bear in mind, also, that on unattended execution (e.g. when scheduled in a job), your package will be running in a different security context from the one you work in to develop the package (depending on the SSIS service settings). If your access to OpenVPN has any dependency on ActiveDirectory, you may have to set up another OpenVPN account for the SSIS service's Logon user.
If this sounds horrible, nightmarishly complicated and unstable, it's because it is. Really, the TL;DR version is: tell whoever administers the server that if they really want you to access it using SSIS, they should put some more sensible access policy in place.
After working with my IT colleague, I think I solve the problem. Here are my answer and opinion to this question, and please correct me if I am wrong.
I think opening a VPN automatically is a case by case issue. In this case I am still using Execute Process Task in SSIS. Inside the task editor, what we are executing is not openVPN.exe, it should be a batch file. In the batch file (.bat file) it contains the command of opening .ovpn file with OpenVPN.exe
The process below gives you more detailed info.
The command line is what I use in the batch file:
#echo off
CD /D "C:\Program Files\OpenVPN\config"
start openvpn client.ovpn
TIMEOUT /T 10
exit
In SSIS execute process task
Hopefully this is useful
I set up a job to run a SSIS package in the SSIS Catalog, but the SSIS package doesn't start. If I right-click the package and run it, it completes successfully.
When I look in the execution history of the package in the Catalog, there is no sign that it the package was even started by the job.
I created a test package to make sure it isn't the specific package failing, so the package I'm testing with has a SQL Task that does a 'SELECT 1'. Still fails.
The job just stays in the "Executing" state until I stop the job.
I am running SQL Server 2014 Standard Edition, and running the job with a sysadmin account.
Why is the SQL Agent Job not running my package?
Edit: I had a look through this article but it either doesn't apply or I gave it a shot and it didn't work either.
Edit 2: I found an error under the SQL Server Agent Job History: Unable to terminate process 1850 launched by step 1 of job 0x5DAD5416FA09C445B82ABDBB49F75E38 (reason: Access is denied)
Does that mean my service account doesn't have enough permissions on the local machine? However, I AM able to run the job with DTEXEC using the service account without issues...
I have also faced this scenario many times but when I checked running the package manually,its completing successfully because I was using for each loop container and sequence container as well.In both cases for each loop and sequence were completing without validating other ones.So I checked precedence constraint and change it,Now it working and all the component ran successfully. Sometimes we miss to choose appropriate precedence constraint, there are many option like on Success ,failure,completion and then for you can choose values from Constraint ,Expression,Expression AND Constraint and Expression OR Constraint. Initially i was using Expression OR Constraint for success and now changed it to Expression AND Constraint, its working fine for me. You also need to do this,it will definitely work please try and let me know.
When you look at the connections in the SSIS package, did you put in a user id and password? If so, did you add a package password? The reason I ask is that if you don't add a package password, it will strip out things it thinks should be secure (like passwords). You might want to try putting the connection password in a configuration file, and add that file to the job. Either that or password encrypt the SSIS package but you will have to take special care when you run the package for the first time (or when you create the job) because you will need to provide the password that you used to encrypt the package.
Reboot fixes this issue. Reason is because if you look at this message. Unable to terminate process 1850. That is a windows process. I would say reboot. If you really like to enquire before rebooting. You could use sysinternals process explorer. Download it and use that to find process id 1850 which may be hung process in windows.
Thanks
Ali
What is the simplest way to schedule a batch file to run on a remote machine using Hudson (latest and greatest version)? I was exploring the master slave setup. I created a dumb slave but I am not sure what the parameters should be so that I can trigger the batch file in the remote slave machine.
Basically, I am trying to run 2 different batch files on two different remote machines sequentially, triggered from my machine (the master). The Step by step guide on the Hudson website is a dead link. There are similar questions posted on SO but it does not quite work for me when I use the parameters they mention.
If anyone has done something similar please suggest ways to make this work.
(I know how to set up jobs, and add a step to run a batch file etc what I am having trouble configuring is doing this on a remote machine using hudson in built features)
UPDATE
Thank you all for the suggestions. Quick update on this:
What I wanted to get done is partially working, below are the steps followed to get to it -
Created new Node from Manage Nodes -> New Node -> set # of Executors as 1, Remote FS root set as '/var/hudson', set Launch method as using JNLP, set slavename and saved.
Once slave was set up (from master machine), I logged into the Slave physical machine, I downloaded the _slave.jar from http://masterserver:port/jnlpJars/slave.jar, and ran the following from command line at the download location -> java -jar _slave.jar -jnlpUrl http://masterserver:port/computer/slavename/slave-agent.jnlp. The connection was made successfully.
Checked 'Restrict where this project can be run' in the Master job configuration, and set paramater as slavename.
Checked "Add Build Step" for adding my batch job script
What I am still missing now is a way to connect to 2 slaves from one job in sequence, is that possible?
It is fairly easy and straight forward. Lets assume you already have a slave running. Then you configure the job as if you are locally on the target box. The setting for Restrict where this project can be run needs to be the node that you want to on. This is all for the job configuration.
For the slave configuration read the following pages.
Installing Hudson as a Windows service
Distributed builds
On windows I prefer to run the slave as a service and let the remote machine manage the start up and shut down of the slave. The only disadvantage with this is, you need to upgrade the client every time you update the server Just get the new client.jar from the server, after the upgrade and put it on the slave. Then restart the slave and you are done.
I had troubles using the install as a service option for the slave even though I did it as a local administrator. I used then srvany to wrap the jar into a service. Here is a blog about it. The command that you need to wrap, you will get from your Hudson server from the slave page. For all of this to work, you should set up the slave management as jnlp.
If you have an ssh server on your target machine, you can use the ssl slave settings. These work for me like a charm. I use them with my unix slaves. So far the ssl option with unix is less of an hassle, than the windows service clients.
I had some similar trouble with slave setup and wrote up this blog post - I was running on Linux rather than Windows, but hopefully this will help.
I dont know about how to use built-in hudson features for this job - but in one of my project builds, i run a batch file that in turn uses PSTools
to run the job on a remote server. I found PS tools extremely easy to use - download, unpack and run the command with the right parameters, hence opted to use this.