Curious if this is feasible. I am currently in the process of building a number of SQL Server agent jobs. These jobs are just SFTP jobs that pass files between the servers of 2 different clients, making a pit-stop at my local server for some pre and post processing. Yes, this setup is from one standpoint unnecessarily complicated, but it is necessary from a security standpoint. All of these jobs have identical structure:
SFTP a file from the client1server to the local server.
Run an executable on the file
SFTP the processed file to client2server.
Wait a predetermined amount of time so that client2 can perform their query on the input.
SFTP the response file from client2server to the local server.
Run a second executable on the file.
SFTP the processed response file back to client1server.
Pretty straight forward.
There are only a handful of values that change between each job:
- Input/output file path on client1server
- Input/output file path on client2server
- Directory on local server
These jobs are not complicated, so If necessary I can just create them all by hand. It seems like an unnecessary amount of work though. I had the thought that maybe I could create a stored procedure that would generate the SQL script that creates the job, and that stored procedure could simple accept the variables that change from job to job. Is this feasible?
Broadly, heres what I'm thinking:
CREATE PROCEDURE create_ftp_interface_job
#client1input_fp nvarchar(100),
#client1output_fp nvarchar(100),
#client2input_fp nvarchar(100),
#client2output_fp nvarchar(100),
etc...
AS
<SQL Script for creating SQL Server Agent jobs, with parameters inserted>
GO
I've tried an early version of this, and I seem to be having trouble referencing the variables I declare in the stored procedure definition inside of the SSA job script. I came here to ask if what I'm attempting is feasible, and I just have a run of the mill reference error, or if what I'm doing is not allowed.
You can use msdb.dbo.sp_add_job, msdb.dbo.sp_add_jobstep, msdb.dbo.sp_update_job, msdb.dbo.sp_add_jobschedule and msdb.dbo.sp_add_jobserver to create and manage SQL Agent Jobs programatically.
In fact there are even more stored procedures here which relate to managing SQL Agent Jobs.
Related
Is there an alternative to xp_cmdshell BCP to write to a file from MSSQL?
Some context: This is to be built into a stored procedure called by an external program. I do not have the luxury to work through executable files, export functions of ssms, or any such things which require more than the calling of this stored procedure.
The reason; there's a lot of odd stuff on this server to do with user rights, and I'm not the SA. I cannot create the ##xp_cmdshell_proxy_account## (let alone assign credentials to it), and xp_cmdshell 'whoami' returns a user noone has ever seen or heard from. I've tried creating a new user based on an existing windows user and granting xp_cmdshell execute rights to it, but this still did nothing. I'm not sure if I don't have the rights or if it's something else.
So long story short, I'm fed up with trying to get this to work on this environment and am looking for an alternative. Is there one?
Write a SQL Agent Job and kick it off with sp_start_job. You can control the identity the job uses with a SQL Agent Proxy.
Or write an SSIS package, deploy it to the SSIS Catalog and run it from your stored procedure.
System Error: Cannot bulk load because the file "XYZ.txt" could not be opened. Operating system error code 1311(There are currently no logon servers available to service the logon request.)
I have a stored procedure in SQL Server 2008 R2 which is using Bulk Insert command to load data from txt files into SQL Server tables. These files are on a shared folder on a drive located on different domain. I have full access to the drive. I tried copying files to different directory on that drive, moving files, deleting files and everything works.
When I execute the stored procedure from a ssms session from local computer it works like a champ. It is able to open the files on shared drive, read it and load the data into the SQL Server tables without any issue. When I call the stored procedure from a SQL Server Agent job, it throws this error.
SQL Server Agent is using the account which is very powerful with lot more permissions than mine. But the job fails.
To find a workaround I created a SSIS package which calls the stored procedure from "Execute SQL Task". It uses Windows authentication to connect to the database. I tried executing the package and it ran successfully. It is able to upload the data from txt to table.
So, then I created a SQL Server user with my account details, and then used that credentials to create a proxy with ssis sub system. I then scheduled the job to execute the step with newly created proxy to see if it can upload the data. But it failed with the same error.
I am confused what am I doing wrong..? I even added myself to bulkdmin role and ran the job again with no success.
I'll appreciate if someone can help.
Thanks.
Just out of curiosity I tried replacing Bulk Insert command with BCP. For some reason BCP worked. It is able to Open the files on network drive and read through it to insert the data in sql server tables. I can even call the same stored proc from sql agent job and it works perfectly fine. I didn't need to use SSIS package to solve this.
I was wondering if it is possible for SQL Server to check a directory for files and run a stored procedure. I did some research and found this, but I am wondering if there is a way to do what I want WITHOUT SSIS.
EDIT: After reading my post, I realized I should have been more specific. Is there a way to AUTOMATICALLY or set SQL Server to check for files in a directory and run a stored procedure?
You can use xp_cmdshell to run file related commands. To get a directly listing:
exec xp_cmdshell 'dir *.csv';
You can also use bulk insert to load a file from disk into a table and take actions based on the loaded contents.
Normally you'd use the File Watcher Task with SSIS. But you can also use SQL Server Agent to schedule a task for periodic execution, schedule a task with Windows Task Scheduler, or configure a stored procedure to runs at startup with sp_procoption that pauses (using waitfor) between processing times.
I have a collection of .sql files containing ddl for various database objects.
For example:
User.sql
Group.sql
GroupUser.sql
Is there a way I can use a stored procedure to easily/elegantly load/execute these files in sequence? For example, GroupUser.sql depends on the existence of User and Group, so I need to execute the .sql files in the order listed above.
I know I could concatenate the contents of all of the .sql scripts above into a stored procedure, but I would prefer to take a more modular approach. I could also put each script into its own stored procedure but I'd rather not clutter the stored procedure collection in my app database with DDL setup scripts.
From SSMS, go to the "Query" menu, and select "SQLCMD Mode". Once there, you can run commands like this. I script out stuff like this all the time.
use test
go
:r "D:\SomeDataDirectory\SomeSQLFile.sql"
EDIT: Didn't see you wanted to do this within a stored procedure. That's a bit of a dicey proposition. Assuming you have permissions to execute it, you could put the same SQLCMD code in calls to xp_cmdshell, but in many circumstances that won't be an option for you unless you've got admin-like permissions on the server.
Ideally would like to run something from a SQL query or SQL agent job to FTP upload a file to an external site, but cannot use xp_cmdshell.
Yes. You need to split your work into two separate tasks:
How to run executable or a batch program from within SQL Server without resorting to xp_cmdshell.
An example of how to do it can be found in:
https://www.mssqltips.com/sqlservertip/2014/replace-xpcmdshell-command-line-use-with-sql-server-agent/.
You should modify this example to suit your particular needs. Suggested stored procedure would:
run command passed as a parameter in created on-the-fly SQL job (indicate CmdExec subsystem)
wait for SQL job completion (query msdb.dbo.sysjobactivity) or kill the job if predefined timeout value has been reached
return results of job execution (query msdb.dbo.sysjobhistory)
delete the job
Note: Full version of SQL Server required. If you are using express version, you would have to manually define a windows scheduled task.
How to send a file via ftp using a batch program.
Please see:
How to ftp with a batch file?