I have a requirement. We have a FTP server where the data will change everyday. There are around 9 files. Each file is data for MS SQL ETL. Now what i want to do is. As soon as file arrives in FTP location. Powershell should read that date modified of the file and trigger the job in SQL Server. Is that possible with powershell?
Challenges involved
Limited with technology (Only powershell and TSQL Can be used)
Old file (Day - 1) Data. to get each file completely replace it
will take 15 Minutee, before that job should not triggered.
Need your inputs on this.
You may want to try a FileSystemWatcher. A similar-ish question has been asked before, so I won't try to regurgitate the answer:
Watch file for changes and run command with powershell
See also on MSDN:
FileSystemWatcher class
FileSystemWatcher events
Related
Hopefully this question is unique enough not to be a duplicate. I have a PowerShell script which does two things.
Inserts records into a SQL Server table
Writes text to a text file
For the purpose of this post, I have simplified the script. On my computer, the script is located at C:\Temp\ssis.ps1. Following is the contents of the script.
DTEXEC.EXE /F "C:\Temp\ssisjob.dtsx"
$date = Get-Date
Write-Output "This PowerShell script file was last run on $date" >> C:\Temp\test.txt
When I manually run this PowerShell script, records are inserted into the SQL Server table, and a line of text is written to the test.txt file. If I schedule this script to run using Windows Task Scheduler, a new line of text is written to the text file, but the records are not inserted into the SQL Server table. This tells me that Windows Task Scheduler is able to run the PowerShell script. However, for some unknown reason, Windows Task Scheduler seems to not want to run the SSIS job (DTEXEC.EXE) part of the script. Event Viewer confirms there is an issue with the SSIS job. I am running Microsoft SQL Server 2014, Developer Version.
In my task, on the Actions tab, the Add arguments field has the following reference: C:\Temp\ssis.ps1. Task Scheduler is configured to run with the highest privileges.
I have tried all of the following Execution Policies in PowerShell. Regardless of the Execution Policy I select, my experience does not change.
Bypass
Unrestricted
RemoteSigned
The History tab in Task Scheduler has information events, but no error events.
I do not have the permission to view the SQL Server logs (this is a production server).
I have been debugging this issue for a few weeks, and I have read numerous posts here on Stack Overflow, yet I still cannot seem to find the answer, so hopefully I have done my due diligence before making a new post here. I could add some additional observations, but I do not want my post here to get extensively long. If anyone has any hints or tips or insight that might lead me down the right path, it would be greatly appreciated.
Here is the solution I came up with. Instead of exporting the file to Excel, I exported to a flat file (txt file). Also, using Nick McDermaids excellent recommendations, instead of using PowerShell in Task Scheduler, I started the dtexec.exe file in Task Scheduler.
Task Sheduler Actions Tab
Keep the action as Start a program
In Program/script, type dtexec.exe
In Add arguments, type /f "C:\path\to\example.dtsx
Leave the Start In box empty
I need to import a flat file daily. The file changes its name every day. After the file is processed, it needs to be moved to another folder.
I noticed I can schedule jobs in the SQL Server Agent, and that I can tell it to run every hour or so and that I am able to add CMD commands to it.
The solution I found was to run a script to check if the file exists, since the folder should be empty or have at least one file.
If the file exists, the script renames the file to one used in the SSIS package and then it runs the SSIS package.
After the whole thing is done, it should rename the file again based on today's date and move it to another folder.
If the file does not exist, then it should do nothing and wait another hour or so to run again.
What's the best solution to this scenario? Is the script a good idea? Maybe is it possible to add the if/else -for the file exists- into the SSIS package? Or even make the script run from the SSIS package itself instead of adding it to the Server Agent?
EDIT:
It seems I was a little naïve, it's possible to run VB scripts from the server. Would that be the recommended solution? It does solve my problem, but I'm just wondering if it's a good idea.
This solves all my questions:
http://www.sqlservercentral.com/articles/Integration+Services+%28SSIS%29/90571/
I'm really bad at SQL and couldn't find anything near what I really need. I'm trying to create a Stored Procedure that should run each night to check if records in my database have an equivalent file on a server with all our data.
Example: Record with a mp4 has: [Spotnumber] -> 0000001. Then my procedure should check (not locally) if the file exists on the other server with this number.
Also the place where it should look could be fore exemple (not locally) C:/Spots. And in this directory there'll be subdirectories like: 2013, 2012, 2011. It should check in each directory if it doesn't exists.
For this I was thinking to make something like this: Single check. But this one searches locally and already has the url in a table-field. This won't be possible for mine.
So my question is: Is it even possible to do this with just a SQL procedure? If yes how should I make it check all the files on another server (what path should I use?) + How can I make it check for each record in each subdirectory?
I would suggest another approach.
Instead of using Sql server to check if the file exists then update the db.
Why don't you use a powershell script checking if a file exist, then in this powershell script update the database. With a little search on google you can find all functions on microsoft blogs explaining how to check if a file exists and update file.
Another solution, you could create an assembly in your database with a .net language and work with that.
Last possibility, i think it can be possible too with SSRS.
If you really need to do that with tsql, you should allow xcmd command on your server, then use xcmd... but it means every body could use xcmd. It's not designed too and not suggested :)
I want to create a SQL Server SSIS package where I can watch a folder and once I have all (20 files) the required files I want to execute a sql statement. The files may come at different times and sometime they will be in csv and sometime they can come in zip. I know ssis has a wmi event watcher task but I’m not sure how I can specify to look for all 20 files. I guess I want wmi event watcher to look into that folder every 30 minutes and once it sees all the files move to the next step (execute sql task). Can someone tell me how I can specify the file name in wmi event watcher task? Thanks.
This article seems relevant to your plan. You need to create the proper WQL code.
http://blogs.technet.com/b/heyscriptingguy/archive/2007/05/22/how-can-i-monitor-the-number-of-files-in-a-folder.aspx
("ASSOCIATORS OF {Win32_Directory.Name='C:\Logs'} Where " _
& "ResultClass = CIM_DataFile")
I'm not sure how that will behave in the WMI Event watcher though. Have you looked at the docs for the SSIS task?
Here is a more step-by-step approach:
http://microsoft-ssis.blogspot.com/2010/12/continuously-watching-files-with-wmi.html
Some good points there, even if it doesn't address the pesky 20 file requirement.
You could also have a powershell script on the server monitor the files and then chuck them into a subfolder when they are all there, which SSIS would be monitoring.
Here is a doc page showing how to specify one file:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa394594(v=vs.85).aspx
With that, I'm sure you could set up a chain of WMI checks in your SSIS package.
I have a PowerShell script do the following tasks:
Loop a big database table
Generate text file
Zip the text file
FTP upload the zipped file
Write to the log table
The step generating text file may take short or longer time depends on the data. And the FTP uploading time takes a while. So I want to make at least these two steps asynchronous. Is SQL Server Service Broker a viable choice? Is there any example? Any other options?
You cant make them aysnc in PowerShell, but you could use the Start-Job cmdlet to put them on another thread and wait until they complete.
Using Service Broker will will by default make them work asynchronously. The tricky thing would be if you still want to run some of then sequentially, for which you need to add a conversation-group id for them.