I am running a daily job in MSSQL agent, that runs a package to export a certain table to an excel file. After completion it sends me an email.
It happened couple of times that the table was empty and it exported 0 rows.
I did not receive a job failed notification, because the job didn't fail.
I would like to receive a failed notification when it's exporting 0 rows.
How would you achieve this?
Thanks
There are a number of ways to force a package to return a failure (which will then cause the calling Agent job to report a failure).
One simple way would be to put a Row Count task between the source and destination in the data flow task that populates the spreadsheet. Assign the value to a variable, say #RowsExported.
Back on the Control Flow tab, if there's more to the package, put a condition on the precedent constraint leading to the rest of the work where #RowsExported > 0 so the rest of the package will only continue if there were rows sent. Whether or not there's more to the package, add a new precedent constraint coming off the data flow with a condition #RowsExported == 0. Send that constraint to an Execute SQL task that just contains the script SELECT 1/0;.
Now, if zero rows are exported, the package will throw a division by zero error, and the calling job will fail.
There are other ways, of course, but this one can be implemented very quickly.
Related
I have an SSIS package that consist of 2 main blocks within Begin and Commit/Rollback transaction blocks
1. Truncate tables (with truncate table query)
2. Import Data (Import data from flat file and insert to these truncated tables)
When I run the package the job is getting hung. The activity monitor shows that there is a LCK_M_SCH_S lock created that blocks further execution.
Sometimes this work and sometimes not.
To add, if I truncate these table separately and run the package removing the truncate block it executes fine.
Also to add there are not just one Import Data component. We have around 6 import data component for 6 different table. For time being i kept only one in the screen shot
,
Looking at your screenshots, the first thing I'd verify is that the property RetainSameConnection is set to true on your OLE DB Connection manager (right click on CM, select Properties, find RetainSameConnection). The default for this is False.
If that resolves the issue, then the root cause was you had two requests in different transactions attempting to modify the same resource.
If you had already switched the Connection Manager's property to true, then my next guess would be that you want to set the DelayValidation property for the Data Flow "Import Data" to True.
If that resolves the issue, then the root cause was the component was attempting to validate the metadata for the target table and was getting blocked by the truncate statement (or vice versa). Setting DelayValidation will prevent the package from validating that specific task until the last possible second - giving other processes time to get out of the way. This seems less likely but it's the only other opportunity for the package to be blocking itself.
we have a requirement where SSIS job should trigger based on the availability of value in the status table maintained,point to remember here that we are not sure about the exact time when the status is going to be available so my SSIS process must continuously look for the value in status table,if value(ex: success) is available in status table then job should trigger.here we have 20 different ssis batch processes which should invoke based on respective/related status value is available.
What you can do is:
Scheduled the SSIS package that run frequently.
For that scheduled package, assign the value from the table to a package variable
Use either expression for disabling the task or constraint expression to let the package proceeds.
Starting a SSIS package takes some time. So I would recommend to create a package with the following structure:
Package variable Check_run type int, initial value 1440 (to stop run after 24 hours if we run check every minute). This is to avoid infinite package run.
Set For Loop, check if Check_run is greater than zero and decrement it on each loop run.
In For loop check your flag variable in Exec SQL task, select single result value and assign its result to a variable, say, Flag.
Create conditional execution branches based on Flag variable value. If Flag variable is set to run - start other packages. Otherwise - wait for a minute with Exec SQL command waitfor delay '01:00'
You mentioned the word trigger. How about you create a trigger when that status column meets the criteria to run the packages:
Also this is how to run a package from T-SQL:
https://www.timmitchell.net/post/2016/11/28/a-better-way-to-execute-ssis-packages-with-t-sql/
You might want to consider creating a master package that runs all the packages associated with this trigger.
I would take #Long's approach, but enhance it by doing the following:
1.) use Execute SQL Task to query the status table for all records that pertain to the specific job function and load the results into a recordset. Note: the variable that you are loading the recordset into must be of type object.
2.) Create a Foreach Loop enumerator of type ADO to loop over the recordset.
3.) Do stuff.
4.) When the job is complete, go back to the status table and mark the record complete so that it is not processed again.
5.) Set the job to run periodically (e.g., minute, hourly, daily, etc.).
The enhancement hear is that no flags are needed to govern the job. If a record exists then the foreach loop does its job. If no records exist within the recordset then the job exits successfully. This simplifies the design.
I need to create a trigger on one of the sysjobxxx tables in MS SQL Server which when a record of a job is inserted, needs to be triggered.
For eg. when jobA is executed, some of the sysjobxxx tables (sysjobhistory, sysjobs, sysjobsteps ) get records inserted. My trigger needs to be
built on the table that inserts the success/failure info of jobA. I'd thought that sysjobhistory would be the one but when I tested with a dummy job that failed, it inserted 2 records, instead of 1 - this would run my trigger (upon insert) twice, instead of just once.
What I'm trying to accomplish is to get the full description of job failure, everytime a job fails and is inserted into the sysjobxxx tables.
I know there's an email notification that can be sent out but the description of failure is truncated (to 1024 chars) and unless a full 'View history' is done on the job, it's not possible to see the complete job failure description.
Is there a way I can create a trigger on one of the sysjobxxx tables, that upon a job record insertion, checks a column (I don't know which one
indicates failure of the job), then sends out an email (directly from within the trigger via sp_sendmail or calls another stored proc that then
executes sp_sendmail) to a list of recipients, with the full job failure description?
I'm trying to create SSIS package which will periodically send data to other database. I want to send only new records(I need to keep sent records) so I created status column in my source table.
I want my package to update this column after successfuly sending data, but I can't update all rows wih "unsent" status because during package execution some rows may have been added, and I also can't use transactions(I mean on isolation levels that would solve my problem: I can't use Serializable beacause i musn't prevent users from adding new rows, and Sequence Container doesn't support Snapshot).
My next idea was to use recordset and after sending data to other db use it to get ids of sent rows, but I couldn't find a way to use it as datasource.
I don't think I should set status "to send" and then update it to "sent", I believe it would be to costly.
Now I'm thinking about using temporary table, but I'm not convinced that this is the right way to do it, am I missing something?
Record Set is a destination. You cannot use it in Data Flow task.
But since the data is saved to a variable, it is available in the Control flow.
After completing the DataFlow, come to the control flow and create a foreach component that can run on the ResultSet varialbe.
Read each Record Set value into a variable and use it to run an update query.
Also, see if "Lookup Transform" can be useful to you. You can generate rows that match or doesn't match.
I will improve the answer based on discussions
What you have here is a very typical data mirroring problem. To start with, I would not simply have a boolean that signifies that a record was "sent" to the destination (mirror) database. At the very least, I would put a LastUpdated datetime column in the source table, and have triggers on that table, on insert and update, that put the system date into that column. Then, every day I would execute an SSIS package that reads the records updated in the last week, checks to see if those records exist in the destination, splitting the datastream into records already existing and records that do not exist in the destination. For those that do exist, if the LastUpdated in the destination is less than the LastUpdated in the source, then update them with the values from the source. For those that do not exist in the destination, insert the record from the source.
It gets a little more interesting if you also have to deal with record deletions.
I know it may seem wasteful to read and check a week's worth, every day, but your database should hardly feel it, it provides a lot of good double checking, and saves you a lot of headaches by providing a simple, error tolerant algorithm. Some record does not get transferred because of some hiccup on the network, no worries, it gets picked up the next day.
I would still set up the SSIS package as a server task that sends me an email with any errors, so that I can keep track. Most days, you get no errors, and when there are errors, you can wait a day or resolve the cause and let the next days run pick up the problems.
I am doing a similar thing, in my case, I have a status on the source record.
I read in all records with a status of new.
Then use a OLE DB Command to execute SQL on each row, changing
the status to "In progress"(in you where, enter a ? as the value in
the Component Property tab, and you can configure it as a parameter
from the table row like an ID or some pk in the Column Mappings
tab).
Once the records are processed, you can change all "In Progress"
records to "Success" or something similar using another OLE DB
Command.
Depending on what you are doing, you can use the status to mark records that errored at some point, and require further attention.
In SSIS 2008 I have a Script Task that checks if a table exists in a database and sets a boolean variable.
In my Data Flow I do a Conditional Split based on that variable, so that I can do the appropriate OLE DB Commands based on whether that table exists or not.
If the table does exist, the package will run correctly. But if the table doesn't exist, SSIS is checking metadata on the OLE DB Command that isn't being run, determine the table isn't there, and failing with an error before doing anything.
There doesn't seem to be any way to catch or ignore that error (e.g. I tried increasing MaximumErrorCount and various different ErrorRowDescription settings), or to stop it ever validating the command (ValidateExternalMetadata only seems to affect the designer, by design).
I don't have access to create stored procedures to wrap this kind of test, and OLE DB Commands do not let you use IF OBJECT_ID('') IS NOT NULL prefixes on any statements you're doing (in this case, a DELETE FROM TableName WHERE X = ?).
Is there any other way around this, short of using a script component to fire off the DELETE command row-by-row manually?
You can use Script component to execute DELETE statement for each row in input path but that might be very slow depending on number of rows to be deleted.
You can:
Store PKs of records that should be deleted to a database table (for instance: TBL_TO_DEL)
Add Execute SQL Task with SQL query to delete records by joining TBL_TO_DEL with table that You want to delete records from
Put precedence constraint on path between your data flow and Execute SQL task (constraint based on your variable)
This solution is much faster than deleting row by row.
If for some reason You can't create new table, check my answer on SSIS Pass Datasource Between Control Flow Tasks to see other ways to pass data to next data flow where You can use OleDb source and OleDb command. Whichever way You choose, key is in constraint that will or will not execute following task (Execute SQL task or data flow) depending on value in variable.
Note that Execute SQL task will not validate query and as such will fail at runtime if constraint is satisfied and table doesn't exist. If You use another Data Flow instead of Execute SQL Task, set DelayedValidation property to true. It means that task will be validated at the moment prior to executing particular task, not anytime earlier.