In my Informatica mapping, when an SP is called via an unconnected stored procedure transformation, the workflow succeeds. However there is a divide-by-zero error in the SP and ideally the workflow should fail. The source and target used in this mapping are dummy flat-files.
However, when I use dummy tables instead for source and target, with a connected SP transformation this time, the error bubbles up successfully.
Any idea why this would happen? Why does the error show only with source table and connected SP transformation, and not with flat-file source unconnected SP transformation?
I've been dealing with it. One thing you can try is using SET NOCOUNT ON in Stored Procedure. Otherwise the x rows affected message can be treated by PowerCenter as the indicator that Stored Procedure has been successfully executed. No matter what else would be returned.
The other thing is it depends on whether you're using Native or ODBC Connector - with the latter the errors are not properly escalated. You can read more about my tials here: http://powercenternotes.blogspot.com/2014/09/ms-sql-server-stored-procedure-error.html
Related
I am trying to execute (call) a SQL Server stored procedure from Infa Developer, I created a mapping (new mapping from SQL Query). I am trying to pass it runtime variables from the previous mapping task in order to log these to a SQL Server table (the stored procedure does an INSERT). It generated the following T-SQL query:
?RETURN_VALUE? = call usp_TempTestInsertINFARunTimeParams (?Workflow_Name?, ?Instance_Id?, ?StartTime?, ?EndTime?, ?SourceRows?, ?TargetRows?)
However, it does not validate, the validation log states 'the mapping must have a source' and '... must have a target'. I have a feeling I'm doing this completely wrong. And: this is not Power Center (no sessions, as far as I can tell).
Any help is appreciated! Thanks
Now with the comments I can confirm and answer your question:
Yes, Soure and Target transformations in Informatica are mandatory elements of the mapping. It will not be a valid mapping without them. Let me try to explain a bit more.
The whole concept of ETL tool is to Extract data from the Source, do all the needed Transformations outside the database and Load the data to required Target. It is possible - and quite often necessary - to invoke Stored Procedures before or after the data load. Sometimes even use the exisitng Stored Procedures as part of the dataload. However, from ETL perspective, this is the additional feature. ETL tool - here Informatica being a perfect example - is not meant to be a tool for invoking SPs. This reminds me a question any T-SQL developer asks with his first PL-SQL query: what in the world is this DUAL? Why do I need 'from dual' if I just want to do some calculation like SELECT 123*456? That is the theory.
Now in real world it happens quite often that you NEED to invoke a stored procedure. And that it is the ONLY thing you need to do. Then you do use the DUAL ;) Which in PowerCenter world means you use DUAL as the Source (or actually any table you know that exists in the source system), you put 1=2 in the Source Filter property (or put the Filter Transforation in the mapping with FALSE as the condition), link just one port with the target. Next, you put the Stored Procedure call as Pre- or Post-SQL property on your source or target - depending on where you actually want to run it.
Odd? Well - the odd part is where you want to use the ETL tool as a trigger, not the ETL tool ;)
Having seen other questions with answers that don't totally address what I am after, I am wondering how in SSIS to use an OLE DB Command transformation to do an Insert and immediately get the resulting primary key for each row inserted as a new column, all within the same Data Flow Task. That sounds like it should be a common, built-in, fairly simple thing to ask for in SSIS, right?
So the obvious first choice for me would be to use an OLE DB Command where I do a SELECT and include an OUTPUT clause in my command:
INSERT INTO dbo.MyReleaseTable(releaseDate)
OUTPUT ?=Inserted.id
VALUES (?)
Only I can't figure out how to do this in an OLE DB Command (with an output) and it not complain. I've read about using stored procedures to do this, so am I required to use a stored procedure if I want to do this?
Let's say this won't work. I could use a Script Transformation and execute direct SQL in that, right? Well if that's what I must do, then the line between using custom code and SSIS block-components gets blurred and I am tempted to throw SSIS away and just do the whole ETL in code.
Then I hear talk about using an Execute SQL task. So now I can't even do 1 data flow within 1 data flow task? Am I getting that right? I'd like to keep 1 single data flow contained within 1 data flow task and not have to break my 1 flow out between separate tasks.
If it turns out that this seemingly simple data flow objective is not built into SSIS then I will consider dumping SSIS altogether. Talend has a free ETL offering, don't they?
Well, this can be done with SSIS inside DataFlow, but with some tricks. You need to create a stored procedure with input and output parameters and reuse it in DataFlow, as described here, fetching result value.
Drawbacks of this approach:
You need to create a Stored Procedure
Each row is processed with SP, which causes implicit transactions, instead of batch processing. This can slow down your package.
Solution without performance penalty - do it in two DataFlows, first doing value insert into some temp table, and the second DF - doing SQL MERGE command at OLE DB source and handling output data as you wish. All this inside transaction, handled either by MSDTC or by your own.
I have an SSIS package with a data flow task. The OLE DB source has an execute proc statement. It fails while saving with below error message.
an OLEDB record is available... The metadata could not be determined because the statement 'select appname....' in procedure is not compatible with the statement 'select appid....' in procedure
This proc has several select statements and returns the appropriate result set as per parameters passed. Any pointers to bypass this error?
So you're saying that the SP will return different meta data depending on the parameter passed? SSIS doesn't like this - it can't update the meta data dynamically at run time. i.e. if you create a package that splits or sorts on a certain column, then you run the SP and it doesn't return that column, or the same column is a different data type, what should SSIS do? It can't automatically work it out.
I suggest you create a data source for each possibility of result set returned and conditionally execute each on as required.
In short SP's returning optionally different datasets is often not a good idea, definitely not from an ETL perspective.
Here is some code that shows how to create dynamically built output, (you could use the same method with just one output), but you'll still face the same problems downstream.
http://www.codeproject.com/Articles/32151/How-to-Use-a-Multi-Result-Set-Stored-Procedure-in
I ran into this issue as well. In my case, the result returned looked identical no matter which branch was executed, the difference was just in how that result was obtained (including different source tables). I simply executed all the cases with a union, and each "where" clause included the conditions for its execution instead of using "if" logic to choose a query.
These components have the ability to retrieve multiple result sets (e.g. from a stored proc) in one go, and using D5 I can successfully use NextRecordSet to get at the second and subsequent ones from a Sql Server.
However, this only works if I specify the cursor location as clClient; if I use clServer, I get a "Does not return multiple result sets" error. Is this an inherent limitation (e.g imposed by the MDac layer on the client) or can multiple recordsets be successfully retrieved from a server-side cursor?
It is an inherent limitation to server side cursors. As stated in following MSDN link:
Server cursors cannot be used with statements that generate more than
one recordset.
This restriction applies to all statements described in Generating
Multiple Recordsets. For more information, see Generating Multiple
Recordsets. If a server cursor is used with any statement that
generates multiple recordsets, an application can return one of the
following errors:
Cannot open a cursor on a stored procedure that has anything other than a single SELECT statement in it.
sp_cursoropen. The statement parameter can only be a single SELECT statement or stored procedure.
Recently I noticed that a stored proc we are trying to profile failed to appear in the profiling output.
After adding in SP:StmtStarting and SP:StmtCompleted events, I noticed the TextData reported as
-- Encrypted text
.. but the stored procedure is not encrypted.
This has only recently started happening - we used to profile this SP perfectly fine, and I can't figure what has changed.
Any suggestions would be gratefully received.
UPDATE: The SP is definitely not encrypted. I've created new SP's on the box, and I see SP:BatchStarting event with the new SP's name. With the old SP, I don't see the BatchStarting event, but I do see the statements within the SP executing.
However I need to see the values of the parameters the SP is being called with, as they are table types. Originally I could see the table types being instantiated and populated before the SP is called.
So I figured this out in case anyone finds it useful.
I have table type parameters to this stored procedure. One of the parameters is passed a lot of data (i.e. a C# DataTable with >5000 rows). Without this quantity of data the stored proc profiled fine.
I guess there must be some cut-off at which point Profiler does not show all of the data being passed in.
Someone has altered the stored procedure and added the 'WITH ENCRYPTION' hint, which will cause this behavior. Alter the stored procedure and remove that hint and you'll start seeing the text of the proc again.
Also to note, if you don't have the original code, you will not be able to decrypt the text of the proc to issue the ALTER statement, so hopefully you have that handy.
Here's a decent run down of this option: Options for hiding SQL Server code
Moving the Trace Properties from the default of OnlySP(<your database here>)(user,default) to TSQL or TSQL_Replay unveiled the SQL being used for me, ... Go to File|Properties... and change the [Use the template:] drop-down combobox.