I've got an SSIS package - the primary function of which is to precalculate some data and invoke an parameterized SSRS report. The SSRS report has multiple datasets, that it retrieves through stored procedures. It takes around 2-2.5 seconds to generate.
When I loop through the report within the package, the loop obviously executes one report at a time. To speed up this process, I split up the dataset into two and tried passing each dataset into its own loop container. The problem is that although the loops process simultaneously, the step at which the report is generated (script task) halts the process for the other loop - that is, while one report is generating, the other waits.
Given this, it seems that SSRS locks and only allows for one execution at a time. The profiler showed "sp_WriteLockSession" being invoked but according to this it appears that that is based on the design. I've also read up on the "no lock" hint but I'm not sure that's the route I want to go down either.
I'm not sure if I'm approaching this in the right way. Am I missing something? The only other thing I can think of is to create a second report and invoke that instead but if its locking due to the underlying datasets, then I'm really not sure what else to do. The datasets are primarily just select statements, with a couple of them inserting one row into a single table at the very end.
I'd appreciate any advice, thanks in advance!
Related
Firstly, I am new to SSIS and not sure of what format to explain the problem in a repeatable way for testing
So I have a task that loops through a folder looking for flat files and if in the correct format loads them into a SQL server staging table. This seem to work correctly
The next bit is the strange bit. I have a split for success and a split for failure (for user notification etc) on both legs I run a SQL task to count the number of rows in the staging table to process. IF it greater than zero I want to run my SQL stored procedure that handles the load to into the main database table for the data, then clean up.
In my debug email I see the number of rows is greater than 0 but the task does not proceed. I can not understand why. I have tried a number of constrain combinations (On completion, On Success, On Success and RowsToProcess>0).
I have also tried removing and adding my SQL Task and remapping.
(BTW all my tasks function in SSMS etc)
Nothing seems to work The only thing is that 2 branches re-join at this point, but surely that would not affect it, would it? (see below original screen shots)
Here is my control flow, If I have missed anything please add a comment and I will supply the information if I know it
My results from executions
On further testing two executables does seem to be the problem! So, additional questions.
Is this expected behaviour that two constraints joining back to a SQL task stops the flow in SSIS (in BPA task centre it does not)?
If, this is the case does this mean you have to repeat code in parallel? (for example if I was to run a clean up script at the end of my flow I would have to write it tice once for success leg and once for failure leg). This seems inefficient, am I missing something?
I think I've found the answer you are looking for here (go give it an upvote)
I was not aware of it, but basically, it says that bot branches must complete successfully (logical AND) in order for the flow to proceed, which won' happen as the right branch in this case won't even run.
The default behavior can be changed in the "constraint properties" (right-click on the arrow that links the blocks) under the "Multiple Constraints" section.
That said
I've never used this method, but that's up to the use case. I would have used "sequence containers" to avoid replicating blocks or multiple constraints, which is not the same but works as long as you don't have to pass data from one block to the other.
In your case, I'd put all the blocks inside a sequence container except the last 2, and after the container completes execute the last two as they must be always run after the previous block of operations
note about containers:
Containers do not output data but are useful to group operations logically, and you can even run just a container instead of the whole package (from VS).
Seems today I've learned something new too...
Hope this helps you solve your issue
Have a SQL Server stored procedure where I want to loop through a set of options that I read from a table. So say a table has 100 options. My stored procedure will loop through these options and for each option I need to do some checks - by querying few specific tables based on the option and flag a status related to it.
Is it possible for me to split the for loop such that row 1 -50 are processed in one loop and row 51-100 in another loop and I am able to run both of these in parallel?. I see a way where you can run multiple stored procedure in parallel through a SQL job or other means but not able to see if I can get a for loop to execute in parallel by splitting it.
Treating your question as academic, and not considering whether a set-based solution might exist, since there isn't nearly enough information to do that.
No you can't do this in a single loop (or in two separate loops for that matter) using standard TSQL, because TSQL is synchronous. Even if you "split" the loop, the second procedure call could not start until the first call finished. They would not run in parallel.
To run two loops in parallel, you would have to introduce some other language. The results of this search turned up quite a few ideas but the first few I looked at had lots of warnings of pitfalls and unexpected results. Up to you if you want to experiment with any of them.
We have a list of stored procedures (more than 1000) in a table which need to be executed every morning.
The stored procedures do not have any dependency with each other.
We have tried while loop and cursor it used to takes a lot of time in execution.
We taught of creating job for each stored procedure and call them using sp_start_job (sp_start_job is called in async manner) we got level of parallelism.
Problem arise when a new stored procedure is added to list and it became huge.
some time ppl missed to create job related new stored procedure
DB got bombarded with a no of jobs (manageability issue for DBA)
Note: list of may altered any day (stored procedures can be added or removed from list).
If the SPs run for longer, I would have categorized the 1000 SPs into 5-10 numbers, then 1 SSIS package for each category and then Agent Jobs for each package. Then, schedule those jobs at same time.
There are many ways like Loops, Scripting and multiple factors to achieve it. You can test with different ways and go with the best one.
Note: Performance of the SSIS execution depends on your Memory, Processor and Hardware.
Adding to # Nick.MacDermaid - you can utilize MaxConcurrentExecutables property of package to implement custom parallelism. Of course you would need to have multiple containers and corresponding stored proc groups.
Parallel Execution in SSIS
MaxConcurrentExecutables, a property of the package. It defines how
many tasks (executables) can run simultaneously. It defaults to -1
which is translated to the number of processors plus 2. Please note
that if your box has hyperthreading turned on, it is the logical
processor rather than the physically present processor that is
counted.
Hi you can use the following piece of code to get basically write the script of running all your stored procedures if you add a new procedure it will automatically be added to the list
SELECT'EXEC '+SPECIFIC_NAME [Command] + ';'
FROM information_schema.routines
WHERE routine_type = 'PROCEDURE'
After this you take the result set put it into a tab delimited text file and save the file in a location.
use this link to import the text into a execute SQL task the first answer works well
SSIS: How do I pull a SQL statement from a file into a string variable?
execute the task and it should work, if you need to narrow the ;ist of procedures you can specify a specific prefix in the name of the procedure and use that in the where clause
It will run in serial, sorry i dont have enough rep to comment yet
I'm trying to optimize a report that uses multiple stored procedures on the same table. Unfortunately, each procedure is reading millions of records and aggregating the results. It's a very intense read for a report, but each stored procedure is optimized to run pretty fast within SSMS.
I can run each stored procedure and get a result set within 10 to 20 seconds. When I put them all into one report within SSRS, the report times out.
There is a total of 4 parameters per stored procedure. All targeting the same table, just aggregating the data in different ways. Indexes on those tables are inline with the query. It's based on time, user and the one dimension I'm using to COUNT() both DISTINCT and NONDISTINCT.
I'm thinking the issue is the fact SSRS is running 4 procedures at the same time on the same table as opposed to one after the other. Is this true? If so, is there anyway to ensure SSRS does not run them in parallel?
My only option is to create summary table that is already preaggregated. Then just run the report off that table. Otherwise, I guess param sniffing is possible too.
By default, datasets in SSRS are executed in parallel.
If all of your datasets refer to the same datasource, then you can configure for serialized execution of the datasets on a single connection this way:
open the data source dialog in report designer
ensure that the Use Single Transaction checkbox is checked
Once that checkbox is selected, datasets that use the same data source are no longer executed in parallel.
I hope that solves your problem.
I'm trying to optimize a stored procedure I'm maintaining, and am wondering if anyone can clue me in to the performance benefits/penalities of the options below. For my solution, I basically need to run a conversion program on an image stored in an IMAGE column in a table. The conversion process lives in an external .EXE file. Here are my options:
Pull the results of the target table into a temporary table, and then use a cursor to go over each row in the table and run a stored procedure on the IMAGE column. The stored proc calls out to the .EXE.
Create a UDF that calls the .EXE file, and run a SQL query similar to "select UDFNAME(Image_Col) from TargetTable".
I guess what I'm looking for is an idea of how much overhead would be added by the creation of the cursor, instead of doing it as a set?
Some additional info:
The size of the set in this case is max. 1000
As an answer mentions below, if done as a set with a UDF, will that mean that the external program is opened 1000 times all at once? Or are there optimizations in place for that? Obviously, on a multi-processor system, it may not be a bad thing to have multiple instances of the process running, but 1000 might be a bit much.
define set base in this context?
If you have 100 rows will this open up the app 100 times in one shot? I would say test and just because you can call an extended proc from a UDF I would still use a cursor for this because setbased doesn't matter in this case since you are not manipulating data in the tables directly
I did a little testing and experimenting, and when done in a UDF, it does indeed process each row at a time - SQL server doesn't run 100 processes for each of the 100 rows (I didn't think it would).
However, I still believe that doing this as a UDF instead of as a cursor would be better, because my research tends to show that the extra overhead of having to pull the data out in the cursor would slow things down. It may not make a huge difference, but it might save time versus pulling all of the data out into a temporary table first.