PBI Service dataflow - SQL Server data query timeout error - sql-server

Have you had experience with SQL queries in Power BI dataflows that run for longer than 10 minutes for one SQL query?
I have migrated an Excel Power Query script to PBI dataflows. One of the migrated queries fails and it fails consistently at 10:00 minutes exactly. But there is not "timeout" error displayed, just a message that says, "Query has cancelled". My query does have the [Command Timeout] property set to 60 minutes, but the property looks like it's being ignored in the PBI service dataflow.
I have written an Excel Power Query script with multiple SQL connections, most connection are stored procedure that return a dataset. One SQL proc takes about 30 minutes to complete. The entire of these queries takes about 50 minutes. We're trying to stage datasets in dataflows and improve performance for reporting for our end users, who are currently pulling datasets from SSRS reports. This dataflow query failure is a major road block.
I'm curious if anyone has had a similar experience. To me it seems ridiculous that a dataflow SQL connection can't run past 10 minutes.
Thanks for your input!

Related

SSRS accessing SQL ReportServer DB every 10 seconds

In SQL Profiler (for SPs), it seems my (single) SSRS Report Server instance is repeatedly accessing the ReportServer SQL DB every 10 seconds. The SSRS Stored Procedures being run are:
GetCurrentProductInfo
GetAllConfigurationInfo
GetAnnouncedKey
AnnounceOrGetKey
Is there any way to reduce the frequency of these SSRS requests ?
I've changed various settings in RSReportService.config, inc. PollingInterval (from 10s to 28800s), but none of this has had any effect on the above, even after re-start, re-boot.
Thanks for info/help.

How to achieve real time reporting in Azure SQL server in one database?

We have a stored procedure in Azure SQL database (Pricing tier is premium with 250 DTU) which processes around 1.3 billion records and inserts results in tables which we display in reporting page. To run this stored procedure, it takes around 15 minutes and we have scheduled it weekly as Azure webjobs because we use same database for writing actual user logs.
But now, we want real time reporting max 5 minutes of differences and if I schedule webjobs to execute the stored procedure every 5 minutes then my application will shutdown.
Is there any other approach to achieve real time reporting?
Is there any Azure services available for it?
Can I use azure databricks to execute the stored procedure? Will it help?
Yes, you can use read queries on Premuim replica databases, by adding this to your connection string:
ApplicationIntent=ReadOnly;
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-read-scale-out

Having exception in Win Form Application because SQL query takes too long time to execute

I have millions records in a database. When I fetch with conditions that are not indexed, it takes 2-5 minutes SQL query to be executed.
But, C# windows form application gives exception because it thinks SQL server is not responding it.
What is the solution?

SQL Server Agent Job Running Slow

I am executing a stored procedure using SQL Server Agent Job in SQL Server 2005.
This job was running fast until yesterday. Since yesterday this job is taking more than 1 hour instead of 2 mins.
I executed the stored procedure in SSMS, it just took less than 1 minute to execute.
I could not figure out why it is taking more than 1 hour when executed as a SQL Server Agent job?
After some time commenting and assuming that the SP performs with the same input parameters and data well when executed in SSMS, I finnaly think I can give a last tip:
Depending on what actions are performed within the SP (e.g. inserting/updating/deleting a lot of data within a loop or cursor), you should set nocount on at the beginning of your code.
set nocount on
If this is not the case or does not help, please add more information, already mentioned in the comments (e.g. all settings of the Job and each Jobstep, what has been logged, what is in the Jobhistory, check SQLerrorlogs, eventlogs,....).
Also take a look at the "SQL Server Logs" maybe you can gather some info here. Also a look into the Application/System eventlo of the Databaseserver is always a good idea.
To get a basic overview you can use the Activitymonitor in SSMS, by selecting the Databaseserver and selecting "Activity monitor" from contextmenu and search for the sql agent.
My last try would be to try to run a sql trace for the agent. In this case you would start a trace and filter e.g. by the user that the SQLAgent Service runs. There are so many options you can set for traces, so I would recommend to google for it, search on MSDN or ask another question here on stackoverflow.
We have a large proc that runs in 88 seconds in SSMS and 30-45 minutes in SQL Server Agent. I added the dbo. prefix on all the table names and now it runs just as fast as SSMS.
I've noticed that SQL Agent jobs ignore the server's MAXDOP setting and run everything with a MAXDOP of 1. If I run a stored procedure in a query windows, it obeys the server settings and uses 4 processes. If I use SQL Agent, any stored procedure I run uses only one process.
I have a similar issue with a script that calls a number of UDFs that I created. The UDF's themselves normally run subsecond under SSMS. Likewise, running the reports I generate with them is bearable under SSMS (30d data in 8s, 365d data in 22s). I've always done NOCOUNT ON with my SQL Agent jobs as they normally generate text files out for pickin up by other processes or Excel and I do not want the extra data at the end, so it was not a solution for me.
In this case, when we run the exact same script under SQL Agent as a job, my times grow exponentially. My 8s script takes 2m30s and my 22s script takes 2h20m. This is the same whether I run it midday with other user activity and jobs or after hours with no user activity, nor jobs or backups running. Our server is idle and at best I get one of the 8 cores being utilized when run. DB is only about 10GB running on SSD with a cached RAID card and 16 of 32GB RAM is free. Since my SQL runs efficiently in SSMS, I am pretty well of the belief that I am hitting a threading limit of some sort. I have researched and tried adjusting MAXDOP just prior to the scripts in the SQL Agent with no luck.
Since this is an activity I want to schedule, it needs to be automated one way or another. I could let these scripts take the hours they need to run as SQL steps in SQL Agent jobs, but I decided to run from command line instead and I get the same performance I see in SSMS.
sqlcmd -S SQLSRVRHost -i "C:\My Script Loc With Spaces.sql" -v MyVar="VarValue" >"C:\MyOutputFile.txt"
So I created a batch script with the SQL jobs run from sqlcmd. Then I run the batch script from a SQL Agent job, so I still have the same management and control in place. My 4 SQL jobs that collectively took over 3 hours to run complete in 1 min and a few seconds from a single batch script executed by SQL Agent.
I hope this helps...

SQL Server job (stored proc) trace

I need your suggestion on tracing the issue.
We are running data load jobs at early morning and loading the data from Excel file into SQL Server 2005 db. When job runs on production server, many times it takes 2 to 3 hours to complete the tasks. We could drill down to one job step which is taking 99% of the total time to finish.
While running the job step (stored procs) on staging environment (with the same production database restored) takes 9 to 10 minutes, the same takes hours on production server when it run at early morning as part of job. The production server always stuck up at the very job step.
I would like to run trace on the very job step (around 10 stored procs run for each user in while loop within the job step) and collect the info to figure out the issue.
What are the ways available in SQL Server 2005 to achieve the same? I want to run the trace only for these SPs and not for certain period time period on production server, as trace give lots of information and it becomes very difficult for me (as not being DBA) to analyze that much of trace information and figure out the issue. So I want to collect info about specific SPs only.
Let me know what you suggest.
Appreciate your time and help.
Thanks.
Use SQL Profiler. It allows you to trace plenty of events, including stored procedures, and even apply filters to the trace.
Create a new trace
Select just stored procedures (RPC:Completed)
Check "TextData" for columns to read
Click the "Column Filters" button
Select "TextData" from the left hand nav
Expand the "Like" tree view, and type in your procedure name
Check "Exclude Rows that Do Not Contain Values"
Click "OK", then "Run"
What else is happening on the server at that time is important when it is faster on other servers but not prod. Maybe you are running into the daily backup or maintenance of statistics or indexes jobs?

Resources