I'm trying to design a Matrix report through SSRS to aggregate a column for a range of dynamic values in another column (i.e. a pivot). This data consists of just over 13 million rows, so it's a large dataset.
When doing a PIVOT on this data via T-SQL, it's able to aggregate all of these rows in about ~1min, however when getting SSRS to do the pivoting for me through a Matrix report, I'm getting an OutOfMemory exception when trying to preview the report on my PC.
The query returning the dataset itself isn't complicated, it's as simple as:
SELECT
ID
,Test_Ref
,Data_issue_indicator
FROM MyTable
Where we're trying to do the sum of Data_issue_indicator (which can be either a 1 or 0) for values in Test_Ref, in which there is a dynamic range of values to aggregate against; in other words we cannot use a standard Tablix report because the amount of columns can increase at any time should a new Test_Ref value be introduced into the dataset.
I'm using Visual Studio Enterprise 2019, and my PC is a Windows 10, i7-8850H, with 16GB memory.
Is there a suggestion on getting around this issue?
When using SSRS, its recommended to grab more data once in case of using the dataset multiple times. but when you have a larger dataset it needs to be a trade off between what you want to achieve against do you need all the data.
So in this situation i would suggest to use a procedure to restrict the amount of data that you are grabbing to the report.
I have gone through this sort of scenario, and i had to do the same, because its not the query that is timing out but the huge amount of data that is loaded to the report which fails the report.
If you have SQL server profiler , you would see the SQL executed and completed, but the report times out rendering.
Two ideas, assuming that you plan to deploy the report to a server that will have the memory to handle this, and that you'd prefer to do this processing on the report server rather than the SQL server for some reason:
Don't test the functionality on your PC in Visual Studio. Design the report, deploy it to your Report Server, and test it there to see if it works.
When testing on your PC, force it somehow to use a much smaller dataset: one just large enough to verify that the pivoting Matrix works, but small enough that your PC's memory can handle it.
Or better yet, do option 2, and then option 1.
Related
I need to create report using SQL Server data from couple million records. Getting this error in Excel any way to resolve this? Or should I switch to SSRS which I don't know if it's better for handling large data sets since I never used it before.
I am currently working on developing reports on SSRS. This specific report was initially built on excel using a query that is quite comple. I noticed it pulls about 2 million records and thus slow due to several joins and subqueries. I was trying to create a table instead dedicated to have this data readily but i got an error (query taking too long ) and timed out while trying to execute it as a SELECT * INTO ReportTable From. Please any assistance is well appreciated. Thanks
You can either run this directly on the DB host, or you can create an SQL Server Agent job that performs this. It should prevent the time out.
I'm using the Fuzzy Matching SSIS component and want to do a fuzzy match on new names (from input columns Name1) against known names (Alias from Lookup Columns). Doing this alone works great (and is fast), but when I want to restrict the matching to only those records that have the same Country code as shown below, BTW both columns are char(3) is ISO codes, SSIS performance is so slow that it never completes.
I've attempted every variation of indexing available on the [Reference Table] tab and I believe I'm using the combination of Relationships correctly per https://msdn.microsoft.com/en-us/library/ms186488.aspx
Anyone run into a similar issue and figure out a workable solution?
Adding answer to question asked in comments here (better formatting);
SQL Server version : 2014 Ent
SSIS Version : not sure, but created w/ SQL Server data tool for VS 2013
Data Volume Source : 65K
Data Volume Data to Match : 105K (but SSIS pipeline gets stuck around 10k)
SQL Server indicates it's waiting for SSIS to pull more data in
Looking at task manager DTSdebug is using ~15% CPU and will do so indefinitely
The really odd thing is if I remove the country matching (which is set to exact) and use a larger source set (172K vs 65K) SSIS runs wonderfully.
I'd like to get some advice on a reporting situation that I have. I am working in SQL Server. I have a ton of data validation queries that I run against a database. In general, for each query, I return two things -- one is the count of the offending records, and the other is the offending records themselves.
My goal is produce a report that gives the counts of the offending records for all data validation queries (ideally, on one sheet in an Excel workbook) and the offending records themselves (ideally, on separate sheets in an Excel workbook).
How is this best achieved? That is, what technology is best for this situation? For example, in the past, I have prototyped the queries in SSMS, copied them into a Windows batch file (and added code to write the results to separate text files), and called the batch file via the sqlcmd utility (using command prompt). However, I know that other solutions exist (e.g., SSRS). Would something like SSRS be a better tool for this situation? I'm hesitant to go the SSRS route, since I'm only giving metrics on one issue (i.e., counts of offending records) and the rest of the report consists of offending records.
This might get closed because it is a matter of opinion, but SSRS would be a good solution for this requirement. I think SSRS is a good fit if you have the following criteria:
You need to visualize the data in some kind of a table, chart, or graph
You want to send out automated emails every morning / week / month to a group of users (as opposed to just individual consumption)
You want to be able to export the report to other formats (excel or pdf) for additional analysis or sharing.
Otherwise, if it's just for you and you currently don't have SSRS running on the server, save yourself the overhead of running another service and just keep doing it in batch files.
I have a problems with 3rd dimension in MDX Query (on MS SQL Server 2005). I can use 3rd dimension in Visual Basic (I have a cube there, using browser I can make 3 dim. queries -- owing to ON PAGES). I snooped it via MS SQL Profiler (it records databases queries). But when I tried to put the query into MS SQL SERVER, only thing what I've received was:
Executing the query ...
Obtained object of type: Microsoft.AnalysisServices.AdomdClient.CellSet
Formatting.
Results cannot be displayed for cellsets with more than two axes.
Execution complete
I tried a few different ways to implement query, but this^ answer, was an only answer from a server.
The question is: What I need to do to use third dim in my OLAP?
When you query Analysis Services with SQL Server Management Studio (SSMS) you get an ADOMD.NET Cellset object. This object represents your query results in a multidimensional fashion, laying out the information in several axes: Axis 0 for columns, axis 1 for rows, axis 2 for pages, etc.
Although your query may be correct with three axes, SSMS can only render bidimensional restults, hence the error message you get. You need to build an application using ADOMD.NET to consume the cellset, or use a third party tool.
You say you "put the query into MS SQL SERVER" - what exactly do you mean? Did you try the cube browser that comes with Analysis Services? It's pretty rubbish.
I think the problem is as simple as the error message you got - "Results cannot be displayed". In other words, the viewing software can show tables (i.e. 2D info) but doesnt have a way to display 3D info (whether it be in multiple tables, or whatever).
You need to find software that can present cube data more elaborately.