Here is my problem, I need to fetch some large record from various tables; to be exact, it consists of 30 tables. I did the join for the 30 tables, and it took 20 min just to fetch 200 rows.
I was thinking of creating a stored procedure to do some transactional DB call to fetch bit by bit of data and store it to a new report table.
Here is the nature of my business process:
In my web screen, I have 10 tabs of questionnaire need to be fill up by insurance client. Basically I need to fetch all questions and answers and put them in one row
The problem is, my client won't finish all the 10 tabs in one day, they might finish all the tabs in 3 days max
Initially I want to put a trigger for insert on the primary table and fetch all and put in a reporting table. But I only can get record for t+0, not t+1 or t+n. How am I going to update the same row if user updated another tab at another day?
To simplify my requirement, I have 10 tabs of questionnaire and to make it simpler for discussion, each tab has its own table. And to complete all the questionnaire doesn't required you to finish it in one day.
How am I going to fetch all the data using transactional SQL in a stored procedure?
Related
I'm currently writing a web application that displays any sort of changes that happen within a database to the webpage. I had to write a few triggers already that insert and update a table from the table I need the information from. Because the SQL is live, I needed to create a back up table to have my webpage watch so there will be no deadlocks in the live database as it is extremely active.
I need a trigger that deletes rows from my back up table when the row amount reaches 100, is there a way to do this? I want the table to be lightweight and only display a certain amount of information at a time without taking up unnecessary space in my database
Background
A customer is collecting prices of different products and wants to update the price in their database if there is any change. We are talking about ~100 million rows of data that comes from scrapping which is stored in spreadsheets.
We are using SQL Server to store this data.
Issue
Problem starts when we upload those spreadsheets so the data can be inserted into staging table and updates curated price table.
Following are two steps behind the operation-
step 1 data is stored in staging table
step 2 price is checked. The ones that changed since last run then
gets updated in the main curated table "price".
But SQL Server is taking too long
What I have done so far
to help step 2 run faster, we created an index on staging table and but that made step 1 slower.
so to help step 1 faster, we disable the index, then bulk insert into the table and then rebuild the said index. which takes about 15 minutes. We don't see any significant improvement.
What I want help with?
I am trying to improve performance. Any ideas on how to accomplish this?
(This question is "focused" towards performance improvement)
Have you considered using MERGE in your query. It would reduce the I/O operations on the disk drastically. Here is an example I found on the very first google search of your problem
I have to fetch n number of records from the database. The count of this record may very manager to manager. It means one manager can have 100+ records and other may have only 50+ records.
If it is all about fetching data from only one table then its super easy to get.
In my case the main pain point is I will get my result after using so many joins , temp tabels , functions , maths on some column and dates filter using switch cases and many more and yes each tables has 100k+ records with proper indexing.
I have added pagination in UI side so that I can get only 20 records at a time on screen. once I clicked on page number based on that I should offset the records for next 20. Supposed clicked on page number 3 then from db I should get only records from 41- 60.
UI part is not a big deal the point is how to optimise your query so that every time I should get only 20 records.
My current implementation is every time I am calling the same procedure with index value to offset the data. Is that correct way to run same complex with all functions , cte, cases in filters and inner/left joins again and again to fetch only piece of data from recordset.
We have a lot of operations that time out in our site log.
After installing Redgate SQL Monitor on the server, we figured out we have many blocked processes and some times deadlock.
With Redgate we realized problem is for a stored procedure. It's a simple stored procedure and just increase view count of product (simple update)
ALTER PROCEDURE [dbo].[SP_IncreaseProductView]
#PID int
AS
BEGIN
UPDATE dbo.stProduct
SET ViewCount = ViewCount + 1
WHERE ID = #PID
END
When that stored procedure is off, everything is fine but some times block process error is back.
This table(product) hasn't any trigger. but its like heart of system and has 12000 records.
It has 3 indexes, 1 clustered and 2 non-clustered, and many statistics
We don't have any transaction. block process mostly happens in update query.
How can I figure out where the problem is?
Sorry for my bad languages
Thanks
Edit:
I think the problem isn't the SP, its about update on product table (Its my opinion). Its large table. I still get block process when SP is off but less.
We have a lot of select and update on this table.
Also i rewrite the increase view count with LINQ to SQL, but still get block process like when SP is on.
Edit 2:
I set profiler and get all query on product table.
530 select (most with join with another 2 table) and 25 update per minute (on product table only).
For now, [SP_IncreaseProductView] is off. Because when its on, we getting block process and operation timed out about every 10 second and web site stopped.
After that (set SP to off) block process still exist but roughly 50 per day.
I would go with Ingaz' second solution, but further optimizations or simplification can be performed:
1) store view count in a product 1:1 table. This is particularly useful when some queries do not need view count
2) view count redundancy
- keep view count in product table and read it from there
- also define view count in another table (just productid and viewcount columns)
- application can update asynchronously directly in the second table
- a job updates in the product table based on data from the second table
This ensures that locking is affecting product table much less than independent updates.
It is expected: I suppose that you have a lot of updates into single small (12K rows) table.
You can:
Work around your problem
Put ROWLOCK hint in your UPDATE
Change database option to READ_COMMITED_SNAPSHOT
Be warned though: it can create another problems.
More complex recipe to eliminate blocking completely
Not for faint of heart.
Create table dbo.stProduct_Increment.
Modify your [SP_IncreaseProductView] to INSERT into increment table.
Create periodic task that UPDATEs your dbo.stProduct and clear increment table.
Create view that combines stProduct and stProduct_Increment.
Modify all SELECT statements for stProduct to created view.
I have some troubles trying to move data from SQL Server 2000 (SP4) to Oracle10g, so the link is ready and working, now my issue is how to move detailed data, my case is the following:
Table A is Master
Table B is Detail
Both relationed for work with the trigger (FOR INSERT)
So My query needs to query both for create a robust query, so when trigger get fired on first insert of Master it passed normal, in the next step the user will insert one or more details in Table B, so the trigger will be fired any time the record increment, my problem is that I need to send for example :
1 Master - 1 Detail = 2 rows (Works Normal)
1 Master - 2 Details = 4 rows (Trouble)
In the second case I work around the detail that in each select for each insert it duplicates data, I said if Detail have 2 details the normal is that it will be 2 selects with 1 row each one, but in the second select the rows get doubled (query the first detail inserted)
How can I move one row per insert using triggers on Table B?
Most of the time this boils down to a coding error, and I blogged about it here:
http://www.brentozar.com/archive/2009/01/triggers-need-to-handle-multiple-records/
However, I'm concerned about what's going to happen with rollbacks. If you have a program on your SQL Server that does several things in a row to different tables, and they're encapsulated in different transactions, I can envision scenarios where data will get inserted into Oracle but it won't be in SQL Server. I would advise against using triggers for cross-server data synchronization.
Instead, consider using something like DTS or SSIS to synchronize the two servers regularly.