I have a big table on sql server, about 800k records.
How could I navigate through all the records, x amount at a time?
For example, I would like to open FDQuery, but browse 1000 records at a time.
If I use:
FDQuery.First;
while not FDQuery.eof do
begin
//do something
FDQuery.Next;
end;
I believe all records are brought;
I have read about fetching records, properties like FetchOptions:
Mode, RowsetSize, RecsMax, RecsSkip ... but I can't browse all the records, a fixed number of records at a time.
You can use some creativity, provide pagination and fetch let's say 100 rows per page using a thread which is equipped with a nice progress bar in the UI. in this method you must manage search and filters by some smart queries, reloading data sometimes, etc...
IF you are using SQL server one of the options could be using some ranking functions like ROW_NUMBER to fetch each page.
Something like this:
;WITH CTE_Test AS
(
SELECT *, ROW_NUMBER() OVER (ORDER BY F1) AS Row_No FROM Tbl_Test
)
SELECT * FROM CTE_Test
WHERE Row_No BETWEEN 1 AND 100
Related
When creating ad-hoc queries to look for information in a table I have run into this issue over and over.
Let's say I have a table with a million records with fields id - int, createddatetime - timestamp, category - varchar(50) and content - varchar(max). I want to find all records in the last day that have a certain string in the content field. If I create a query like this...
select *
from table
where createddatetime > '2018-1-31'
and content like '%something%'
it may complete in a second because in the last day there may only be 100 records so the LIKE clause is only operating on a small number of records
However if I add one more item to the where clause...
select *
from table
where createddatetime > '2018-1-31'
and content like '%something%'
and category = 'testing'
then it could take many minutes to complete while locking up the table.
It appears to be changing from performing all the straight forward WHERE clause items first and then the LIKE on the limited set of records, over to having the LIKE clause first. There are even times where there are multiple LIKE statements and adding one more causes the query to go from a split second to minutes.
The only solutions I've found are to either generate an intermediate table (maybe temp tables would work), insert records based on the basic WHERE clause items, then run a separate query to filter by one or more LIKE statements. I've tried various JOIN and CTE approaches which usually have no improvement. Alternatively CHARINDEX also appears to work though difficult to use if trying to convert the logic of multiple LIKE statements.
Is there any hint or something that can be placed in the query statement to tell sql server to wait until records are filtered by the basic WHERE clause items before filtering by the LIKE?
I actually just tried this approach and it had the same issue...
select *
from (
select *, charindex('something', content) as found
from bounce
where createddatetime > '2018-1-31'
) t
where found > 0
while the subquery independently returns in a couple seconds, the overall query just never returns. Why is this so bad
Not fancy, but I've had better luck with temp tables than nested select statements... It will isolate the first data set, and then you can select just from that. If you're looking for quick and dirty, which usually serves my purposes for ad-hoc, this may help. If this is a permanent stored proc, the indexing suggestions may serve you better in the long run.
select *
into #like
from table
where createddatetime > '2018-1-31'
and content like '%something%'
select *
from #like
where category = 'testing'
A rock band has currently 100 songs :
select count(songName) from bands where name='Beatles'
result : 100.
I display those songs in my app via paging ( each time - 10 results)
How do I get the relevant page and its rows ?
like this : (SP)
declare #pageItems int=10
declare #pagenum int=1
select * from (
SELECT [id] , row_number() over ( order by songName) as n
FROM Bands where name='Beatles'
) a
where a.n > (#pagenum-1)*#pageItems and a.n<= (#pagenum)*#pageItems
But there is a problem.
Suppose a user is at page 3.
And the beatles publish a new song named : "aaa"
So now , there will be a mismatch because there is a new row which is inserted at the top ( and pushes all rows below).
I dont want to get all the rows into my app.
What is the correct way / (feature?) to get the first criteria results ?
I could use a temp table but it will be only for the current connection. ( and each time a user press "next" at paging - it's a new session).
This is a common problem in paging, whether or not to adjust to data that is moving. Usually your users won't notice, or care, that the total number of records changes during pagination or if a record that was at the bottom of page 2 suddenly is at the top of page 3 because a new record was added. But, if you really need the consistency, then if the dataset is small enough you can cache the entire list of PK's if a temp table or in your apps memory and then just join to that when fetching.
-> What is the best way to paginate results in SQL Server
I've found a lot of examples of paging in Oracle. The particular one I'm using now looks a like this:
SELECT * FROM (
SELECT a.*, ROWNUM RNUM FROM (
**Select * From SomeTable**) a
WHERE ROWNUM <= 500) b
WHERE b.RNUM >= 1
The line in bold represents the 'original' query. The rest of the SQL there is to implement the paging. The problem I'm running into is a query that is perfectly valid by itself; will fail when I place it inside of my paging code.
As an example - this query will fail:
SELECT TABLE1.*, TABLE1.SomeValue FROM TABLE1
With a ambiguous column error. But, without my extra code; it will run just fine. I have a large number of 'saved' queries, but I have to ensure that my paging solution doesn't invalidate them.
I've used SQL Developer as my Oracle querying tool and it manages to implement paging that works even with the above query that fails when I wrap it in the paging code. Can anyone tell me how they manage to pull it off?
First off, the original query would need to have an ORDER BY clause in order to make the paging solution work reasonably. Otherwise, it would be perfectly valid for Oracle to return the same 500 rows for the first page, the second page, and the Nth page.
SQL Developer is not changing your query to implement paging. It is simply sending the full query to Oracle and paging the results itself using JDBC. The JDBC client application can specify a fetch size which controls how many rows are returned from the database to the client at a time. The client application can then wait for the user to either decide to go to the next page or to do something else in which case the cursor is closed.
Whether the SQL Developer approach makes sense depends heavily on the architecture of your application. If you're trying to page data in a stateless web application, it probably doesn't work because you're not going to hold a database session open across multiple page requests. On the other hand, if you've got a fat client application with a dedicated Oracle database connection, it's quite reasonable.
First of all,
What's the point in doing
SELECT TABLE1.*, TABLE1.someValue from TABLE1
Wouldn't TABLE1.* automatically select "someValue", so why query it redundantly ?
Secondly for pagination, try the analytical query approach
SELECT * FROM {
SELECT col1, col2, col3
, row_number() OVER (order by col1) position
FROM TABLE1
} WHERE rn >= p_seek and rn < (p_seek+p_count)
p_seek is the starting position and p_count is the number of rows to fetch.
Here instead of col1, col2, col3, etc you can do TABLE1.*, TABLE1.someValue etc.
There are a number of sql server paging questions on stackoverflow and many of them talk about using ROW_NUMBER() OVER (ORDER BY ...) AND CTE. Once you get into the hundreds of thousands of rows and start adding sorting on non-primary key values and adding custom WHERE clauses, these methods become very inneficient. I have a dataset of several million rows I am trying to page through with custom sorting and filtering, but I am getting poor performance, even with indexes on all the fields that I sort by and filter by. I even went as far as to include my SELECT columns in each of the indexes, but this barely helped and severely bloated my database.
I noticed the stackoverflow paging only takes about 500 milliseconds no matter what sorting criteria or page number you click on. Anyone know how to make paging work efficiently in SQL Server 2008 with millions of rows? This would include getting the total rows as efficiently as possible.
My current query has the exact same logic as this stackoverflow question about paging:
Best paging solution using SQL Server 2005?
Anyone know how to make paging work efficiently in SQL Server 2008 with millions of rows?
If you want accurate perfect paging, there is no substitute for building an index key (position row number) for each record. However, there are alternatives.
(1) total number of pages (records)
You can use an approximation from sysindexes.rows (almost instant) assuming the rate of change is small.
You can use triggers to maintain a completely accurate, to the second, table row count
(2) paging
(a)
You can show page jumps within say the next five pages to either side of a record. These need to scan at most {page size} x 5 on each side. If your underlying query lends itself to travelling along the sort order quickly, this should not be slow. So given a record X, you can go to the previous page using (assuming sort order is a asc, b desc
select top(#pagesize) t.*
from tbl x
inner join tbl t on (t.a = x.a and t.b > x.b) OR
(t.a < a.x)
where x.id = #X
order by t.a asc, t.b desc
(i.e. the last {page size} of records prior to X)
To go five pages back, you increase it to TOP(#pagesize*5) then further TOP(#pagesize) from that subquery.
Downside: This option requires that you cannot directly jump to a particular location, your options are only FIRST (easy), LAST (easy), NEXT/PRIOR, <5 pages either side
(b)
If the paging is always going to be quite specific and predictable, maintain an INDEXED view or trigger-updated table that does not contain gaps in the row number. This may be an option if the tables normally only see updates at one end of the spectrum, with gaps from deletes easily filled quickly by shifting not-so-many records.
This approach gives you a rowcount (last row) and also direct access to any page.
try this, let say you have country table as below:
DECLARE #pageIndex INT=0;
DECLARE #pageSize INT= 10;
DECLARE #sortByColumn NVARCHAR(200)='Code';
DECLARE #sortByDesc BIT=0;
;WITH tbl AS (
SELECT COUNT(id) OVER() [RowTotal], c.Id, c.Code, c.Name
FROM dbo.[Country] c
ORDER BY
CASE WHEN #sortByColumn='Code' AND #sortByDesc=0 THEN c.Code END ASC,
CASE WHEN #sortByColumn='Code' AND #sortByDesc<>0 THEN c.Code END DESC,
CASE WHEN #sortByColumn='Name' AND #sortByDesc=0 THEN c.Name END ASC,
CASE WHEN #sortByColumn='Name' AND #sortByDesc<>0 THEN c.Name END DESC,
,c.Name ASC --DEFAULT SORTING ORDER
OFFSET #PageIndex*#pageSize ROWS
FETCH NEXT #pageSize ROWS ONLY
) SELECT (#PageIndex*#pageSize)+(ROW_NUMBER() OVER(ORDER BY Id))[RowNo],* from tbl;
In Sql Server 2008, many options are available for database paging via stored procedure. For example, see here and here.
OPTIONS:
ROW_NUMBER() function
ROWCOUNT
CURSORS
temporary tables
Nested SQL queries
OTHERS
Paging using ROW_NUMBER() is known to have performance issues:
Please advise, which paging method has the best performance (for large tables with JOINs) ?
Please also provide links to relevant article(s), if possible.
Thank You.
One question you have to answer is if you want to display the total number of rows to the end user. To calculate the number of the last page, you also need the last row number.
If you can do without that information, a temporary table is a good option. You can select the pirmary key and use LIMIT to retrieve keys up to the key you're interested in. If you do this right, the typical use case will only retrieve the first few pages.
If you need the last page number, you can use ROW_NUMBER(). Using a temporary table won't be much faster because you can't use the LIMIT clause, making this strategy the equivalent of a ROW_NUMBER() calculation.
We can get a rowcount using following query.
WITH data AS
(
SELECT ROW_NUMBER() OVER (order by memberid ) AS rowid, memberid
FROM Customer
)
SELECT *, (select count(*) from data) AS TotalCount
FROM data
WHERE rowid > 20 AND rowid <= 30