Is a compound index sufficient for queries against a subset of columns ?
CREATE TABLE [FILE_STATUS_HISTORY]
(
[FILE_ID] [INT] NOT NULL,
[STATUS_ID] [INT] NOT NULL,
[TIMESTAMP_UTC] [DATETIME] NOT NULL,
CONSTRAINT [PK_FILE_STATUS_HISTORY]
PRIMARY KEY CLUSTERED ([FILE_ID] ASC, [STATUS_ID] ASC)
) ON [PRIMARY]
CREATE UNIQUE NONCLUSTERED INDEX [IX_FILE_STATUS_HISTORY]
ON [FILE_STATUS_HISTORY] ([FILE_ID] ASC,
[STATUS_ID] ASC,
[TIMESTAMP_UTC] ASC) ON [PRIMARY]
GO
SELECT TOP (1) *
FROM [FILE_STATUS_HISTORY]
WHERE [FILE_ID] = 382748
ORDER BY [TIMESTAMP_UTC] DESC
A composite index on ( File_Id, Timestamp_UTC desc ) should optimize handling the where and top/order by clauses. The actual execution plan will show whether the query optimizer agrees.
A covering index would also have Status_Id as an included column so that the index could satisfy the entire query in a single lookup.
Related
I have SQL Server 2019 where I want to partition one of my tables. Let's say we have a simple table like so:
IF OBJECT_ID('dbo.t') IS NOT NULL
DROP TABLE t;
CREATE TABLE t
(
PKID INT NOT NULL,
PeriodId INT NOT NULL,
ColA VARCHAR(10),
ColB INT
);
Let's also say that I have defined partition function and schema. The schema is called [PS_PartitionKey]
Now I can partition this table by building a clustered index in a couple of ways.
Like this:
CREATE CLUSTERED INDEX IX_1 ON t ([PKId] ASC )
ON [PS_PartitionKey]([PeriodID])
Or like this:
CREATE CLUSTERED INDEX IX_1 ON t ([PKId] ASC, [PeriodId] ASC )
ON [PS_PartitionKey]([PeriodID])
As you can see, in the first case I did not explicitly specify my partitioning column as part of the index key, but in the second case I did. Both of these work, but what's the difference?
A similar question would apply if I were building these as non-clustered indexes. Using the same table as an example. Let's say I start by creating a clustered PK:
ALTER TABLE [dbo].t
ADD CONSTRAINT PK_t
PRIMARY KEY CLUSTERED ([PKId] ASC, [PeriodId]) ON [PS_PartitionKey]([PeriodID])
Now I want to define additional non-clustered index. Once again, I can do it in two ways:
CREATE NONCLUSTERED INDEX IX_1 ON t ([ColA] ASC)
ON [PS_PartitionKey]([PeriodID])
or:
CREATE NONCLUSTERED INDEX IX_1 ON t ([ColA] ASC, [PeriodId] ASC)
ON [PS_PartitionKey]([PeriodID])
What difference would it make?
Problem
Which is correct when join section table with class table OR with course table OR with instructor Table .
Details
section is group of student classified to ( aa,bb,cc )can take one course or more courses.
section can teach in one or more class(lab or class room) .
Instructor can teach to more sections and section can have more instructor
raltion is many to many and made in third table Inst_Course
My ER diagram as following :
section table join
Database Schema as following :
CREATE TABLE [dbo].[Instructor](
[InstructorID] [int] NOT NULL,
[InstructorName] [nvarchar](50) NULL,
CONSTRAINT [PK_Instructor] PRIMARY KEY CLUSTERED
(
[InstructorID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Course](
[CourseID] [int] NOT NULL,
[CourseName] [nvarchar](50) NULL,
CONSTRAINT [PK_Course] PRIMARY KEY CLUSTERED
(
[CourseID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Class](
[ClassID] [int] NOT NULL,
[ClassName] [nvarchar](50) NULL,
CONSTRAINT [PK_Class] PRIMARY KEY CLUSTERED
(
[ClassID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Section](
[SectionID] [int] NOT NULL,
[SectionName] [nvarchar](50) NULL,
CONSTRAINT [PK_Section] PRIMARY KEY CLUSTERED
(
[SectionID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Inst_Course](
[InstID] [int] NOT NULL,
[CourseID] [int] NOT NULL,
CONSTRAINT [PK_Inst_Course] PRIMARY KEY CLUSTERED
(
[InstID] ASC,
[CourseID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Course_Class](
[ClassID] [int] NOT NULL,
[CourseID] [int] NOT NULL,
[Fromtime] [int] NULL,
[Totime] [int] NULL,
[day] [nvarchar](50) NULL,
CONSTRAINT [PK_Course_Class] PRIMARY KEY CLUSTERED
(
[ClassID] ASC,
[CourseID] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
Relation between tables as following :
Class table and courses table has many to many relation ship in tableCourse_Class .
Instructor table and courses table has relation many to many in table
Inst_Course .
Section is have many to many with instructor table and course table and class table
which is correct for join section with instructor or course or class
Notes :this diagram not have student courses table because the goal from diagram is design schedule for instructor .
sample data
Sample data to table Course_Class for instructor schedule
join between tables as following :
SELECT dbo.Class.ClassName, dbo.Course_Class.CourseID, dbo.Course_Class.Fromtime, dbo.Course_Class.Totime, dbo.Course_Class.day, dbo.Course.CourseName,
dbo.Inst_Course.InstID, dbo.Inst_Course.CourseID AS Expr3, dbo.Instructor.InstructorID, dbo.Instructor.InstructorName
FROM dbo.Class INNER JOIN
dbo.Course_Class ON dbo.Class.ClassID = dbo.Course_Class.ClassID INNER JOIN
dbo.Course ON dbo.Course_Class.CourseID = dbo.Course.CourseID INNER JOIN
dbo.Inst_Course ON dbo.Course.CourseID = dbo.Inst_Course.CourseID INNER JOIN
dbo.Instructor ON dbo.Inst_Course.InstID = dbo.Instructor.InstructorID
WHERE (dbo.Inst_Course.InstID = 1)
Question is :Actually what i need is which table must join with section table class or course or instructor tables
Update
Class in my case represent classroom or lab meaning class is the place teach courses in it
Section :(group of student)represent who teaches .
I can take course c# in class 1 meaning lab 1 or lab 2 or lab 3
and in lab1 i can get course c# OR c++ OR java in my case .
Here i treat with section to represent group of students .
section can teach more courses c# and c++ and java .
c# course can have more section aa,bb,cc .
Update2
student participate in one section only and cannot more section meaning relation one to many .
relation between section and class is many to many because section aa can take course c# in class a and class bb
and class bb can have course c# and c++
if you mean by session is course you are right .
Courses teaches in more classes in different times from 9-11,11-1,1-3,3-4.30
in different classes .
courses include more sections and every section can have more course
OK, updated based upon your update, I'd suggest you have the following structures:
create table dbo.Instructors (
InstructorID int identity(1,1) not null ,
constraint pkc_Instructors
primary key clustered ( InstructorID ) ,
InstructorName nvarchar(48) not null ,
constraint uni_InstructorName#Instructors
unique nonclustered ( InstructorName )
)
create table dbo.Courses (
CourseID int identity(1,1) not null ,
constraint pkc_Courses
primary key clustered ( CourseID ) ,
CourseName nvarchar(48) not null ,
constraint uni_CourseName#Courses
unique nonclustered ( CourseName )
)
create table dbo.ClassRooms (
ClassRoomID int identity(1,1) not null ,
constraint pkc_ClassRooms
primary key clustered ( ClassRoomID ) ,
ClassRoomName nvarchar(48) not null ,
constraint uni_ClassRoomName#ClassRooms
unique nonclustered ( ClassRoomName )
)
create table dbo.Sections (
SectionID int identity(1,1) not null ,
constraint pkc_Sections
primary key clustered ( SectionID ) ,
CourseID int not null ,
constraint fk_CourseID#Sections
foreign key ( CourseID )
references dbo.Courses ( CourseID ) ,
SectionName nvarchar(48) not null ,
constraint uni_SectionName#Sections
unique nonclustered ( SectionName )
)
create table dbo.StudentSections (
StudentSectionID int identity(1,1) not null ,
constraint pkn_StudentSections
primary key nonclustered ( StudentSectionID ) ,
StudentID int not null ,
constraint fk_StudentID#StudentSections
foreign key ( StudentID )
references dbo.Students ( StudentID ) ,
SectionID int not null ,
constraint fk_SectionID#StudentSections
foreign key ( SectionID )
references dbo.Sections ( SectionID ) ,
constraint uci_StudentID_SectionID#StudentSections
unique clustered ( StudentID , SectionID )
)
create table dbo.SectionClassRooms (
SectionClassRoomID int identity(1,1) not null ,
constraint pkn_SectionClassRooms
primary key nonclustered ( SectionClassRoomID ) ,
SectionID int not null ,
constraint fk_SectionID#SectionClassRooms
foreign key ( SectionID )
references dbo.Sections ( SectionID ) ,
ClassRoomID int not null ,
constraint fk_ClassRoomID#SectionClassRooms
foreign key ( ClassRoomID )
references dbo.ClassRooms ( ClassRoomID ) ,
constraint uci_SectionID_ClassRoomID#SectionClassRooms
unique clustered ( SectionID , ClassRoomID )
)
create table dbo.InstructorSections (
InstructorSectionID int identity(1,1) not null ,
constraint pkn_InstructorSections
primary key nonclustered ( InstructorSectionID ) ,
InstructorID int not null ,
constraint fk_InstructorID#InstructorSections
foreign key ( InstructorID )
references dbo.Instructors ( InstructorID ) ,
SectionID int not null ,
constraint fk_SectionID#InstructorSections
foreign key ( SectionID )
references dbo.Sections ( SectionID ) ,
constraint uci_InstructorID_SectionID#InstructorSections
unique clustered ( InstructorID , SectionID )
)
What this says:
Instructors are their own entities, with each instructor having a unique name.
Courses are similar, with uniqueness on their name.
ClassRooms, ditto.
Sections must be a section of a Course and cannot exist in isolation.
Students participate in one or more Sections.
Sections may take place in one or more ClassRooms (if this is not true, then we need to make ClassRoomID an attribute of each Section, rather than putting this in its own table SectionClassRooms).
An Instructor may teach many Sessions and also a Session may be taught by multiple Instructors (if this is not the case, then InstructorID should be an attribute of Sessions).
Obviously, you're going to have a few more fields in some of these, as needed. For example: when do the Sessions take place? Does a Session have the same time regardless of the classroom in which it takes place? There are nuances here, I'm sure.
Lastly, I'd encourage you to review the clustered vs nonclustered indexing, as this should be optimized for how your information will be retrieved. Not critical, but I made a stab at how I thought it should work, not knowing your application requirements.
I'll modify this again if needed.
This design seems to arise from the relationship/table in this question you recently asked. (Unfortunately both questions are extremely unclear.) From my answer:
Probably you want a table telling you the same thing as all reports for all instructors: instructorIteaches courseCin classroomCRto sectionSfor departmentDin timeslotTSon weekdayWD.
I told you you should learn normalization:
By identifying all FDs (functional dependencies) we determine all CKs (candidate keys). Then normalization uses these to possibly suggest "better" choices for base tables.
In that question and in this question it is not possible to express the big table as a join of smaller tables.
A prerequisite for decomposing a table into smaller ones, ie for joining smaller ones to get it, is that the statement that a row makes when it is in the big table can be expressed as the AND of the statements of the smaller tables.
Apparently Course_Class holds rows that make a true statement from course COURSEID is taught in classroom CLASSID from FROMTIME to TOTIME on day WEEKDAY. Apparently Inst_Course holds rows that make a true statement from instructor INSTID teaches course COURSEID.
Maybe you want the rows where instructor INSTID teaches course COURSEID in classroom CLASSID from FROMTIME to TOTIME on day WEEKDAY. If every instructor that teaches a course teaches every lecture of it then you can join Course_Class and Inst_Course to get this. That seems very unlikely. So more likely, this cannot be rephrased using the AND of smaller statements. So you can't get this table from joining smaller tables.
Maybe you want the rows where instructor INSTID teaches section SECTIONID of course COURSEID in classroom CLASSID from FROMTIME to TOTIME on day WEEKDAY. Then, similarly, adding a Course_Section table for course COURSEID has section SECTIONID is no help unless also every section of a course is taught in every lecture for it.
The information in the separate tables just doesn't collectively tell you what that big tables do. (Although it happens that here the smaller tables can be probably be generated from the big tables.)
You need to learn about (FKs & CKs &) normalization, which replaces big tables by smaller ones that join back to them.
I took a SQL assessment test this week. And this question in specific is one I did not understand since I am not familiar with clustered, non-clustered indexes yet.
The SQL server table below is used to manage a company’s product purchases. The table contains 17 million rows. Which of the following SQL statements can be used to create an index such to calculate the total purchases for a given data will run the shortest amount of time?
CREATE TABLE [Production].[TransactionHistory]
(
[TransactionID][int] IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED,
[ProductionID][int] NOT NULL,
[TransactionType][nchar](1) NOT NULL,
[Quantity][int] NOT NULL,
[ActualCost][money] NOT NULL,
[ProductionDate][dateTime] NOT NULL,
)
Which of the following queries can return data in the shortest amount of processing time? This will give me a good understanding of how indexes work. And there can be up to 3 valid answers in this question. Thanks in advance, I appreciate the help.
Option 1
CREATE COVERING INDEX IX_TranHistory_Covered
ON [Production].[TransactionHistory]
(
[ProductionDate] ASC,
[ActualCost] ASC,
[Quantity] ASC
);
Option 2
CREATE NONCLUSTERED INDEX IX_TranHistory_Covered
ON [Production].[TransactionHistory]
(
[ActualCost] ASC,
[ProductionDate] ASC,
[Quantity] ASC
);
Option 3
CREATE NONCLUSTERED INDEX IX_TranHistory_Covered
ON [Production].[TransactionHistory]
(
[Quantity]
)
INCLUDE
(
[ProductionDate],
[ActualCost]
);
Option 4
CREATE NONCLUSTERED INDEX IX_TranHistory_Covered
ON [Production].[TransactionHistory]
(
[ProductionDate]
)
INCLUDE
(
[ActualCost] ASC,
[Quantity] ASC
);
Last option
CREATE INDEX IX_TranHistory_Covered
ON [Production].[TransactionHistory]
(
[ActualCost] ASC,
[Quantity] ASC,
[ProductionDate] ASC
);
You want option 4. The key (Production Date) will induce index seeks and by creating a covered index the information needed to do satisfy query is right there in the index tree and SQL Server does not have to retrieve the entire row to calculate the result. You don't want 'asc' in the include part of the index.
In a SQL Server 2012 version 11.0.5058 I've a query like this
SELECT TOP 30
row_number() OVER (ORDER BY SequentialNumber ASC) AS [row_number],
o.Oid, StopAzioni
FROM
tmpTestPerf O
INNER JOIN
Stati s on O.Stato = s.Oid
WHERE
StopAzioni = 0
When I use ORDER BY SequentialNumber ASC it takes 400 ms
When I use ORDER BY DESC in the row_number function it takes only 2 ms
(This is in a test environment, in production it is 7000, 7 seconds vs 15 ms!)
Analyzing the execution plan, I found that it's the same for both queries. The interesting difference is that in the slower it works with all the rows filtered by the stopazioni = 0 condition, 117k rows
In the faster it only uses 53 rows
There are a primary key on the tmpTestPerf query and an indexed ASC key on the sequential number column.
How it could be explained?
Regards.
Daniele
This is the script of the tmpTestPerfQuery and Stati query with their indexes
CREATE TABLE [dbo].[tmpTestPerf]
(
[Oid] [uniqueidentifier] NOT NULL,
[SequentialNumber] [bigint] NOT NULL,
[Anagrafica] [uniqueidentifier] NULL,
[Stato] [uniqueidentifier] NULL,
CONSTRAINT [PK_tmpTestPerf]
PRIMARY KEY CLUSTERED ([Oid] ASC)
)
CREATE NONCLUSTERED INDEX [IX_2]
ON [dbo].[tmpTestPerf]([SequentialNumber] ASC)
CREATE TABLE [dbo].[Stati]
(
[Oid] [uniqueidentifier] ROWGUIDCOL NOT NULL,
[Descrizione] [nvarchar](100) NULL,
[StopAzioni] [bit] NOT NULL
CONSTRAINT [PK_Stati]
PRIMARY KEY CLUSTERED ([Oid] ASC)
) ON [PRIMARY]
CREATE NONCLUSTERED INDEX [iStopAzioni_Stati]
ON [dbo].[Stati]([StopAzioni] ASC)
GO
The query plans are not exactly the same.
Select the Index Scan operator.
Press F4 to view the properties and have a look at Scan Direction.
When you order ascending the Scan Direction is FORWARD and when you order descending it is BACKWARD.
The difference in number of rows is there because it takes only 53 rows to find 30 rows when scanning backwards and it takes 117k rows to find 30 matching rows scanning forwards in the index.
Note, without an order by clause on the main query there is no guarantee on what 30 rows you will get from your query. In this case it just happens to be the first thirty or the last thirty depending on the order by used in row_number().
ALTER PROCEDURE ReadNews
#CategoryID INT,
#Culture TINYINT = NULL,
#StartDate DATETIME = NULL,
#EndDate DATETIME = NULL,
#Start BIGINT, -- for paging
#Count BIGINT -- for paging
AS
BEGIN
SET NOCOUNT ON;
--ItemType for news is 0
;WITH Paging AS
(
SELECT news.ID,
news.Title,
news.Description,
news.Date,
news.Url,
news.Vote,
news.ResourceTitle,
news.UserID,
ROW_NUMBER() OVER(ORDER BY news.rank DESC) AS RowNumber, TotalCount = COUNT(*) OVER()
FROM dbo.News news
JOIN ItemCategory itemCat ON itemCat.ItemID = news.ID
WHERE itemCat.ItemType = 0 -- news item
AND itemCat.CategoryID = #CategoryID
AND (
(#StartDate IS NULL OR news.Date >= #StartDate) AND
(#EndDate IS NULL OR news.Date <= #EndDate)
)
AND news.Culture = #Culture
and news.[status] = 1
)
SELECT * FROM Paging WHERE RowNumber >= #Start AND RowNumber <= (#Start + #Count - 1)
OPTION (OPTIMIZE FOR (#CategoryID UNKNOWN, #Culture UNKNOWN))
END
Here is the structure of News and ItemCategory tables:
CREATE TABLE [dbo].[News](
[ID] [bigint] NOT NULL,
[Url] [varchar](300) NULL,
[Title] [nvarchar](300) NULL,
[Description] [nvarchar](3000) NULL,
[Date] [datetime] NULL,
[Rank] [smallint] NULL,
[Vote] [smallint] NULL,
[Culture] [tinyint] NULL,
[ResourceTitle] [nvarchar](200) NULL,
[Status] [tinyint] NULL
CONSTRAINT [PK_News] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [ItemCategory](
[ID] [bigint] IDENTITY(1,1) NOT NULL,
[ItemID] [bigint] NOT NULL,
[ItemType] [tinyint] NOT NULL,
[CategoryID] [int] NOT NULL,
CONSTRAINT [PK_ItemCategory] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
This query reads news of a specific category (sport, politics, ...).
#Culture parameter specifies the language of news, like 0 (english), 1 (french), etc.
ItemCategory table relates a news record to one or more categories.
ItemType column in ItemCategory table specifies which type of itemID is there. for now, we have only ItemType 0 indicating that ItemID refers to a record in News table.
Currently, I have the following index on ItemCategory table:
CREATE NONCLUSTERED INDEX [IX_ItemCategory_ItemType_CategoryID__ItemID] ON [ItemCategory]
(
[ItemType] ASC,
[CategoryID] ASC
)
INCLUDE ( [ItemID])
and the following index for News table (suggested by query analyzer):
CREATE NONCLUSTERED INDEX [_dta_index_News_8_1734000549__K1_K7_K13_K15] ON [dbo].[News]
(
[ID] ASC,
[Date] ASC,
[Culture] ASC,
[Status] ASC
)
With these indexes, when I execute the query, the query executes in less than a second for some parameters, and for another parameters (e.g. different #Culture or #CategoryID) may take up to 2 minutes! I have used OPTIMIZE FOR (#CategoryID UNKNOWN, #Culture UNKNOWN) to prevent parameter sniffing for #CategoryID and #Culture parameters but seems not working for some parameters.
There are currently around 2,870,000 records in News table and 4,740,000 in ItemCategory table.
Now I greatly appreciate any advice on how to optimize this query or its indexes.
update:
execution plan: (in this image, ItemNetwork is what I referred to as ItemCategory. they are the same)
Have you had a look at some of the inbuilt SQL tools to help you with this:
I.e. from the management studio menu:
'Query'->'Display Estimated Execution Plan'
'Query'->'Include Actual Execution Plan'
'Tools'->'Database Engine Tuning Advisor'
Shouldn't the OPTION OPTIMIZE clause be part of the inner SQL, rather than of the SELECT on the CTE?
You should look at indexing the culture field in the news table, and the itemid and categoryid fields in the item category table. You may not need all these indexes - I would try them one at a time, then in combination until you find something that works. Your existing indexes do not seem to help your query very much.
Really need to see the query plan - one thing of note is you put the clustered index for News on News.ID, but it is not an identity field but the FK for the ItemCategory table, this will result in some fragmentation on the news table over time, so it less than ideal.
I suspect the underlying problem is your paging is causing the table to scan.
Updated:
Those Sort's are costing you 68% of the query execution time from the plan, and that makes sense, one of those sorts at least must be to support the ranking function you are using that is based on news.rank desc, but you have no index that can support that ranking natively.
Getting an index in to support that will be interesting, you can try a simple NC index on news.rank first off, SQL may chose to join indexes and avoid the sort, but it will take some experimentation.
Try using for ItemCategory table nonclustered index on itemId,categoryId and on News table also nonclustered index on Rank,Culture.
I have finally come up with the following indexes which are working great and the stored procedure executes in less than a second. I have just removed TotalCount = COUNT(*) OVER() from the query and I couldn't find any good index for that. Maybe I write a separate stored procedure to calculate the total number of records. I may even decide to use a "more" button like in Twitter and Facebook without pagination buttons.
for news table:
CREATE NONCLUSTERED INDEX [IX_News_Rank_Culture_Status_Date] ON [dbo].[News]
(
[Rank] DESC,
[Culture] ASC,
[Status] ASC,
[Date] ASC
)
for ItemNetwork table:
CREATE NONCLUSTERED INDEX [IX_ItemNetwork_ItemID_NetworkID] ON ItemNetwork
(
[ItemID] ASC,
[NetworkID] ASC
)
I just don't know whether ItemNetwork needs a clustered index on ID column. I am never retrieving a record from this table using the ID column. Do you think it's better to have a clustered index on (ItemID, NetworkID) columns?
Please try to change
FROM dbo.News news
JOIN ItemCategory itemCat ON itemCat.ItemID = news.ID
to
FROM dbo.News news
HASH JOIN ItemCategory itemCat ON itemCat.ItemID = news.ID
or
FROM dbo.News news
LOOP JOIN ItemCategory itemCat ON itemCat.ItemID = news.ID
I don't really know what is in your data, but the joining of this tables may be a bottleneck.