Related
I am using SQL Server version 2012. I have a table which has more than 10 million rows. I have to count records using a SQL filter.
My query is this:
select count(*)
from reconcil
where tenantid = 101
which is taking more than 5 minutes for 5 millions records.
Is there any fastest way to count records?
Reconcil table structure is
CREATE TABLE [dbo].[RECONCIL]
(
[AckCode] [nvarchar](50) NULL,
[AckExpireTime] [int] NULL,
[AckFileName] [nvarchar](255) NULL,
[AckKey] [int] NULL,
[AckState] [int] NULL,
[AppMsgKey] [nvarchar](30) NULL,
[CurWrkActID] [nvarchar](50) NULL,
[Date_Time] [datetime] NULL,
[Direction] [nvarchar](1) NULL,
[ErrorCode] [nvarchar](50) NULL,
[FGLOGKEY] [int] NOT NULL,
[FolderID] [int] NULL,
[FuncGCtrlNo] [nvarchar](14) NULL,
[INLOGKEY] [int] NULL,
[InputFileName] [nvarchar](255) NULL,
[IntCtrlNo] [nvarchar](14) NULL,
[IsAssoDataPresent] [nvarchar](1) NULL,
[JobState] [int] NULL,
[LOGDATA] [nvarchar](max) NULL,
[MessageID] [nvarchar](25) NULL,
[MessageState] [int] NULL,
[MessageType] [int] NULL,
[NextWrkActID] [nvarchar](50) NULL,
[NextWrkHint] [nvarchar](20) NULL,
[NONFAERRORLOG] [nvarchar](max) NULL,
[NumberOfBytes] [int] NULL,
[NumberOfSegments] [int] NULL,
[OutputFileName] [nvarchar](255) NULL,
[Priority] [nvarchar](1) NULL,
[ReceiverID] [nvarchar](30) NULL,
[RecNo] [int] NULL,
[RecordID] [int] IDENTITY(1,1) NOT NULL,
[RelationKey] [int] NULL,
[SEGLOG] [nvarchar](max) NULL,
[SenderID] [nvarchar](30) NULL,
[ServerID] [nvarchar](255) NULL,
[Standard] [int] NULL,
[TenantID] [int] NULL,
[TPAgreementKey] [int] NULL,
[TSetCtrlNo] [nvarchar](35) NULL,
[UserKey1] [nvarchar](255) NULL,
[UserKey2] [nvarchar](255) NULL,
[UserKey3] [nvarchar](255) NULL,
CONSTRAINT [RECONCIL_PK]
PRIMARY KEY CLUSTERED ([RecordID] ASC)
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
Unless you materialized the count, this non-clustered index on TenentID will provide better performance because it is narrower than the clustered primary key index and will scan only the matching rows:
CREATE INDEX idx ON [dbo].[RECONCIL](TenantID);
If performance of the aggregate query with this index isn't acceptable, you could create an indexed view with the count. The indexed view will provide the fastest performance for this query but will incur additional costs for storage and index maintenance for inserts and deletes. Also, queries that modify the table must have required SET options for indexed views. Those costs may be justified if the count query is executed often.
SQL Server can use the indexed view automatically in Enterprise (or Developer) editions even if not directly referenced in the query as long as the optimizer can match the semantics of the query using the view. In lesser editions, you'll need to query the indexed view directly and specify the NOEXPAND hint.
CREATE VIEW dbo.VW_RECONCIL_COUNT
WITH SCHEMABINDING
AS
SELECT
TenantID
, COUNT_BIG(*) AS TenentRowCount
FROM [dbo].[RECONCIL]
GROUP BY TenantID;
GO
CREATE UNIQUE CLUSTERED INDEX cdx ON dbo.VW_RECONCIL_COUNT(TenantID);
GO
--Enterprise Edition can use the view index automatically
SELECT COUNT_BIG(*) AS TenentRowCount
FROM [dbo].[RECONCIL]
WHERE TenantID = 101
GROUP BY TenantID;
GO
--other editions require the view to be specified plus the NOEXPAND hint
SELECT TenentRowCount
FROM dbo.VW_RECONCIL_COUNT WITH (NOEXPAND)
WHERE TenantID = 101;
GO
As being suggested, create an index or even partition your table by tenantId if you have so many items. This way you would have one data file per partition which increases performance.
select count(tenantid)
from reconcil
where tenantid = 101 group by tenantid ;
not sure but try using this.
I checked every link of stackoverflow (I Rebuild table too) and others link but no luck.
I have to analyze one application. So I I got backup file, via application utility on particular path. I restored backup file in Sqlserver 2014 full version properly. Then I change the config file of the application to this sqlserver.
After that, when run the application and submit employee form via applicaion its gives the above error.
Employee table db structure as below, it contains 47 columns and does not have any rows( This is first entry).
CREATE TABLE [dbo].[employee](
[wcsuid] [nchar](30) NULL,
[lastupdatedate] [nchar](8) NULL,
[updatedatetime] [varchar](17) NULL,
[locationid] [nchar](20) NULL,
[isactive] [nchar](1) NULL,
[CreatedDate] [varchar](10) NULL,
[sdate] [nchar](90) NULL,
[identityemployeeno] [nchar](45) NULL,
[employeeid] [nchar](180) NULL,
[firstname] [nchar](120) NULL,
[lastname] [nchar](120) NULL,
[ssn] [nchar](99) NULL,
[dob] [nchar](90) NULL,
[gender] [nchar](9) NULL,
[address] [nchar](180) NULL,
[addressLine2] [nchar](180) NULL,
[city] [nchar](105) NULL,
[state] [nchar](180) NULL,
[zipcode] [nchar](135) NULL,
[isinternationaladdress] [nchar](9) NULL,
[telephone] [nchar](126) NULL,
[cellphone] [nchar](126) NULL,
[IdentityJobTitleNo] [nchar](45) NULL,
[manager] [nchar](120) NULL,
[hiredate] [nchar](90) NULL,
[terminationdate] [nchar](90) NULL,
[username] [nchar](90) NULL,
[userpassword] [nchar](90) NULL,
[userpassword_required] [nchar](9) NULL,
[employeestanding] [nchar](180) NULL,
[identityroleid] [nchar](45) NULL,
[loggedin] [nchar](9) NULL,
[punchedin] [nchar](9) NULL,
[punchedinTime] [nchar](90) NULL,
[punchedinDate] [nchar](90) NULL,
[contactname] [nchar](120) NULL,
[contactaddress] [nchar](180) NULL,
[contactaddressLine2] [nchar](180) NULL,
[contactcity] [nchar](105) NULL,
[contactstate] [nchar](30) NULL,
[contactzipcode] [nchar](135) NULL,
[isinternationalcontactaddress] [nchar](9) NULL,
[contactphone] [nchar](126) NULL,
[contactcellphone] [nchar](126) NULL,
[relationtoemployee] [nchar](180) NULL,
[worksoncommission] [nchar](9) NULL,
[overrideitemcommission] [nchar](9) NULL
) ON [PRIMARY]
And insert statement (also have 47 values) with error also show in application as below :
INSERT INTO employee
(
wcsuid,lastupdatedate,updatedatetime,locationid,isactive,CreatedDate,sdate,identityemployeeno,employeeid,firstname,lastname,ssn,dob,gender,address,addressLine2,city,state,zipcode,isinternationaladdress,telephone,cellphone,IdentityJobTitleNo,manager,hiredate,terminationdate,username,
userpassword,userpassword_required,employeestanding,identityroleid,loggedin,punchedin,punchedinTime,punchedinDate,contactname,contactaddress,contactaddressLine2,contactcity,contactstate,contactzipcode,isinternationalcontactaddress,contactphone,contactcellphone,relationtoemployee,worksoncommission,overrideitemcommission)
VALUES
(
'2015121512201580210002','','20151215122015802','','','20151215','','4','','Ajayendra','Raghuvanshi','','19782707','M','','','','','','','0792762063','','','','00000000','00000000','ajayendra',
'SdYHcqaxf1gMjjSkjmpUiw==','N','','','','','','','','','','','','','','','','','N','N')
Is the limit of char fields, but I checked all length with the values and does not get any logic behind error.
So question is How to solved the error and why it is comes?
I am not an SQL Server expert but... Adding all the field length above I got 4,046 (I might have made few mistakes - it is too early here and I am still on the first coffee...). Now looking at the manual:
nchar [ ( n ) ]
Fixed-length Unicode string data. n defines the string length and must be a value from 1 through 4,000. The storage size is two times n bytes. When the collation code page uses double-byte characters, the storage size is still n bytes. Depending on the string, the storage size of n bytes can be less than the value specified for n. The ISO synonyms for nchar are national char and national character..
Note that storing each characters requires two bytes. Doing 2 * 4046 we have 8092 bytes line length! Also note that nchar is fixed length which I assume is allocated when an entry/row is created and thus it generates the error that you see, telling you that the line is too long. I would expect it to warn you at least when you create the table...
Solutions
normalize more: Split the table into two, for example have an emploeeyAddress table which can make sense if you later need home and work addresses for each employee
Use nvarchar which is variable or dynamic length. The storage is still 2 bytes per character but only for the data inserted (not pre-allocated). You will hit the same limit if you try to populate all fields to their maximum length
For nchar values, each character uses up two bytes, so you should multiply your character count by two to get the total row size.
Finally I got the solution. I am in analyze phase, so I can not change in normalize table.
After reading this below link, I understand the issue. So I just recreated a "Employee" table with nvarchar datatype as I have only option.
Thanks #urban to give the right direction and for this I accept his answer.
What is the difference between char, nchar, varchar, and nvarchar in SQL Server?
I have two data sets (190 Million Rows and 100 Million Rows). I need to find the most effective way to merge the two together with no duplicate rows. I can do this via TSQL commands or an SSIS process. Does anyone have any input/experience as to what the most effective way to complete this process is?
Both tables have the same formatting:
CREATE TABLE [dbo].[Table01](
[StudentId] [char](10) NOT NULL,
[CollegeId] [char](6) NOT NULL,
[TermId] [char](3) NOT NULL,
[CourseId] [char](12) NULL,
[Title] [char](68) NULL,
[SectionId] [char](6) NULL,
[UnitsEarned] [decimal](5, 2) NULL,
[Grade] [char](3) NULL,
[CreditFlag] [char](1) NULL,
[UnitsAttempted] [decimal](5, 2) NULL,
[TopCode] [char](6) NULL,
[TransferStatus] [char](1) NULL,
[UnitsMax] [decimal](5, 2) NULL,
[BSStatus] [char](1) NULL,
[SamCode] [char](1) NULL,
[ClassCode] [char](1) NULL,
[CollegeLevel] [char](1) NULL,
[NCRCategory] [char](1) NULL,
[CCLongTermId] [char](5) NULL,
[batch_id] [int] NULL
)
These are the fields that need to be distinct to eliminate duplicates:
[StudentId]
[CollegeId]
[TermId]
[CourseId]
The server that will be running this process has 8 cores, 32GB RAM, and SQL Server 2012.
Create a composite clustered index on all 4 columns on both tables. Create the destination table with a standard identity field primary key clustered index, but with a composite nonclustered index on those 4 columns. Insert into destination table from table01, using a derived table that also exposes ROW_NUMBER() OVER (PARTITION BY StudentId, CollegeId, TermId, CourseId), and filter out all rows except for those with ROW_NUMBER of 1. That will dedupe table01. Then do the same thing with table02, but also using a NOT EXISTS to check against the destination table to make sure that a row does not already exist.
Actually I want some records from my database. There is a field Time_From. I just want to show only those records which are after DateTime.Now().
But it is not showing, it shows records before and after DateTime.Now()
Select *
from mytable
where Vehicle_Booking_Date= #Vehicle_Booking_Date
AND Time_From > #Time_From order by Time_From"
The table definition (from the comment)
CREATE TABLE [dbo].[mtblVehicle_Booking]
(
[Sno] [int] IDENTITY(1,1) NOT NULL,
[Vehicle_Number] [nvarchar](100) NULL,
[Vehicle_Booking_Date] [datetime] NULL,
[Time_From] [datetime] NULL,
[Time_To] [datetime] NULL,
[Vehicle_Used_By] [varchar](100) NULL,
[Vehicle_Booked_By] [varchar](100) NULL,
[Cost_Code] [nvarchar](50) NULL,
[Budget_Line] [nvarchar](50) NULL,
[Purpose] [varchar](20) NULL,
[Destination] [nvarchar](500) NULL,
[Trip_Details] [nvarchar](500) NULL
)
ON [PRIMARY]
If you want to select all records whose Time_From is greater than current time and date is equal to specified date,
Select *
from mytable
where Vehicle_Booking_Date= #Vehicle_Booking_Date
AND Time_From > now() order by Time_From'
You have not mentioned what database you are using. But concept remains same. Just substract the other date time from current time. If it is greater than 0 it means that the given date time is greater than the current date time.
Sample SQL
Select *
from mytable
where Vehicle_Booking_Date= #Vehicle_Booking_Date
AND DATEDIFF(Time_From,SYSDATETIME()) > 0
Hope this helps.
I have been playing around with a quite complex SQL Statement for a few days, and have gotten most of it working correctly.
I am having trouble with one last part, and was wondering if anyone could shed some light on the issue, as I have no idea why it isnt working:
INSERT INTO ExistingClientsAccounts_IMPORT
SELECT DISTINCT
cca.AccountID, cca.SKBranch, cca.SKAccount, cca.SKName, cca.SKBase,
cca.SyncStatus, cca.SKCCY, cca.ClientType, cca.GFCID, cca.GFPID, cca.SyncInput,
cca.SyncUpdate, cca.LastUpdatedBy, cca.Deleted, cca.Branch_Account, cca.AccountTypeID
FROM ClientsAccounts AS cca
INNER JOIN
(SELECT DISTINCT ClientAccount, SKAccount, SKDesc,
SKBase, SKBranch, ClientType, SKStatus, GFCID,
GFPID, Account_Open_Date, Account_Update
FROM ClientsAccounts_IMPORT) AS ccai
ON cca.Branch_Account = ccai.ClientAccount
Table definitions follow:
CREATE TABLE [dbo].[ExistingClientsAccounts_IMPORT](
[AccountID] [int] NOT NULL,
[SKBranch] [varchar](2) NOT NULL,
[SKAccount] [varchar](12) NOT NULL,
[SKName] [varchar](255) NULL,
[SKBase] [varchar](16) NULL,
[SyncStatus] [varchar](50) NULL,
[SKCCY] [varchar](5) NULL,
[ClientType] [varchar](50) NULL,
[GFCID] [varchar](10) NULL,
[GFPID] [varchar](10) NULL,
[SyncInput] [smalldatetime] NULL,
[SyncUpdate] [smalldatetime] NULL,
[LastUpdatedBy] [varchar](50) NOT NULL,
[Deleted] [tinyint] NOT NULL,
[Branch_Account] [varchar](16) NOT NULL,
[AccountTypeID] [int] NOT NULL
) ON [PRIMARY]
CREATE TABLE [dbo].[ClientsAccounts_IMPORT](
[NEWClientIndex] [bigint] NOT NULL,
[ClientGroup] [varchar](255) NOT NULL,
[ClientAccount] [varchar](255) NOT NULL,
[SKAccount] [varchar](255) NOT NULL,
[SKDesc] [varchar](255) NOT NULL,
[SKBase] [varchar](10) NULL,
[SKBranch] [varchar](2) NOT NULL,
[ClientType] [varchar](255) NOT NULL,
[SKStatus] [varchar](255) NOT NULL,
[GFCID] [varchar](255) NULL,
[GFPID] [varchar](255) NULL,
[Account_Open_Date] [smalldatetime] NULL,
[Account_Update] [smalldatetime] NULL,
[SKType] [varchar](255) NOT NULL
) ON [PRIMARY]
The error message I get is:
Msg 8152, Level 16, State 14, Line 1
String or binary data would be truncated.
The statement has been terminated.
The error is because you are trying to insert data into a column in ExistingClientsAccounts_IMPORT where the column size is smaller than the length of data attempting to be inserted into it.
e.g.
SKAccount column is VARCHAR(12) in the ExistingClientsAccounts_IMPORT table but is VARCHAR(255) in ClientsAccounts_IMPORT.
So if ClientsAccounts_IMPORT contains any rows where that field is longer than 12 characters, you will get that error as obv. e.g. 100 characters will not fit into a 12 character field.
You need to make sure all the columns in the table you are inserting into, are big enough - make sure each column definition matches the source table.
The third column of your SELECT column list means that ExistingClientsAccounts_IMPORT.SKAccount is populated from ClientsAccounts.SKAccount - however, the source is up to 255 characters, while the destination has a capacity of 12. If there's any data that wouldn't fit, you'll get this message.
I haven't gone through all the other columns.
You are trying to insert values which are greater than tha max length specified for a column. Use a profiler to check the data being passed to this query and verify the length of data against the permissible length for all columns.
There is a clear mismatch in the column lenghts of common columns of these two tables.
ClientsAccounts_IMPORT.SKBase is 10 whereas it is 16 in the other table.
Check the field definitions. You can see some are smaller than the original ones. Now run a query on the old data - you will find some of the larger fields were used, so the insert is not possible without losing data.
Example: SKAccount - from 255 length to 12.