Related
I have an issue with one of my MSSQL tables. The table has 1.2 Trillion rows about 1TB of data and growing. The table is partitioned into 8000 partitions, only about 800 are used. The others were created for expanded growth.
Inserts and selects are < 1s
My problem is Updates are very slow. To updated 1 record varchar(100) using the exact partition key and Identity Column key, it is 3s.
If I place the update code in a Stored Procedure, goes in less than 1s. If I add option (recompile), about 1s. Is there a way to fix this without adding recomile?
Thank you.
CREATE TABLE [dbo].[_tabletest](
[data_id] [bigint] IDENTITY(10000,1) NOT NULL,
[idx1_id] [smallint] NOT NULL,
[idx2_id] [bigint] NOT NULL,
[idx3_id] [bigint] NOT NULL,
[template_id] [bigint] NOT NULL,
[reference_id] [varchar](200) NOT NULL,
[data] [nvarchar](400) NULL,
[data_1] [tinyint] NULL,
[data_2] [varchar](max) NULL,
[data_3] [real] NULL,
[data_4] [tinyint] NULL,
[data_5] [tinyint] NULL,
[data_6] [bit] NULL,
[data_7] [nvarchar](50) NULL,
[data_8] [datetime] NULL,
[data_9] [nvarchar](50) NULL,
[data_10] [varchar](100) NULL,
[data_11] [varchar](100) NULL,
[data_12] [varchar](100) NULL,
[data_13] [varchar](300) NULL,
[data_14] [varchar](200) NULL,
[data_15] [uniqueidentifier] NULL,
[data_16] [varchar](600) NULL,
[data_17] [varchar](100) NULL,
[data_18] [varchar](100) NULL,
[data_19] [decimal](10, 5) NULL,
[data_20] [decimal](10, 5) NULL,
[data_21] [decimal](10, 5) NULL,
[data_22] [decimal](10, 5) NULL,
[data_23] [decimal](10, 5) NULL,
[data_24] [int] NULL,
[data_25] [int] NULL,
[data_26] [int] NULL,
[data_27] [int] NULL,
[data_28] [decimal](10, 5) NULL,
[data_29] [decimal](10, 5) NULL,
[data_30] [decimal](10, 5) NULL,
[data_31] [datetime] NULL,
[data_32] [decimal](10, 5) NULL,
[data_3] [bit] NULL,
[data_34] [varchar](max) NULL,
[data_35] [smallint] NULL,
[data_36] [bigint] NULL,
[data_37] [int] NULL,
[data_38] [real] NULL,
[data_39] [datetime] NULL,
[data_40] [varchar](2500) NULL,
CONSTRAINT [PK_data_id] PRIMARY KEY CLUSTERED
(
[idx1_id] ASC,
[idx2_id] ASC,
[idx3_id] ASC,
[data_id] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90)
)
GO
ALTER TABLE [dbo].[_tabletest] SET (LOCK_ESCALATION = AUTO)
GO
ALTER TABLE [dbo].[_tabletest] ADD CONSTRAINT [DF_1] DEFAULT ((0)) FOR [data_1]
GO
ALTER TABLE [dbo].[_tabletest] ADD CONSTRAINT [DF_2] DEFAULT (getutcdate()) FOR [data_39]
GO
ALTER TABLE [dbo].[_tabletest] ADD DEFAULT (newid()) FOR [data_15]
GO
UPDATE [_tabletest] SET [data_40] = 'test-data'
WHERE [idx1_id] = 1209 AND [idx2_id] = 113795 AND [idx3] = 41195716 AND [data_id] = 1329110156
;
I have the following query that will be used to fetch data from legacy tables. It's no surprise but the amount of data is huge and thus it takes a long time. The first select takes 40 minutes to run using an empty dbo.commodities_copy table as a starting point and yields around 26,000 rows. Keep in mind that there are separate databases: STAGING and PRESTAGING and that some joins are made using non-PK fields, which is most definately making an impact in its performance. This is something that I cannot fix, due to the way data was organized from the start. Also the transaction table has around 1 million rows, which also impacts heavily on performance. The entire script takes a total of 3.5 hours to execute when using an EMPTY dbo.commodities_copy table. I have not tested on insertion to a table with data.
The goal of the query is to get commodity information from the transaction table (if you guessed this was supposed to be noSQL data, you guessed right) and if the commodity code exists in the commodity table, do not insert a commodity in it.
The group bys are absolutely needed to get around duplicates, since a transactions may share the same commodity. The commodity code should be unique in the commodities table, but currently it is not - though if it helps, it's possible we could alter it.
What can I do to speed it up?
INSERT INTO STAGING.dbo.commodities_copy
(commodity_code,
short_description_sched_b,
short_description_sched_tsusa,
long_description_sched_b,
long_description_sched_tsusa,
measurement_unit_1_sched_b,
measurement_unit_1_sched_tsusa,
measurement_unit_2_sched_b,
measurement_unit_2_sched_tsusa,
end_use_sched_b,
end_use_sched_tsusa,
year,
created_by,
created_on,
taxable_sched_b,
taxable_sched_tsusa,
non_taxable_sched_b,
non_taxable_sched_tsusa,
fk_sic_sched_b,
fk_sic_sched_tsusa,
chapter,
header,
sub_header,
needs_validation)
SELECT
--Distinct
Commodity_Code,
iif(miob2.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), miob2.DESC_COMM) as short_commmodity_description_b,
iif(mio2tsusa.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), mio2tsusa.DESC_COMM) as short_commmodity_description_tsusa,
socrata.Commodity_description as long_commodity_description_b,
socrata.Commodity_description as long_commodity_description_tsusa,
iif(miob2.UNIDAD is null, socrata.unit_1, miob2.UNIDAD) as unit_1_b,
iif(mio2tsusa.UNIDAD is null, socrata.unit_1, mio2tsusa.UNIDAD) as unit_1_tsusa,
MAX(socrata.unit_2) as unit_2_b,
MAX(socrata.unit_2) as unit_2_tsusa,
socrata.end_use_e as end_use_b,
socrata.end_use_i as end_use_tsusa,
MAX(socrata.[year]),
'system' as created_by,
getdate() as created_on,
miob.TRIBUTA as taxable_b,
miotsusa.TRIBUTA as taxable_tsusa,
miob.NTRIBUTA as non_taxable_b,
miotsusa.NTRIBUTA as non_taxable_tsusa,
sicb.id as sic_id_b,
sictsusa.id as sic_id_tsusa,
SUBSTRING(Commodity_Code, 1, 2) as chapter,
SUBSTRING(Commodity_Code, 1, 4) as header,
SUBSTRING(Commodity_Code, 1, 6) as sub_header,
0 as needs_validation
FROM PRE_STAGING.dbo.TRANSACTIONS_FROM_SOCRATA socrata
Left join PRE_STAGING.DBO.MIOB_TBL miob ON miob.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MSCHB_TBL miob2 ON miob2.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MIOTSUSA_TBL miotsusa ON miotsusa.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MTSUSA_TBL mio2tsusa ON mio2tsusa.COMM=socrata.Commodity_Code
Left join STAGING.dbo.sics_altered sicb ON sicb.sic_code = miob.SIC
Left join STAGING.dbo.sics_altered sictsusa ON sictsusa.sic_code = miotsusa.SIC
WHERE NOT EXISTS
(Select Distinct commodity_code from STAGING.dbo.commodities_copy)
group by
Commodity_Code,
iif(miob2.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), miob2.DESC_COMM),
iif(mio2tsusa.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), mio2tsusa.DESC_COMM),
socrata.Commodity_description,
socrata.Commodity_description,
iif(miob2.UNIDAD is null, socrata.unit_1, miob2.UNIDAD),
iif(mio2tsusa.UNIDAD is null, socrata.unit_1, mio2tsusa.UNIDAD),
socrata.end_use_e,
socrata.end_use_i,
miob.TRIBUTA,
miotsusa.TRIBUTA,
miob.NTRIBUTA,
miotsusa.NTRIBUTA,
sicb.id,
sictsusa.id,
SUBSTRING(Commodity_Code, 1, 2),
SUBSTRING(Commodity_Code, 1, 4),
SUBSTRING(Commodity_Code, 1, 6)
The tables used are the following:
STAGING.dbo.commodities_copy:
CREATE TABLE [dbo].[commodities_copy](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[chapter] [varchar](5) NULL,
[header] [varchar](5) NULL,
[sub_header] [varchar](10) NULL,
[commodity_code] [varchar](20) NULL,
[short_description_sched_b] [varchar](100) NULL,
[long_description_sched_b] [varchar](200) NULL,
[measurement_unit_1_sched_b] [varchar](5) NULL,
[measurement_unit_2_sched_b] [varchar](5) NULL,
[end_use_sched_b] [int] NULL,
[sitc_sched_b] [varchar](20) NULL,
[usda_sched_b] [int] NULL,
[hitech_sched_b] [int] NULL,
[naics_fk_id_sched_b] [bigint] NULL,
[short_description_sched_tsusa] [varchar](100) NULL,
[long_description_sched_tsusa] [varchar](200) NULL,
[measurement_unit_1_sched_tsusa] [varchar](5) NULL,
[measurement_unit_2_sched_tsusa] [varchar](5) NULL,
[end_use_sched_tsusa] [int] NULL,
[sitc_sched_tsusa] [varchar](20) NULL,
[usda_sched_tsusa] [int] NULL,
[hitech_sched_tsusa] [int] NULL,
[naics_fk_id_sched_tsusa] [bigint] NULL,
[year] [int] NOT NULL,
[created_on] [datetime] NOT NULL,
[created_by] [varchar](50) NULL,
[updated_on] [datetime] NULL,
[updated_by] [varchar](50) NULL,
[needs_validation] [bit] NOT NULL,
[taxable_sched_b] [nchar](3) NULL,
[non_taxable_sched_b] [nchar](3) NULL,
[taxable_sched_tsusa] [nchar](3) NULL,
[non_taxable_sched_tsusa] [nchar](3) NULL,
[fk_sic_sched_b] [bigint] NULL,
[fk_sic_sched_tsusa] [bigint] NULL
) ON [PRIMARY]
STAGING.dbo.sics_altered:
CREATE TABLE [dbo].[sics_altered](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[sic_code] [varchar](4) NULL,
[sic_description] [varchar](max) NULL,
[created_on] [datetime] NOT NULL,
[created_by] [varchar](50) NOT NULL,
PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
The rest are in PRESTAGING:
PRESTAGING.dbo.TRANSACTIONS_FROM_SOCRATA:
This is the table with 1.3 million rows
CREATE TABLE [dbo].[TRANSACTIONS_FROM_SOCRATA](
[Trade] [varchar](255) NULL,
[Year] [varchar](255) NULL,
[Month] [varchar](50) NULL,
[Commodity_Code] [varchar](50) NULL,
[Commodity_Short_Name] [varchar](255) NULL,
[Commodity_description] [varchar](255) NULL,
[cty_code] [varchar](50) NULL,
[Country] [varchar](50) NULL,
[Subcountry_code] [varchar](50) NULL,
[district] [varchar](50) NULL,
[dist_name] [varchar](255) NULL,
[data] [varchar](50) NULL,
[sitc] [varchar](50) NULL,
[SITC_Short_Desc] [varchar](255) NULL,
[SITC_Long_Desc] [varchar](255) NULL,
[naics] [varchar](50) NULL,
[NAICS_description] [varchar](255) NULL,
[end_use_i] [varchar](50) NULL,
[end_use_e] [varchar](50) NULL,
[hts_desc] [varchar](255) NULL,
[unit_1] [varchar](50) NULL,
[qty_1] [varchar](50) NULL,
[unit_2] [varchar](50) NULL,
[qty_2] [varchar](50) NULL,
[ves_val_mo] [varchar](50) NULL,
[ves_wgt_mo] [varchar](50) NULL,
[cards_mo] [varchar](50) NULL,
[air_val_mo] [varchar](50) NULL,
[air_wgt_mo] [varchar](50) NULL,
[dut_val_mo] [varchar](50) NULL,
[cal_dut_mo] [varchar](50) NULL,
[con_cha_mo] [varchar](50) NULL,
[con_cif_mo] [varchar](50) NULL,
[gen_val_mo] [varchar](50) NULL,
[gen_cha_mo] [varchar](50) NULL,
[gen_cif_mo] [varchar](50) NULL,
[air_cha_mo] [varchar](50) NULL,
[ves_cha_mo] [varchar](50) NULL,
[cnt_cha_mo] [varchar](50) NULL,
[rev_data] [varchar](50) NULL
) ON [PRIMARY]
PRESTAGING.dbo.MIOB_TBL:
CREATE TABLE [dbo].[MIOB_TBL](
[id] [int] IDENTITY(1,1) NOT NULL,
[COMM] [nchar](10) NOT NULL,
[INSUMO] [nchar](3) NULL,
[PBTO] [nchar](4) NULL,
[SIC] [nchar](4) NULL,
[NAICS] [nchar](6) NULL,
[TRIBUTA] [nchar](3) NULL,
[NTRIBUTA] [nchar](3) NULL,
[LAST_UPDATE] [date] NULL,
[LAST_UPDATED_BY] [nchar](20) NULL,
[CREATION_DATE] [date] NULL,
[CREATED_BY] [nchar](15) NULL,
[migrated_on] [datetime] NOT NULL,
PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
PRESTAGING.dbo.MIOTSUSA_TBL:
CREATE TABLE [dbo].[MIOTSUSA_TBL](
[COMM] [nchar](10) NOT NULL,
[INSUMO] [nchar](3) NULL,
[PBTO] [nchar](4) NULL,
[SIC] [nchar](4) NULL,
[NAICS] [nchar](6) NULL,
[TRIBUTA] [nchar](3) NULL,
[NTRIBUTA] [nchar](3) NULL,
[id] [int] IDENTITY(1,1) NOT NULL,
[migrated_on] [datetime] NOT NULL
) ON [PRIMARY]
PRESTAGING.dbo.MSCHB_TBL:
CREATE TABLE [dbo].[MSCHB_TBL](
[id] [int] IDENTITY(1,1) NOT NULL,
[COMM] [nchar](10) NOT NULL,
[DESC_COMM] [nchar](50) NULL,
[UNIDAD] [nchar](3) NULL,
[LAST_UPDATE] [date] NULL,
[LAST_UPDATED_BY] [nchar](20) NULL,
[CREATION_DATE] [date] NULL,
[CREATED_BY] [nchar](15) NULL,
[migrated_on] [datetime] NOT NULL,
PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
PRESTAGING.dbo.MTSUSA_TBL
CREATE TABLE [dbo].[MTSUSA_TBL](
[COMM] [nchar](10) NOT NULL,
[DESC_COMM] [nchar](50) NULL,
[UNIDAD] [nchar](3) NULL,
[id] [int] IDENTITY(1,1) NOT NULL,
[migrated_on] [datetime] NOT NULL
) ON [PRIMARY]
Let me know if there's anything else I need to provide.
With all those left outer joins, the query optimizer has to start with TRANSACTION_FROM_SOCRATA, so I would start with that. The only filtering is the NOT IN clause--would that cut down the 1MM rows to something more reasonable? If not, you're pretty much doomed to running at least one table scan (and possibly several) on the entire table.
If filtering on Commodity_Code would significantly cut things down, that can only be done if the column is indexed, so that SQL can find and read only those rows. It can only do that if there is an index on column--otherwise you're back to a table scan. Similarly, having an index on commodity_code in table commodities_copy` would help as well, if that table is large.
As discussed in the comments, a NOT EXISTS check would be most efficient, written as a correlated subquery:
WHERE NOT EXISTS (select commodity_code
from STAGING.dbo.commodities_copy
where commodity_code = socrtata.Commodity_Code)
(I'd want to do a lot of testing on this, checking and double-checking everything. Improving performance is tricky, doubly so when done through SO.)
Try this,
create table #socrata(Commodity_Code varchar(100),unit_2_b varchar(50),unit_2_tsusa varchar(50),[year] varchar(50))
insert into #socrata
SELECT
Commodity_Code,
MAX(socrata.unit_2) as unit_2_b,
MAX(socrata.unit_2) as unit_2_tsusa,
MAX(socrata.[year]),
FROM PRE_STAGING.dbo.TRANSACTIONS_FROM_SOCRATA socrata
group by Commodity_Code
SELECT
--Distinct
Commodity_Code,
iif(miob2.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), miob2.DESC_COMM) as short_commmodity_description_b,
iif(mio2tsusa.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), mio2tsusa.DESC_COMM) as short_commmodity_description_tsusa,
socrata.Commodity_description as long_commodity_description_b,
socrata.Commodity_description as long_commodity_description_tsusa,
iif(miob2.UNIDAD is null, socrata.unit_1, miob2.UNIDAD) as unit_1_b,
iif(mio2tsusa.UNIDAD is null, socrata.unit_1, mio2tsusa.UNIDAD) as unit_1_tsusa,
unit_2_b,
unit_2_tsusa,
socrata.end_use_e as end_use_b,
socrata.end_use_i as end_use_tsusa,
[year],
'system' as created_by,
getdate() as created_on,
miob.TRIBUTA as taxable_b,
miotsusa.TRIBUTA as taxable_tsusa,
miob.NTRIBUTA as non_taxable_b,
miotsusa.NTRIBUTA as non_taxable_tsusa,
sicb.id as sic_id_b,
sictsusa.id as sic_id_tsusa,
SUBSTRING(Commodity_Code, 1, 2) as chapter,
SUBSTRING(Commodity_Code, 1, 4) as header,
SUBSTRING(Commodity_Code, 1, 6) as sub_header,
0 as needs_validation
FROM #socrata socrata
Left join PRE_STAGING.DBO.MIOB_TBL miob ON miob.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MSCHB_TBL miob2 ON miob2.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MIOTSUSA_TBL miotsusa ON miotsusa.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MTSUSA_TBL mio2tsusa ON mio2tsusa.COMM=socrata.Commodity_Code
Left join STAGING.dbo.sics_altered sicb ON sicb.sic_code = miob.SIC
Left join STAGING.dbo.sics_altered sictsusa ON sictsusa.sic_code = miotsusa.SIC
WHERE NOT EXISTS
(Select commodity_code from STAGING.dbo.commodities_copy where commodity_code = socrtata.Commodity_Code)
if Read uncommitted data is not a concern then you can use with (nolock)
Also your exists clause was wrong and no need of distinct.check rest of the changes.
UPDATE MysterySpells SET SpellId = (SELECT Id FROM Spells
WHERE Name = 'unseen servant')
WHERE MysteryId = 1 And ClassLevel = 2
I know the subquery returns 589. Anyone have any clues why I would be getting this error:
The UPDATE statement conflicted with the FOREIGN KEY constraint
"FK_MysterySpells_Spells". The conflict occurred in database "x",
table "dbo.Spells", column 'Id'.
CODE UPDATE
CREATE TABLE [dbo].[Spells](
[Id] [int] IDENTITY(1,1) NOT NULL,
[OldId] [int] NULL,
[Name] [nvarchar](100) NOT NULL,
[School] [nvarchar](50) NULL,
[SubSchool] [nvarchar](50) NULL,
[SchoolID] [int] NOT NULL,
[SubSchoolID] [int] NULL,
[CastingTime] [nvarchar](250) NULL,
[Components] [nvarchar](250) NULL,
[IsCostly] [bit] NOT NULL,
[Cost] [int] NULL,
[RangeDescription] [nvarchar](250) NULL,
[Range] [int] NOT NULL,
[RangeIncrement] [int] NOT NULL,
[RangeGap] [int] NOT NULL,
[Area] [int] NOT NULL,
[AreaIncrement] [int] NOT NULL,
[AreaGap] [int] NOT NULL,
[AreaNote] [nvarchar](250) NULL,
[Description] [nvarchar](max) NULL,
[ShortDescription] [nvarchar](250) NULL,
[Targets] [nvarchar](250) NULL,
[Effect] [nvarchar](250) NULL,
[Duration] [nvarchar](250) NULL,
[SavingThrow] [nvarchar](250) NULL,
[SpellResistence] [nvarchar](250) NULL,
[HasVerbal] [bit] NOT NULL,
[HasSomatic] [bit] NOT NULL,
[HasMaterial] [bit] NOT NULL,
[HasFocus] [bit] NOT NULL,
[HasDivineFocus] [bit] NOT NULL,
[WizardLevel] [int] NULL,
[ClericLevel] [int] NULL,
[AdeptLevel] [int] NULL,
[AlchemistLevel] [int] NULL,
[AntipaladinLevel] [int] NULL,
[BardLevel] [int] NULL,
[BloodragerLevel] [int] NULL,
[DruidLevel] [int] NULL,
[InquisitorLevel] [int] NULL,
[MagusLevel] [int] NULL,
[OracleLevel] [int] NULL,
[PaladinLevel] [int] NULL,
[RangerLevel] [int] NULL,
[ShamanLevel] [int] NULL,
[SorcererLevel] [int] NULL,
[SummonerLevel] [int] NULL,
[WitchLevel] [int] NULL,
[SLALevel] [int] NULL,
[IsDismissible] [bit] NOT NULL,
[IsLanguageDependent] [bit] NOT NULL,
[IsShapeable] [bit] NOT NULL,
[IsAcid] [bit] NOT NULL,
[IsAir] [bit] NOT NULL,
[IsCold] [bit] NOT NULL,
[IsCurse] [bit] NOT NULL,
[IsDarkness] [bit] NOT NULL,
[IsDeath] [bit] NOT NULL,
[IsDisease] [bit] NOT NULL,
[IsEarth] [bit] NOT NULL,
[IsElectricity] [bit] NOT NULL,
[IsEmotion] [bit] NOT NULL,
[IsFear] [bit] NOT NULL,
[IsFire] [bit] NOT NULL,
[IsForce] [bit] NOT NULL,
[IsLight] [bit] NOT NULL,
[IsPain] [bit] NOT NULL,
[IsPoison] [bit] NOT NULL,
[IsShadow] [bit] NOT NULL,
[IsSonic] [bit] NOT NULL,
[IsWater] [bit] NOT NULL,
[IsChaotic] [bit] NOT NULL,
[IsEvil] [bit] NOT NULL,
[IsGood] [bit] NOT NULL,
[IsLawful] [bit] NOT NULL,
[IsMindAffecting] [bit] NOT NULL,
[IsMythic] [bit] NULL,
[MythicDescription] [nvarchar](max) NULL,
[Augmented] [nvarchar](max) NULL,
CONSTRAINT [PK_Spells] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
CREATE TABLE [dbo].[MysterySpells](
[Id] [int] IDENTITY(1,1) NOT NULL,
[MysteryId] [int] NOT NULL,
[ClassLevel] [int] NOT NULL,
[SpellId] [int] NOT NULL,
[SpellLevel] [int] NULL,
[Restrictions] [varchar](100) NULL,
CONSTRAINT [PK_MysterySpells] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[MysterySpells] WITH NOCHECK ADD CONSTRAINT [FK_MysterySpells_Mysteries] FOREIGN KEY([MysteryId])
REFERENCES [dbo].[Mysteries] ([Id])
GO
ALTER TABLE [dbo].[MysterySpells] CHECK CONSTRAINT [FK_MysterySpells_Mysteries]
GO
ALTER TABLE [dbo].[MysterySpells] WITH NOCHECK ADD CONSTRAINT [FK_MysterySpells_Spells] FOREIGN KEY([SpellId])
REFERENCES [dbo].[Spells] ([Id])
GO
ALTER TABLE [dbo].[MysterySpells] CHECK CONSTRAINT [FK_MysterySpells_Spells]
GO
If I were you, here is how I debug
If you expect
SELECT Id FROM Spells WHERE Name = 'unseen servant'
to return 589
I will hard code it to 1 first.
UPDATE MysterySpells SET SpellId = 589
WHERE MysteryId = 1 And ClassLevel = 2
If the error still there, I will check if table Spells contains value where Id = 589 and Name = 'unseen servant'.
I believe you will find the root cause somewhere in the process.
Note: It would be better if you show us the table structure and data inside.
The issue is that it does not know what value to grab. There is no mapping defined between spells and mystery spells in your query. The current update is actually trying to update each row in MysterySpells with 589 records.
If you simply wanted to map all of your mystery spells to a single spell you can do:
UPDATE MysterySpells SET SpellId = (SELECT TOP 1 Id FROM Spells
WHERE Name = 'unseen servant')
WHERE MysteryId = 1 And ClassLevel = 2
If there is a pre-existing mapping between these 2 tables and you are expecting to update MysterySpells based on a relationship with spells table:
UPDATE MysterySpells
SET SpellID = t2.ID
FROM MysterySpells t1
JOIN Spells t2 ON t2.relation = t1.relation
If there isn't any existing mapping and you are trying to do this to create this as the mapping you have few options.
1) If no other relationship exists map by hand
2) If their is a relationship that can be calculated, but is not available through an existing join, you can take advantage of the row and partition functions to relate the two. Then use an update similar to the 2nd code block I posted.
I have a table with approximately 1 million rows. Part of our maintenance involves deleting old row each day, but this is taking about 40 minutes.
The delete statement is:
DELETE
FROM [dbGlobalPricingMatrix].[dbo].[tblPricing]
WHERE type = 'car'
AND capid NOT IN
(SELECT cder_id FROM PUB_CAR.dbo.CapDer WHERE cder_discontinued
IS NULL OR DATEDIFF(dd,cder_discontinued,GETDATE()) <= 7)
AND source = #source
Is there anything I can do to improve the performance?
Thanks
As Requested:
CREATE TABLE [dbo].[tblPricing](
[id] [int] IDENTITY(1,1) NOT NULL,
[type] [varchar](50) NULL,
[capid] [int] NULL,
[source] [varchar](50) NULL,
[product] [varchar](50) NULL,
[term] [int] NULL,
[milespa] [int] NULL,
[maintained] [bit] NULL,
[price] [money] NULL,
[created] [datetime] NULL,
[updated] [datetime] NULL,
[notes] [varchar](1000) NULL,
[painttype] [char](1) NULL,
[activeflag] [bit] NULL,
[DealerId] [int] NULL,
[FunderId] [int] NULL,
[IsSpecial] [bit] NULL,
[username] [varchar](50) NULL,
[expiry] [datetime] NULL,
CONSTRAINT [PK_tblPricing] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[CAPDer](
[cder_ID] [int] NOT NULL,
[cder_capcode] [char](20) NULL,
[cder_mancode] [int] NULL,
[cder_rancode] [int] NULL,
[cder_modcode] [int] NULL,
[cder_trimcode] [int] NULL,
[cder_name] [varchar](50) NULL,
[cder_introduced] [datetime] NULL,
[cder_discontinued] [datetime] NULL,
[cder_orderno] [int] NULL,
[cder_vehiclesector] [tinyint] NULL,
[cder_doors] [tinyint] NULL,
[cder_drivetrain] [char](1) NULL,
[cder_fueldelivery] [char](1) NULL,
[cder_transmission] [char](1) NULL,
[cder_fueltype] [char](1) NULL,
CONSTRAINT [PK_CapDer] PRIMARY KEY CLUSTERED
(
[cder_ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
) ON [PRIMARY]
Okay, as I suspected, you do not seem to have indexes for the fields that you reference in your DELETE query. So, add indexes for type, capid, cder_discontinued, and source.
Additionally, you might want to try AND capid IN (SELECT cder_id FROM PUB_CAR.dbo.CapDer WHERE cder_discontinued IS NOT NULL AND DATEDIFF(dd,cder_discontinued,GETDATE()) > 7). The optimizer of MS-SQL-Server should actually be doing this for you, but you never know, it is worth trying.
I have the following query that I am running on my database server but it takes about 30 seconds to run and I can't work out why this is.
SELECT *
FROM [dbo].[PackageInstance] AS packInst
INNER JOIN [dbo].[PackageDefinition] AS packageDef
ON packInst.[PackageDefinitionID] = packageDef.[PackageDefinitionID]
LEFT OUTER JOIN [dbo].[PackageInstanceContextDef] AS contextDef
ON packInst.[PackageInstanceID] = contextDef.[PackageInstanceID]
This produced the following execution plan which to me looks to be good....so I can't understand why it takes so much time to execute where the resulting data is only 100,000 records (which should be a walk in the park for SQL Server).
Any ideas what could be causing this long execution time?
I have looked at the query in Profiler to see what the stats where on it and they are as follows:
CPU - 4711
Reads - 744453
Writes - 9
Duration - 26329
The following are the table definitions:
CREATE TABLE [dbo].[PackageDefinition](
[PackageDefinitionID] [int] IDENTITY(1,1) NOT NULL,
[ts] [timestamp] NOT NULL,
[ProgramID] [int] NULL,
[VendorID] [int] NULL,
[PackageExecutionTypeID] [int] NULL,
[PackageDefinitionStatusID] [int] NOT NULL,
[IsInternal] [bit] NOT NULL,
[Name] [dbo].[D_Name] NOT NULL,
[Description] [dbo].[D_Description] NOT NULL,
[CreatedDate] [datetime] NOT NULL,
[PublishedDate] [datetime] NULL,
[OwnerUserGuid] [uniqueidentifier] NOT NULL,
[ProcessDefinitionMainID] [int] NULL,
[KeyInfoHtml] [nvarchar](max) NULL,
[DescriptionHtml] [nvarchar](max) NULL,
[WhatToExpectHtml] [nvarchar](max) NULL,
[BestPracticesHtml] [nvarchar](max) NULL,
[RecommendedJourneysHtml] [nvarchar](max) NULL,
[RequiresSLAAgreement] [bit] NOT NULL,
[SLAFileAssetID] [int] NULL,
[ImageDataID] [int] NULL,
[VideoHtml] [nvarchar](max) NULL,
[VideoAssetID] [int] NULL,
[UseMapCosts] [bit] NOT NULL,
[CostMin] [money] NOT NULL,
[CostMax] [money] NOT NULL,
[LandingPageVisitCount] [int] NOT NULL,
[IsDeleted] [dbo].[D_IsDeleted] NOT NULL,
[CreatedByUserGuid] [uniqueidentifier] NOT NULL,
[OrderHtml] [nvarchar](max) NULL,
CONSTRAINT [PK_PackageDefinition] PRIMARY KEY CLUSTERED
(
[PackageDefinitionID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[PackageInstance](
[PackageInstanceID] [int] IDENTITY(1,1) NOT NULL,
[ts] [timestamp] NOT NULL,
[PackageDefinitionID] [int] NOT NULL,
[PackageStatusID] [int] NOT NULL,
[Name] [dbo].[D_Description] NOT NULL,
[CampaignID] [int] NULL,
[MarketingPlanID] [int] NULL,
[CountryID] [int] NULL,
[DateEntered] [datetime] NULL,
[DateExecuted] [datetime] NULL,
[ProcessID] [int] NULL,
[OrderedByUserGuid] [uniqueidentifier] NULL,
[RequestedByUserGuid] [uniqueidentifier] NULL,
[SLAEndDate] [datetime] NULL,
CONSTRAINT [PK_PackageInstance] PRIMARY KEY CLUSTERED
(
[PackageInstanceID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[PackageInstanceContextDef](
[PackageInstanceContextDefID] [int] IDENTITY(1,1) NOT NULL,
[ts] [timestamp] NOT NULL,
[PackageInstanceID] [int] NOT NULL,
[ContextObjectDefID] [int] NOT NULL,
[EnteredFieldValue] [varchar](max) NULL,
[SelectedListValueID] [int] NULL,
[AssetIdsString] [nvarchar](max) NULL,
[SelectedListValueIdsString] [nvarchar](max) NULL,
[ContextObjectFieldName] [nvarchar](30) NOT NULL,
CONSTRAINT [PK_PackageInstanceContextDef] PRIMARY KEY CLUSTERED
(
[PackageInstanceContextDefID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
Remove the * in SELECT *
It will always scan because you ask for all columns. And do you have clustered indexes?
The answer turned out to be what #MartinSmith suggested. Because the PackageDefinition table contained about 8 NVARCHAR(MAX) columns, when the resulting join was created and that was over 100k rows, this was causing the varchar(max) values to be re-read over and over and they exist in out of row pages. Hence the large number of logical reads.
Thanks all for your support, just have to figure out to make the entity framework produce the query that I want.
What happens if you add the following index...
CREATE NONCLUSTERED INDEX ix ON PackageDefinition(PackageDefinitionID)
...and try the following to reduce the width of the data going into the sort?
SELECT packInst.*,
packageDef2.*,
contextDef.*
FROM [dbo].[PackageInstance] AS packInst
INNER MERGE JOIN [dbo].[PackageDefinition] AS packageDef
ON packInst.[PackageDefinitionID] = packageDef.[PackageDefinitionID]
LEFT OUTER MERGE JOIN [dbo].[PackageInstanceContextDef] AS contextDef
ON packInst.[PackageInstanceID] = contextDef.[PackageInstanceID]
INNER MERGE JOIN [dbo].[PackageDefinition] AS packageDef2
ON packageDef.[PackageDefinitionID] = packageDef2.[PackageDefinitionID]
OF course * should not be used as even if you need all columns you definitely won't need the same columns twice as the result of the JOIN but this is just to maintain the semantics of your original query.