Related
I have the following query that will be used to fetch data from legacy tables. It's no surprise but the amount of data is huge and thus it takes a long time. The first select takes 40 minutes to run using an empty dbo.commodities_copy table as a starting point and yields around 26,000 rows. Keep in mind that there are separate databases: STAGING and PRESTAGING and that some joins are made using non-PK fields, which is most definately making an impact in its performance. This is something that I cannot fix, due to the way data was organized from the start. Also the transaction table has around 1 million rows, which also impacts heavily on performance. The entire script takes a total of 3.5 hours to execute when using an EMPTY dbo.commodities_copy table. I have not tested on insertion to a table with data.
The goal of the query is to get commodity information from the transaction table (if you guessed this was supposed to be noSQL data, you guessed right) and if the commodity code exists in the commodity table, do not insert a commodity in it.
The group bys are absolutely needed to get around duplicates, since a transactions may share the same commodity. The commodity code should be unique in the commodities table, but currently it is not - though if it helps, it's possible we could alter it.
What can I do to speed it up?
INSERT INTO STAGING.dbo.commodities_copy
(commodity_code,
short_description_sched_b,
short_description_sched_tsusa,
long_description_sched_b,
long_description_sched_tsusa,
measurement_unit_1_sched_b,
measurement_unit_1_sched_tsusa,
measurement_unit_2_sched_b,
measurement_unit_2_sched_tsusa,
end_use_sched_b,
end_use_sched_tsusa,
year,
created_by,
created_on,
taxable_sched_b,
taxable_sched_tsusa,
non_taxable_sched_b,
non_taxable_sched_tsusa,
fk_sic_sched_b,
fk_sic_sched_tsusa,
chapter,
header,
sub_header,
needs_validation)
SELECT
--Distinct
Commodity_Code,
iif(miob2.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), miob2.DESC_COMM) as short_commmodity_description_b,
iif(mio2tsusa.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), mio2tsusa.DESC_COMM) as short_commmodity_description_tsusa,
socrata.Commodity_description as long_commodity_description_b,
socrata.Commodity_description as long_commodity_description_tsusa,
iif(miob2.UNIDAD is null, socrata.unit_1, miob2.UNIDAD) as unit_1_b,
iif(mio2tsusa.UNIDAD is null, socrata.unit_1, mio2tsusa.UNIDAD) as unit_1_tsusa,
MAX(socrata.unit_2) as unit_2_b,
MAX(socrata.unit_2) as unit_2_tsusa,
socrata.end_use_e as end_use_b,
socrata.end_use_i as end_use_tsusa,
MAX(socrata.[year]),
'system' as created_by,
getdate() as created_on,
miob.TRIBUTA as taxable_b,
miotsusa.TRIBUTA as taxable_tsusa,
miob.NTRIBUTA as non_taxable_b,
miotsusa.NTRIBUTA as non_taxable_tsusa,
sicb.id as sic_id_b,
sictsusa.id as sic_id_tsusa,
SUBSTRING(Commodity_Code, 1, 2) as chapter,
SUBSTRING(Commodity_Code, 1, 4) as header,
SUBSTRING(Commodity_Code, 1, 6) as sub_header,
0 as needs_validation
FROM PRE_STAGING.dbo.TRANSACTIONS_FROM_SOCRATA socrata
Left join PRE_STAGING.DBO.MIOB_TBL miob ON miob.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MSCHB_TBL miob2 ON miob2.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MIOTSUSA_TBL miotsusa ON miotsusa.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MTSUSA_TBL mio2tsusa ON mio2tsusa.COMM=socrata.Commodity_Code
Left join STAGING.dbo.sics_altered sicb ON sicb.sic_code = miob.SIC
Left join STAGING.dbo.sics_altered sictsusa ON sictsusa.sic_code = miotsusa.SIC
WHERE NOT EXISTS
(Select Distinct commodity_code from STAGING.dbo.commodities_copy)
group by
Commodity_Code,
iif(miob2.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), miob2.DESC_COMM),
iif(mio2tsusa.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), mio2tsusa.DESC_COMM),
socrata.Commodity_description,
socrata.Commodity_description,
iif(miob2.UNIDAD is null, socrata.unit_1, miob2.UNIDAD),
iif(mio2tsusa.UNIDAD is null, socrata.unit_1, mio2tsusa.UNIDAD),
socrata.end_use_e,
socrata.end_use_i,
miob.TRIBUTA,
miotsusa.TRIBUTA,
miob.NTRIBUTA,
miotsusa.NTRIBUTA,
sicb.id,
sictsusa.id,
SUBSTRING(Commodity_Code, 1, 2),
SUBSTRING(Commodity_Code, 1, 4),
SUBSTRING(Commodity_Code, 1, 6)
The tables used are the following:
STAGING.dbo.commodities_copy:
CREATE TABLE [dbo].[commodities_copy](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[chapter] [varchar](5) NULL,
[header] [varchar](5) NULL,
[sub_header] [varchar](10) NULL,
[commodity_code] [varchar](20) NULL,
[short_description_sched_b] [varchar](100) NULL,
[long_description_sched_b] [varchar](200) NULL,
[measurement_unit_1_sched_b] [varchar](5) NULL,
[measurement_unit_2_sched_b] [varchar](5) NULL,
[end_use_sched_b] [int] NULL,
[sitc_sched_b] [varchar](20) NULL,
[usda_sched_b] [int] NULL,
[hitech_sched_b] [int] NULL,
[naics_fk_id_sched_b] [bigint] NULL,
[short_description_sched_tsusa] [varchar](100) NULL,
[long_description_sched_tsusa] [varchar](200) NULL,
[measurement_unit_1_sched_tsusa] [varchar](5) NULL,
[measurement_unit_2_sched_tsusa] [varchar](5) NULL,
[end_use_sched_tsusa] [int] NULL,
[sitc_sched_tsusa] [varchar](20) NULL,
[usda_sched_tsusa] [int] NULL,
[hitech_sched_tsusa] [int] NULL,
[naics_fk_id_sched_tsusa] [bigint] NULL,
[year] [int] NOT NULL,
[created_on] [datetime] NOT NULL,
[created_by] [varchar](50) NULL,
[updated_on] [datetime] NULL,
[updated_by] [varchar](50) NULL,
[needs_validation] [bit] NOT NULL,
[taxable_sched_b] [nchar](3) NULL,
[non_taxable_sched_b] [nchar](3) NULL,
[taxable_sched_tsusa] [nchar](3) NULL,
[non_taxable_sched_tsusa] [nchar](3) NULL,
[fk_sic_sched_b] [bigint] NULL,
[fk_sic_sched_tsusa] [bigint] NULL
) ON [PRIMARY]
STAGING.dbo.sics_altered:
CREATE TABLE [dbo].[sics_altered](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[sic_code] [varchar](4) NULL,
[sic_description] [varchar](max) NULL,
[created_on] [datetime] NOT NULL,
[created_by] [varchar](50) NOT NULL,
PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
The rest are in PRESTAGING:
PRESTAGING.dbo.TRANSACTIONS_FROM_SOCRATA:
This is the table with 1.3 million rows
CREATE TABLE [dbo].[TRANSACTIONS_FROM_SOCRATA](
[Trade] [varchar](255) NULL,
[Year] [varchar](255) NULL,
[Month] [varchar](50) NULL,
[Commodity_Code] [varchar](50) NULL,
[Commodity_Short_Name] [varchar](255) NULL,
[Commodity_description] [varchar](255) NULL,
[cty_code] [varchar](50) NULL,
[Country] [varchar](50) NULL,
[Subcountry_code] [varchar](50) NULL,
[district] [varchar](50) NULL,
[dist_name] [varchar](255) NULL,
[data] [varchar](50) NULL,
[sitc] [varchar](50) NULL,
[SITC_Short_Desc] [varchar](255) NULL,
[SITC_Long_Desc] [varchar](255) NULL,
[naics] [varchar](50) NULL,
[NAICS_description] [varchar](255) NULL,
[end_use_i] [varchar](50) NULL,
[end_use_e] [varchar](50) NULL,
[hts_desc] [varchar](255) NULL,
[unit_1] [varchar](50) NULL,
[qty_1] [varchar](50) NULL,
[unit_2] [varchar](50) NULL,
[qty_2] [varchar](50) NULL,
[ves_val_mo] [varchar](50) NULL,
[ves_wgt_mo] [varchar](50) NULL,
[cards_mo] [varchar](50) NULL,
[air_val_mo] [varchar](50) NULL,
[air_wgt_mo] [varchar](50) NULL,
[dut_val_mo] [varchar](50) NULL,
[cal_dut_mo] [varchar](50) NULL,
[con_cha_mo] [varchar](50) NULL,
[con_cif_mo] [varchar](50) NULL,
[gen_val_mo] [varchar](50) NULL,
[gen_cha_mo] [varchar](50) NULL,
[gen_cif_mo] [varchar](50) NULL,
[air_cha_mo] [varchar](50) NULL,
[ves_cha_mo] [varchar](50) NULL,
[cnt_cha_mo] [varchar](50) NULL,
[rev_data] [varchar](50) NULL
) ON [PRIMARY]
PRESTAGING.dbo.MIOB_TBL:
CREATE TABLE [dbo].[MIOB_TBL](
[id] [int] IDENTITY(1,1) NOT NULL,
[COMM] [nchar](10) NOT NULL,
[INSUMO] [nchar](3) NULL,
[PBTO] [nchar](4) NULL,
[SIC] [nchar](4) NULL,
[NAICS] [nchar](6) NULL,
[TRIBUTA] [nchar](3) NULL,
[NTRIBUTA] [nchar](3) NULL,
[LAST_UPDATE] [date] NULL,
[LAST_UPDATED_BY] [nchar](20) NULL,
[CREATION_DATE] [date] NULL,
[CREATED_BY] [nchar](15) NULL,
[migrated_on] [datetime] NOT NULL,
PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
PRESTAGING.dbo.MIOTSUSA_TBL:
CREATE TABLE [dbo].[MIOTSUSA_TBL](
[COMM] [nchar](10) NOT NULL,
[INSUMO] [nchar](3) NULL,
[PBTO] [nchar](4) NULL,
[SIC] [nchar](4) NULL,
[NAICS] [nchar](6) NULL,
[TRIBUTA] [nchar](3) NULL,
[NTRIBUTA] [nchar](3) NULL,
[id] [int] IDENTITY(1,1) NOT NULL,
[migrated_on] [datetime] NOT NULL
) ON [PRIMARY]
PRESTAGING.dbo.MSCHB_TBL:
CREATE TABLE [dbo].[MSCHB_TBL](
[id] [int] IDENTITY(1,1) NOT NULL,
[COMM] [nchar](10) NOT NULL,
[DESC_COMM] [nchar](50) NULL,
[UNIDAD] [nchar](3) NULL,
[LAST_UPDATE] [date] NULL,
[LAST_UPDATED_BY] [nchar](20) NULL,
[CREATION_DATE] [date] NULL,
[CREATED_BY] [nchar](15) NULL,
[migrated_on] [datetime] NOT NULL,
PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
PRESTAGING.dbo.MTSUSA_TBL
CREATE TABLE [dbo].[MTSUSA_TBL](
[COMM] [nchar](10) NOT NULL,
[DESC_COMM] [nchar](50) NULL,
[UNIDAD] [nchar](3) NULL,
[id] [int] IDENTITY(1,1) NOT NULL,
[migrated_on] [datetime] NOT NULL
) ON [PRIMARY]
Let me know if there's anything else I need to provide.
With all those left outer joins, the query optimizer has to start with TRANSACTION_FROM_SOCRATA, so I would start with that. The only filtering is the NOT IN clause--would that cut down the 1MM rows to something more reasonable? If not, you're pretty much doomed to running at least one table scan (and possibly several) on the entire table.
If filtering on Commodity_Code would significantly cut things down, that can only be done if the column is indexed, so that SQL can find and read only those rows. It can only do that if there is an index on column--otherwise you're back to a table scan. Similarly, having an index on commodity_code in table commodities_copy` would help as well, if that table is large.
As discussed in the comments, a NOT EXISTS check would be most efficient, written as a correlated subquery:
WHERE NOT EXISTS (select commodity_code
from STAGING.dbo.commodities_copy
where commodity_code = socrtata.Commodity_Code)
(I'd want to do a lot of testing on this, checking and double-checking everything. Improving performance is tricky, doubly so when done through SO.)
Try this,
create table #socrata(Commodity_Code varchar(100),unit_2_b varchar(50),unit_2_tsusa varchar(50),[year] varchar(50))
insert into #socrata
SELECT
Commodity_Code,
MAX(socrata.unit_2) as unit_2_b,
MAX(socrata.unit_2) as unit_2_tsusa,
MAX(socrata.[year]),
FROM PRE_STAGING.dbo.TRANSACTIONS_FROM_SOCRATA socrata
group by Commodity_Code
SELECT
--Distinct
Commodity_Code,
iif(miob2.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), miob2.DESC_COMM) as short_commmodity_description_b,
iif(mio2tsusa.DESC_COMM is null, UPPER(socrata.Commodity_Short_Name), mio2tsusa.DESC_COMM) as short_commmodity_description_tsusa,
socrata.Commodity_description as long_commodity_description_b,
socrata.Commodity_description as long_commodity_description_tsusa,
iif(miob2.UNIDAD is null, socrata.unit_1, miob2.UNIDAD) as unit_1_b,
iif(mio2tsusa.UNIDAD is null, socrata.unit_1, mio2tsusa.UNIDAD) as unit_1_tsusa,
unit_2_b,
unit_2_tsusa,
socrata.end_use_e as end_use_b,
socrata.end_use_i as end_use_tsusa,
[year],
'system' as created_by,
getdate() as created_on,
miob.TRIBUTA as taxable_b,
miotsusa.TRIBUTA as taxable_tsusa,
miob.NTRIBUTA as non_taxable_b,
miotsusa.NTRIBUTA as non_taxable_tsusa,
sicb.id as sic_id_b,
sictsusa.id as sic_id_tsusa,
SUBSTRING(Commodity_Code, 1, 2) as chapter,
SUBSTRING(Commodity_Code, 1, 4) as header,
SUBSTRING(Commodity_Code, 1, 6) as sub_header,
0 as needs_validation
FROM #socrata socrata
Left join PRE_STAGING.DBO.MIOB_TBL miob ON miob.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MSCHB_TBL miob2 ON miob2.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MIOTSUSA_TBL miotsusa ON miotsusa.COMM=socrata.Commodity_Code
Left join PRE_STAGING.dbo.MTSUSA_TBL mio2tsusa ON mio2tsusa.COMM=socrata.Commodity_Code
Left join STAGING.dbo.sics_altered sicb ON sicb.sic_code = miob.SIC
Left join STAGING.dbo.sics_altered sictsusa ON sictsusa.sic_code = miotsusa.SIC
WHERE NOT EXISTS
(Select commodity_code from STAGING.dbo.commodities_copy where commodity_code = socrtata.Commodity_Code)
if Read uncommitted data is not a concern then you can use with (nolock)
Also your exists clause was wrong and no need of distinct.check rest of the changes.
We are trying to insert 20000 or more records using SqlBulkCopy.WriteToServer method, but it is taking 140 sec to insert into the table.
Is there any way we can improve the time using SqlBulkCopy as this is widely used for bulk insertion?
Calling code:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(dConn))
{
bulkCopy.ColumnMappings.Add("SamplingFileId", "SamplingFileId");
/*... 74 more columns ...*/
bulkCopy.BulkCopyTimeout = 500; //Assign Destination Table Name
bulkCopy.DestinationTableName = "cvr_ReviewSamplingLoans";
bulkCopy.BatchSize = 5000;
bulkCopy.WriteToServer(dtLoansToUpload);
}
Sql Table Definition:
CREATE TABLE [dbo].[ReviewSamplingLoans](
[Id] [int] IDENTITY(1000,1) NOT NULL,
[SamplingFileId] [int] NOT NULL,
[ReviewId] [int] NOT NULL,
[LoanType] [nvarchar](10) NOT NULL,
[LoanNumber] [nvarchar](50) NOT NULL,
[BorrowerName] [nvarchar](150) NOT NULL,
[OriginalUPB] [money] NOT NULL,
[CurrentUPB] [money] NOT NULL,
[OriginalMonthlyPayment] [money] NULL,
[CurrentMonthlyP&I] [money] NULL,
[PaymentFrequency] [nvarchar](2) NULL,
[OriginalNoteRate] [float] NULL,
[CurrentInterestRate] [float] NULL,
[OriginationDate] [date] NULL,
[MaturityDate] [date] NULL,
[FirstPaymentDate] [date] NULL,
[InterestPaidThruDate] [date] NULL,
[NextPaymentDate] [date] NULL,
[DelinquencyStatus] [nvarchar](10) NULL,
[LatePaymentCount] [int] NULL,
[StatedTerm] [int] NULL,
[OriginalAmortizationTerm] [int] NULL,
[NoteType] [nvarchar](10) NULL,
[RateType] [nvarchar](10) NULL,
[LoanProductType] [nvarchar](10) NULL,
[BalloonDueDate] [date] NULL,
[BalloonPaymentAmount] [money] NULL,
[PropertyCity] [nvarchar](50) NULL,
[PropertyState] [nchar](2) NULL,
[PropertyZipcode] [nvarchar](10) NULL,
[PropertyType] [nvarchar](10) NULL,
[PropertyTypeDescription] [nvarchar](50) NULL,
[LienPosition] [tinyint] NULL,
[OriginalLTV] [float] NULL,
[CurrentLTV] [float] NULL,
[SeniorLienBalance] [money] NULL,
[OriginalCommitmentAmount] [money] NULL,
[CurrentCommitmentAmount] [money] NULL,
[OriginalCombinedLTV] [float] NULL,
[CurrentCombinedLTV] [float] NULL,
[OriginalPropertyValue] [money] NULL,
[CurrentPropertyValue] [money] NULL,
[AppraisalDate] [date] NULL,
[AppraisalType] [nvarchar](10) NULL,
[PMI] [nchar](1) NULL,
[LoanPurpose] [nchar](1) NULL,
[OccupancyType] [nchar](1) NULL,
[DocumentationType] [nvarchar](10) NULL,
[OriginalFICO] [smallint] NULL,
[CurrentFICO] [smallint] NULL,
[LastFICOUpdatedDate] [date] NULL,
[DebtToIncomeRatio] [float] NULL,
[InterestOnlyPeriod] [smallint] NULL,
[Modication] [nchar](1) NULL,
[Foreclosure] [nchar](1) NULL,
[ArmProductType] [nvarchar](10) NULL,
[ArmMargin] [float] NULL,
[ArmIndexType] [nvarchar](10) NULL,
[FirstRateAdjustmentDate] [date] NULL,
[NextRateAdjustmentDate] [date] NULL,
[LifeFloor] [float] NULL,
[LifeCeiling] [float] NULL,
[InitialRateCap] [float] NULL,
[PeriodicRateCap] [float] NULL,
[InternalRiskRating] [nvarchar](10) NULL,
[CurrentDebtServiceRatio] [float] NULL,
[CurrentNetOperIncome] [float] NULL,
[OriginalDebtServiceRatio] [float] NULL,
[OriginalNetOperIncome] [float] NULL,
[CurrentOccupancy] [float] NULL,
[PropertyTotalSquareFootage] [int] NULL,
[UnitCount] [smallint] NULL,
[PrepaymentPenalty] [nchar](1) NULL,
[PrepaymentPenaltyType] [int] NULL,
[PrepaymentPenaltyTerm] [smallint] NULL,
CONSTRAINT [PK_ReviewSamplingLoans] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[ReviewSamplingLoans] WITH NOCHECK ADD CONSTRAINT [FK_ReviewSamplingLoans_ReviewCollateralTypes] FOREIGN KEY([LoanType])
REFERENCES [dbo].[ReviewSamplingLoans] ([Code])
GO
ALTER TABLE [dbo].[ReviewSamplingLoans] CHECK CONSTRAINT [FK_ReviewSamplingLoans_ReviewCollateralTypes]
GO
ALTER TABLE [dbo].[ReviewSamplingLoans] WITH NOCHECK ADD CONSTRAINT [FK_ReviewSamplingLoans_ReviewSamplingFiles] FOREIGN KEY([ReviewId])
REFERENCES [dbo].[Reviews] ([Id])
GO
ALTER TABLE [dbo].[ReviewSamplingLoans] CHECK CONSTRAINT [FK_ReviewSamplingLoans_ReviewSamplingFiles]
GO
Regarding Size, excel file contains 20000 rows which are added to datatable upon validation and using sqlbulkcopy inserting into table.
SqlBulkCopy can be performed faster when you enable the Tablelock on the options.
Have a look here SqlBulkCopyOptions Enumeration
But keep in mind that once the operation has started, no one will be able to use the table until it is finished. Microsoft says that this tenchnique improves the speed when inserting large amounts of data.
select
a.COUNTY_FIPS
,COUNT(e.PROPERTY_ID) as house_count
,AVG(cast(e.AVM_FINAL_VALUE as bigint)) as avg_avm
,max(cast(e.AVM_FINAL_VALUE as bigint)) as max_avm
,min(cast(e.AVM_FINAL_VALUE as bigint)) as min_avm
from
RAW_Equity e
left join
(SELECT
SA_PROPERTY_ID, MM_FIPS_STATE_CODE, MM_FIPS_MUNI_CODE,
CASE
WHEN MM_FIPS_STATE_CODE < 10
THEN '0' + CAST(MM_FIPS_STATE_CODE as VARCHAR)
ELSE CAST(MM_FIPS_STATE_CODE as VARCHAR)
END
+ CASE
WHEN MM_FIPS_MUNI_CODE < 10
THEN '00' + CAST(MM_FIPS_MUNI_CODE as VARchar)
WHEN MM_FIPS_MUNI_CODE < 100
THEN '0' + CAST(MM_FIPS_MUNI_CODE as VARchar)
ELSE CAST(MM_FIPS_MUNI_CODE as VARchar)
END AS COUNTY_FIPS
FROM
RAW_Address) a ON a.SA_PROPERTY_ID = e.PROPERTY_ID
where
AVM_CONFIDENCE_SCORE >= 70
group by
a.COUNTY_FIPS
Is there any way I can improve the performance of this query? Schema for both the tables are shown below. I am was thinking about creating non clustered index on AVM_CONFIDENCE_SCORE but I think it will only going to increase the query time. Any help will be greatly appreciated.
RAWADDRESS table:
CREATE TABLE [dbo].[RAW_Address]
(
[SA_PROPERTY_ID] [int] NOT NULL,
[SA_SCM_ID] [int] NOT NULL,
[MM_STATE_CODE] [varchar](2) NOT NULL,
[MM_MUNI_NAME] [varchar](24) NOT NULL,
[MM_FIPS_STATE_CODE] [tinyint] NOT NULL,
[MM_FIPS_MUNI_CODE] [smallint] NOT NULL,
[MM_FIPS_COUNTY_NAME] [varchar](35) NOT NULL,
[SA_SITE_HOUSE_NBR] [varchar](20) NULL,
[SA_SITE_FRACTION] [varchar](10) NULL,
[SA_SITE_DIR] [varchar](2) NULL,
[SA_SITE_STREET_NAME] [varchar](40) NULL,
[SA_SITE_SUF] [varchar](4) NULL,
[SA_SITE_POST_DIR] [varchar](2) NULL,
[SA_SITE_UNIT_PRE] [varchar](10) NULL,
[SA_SITE_UNIT_VAL] [varchar](6) NULL,
[SA_SITE_CITY] [varchar](30) NULL,
[SA_SITE_STATE] [varchar](2) NOT NULL,
[SA_SITE_ZIP] [int] NULL,
[SA_SITE_PLUS_4] [smallint] NULL,
[SA_SITE_CRRT] [varchar](4) NULL,
[SA_MAIL_HOUSE_NBR] [varchar](20) NULL,
[SA_MAIL_FRACTION] [varchar](10) NULL,
[SA_MAIL_DIR] [varchar](2) NULL,
[SA_MAIL_STREET_NAME] [varchar](50) NULL,
[SA_MAIL_SUF] [varchar](4) NULL,
[SA_MAIL_POST_DIR] [varchar](2) NULL,
[SA_MAIL_UNIT_PRE] [varchar](10) NULL,
[SA_MAIL_UNIT_VAL] [varchar](6) NULL,
[SA_MAIL_CITY] [varchar](50) NULL,
[SA_MAIL_STATE] [varchar](2) NULL,
[SA_MAIL_ZIP] [int] NULL,
[SA_MAIL_PLUS_4] [smallint] NULL,
[SA_MAIL_CRRT] [varchar](4) NULL,
[SA_SITE_MAIL_SAME] [varchar](1) NULL
) ON [PRIMARY]
RAW Equity table:
CREATE TABLE [dbo].[RAW_Equity]
(
[PROPERTY_ID] [int] NOT NULL,
[SCM_ID] [int] NOT NULL,
[MM_STATE_CODE] [varchar](2) NOT NULL,
[MM_MUNI_NAME] [varchar](24) NOT NULL,
[MM_FIPS_STATE_CODE] [int] NOT NULL,
[MM_FIPS_MUNI_CODE] [int] NOT NULL,
[MM_FIPS_COUNTY_NAME] [varchar](35) NOT NULL,
[AVM_FINAL_VALUE] [int] NULL,
[AVM_LOW_VALUE] [int] NULL,
[AVM_HIGH_VALUE] [int] NULL,
[AVM_CONFIDENCE_SCORE] [int] NULL,
[FINAL_VALUE] [float] NULL,
[FIRST_POSITION_SR_UNIQUE_ID] [int] NULL,
[FIRST_POSITION_LOAN_DATE] [int] NULL,
[FIRST_POSITION_DOC_NBR] [varchar](20) NULL,
[FIRST_POSITION_LOAN_VAL] [int] NULL,
[FIRST_POSITION_LENDER_CODE] [int] NULL,
[FIRST_POSITION_LNDR_LAST_NAME] [varchar](50) NULL,
[FIRST_POSITION_LNDR_FIRST_NAME] [varchar](50) NULL,
[FIRST_POSITION_LENDER_TYPE] [varchar](1) NULL,
[FIRST_POSITION_LOAN_TYPE] [varchar](1) NULL,
[FIRST_POSITION_INTEREST_RATE_TYPE] [varchar](1) NULL,
[FIRST_POSITION_ESTIMATED_INTEREST_RATE] [float] NULL,
[FIRST_POSITION_LNDR_CREDIT_LINE] [varchar](1) NULL,
[FIRST_POSITION_MODELED_MORTGAGE_TYPE] [varchar](1) NULL,
[SECOND_POSITION_SR_UNIQUE_ID] [int] NULL,
[SECOND_POSITION_LOAN_DATE] [int] NULL,
[SECOND_POSITION_DOC_NBR] [varchar](20) NULL,
[SECOND_POSITION_LOAN_VAL] [int] NULL,
[SECOND_POSITION_LENDER_CODE] [int] NULL,
[SECOND_POSITION_LNDR_LAST_NAME] [varchar](50) NULL,
[SECOND_POSITION_LNDR_FIRST_NAME] [varchar](50) NULL,
[SECOND_POSITION_LENDER_TYPE] [varchar](1) NULL,
[SECOND_POSITION_LOAN_TYPE] [varchar](1) NULL,
[SECOND_POSITION_INTEREST_RATE_TYPE] [varchar](1) NULL,
[SECOND_POSITION_ESTIMATED_INTEREST_RATE] [float] NULL,
[SECOND_POSITION_LNDR_CREDIT_LINE] [varchar](1) NULL,
[SECOND_POSITION_MODELED_MORTGAGE_TYPE] [varchar](1) NULL,
[THIRD_POSITION_SR_UNIQUE_ID] [int] NULL,
[THIRD_POSITION_LOAN_DATE] [int] NULL,
[THIRD_POSITION_DOC_NBR] [varchar](20) NULL,
[THIRD_POSITION_LOAN_VAL] [int] NULL,
[THIRD_POSITION_LENDER_CODE] [int] NULL,
[THIRD_POSITION_LNDR_LAST_NAME] [varchar](50) NULL,
[THIRD_POSITION_LNDR_FIRST_NAME] [varchar](50) NULL,
[THIRD_POSITION_LENDER_TYPE] [varchar](1) NULL,
[THIRD_POSITION_LOAN_TYPE] [varchar](1) NULL,
[THIRD_POSITION_INTEREST_RATE_TYPE] [varchar](1) NULL,
[THIRD_POSITION_ESTIMATED_INTEREST_RATE] [float] NULL,
[THIRD_POSITION_LNDR_CREDIT_LINE] [varchar](1) NULL,
[THIRD_POSITION_MODELED_MORTGAGE_TYPE] [varchar](1) NULL,
[TOTAL_OUTSTANDING_LOANS] [bigint] NULL,
[LTV] [int] NULL,
[AVAILABLE_EQUITY] [int] NULL,
[LENDABLE_EQUITY] [int] NULL,
[PROCESS_ID] [int] NOT NULL,
[FILLER] [varchar](4) NULL,
CONSTRAINT [PK_RAW_Equity] PRIMARY KEY CLUSTERED
(
[PROPERTY_ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
You could try this way :-
(willing to know, how much this one, helpful to you)
Declare #result Table
(
RowId Int Identity(1,1) Primary Key
,COUNTY_FIPS Varchar(100)
,MM_FIPS_STATE_CODE Int
,MM_FIPS_MUNI_CODE Int
,house_count Int
,avg_avm Int
,max_avm Int
,min_avm Int
)
Insert Into #result(MM_FIPS_STATE_CODE,MM_FIPS_MUNI_CODE,house_count,avg_avm,max_avm,min_avm)
Select a.MM_FIPS_STATE_CODE
,a.MM_FIPS_MUNI_CODE
,Count(e.PROPERTY_ID) as house_count
,Avg(Cast(e.AVM_FINAL_VALUE as bigint)) as avg_avm
,Max(Cast(e.AVM_FINAL_VALUE as bigint)) as max_avm
,Min(Cast(e.AVM_FINAL_VALUE as bigint)) as min_avm
From RAW_Equity As e With (Nolock)
Left Join RAW_Address As a With (Nolock) On e.PROPERTY_ID = a.SA_PROPERTY_ID
Where e.AVM_CONFIDENCE_SCORE >= 70
Group by a.MM_FIPS_STATE_CODE
,a.MM_FIPS_MUNI_CODE
Update r
Set r.COUNTY_FIPS = REPLICATE('0',2-LEN(RTRIM(r.MM_FIPS_STATE_CODE))) + RTRIM(r.MM_FIPS_STATE_CODE) + REPLICATE('0',3-LEN(RTRIM(r.MM_FIPS_MUNI_CODE))) + RTRIM(r.MM_FIPS_MUNI_CODE)
From #result As r
Select r.COUNTY_FIPS
,r.house_count
,r.avg_avm
,r.max_avm
,r.min_avm
From #result As r
1st try without any index, and after that create clustered index as mentioned and try AGAIN the above same query
CREATE INDEX IX_RAW_Address_SA_PROPERTY_ID ON RAW_Address(SA_PROPERTY_ID)
I would put an index on:
Table: RAW_Equity
Columns: PROPERTY_ID, AVM_CONFIDENCE_SCORE
and
Table: RAW_Address
Columns: SA_PROPERTY_ID
Include: MM_FIPS_STATE_CODE,MM_FIPS_MUNI_CODE
Your query can be simplified, which will likely make it faster:
select
REPLICATE('0',2-LEN(RTRIM(a.MM_FIPS_STATE_CODE)))
+ RTRIM(a.MM_FIPS_STATE_CODE)
+ REPLICATE('0',3-LEN(RTRIM(a.MM_FIPS_MUNI_CODE)))
+ RTRIM(a.MM_FIPS_MUNI_CODE)
AS COUNTY_FIPS
,COUNT(e.PROPERTY_ID) as house_count
,AVG(cast(e.AVM_FINAL_VALUE as bigint)) as avg_avm
,max(cast(e.AVM_FINAL_VALUE as bigint)) as max_avm
,min(cast(e.AVM_FINAL_VALUE as bigint)) as min_avm
from
RAW_Equity e
left join RAW_Address a
ON a.SA_PROPERTY_ID = e.PROPERTY_ID
where
e.AVM_CONFIDENCE_SCORE >= 70
group by
a.MM_FIPS_STATE_CODE, a.MM_FIPS_MUNI_CODE
If RAW_Address doesn't already have a CLUSTERED INDEX with SA_PROPERTY_ID as the first key of the index, then this may help:
CREATE INDEX IX_RAW_Address_SA_PROPERTY_ID ON RAW_Address(SA_PROPERTY_ID)
INCLUDE (MM_FIPS_STATE_CODE, MM_FIPS_MUNI_CODE)
UPDATE MysterySpells SET SpellId = (SELECT Id FROM Spells
WHERE Name = 'unseen servant')
WHERE MysteryId = 1 And ClassLevel = 2
I know the subquery returns 589. Anyone have any clues why I would be getting this error:
The UPDATE statement conflicted with the FOREIGN KEY constraint
"FK_MysterySpells_Spells". The conflict occurred in database "x",
table "dbo.Spells", column 'Id'.
CODE UPDATE
CREATE TABLE [dbo].[Spells](
[Id] [int] IDENTITY(1,1) NOT NULL,
[OldId] [int] NULL,
[Name] [nvarchar](100) NOT NULL,
[School] [nvarchar](50) NULL,
[SubSchool] [nvarchar](50) NULL,
[SchoolID] [int] NOT NULL,
[SubSchoolID] [int] NULL,
[CastingTime] [nvarchar](250) NULL,
[Components] [nvarchar](250) NULL,
[IsCostly] [bit] NOT NULL,
[Cost] [int] NULL,
[RangeDescription] [nvarchar](250) NULL,
[Range] [int] NOT NULL,
[RangeIncrement] [int] NOT NULL,
[RangeGap] [int] NOT NULL,
[Area] [int] NOT NULL,
[AreaIncrement] [int] NOT NULL,
[AreaGap] [int] NOT NULL,
[AreaNote] [nvarchar](250) NULL,
[Description] [nvarchar](max) NULL,
[ShortDescription] [nvarchar](250) NULL,
[Targets] [nvarchar](250) NULL,
[Effect] [nvarchar](250) NULL,
[Duration] [nvarchar](250) NULL,
[SavingThrow] [nvarchar](250) NULL,
[SpellResistence] [nvarchar](250) NULL,
[HasVerbal] [bit] NOT NULL,
[HasSomatic] [bit] NOT NULL,
[HasMaterial] [bit] NOT NULL,
[HasFocus] [bit] NOT NULL,
[HasDivineFocus] [bit] NOT NULL,
[WizardLevel] [int] NULL,
[ClericLevel] [int] NULL,
[AdeptLevel] [int] NULL,
[AlchemistLevel] [int] NULL,
[AntipaladinLevel] [int] NULL,
[BardLevel] [int] NULL,
[BloodragerLevel] [int] NULL,
[DruidLevel] [int] NULL,
[InquisitorLevel] [int] NULL,
[MagusLevel] [int] NULL,
[OracleLevel] [int] NULL,
[PaladinLevel] [int] NULL,
[RangerLevel] [int] NULL,
[ShamanLevel] [int] NULL,
[SorcererLevel] [int] NULL,
[SummonerLevel] [int] NULL,
[WitchLevel] [int] NULL,
[SLALevel] [int] NULL,
[IsDismissible] [bit] NOT NULL,
[IsLanguageDependent] [bit] NOT NULL,
[IsShapeable] [bit] NOT NULL,
[IsAcid] [bit] NOT NULL,
[IsAir] [bit] NOT NULL,
[IsCold] [bit] NOT NULL,
[IsCurse] [bit] NOT NULL,
[IsDarkness] [bit] NOT NULL,
[IsDeath] [bit] NOT NULL,
[IsDisease] [bit] NOT NULL,
[IsEarth] [bit] NOT NULL,
[IsElectricity] [bit] NOT NULL,
[IsEmotion] [bit] NOT NULL,
[IsFear] [bit] NOT NULL,
[IsFire] [bit] NOT NULL,
[IsForce] [bit] NOT NULL,
[IsLight] [bit] NOT NULL,
[IsPain] [bit] NOT NULL,
[IsPoison] [bit] NOT NULL,
[IsShadow] [bit] NOT NULL,
[IsSonic] [bit] NOT NULL,
[IsWater] [bit] NOT NULL,
[IsChaotic] [bit] NOT NULL,
[IsEvil] [bit] NOT NULL,
[IsGood] [bit] NOT NULL,
[IsLawful] [bit] NOT NULL,
[IsMindAffecting] [bit] NOT NULL,
[IsMythic] [bit] NULL,
[MythicDescription] [nvarchar](max) NULL,
[Augmented] [nvarchar](max) NULL,
CONSTRAINT [PK_Spells] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
CREATE TABLE [dbo].[MysterySpells](
[Id] [int] IDENTITY(1,1) NOT NULL,
[MysteryId] [int] NOT NULL,
[ClassLevel] [int] NOT NULL,
[SpellId] [int] NOT NULL,
[SpellLevel] [int] NULL,
[Restrictions] [varchar](100) NULL,
CONSTRAINT [PK_MysterySpells] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[MysterySpells] WITH NOCHECK ADD CONSTRAINT [FK_MysterySpells_Mysteries] FOREIGN KEY([MysteryId])
REFERENCES [dbo].[Mysteries] ([Id])
GO
ALTER TABLE [dbo].[MysterySpells] CHECK CONSTRAINT [FK_MysterySpells_Mysteries]
GO
ALTER TABLE [dbo].[MysterySpells] WITH NOCHECK ADD CONSTRAINT [FK_MysterySpells_Spells] FOREIGN KEY([SpellId])
REFERENCES [dbo].[Spells] ([Id])
GO
ALTER TABLE [dbo].[MysterySpells] CHECK CONSTRAINT [FK_MysterySpells_Spells]
GO
If I were you, here is how I debug
If you expect
SELECT Id FROM Spells WHERE Name = 'unseen servant'
to return 589
I will hard code it to 1 first.
UPDATE MysterySpells SET SpellId = 589
WHERE MysteryId = 1 And ClassLevel = 2
If the error still there, I will check if table Spells contains value where Id = 589 and Name = 'unseen servant'.
I believe you will find the root cause somewhere in the process.
Note: It would be better if you show us the table structure and data inside.
The issue is that it does not know what value to grab. There is no mapping defined between spells and mystery spells in your query. The current update is actually trying to update each row in MysterySpells with 589 records.
If you simply wanted to map all of your mystery spells to a single spell you can do:
UPDATE MysterySpells SET SpellId = (SELECT TOP 1 Id FROM Spells
WHERE Name = 'unseen servant')
WHERE MysteryId = 1 And ClassLevel = 2
If there is a pre-existing mapping between these 2 tables and you are expecting to update MysterySpells based on a relationship with spells table:
UPDATE MysterySpells
SET SpellID = t2.ID
FROM MysterySpells t1
JOIN Spells t2 ON t2.relation = t1.relation
If there isn't any existing mapping and you are trying to do this to create this as the mapping you have few options.
1) If no other relationship exists map by hand
2) If their is a relationship that can be calculated, but is not available through an existing join, you can take advantage of the row and partition functions to relate the two. Then use an update similar to the 2nd code block I posted.
This is maddening! Code in question has been running for over 5 years.
Here's the scoop....
I am doing an INSERT...SELECT into a table with a primary key that is an identity column. I do not specify the key when I insert - SQL Server generates it as expected.
I am doing the insert in a stored procedure that I call in a loop (for loop in SSIS, actually). The stored procedure will insert rows in batches (configurable). It might insert 1000 rows at a time or it might insert 50,000 - doesn't matter. It will work for a random number of calls (inserting thousands of rows) and then it will fail, out of the blue, with a
Violation of primary key / duplicate
error. If I check the identity seed - it is correct. If I kick off the process again it will work fine, for a while.
The values being inserted are coming from 2 tables that I join together, as if that matters.
The bulk of my code is below:
WHILE #pk <= #max_pk
BEGIN
INSERT INTO tbl_claim_line (fk_batch_control_group, fk_claim, fk_provider, service_from_date, service_to_date, allowed, net_paid, COB, flex_1, flex_2, flex_3, flex_4)
SELECT
#fk_batch_control_group
, c.pk_claim
, p.pk_provider
, i.date_of_service_from
, i.date_of_service_to
, i.allowed_amount
, i.net_paid_amount
, i.cob_amount
, i.claimline_flex_1
, i.claimline_flex_2
, i.claimline_flex_3
, i.claimline_flex_4
FROM
tbl_import i
INNER JOIN
tbl_import__claim c ON i.claim_number = c.claim_number
LEFT JOIN
tbl_import__provider p ON ISNULL(i.provider_type,'') = ISNULL(p.provider_type,'')
AND ISNULL(i.provider_specialty,'') = ISNULL(p.provider_specialty,'')
AND ISNULL(i.provider_zip_code,'') = ISNULL(p.provider_zip_code,'')
WHERE
pk_import = #pk
UPDATE tbl_import
SET fk_claim_line = SCOPE_IDENTITY()
WHERE pk_import = #pk
SET #pk += 1
END
--TABLE DEFINITIONS...
CREATE TABLE [dbo].[tbl_claim_line](
[fk_batch_control_group] [int] NOT NULL,
[fk_claim] [int] NOT NULL,
[fk_provider] [int] NULL,
[service_from_date] [date] NULL,
[service_to_date] [date] NULL,
[allowed] [money] NULL,
[net_paid] [money] NULL,
[COB] [money] NULL,
[flex_1] [varchar](200) NULL,
[flex_2] [varchar](200) NULL,
[flex_3] [varchar](200) NULL,
[flex_4] [varchar](200) NULL,
[pk_claim_line] [int] IDENTITY(1,1) NOT NULL,
[insert_date] [datetime] NOT NULL,
CONSTRAINT [PK_tbl_claim_line] PRIMARY KEY NONCLUSTERED
(
[pk_claim_line] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[tbl_claim_line] WITH CHECK
ADD CONSTRAINT [FK_tbl_claim_line_tbl_batch_control_group]
FOREIGN KEY([fk_batch_control_group])
REFERENCES [dbo].[tbl_batch_control_group] ([pk_batch_control_group])
GO
ALTER TABLE [dbo].[tbl_claim_line] CHECK CONSTRAINT [FK_tbl_claim_line_tbl_batch_control_group]
GO
ALTER TABLE [dbo].[tbl_claim_line] WITH CHECK
ADD CONSTRAINT [FK_tbl_claim_line_tbl_claim]
FOREIGN KEY([fk_claim])
REFERENCES [dbo].[tbl_claim] ([pk_claim])
ON DELETE CASCADE
GO
ALTER TABLE [dbo].[tbl_claim_line] CHECK CONSTRAINT [FK_tbl_claim_line_tbl_claim]
GO
ALTER TABLE [dbo].[tbl_claim_line] WITH CHECK
ADD CONSTRAINT [FK_tbl_claim_line_tbl_provider]
FOREIGN KEY([fk_provider])
REFERENCES [dbo].[tbl_provider] ([pk_provider])
GO
ALTER TABLE [dbo].[tbl_claim_line] CHECK CONSTRAINT [FK_tbl_claim_line_tbl_provider]
GO
ALTER TABLE [dbo].[tbl_claim_line] ADD CONSTRAINT [DF_tbl_claim_line__insert_date] DEFAULT (getdate()) FOR [insert_date]
GO
----second table
CREATE TABLE [dbo].[tbl_import](
[fk_claim_line] [int] NULL,
[member_id] [varchar](50) NULL,
[member_card_id] [varchar](50) NULL,
[member_first_name] [varchar](50) NULL,
[member_last_name] [varchar](50) NULL,
[member_dob] [varchar](50) NULL,
[member_gender] [varchar](50) NULL,
[member_subscriber_relationship_code] [varchar](50) NULL,
[member_address_line_1] [varchar](100) NULL,
[member_address_line_2] [varchar](100) NULL,
[member_city] [varchar](50) NULL,
[member_state] [varchar](50) NULL,
[member_zip] [varchar](50) NULL,
[member_phone] [varchar](50) NULL,
[member_email] [varchar](50) NULL,
[subscriber_id] [varchar](50) NULL,
[group_line_of_business] [varchar](50) NULL,
[group_product] [varchar](50) NULL,
[group_employer] [varchar](50) NULL,
[provider_first_name] [varchar](50) NULL,
[provider_last_or_full_name] [varchar](200) NULL,
[provider_type] [varchar](200) NULL,
[provider_specialty] [varchar](400) NULL,
[provider_zip_code] [varchar](50) NULL,
[provider_tax_id] [varchar](50) NULL,
[medical_code_1] [varchar](10) NULL,
[medical_code_1_description] [varchar](500) NULL,
[medical_code_2] [varchar](10) NULL,
[medical_code_2_description] [varchar](500) NULL,
[medical_code_3] [varchar](10) NULL,
[medical_code_3_description] [varchar](500) NULL,
[medical_code_4] [varchar](10) NULL,
[medical_code_4_description] [varchar](500) NULL,
[medical_code_5] [varchar](10) NULL,
[medical_code_5_description] [varchar](500) NULL,
[medical_code_6] [varchar](10) NULL,
[medical_code_6_description] [varchar](500) NULL,
[medical_code_7] [varchar](10) NULL,
[medical_code_7_description] [varchar](500) NULL,
[medical_code_8] [varchar](10) NULL,
[medical_code_8_description] [varchar](500) NULL,
[medical_code_9] [varchar](10) NULL,
[medical_code_9_description] [varchar](500) NULL,
[medical_code_10] [varchar](10) NULL,
[medical_code_10_description] [varchar](500) NULL,
[medical_code_11] [varchar](10) NULL,
[medical_code_11_description] [varchar](500) NULL,
[medical_code_12] [varchar](10) NULL,
[medical_code_12_description] [varchar](500) NULL,
[medical_code_13] [varchar](10) NULL,
[medical_code_13_description] [varchar](500) NULL,
[medical_code_14] [varchar](10) NULL,
[medical_code_14_description] [varchar](500) NULL,
[medical_code_15] [varchar](10) NULL,
[medical_code_15_description] [varchar](500) NULL,
[medical_code_16] [varchar](10) NULL,
[medical_code_16_description] [varchar](500) NULL,
[date_of_service_from] [varchar](50) NULL,
[date_of_service_to] [varchar](50) NULL,
[claim_number] [varchar](50) NULL,
[claim_line_number] [varchar](50) NULL,
[original_claim_number] [varchar](50) NULL,
[allowed_amount] [varchar](50) NULL,
[net_paid_amount] [varchar](50) NULL,
[cob_amount] [varchar](50) NULL,
[date_paid] [varchar](50) NULL,
[member_flex_1] [varchar](200) NULL,
[member_flex_2] [varchar](200) NULL,
[member_flex_3] [varchar](200) NULL,
[member_flex_4] [varchar](200) NULL,
[claim_flex_1] [varchar](200) NULL,
[claim_flex_2] [varchar](200) NULL,
[claim_flex_3] [varchar](200) NULL,
[claim_flex_4] [varchar](200) NULL,
[claimline_flex_1] [varchar](200) NULL,
[claimline_flex_2] [varchar](200) NULL,
[claimline_flex_3] [varchar](200) NULL,
[claimline_flex_4] [varchar](200) NULL,
[pk_import] [int] IDENTITY(1,1) NOT NULL,
CONSTRAINT [PK_tbl_import] PRIMARY KEY NONCLUSTERED
(
[pk_import] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
I ran into this and much like user3170349, it was a seed issue on the column. I'm adding some additional info, however.
First, you can run this to figure out if you have a seed problem:
DBCC CHECKIDENT ('TABLE_NAME_GOES_HERE', NORESEED);
This will give you information which will read something like this:
Checking identity information: current identity value 'XXXX', current column value 'YYYY'.
If YYYY is larger than XXXX, then you have a problem and need to RESEED the table to get things going again. You can do so with the following command:
DBCC CHECKIDENT ('TABLE_NAME_GOES_HERE', RESEED, ZZZZZ);
Where ZZZZ is the reseed value. That value should be at least one higher than YYYY. YMMV, so pick a value that is appropriate for your situation.
"Code in question has been running for over 5 years."
"It might insert 1000 records at a time or it might insert 50,000 "
Is it possible you have finally overflowed the integer type of the primary key?
Did it wrap around and is now starting over? That would cause duplicate primary keys.