Multiple Table Spools (Eager spools) during index inserts - sql-server

Question

I am performing a simple insert of a few hundred rows, e.g.:
INSERT INTO Foo
SELECT * FROM Bar
The table has a handful of secondary indexes. With these indexes disabled the query runs nearly instantly. With the secondary indexes enabled, the query takes seconds to run, with a relatively high subtree cost.
The issue is that for every secondary index, the database performs:
Physical Operation: Table spool
Logical Operation: Lazy spool
where it caches:
all columns in the destiation table (when it only needs the values it needs)
multiple times values (rather than just once)
While it may be interesting to know why SQL Server (2008 R2 SP2) thinks it needs to do this, what i really need is a way to make inserting 100 rows in a live server not take six seconds.
The really, really, horrible part is that every for every table spool, SQL Server caches the value of every column, every time:
Which is just burning logical IO.
Without these problematic index updates, the complete import of 60,000 rows happens in a second or two
With these indexes, the complete import takes literally dozens of minute
Steps to reproduce
Of course, my real AuditLog table contains 4M rows. But we can reproduce the exact same operators, with a high subtree cost, using an empty AuditLog table:
CREATE TABLE [dbo].[AuditLog](
[AuditLogID] [int] IDENTITY(216,1) NOT NULL,
[ChangeDate] [datetime] NOT NULL CONSTRAINT [DF_AuditLog_ChangeDate] DEFAULT (getdate()),
[RowGUID] [uniqueidentifier] NOT NULL,
[ChangeType] [varchar](50) NOT NULL,
[TableName] [varchar](128) NOT NULL,
[FieldName] [varchar](128) NOT NULL,
[OldValue] [varchar](max) NULL,
[NewValue] [varchar](max) NULL,
[SystemUser] [varchar](128) NULL CONSTRAINT [DF_AuditLog_SystemUser] DEFAULT (suser_sname()),
[Username] [varchar](128) NOT NULL CONSTRAINT [DF_AuditLog_Username] DEFAULT (user_name()),
[Hostname] [varchar](50) NOT NULL CONSTRAINT [DF_AuditLog_Hostname] DEFAULT (host_name()),
[AppName] [varchar](128) NULL CONSTRAINT [DF_AuditLog_AppName] DEFAULT (app_name()),
[UserGUID] [uniqueidentifier] NULL,
[TagGUID] [uniqueidentifier] NULL,
[Tag] [varchar](max) NULL,
[timestamp] [timestamp] NOT NULL,
CONSTRAINT [PK_AuditLog] PRIMARY KEY CLUSTERED ([AuditLogID] ASC)
)
And we have the painful indexes:
SET ANSI_PADDING OFF
GO
/****** Object: Index [IX_AuditLog_ChangeDate] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_ChangeDate] ON [dbo].[AuditLog]
(
[ChangeDate] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
SET ANSI_PADDING ON
GO
/****** Object: Index [IX_AuditLog_FieldName] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_FieldName] ON [dbo].[AuditLog]
(
[FieldName] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
SET ANSI_PADDING ON
GO
/****** Object: Index [IX_AuditLog_LastRowActionByTable] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_LastRowActionByTable] ON [dbo].[AuditLog]
(
[TableName] ASC,
[ChangeType] ASC,
[RowGUID] ASC,
[UserGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
/****** Object: Index [IX_AuditLog_RowGUID] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_RowGUID] ON [dbo].[AuditLog]
(
[RowGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
SET ANSI_PADDING ON
GO
/****** Object: Index [IX_AuditLog_RowInsertedByUserGUID] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_RowInsertedByUserGUID] ON [dbo].[AuditLog]
(
[ChangeType] ASC,
[RowGUID] ASC,
[UserGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
/****** Object: Index [IX_AuditLog_RowLastModifiedByUserGUID] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_RowLastModifiedByUserGUID] ON [dbo].[AuditLog]
(
[RowGUID] ASC,
[ChangeDate] ASC,
[UserGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
SET ANSI_PADDING ON
GO
/****** Object: Index [IX_AuditLog_TableName] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_TableName] ON [dbo].[AuditLog]
(
[TableName] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
/****** Object: Index [IX_AuditLog_TagGUID] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_TagGUID] ON [dbo].[AuditLog]
(
[TagGUID] ASC,
[RowGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
/****** Object: Index [IX_AuditLog_UserGUID] Script Date: 11/17/2016 2:58:43 PM ******/
CREATE NONCLUSTERED INDEX [IX_AuditLog_UserGUID] ON [dbo].[AuditLog]
(
[ChangeDate] ASC,
[UserGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
And then we create our insert:
INSERT INTO AuditLog(
RowGUID,
ChangeType,
UserGUID,
TableName,
FieldName,
TagGUID,
Tag)
SELECT
'E5E31EDD-7D39-47FD-BCFF-4B7044AC433D',
'INSERTED',
'4A2FDACD-0209-403B-ADBC-1B8A68E90350', --UserGUID
'Customers', --TableName
'', --FieldName
'7A74267D-64F9-44D7-A1D7-1490A66136BF', --TagGUID
'Contoso'
FROM (
--A dummy derived table that lets us select the above row 100 times
SELECT TOP 400 (a.Number * 256) + b.Number AS Number
FROM (
SELECT number FROM master..spt_values WHERE type = 'P' AND number <= 255) a (Number),
(SELECT number FROM master..spt_values WHERE type = 'P' AND number <= 255) b (Number)
) dt
Wait Times You Ask?
| Wait Type | Wait Time (s) | Wait Count |
|----------------|---------------|------------|
| IO_COMPLETION | 4.55 s | 211 |
| WRITELOG | 0.79 s | 37 |
| PAGEIOLATCH_UP | 0.36 s | 1 |
| PAGELATCH_UP | 0.09 s | 2 |
| PAGEIOLATCH_EX | 0.07 s | 4 |
4.55s of a 6s execution in IO_COMPLETION:
Occurs while waiting for I/O operations to complete. This wait type generally represents non-data page I/Os. Data page I/O completion waits appear as PAGEIOLATCH_* waits.
Non-redundant indexes you say?
| Index Name | Columns | Index Entry Size |
|---------------------------------------|------------------------------------------|--------------------------|
| IX_AuditLog_ChangeDate | ChangeDate | 12 bytes per entry |
| IX_AuditLog_UserGUID | ChangeDate, UserGUID | 28 bytes per entry |
| IX_AuditLog_FieldName | FieldName | 4 bytes per entry (avg) |
| IX_AuditLog_TableName | TableName | 13 bytes per entry (avg) |
| IX_AuditLog_LastRowActionByTable | TableName, ChangeType, RowGUID, UserGUID | 52 bytes per entry (avg) |
| IX_AuditLog_RowGUID | RowGUID | 20 bytes per entry |
| IX_AuditLog_RowLastModifiedByUserGUID | RowGUID, ChangeDate, UserGUID | 44 bytes per entry |
| IX_AuditLog_RowInsertedByUserGUID | ChangeType, RowGUID, UserGUID | 43 bytes per entry (avg) |
| IX_AuditLog_TagGUID | TagGUID, RowGUID | 36 bytes per entry |
No Sort Warnings
SQL Server Profiler results for the batch
Duration: 7,401 ms
Reads: 233,597
Writes: 17,077
CPU: 1,141 ms
No sort warnings. Nor is there any Attention, Bitmap Warning, Execution Warning, Hash Warning, Missing Column Statistics, Missing Join Predicate, Sort Warning, User Error Message.
Indexes were all rebuilt. All statistics were updated.

Answer 1

You have problem with overlaping and redundant indexes.
This qry will help: T-SQL for finding Redundant Indexes