Is using a SqlCacheDependency per user a good idea? - sql-server

Question

I'm thinking of caching permissions for every user on our application server. Is it a good idea to use a SqlCacheDependency for every user?
The query would look like this
SELECT PermissionId, PermissionName From Permissions Where UserId = #UserId
That way I know if any of those records change then to purge my cache for that user.

Answer 1

If you read how Query Notifications work you'll see why createing many dependency requests with a single query template is good practice. For a web app, which is implied by the fact that you use SqlCacheDependency and not SqlDependency, what you plan to do should be OK. If you use Linq2Sql you can also try LinqToCache:
var queryUsers = from u in repository.Users
where u.UserId = currentUserId
select u;
var user= queryUsers .AsCached("Users:" + currentUserId.ToString());
For a fat client app it would not be OK. Not because of the query per-se, but because SqlDependency in general is problematic with a large number of clients connected (it blocks a worker thread per app domain connected):
SqlDependency was designed to be used in ASP.NET or middle-tier
services where there is a relatively small number of servers having
dependencies active against the database. It was not designed for use
in client applications, where hundreds or thousands of client
computers would have SqlDependency objects set up for a single
database server.
Updated
Here is the same test as #usr did in his post. Full c# code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.SqlClient;
using DependencyMassTest.Properties;
using System.Threading.Tasks;
using System.Threading;
namespace DependencyMassTest
{
class Program
{
static volatile int goal = 50000;
static volatile int running = 0;
static volatile int notified = 0;
static int workers = 50;
static SqlConnectionStringBuilder scsb;
static AutoResetEvent done = new AutoResetEvent(false);
static void Main(string[] args)
{
scsb = new SqlConnectionStringBuilder(Settings.Default.ConnString);
scsb.AsynchronousProcessing = true;
scsb.Pooling = true;
try
{
SqlDependency.Start(scsb.ConnectionString);
using (var conn = new SqlConnection(scsb.ConnectionString))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand(#"
if object_id('SqlDependencyTest') is not null
drop table SqlDependencyTest
create table SqlDependencyTest (
ID int not null identity,
SomeValue nvarchar(400),
primary key(ID)
)
", conn))
{
cmd.ExecuteNonQuery();
}
}
for (int i = 0; i < workers; ++i)
{
Task.Factory.StartNew(
() =>
{
RunTask();
});
}
done.WaitOne();
Console.WriteLine("All dependencies subscribed. Waiting...");
Console.ReadKey();
}
catch (Exception e)
{
Console.Error.WriteLine(e);
}
finally
{
SqlDependency.Stop(scsb.ConnectionString);
}
}
static void RunTask()
{
Random rand = new Random();
SqlConnection conn = new SqlConnection(scsb.ConnectionString);
conn.Open();
SqlCommand cmd = new SqlCommand(
#"select SomeValue
from dbo.SqlDependencyTest
where ID = #id", conn);
cmd.Parameters.AddWithValue("#id", rand.Next(50000));
SqlDependency dep = new SqlDependency(cmd);
dep.OnChange += new OnChangeEventHandler((ob, qnArgs) =>
{
Console.WriteLine("Notified {3}: Info:{0}, Source:{1}, Type:{2}", qnArgs.Info, qnArgs.Source, qnArgs.Type, Interlocked.Increment(ref notified));
});
cmd.BeginExecuteReader(
(ar) =>
{
try
{
int crt = Interlocked.Increment(ref running);
if (crt % 1000 == 0)
{
Console.WriteLine("{0} running...", crt);
}
using (SqlDataReader rdr = cmd.EndExecuteReader(ar))
{
while (rdr.Read())
{
}
}
}
catch (Exception e)
{
Console.Error.WriteLine(e.Message);
}
finally
{
conn.Close();
int left = Interlocked.Decrement(ref goal);
if (0 == left)
{
done.Set();
}
else if (left > 0)
{
RunTask();
}
}
}, null);
}
}
}
After 50k subscriptions are set up (takes about 5 min), here are the stats io of a single insert:
set statistics time on
insert into Test..SqlDependencyTest (SomeValue) values ('Foo');
SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 0 ms.
SQL Server Execution Times:
CPU time = 16 ms, elapsed time = 16 ms.
Inserting 1000 rows takes about 7 seconds, which includes firing several hundred notifications. CPU utilization is about 11%. All this is on my T420s ThinkPad.
set nocount on;
go
begin transaction
go
insert into Test..SqlDependencyTest (SomeValue) values ('Foo');
go 1000
commit
go

Answer 2

the documentation says:
SqlDependency was designed to be used in ASP.NET or middle-tier
services where there is a relatively small number of servers having
dependencies active against the database. It was not designed for use
in client applications, where hundreds or thousands of client
computers would have SqlDependency objects set up for a single
database server.
It tells us not to open thousands of cache dependencies. That is likely to cause resource problems on the SQL Server.
There are a few alternatives:
Have a dependency per table
Have 100 dependencies per table, one for every percent of rows. This should be an acceptable number for SQL Server yet you only need to invalidate 1% of the cache.
Have a trigger output the ID of all changes rows into a logging table. Create a dependency on that table and read the IDs. This will tell you exactly which rows have changed.
In order to find out if SqlDependency is suitable for mass usage I did a benchmark:
static void SqlDependencyMassTest()
{
var connectionString = "Data Source=(local); Initial Catalog=Test; Integrated Security=true;";
using (var dependencyConnection = new SqlConnection(connectionString))
{
dependencyConnection.EnsureIsOpen();
dependencyConnection.ExecuteNonQuery(#"
if object_id('SqlDependencyTest') is not null
drop table SqlDependencyTest
create table SqlDependencyTest (
ID int not null identity,
SomeValue nvarchar(400),
primary key(ID)
)
--ALTER DATABASE Test SET ENABLE_BROKER with rollback immediate
");
SqlDependency.Start(connectionString);
for (int i = 0; i < 1000 * 1000; i++)
{
using (var sqlCommand = new SqlCommand("select ID from dbo.SqlDependencyTest where ID = #id", dependencyConnection))
{
sqlCommand.AddCommandParameters(new { id = StaticRandom.ThreadLocal.GetInt32() });
CreateSqlDependency(sqlCommand, args =>
{
});
}
if (i % 1000 == 0)
Console.WriteLine(i);
}
}
}
You can see the amount of dependencies created scroll through the console. It gets slow very quickly. I did not do a formal measurement because it was not necessary to prove the point.
Also, the execution plan for a simple insert into the table shows 99% of the cost being associated with maintaining the 50k dependencies.
Conclusion: Does not work at all for production use. After 30min I have 55k dependencies created. Machine at 100% CPU all the time.