I sync data from an api and detect if an insert or update is necessary.
From time to time I receive DbUpdateExceptions and then fallback to single insert/update + savechanges instead of addrange/updaterange + savechanges.
Because single entities are so slow I wanted to only remove the failing entity from changetracking and try to save it all again, but unfortunately mssql returns all entities instead of only the one that is failing in DbUpdateException.Entries.
Intellisense tells me
Gets the entries that were involved in the error. Typically this is a single entry, but in some cases it may be zero or multiple entries.
Interestingly this is true if I try it on a mysql server. There only one entity is returned, but mssql returns all, which makes it impossible for me to exclude only the failing one.
Is there any setting to change mssql behaviour?
Both mysql and mssql are azure hosted resources.
Here an example:
var addList = new List<MyEntity>();
var updateList = new List<MyEntity>();
//load existing data from db
var existingData = Context.Set<MyEntity>()
.AsNoTracking()
.Take(2).ToList();
if (existingData.Count < 2)
return;
//addList
addList.Add(new MyEntity
{
NotNullableProperty = "Value",
RequiredField1 = Guid.Empty,
RequiredField2 = Guid.Empty,
});
addList.Add(new MyEntity
{
NotNullableProperty = "Value",
RequiredField1 = Guid.Empty,
RequiredField2 = Guid.Empty,
});
addList.Add(existingData.ElementAt(0)); //this should fail due to duplicate key
addList.Add(new MyEntity
{
NotNullableProperty = "Value",
RequiredField1 = Guid.Empty,
RequiredField2 = Guid.Empty,
});
//updateList
existingData.ElementAt(1).NotNullableProperty = null; //this should fail due to invalid value
updateList.Add(existingData.ElementAt(1));
//save a new entity that should fail
var newKb = new MyEntity
{
NotNullableProperty = "Value",
RequiredField1 = Guid.Empty,
RequiredField2 = Guid.Empty,
};
Context.Add(newKb);
Context.SaveChanges();
newKb.NotNullableProperty = "01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"; //this should fail due to length
updateList.Add(newKb);
try
{
if (addList.IsNotNullOrEmpty())
context.Set<MyEntity>().AddRange(addList);
if (updateList.IsNotNullOrEmpty())
context.Set<MyEntity>().UpdateRange(updateList);
context.SaveChanges();
}
catch (DbUpdateException updateException)
{
//updateException.Entries contains all entries, that were added/updated although only three should fail
}
Related
I am using Akka.Net (v 1.3.2) and am trying to query the event journal for all events with a specific tag. I only want the events that exist at the time I query the journal. Inside an actor, I have the following code:
var readJournal = PersistenceQuery.Get(Context.System).ReadJournalFor<SqlReadJournal>(SqlReadJournal.Identifier);
var stream = readJournal.CurrentEventsByTag("The Tag Name", Offset.NoOffset());
var materializer = ActorMaterializer.Create(Context.System);
stream.RunForeach(envelope =>
{
// Do some stuff with the EventEnvelope
}, materializer).Wait();
This will successfully query the event journal. However, the problem is it will only return the first 100 events. I need all of them that match the query!
Question: How do I remove the limit/filter that exists when querying the event journal by tag name?
If you need it, here is my akka.persistence configuration:
var config = Akka.Configuration.ConfigurationFactory.ParseString(#"
akka.persistence {
journal {
plugin = ""akka.persistence.journal.sql-server""
sql-server {
class = ""Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer""
connection-string = """ + connectionString + #"""
schema-name = dbo
table-name = __akka_EventJournal
metadata-table-name = __akka_Metadata
auto-initialize = on
}
}
snapshot-store {
plugin = ""akka.persistence.snapshot-store.sql-server""
sql-server {
class = ""Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer""
connection-string = """ + connectionString + #"""
schema-name = dbo
table-name = __akka_SnapshotStore
auto-initialize = on
}
}
}"
);
There are two things to check out:
You can set the maximum number of messages returned in one query by setting up akka.persistence.query.journal.sql.max-buffer-size value (see: reference.conf).
Use readJournal.EventsByTag instead of readJournal.CurrentEventsByTag to get a continuous stream of events. Just keep in mind, that it won't complete by itself, but will live on waiting for new events to arrive. You can stop it explicitly i.e. by using KillSwitch.
I have a table that consists of a column of pre-populated numbers. My API using Nhibernate grabs the first 10 rows where 'Used' flag is set as false.
What would be the best possible way to avoid concurrency issue when multiple session try to grab row from the table?
After selecting the row, I can update the flag column to be True so subsequent calls will not use the same numbers.
With such a general context, it could be done that way:
// RepeatableRead ensures the read rows does not get concurrently updated by another
// session.
using (var tran = session.BeginTransaction(IsolationLevel.RepeatableRead))
{
var entities = session.Query<Entity>()
.Where(e => !e.Used)
.OrderBy(e => e.Id)
.Take(10)
.ToList();
foreach(var entity in entities)
{
e.Used = true;
}
// If your session flush mode is not the default one and does not cause
// commits to flush the session, add a session.Flush(); call before committing.
tran.Commit();
return entities;
}
It is simple. It may fail with a deadlock, in which case you would have to throw away the session, get a new one, and retry.
Using an optimistic update pattern could be an alternate solution, but this requires some code for recovering from failed attempts too.
Using a no explicit lock solution, which will not cause deadlock risks, could do it, but it will require more queries:
const int entitiesToObtain = 10;
// Could initialize here with null instead, but then, will have to check
// for null after the while too.
var obtainedEntities = new List<Entity>();
while (obtainedEntities.Count == 0)
{
List<Entity> candidates;
using (var tran = session.BeginTransaction())
{
candidatesIds = session.Query<Entity>()
.Where(e => !e.Used)
.Select(e => e.Id)
.OrderBy(id => id)
.Take(entitiesToObtain)
.ToArray();
}
if (candidatesIds.Count == 0)
// No available entities.
break;
using (var tran = session.BeginTransaction())
{
var updatedCount = session.CreateQuery(
#"update Entity e set e.Used = true
where e.Used = false
and e.Id in (:ids)")
.SetParameterList("ids", candidatesIds)
.ExecuteUpdate();
if (updatedCount == candidatesIds.Length)
{
// All good, get them.
obtainedEntities = session.Query<Entity>()
.Where(e => candidatesIds.Contains(e.Id))
.ToList();
tran.Commit();
}
else
{
// Some or all of them were no more available, and there
// are no reliable way to know which ones, so just try again.
tran.Rollback();
}
}
}
This uses NHibernate DML-style operations as suggested here. A strongly typed alternative is available in NHibernate v5.0.
Instead of using Breeze server side to save JObject, I'm using a dummy contextprovider to extract the EntityMaps and then performing custom validations on each entity and saving them myself. If the save succeeds, how do I reconstruct the SaveResult object to return back to the client so that BreezeJS client knows about my changes?
Currently I'm returning the following SaveResult:
// Using example here (https://github.com/Breeze/breeze.js.samples/issues/33)
// to extract EntityMaps from JObject.
// The return result is a Dictionary<Type, EntityInfo>.
var entityMaps = SaveBundleToSaveMap.Convert(saveBundle);
// ... Code to save entities to DB
// SaveResult to be returned to the client.
return new SaveResult()
{
Entities = entityMaps.SelectMany(innerEi => innerEi.Value.Select(ie => ie.Entity)).ToList<object>(),
Errors = null,
KeyMappings = new List<KeyMapping>()
};
How do I construct the KeyMapping list for single primary keys? How do I construct the KeyMapping for composite keys?
After some trial and error, I found that the SaveResult.KeyMapping list contain only the old and new keys generated from inserted entities. This makes sense because the client has the key of updates and do not need to worry about deleted entities.
To construct the SaveResult.KeyMapping list, first need to set the EntityInfo.AutoGeneratedKey.TempValue of each insert entity to the key sent from the client side, save the entities to get the new keys, and then create KeyMapping list and return back to the client.
1.Loop the entityMaps and set the TempValue for each EntityInfo to the key/id sent from the client side.
// Extract out from loop to make it more readable.
Action<EntityInfo> processEntityInfo = (ei) =>
{
if (ei.EntityState == EntityState.Added && (ei.AutoGeneratedKey != null && ei.AutoGeneratedKey.AutoGeneratedKeyType == AutoGeneratedKeyType.Identity))
{
var entity = ei.Entity;
var tempValue = ei.AutoGeneratedKey.Property.GetValue(entity);
ei.AutoGeneratedKey.TempValue = tempValue;
}
};
entityMaps.ToList().ForEach(map => map.Value.ForEach(ei => processEntityInfo(ei)));
2.Save the entities by calling the the Context.SaveChanges() on the EntityFramework Context. This will generate keys for all the inserted entities.
3.Loop through the saved entities in the entity maps and construct and return the KeyMapping list.
return entityMaps.SelectMany(entityMap =>
entityMap.Value
.Where(entityInfo => entityInfo.EntityState == EntityState.Added && (entityInfo.AutoGeneratedKey != null && entityInfo.AutoGeneratedKey.AutoGeneratedKeyType == AutoGeneratedKeyType.Identity))
.Select(entityInfo => new KeyMapping
{
EntityTypeName = entityInfo.Entity.GetType().FullName,
RealValue = entityInfo.AutoGeneratedKey.Property.GetValue(entityInfo.Entity),
TempValue = entityInfo.AutoGeneratedKey.TempValue
}));
I am trying to determine a way to audit which records a given user can see by;
Object Type
Record Type
Count of records
Ideally would also be able to see which fields for each object/record type the user can see.
We will need to repeat this often and for different users and in different orgs, so would like to avoid manually determining this.
My first thought was to create an app using the partner WSDL, but would like to ask if there are any easier approaches or perhaps existing solutions.
Thanks all
I think that you can follow the documentation to solve it, using a query similar to this one:
SELECT RecordId
FROM UserRecordAccess
WHERE UserId = [single ID]
AND RecordId = [single ID] //or Record IN [list of IDs]
AND HasReadAccess = true
The following query returns the records for which a queried user has
read access to.
In addition, you should add limit 1 and get from record metadata the object type,record type, and so on.
I ended up using the below (C# using the Partner WSDL) to get an idea of what kinds of objects the user had visibility into.
Just a quick'n'dirty utility for my own use (read - not prod code);
var service = new SforceService();
var result = service.login("UserName", "Password");
service.Url = result.serverUrl;
service.SessionHeaderValue = new SessionHeader { sessionId = result.sessionId };
var queryResult = service.describeGlobal();
int total = queryResult.sobjects.Count();
int batcheSize = 100;
var batches = Math.Ceiling(total / (double)batcheSize);
using (var output = new StreamWriter(#"C:\test\sfdcAccess.txt", false))
{
for (int batch = 0; batch < batches; batch++)
{
var toQuery =
queryResult.sobjects.Skip(batch * batcheSize).Take(batcheSize).Select(x => x.name).ToArray();
var batchResult = service.describeSObjects(toQuery);
foreach (var x in batchResult)
{
if (!x.queryable)
{
Console.WriteLine("{0} is not queryable", x.name);
continue;
}
var test = service.query(string.Format("SELECT Id FROM {0} limit 100", x.name));
if(test == null || test.records == null)
{
Console.WriteLine("{0}:null records", x.name);
continue;
}
foreach (var record in test.records)
{
output.WriteLine("{0}\t{1}",x.name, record.Id);
}
Console.WriteLine("{0}:\t{1} records(0)", x.name, test.size);
}
}
output.Flush();
}
I am currently building a web site and I just implemented SqlCacheDependency using LinqToSQL like so.
public IQueryable<VictoryList> GetVictoryList()
{
string cacheKey = HttpContext.Current.User.Identity.Name + "victoryCacheKey";
IQueryable<VictoryList> cachednews = (IQueryable<VictoryList>)HttpContext.Current.Cache.Get(cacheKey);
if (cachednews == null)
{
var results = from v in _datacontext.ViewVictoryLists
orderby _datacontext.GetNewId()
select new VictoryList
{
MemberID = v.MemberID,
Username = v.Aspnetusername,
Location = v.Location,
DaimokuGoal = v.DaimokuGoal,
PreviewImageID = v.PreviewImageID,
TotalDaimoku = v.TotalDaimoku,
TotalDeterminations = v.TotalDeterminations,
DeterminationID = v.DeterminationID,
DeterminationName = v.DeterminationName
};
SqlCacheDependency dependency =
new SqlCacheDependency(_datacontext.GetCommand(results) as SqlCommand);
HttpContext.Current.Cache.Add(cacheKey, results, dependency, DateTime.MaxValue,
TimeSpan.Zero, CacheItemPriority.Normal, null);
return results.ToList().AsQueryable();
}
return cachednews;
}
It appears to be working as things are noticbly faster especially on some complex queries, however while looking at things in SQLProfiler I still see the query run through, I'm using the CommandBroker mode of SqlCacheDependency. Should I still see the query even though the data is obviously coming from a cached object?
I think that the problem is that you are storing IQueryable's in your cache, and then cachednews contains an IQueryable that hits the database.
Try the following changes.
public IQueryable<VictoryList> GetVictoryList() {
// ...
if (cachednews == null)
{
var results = from // ...
results = results.ToList().AsQueryable(); // force query execution
SqlCacheDependency dependency = // ...;
HttpContext.Current.Cache.Add(cacheKey,
results, // now just the result are stored
dependency,
DateTime.MaxValue,
TimeSpan.Zero,
CacheItemPriority.Normal,
null);
return results;
}
return cachednews;
}