I'm trying to parse query results returned from an Azure Elastic Scale MultiShardConnection. It doesn't look like it inherits from SqlConnection or DbConnection so the Dapper methods are not available. This make sense when you consider that it's executing a fan-out query that is union'ed together. What I was hoping to do was to use existing Dapper functionality to just handle the parser of the reader results to a type.
Are those mapping features available if I don't use Dapper for the original connection?
Below are the types I'm working with:
MultiShardConnection : IDisposable
MultiShardCommand : DbCommand
MultiShardDataReader : DbDataReader, IDataReader, IDisposable, IDataRecord
Here's an example query where I'm trying to use the Dapper mapper.
Customer customer = null;
using (MultiShardConnection conn = GetMultiShardConnection())
using (MultiShardCommand cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT * FROM [Customer] WHERE ...";
cmd.CommandTimeout = 120;
cmd.CommandTimeoutPerShard = 120;
using (MultiShardDataReader reader = await cmd.ExecuteReaderAsync())
{
while (reader.Read())
{
// Replace this with mapper...
customer = new Customer()
{
CustomerId = reader.GetInt32(0)
//etc...
};
}
}
}
return customer;
Update
I ended up needing to use sp_execute_fanout
using (var con = GetConnection())
{
await con.OpenAsync();
return (await con.QueryAsync<Customer>("sp_execute_fanout ", new
{
shard_map_manager_server = "my_server.database.windows.net",
shard_map_manager_database = "my_shardmapmananger",
user_id = "my_username",
password = "my_password",
shard_map_name = "my_shardmap",
statement = "SELECT * FROM Customer"
}, commandTimeout: 120, commandType: CommandType.StoredProcedure)).FirstOrDefault();
}
Currently, MultiShardConnection is not integrated with Dapper. The reason is exactly as you point out that it does not implement DbConnection. As an alternative solution, I would recommend to try elastic database query (EDQ): https://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-query-overview/. With EDQ, you can simply connect to a single database in Azure DB and use regular Dapper over the EDQ external tables to query across your shards. EDQ is now available on all service tiers in Azure SQL DB.
Let us know how that works for you.
Thanks,
Torsten
I tried the solution from OP today, because I also wanted to use Dapper to map query results from multiple shards, but I noticed that sp_execute_fanout is deprecated and replaced by sp_execute_remote.
Before you can use this sp, you need to create an external source as a reference to the Shard Map Manager.
You can then use the name of this external source as the data source name ('TestExtScr' in my example) and do something like this in your code:
using (var con = new SqlConnection(connString))
{
return await con.QueryAsync<Customer>("sp_execute_remote", new
{
data_source_name = "TestExtSrc",
statement = "SELECT foo FROM bar"
}, commandTimeout: 120, commandType: CommandType.StoredProcedure);
}
This way you can use Dapper while querying multiple shards.
I know I'm bumping an old post, but when I was looking for a solution to my problem, this post kept showing up. So I added this bit in case someone in the future is looking for the same thing :) .
Related
I'm writing a new ASP.NET Core Web API, and one of my requirements is to be able to leverage EF Core 3.1 to grab the next value of a sequence defined in my SQL Server as the ID for a record I need to store.
I'm struggling to find a way to do this - in EF 6.x, I used a method directly on the DbContext descendant like this:
public int GetNextSequenceValue()
{
var rawQuery = Database.SqlQuery<int>("SELECT NEXT VALUE FOR dbo.TestSequence;");
var task = rawQuery.SingleAsync();
int nextVal = task.Result;
return nextVal;
}
and for EF Core up to 2.1, I would have been able to use Database.ExecuteSqlCommand() to run a SQL snippet and get back results. But it seems, in EF Core 3.x, I'm out of luck....
I know there are the .FromSqlRaw() and .FromSqlInterpolated methods on the DbSet - but since I only need to return the next value of a sequence (an INT), that's not going to fly. And I also know these methods also exist on the context.Database level which looks like it would be really close to what I had in EF 6.x - but here, those methods will only return the number of rows affected - I haven't found a way to send back the new value from the SEQUENCE.
Can it really be that in EF Core 3.x, I have to actually resort back to way-old ADO.NET code to fetch that value?? Is there REALLY no way to execute an arbitrary SQL snippet and get back some results from the context??
If you want to run an arbitrary TSQL batch and return a scalar value, you can do it like this:
var p = new SqlParameter("#result", System.Data.SqlDbType.Int);
p.Direction = System.Data.ParameterDirection.Output;
context.Database.ExecuteSqlRaw("set #result = next value for some_seq", p);
var nextVal = (int)p.Value;
Looks like executing raw SQL is not priority for EF Core, so up to now (EF Core 3.1) it's providing publicly just few basic limited methods. FromSql requires entity type or keyless entity type, and ExecuteSqlRaw / ExecuteSqlInterpolated are the "modern" bridge to ADO.NET ExecuteNonQuery which returns the affected rows.
The good thing is that EF Core is built on top of a public service architecture, so it can be used to add some missing functionalities. For instance, services can be used to build the so called IRelationalCommand, which has all the DbCommand execute methods, in particular ExecuteScalar needed for SQL in question.
Since EF Core model supports sequences, there is also a service for building the IRelationalCommand needed to retrieve the next value (used internally by HiLo value generators).
With that being said, following is a sample implementation of the custom method in question using the aforementioned concepts:
using System;
using System.Globalization;
using Microsoft.EntityFrameworkCore.Diagnostics;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Microsoft.EntityFrameworkCore.Storage;
using Microsoft.EntityFrameworkCore.Update;
namespace Microsoft.EntityFrameworkCore
{
public static partial class CustomExtensions
{
public static long GetNextSequenceValue(this DbContext context, string name, string schema = null)
{
var sqlGenerator = context.GetService<IUpdateSqlGenerator>();
var sql = sqlGenerator.GenerateNextSequenceValueOperation(name, schema ?? context.Model.GetDefaultSchema());
var rawCommandBuilder = context.GetService<IRawSqlCommandBuilder>();
var command = rawCommandBuilder.Build(sql);
var connection = context.GetService<IRelationalConnection>();
var logger = context.GetService<IDiagnosticsLogger<DbLoggerCategory.Database.Command>>();
var parameters = new RelationalCommandParameterObject(connection, null, null, context, logger);
var result = command.ExecuteScalar(parameters);
return Convert.ToInt64(result, CultureInfo.InvariantCulture);
}
}
}
In your fluent api configs you can create migration that set ID automatically to be next value from Sequence
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.HasSequence<int>("OrderNumbers");
modelBuilder.Entity<Order>()
.Property(o => o.OrderNo)
.HasDefaultValueSql("NEXT VALUE FOR shared.OrderNumbers");
}
For creating sequence:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.HasSequence<int>("OrderNumbers", schema: "shared")
.StartsAt(1000)
.IncrementsBy(5);
}
Read more from here: https://www.talkingdotnet.com/use-sql-server-sequence-in-entity-framework-core-primary-key/
For people suffering with oracle version of this problem, here's a solution:
var p = new OracleParameter("result", OracleDbType.Decimal, null, System.Data.ParameterDirection.Output);
Database.ExecuteSqlRaw($"BEGIN :result := my_seq.nextval; END;", p);
var nextVal = p.Value;
Ugly, but the best thing I could come up with:
var connection = repcontext.Database.GetDbConnection();
connection.Open();
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT NEXT VALUE FOR AA.TransSeq;";
var obj = cmd.ExecuteScalar();
connection.Close();
seqnum = (int)obj;
This code should work in a variety of situations:
public static class DbSequence
{
private const string sqlCode = "SELECT NEXT VALUE FOR {0}.{1};";
public static T GetNextSeq<T>(this DbContext dbContext, string seqName)
{
var sqlCnn = dbContext.Database.GetDbConnection();
bool cnnClosed = sqlCnn.State != ConnectionState.Open;
if (cnnClosed) sqlCnn.Open();
try
{
using (var sqlCmd = sqlCnn.CreateCommand())
{
sqlCmd.Transaction = dbContext.Database.CurrentTransaction?.GetDbTransaction();
sqlCmd.CommandText = string.Format(sqlCode, "dbo", seqName);
var result = sqlCmd.ExecuteScalar();
if ((result == null) || (result == DBNull.Value)) throw new InvalidOperationException();
return (T)result;
}
}
finally
{
if (cnnClosed) sqlCnn.Close();
}
}
}
This code works when the connection is closed, opening it when needed and closing it after itself. It should also work when a transaction has been initiated. According to this source: https://learn.microsoft.com/en-us/sql/t-sql/statements/create-sequence-transact-sql?view=sql-server-ver16#general-remarks sequences run outside of transactions. Still, if it's available, I set up a transaction for the command. I also use generics and extension methods.
In a 3 tier architecture (C# , ADO.NET), are datasets and datatables the only option to return data from the Data Layer to the Presentation Layer?. I've been working with Datatables but when I will ttry to do this
public User getUserByP(User user)
{
User t = new User();
using(SqlConnection con = new SqlConnection(Conexion.Cn))
{
con.Open();
SqlCommand command = new SqlCommand("spLogIn_User", con);
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#user", User.user);
command.Parameters.AddWithValue("#password", User.Pass);
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
t.IdUser = reader.GetInt32(0);
t.Name = reader.GetString(1);
t.LastName = reader.GetString(2);
t.Access = reader.GetString(3);
t.user = reader.GetString(4);
t.Pass = reader.GetString(5);
}
}
return t;
}
I ill have an error cause there is no communication between Data Layer and Presentation Layer. This is posible in MVC (i think) but no here. So if a just want return even 1 result , datatble or dataset are only option?
No. There are other alternatives to retrieve data from databases into your data layer.
Linq to sql
Entity Framework
nHibernate
Dapper.Net
If on the other hand, you want to know how to access the data layer from the presentation layer without additional dependencies, you could do the following.
In the data layer, always return a simple object instead of an specialized type (datatable/dataset) to avoid unnecessary dependencies to system.data or other dlls
Add a business layer. In a 3 tier application, this layer uses the data from the data layer and adds validation, business rules and logic to access and manipulate that data.
Reference the business layer in the presentation layer to display the information desired.
For more details on how to achieve a 3-tier/layer architecture refer to this explanation
All that I can find on net is for asp.net use.
All that I want is to retrieve only the page count amount of records from database every time. Isn't this possible?
The bottleneck seems to be the disposal of the context. Dunno how to resolve this.
Thanks.
Nico
if you just need a simple sql query you can use this example:
using System;
using System.Data.SqlClient;
class Program
{
static void Main()
{
//
// The name we are trying to match.
//
string dogName = "Fido";
//
// Use preset string for connection and open it.
//
string connectionString = ConsoleApplication1.Properties.Settings.Default.ConnectionString;
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
//
// Description of SQL command:
// 1. It selects all cells from rows matching the name.
// 2. It uses LIKE operator because Name is a Text field.
// 3. #Name must be added as a new SqlParameter.
//
using (SqlCommand command = new SqlCommand("SELECT count(*) FROM Dogs1 WHERE Name LIKE #Name", connection))
{
//
// Add new SqlParameter to the command.
//
command.Parameters.Add(new SqlParameter("Name", dogName));
//
// Read in the SELECT results.
//
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
int count = reader[0];
}
// Call Close when done reading.
reader.Close();
}
}
}
}
keep in mind you'll need to find out the connectionString to your database. depend on what database you have you can use google to find out how to do it.
Some times, sample data is important but boring to generate due to many related tables.
So, in Entity Framework Code First, I think it's an elegant way to insert them in DropCreateDatabaseIfModelChanges::Seed().
But is there a way to export existing data from database back to a string as C# / VB sentences that insert poco objects?
If it works, we may use it to backup data gracefully, or save to multiple .cs by scenes, switch them as needed to better our test, etc.
DATABASE EXPORTS AS THE FOLLOWING CODE:
var students = new List<Student>
{
new Student { FirstMidName = "Carson", LastName = "Alexander", },
new Student { FirstMidName = "Meredith", LastName = "Alonso", },
new Student { FirstMidName = "Arturo", LastName = "Anand", },
// ...
};
students.ForEach(s => context.Students.Add(s));
context.SaveChanges();
As #Luke suggested in his comment. the best way is to generate Json files for your seed data.
in this article, you can find the easy way to generate your Json from SQL query.
Then all you need is to read them by Newtonsoft.Json
e.g.
var countriesStream = new StreamReader(HostingEnvironment.MapPath("jsonFile.json"));
try
{
countriesList = JsonConvert.DeserializeObject<List<Country>>(countriesStream.ReadToEnd());
}
In the earlier versions of Entity Framework, we were able to reach the Context out of ObjectQuery in order to read Parameters, Connection, etc. as below:
var query = (ObjectQuery<T>)source;
cmd.Connection = (SqlConnection)((EntityConnection)query.Context.Connection).StoreConnection;
cmd.Parameters.AddRange(
query.Parameters.Select(x => new SqlParameter(
x.Name, x.Value ?? DBNull.Value)
).ToArray()
);
When I look at the DbSet<T> object, I am unable to find any equivalent of this. My purpose here is to create extensions which will manipulate the query and get the result out of it.
Here is an instance: http://philsversion.com/2011/09/07/async-entity-framework-queries
Or should I write the extension for DbContext class and work with Set method?
Any idea?
Edit
Here is what I did so far. Basic implementation so far but certainly not ready for production. Any suggestions on this?
public static async Task<IEnumerable<T>> QueryAsync<T>(this DbContext #this, System.Linq.Expressions.Expression<Func<T, bool>> predicate = null)
where T : class {
var query = (predicate != null) ? #this.Set<T>().Where(predicate) : #this.Set<T>();
var cmd = new SqlCommand();
cmd.Connection = (SqlConnection)(#this.Database.Connection);
cmd.CommandText = query.ToString();
if (cmd.Connection.State == System.Data.ConnectionState.Closed) {
cmd.Connection.ConnectionString = new SqlConnectionStringBuilder(cmd.Connection.ConnectionString) {
AsynchronousProcessing = true
}.ToString();
cmd.Connection.Open();
}
cmd.Disposed += (o, e) => {
cmd.Clone();
};
var source = ((IObjectContextAdapter)#this).ObjectContext.Translate<T>(
await cmd.ExecuteReaderAsync()
);
return source;
}
This is a nice workaround, although I don't think you can make it much more generally applicable than what you already have.
A few things to keep in mind:
- Depending on the EF query, e.g. if you are using Include or not, the columns returned in the reader might not match the properties in the type T you are passsing.
- Depending on whether you have inheritance in your model, the T that you pass to translate may not always be the right thing to materialize for every row returned.
- After the task returned by ExecuteReaderAsync completes, you still have to retrieve each row, which depending on the execution plan for the query and the latency you are getting with the server is potentially also a blocking operation.
Async support is not coming to EF in 5.0 but we worked with other teams to make sure we have all the necessary building blocks included in .NET 4.5 and the feature is pretty high in our priority list. I encourage you to vote for it in our UserVoice site.