I have thousands of queries I need to run on thousands of different Schema over 10 databases. I am trying to thread these queries and use a BlockingCollection to write results into, while also using another thread to read from this collection and write it to disk as the result sets of these queries are too large to store in memory.
Here is the problem area in my code:
public class Node {
public string ConnectionString;
public string Query;
public Node(string databaseDetails, string query) {
//Cannot put in actual logic, but this part is fine
ConnectionString = {logic for connection string}
Query = "set search_path to {schema from databaseDetails};" + query
}
}
public void runQuery(string query, BlockingCollection<Dictionary<string, object>> producer) {
List<Node> nodes = getNodes(query);
Parallel.ForEach(nodes, node => {
NpgsqlConnection conn = new NpgsqlConnection(node.ConnectionString);
conn.Open();
NpgsqlCommand npgQuery = new NpgsqlCommand(node.Query, conn);
NpgsqlDataReader reader = npgQuery.ExecuteReader();
while (reader.Read()) {
Dictionary<string, object> row = new Dictionary<string, object>();
for (int i = 0; i < reader.FieldCount; i++) {
row[reader.GetName(i)] = reader.GetValue(i);
}
producer.Add(row);
}
conn.Close();
});
producer.CompleteAdding();
}
This code runs, and retrieves all of the results, but it duplicates a lot of the results as well, so the blocking collection has 5-10 times more records than it should. Any help would be greatly appreaciated.
So I was just an idiot and was comparing my generated result set to the UNION of all the queries I was running not the UNION ALL, so my "true" result set had no duplicates in it because the union was removing them :/
Related
I am trying to get data using a SqlDataReader. When I look at the result, I do see the number of columns but rows are missing and I am seeing only blanks with "the enumeration yielded no results".
After spending much time and going through similar issues on the net I couldn't resolve my issue. Then I tried the same stored procedure but used the DataAdapter. Using a DataAdapter, all the rows were returned. I am not sure what is causing the issue. I tried to rewrite my methods differently but no avail.
public static SqlDataReader ExecuteDataReader(string connection, string storedProcedure)
{
using (var sqlCon = new SqlConnection(connection))
{
// sqlCon.Open();
var cmd = new SqlCommand(storedProcedure, sqlCon)
{
CommandType = CommandType.StoredProcedure
};
sqlCon.Open();
using (var reader = cmd.ExecuteReader())
{
var count = reader.FieldCount;
while (reader.Read())
{
for (var i = 0; i < count; i++)
{
Console.WriteLine(reader.GetValue(i));
}
}
}
//reader.Close();
return null; // for testing purpose
}
}
--stored procedure. (kept it simple)
Create PROC [dbo].GetTheData
As
select '1' AS one,'2' as two,'3' as three,'4' as four,'5'as five
Earlier I had a table named ApplicationConfiguration which simply had [Key],[Value] columns to store some config data. This was queried straight away using SQL queries.
Now I intend to make use of Entity Framework (EF) Code First approach to query this table. The specialty of this table is that the table will have only a fixed number of rows in its lifetime. Only the Value column can be updated.
So as per the code first approach, we have to first write our POCO classes with its properties that will be mapped to columns in the underlying table. However, I wish to have a Dictionary<> structure to represent these configuration KV pairs. My concern is, will EF be able to fire update queries against any updation to the the value of a particular pair.
Also since I am using Code First approach, I would want some seed data(i.e the fixed number of rows and its initial content) to the added after the table itself is created on the fly when the application is first executed.
If Dictionary<> cannot be used, please suggest some alternative. Thanks in advance.
Coded this way:
public class ApplicationConfiguration
{
public int Id { get; set; }
public string Key { get; set; }
public int Value { get; set; } // should be string, but I'm lazy
}
class Context : DbContext
{
internal class ContextInitializer : DropCreateDatabaseIfModelChanges<Context>
{
protected override void Seed(Context context)
{
var defaults = new List<ApplicationConfiguration>
{
new ApplicationConfiguration {Key = "Top", Value = 5},
new ApplicationConfiguration {Key = "Bottom", Value = 7},
new ApplicationConfiguration {Key = "Left", Value = 1},
new ApplicationConfiguration {Key = "Right", Value = 3}
};
// foreach (var c in defaults)
// context.ConfigurationMap.Add(c.Key, c); // by design, no IReadOnlyDictionary.Add
foreach (var c in defaults)
context.ApplicationConfigurations.Add(c);
base.Seed(context);
}
}
public Context()
{
Database.SetInitializer(new ContextInitializer());
}
private IDbSet<ApplicationConfiguration> ApplicationConfigurations
{
get { return Set<ApplicationConfiguration>(); }
}
public IReadOnlyDictionary<string, ApplicationConfiguration> ConfigurationMap
{
get { return ApplicationConfigurations.ToDictionary(kvp => kvp.Key, kvp => kvp); }
}
}
Used this way:
using (var context = new Context())
{
ReadConfigurationOnly(context.ConfigurationMap);
}
using (var context = new Context())
{
ModifyConfiguration(context.ConfigurationMap);
context.SaveChanges();
}
static void ReadConfigurationOnly(IReadOnlyDictionary<string, ApplicationConfiguration> configuration)
{
foreach (var k in configuration.Keys)
Console.WriteLine("{0} = {1}", k, configuration[k].Value);
}
static void ModifyConfiguration(IReadOnlyDictionary<string, ApplicationConfiguration> configuration)
{
foreach (var k in configuration.Keys)
configuration[k].Value++; // this is why I was lazy, using an int for a string
}
So, I wrote it up this way — using an int Value property rather than a string — just so I could run the "Used this way" code over and over, and see the database update each time, without having to come up with some other way to change Value in an interesting way.
It's not quite as nifty here to use a IReadOnlyDictionary<string, ApplicatonConfiguration> instead of a IReadOnlyDictionary<string, string>, the way we'd really like, but that's more than made up for by the fact that we can easily modify our collection values without resorting to a clumsier Set method taking a dictionary as input. The drawback, of course, is that we have to settle for configuration[key].Value = "new value" rather than configuration[key] = "new value", but — as I say — I think it's worth it.
EDIT
Dang! I wrote this code up specifically to answer this question, but I think I like it so much, I'm going to add it to my bag of tricks ... this would fit in really well when my company goes from local databases to Azure instances in the cloud, and the current app.config has to go into the database.
Now all I need is a ContextInitializer taking a System.Configuration.ConfigurationManager as a ctor parameter in order to seed a new database from an existing app.config ...
I don't think you can map a table directly to a Dictionary; you will probably have to write your own wrapper to fill a dictionary from the table and update it back to the DB. Entities are each a row of a given table... Something like this (untested):
public Dictionary<string, string> GetDictionary()
{
Dictionary<string, string> dic = new Dictionary<string, string>();
using (var db = new Context())
{
var configs = db.ApplicationConfiguration.Select();
foreach (var entry in configs)
{
dic.Add(config.Key, config.Value);
}
}
return dic;
}
public void SaveConfig(Dictionary<string, string> dic)
{
using (var db = new Context())
{
foreach (KeyValuePair kvp in dic)
{
if (!db.ApplicationConfiguration.First(a => a.Key == kvp.Key).Value == kvp.Value)
{
var ac = new ApplicationConfiguration();
ac.Key = kvp.Key;
ac.Value = kvp.Value;
db.Entry(ac).State = EntityState.Modified;
}
}
db.SaveChanges();
}
}
For your second question, you want to use the Seed() method to add initial values to the database. See here for an example implementation.
When I call
connection.Execute(sql);
Dapper executes and everything is fine. When I call
connection.Execute(sql, new { UserId = _userId });
it executes with sp_executesql.
The issue is when it uses sp_executesql it's in its own scope. If it creates a temporary table, it's not accessible to subsequent queries that use the same connection. I could get around it by using global temporary tables, but I don't want to risk having two processes interfere with each other.
Does anybody know a way around that?
Update: I have the same problem when I use SqlCommand objects without Dapper. I wrote a unit test that illustrates the problem I'm having. WorksWithParameters fails with System.Data.SqlClient.SqlException : Invalid object name '#TEMP_OBJECTLIST'.
[TestFixture]
public class DapperTest
{
private const string TestObjectType = "S";
private const string ConnectionString = "XXXXXXXXX";
private static void CreateTempTableWithoutParameters(SqlConnection connection)
{
const string sql = "SELECT TOP 10 * INTO #TEMP_OBJECTLIST FROM sys.objects WHERE TYPE = 'S'";
connection.Execute(sql);
}
private static void UseTempTableWithoutParameters(SqlConnection connection)
{
const int expectedCount = 10;
const string sql = "SELECT COUNT(*) FROM #TEMP_OBJECTLIST WHERE TYPE = 'S'";
var count = connection.Query<int>(sql).First();
Assert.AreEqual(expectedCount, count);
}
private static void CreateTempTableWithParameters(SqlConnection connection)
{
const string sql = "SELECT TOP 10 * INTO #TEMP_OBJECTLIST FROM sys.objects WHERE TYPE = #OBJECT_TYPE";
connection.Execute(sql, new {OBJECT_TYPE = TestObjectType});
}
private static void UseTempTableWithParameters(SqlConnection connection)
{
const int expectedCount = 10;
const string sql = "SELECT COUNT(*) FROM #TEMP_OBJECTLIST WHERE TYPE = #OBJECT_TYPE";
var param = new {OBJECT_TYPE = TestObjectType};
var count = connection.Query<int>(sql, param).First();
Assert.AreEqual(expectedCount, count);
}
[Test]
public void WorksWithParameters()
{
using (var connection = new SqlConnection(ConnectionString))
{
connection.Open();
CreateTempTableWithParameters(connection);
UseTempTableWithParameters(connection);
}
}
[Test]
public void WorksWithoutParameters()
{
using (var connection = new SqlConnection(ConnectionString))
{
connection.Open();
CreateTempTableWithoutParameters(connection);
UseTempTableWithoutParameters(connection);
}
}
}
One way around the temp table scope problem is to create the temp table with one dummy column in the outer scope, then use alter table statements to add all the desired columns and use it.
Additionally, How to share data between procedures by Erland Sommarskog may be useful to you or another person looking for different options for sharing data.
I ran into the same problem with Dapper, but it's not Dapper's fault. sp_executesql is called by ADO.NET and this switches the "scope" so temp tables become invisible.
As a workaround:
//no parameters, so it runs without sp_executesql
conn.Execute("CREATE TABLE #temp BLAHBLAH");
//do your thing
conn.Execute("INSERT INTO #temp BLAHBLAH", parameters);
//cleanup (no parameters again)
conn.Execute("DROP TABLE #temp");
i have the following code but don't understand where am i getting wrong. There are five items in the xml file but this code shows none.
public static List<string> LoadLedgersString()
{
List<string> ListLedgerStringRecords = new List<string>();
// Execute the query using the LINQ to XML
XDocument doc = XDocument.Load(#"Ledgers.xml");
var records = from r in doc.Element("Ledgers").Elements("Ledger") select r;
foreach (var record in records)
{
string lLedgerString = record.Element("Name").Value;
ListLedgerStringRecords.Add(lLedgerString);
}
return ListLedgerStringRecords;
}
The code that is calling and utilizing its result is following
List<string> ledgerList = new List<string>();
ledgerList = DAL_Ledgers.LoadLedgersString();
ledgerListView.DataContext = ledgerList;
ICollectionView view =
CollectionViewSource.GetDefaultView(ledgerList);
new TextSearchFilter(view, this.searchTextBox);
This will probably be a problem with namespacing or with pathing.
Please post some example XML, then we can try to help.
I want to pass a collection of ids to a stored procedure that will be mapped using NHibernate. This technique was introduced in Sql Server 2008 ( more info here => Table-Valued Parameters ). I just don't want to pass multiple ids within an nvarchar parameter and then chop its value on the SQL Server side.
My first, ad hoc, idea was to implement my own IType.
public class Sql2008Structured : IType {
private static readonly SqlType[] x = new[] { new SqlType(DbType.Object) };
public SqlType[] SqlTypes(NHibernate.Engine.IMapping mapping) {
return x;
}
public bool IsCollectionType {
get { return true; }
}
public int GetColumnSpan(NHibernate.Engine.IMapping mapping) {
return 1;
}
public void NullSafeSet(DbCommand st, object value, int index, NHibernate.Engine.ISessionImplementor session) {
var s = st as SqlCommand;
if (s != null) {
s.Parameters[index].SqlDbType = SqlDbType.Structured;
s.Parameters[index].TypeName = "IntTable";
s.Parameters[index].Value = value;
}
else {
throw new NotImplementedException();
}
}
#region IType Members...
#region ICacheAssembler Members...
}
No more methods are implemented; a throw new NotImplementedException(); is in all the rest. Next, I created a simple extension for IQuery.
public static class StructuredExtensions {
private static readonly Sql2008Structured structured = new Sql2008Structured();
public static IQuery SetStructured(this IQuery query, string name, DataTable dt) {
return query.SetParameter(name, dt, structured);
}
}
Typical usage for me is
DataTable dt = ...;
ISession s = ...;
var l = s.CreateSQLQuery("EXEC some_sp #id = :id, #par1 = :par1")
.SetStructured("id", dt)
.SetParameter("par1", ...)
.SetResultTransformer(Transformers.AliasToBean<SomeEntity>())
.List<SomeEntity>();
Ok, but what is an "IntTable"? It's the name of SQL type created to pass table value arguments.
CREATE TYPE IntTable AS TABLE
(
ID INT
);
And some_sp could be like
CREATE PROCEDURE some_sp
#id IntTable READONLY,
#par1 ...
AS
BEGIN
...
END
It only works with Sql Server 2008 of course and in this particular implementation with a single-column DataTable.
var dt = new DataTable();
dt.Columns.Add("ID", typeof(int));
It's POC only, not a complete solution, but it works and might be useful when customized. If someone knows a better/shorter solution let us know.
A simpler solution than the accepted answer would be to use ADO.NET. NHibernate allows users to enlist IDbCommands into NHibernate transactions.
DataTable myIntsDataTable = new DataTable();
myIntsDataTable.Columns.Add("ID", typeof(int));
// ... Add rows to DataTable
ISession session = sessionFactory.GetSession();
using(ITransaction transaction = session.BeginTransaction())
{
IDbCommand command = new SqlCommand("StoredProcedureName");
command.Connection = session.Connection;
command.CommandType = CommandType.StoredProcedure;
var parameter = new SqlParameter();
parameter.ParameterName = "IntTable";
parameter.SqlDbType = SqlDbType.Structured;
parameter.Value = myIntsDataTable;
command.Parameters.Add(parameter);
session.Transaction.Enlist(command);
command.ExecuteNonQuery();
}
For my case, my stored procedure needs to be called in the middle of an open transaction.
If there is an open transaction, this code works because it is automatically reusing the existing transaction of the NHibernate session:
NHibernateSession.GetNamedQuery("SaveStoredProc")
.SetInt64("spData", 500)
.ExecuteUpdate();
However, for my new Stored Procedure, the parameter is not as simple as an Int64. It's a table-valued-parameter (User Defined Table Type)
My problem is that I cannot find the proper Set function.
I tried SetParameter("spData", tvpObj), but it's returning this error:
Could not determine a type for class: …
Anyways, after some trial and error, this approach below seems to work.
The Enlist() function is the key in this approach. It basically tells the SQLCommand to use the existing transaction. Without it, there will be an error saying
ExecuteNonQuery requires the command to have a transaction when the
connection assigned to the command is in a pending local transaction…
using (SqlCommand cmd = NHibernateSession.Connection.CreateCommand() as SqlCommand)
{
cmd.CommandText = "MyStoredProc";
NHibernateSession.Transaction.Enlist(cmd); // Because there is a pending transaction
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter("#wiData", SqlDbType.Structured) { Value = wiSnSqlList });
int affected = cmd.ExecuteNonQuery();
}
Since I am using the SqlParameter class with this approach, SqlDbType.Structured is available.
This is the function where wiSnList gets assigned:
private IEnumerable<SqlDataRecord> TransformWiSnListToSql(IList<SHWorkInstructionSnapshot> wiSnList)
{
if (wiSnList == null)
{
yield break;
}
var schema = new[]
{
new SqlMetaData("OriginalId", SqlDbType.BigInt), //0
new SqlMetaData("ReportId", SqlDbType.BigInt), //1
new SqlMetaData("Description", SqlDbType.DateTime), //2
};
SqlDataRecord row = new SqlDataRecord(schema);
foreach (var wi in wiSnList)
{
row.SetSqlInt64(0, wi.OriginalId);
row.SetSqlInt64(1, wi.ShiftHandoverReportId);
if (wi.Description == null)
{
row.SetDBNull(2);
}
else
{
row.SetSqlString(2, wi.Description);
}
yield return row;
}
}
You can pass collections of values without the hassle.
Example:
var ids = new[] {1, 2, 3};
var query = session.CreateQuery("from Foo where id in (:ids)");
query.SetParameterList("ids", ids);
NHibernate will create a parameter for each element.