I use a similar style of code many times in my application to read in records from a database
WorkoutResultsRecord is inherited from a class called BaseRecord. One of the base constructors takes a IDataReader parameter to read fields into the class (as seen below).
What I want to do is define a generic function that will do the following for any/all of my 60+ xxxRecord type classes ie I can pass in a type as a parameter and it will return the correct type of objects as a typed List. Is it possible with Activator class? I've not used it before and my results just wouldn't compile
protected List<WorkoutResultsRecord> ReadRecordList(string sql,
IDbConnection connection)
{
var results = new List<WorkoutResultsRecord>();
using (IDbCommand command = GetCommand(sql, connection))
using (IDataReader reader = command.ExecuteReader())
while (reader.Read())
results.Add(new WorkoutResultsRecord(reader));
return results;
}
My really bad, failed attempt :(
private void sfsdf(Type type)
{
List<typeof(type)> lst = new List<type>();
Activator.CreateInstance(List<typeof(type)>);
}// function
this should work:
private void sfsdf(Type type)
{
Type genericType = typeof(List<>).MakeGenericType(type);
System.Collections.IList theList = (IList) Activator.CreateInstance(genericType);
// do whatever you like with this list...
}
Note: as the type is known at runtime only, it's not possible for you to declare a List when you write the code, so rather, use IList interface instead, but the created object theList should be of the expected type...
The following is the full function with all the generics done as I wanted. This will save a lot of typing!! thanks very much
allResults = (List<WorkoutResultsRecord>)FillList(typeof(WorkoutResultsRecord),
sql, connection, new KVP("FROMDATE", fromUtf.Date),
new KVP("TODATE", endDate.AddDays(1).Date));
IList FillList(Type type,string sql,IDbConnection connection,
params KVP[] parameters)
{
Type genericType = typeof(List<>).MakeGenericType(type);
IList results = (IList)Activator.CreateInstance(genericType);
using (var command= Command(sql,connection))
{
foreach(KVP parameter in parameters)
CreateParam(command,parameter.Key,parameter.Value);
using (IDataReader reader = command.ExecuteReader())
while (reader.Read())
results.Add(Activator.CreateInstance(type,reader));
}
return results;
}
Related
I am writing an application using C#, WPF, MVVM and SQL Server, and I am using Dapper as an ORM. Below is a snippet of the sample code I am trying to make work:
public List<MAKE THIS GENERIC> ExecuteSQLSPROC(string SPROCName, SqlParameter[] parameters, IEnumerable<MAKE THIS GENERIC> model)
{
using (IDbConnection connection = new System.Data.SqlClient.SqlConnection(DBHelper.CNNVal("MyDatabaseName")))
{
System.Data.SqlClient.SqlCommand command = new System.Data.SqlClient.SqlCommand();
command.Connection = (SqlConnection)connection;
command.CommandText = SPROCName;
DynamicParameters parms = new DynamicParameters();
for (int i = 0; i < parameters.Length; i++)
{
var parmname = parameters[i].ParameterName;
var parmvalue = parameters[i].Value;
// I KNOW I can put DIRECTION as a parameter variable but right now I don't want to!!
parms.Add(parmname, parmvalue);
}
var output = connection.Query<Make THIS GENERIC>(SPROCName, parms, commandType: CommandType.StoredProcedure).ToList();
return output; // return the recordset to the caller - a List<> of one object/model is acceptable - NOT POPULATING EXCEL spreadsheet with
// the results
}
}
I want to call this piece of code (which does not compile right now for obvious reasons) by supplying any ViewModel and be able to execute this code as if the ViewModel is generic within the routine. I need to know how to correctly code the above and also make the call to this routine with generic ViewModels (probably using a Lambda expression of some sort).
Basically, I want to be able to say "execute this SPROC, with these dynamic parameters and map the output to a ViewModel that is generic and return the results to the caller. Next time you're called it will be same logic, different SPROC, different parameters and ViewModel". I do NOT want extraneous comments that DO NOT directly answer this question. I know that I can add the parameter direction in a call the SQL SPROCs; but that knowledge is irrelevant to this question I am asking!
You can put your method in a generic class if you have other methods:
public class DbStuff<T>
{
public IList<T> ExecuteSQLSPROC(string SPROCName, SqlParameter[] parameters, IEnumerable<T> model)
{
using (IDbConnection connection = new System.Data.SqlClient.SqlConnection(DBHelper.CNNVal("MyDatabaseName")))
{
System.Data.SqlClient.SqlCommand command = new System.Data.SqlClient.SqlCommand();
command.Connection = (SqlConnection)connection;
command.CommandText = SPROCName;
DynamicParameters parms = new DynamicParameters();
for (int i = 0; i < parameters.Length; i++)
{
var parmname = parameters[i].ParameterName;
var parmvalue = parameters[i].Value;
// I KNOW I can put DIRECTION as a parameter variable but right now I don't want to!!
parms.Add(parmname, parmvalue);
}
var output = connection.Query<T>(SPROCName, parms, commandType: CommandType.StoredProcedure).ToList();
return output; // return the recordset to the caller - a List<> of one object/model is acceptable - NOT POPULATING EXCEL spreadsheet with
// the results
}
}
}
Then call it via:
var stuff = new DbStuff<MyViewModel>();
var results = stuff.ExecuteSQLSPROC("", blah, blah);
Or just make the method generic:
public IList<T> ExecuteSQLSPROC<T>(string SPROCName, SqlParameter[] parameters, IEnumerable<T> model)
I have thousands of queries I need to run on thousands of different Schema over 10 databases. I am trying to thread these queries and use a BlockingCollection to write results into, while also using another thread to read from this collection and write it to disk as the result sets of these queries are too large to store in memory.
Here is the problem area in my code:
public class Node {
public string ConnectionString;
public string Query;
public Node(string databaseDetails, string query) {
//Cannot put in actual logic, but this part is fine
ConnectionString = {logic for connection string}
Query = "set search_path to {schema from databaseDetails};" + query
}
}
public void runQuery(string query, BlockingCollection<Dictionary<string, object>> producer) {
List<Node> nodes = getNodes(query);
Parallel.ForEach(nodes, node => {
NpgsqlConnection conn = new NpgsqlConnection(node.ConnectionString);
conn.Open();
NpgsqlCommand npgQuery = new NpgsqlCommand(node.Query, conn);
NpgsqlDataReader reader = npgQuery.ExecuteReader();
while (reader.Read()) {
Dictionary<string, object> row = new Dictionary<string, object>();
for (int i = 0; i < reader.FieldCount; i++) {
row[reader.GetName(i)] = reader.GetValue(i);
}
producer.Add(row);
}
conn.Close();
});
producer.CompleteAdding();
}
This code runs, and retrieves all of the results, but it duplicates a lot of the results as well, so the blocking collection has 5-10 times more records than it should. Any help would be greatly appreaciated.
So I was just an idiot and was comparing my generated result set to the UNION of all the queries I was running not the UNION ALL, so my "true" result set had no duplicates in it because the union was removing them :/
Using the example code below as context... When I run this query I get the 'Id' field coming back as default value (which is 0 for an int). I would like to tell dapper to run in a manner where it would throw an exception if there is a column in the result set that does not get mapped to a property on my result object. (I understand that the issue is just that I need to remove the extra 'd' in the SQL query but I'm interested in having this expose itself more explicitly)
I've been unable to find anything on this topic. Please let me know if this is even possible with Dapper.
Thanks in advance (besides this issue, and for anyone who hasn't taken the plunge, Dapper really is the greatest thing since sliced bread!).
class CustomerRecord
{
public int Id { get; set; }
public string Name { get; set; }
}
CustomerRecord[] GetCustomerRecords()
{
CustomerRecord[] ret;
var sql = #"SELECT
CustomerRecordId AS Idd,
CustomerName as Name
FROM CustomerRecord";
using (var connection = new SqlConnection(this.connectionString))
{
ret = connection.Query<CustomerRecord>(sql).ToArray();
}
return ret;
}
You could create your own type map where you use Dapper's DefaultTypeMap and throw an exception when it cannot find the member:
public class ThrowWhenNullTypeMap<T> : SqlMapper.ITypeMap
{
private readonly SqlMapper.ITypeMap _defaultTypeMap = new DefaultTypeMap(typeof(T));
public ConstructorInfo FindConstructor(string[] names, Type[] types)
{
return _defaultTypeMap.FindConstructor(names, types);
}
public ConstructorInfo FindExplicitConstructor()
{
return _defaultTypeMap.FindExplicitConstructor();
}
public SqlMapper.IMemberMap GetConstructorParameter(ConstructorInfo constructor, string columnName)
{
return _defaultTypeMap.GetConstructorParameter(constructor, columnName);
}
public SqlMapper.IMemberMap GetMember(string columnName)
{
var member = _defaultTypeMap.GetMember(columnName);
if (member == null)
{
throw new Exception();
}
return member;
}
}
Downside of this, is that you have to configure all the type maps for every entity:
SqlMapper.SetTypeMap(typeof(CustomerRecord), typeof(ThrowWhenNullTypeMap<CustomerRecord>));
This could be configured using reflection, however.
I came here after I solved this same problem for the IEnumerable<dynamic> methods in Dapper. Then I found the proposal to solve the issue for Query<T>; but that doesn't seem to be going anywhere.
My answer builds on the answer proposed by #HenkMollema, and uses his class in the solution, so credit to him for that...
To solve the IEnumerable<dynamic> scenario, I had created a "SafeDynamic" class (follow the link above to see that). I refactored the static "Create" method into an extension method:
public static class EnumerableDynamicExtensions
{
public static IEnumerable<dynamic> Safe(this IEnumerable<dynamic> rows)
{
return rows.Select(x => new SafeDynamic(x));
}
}
and then I created a DapperExtensions class to provide 'Safe' versions of Query and Read (Read is used after QueryMultiple), to give me...
internal static class DapperExtensions
{
public static IEnumerable<dynamic> SafeQuery(this IDbConnection cnn, string sql, object param = null, IDbTransaction transaction = null, bool buffered = true, int? commandTimeout = default(int?), CommandType? commandType = default(CommandType?))
{
return cnn.Query(sql, param, transaction, buffered, commandTimeout, commandType).Safe();
}
public static IEnumerable<dynamic> SafeRead(this SqlMapper.GridReader gridReader, bool buffered = true)
{
return gridReader.Read(buffered).Safe();
}
}
So to solve this issue I added a "SafeQuery<T>" method to DapperExtensions, which takes care of setting up that type mapping for you:
private static readonly IDictionary<Type, object> TypesThatHaveMapper = new Dictionary<Type, object>();
public static IEnumerable<T> SafeQuery<T>(this IDbConnection cnn, string sql, object param = null, IDbTransaction transaction = null, bool buffered = true, int? commandTimeout = default(int?), CommandType? commandType = default(CommandType?))
{
if (TypesThatHaveMapper.ContainsKey(typeof(T)) == false)
{
SqlMapper.SetTypeMap(typeof(T), new ThrowWhenNullTypeMap<T>());
TypesThatHaveMapper.Add(typeof(T), null);
}
return cnn.Query<T>(sql, param, transaction, buffered, commandTimeout, commandType);
}
So if the original poster changes the call to Query to become SafeQuery, it should do what he requested
Edit 25/1/17
Improvements to avoid threading issues on the static dictionary:
private static readonly ConcurrentDictionary<Type, object> TypesThatHaveMapper = new ConcurrentDictionary<Type, object>();
public static IEnumerable<T> SafeQuery<T>(this IDbConnection cnn, string sql, object param = null, IDbTransaction transaction = null, bool buffered = true, int? commandTimeout = default(int?), CommandType? commandType = default(CommandType?))
{
TypesThatHaveMapper.AddOrUpdate(typeof(T), AddValue, UpdateValue);
return cnn.Query<T>(sql, param, transaction, buffered, commandTimeout, commandType);
}
private static object AddValue(Type type)
{
SqlMapper.SetTypeMap(type, XXX); // Apologies... XXX is left to the reader, as my implementation has moved on significantly.
return null;
}
private static object UpdateValue(Type type, object existingValue)
{
return null;
}
I'd like to expand on #Richardissimo 's answer by providing a visual studio project that includes his "SafeQuery" extention to Dapper, wrapped up nice and neat and tested.
https://github.com/LarrySmith-1437/SafeDapper
I use this in all my projects now to help keep the DAL clean of mismapped data, and felt the need to share. I would have posted up a Nuget, but the dependency on Dapper itself makes it much easier to post the project where consumers can update the reference to the Dapper version they want. Consume in good health, all.
Based on this thread and some other resources on SO, I've created an extension method without any custom mapper. What I needed was to throw when some property of my DTO was not set because for example SQL query has some column missing in SELECT statement.
This way my DTO would be set with default property silently and that's kinda dangerous.
The code can be simplified a little by not checking firstly for all properties being present in result, but throwing exception in the last Select call where we could iterate through properties of our type and check if query result has this property as well.
public static class Extensions
{
public static async Task<IEnumerable<T>> SafeQueryAsync<T>(
this IDbConnection cnn,
string sql,
object param = null,
IDbTransaction transaction = null,
int? commandTimeout = default(int?),
CommandType? commandType = default(CommandType?))
where T : new()
{
Dictionary<string, PropertyInfo> propertySetters = typeof(T)
.GetProperties().Where(p => p.CanRead && p.CanWrite)
.ToDictionary(p => p.Name.ToLowerInvariant(), p => p);
HashSet<string> typeProperties = propertySetters
.Select(p => p.Key)
.ToHashSet();
var rows = (await cnn.QueryAsync(sql, param, transaction, commandTimeout, commandType)).ToArray();
if (!rows.Any())
{
return Enumerable.Empty<T>();
}
var firstRow = rows.First();
HashSet<string> rowColumns = ((IDictionary<string, object>) firstRow)
.Select(kvp=>kvp.Key.ToLowerInvariant()).ToHashSet();
var notMappedColumns = typeProperties.Except(rowColumns).ToArray();
if (notMappedColumns.Any())
{
throw new InvalidOperationException(
$"Not all type properties had corresponding columns in SQL query. Query result lacks [{string.Join(", ", notMappedColumns)}]");
}
return rows.Select(row =>
{
IDictionary<string, object> rowDict = (IDictionary<string, object>) row;
T instance = new T();
rowDict.Where(o => propertySetters.ContainsKey(o.Key.ToLowerInvariant()))
.ToList().ForEach(o => propertySetters[o.Key.ToLowerInvariant()].SetValue(instance, o.Value));
return instance;
}).AsEnumerable();
}
}
When reading from an SQLiteDataReader I'm experiencing some odd behaviour whereby GetFieldType(0) returns typeof(Int64), GetValue(0) returns an Int64, but GetInt64(0) fails with an System.InvalidCastException exception.
It has taken me a rather long time to reproduce this behaviour:
using System;
using System.Data.SQLite;
using NUnit.Framework;
namespace Test
{
[TestFixture]
public class SQLiteType
{
[Test]
public void A()
{
var sqlConnection = new SQLiteConnection("Data Source=:memory:;Version=3;");
sqlConnection.Open();
var create = sqlConnection.CreateCommand();
create.CommandText = "CREATE TABLE FOO (x INTEGER)";
create.ExecuteNonQuery();
var insert = sqlConnection.CreateCommand();
insert.CommandText = "INSERT INTO FOO VALUES (?)";
var param = insert.CreateParameter();
param.Value = new TimeSpan(0); // NOTE INSERTING TIMESPAN DIRECTLY instead of .Ticks
insert.Parameters.Add(param);
insert.ExecuteNonQuery();
var select = sqlConnection.CreateCommand();
select.CommandText = "SELECT x FROM FOO";
var dr = select.ExecuteReader();
while (dr.Read())
{
var valueObject = dr.GetValue(0);
Assert.AreEqual(typeof (Int64), valueObject.GetType());
var valueType = dr.GetFieldType(0);
Assert.AreEqual(typeof (Int64), valueType);
var value = dr.GetInt64(0); // throws System.InvalidCastException
}
}
}
}
It seems to occur when the row is created by inserting a TimeSpan value directly into an INTEGER column (instead of e.g. TimeSpan.Ticks which might be more meaningful). Despite this, the datareader is still telling me that column is an Int64.
I'm not exactly sure what the contract is for SQLiteDataReader but I had previously assumed that if GetFieldType() returns a typeof(Int64), then GetInt64() should not fail. Perhaps this is not the case? (It seems quite odd that GetValue() still returns an Int64) Maybe it is an artifact of SQLite's unique dynamic typing system.
Certainly it is not hard to avoid, but for pedagogical reasons I am just curious why this is happening?
The root cause may have to do with how types are handled with SQLite:
http://www.sqlite.org/datatype3.html#affinity
Even then, this looks like a bug to me; if:
dr.GetValue(0).GetType() == typeof(System.Int64)
then it should certainly follow that dr.GetInt64(0) doesn't throw an exception. I would send an email to sqlite-users#sqlite.org as described here: http://www.sqlite.org/src/wiki?name=Bug+Reports
Please note though that if you replace:
param.Value = new TimeSpan(0);
with
param.Value = new TimeSpan(0).Ticks;
then
var value = dr.GetInt64(0);
works fine. I'm bringing this up because I'm not sure there is any conversion assumption to make when you assign that TimeSpan. For instance, there is no implicit or explicit conversion from TimeSpan to long.
I'm using Spring JdbcTemplate, and I'm stuck at the point where I have a query that updates a column that is actually an array of int. The database is postgres 8.3.7.
This is the code I'm using :
public int setUsersArray(int idUser, int idDevice, Collection<Integer> ids) {
int update = -666;
int[] tipi = new int[3];
tipi[0] = java.sql.Types.INTEGER;
tipi[1] = java.sql.Types.INTEGER;
tipi[2] = java.sql.Types.ARRAY;
try {
update = this.jdbcTemplate.update(setUsersArrayQuery, new Object[] {
ids, idUser, idDevice }, tipi);
} catch (Exception e) {
e.printStackTrace();
}
return update;
}
The query is "update table_name set array_column = ? where id_user = ? and id_device = ?".
I get this exception :
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [update acotel_msp.users_mau set denied_sub_client = ? where id_users = ? and id_mau = ?]; The column index is out of range: 4, number of columns: 3.; nested exception is org.postgresql.util.PSQLException: The column index is out of range: 4, number of columns: 3.
Caused by: org.postgresql.util.PSQLException: The column index is out of range: 4, number of columns: 3.
I've looked into spring jdbc template docs but I can't find any help, I'll keep looking, anyway could someone point me to the right direction? Thanks!
EDIT :
Obviously the order was wrong, my fault...
I tried both your solutions, in the first case I had this :
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [update users set denied_sub_client = ? where id_users = ? and id_device = ?]; nested exception is org.postgresql.util.PSQLException: Cannot cast an instance of java.util.ArrayList to type Types.ARRAY
Trying the second solution I had this :
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [update users set denied_sub_client = ? where id_users = ? and id_device = ?]; nested exception is org.postgresql.util.PSQLException: Cannot cast an instance of [Ljava.lang.Object; to type Types.ARRAY
I suppose i need an instance of java.sql.Array, but how can I create it using JdbcTemplate?
After struggling with many attempts, we settled to use a little helper ArraySqlValue to create Spring SqlValue objects for Java Array Types.
usage is like this
jdbcTemplate.update(
"UPDATE sometable SET arraycolumn = ?",
ArraySqlValue.create(arrayValue))
The ArraySqlValue can also be used in MapSqlParameterSource for use with NamedParameterJdbcTemplate.
import static com.google.common.base.Preconditions.checkNotNull;
import java.sql.Array;
import java.sql.JDBCType;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Locale;
import org.springframework.jdbc.core.StatementCreatorUtils;
import org.springframework.jdbc.support.SqlValue;
public class ArraySqlValue implements SqlValue {
private final Object[] arr;
private final String dbTypeName;
public static ArraySqlValue create(final Object[] arr) {
return new ArraySqlValue(arr, determineDbTypeName(arr));
}
public static ArraySqlValue create(final Object[] arr, final String dbTypeName) {
return new ArraySqlValue(arr, dbTypeName);
}
private ArraySqlValue(final Object[] arr, final String dbTypeName) {
this.arr = checkNotNull(arr);
this.dbTypeName = checkNotNull(dbTypeName);
}
#Override
public void setValue(final PreparedStatement ps, final int paramIndex) throws SQLException {
final Array arrayValue = ps.getConnection().createArrayOf(dbTypeName, arr);
ps.setArray(paramIndex, arrayValue);
}
#Override
public void cleanup() {}
private static String determineDbTypeName(final Object[] arr) {
// use Spring Utils similar to normal JdbcTemplate inner workings
final int sqlParameterType =
StatementCreatorUtils.javaTypeToSqlParameterType(arr.getClass().getComponentType());
final JDBCType jdbcTypeToUse = JDBCType.valueOf(sqlParameterType);
// lowercasing typename for Postgres
final String typeNameToUse = jdbcTypeToUse.getName().toLowerCase(Locale.US);
return typeNameToUse;
}
}
this code is provided in the Public Domain
private static final String ARRAY_DATATYPE = "int4";
private static final String SQL_UPDATE = "UPDATE foo SET arr = ? WHERE d = ?";
final Integer[] existing = ...;
final DateTime dt = ...;
getJdbcTemplate().update(new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(final Connection con) throws SQLException {
final PreparedStatement ret = con.prepareStatement(SQL_UPDATE);
ret.setArray(1, con.createArrayOf(ARRAY_DATATYPE, existing));
ret.setDate(2, new java.sql.Date(dt.getMillis()));
return ret;
}
});
This solution is kind of workaround using postgreSQL built-in function, which definitely worked for me.
reference blog
1) Convert String Array to Comma Separated String
If you are using Java8, it's pretty easy. other options are here
String commaSeparatedString = String.join(",",stringArray); // Java8 feature
2) PostgreSQL built-in function string_to_array()
you can find other postgreSQL array functions here
// tableName ( name text, string_array_column_name text[] )
String query = "insert into tableName(name,string_array_column_name ) values(?, string_to_array(?,',') )";
int[] types = new int[] { Types.VARCHAR, Types.VARCHAR};
Object[] psParams = new Object[] {"Dhruvil Thaker",commaSeparatedString };
jdbcTemplate.batchUpdate(query, psParams ,types); // assuming you have jdbctemplate instance
The cleanest way I found so far is to first convert the Collection into an Integer[] and then use the Connection to convert that into an Array.
Integer[] idArray = ids.toArray(new Integer[0]);
Array idSqlArray = jdbcTemplate.execute(
(Connection c) -> c.createArrayOf(JDBCType.INTEGER.getName(), idArray)
);
update = this.jdbcTemplate.update(setUsersArrayQuery, new Object[] {
idSqlArray, idUser, idDevice })
This is based on information in the documentation: https://jdbc.postgresql.org/documentation/head/arrays.html
The argument type and argument is not matching.
Try changing the argument type order
int[] tipi = new int[3];
tipi[0] = java.sql.Types.ARRAY;
tipi[1] = java.sql.Types.INTEGER;
tipi[2] = java.sql.Types.INTEGER;
or use
update = this.jdbcTemplate.update(setUsersArrayQuery, new Object[] {
ids.toArray(), idUser, idDevice })
and see if it works
http://valgogtech.blogspot.com/2009/02/passing-arrays-to-postgresql-database.html explains how to create java.sql.Array postgresql
basically Array.getBaseTypeName should return int and Array.toString should return the array content in "{1,2,3}" format
after you create the array you can set it using preparedstatement.setArray(...)
from PreparedStatementCreator e.g.
jdbcTemplate.update(
new PreparedStatementCreator() {
public PreparedStatement createPreparedStatement(Connection connection) throws SQLException {
Good Luck ..
java.sql.Array intArray = connection.createArrayOf("int", existing);
List<Object> values= new ArrayList<Object>();
values.add(intArray);
values.add(dt);
getJdbcTemplate().update(SQL_UPDATE,values);