Related
I am creating a new SSIS 2008 ETL report that will read data from a SQL Server and append it to an ACCESS 2010 database table. How can I append records to an existing access table when there is an identity column in the Access table?
My current solution has an OLE DB source with SQL that reads the data. It connects to a data conversion task and then that connects to an OLE DB destination task. The identity field in the access table is named "Id" and it is an autonumber field.
I am able to insert the values using an OLE DB destination when the Access table is empty. The problem has to do with the identity field and when there are already records in the table.
So, the problem is that my Id column always starts at 1, which means I get an error as the engine thinks I am trying to insert a duplicate key.
I created a SQL Task that reads the max ID from the table and stores it into a variable, but now I am stuck trying to figure out how to use this value in my data flow export process.
Will someone please show me how to insert data into an Access DB when there are already records in the table?
Edited for clarity. I have added the SQL that I use to obtain the desired SQL records below. Note that I am NOT including the ID identity field in my SQL:
DECLARE #STARTDATE DATETIME = (SELECT CONVERT(VARCHAR(25),DATEADD(DD,-(DAY(DATEADD(MM,4,GETDATE()))-1),
DATEADD(MM,4,GETDATE())),101))
DECLARE #ENDDATE DATETIME = (SELECT CONVERT(VARCHAR(25),DATEADD(DD,-(DAY(DATEADD(MM,5,GETDATE()))),
DATEADD(MM,5,GETDATE())),101))
SELECT DISTINCT
ISNULL(CONVERT(VARCHAR(3),PP.LOBCD),'CPP') 'LOB_Name_Product'
, POLICYID 'Policy_#'
, P.PRODUCERID 'Agent_#'
, CONVERT(VARCHAR(14),POLICYEFFDT,101) 'Policy_Eff_Date'
, CONVERT(VARCHAR(14),TS.POLICYEXPDT,101) 'Policy_Exp_Date'
, CONVERT(NUMERIC(15,2),TS.TERMPREMAMT) 'Inforce_Prem_Sum'
, REPLACE(CONVERT(CHAR(100),REPLACE(REPLACE(N.INSUREDNM,CHAR(10),' '),CHAR(13),'')),',',' ') AS 'Insured_Name'
, REPLACE(P.PRODUCERNM1TX,',',' ') AS 'Agent_Name'
, PD.PREDSTATECD 'Policy_State'
, REPLACE(II.ADDRLINE1TX ,',',' ') AS 'Insured_Address_1'
, REPLACE(ISNULL(II.ADDRLINE2TX,''),',',' ') AS 'Insured_Address_2'
, II.CITYNM 'Insured_City'
, II.STATECD 'Insured_State'
, CASE WHEN LEN(RTRIM(II.ZIPCD)) > 5 THEN (SUBSTRING(II.ZIPCD,1,5) + '-' + SUBSTRING(II.ZIPCD,6,5))
ELSE II.ZIPCD
END 'Insured_Zip'
, REPLACE(P.PRODUCERADDRLINE1TX,',',' ') AS 'Agent_Address_1'
, REPLACE(ISNULL(P.PRODUCERADDRLINE2TX,''),',',' ') AS 'Agent_Address_2'
, P.PRODUCERCITYNM 'Agent_City'
, P.STATECD 'Agent_State'
, CASE WHEN LEN(RTRIM(P.ZIPCD)) > 5 THEN SUBSTRING(RTRIM(P.ZIPCD),1,5) + '-' + SUBSTRING(RTRIM(P.ZIPCD),6,5)
ELSE P.ZIPCD
END 'Agent_Zip'
, CONVERT(VARCHAR(10), GETDATE(), 101) AS 'Upload_Date'
, 'Open' AS 'Status'
FROM COPOLICYPOINTER PP
JOIN COTRANSACTIONSUMMARY TS ON TS.SYSTEMASSIGNID = PP.SYSTEMASSIGNID
AND TS.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COTRANSACTIONSUMMARY TS2
WHERE TS2.SYSTEMASSIGNID = TS.SYSTEMASSIGNID)
AND TS.TRANSEFFDT = ( SELECT MAX(TRANSEFFDT) FROM COTRANSACTIONSUMMARY TS2
WHERE TS2.SYSTEMASSIGNID = TS.SYSTEMASSIGNID)
JOIN COPOLICYDETAIL PD ON TS.SYSTEMASSIGNID = PD.SYSTEMASSIGNID
AND PD.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COPRODUCER PD2
WHERE PD2.SYSTEMASSIGNID = PD.SYSTEMASSIGNID)
JOIN COPRODUCER P ON P.SYSTEMASSIGNID = TS.SYSTEMASSIGNID
AND P.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COPRODUCER P2
WHERE P2.SYSTEMASSIGNID = P.SYSTEMASSIGNID)
JOIN COINSUREDNAME N ON N.SYSTEMASSIGNID = P.SYSTEMASSIGNID
AND N.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COINSUREDNAME N2
WHERE N2.SYSTEMASSIGNID = N.SYSTEMASSIGNID)
JOIN COINSUREDINFO II ON II.SYSTEMASSIGNID = N.SYSTEMASSIGNID
AND II.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COINSUREDINFO I2
WHERE I2.SYSTEMASSIGNID = II.SYSTEMASSIGNID)
WHERE TS.POLICYEXPDT BETWEEN #STARTDATE AND #ENDDATE
AND PP.CANCEFFDT IS NULL
AND PD.PREDSTATECD IN ('CT', 'RI', 'GA','NH','NY')
ORDER BY POLICYID
The results are linked to a Data Conversion Task.
The Data Conversion Task is then linked to an OLE DB Destination task.
The OLE DB Destination task uses an OLE DB Provider that connects to an Access Database. One of the Tables in this Database is called MasterTable and it has an autonumber field named ID.
I have added screenshots of the entire workflow below.
High-level data workflow
OLE DB Destination connection info to Access Database
I found a way to do what I need so I am posting the answer here in case it helps anyone else. I will summarize the steps below and then post the code. I am using SSIS 2008.
I derived this answer from a related answer I found on this site:
What is the fastest way to insert 100 000 records into an MDB file in C#
Create a data flow task
In the data flow task, create an OLE Db source with your SQL code. This is the query that will give you the results to put into your Access Table.
Once I got the SQL working, I created a data conversion task and converted most of the columns to unicode (except the date columns). Pay attention to the names in the "Output Alias" column of the converter. These are the names you use in the C# script task shown below.
Save the results to a recordset destination. When you create the recordset, the "Component Properties" tab has a field named VariableName. Put a variable there. This variable will hold the results of the SQL query. I named mine "rsSourceTable." This variable is what the C# code will read to get our resultset.
Once you get the Data Flow task working, create a C# Script Task. I created several variables for use with the script task.
The read-only variables:
AccessPath - Holds the path where the Access file is located.
rsSourceTable - the variable that holds the results of our data flow task.
The read/write variables:
MasterTableRowCount - I use this to make it easy to report the number of files inserted in log files and email tasks.
6. The c# code for the script task is shown below. I did not have to add any references to this.
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Data.OleDb;
namespace ST_afd8e3cca5534e51ba5855e82f502e92.csproj
{
[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
OleDbConnection myConnection = new OleDbConnection();
try
{
string accessPath = Dts.Variables["AccessPath"].Value.ToString();
string materTableId = Dts.Variables["MasterTableId"].Value.ToString();
myConnection.ConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + accessPath;
DataTable dt = new DataTable();
OleDbDataAdapter adapter = new OleDbDataAdapter();
adapter.Fill(dt, Dts.Variables["User::rsSourceTable"].Value);
int rowCount = 0;
if (dt.Rows.Count > 0)
{
rowCount = dt.Rows.Count;
Dts.Variables["MasterTableRowCount"].Value.ToString();
myConnection.Open();
//1. When building the INSERT statement, remember to enclose column names in square brackets. This prevents errors because Access allows special characters in column names and OLE DB doesn't
//2. Also remember that the order the column names appear in the INSERT statement is important for the code that adds parameters below.
//3. To prevent an error, the INSERT statement is first constructed with a ? for each parameter. The parameter is replaced with the
// appropriate column name in the for loop below.
string insertString = "INSERT INTO MasterTable ([LOB_Name_Product], [Policy_#], [Policy_State], [Policy_Eff_Date], [Policy_Exp_Date], [Insured_Name], [Insured_Address_1], ";
insertString += "[Insured_Address_2], [Insured_City], [Agent_#], [Agent_Name], [Inforce_Prem_Sum], [Status], [Upload_date], [Insured_Zip], [Insured_State], [Agent_Address_1], [Agent_Address_2], [Agent_City], [Agent_Zip], [Agent_State])";
insertString += " Values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
OleDbCommand cmmd = new OleDbCommand(insertString, myConnection);
if (myConnection.State == ConnectionState.Open)
{
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //LOB_Name_Product
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Policy_#
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Policy_State
cmmd.Parameters.Add("?", OleDbType.DBDate, 10); //Policy_Eff_Date
cmmd.Parameters.Add("?", OleDbType.DBDate, 10); //Policy_Exp_Date
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Name
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Address_1
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Address_2
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_City
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_#
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Name
cmmd.Parameters.Add("?", OleDbType.Currency, 255); //Inforce_Prem_Sum
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Status
cmmd.Parameters.Add("?", OleDbType.Date, 10); //Upload_date
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Zip
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_State
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Address_1
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Address_2
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_City
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Zip
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_State
cmmd.Prepare();
OleDbTransaction trans = myConnection.BeginTransaction();
cmmd.Transaction = trans;
foreach(DataRow dr in dt.Rows)
{
cmmd.Parameters[0].Value = dr["Copy of LOB_Name_Product"];
cmmd.Parameters[1].Value = dr["Copy of Policy_#"];
cmmd.Parameters[2].Value = dr["Copy of Policy_State"];
cmmd.Parameters[3].Value = dr["Policy_Eff_Date"];
cmmd.Parameters[4].Value = dr["Policy_Exp_Date"];
cmmd.Parameters[5].Value = dr["Copy of Insured_Name"];
cmmd.Parameters[6].Value = dr["Copy of Insured_Address_1"];
cmmd.Parameters[7].Value = dr["Copy of Insured_Address_2"];
cmmd.Parameters[8].Value = dr["Copy of Insured_City"];
cmmd.Parameters[9].Value = dr["Copy of Agent_#"];
cmmd.Parameters[10].Value = dr["Copy of Agent_Name"];
cmmd.Parameters[11].Value = dr["Copy of Inforce_Prem_Sum"];
cmmd.Parameters[12].Value = "Open";
cmmd.Parameters[13].Value = DateTime.Today.ToString("d");
cmmd.Parameters[14].Value = dr["Copy of Insured_Zip"];
cmmd.Parameters[15].Value = dr["Copy of Insured_State"];
cmmd.Parameters[16].Value = dr["Copy of Agent_Address_1"];
cmmd.Parameters[17].Value = dr["Copy of Agent_Address_2"];
cmmd.Parameters[18].Value = dr["Copy of Agent_City"];
cmmd.Parameters[19].Value = dr["Copy of Agent_Zip"];
cmmd.Parameters[20].Value = dr["Copy of Agent_State"];
cmmd.ExecuteNonQuery();
}
trans.Commit();
myConnection.Close();
Dts.TaskResult = (int)ScriptResults.Success; //add logging here for successful operation
}
else
Dts.TaskResult = (int)ScriptResults.Failure;
}
else
Dts.TaskResult = (int)ScriptResults.Success; //add logging here for no records
}
catch (OleDbException oleEx)
{
myConnection.Close();
Dts.TaskResult = (int)ScriptResults.Failure; //add logging here for unable to connect
}
catch (Exception ex)
{
myConnection.Close();
Dts.TaskResult = (int)ScriptResults.Failure; //add logging here for any other error
}
}
}
}
I am using the following code to export data from excel to Sql server database. What is going on with this code is, its importing complete data into the database.
[HttpPost]
public ActionResult Importexcel()
{
if (Request.Files["FileUpload1"].ContentLength > 0)
{
string extension = System.IO.Path.GetExtension(Request.Files["FileUpload1"].FileName);
string path1 = string.Format("{0}/{1}", Server.MapPath("~/Content/UploadedFolder"), Request.Files["FileUpload1"].FileName);
if (System.IO.File.Exists(path1))
System.IO.File.Delete(path1);
Request.Files["FileUpload1"].SaveAs(path1);
string sqlConnectionString = #"Data Source=xyz-101\SQLEXPRESS;Database=PracDB;Trusted_Connection=true;Persist Security Info=True";
string excelConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + path1 + ";Extended Properties=Excel 12.0;Persist Security Info=False";
OleDbConnection excelConnection = new OleDbConnection(excelConnectionString);
OleDbCommand cmd = new OleDbCommand("Select [ID],[Name],[Designation] from [Sheet1$]", excelConnection);
excelConnection.Open();
OleDbDataReader dReader;
dReader = cmd.ExecuteReader();
SqlBulkCopy sqlBulk = new SqlBulkCopy(sqlConnectionString);
sqlBulk.DestinationTableName = "Excel_Table";
sqlBulk.WriteToServer(dReader);
excelConnection.Close();
}
return RedirectToAction("Index");
}
How to check that any particular record already exist in the database or not. If not then Insert the record into the databse else it should not.
Thanks in Advance !
Since your target is SQL Server, you can use this to your advantage.
What I would do is read the data from the excel into a DataTable (instead of using a DataReader you can use a DataAdapter), Send that DataTable to a stored procedure in the SQL server, and handle the insert there. In order to sand a data table to a stored procedure you first need to create a Table-value user defined type in your sql server, like this:
CREATE TYPE MyType AS TABLE
(
Id int,
Name varchar(20), -- use whatever length best fitted to your data
Designation varchar(max) -- use whatever length best fitted to your data
)
Then you can write a simple stored procedure with an argument of this type:
CREATE PROCEDURE InsertDataFromExcel
(
#ExcelData dbo.MyType readonly -- Note: readonly is a required!
)
AS
INSERT INTO MyTable(Id, Name, Designation)
SELECT a.Id, a.Name, a.Designation
FROM #ExcelData a LEFT JOIN
MyTable b ON(a.Id = b.Id)
WHERE b.Id IS NULL -- this condition with the left join ensures you only select records that has different id values then the records already in your database
in order to send this parameter to the stored procedure from your c# code you will have to use a SqlCommand object and add the DataTable as a parameter, something like this:
using(SqlConnection Con = new SqlConnection(sqlConnectionString))
{
using(SqlCommand InsertCommand = new SqlCommand("InsertDataFromExcel", Con))
{
SqlParameter MyParam = new SqlParameter("#ExcelData", SqlDBType.Structured);
MyParam.Value = MyDataTable; // this is the data table from the
InsertCommand.Parameters.Add(MyParam);
Con.Open();
InsertCommand.ExecuteNoQuery();
Con.Close();
}
}
Note: Code was writen directly here, some errors might be found.
I have this scenario :
I have a user table that has two column: Name and family.
what I want is that when user search a keyword ,for example 'name1 family1' ,all the result that have name: name1 and family: family1.
this is only one example and and my scenario is very complicated.(I want search name,lastname in one tables and phone number,address,so on related to current table).:(
is this search like google in an sql database possible? how? can I use fulltextsearch for this? how?
thank you.
Yes, it is possible. Create a view with schemabinding on your table with two columns: The ID (as PK) and a combined string of all fields you want to include in the full text search.
create view [dbo].[View_FamilyData]
with schemabinding as
select ID,
LastName+' '+FirstName+' '+Phone+' '+Address as SearchText
from dbo.YourTable
Then put a fulltext index on the the column SearchText.
Finally, play around with the various possiblities of CONTAINS,FREETEXT, CONTAINSTABLE and FREETEXTTABLE.
select T.*
from YourTable T
join View_FamilyData v
on v on T.ID = v.ID
where contains (SearchText,'Smith 12345')
private void tbautocomplete_TextChanged(object sender, EventArgs e)
{
AutoCompleteStringCollection namecollection = new AutoCompleteStringCollection();
SqlConnection con = new SqlConnection(#"Data Source=88888;Initial Catalog=contrynames;Integrated Security=True");
SqlCommand cmd = new SqlCommand();
cmd.Connection = con;
cmd.CommandType = CommandType.Text;
cmd.CommandText = "SELECT distinct(person_Firstname+''+person_Lastname) AS name FROM persondetails WHERE name Like '%'+#name+'%'";
con.Open();
SqlDataReader rea = cmd.ExecuteReader();
if (rea.HasRows == true)
{
while (rea.Read())
namecollection.Add(rea["name"].ToString());
}
rea.Close();
tbautocomplete.AutoCompleteMode = AutoCompleteMode.Suggest;
tbautocomplete.AutoCompleteSource = AutoCompleteSource.CustomSource;
tbautocomplete.AutoCompleteCustomSource = namecollection;
SQL Server CE 4 (SQL Server Compact Edition 4.0) is not news already (If it is, you could read this article)
But it is very interesting to see SQL Server CE 4 performance comparison to other databases.
Especially with:
SQLite
SQL Server (1)
SQL Server Express *
maybe Firebird
(1) for applications where functionality is comparable.
Unfortunately there are not so much links about the subject that google provides right now. Actually I was unable to find any (for proper SQL CE version).
If one could find or share such information lets collect it here for future humanity.
In my opinion, it is incorrect to compare the embedded database (like SQL CE) versus server-side relational database (like all the rest, except for SQLite and the Embedded version of Firebird).
The main difference between them is that the general-purpose server-side relational databases (like MS SQL, MySQL, Firebird Classic and SuperServer etc.) are installed as an independent service and run outside of the scope of your main application. That is why they can perform much better because of the intrinsic support for multi-core and multi-CPU architectures, using OS features like pre-caching, VSS etc to increase the throughput in case of intensive database operation and can claim as much memory as your OS can provide for a single service/application. It also means that the performance indicators for them are more or less independent from your application, but largely depend upon your hardware. In this respect I would say that the server versions of any database are always more performance compared to the embedded ones.
SQL CE (along with Firebird Embedded, SQLite, TurboSQL and some other) are embedded DB engines, meaning that the complete database is packed into a single (or maximally 2) DLL-files that are distributed together with your application. Due to the evident size limitations (would you like to have to distribute a 30 MB DLL together with your 2-3 MB long application?) they also run directly in the context of your application and the total memory and performance for data access operations are shared with other parts of your application -- that regards both available memory, CPU time, disk throughput etc. Having a computation-intensive threads running in parallel with your data access thread might lead to dramatic decrease of your database performance.
Due to the different areas of application these databases have different palette of options: server-db provide extensive user and right management, support for views and stored procedures, whereas embedded database normally lack any support for users and rights management and have limited support for views and stored procedures (latter ones lose the majority of their benefits of running on server side). Data throughput is a usual bottlenecks of RDBMS, server versions are usually installed on striped RAID volumes, whereas embedded DB are often memory-oriented (try to keep all the actual data in the memory) and minimize the data storage access operations.
So, what would make sense probably is to compare different embedded RDBMS for .Net for their performance, like MS SQL CE 4.0, SQLite, Firebird Embedded, TurboSQL. I wouldn't expect drastic differences during usual non-peak operation, whereas some database may provide better support for large BLOBs due to better integration with OS.
-- update --
I have to take back my last words, for my quick implementation shows very interesting results.
I wrote a short console application to test both data providers, here is the source code for you if you want to experiment with them on your own.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.SQLite;
using System.Data.SqlServerCe;
using System.Data.Common;
namespace TestSQL
{
class Program
{
const int NUMBER_OF_TESTS = 1000;
private static string create_table;
private static string create_table_sqlce = "CREATE TABLE Test ( id integer not null identity primary key, textdata nvarchar(500));";
private static string create_table_sqlite = "CREATE TABLE Test ( id integer not null primary key, textdata nvarchar(500));";
private static string drop_table = "DROP TABLE Test";
private static string insert_data = "INSERT INTO Test (textdata) VALUES ('{0}');";
private static string read_data = "SELECT textdata FROM Test WHERE id = {0}";
private static string update_data = "UPDATE Test SET textdata = '{1}' WHERE id = {0}";
private static string delete_data = "DELETE FROM Test WHERE id = {0}";
static Action<DbConnection> ACreateTable = (a) => CreateTable(a);
static Action<DbConnection> ATestWrite = (a) => TestWrite(a, NUMBER_OF_TESTS);
static Action<DbConnection> ATestRead = (a) => TestRead(a, NUMBER_OF_TESTS);
static Action<DbConnection> ATestUpdate = (a) => TestUpdate(a, NUMBER_OF_TESTS);
static Action<DbConnection> ATestDelete = (a) => TestDelete(a, NUMBER_OF_TESTS);
static Action<DbConnection> ADropTable = (a) => DropTable(a);
static Func<Action<DbConnection>,DbConnection, TimeSpan> MeasureExecTime = (a,b) => { var start = DateTime.Now; a(b); var finish = DateTime.Now; return finish - start; };
static Action<string, TimeSpan> AMeasureAndOutput = (a, b) => Console.WriteLine(a, b.TotalMilliseconds);
static void Main(string[] args)
{
// opening databases
SQLiteConnection.CreateFile("sqlite.db");
SQLiteConnection sqliteconnect = new SQLiteConnection("Data Source=sqlite.db");
SqlCeConnection sqlceconnect = new SqlCeConnection("Data Source=sqlce.sdf");
sqlceconnect.Open();
sqliteconnect.Open();
Console.WriteLine("=Testing CRUD performance of embedded DBs=");
Console.WriteLine(" => Samplesize: {0}", NUMBER_OF_TESTS);
create_table = create_table_sqlite;
Console.WriteLine("==Testing SQLite==");
DoMeasures(sqliteconnect);
create_table = create_table_sqlce;
Console.WriteLine("==Testing SQL CE 4.0==");
DoMeasures(sqlceconnect);
Console.ReadKey();
}
static void DoMeasures(DbConnection con)
{
AMeasureAndOutput("Creating table: {0} ms", MeasureExecTime(ACreateTable, con));
AMeasureAndOutput("Writing data: {0} ms", MeasureExecTime(ATestWrite, con));
AMeasureAndOutput("Updating data: {0} ms", MeasureExecTime(ATestUpdate, con));
AMeasureAndOutput("Reading data: {0} ms", MeasureExecTime(ATestRead, con));
AMeasureAndOutput("Deleting data: {0} ms", MeasureExecTime(ATestDelete, con));
AMeasureAndOutput("Dropping table: {0} ms", MeasureExecTime(ADropTable, con));
}
static void CreateTable(DbConnection con)
{
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = create_table;
sqlcmd.ExecuteNonQuery();
}
static void TestWrite(DbConnection con, int num)
{
for (; num-- > 0; )
{
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = string.Format(insert_data,Guid.NewGuid().ToString());
sqlcmd.ExecuteNonQuery();
}
}
static void TestRead(DbConnection con, int num)
{
Random rnd = new Random(DateTime.Now.Millisecond);
for (var max = num; max-- > 0; )
{
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = string.Format(read_data, rnd.Next(1,num-1));
sqlcmd.ExecuteNonQuery();
}
}
static void TestUpdate(DbConnection con, int num)
{
Random rnd = new Random(DateTime.Now.Millisecond);
for (var max = num; max-- > 0; )
{
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = string.Format(update_data, rnd.Next(1, num - 1), Guid.NewGuid().ToString());
sqlcmd.ExecuteNonQuery();
}
}
static void TestDelete(DbConnection con, int num)
{
Random rnd = new Random(DateTime.Now.Millisecond);
var order = Enumerable.Range(1, num).ToArray<int>();
Action<int[], int, int> swap = (arr, a, b) => { int c = arr[a]; arr[a] = arr[b]; arr[b] = c; };
// shuffling the array
for (var max=num; max-- > 0; ) swap(order, rnd.Next(0, num - 1), rnd.Next(0, num - 1));
foreach(int index in order)
{
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = string.Format(delete_data, index);
sqlcmd.ExecuteNonQuery();
}
}
static void DropTable(DbConnection con)
{
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = drop_table;
sqlcmd.ExecuteNonQuery();
}
}
}
Necessary disclaimer:
I got these results on my machine: Dell Precision WorkStation T7400 equipped with 2 Intel Xeon E5420 CPUs and 8GB of RAM, running 64bit Win7 Enterprise.
I used the default settings for both DBs with connection string "Data Source=database_file_name".
I used the latest versions of both SQL CE 4.0 and SQLite/System.Data.SQLite (from today, June 3rd 2011).
Here are the results for two different samples:
> =Testing CRUD performance of embedded DBs=
> => Samplesize: 200
> ==Testing SQLite==
> Creating table: 396.0396 ms
> Writing data: 22189.2187 ms
> Updating data: 23591.3589 ms
> Reading data: 21.0021 ms
> Deleting data: 20963.0961 ms
> Dropping table: 85.0085 ms
> ==Testing SQL CE 4.0==
> Creating table: 16.0016 ms
> Writing data: 25.0025 ms
> Updating data: 56.0056 ms
> Reading data: 28.0028 ms
> Deleting data: 53.0053 ms
> Dropping table: 11.0011 ms
... and a bigger sample:
=Testing CRUD performance of embedded DBs=
=> Samplesize: 1000
==Testing SQLite==
Creating table: 93.0093 ms
Writing data: 116632.6621 ms
Updating data: 104967.4957 ms
Reading data: 134.0134 ms
Deleting data: 107666.7656 ms
Dropping table: 83.0083 ms
==Testing SQL CE 4.0==
Creating table: 16.0016 ms
Writing data: 128.0128 ms
Updating data: 307.0307 ms
Reading data: 164.0164 ms
Deleting data: 306.0306 ms
Dropping table: 13.0013 ms
So, as you can see, any writing operations (create, update, delete) require almost 1000x more time in SQLite compared to SQLCE. It does not necessarily reflect the general bad performance of this database and might be due to the following:
The data provider I use for SQLite is the System.Data.SQLite, that is a mixed assembly containing both managed and unmanaged code (SQLite is originally written completely in C and the DLL only provides bindings). Probably P/Invoke and data marshaling eats up a good piece of the operation time.
Most likely SQLCE 4.0 caches all the data in memory by default, whereas SQLite flushes most of the data changes directly to the disk storage every time the change happens. One can supply hundreds of parameters for both databases via connection string and tune them appropriately.
I used a series of single queries to test the DB. At least SQLCE supports bulk operations via special .Net classes that would be better suited here. If SQLite supports them too (sorry, I am not an expert here and my quick search yielded nothing promising) it would be nice to compare them as well.
I have observed many problems with SQLite on x64 machines (using the same .net adapter): from data connection being closed unexpectedly to database file corruption. I presume there is some stability problems either with the data adapter or with the library itself.
Here is my freshly baked article about the benchmarking on CodeProject webpage:
Benchmarking the performance of embedded DB for .Net: SQL CE 4.0 vs SQLite
(the article has a pending status now, you need to be logged-in on CodeProject to access its content)
P.S.: I mistakenly marked my previous answer as a community wiki entry and won't get any reputation for it. This encouraged me to write the article for Code Project on this topic, with a somewhat optimized code, more additional information about embedded dbs and statistical analysis of the results. So, please, vote this answer up if you like the article and my second answer here.
Because I'm having a real hard time with Alaudo's tests, test results, and ultimately with his conclusion, I went ahead and played around a bit with his program and came up with a modified version.
It tests each of the following 10 times and outputs the average times:
sqlite without transactions, using default jounal_mode
sqlite with transactions, using default journal_mode
sqlite without transactions, using WAL jounal_mode
sqlite with transactions, using WAL journal_mode
sqlce without transactions
sqlce with transactions
Here is the program (it's a class actually):
using System;
using System.Collections.Generic;
using System.Data.Common;
using System.Data.SqlServerCe;
using System.Data.SQLite;
using System.Diagnostics;
using System.IO;
using System.Linq;
class SqliteAndSqlceSpeedTesting
{
class Results
{
public string test_details;
public long create_table_time, insert_time, update_time, select_time, delete_time, drop_table_time;
}
enum DbType { Sqlite, Sqlce };
const int NUMBER_OF_TESTS = 200;
const string create_table_sqlite = "CREATE TABLE Test (id integer not null primary key, textdata nvarchar(500));";
const string create_table_sqlce = "CREATE TABLE Test (id integer not null identity primary key, textdata nvarchar(500));";
const string drop_table = "DROP TABLE Test";
const string insert_data = "INSERT INTO Test (textdata) VALUES ('{0}');";
const string read_data = "SELECT textdata FROM Test WHERE id = {0}";
const string update_data = "UPDATE Test SET textdata = '{1}' WHERE id = {0}";
const string delete_data = "DELETE FROM Test WHERE id = {0}";
public static void RunTests()
{
List<Results> results_list = new List<Results>();
for (int i = 0; i < 10; i++) {
results_list.Add(RunTest(DbType.Sqlite, false, false));
results_list.Add(RunTest(DbType.Sqlite, false, true));
results_list.Add(RunTest(DbType.Sqlite, true, false));
results_list.Add(RunTest(DbType.Sqlite, true, true));
results_list.Add(RunTest(DbType.Sqlce, false));
results_list.Add(RunTest(DbType.Sqlce, true));
}
foreach (var test_detail in results_list.GroupBy(r => r.test_details)) {
Console.WriteLine(test_detail.Key);
Console.WriteLine("Creating table: {0} ms", test_detail.Average(r => r.create_table_time));
Console.WriteLine("Inserting data: {0} ms", test_detail.Average(r => r.insert_time));
Console.WriteLine("Updating data: {0} ms", test_detail.Average(r => r.update_time));
Console.WriteLine("Selecting data: {0} ms", test_detail.Average(r => r.select_time));
Console.WriteLine("Deleting data: {0} ms", test_detail.Average(r => r.delete_time));
Console.WriteLine("Dropping table: {0} ms", test_detail.Average(r => r.drop_table_time));
Console.WriteLine();
}
}
static Results RunTest(DbType db_type, bool use_trx, bool use_wal = false)
{
DbConnection conn = null;
if (db_type == DbType.Sqlite)
conn = GetConnectionSqlite(use_wal);
else
conn = GetConnectionSqlce();
Results results = new Results();
results.test_details = string.Format("Testing: {0}, transactions: {1}, WAL: {2}", db_type, use_trx, use_wal);
results.create_table_time = CreateTable(conn, db_type);
results.insert_time = InsertTime(conn, use_trx);
results.update_time = UpdateTime(conn, use_trx);
results.select_time = SelectTime(conn, use_trx);
results.delete_time = DeleteTime(conn, use_trx);
results.drop_table_time = DropTableTime(conn);
conn.Close();
return results;
}
static DbConnection GetConnectionSqlite(bool use_wal)
{
SQLiteConnection conn = new SQLiteConnection("Data Source=sqlite.db");
if (!File.Exists(conn.Database))
SQLiteConnection.CreateFile("sqlite.db");
conn.Open();
if (use_wal) {
var command = conn.CreateCommand();
command.CommandText = "PRAGMA journal_mode=WAL";
command.ExecuteNonQuery();
}
return conn;
}
static DbConnection GetConnectionSqlce()
{
SqlCeConnection conn = new SqlCeConnection("Data Source=sqlce.sdf");
if (!File.Exists(conn.Database))
using (var sqlCeEngine = new SqlCeEngine("Data Source=sqlce.sdf"))
sqlCeEngine.CreateDatabase();
conn.Open();
return conn;
}
static long CreateTable(DbConnection con, DbType db_type)
{
Stopwatch sw = Stopwatch.StartNew();
var sqlcmd = con.CreateCommand();
if (db_type == DbType.Sqlite)
sqlcmd.CommandText = create_table_sqlite;
else
sqlcmd.CommandText = create_table_sqlce;
sqlcmd.ExecuteNonQuery();
return sw.ElapsedMilliseconds;
}
static long DropTableTime(DbConnection con)
{
Stopwatch sw = Stopwatch.StartNew();
var sqlcmd = con.CreateCommand();
sqlcmd.CommandText = drop_table;
sqlcmd.ExecuteNonQuery();
return sw.ElapsedMilliseconds;
}
static long InsertTime(DbConnection con, bool use_trx)
{
Stopwatch sw = Stopwatch.StartNew();
var sqlcmd = con.CreateCommand();
DbTransaction trx = null;
if (use_trx) {
trx = con.BeginTransaction();
sqlcmd.Transaction = trx;
}
for (int i = 0; i < NUMBER_OF_TESTS; i++) {
sqlcmd.CommandText = string.Format(insert_data, Guid.NewGuid().ToString());
sqlcmd.ExecuteNonQuery();
}
if (trx != null)
trx.Commit();
return sw.ElapsedMilliseconds;
}
static long SelectTime(DbConnection con, bool use_trx)
{
Stopwatch sw = Stopwatch.StartNew();
var sqlcmd = con.CreateCommand();
DbTransaction trx = null;
if (use_trx) {
trx = con.BeginTransaction();
sqlcmd.Transaction = trx;
}
Random rnd = new Random(DateTime.Now.Millisecond);
for (var max = NUMBER_OF_TESTS; max-- > 0; ) {
sqlcmd.CommandText = string.Format(read_data, rnd.Next(1, NUMBER_OF_TESTS - 1));
sqlcmd.ExecuteNonQuery();
}
if (trx != null)
trx.Commit();
return sw.ElapsedMilliseconds;
}
static long UpdateTime(DbConnection con, bool use_trx)
{
Stopwatch sw = Stopwatch.StartNew();
var sqlcmd = con.CreateCommand();
DbTransaction trx = null;
if (use_trx) {
trx = con.BeginTransaction();
sqlcmd.Transaction = trx;
}
Random rnd = new Random(DateTime.Now.Millisecond);
for (var max = NUMBER_OF_TESTS; max-- > 0; ) {
sqlcmd.CommandText = string.Format(update_data, rnd.Next(1, NUMBER_OF_TESTS - 1), Guid.NewGuid().ToString());
sqlcmd.ExecuteNonQuery();
}
if (trx != null)
trx.Commit();
return sw.ElapsedMilliseconds;
}
static long DeleteTime(DbConnection con, bool use_trx)
{
Stopwatch sw = Stopwatch.StartNew();
Random rnd = new Random(DateTime.Now.Millisecond);
var order = Enumerable.Range(1, NUMBER_OF_TESTS).ToArray<int>();
Action<int[], int, int> swap = (arr, a, b) => { int c = arr[a]; arr[a] = arr[b]; arr[b] = c; };
var sqlcmd = con.CreateCommand();
DbTransaction trx = null;
if (use_trx) {
trx = con.BeginTransaction();
sqlcmd.Transaction = trx;
}
// shuffling the array
for (var max = NUMBER_OF_TESTS; max-- > 0; ) swap(order, rnd.Next(0, NUMBER_OF_TESTS - 1), rnd.Next(0, NUMBER_OF_TESTS - 1));
foreach (int index in order) {
sqlcmd.CommandText = string.Format(delete_data, index);
sqlcmd.ExecuteNonQuery();
}
if (trx != null)
trx.Commit();
return sw.ElapsedMilliseconds;
}
}
Here are the numbers I get:
Testing: Sqlite, transactions: False, WAL: False
Creating table: 24.4 ms
Inserting data: 3084.7 ms
Updating data: 3147.8 ms
Selecting data: 30 ms
Deleting data: 3182.6 ms
Dropping table: 14.5 ms
Testing: Sqlite, transactions: False, WAL: True
Creating table: 2.3 ms
Inserting data: 14 ms
Updating data: 12.2 ms
Selecting data: 6.8 ms
Deleting data: 11.7 ms
Dropping table: 0 ms
Testing: Sqlite, transactions: True, WAL: False
Creating table: 13.5 ms
Inserting data: 20.3 ms
Updating data: 24.5 ms
Selecting data: 7.8 ms
Deleting data: 22.3 ms
Dropping table: 16.7 ms
Testing: Sqlite, transactions: True, WAL: True
Creating table: 3.2 ms
Inserting data: 5.8 ms
Updating data: 4.9 ms
Selecting data: 4.4 ms
Deleting data: 3.8 ms
Dropping table: 0 ms
Testing: Sqlce, transactions: False, WAL: False
Creating table: 2.8 ms
Inserting data: 24.4 ms
Updating data: 42.8 ms
Selecting data: 30.4 ms
Deleting data: 38.3 ms
Dropping table: 3.3 ms
Testing: Sqlce, transactions: True, WAL: False
Creating table: 2.1 ms
Inserting data: 24.6 ms
Updating data: 44.2 ms
Selecting data: 32 ms
Deleting data: 37.8 ms
Dropping table: 3.2 ms
~3 seconds for 200 inserts or updates using sqlite might still seem a little high, but at least it's more reasonable than 23 seconds. Conversely, one might be worried how SqlCe takes too little time to complete the same 200 inserts or update, especially since there seems to be no real speed difference between having each SQL query in individual transactions, or together in one transaction. I don't know enough about SqlCe to explain this, but it worries me. Would it mean that when .Commit() returns, you are not assured that the changes are actually written to disk?
I have recently worked on a project using SQL CE 4 and NHibernate and I found the performance to be really good. With SQL CE 4 we were able to insert 8000 records in a second. With Oracle over the network we were only able to insert 100 records per second even batch-size and seqhilo approaches were used.
I did not test it but looking at some of the performance reports for NoSQL products for .NET, SQL CE 4 seems to be one of the best solutions for stand-alone applications.
Just avoid using Identity columns, we noticed the performance was 40 times better if they are not used. The same 8000 records were taking 40 secs to insert when an Identity column was used as PK.
I need to change some primary keys from non-clustered to clustered but I can't drop the constraint because it is referenced from other foreign keys.
How can I find the tables that reference a primary key in the parent table as part of a foreign relation without looping through all tables in the DB? I need to disable the constraints on those, change the PK and re-enable.
Update:
I do not want to use plain SQL to do this but SMO only.
Marc, I know about ForeignKeys by I need something like:
table.PrimaryKey.ForeignKeys (i.e. which tables are referencing my table's primary key)
I just want to avoid looping through all the tables in the database and check the ForeignKeys property on each and every one of them to see if any of them reference my table.(not scalable)
Ok I think I found it.
table.Columns[0].EnumForeignKeys()
or directly
table.EnumForeignKeys()
I was expecting a property instead of a function. I am pretty sure behind the scenes it does what cmsjr suggested.
Using SMO, you could do this:
using Microsoft.SqlServer.Management.Smo;
Server localServer = new Server("your server name");
Database dasecoDB = localServer.Databases["your database name"];
Table table = dasecoDB.Tables["your table name"];
foreach(ForeignKey fk in table.ForeignKeys)
{
Console.WriteLine("Foreign key {0} references table {1} and key {2}", fk.Name, fk.ReferencedTable, fk.ReferencedKey);
}
Marc
This query should work, and could be executed using Database.ExecuteWithResults
Select fk.Table_Name from
INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS C
INNER JOIN
INFORMATION_SCHEMA.TABLE_CONSTRAINTS FK
ON C.CONSTRAINT_NAME = FK.CONSTRAINT_NAME
INNER JOIN
INFORMATION_SCHEMA.TABLE_CONSTRAINTS PK
ON C.UNIQUE_CONSTRAINT_NAME = PK.CONSTRAINT_NAME
where PK.Table_Name = 'SomeTable'
e.g.
SqlConnection sqlConnection =
new SqlConnection(#"Integrated Security=SSPI; Data Source=SomeInstance");
Server server = new Server(serverConnection);
Database db = server.Databases["somedatabase"];
DataSet ds = db.ExecuteWithResults(thesqlabove);
You could use the INFORMATION_SCHEMA Views.
INFORMATION_SCHEMA.TABLE_CONSTRAINTS will give you the names of the primary keys on that table.
SELECT CONSTRAINT_NAME FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS WHERE TABLE_NAME = #TableName
Given the primary key names you can get the referential constraints that use those keys from INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS
And then the table names by querying INFORMATION_SCHEMA.CONSTRAINT_TABLE_USAGE
Not SMO as such, but given the above you should be able to put together a query that will list the constraints you need to disable.
It doesn't work for me.
Consider the following relations:
Table1 --> master table;
Table2 --> slave table;
Table2.Table1_ID is a foreign key of Table1.ID
Table1.EnumForeignKeys() return null.
Instead I tried with success the DependencyWalker object. The following code list all the tables which dipend from a given collection of tables.
DependencyWalker w = new DependencyWalker(db.Parent);
DependencyTree tree = w.DiscoverDependencies(urns,false);
DependencyCollection depends = w.WalkDependencies(tree);
foreach (DependencyCollectionNode dcn in depends)
{
if (dcn.Urn.Type == "Table")
{
dcn.Urn.GetNameForType("Table");
Console.WriteLine(dcn.Urn.GetNameForType("Table"));
}
}
where "urns" is a collection of table.Urn.
You will have to travel through dependency tree.
Following is the script which use the SMO to generate Create table and insert script.
**
**ServerConnection conn = new ServerConnection( GetConnection() );
Server server = new Server( conn );
Database db = server.Databases[ mDestinationDatabase ];
// Create database script
StringBuilder dbScript = new StringBuilder();
ScriptingOptions dbCreateOptions = new ScriptingOptions();
dbCreateOptions.DriAll = true;
dbCreateOptions.NoCollation = true;
StringCollection coll = db.Script( dbCreateOptions );
foreach( string str in coll )
{
dbScript.Append( str );
dbScript.Append( Environment.NewLine );
}
sqlInsertCommands = dbScript.ToString();
// Create dependency tree
DependencyWalker w = new DependencyWalker(db.Parent);
UrnCollection urnCollection = new UrnCollection();
DataTable table = db.EnumObjects( DatabaseObjectTypes.Table );
string tableName = string.Empty;
foreach( DataRow row in table.Rows )
{
urnCollection.Add( new Urn( ( string )row[ "Urn" ] ) );
}
DependencyTree tree = w.DiscoverDependencies( urnCollection, true );
DependencyCollection depends = w.WalkDependencies(tree);
// walk through the dependency tree and for each table generate create and insert scripts
foreach (DependencyCollectionNode dcn in depends)
{
if (dcn.Urn.Type == "Table")
{
tableName = dcn.Urn.GetNameForType( "Table" );
DataTable dataTableWithData = GetTableWithData( tableName);
ArrayList columnList = new ArrayList();
foreach(DataColumn dataColumn in dataTableWithData.Columns)
{
columnList.Add( dataColumn.ColumnName );
}
sqlInsertCommands = sqlInsertCommands + Environment.NewLine + Environment.NewLine
+ GetCreateTableScript(tableName )
+ Environment.NewLine + Environment.NewLine
+ BuildInsertSQL( columnList, dataTableWithData, tableName );
}
}**
**