I am creating a new SSIS 2008 ETL report that will read data from a SQL Server and append it to an ACCESS 2010 database table. How can I append records to an existing access table when there is an identity column in the Access table?
My current solution has an OLE DB source with SQL that reads the data. It connects to a data conversion task and then that connects to an OLE DB destination task. The identity field in the access table is named "Id" and it is an autonumber field.
I am able to insert the values using an OLE DB destination when the Access table is empty. The problem has to do with the identity field and when there are already records in the table.
So, the problem is that my Id column always starts at 1, which means I get an error as the engine thinks I am trying to insert a duplicate key.
I created a SQL Task that reads the max ID from the table and stores it into a variable, but now I am stuck trying to figure out how to use this value in my data flow export process.
Will someone please show me how to insert data into an Access DB when there are already records in the table?
Edited for clarity. I have added the SQL that I use to obtain the desired SQL records below. Note that I am NOT including the ID identity field in my SQL:
DECLARE #STARTDATE DATETIME = (SELECT CONVERT(VARCHAR(25),DATEADD(DD,-(DAY(DATEADD(MM,4,GETDATE()))-1),
DATEADD(MM,4,GETDATE())),101))
DECLARE #ENDDATE DATETIME = (SELECT CONVERT(VARCHAR(25),DATEADD(DD,-(DAY(DATEADD(MM,5,GETDATE()))),
DATEADD(MM,5,GETDATE())),101))
SELECT DISTINCT
ISNULL(CONVERT(VARCHAR(3),PP.LOBCD),'CPP') 'LOB_Name_Product'
, POLICYID 'Policy_#'
, P.PRODUCERID 'Agent_#'
, CONVERT(VARCHAR(14),POLICYEFFDT,101) 'Policy_Eff_Date'
, CONVERT(VARCHAR(14),TS.POLICYEXPDT,101) 'Policy_Exp_Date'
, CONVERT(NUMERIC(15,2),TS.TERMPREMAMT) 'Inforce_Prem_Sum'
, REPLACE(CONVERT(CHAR(100),REPLACE(REPLACE(N.INSUREDNM,CHAR(10),' '),CHAR(13),'')),',',' ') AS 'Insured_Name'
, REPLACE(P.PRODUCERNM1TX,',',' ') AS 'Agent_Name'
, PD.PREDSTATECD 'Policy_State'
, REPLACE(II.ADDRLINE1TX ,',',' ') AS 'Insured_Address_1'
, REPLACE(ISNULL(II.ADDRLINE2TX,''),',',' ') AS 'Insured_Address_2'
, II.CITYNM 'Insured_City'
, II.STATECD 'Insured_State'
, CASE WHEN LEN(RTRIM(II.ZIPCD)) > 5 THEN (SUBSTRING(II.ZIPCD,1,5) + '-' + SUBSTRING(II.ZIPCD,6,5))
ELSE II.ZIPCD
END 'Insured_Zip'
, REPLACE(P.PRODUCERADDRLINE1TX,',',' ') AS 'Agent_Address_1'
, REPLACE(ISNULL(P.PRODUCERADDRLINE2TX,''),',',' ') AS 'Agent_Address_2'
, P.PRODUCERCITYNM 'Agent_City'
, P.STATECD 'Agent_State'
, CASE WHEN LEN(RTRIM(P.ZIPCD)) > 5 THEN SUBSTRING(RTRIM(P.ZIPCD),1,5) + '-' + SUBSTRING(RTRIM(P.ZIPCD),6,5)
ELSE P.ZIPCD
END 'Agent_Zip'
, CONVERT(VARCHAR(10), GETDATE(), 101) AS 'Upload_Date'
, 'Open' AS 'Status'
FROM COPOLICYPOINTER PP
JOIN COTRANSACTIONSUMMARY TS ON TS.SYSTEMASSIGNID = PP.SYSTEMASSIGNID
AND TS.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COTRANSACTIONSUMMARY TS2
WHERE TS2.SYSTEMASSIGNID = TS.SYSTEMASSIGNID)
AND TS.TRANSEFFDT = ( SELECT MAX(TRANSEFFDT) FROM COTRANSACTIONSUMMARY TS2
WHERE TS2.SYSTEMASSIGNID = TS.SYSTEMASSIGNID)
JOIN COPOLICYDETAIL PD ON TS.SYSTEMASSIGNID = PD.SYSTEMASSIGNID
AND PD.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COPRODUCER PD2
WHERE PD2.SYSTEMASSIGNID = PD.SYSTEMASSIGNID)
JOIN COPRODUCER P ON P.SYSTEMASSIGNID = TS.SYSTEMASSIGNID
AND P.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COPRODUCER P2
WHERE P2.SYSTEMASSIGNID = P.SYSTEMASSIGNID)
JOIN COINSUREDNAME N ON N.SYSTEMASSIGNID = P.SYSTEMASSIGNID
AND N.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COINSUREDNAME N2
WHERE N2.SYSTEMASSIGNID = N.SYSTEMASSIGNID)
JOIN COINSUREDINFO II ON II.SYSTEMASSIGNID = N.SYSTEMASSIGNID
AND II.TRANSSEQNO = ( SELECT MAX(TRANSSEQNO) FROM COINSUREDINFO I2
WHERE I2.SYSTEMASSIGNID = II.SYSTEMASSIGNID)
WHERE TS.POLICYEXPDT BETWEEN #STARTDATE AND #ENDDATE
AND PP.CANCEFFDT IS NULL
AND PD.PREDSTATECD IN ('CT', 'RI', 'GA','NH','NY')
ORDER BY POLICYID
The results are linked to a Data Conversion Task.
The Data Conversion Task is then linked to an OLE DB Destination task.
The OLE DB Destination task uses an OLE DB Provider that connects to an Access Database. One of the Tables in this Database is called MasterTable and it has an autonumber field named ID.
I have added screenshots of the entire workflow below.
High-level data workflow
OLE DB Destination connection info to Access Database
I found a way to do what I need so I am posting the answer here in case it helps anyone else. I will summarize the steps below and then post the code. I am using SSIS 2008.
I derived this answer from a related answer I found on this site:
What is the fastest way to insert 100 000 records into an MDB file in C#
Create a data flow task
In the data flow task, create an OLE Db source with your SQL code. This is the query that will give you the results to put into your Access Table.
Once I got the SQL working, I created a data conversion task and converted most of the columns to unicode (except the date columns). Pay attention to the names in the "Output Alias" column of the converter. These are the names you use in the C# script task shown below.
Save the results to a recordset destination. When you create the recordset, the "Component Properties" tab has a field named VariableName. Put a variable there. This variable will hold the results of the SQL query. I named mine "rsSourceTable." This variable is what the C# code will read to get our resultset.
Once you get the Data Flow task working, create a C# Script Task. I created several variables for use with the script task.
The read-only variables:
AccessPath - Holds the path where the Access file is located.
rsSourceTable - the variable that holds the results of our data flow task.
The read/write variables:
MasterTableRowCount - I use this to make it easy to report the number of files inserted in log files and email tasks.
6. The c# code for the script task is shown below. I did not have to add any references to this.
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Data.OleDb;
namespace ST_afd8e3cca5534e51ba5855e82f502e92.csproj
{
[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
OleDbConnection myConnection = new OleDbConnection();
try
{
string accessPath = Dts.Variables["AccessPath"].Value.ToString();
string materTableId = Dts.Variables["MasterTableId"].Value.ToString();
myConnection.ConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + accessPath;
DataTable dt = new DataTable();
OleDbDataAdapter adapter = new OleDbDataAdapter();
adapter.Fill(dt, Dts.Variables["User::rsSourceTable"].Value);
int rowCount = 0;
if (dt.Rows.Count > 0)
{
rowCount = dt.Rows.Count;
Dts.Variables["MasterTableRowCount"].Value.ToString();
myConnection.Open();
//1. When building the INSERT statement, remember to enclose column names in square brackets. This prevents errors because Access allows special characters in column names and OLE DB doesn't
//2. Also remember that the order the column names appear in the INSERT statement is important for the code that adds parameters below.
//3. To prevent an error, the INSERT statement is first constructed with a ? for each parameter. The parameter is replaced with the
// appropriate column name in the for loop below.
string insertString = "INSERT INTO MasterTable ([LOB_Name_Product], [Policy_#], [Policy_State], [Policy_Eff_Date], [Policy_Exp_Date], [Insured_Name], [Insured_Address_1], ";
insertString += "[Insured_Address_2], [Insured_City], [Agent_#], [Agent_Name], [Inforce_Prem_Sum], [Status], [Upload_date], [Insured_Zip], [Insured_State], [Agent_Address_1], [Agent_Address_2], [Agent_City], [Agent_Zip], [Agent_State])";
insertString += " Values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
OleDbCommand cmmd = new OleDbCommand(insertString, myConnection);
if (myConnection.State == ConnectionState.Open)
{
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //LOB_Name_Product
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Policy_#
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Policy_State
cmmd.Parameters.Add("?", OleDbType.DBDate, 10); //Policy_Eff_Date
cmmd.Parameters.Add("?", OleDbType.DBDate, 10); //Policy_Exp_Date
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Name
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Address_1
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Address_2
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_City
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_#
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Name
cmmd.Parameters.Add("?", OleDbType.Currency, 255); //Inforce_Prem_Sum
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Status
cmmd.Parameters.Add("?", OleDbType.Date, 10); //Upload_date
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_Zip
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Insured_State
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Address_1
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Address_2
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_City
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_Zip
cmmd.Parameters.Add("?", OleDbType.VarWChar, 255); //Agent_State
cmmd.Prepare();
OleDbTransaction trans = myConnection.BeginTransaction();
cmmd.Transaction = trans;
foreach(DataRow dr in dt.Rows)
{
cmmd.Parameters[0].Value = dr["Copy of LOB_Name_Product"];
cmmd.Parameters[1].Value = dr["Copy of Policy_#"];
cmmd.Parameters[2].Value = dr["Copy of Policy_State"];
cmmd.Parameters[3].Value = dr["Policy_Eff_Date"];
cmmd.Parameters[4].Value = dr["Policy_Exp_Date"];
cmmd.Parameters[5].Value = dr["Copy of Insured_Name"];
cmmd.Parameters[6].Value = dr["Copy of Insured_Address_1"];
cmmd.Parameters[7].Value = dr["Copy of Insured_Address_2"];
cmmd.Parameters[8].Value = dr["Copy of Insured_City"];
cmmd.Parameters[9].Value = dr["Copy of Agent_#"];
cmmd.Parameters[10].Value = dr["Copy of Agent_Name"];
cmmd.Parameters[11].Value = dr["Copy of Inforce_Prem_Sum"];
cmmd.Parameters[12].Value = "Open";
cmmd.Parameters[13].Value = DateTime.Today.ToString("d");
cmmd.Parameters[14].Value = dr["Copy of Insured_Zip"];
cmmd.Parameters[15].Value = dr["Copy of Insured_State"];
cmmd.Parameters[16].Value = dr["Copy of Agent_Address_1"];
cmmd.Parameters[17].Value = dr["Copy of Agent_Address_2"];
cmmd.Parameters[18].Value = dr["Copy of Agent_City"];
cmmd.Parameters[19].Value = dr["Copy of Agent_Zip"];
cmmd.Parameters[20].Value = dr["Copy of Agent_State"];
cmmd.ExecuteNonQuery();
}
trans.Commit();
myConnection.Close();
Dts.TaskResult = (int)ScriptResults.Success; //add logging here for successful operation
}
else
Dts.TaskResult = (int)ScriptResults.Failure;
}
else
Dts.TaskResult = (int)ScriptResults.Success; //add logging here for no records
}
catch (OleDbException oleEx)
{
myConnection.Close();
Dts.TaskResult = (int)ScriptResults.Failure; //add logging here for unable to connect
}
catch (Exception ex)
{
myConnection.Close();
Dts.TaskResult = (int)ScriptResults.Failure; //add logging here for any other error
}
}
}
}
Related
This question already has answers here:
How do I return multiple result sets with SqlCommand?
(7 answers)
Closed 1 year ago.
I have a user-defined stored procedure which returns multiple actual tables as result set:
CREATE PROCEDURE uspDemo
(#UserID BIGINT = 0,
#IsAdmin BIT = 0,
#Title VARCHAR(120) = '')
AS
BEGIN
------Retrieve Posts------
SELECT *
FROM tblPost AS MP
INNER JOIN tblUserProfile AS UP ON UP.ID = MP.UserID
WHERE UP.ID = #UserID
AND ((#IsAdmin = 0 AND MP.IsDeleted = 0 AND MP.IsApproved = 1)
OR (#IsAdmin = 1 OR MP.IsDeleted = 0 OR MP.IsApproved = 1))
----- Retrieve Tags------
SELECT *
FROM tblTagMasters AS MT
INNER JOIN tblPostTags AS MP ON MT.TagID = MP.TagID
--------Retrieve User likes-----
SELECT *
FROM tblUserLikes AS UV
INNER JOIN tblPost AS MP ON MP.PostId = UV.PostId
END
How to translate in ASP.NET MVC? Please help me.
Basically, you need something like this:
// define connection string and command for your query
string connectionString = "....."; // typically load from config
string storedProcedureName = "dbo.uspDemo";
// put your disposable ADO.NET objects into proper "using ()" blocks
using (SqlConnection conn = new SqlConnection(connectionString))
using (SqlCommand cmd = new SqlCommand(storedProcedureName, conn))
{
cmd.CommandType = CommandType.StoredProcedure;
// define the parameters
cmd.Parameters.Add("#UserID", SqlDbType.BigInt).Value = .....;
cmd.Parameters.Add("#IsAdmin", SqlDbType.Bit).Value = .....;
cmd.Parameters.Add("#Title", SqlDbType.VarChar, 120).Value = ".....";
// open connection
conn.Open();
// execute stored procedure to get the data reader
using (SqlDataReader reader = cmd.ExecuteReader())
{
// read the first set of data returned
while (reader.Read())
{
// read the values, construct an object from it, store the object into a list or something
}
// get the next set of result data
reader.NextResult();
// read the second set of data returned
while (reader.Read())
{
// read the values, construct an object from it, store the object into a list or something
}
// get the next set of result data
reader.NextResult();
// read the third set of data returned
while (reader.Read())
{
// read the values, construct an object from it, store the object into a list or something
}
}
conn.Close();
}
I leave the remaining details up to you - do some research!
I'm trying to do a "database side" bulk copy (i.e. SELECT INTO/INSERT INTO) using linq2db. However, my code is trying to bring the dataset over the wire which is not possible given the size of the DB in question.
My code looks like this:
using (var db = new MyDb()) {
var list = db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
});
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.BulkCopy(new BulkCopyOptions {
BulkCopyType = BulkCopyType.MultipleRows
}, list);
}
The generated SQL looks like this:
BeforeExecute
-- DBNAME SqlServer.2017
TRUNCATE TABLE DESTINATION_TABLE
DataConnection
Query Execution Time (AfterExecute): 00:00:00.0361209. Records Affected: -1.
DataConnection
BeforeExecute
-- DBNAME SqlServer.2017
DECLARE #take Int -- Int32
SET #take = 1
DECLARE #take_1 Int -- Int32
SET #take_1 = 1
DECLARE #take_2 Int -- Int32
...
SELECT
(
SELECT TOP (#take)
[p].[YEAR]
FROM
[dbo].[SOURCE_TABLE] [p]
WHERE
(([p_16].[YEAR] = [p].[YEAR] OR [p_16].[YEAR] IS NULL AND [p].[YEAR] IS NULL) AND ...
...)
FROM SOURCE_TABLE p_16
WHERE p_16.YEAR > 2012
GROUP BY
...
DataConnection
That is all that is logged as the bulkcopy fails with a timeout, i.e. SqlException "Execution Timeout Expired".
Please note that running this query as an INSERT INTO statement takes less than 1 second directly in the DB.
PS: Anyone have any recommendations as to good code based ETL tools to do large DB (+ 1 TB) ETL. Given the DB size I need things to run in the database and not bring data over the wire. I've tried pyspark, python bonobo, c# etlbox and they all move too much data around. I thought linq2db had potential, i.e. basically just act like a C# to SQL transpiler but it is also trying to move data around.
I would suggest to rewrite your query because group by can not return first element. Also Truncate is a part of the library.
var sourceQuery =
from s in db.SourceTable
where s.Year > 2012
select new
{
Source = s,
Count = Sql.Ext.Count(s.Column3 == 'Y' ? 1 : null).Over()
.PartitionBy(s.Column1, s.Column2).ToValue()
RN = Sql.Ext.RowNumber().Over()
.PartitionBy(s.Column1, s.Column2).OrderByDesc(s.Year).ToValue()
};
db.DestinationTable.Truncate();
sourceQuery.Where(s => s.RN == 1)
.Insert(db.DestinationTable,
e => new DestinationTable
{
Property1 = 'Constant Value',
Property2 = e.Source.Column1,
Property3 = e.Source.Column2,
Property4 = e.Count
});
After some investigation I stumbled onto this issue. Which lead me to the solution. The code above needs to change to:
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
}).Insert(db.DestinationTable, e => e);
Documentation of the linq2db project leaves a bit to be desired however, in terms of functionality its looking like a great project for ETLs (without horrible 1000s of line copy/paste sql/ssis scripts).
I recently migrated a MS Access database to SQL Server, mostly was imported just fine, but there are some datatype differences that I would like to find with some tool if available.
Tools I found so far compare MS Access against MS Access, or SQL Server vs SQL Server only.
At issue is that Access (or JET Red) does not have a single canonical API for working with its data-model, instead you mostly go through the OLE-DB driver or the ODBC driver. I think (but cannot confirm) that the Office Access GUI program probably its own internal API that bypasses the OLE-DB or ODBC abstractions, unfortunately the GUI program does not use specific technical terminology in things like the Table Designer (e.g. Number > Integer doesnt say if it's a 16, 32 or 64-bit integer, and Number > Replication ID is not a number at all but a Win32 GUID).
As of 2019, Microsoft has seemingly de-prioritized OLE-DB compared to the lower-level ODBC API for JET Red, but that's okay because ODBC still provides us with the necessary details for determining a database table's design.
Anyway - the good news is that you don't necessarily need a tool to compare an Access (JET Red) database with a SQL Server database because it's easy to get the ODBC table specifications yourself.
Something like this:
Dictionary<String,DataTable> jetTables = new Dictionary<String,DataTable>();
using( OleDbConnection jetConnection = new OleDbConnection( "your-access-connection-string") )
{
await jetConnection.OpenAsync().ConfigureAwait(false);
DataTable schemaTable = connection.GetOleDbSchemaTable(
OleDbSchemaGuid.Tables,
new object[] { null, null, null, "TABLE" }
);
foreach( DataRow row in schemaTable.Rows.Cast<DataRow>() )
{
String tableName = (String)row.ItemArray[2];
DataTable tableSchema = connection.GetOleDbSchemaTable(
OleDbSchemaGuid.Tables,
new object[] { null, null, tableName, "TABLE" }
);
jetTables.Add( tableName, tableSchema );
}
}
Dictionary<String,DataTable> sqlTables = new Dictionary<String,DataTable>();
using( SqlConnection sqlConnection = new SqlConnection( "your-sql-server-connection-string" ) )
{
await sqlConnection.OpenAsync().ConfigureAwait(false);
DataTable allTables = new DataTable();
using( SqlCommand cmd1 = sqlConnection.CreateCommand() )
{
cmd1.CommandText = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES";
using( SqlDataReader rdr1 = await cmd1.ExecuteReaderAsync.ConfigureAwait(false) )
{
allTables.Load( rdr1 );
}
}
foreach( DataRow row in allTables.Rows.Cast<DataRow>() )
{
String tableName = (String)row.ItemArray[0];
using( SqlCommand cmd2 = sqlConnection.CreateCommand() )
{
cmd2.CommandText = "SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = #tableName";
cmd2.Parameters.Add( "#tableName", SqlDbType.NVarChar ).Value = tableName;
using( SqlDataReader rdr2 = await cmd2.ExecuteReaderAsync.ConfigureAwait(false) )
{
DataTable dt = new DataTable();
dt.Load( rdr2 );
sqlTables.Add( tableName, dt );
}
}
}
}
Then compare jetTables with sqlTables as you so wish.
I am using the following code to export data from excel to Sql server database. What is going on with this code is, its importing complete data into the database.
[HttpPost]
public ActionResult Importexcel()
{
if (Request.Files["FileUpload1"].ContentLength > 0)
{
string extension = System.IO.Path.GetExtension(Request.Files["FileUpload1"].FileName);
string path1 = string.Format("{0}/{1}", Server.MapPath("~/Content/UploadedFolder"), Request.Files["FileUpload1"].FileName);
if (System.IO.File.Exists(path1))
System.IO.File.Delete(path1);
Request.Files["FileUpload1"].SaveAs(path1);
string sqlConnectionString = #"Data Source=xyz-101\SQLEXPRESS;Database=PracDB;Trusted_Connection=true;Persist Security Info=True";
string excelConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + path1 + ";Extended Properties=Excel 12.0;Persist Security Info=False";
OleDbConnection excelConnection = new OleDbConnection(excelConnectionString);
OleDbCommand cmd = new OleDbCommand("Select [ID],[Name],[Designation] from [Sheet1$]", excelConnection);
excelConnection.Open();
OleDbDataReader dReader;
dReader = cmd.ExecuteReader();
SqlBulkCopy sqlBulk = new SqlBulkCopy(sqlConnectionString);
sqlBulk.DestinationTableName = "Excel_Table";
sqlBulk.WriteToServer(dReader);
excelConnection.Close();
}
return RedirectToAction("Index");
}
How to check that any particular record already exist in the database or not. If not then Insert the record into the databse else it should not.
Thanks in Advance !
Since your target is SQL Server, you can use this to your advantage.
What I would do is read the data from the excel into a DataTable (instead of using a DataReader you can use a DataAdapter), Send that DataTable to a stored procedure in the SQL server, and handle the insert there. In order to sand a data table to a stored procedure you first need to create a Table-value user defined type in your sql server, like this:
CREATE TYPE MyType AS TABLE
(
Id int,
Name varchar(20), -- use whatever length best fitted to your data
Designation varchar(max) -- use whatever length best fitted to your data
)
Then you can write a simple stored procedure with an argument of this type:
CREATE PROCEDURE InsertDataFromExcel
(
#ExcelData dbo.MyType readonly -- Note: readonly is a required!
)
AS
INSERT INTO MyTable(Id, Name, Designation)
SELECT a.Id, a.Name, a.Designation
FROM #ExcelData a LEFT JOIN
MyTable b ON(a.Id = b.Id)
WHERE b.Id IS NULL -- this condition with the left join ensures you only select records that has different id values then the records already in your database
in order to send this parameter to the stored procedure from your c# code you will have to use a SqlCommand object and add the DataTable as a parameter, something like this:
using(SqlConnection Con = new SqlConnection(sqlConnectionString))
{
using(SqlCommand InsertCommand = new SqlCommand("InsertDataFromExcel", Con))
{
SqlParameter MyParam = new SqlParameter("#ExcelData", SqlDBType.Structured);
MyParam.Value = MyDataTable; // this is the data table from the
InsertCommand.Parameters.Add(MyParam);
Con.Open();
InsertCommand.ExecuteNoQuery();
Con.Close();
}
}
Note: Code was writen directly here, some errors might be found.
I have a table in C#, the data is coming from an Excel file. I need this data to be inserted into a SQL Server 2000 table. I am not to use stored procedures. How do I program it? Any help would be appreciated.
Do you have a DataTable ?
You'd need something like:
// set up connection to your database
using (SqlConnection con = new SqlConnection("your-connection-string-here"))
{
// define the INSERT statement - of course, I don't know what your table name
// is and which and how many fields you want to insert - adjust accordingly
string insertStmt =
"INSERT INTO dbo.YourTable(field1, field2, field3) " +
"VALUES(#field1, #field2, #field3)";
// create SqlCommand object
using (SqlCommand cmd = new SqlCommand(insertStmt, con))
{
// set up the parameters - again: I don't know your parameter names
// nor the parameters types - adjust to your needs
cmd.Parameters.Add("#field1", SqlDbType.Int);
cmd.Parameters.Add("#field2", SqlDbType.VarChar, 100);
cmd.Parameters.Add("#field3", SqlDbType.VarChar, 250);
// open connection
con.Open();
// iterate over all the Rows in your data table
foreach (DataRow row in YourDataTable.Rows)
{
// assign the values to the parameters, based on your DataRow
cmd.Parameters["#field1"].Value = Convert.ToInt32(row["columnname1"]);
cmd.Parameters["#field2"].Value = row["columnname2"].ToString();
cmd.Parameters["#field3"].Value = row["columnname3"].ToString();
// call INSERT statement
cmd.ExecuteNonQuery();
}
// close connection
con.Close();
}
}
Of course, this has no error checking whatsoever, you will need to add some of that yourself (try....catch and so on). But basically, that's the way I would do it, if I can't use stored procedures.
use System.Data.SqlClient.SqlCommand