Yesterday I learned how to use ADO.NET to take a DataTable and bulk insert all records into a SQL server database. That works great for one part of my project. Now what I need to do is take the records from the DataTable and insert only the records that do not already exist in the SQL server. Im using VB2010 and the code looks something like:
'get the data from our Teradata server and fill up a DataTable
Dim dtCheck As New DataTable("TableCheck")
dtCheck = GetDataTableFromTeradataServer
'connect to our SQL server
Dim connSqlSvr As New System.Data.SqlClient.SqlConnection
connSqlSvr.ConnectionString = "Data Source=DestSqlServer;Initial Catalog=DestDb;Connect Timeout=15"
connSqlSvr.Open()
'how do I take records from the DataTable and insert only those records that do not
'already exist in the SQL server?
'close and dispose the SQL server database connection
connSqlSvr.Close()
connSqlSvr.Dispose()
The SQL server destination table [DestDb].[dbo].[Events] has fields
[CityCode],[CarNum],[VIN],[Fleet],[CarMileage],[EventItm],[EventDate]
and the DataTable in memory has fields
City,CarNo,VIN,Fleet,Mileage,EventDesc,EventDate
For my purposes a unique record is made up of [CityCode],[CarNum],[VIN],[EventItm], and [EventDate]
Typically i will run this routine once a day and will get back about 200 records from the source. Been looking for a couple hours but just not sure how to approach this.
You could try loading data in a staging table in the target server. Then do the unique insert from there.
Related
I have a stored procedure with statements similar to this:
DELETE FROM [LinkedServer].[DB1].[dbo].[Table1]
DELETE FROM [LinkedServer].[DB1].[dbo].[Table2]
DELETE FROM [LinkedServer].[DB1].[dbo].[Table3]
INSERT INTO [LinkedServer].[DB1].[dbo].[Table3]
SELECT * FROM [DB1].[dbo].[Table3]
INSERT INTO [LinkedServer].[DB1].[dbo].[Table2]
SELECT * FROM [DB1].[dbo].[Table2]
INSERT INTO [LinkedServer].[DB1].[dbo].[Table1]
SELECT * FROM [DB1].[dbo].[Table1]
Which is extremely slow. But my goal is simple as this.
I cannot use replication, just a method to empty the tables and fill them again.
If I do the same action using import/export functionality from SSMS, it empties the remote tables and fill them up very quickly.
Is there a way to simulate what import/export is doing, using T-SQL commands?
It would not be a problem to disable restrictions while copying data.
Perhaps exists some kind of BULK INSERT I could use in this scenario? After search info about this option it seems to be useful to transfer data from SQL to file or from file to SQL, but I don't find examples to transfer from table to table.
Is there a way to simulate what import/export is doing, using Transact SQL commands?
Not quite. But try running the INSERT from the other end.
INSERT INTO [DB1].[dbo].[Table3] SELECT * FROM [LinkedServer].[DB1].[dbo].[Table3]
Or install an instance of SQL Server 2008 and backup/restore to upgrade the database; then backup/restore that to the target version.
I finally opted for #Jeroen Mostert's proposal, since no other working solutions have turned up.
I have created a small command line tool that receives the parameters for the source and destination connections. I've tested it with a table and it goes just as fast as the import/export does.
Using sourceConn As New SqlConnection(sourceConnStr)
Using destinationConn As New SqlConnection(destinationConnStr)
Dim cM As New SqlCommand("SELECT * FROM " & sourceTable, sourceConn)
sourceConn.Open()
Using dR As SqlDataReader = cM.ExecuteReader
Dim bC As New SqlClient.SqlBulkCopy(destinationConn)
destinationConn.Open()
'Truncate
cM = New SqlCommand("TRUNCATE TABLE " & destinationTable, destinationConn)
cM.ExecuteNonQuery()
'BulkCopy
bC.DestinationTableName = destinationTable
bC.WriteToServer(dR)
'Close connections
dR.Close()
sourceConn.Close()
destinationConn.Close()
End Using
End Using
End Using
I am trying to add new records to a ODBC linked table in VBA Access. When the table was local, stored within the Access database, there was no issue. But when I uploaded the table to our Azure SQL server and linked it to the Access database the code no longer works. The code is on the "onclick" event of a button.
The code is used to add a quick record of when a user logs in to the Access application. I've linked the other server tables for forms and reports of the database, no problems. But have never had to add records in VBA before. The reason it needs to be done in VBA is because it needs to be hidden from the user.
Dim Rst As Recordset
Set Rst = CurrentDb.OpenRecordset("tblLoginLog", dbOpenDynaset)
With Rst
.AddNew
!User = ID
!LoginDate = Now()
.Update
End With
tblLoginLog is the name of the table,
User is a number field for the employee number,
LoginDate is a date field for when login is successful.
If the table is stored locally in the database, the code works, but when the table is moved to the server nothing happens. When I click on the button, literally nothing happens. No error messages, nothing, and when I open the table, no data has been passed to the table.
I have created an SSIS package to extract and load data from access table (.mdb) named TmShipping to a SQL table say, TmShippingImport, this SQL table has Id and importedDate as additional columns. I have scheduled the package to run for every 30 minutes.
TmShipping.mds
----------------------------------------
OrderId CarrierId TotalCharge
TmShippingImport (SQL table)
-------------------------------------------------------
Id OrderId CarrierId TotalCharge importedDate
In the Data Flow Task:
I am getting the data from source using OLD EB connection and extracting all row data from the access table and the output of this is connected to a Recordset Destination so that I can extract each row.
In the Control flow task:
I have a loop container (connected to data flow task's o/p) which inserts each row into the sql table by a sql query and loads all the row data along with the current datetime.
Package execution
The SSIS package when executed for the first time loads each row into the SQL Table and add a DateTime to imporatDatetime Column. When new records are created in the Access table, the package now takes all the rows in the MS access (rows that were extracted previously and new rows) and loads them in the SQL table again. How to avoid duplicates? My primary key in SQL table is ID which is not present in the MS Access Table.
Tried using Lookup table in the Dataflow in between source and recordset destination but it failed saying I can't connect the available column to BLOB.
Should I be trying with Lookup Merge in the dataflow or should I make changes in the foreachloop container so that it checks for duplicates before inserting the rows into the sql table or...?
First of all, you're approach to inserting the data into the SQL table seems very expensive. Is there a reason you can't simply insert the data in to the SQL table in the data flow using a DataFlow destination, it will be much, much more performant.
If the reason you've not done this is because of the timestamp, you can achieve this be using a derived column transformation and GetDate().
Once you've done that you can implement the pattern used in the answer Tab linked to.
How can I extract the table name from a field in an open ADO recordset?
I can use the .source property to return the select statement, but if I want to know which table a field is associated with, how would I do that?
i.e. if .source returns a something like SELECT Foo, Boo, Shoo FROM table1, table2 how could I query the recordset to see which table Boo is coming from?
My data source is a SQL Server 2005 database and I'm using VBA 7.0/VBA 6.0 (runs in both)
VS2008 project. Created local database (SSCE). Created dataset with tableadapter. Dragged dataset and tableadapter to the form so I can reference it in code.
New records successfully add to the dataset but will not commit back to the database. Gives no error or clue why it won't work.
TableAdapter insert statement was created automatically and is parameterized (#p1, #p2, etc.), but I am trying to avoid those parameter.add statements as I want to be able to use a field-by-field format without having to essentially repeat the schema of my database in code with parameter.add statements. Command object and INSERT statements work fine, but then you always have to construct an INSERT statement -- a pain if they're complicated. I want something as simple as working with a ADO recordset, but in .NET.
I know the last statement with the update is wrong without the parameters, but looking for an alternative method.
What can I do to accomplish the following without parameter.add statements?
DocsTableAdapter1.Fill(Documents1.Docs)
Debug.Print("Starting Row Count is: " & Documents1.Docs.Count.ToString)
Dim dr As DataRow = Documents1.Docs.NewRow
dr("Name") = "John Smith"
dr("Reference") = "My new reference code"
Documents1.Docs.Rows.Add(dr)
Debug.Print("New Row Count is: " & Documents1.Docs.Count.ToString)
DocsTableAdapter1.Update(Documents1.Docs)