I am trying to use change tracking to copy data incrementally from a SQL Server to an Azure SQL Database. I followed the tutorial on Microsoft Azure documentation but I ran into some problems when implementing this for a large number of tables.
In the source part of the copy activity I can use a query that gives me a change table of all the records that are updated, inserted or deleted since the last change tracking version. This table will look something like
PersonID Age Name SYS_CHANGE_OPERATION
---------------------------------------------
1 12 John U
2 15 James U
3 NULL NULL D
4 25 Jane I
with PersonID being the primary key for this table.
The problem is that the copy activity can only append the data to the Azure SQL Database so when a record gets updated it gives an error because of a duplicate primary key. I can deal with this problem by letting the copy activity use a stored procedure that merges the data into the table on the Azure SQL Database, but the problem is that I have a large number of tables.
I would like the pre-copy script to delete the deleted and updated records on the Azure SQL Database, but I can't figure out how to do this. Do I need to create separate stored procedures and corresponding table types for each table that I want to copy or is there a way for the pre-copy script to delete records based on the change tracking table?
You have to use a LookUp activity before the Copy Activity. With that LookUp activity you can query the database so that you get the deleted and updated PersonIDs, preferably all in one field, separated by comma (so its easier to use in the pre-copy script). More information here: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity
Then you can do the following in your pre-copy script:
delete from TableName where PersonID in (#{activity('MyLookUp').output.firstRow.PersonIDs})
This way you will be deleting all the deleted or updated rows before inserting the new ones.
Hope this helped!
In the meanwhile the Azure Data Factory provides the meta-data driven copy task. After going through the dialogue driven setup, a metadata table is created, which has one row for each dataset to be synchronized. I solved this UPSERT problem by adding a stored procedure as well as a table type for each dataset to be synchronized. Then I added the relevant information in the metadata table for each row like this
{
"preCopyScript": null,
"tableOption": "autoCreate",
"storedProcedure": "schemaname.UPSERT_SHOP_SP",
"tableType": "schemaname.TABLE_TYPE_SHOP",
"tableTypeParameterName": "shops"
}
After that you need to adapt the sink properties of the copy task like this (stored procedure, table type, table type parameter name):
#json(item().CopySinkSettings).storedProcedure
#json(item().CopySinkSettings).tableType
#json(item().CopySinkSettings).tableTypeParameterName
If the destination table does not exist, you need to run the whole task once before adding the above variables, because auto-create of tables works only as long as no stored procedure is given in the sink properties.
I have a linked table in my access database (dbo_Billing_denied (DSN=WTSTSQL05_BB;DATABASE=DEPTFINANCE), etc.) and I want to create a table that will store the data from this linked into local table, so I can use it to run other queries. Currently I can use this because it tells me that it can not make connection (ODB--connection to 'WTSTSQL05_BB' failed.
Do I have to create a table first and assign all the fields before I can do this (create a table and fields that are same as what's in the linked table and than create append query to do this...)?
It sounds like you might have two problems. I will address the second one. You will need to reestablish connection to the linked table before this will work.
You can use a "make table query" in Access to make a local copy of the linked table. You can use the GUI for this, or you can structure the SQL something like this:
SELECT <list of various fields, or * for all fields>
INTO <name of new local table>
FROM <name of linked table(s) on the server>
WHERE <any other conditions you want to put on which records are included>;
I mentioned that there might be more than one table. You can also do this with joined tables or unions etc. The "where" clause is optional. Removing it will copy the entire data set.
You will get a warning when you try to execute this query in Access. It will tell you that you are about to write (or overwrite) a table. If you are trying to write a cleaner application with fewer nuisance messages for the end user, call this query from a macro. The macro would need to turn the warnings off, execute the query, then turn the warnings back on.
Microsoft Access does not require you to create this table before you write it; if the table does not exist Access will create this table for you, based on the field definitions in the source data. If a table of the same name does exist, Access will drop this table from your database and then create a new table of that name.
This also implies that the local table you are generating will need a unique name. If your query tries to overwrite the linked table by using the same name, the first thing Access will do is drop the linked table. It will then look for field definitions and input data in the linked table that it just dropped.
Since the new local table will have a new name, queries developed for the linked table will not work with the new local table. One possible work-around would be to rename the linked table in your local Access database. The table name in Access does not need to equal the name in the database it's linking to. The query could then write to a table with the correct name, and previous queries should work. Still, keep in mind that these queries would no longer be working on live data.
I am an application programmer but currently I have a situation in which I need to copy a huge amount of data which is collected for 1 month say approx 653 GB of data from the table in one database to exact similar table in other database (both oracle 11G). Each row size is approx 150 bytes. So the number of rows are approximately 4000 millions. I am not joking.
I have to do this. The table which holds this data (source table) is partitioned based on date column. So there is a partition for each day of the month and hence in total 31 partitions for December month.
The target db is partitioned based on month. So there is a single partition in the target db for complete december month.
I have chosen to copy the data over db link and with the help of dba's I created a db link between this 2 databases.
I have a store procedure in target db which accepts input parameter as (date, tablename). What this procedure does is it creates a temporary table in target db with name as tablename and copies all the data from the source db for the given date into this temporary table in target database. I have done it successfully for 2-3 days. Now I want to insert this data in the temporary table into actual table in the same target database. For that I executed following query:
insert into schemaname.target_table select * from schemaname.temp_table;
But I am getting folloowing ORA error.
ORA-01858: a non-numeric character was found where a numeric was expected
Both the tables have exact same table defination. I searched over internet for copying data and found the above query to insert as simplest. But I don't understand the error. Searching for this errors show it is has something to do with date column. But shouldn't it work as both the tables have same table structure?
Data types used in the table are varchar2(x), date, number(x,y), char(x).
Please help me to get over this error. Let me know if any other information is required.
It means your schemaname.temp_table table has some non-numeric value which doesn't inserted in your new table. Did schemaname.temp_table table populated with some script or any automated tool? There is possibility of empty space or junk character inserted in schemaname.temp_table. Kindly check once again using any sql tool.
There is this table in my Oracle database that is used to store audit information.
When I first did a SELECT * on that table, the audit timestamps were all on the same day, within the same hour (e.g. 18/10/2013 15:06:45, 18/10/2013 15:07:29); the next time I did it, the previous entries were gone, and the table then only contained entries with the 16:mm:ss timestamp.
I think something is acting on that table, such that every interval the table contents is/may be backed up to somewhere - I don't know where, and then the table is cleared. However, as I'm not familiar with databases, I'm not sure what is doing this.
I'd like to know how I can find out what is acting on this table, so that I can in turn retrieve the previous data I need.
EDIT:
What I've tried thus far...
SELECT * FROM DBA_DEPENDENCIES WHERE REFERENCED_NAME='MY_AUDIT_TABLE';
I got back four results, but all of which were (based on my programming skills) talking about putting data into the table, none about backing it up anywhere.
SELECT * FROM MY_AUDIT_TABLE AS OF TIMESTAMP ...
This only gives me a snapshot at a certain time, but since the table is being updated very frequently, it does not make sense for me to query every second.
The dba_dependencies view will give you an idea on what procedures, function etc will act on the table
SELECT * FROM DBA_DEPENDENCIES WHERE REFERENCED_NAME='MY_AUDIT_TABLE';
where MY_AUDIT_TABLE is the audit table name
if the table's synonym is used in the database then
SELECT * FROM DBA_DEPENDENCIES WHERE REFERENCED_NAME='MY_AUDIT_TABLE_SYNONYM';
where MY_AUDIT_TABLE_SYNONYM is the synonym for MY_AUDIT_TABLE
Or if any triggers are acting on the table
Select * from dba_triggers where table_name='MY_AUDIT_TABLE';
for external script to process the table
you can request DBA to turn on DB Fine grained audit for the table
Then query view DBA_FGA_AUDIT_TRAIL with timestamp between 15:00:00 and 16:00:00 to check the external call(OS_PROCESS column will give Operating System Process ID) or what SQL(SQL_TEXT) is executing on the table
If I'm adding a column to a table in Microsoft SQL Server, can I control where the column is displayed logically in queries?
I don't want to mess with the physical layout of columns on disk, but I would like to logically group columns together when possible so that tools like SQL Server Management Studio list the contents of the table in a convenient way.
I know that I can do this through SQL Management Studio by going into their "design" mode for tables and dragging the order of columns around, but I'd like to be able to do it in raw SQL so that I can perform the ordering scripted from the command line.
You can not do this programatically (in a safe way that is) without creating a new table.
What Enterprise Manager does when you commit a reordering is to create a new table, move the data and then delete the old table and rename the new table to the existing name.
If you want your columns in a particular order/grouping without altering their physical order, you can create a view which can be whatever you desire.
I think what everyone here is missing is that although not everyone has to deal with 10's, 20's, or 1000's instances of the same software system installed throughout the country and world, those of us that design commercially sold software do so. As a result, we expand systems over time, expand tables by adding fields as new capability is needed, and as those fields are identified do belong in an existing table, and as such, over a decade of expanding, growing, adding fields, etc to tables, and then having to work with those tables from design, to support, to sometimes digging into raw data/troubleshooting to debug new functionality bugs, it is incredibly aggravating to not have the primary information you want to see within the first handful of fields, when you may have tables with 30, 40, 50, or even 90 fields, and yes, in a strictly normalized database.
I've often wished I could do this, for this exact reason. But short of doing exactly what SQL does, building a Create Script for a new Table the way I want it, writing the Insert to it, then dropping all existing constraints, relationships, keys, index, etc etc from the existing table and renaming the "new" table back to the old name, and then reading all those keys, relationships, index, etc etc ....
It's not only tedious, time-consuming, but ... in five more years, it will need to happen again.
It's so close to worth that massive amount of work, however the point is, it won't be the last time we need this ability, since our systems will continue to grow, expand, and get fields in a wacked ordered driven by need/design additions.
A majority of developers think from a single system standpoint that serves a single company or very specific hard box market.
The "off-the-shelf" but significantly progressive designers and leaders of development in their market space will always have to deal with this problem, over and over, and would love a creative solution if anyone has one. This could easily save my company a dozen hours a week, just not having to scroll over, or remember where "that" field is in the source data table.
When Management Studio does it, it's creating a temporary table, copying everything across, dropping your original table and renaming the temporary table. There's no simple equivalent T-SQL statement.
If you don't fancy doing that, you could always create a view of the table with the columns in the order you'd like and use that?
Edit: beaten!
If I understand your question, you want to affect what columns are returned first, second, third, etc in existing queries, right?
If all of your queries are written with SELECT * FROM TABLE - then they will show up in the output as they are laid out in SQL.
If your queries are written with SELECT Field1, Field2 FROM TABLE - then the order they are laid out in SQL does not matter.
There is one way, but its only temporarily for the query itself. For example,
Lets say you have 5 tables.
Table is called T_Testing
FirstName, LastName, PhoneNumber, Email, and Member_ID
you want it to list their ID, then Last Name, then FirstName, then Phone then Email.
You can do it as per the Select.
Select Member_ID, LastName, FirstName, PhoneNumber, Email
From T_Testing
Other than that, if you just want the LastName to Show before first name for some reason, you can do it also as follows:
Select LastName, *
From T_Testing
The only thing you wanna be sure that you do is that the OrderBy or Where Function needs to be denoted as Table.Column if you are going to be using a Where or OrderBy
Example:
Select LastName, *
From T_Testing
Order By T_Testing.LastName Desc
I hope this helps, I figured it out because I needed to do this myself.
Script your existing table to a query window.
Run this script against a Test database (remove the Use statement)
Use SSMS to make the column changes you need
Click Generate Change Script (left most and bottommost icon on the
buttonbar, by default)
Use this script against your real table
All the script really does is create a second table table with the desired column orders, copies all your data into it, drops the original table and then renames the secondary table to take its place. This does save you writing it yourself though should you want a deploy script.
It is not possible to change the order of the columns without recreating the whole table. If you have a few instances of the database only, you can use SSMS for this (Select the table and click "design").
In case you have too many instances for a manual process, you should try this script:
https://github.com/Epaminaidos/reorder-columns
It can be done using SQL, by modifying the system tables directly. For example, look here:
Alter table - Add new column in between
However, I would not recommend playing with system tables, unless it's absolutely necessary.
Open your table in SSMS in design mode:
Reorder your columns:
It is important to not save your change.
Click the "Generate Change Script" button:
Now a window will open that contains the script to apply this change:
Copy the text from the window.
In this instance, it generated the following code:
/* To prevent any potential data loss issues, you should review this script in detail before running it outside the context of the database designer.*/
BEGIN TRANSACTION
SET QUOTED_IDENTIFIER ON
SET ARITHABORT ON
SET NUMERIC_ROUNDABORT OFF
SET CONCAT_NULL_YIELDS_NULL ON
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
COMMIT
BEGIN TRANSACTION
GO
CREATE TABLE dbo.Tmp_MyTable
(
Id int NOT NULL,
Name nvarchar(30) NULL,
Country nvarchar(50) NOT NULL
) ON [PRIMARY]
GO
ALTER TABLE dbo.Tmp_MyTable SET (LOCK_ESCALATION = TABLE)
GO
IF EXISTS(SELECT * FROM dbo.MyTable)
EXEC('INSERT INTO dbo.Tmp_MyTable (Id, Name, Country)
SELECT Id, Name, Country FROM dbo.MyTable WITH (HOLDLOCK TABLOCKX)')
GO
DROP TABLE dbo.MyTable
GO
EXECUTE sp_rename N'dbo.Tmp_MyTable', N'MyTable', 'OBJECT'
GO
COMMIT
As you can see, what it does is 1) create a new temporary table, 2) copy the data over to the temporary table, 3) delete the original table and 4) rename the temporary table to the original table's name.