How to do automatic data archiving in SQL Server? - sql-server

I have table for which every day I want to do automatic archiving. So to be clear every day I want to take information generated during that day and move it into another partition (of same table) not in another archive table. That's because I want old data to be accessible with same query as new ones.
I'm using SQL Server 2005, I've read http://msdn.microsoft.com/en-us/library/ms345146(SQL.90).aspx article but couldn't find out how can I write partitioning function to satisfy my needs.
So the solution I hope exists should be one time configuration which won't need any further interference. Do you have any suggestions?

You can easily do this with partioned tables; example script below -- it creates a temporary db TestDB; if you dont want to use it change the database to something else. It cleans itself up at the end
IF you run the script as is it
Creates the database;
adds a partitioning funcition based on a Bit.
Creates a table TestTable; applied with the partitioning function
Inerts 3 "Live" rows into the table
Shows that 3 rows are all in one of the partition tables by selecting details from sys.partitions
THen updates one of the records to make it archived
Reselects information from sys.partitions to show that the record has moved to the second schema.
All you would need to do is setup a process to archive the records.
USE master;
GO
--- Step 1 : Create New Test Database with two different filegroups.
IF EXISTS (
SELECT name
FROM sys.databases
WHERE name = N'TestDB')
DROP DATABASE TestDB;
GO
CREATE DATABASE TestDB
ON PRIMARY
(NAME='TestDB_Part1',
FILENAME=
'c:\sqldata\TestDB_Part1.mdf',
SIZE=3,
MAXSIZE=100,
FILEGROWTH=1 ),
FILEGROUP TestDB_Part2
(NAME = 'TestDB_Part2',
FILENAME =
'c:\sqldata\TestDB_Part2.ndf',
SIZE =3,
MAXSIZE=100,
FILEGROWTH=1 );
GO
USE TestDB;
GO
--- Step 2 : Create Partition Range Function
CREATE PARTITION FUNCTION TestDB_PartitionRange (Bit)
AS RANGE right FOR
VALUES (1);
GO
CREATE PARTITION SCHEME TestDB_PartitionScheme
AS PARTITION TestDB_PartitionRange
TO ([PRIMARY], TestDB_Part2);
GO
CREATE TABLE TestTable
(Archived Bit NOT NULL,
Date DATETIME)
ON TestDB_PartitionScheme (Archived);
GO
INSERT INTO TestTable (Archived, Date)
VALUES (0,'2010-01-01');
INSERT INTO TestTable (Archived, Date)
VALUES (0,'2010-02-01');
INSERT INTO TestTable (Archived, Date)
VALUES (0,'2010-03-01');
GO
SELECT * FROM TestTable;
SELECT * FROM sys.partitions
WHERE OBJECT_NAME(OBJECT_ID)='TestTable';
update TestTable
set Archived = 1 where Date = '2010-03-01'
SELECT * FROM sys.partitions
WHERE OBJECT_NAME(OBJECT_ID)='TestTable';
use master
go
drop database testdb

Related

Run update or insert query after select query in SQL Server 2012

I have a large data feed each morning that is dumped into a SQL Server table with a truncate query.
I need to update parts of my master table with any changes from this dump
and append any new rows (basically the rest) to my Master table also from this dump.
To reduce the run time this truncate query is about to be modified to cover only the last 2 years.
How should I go about do this
Both tables have identical structure so hopefully this should be fairly straight forward.
The apt way of doing should be a merge statement.
See here for more
https://learn.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql?view=sql-server-2017
The scenario mentioned in Example C would be the case you should follow.
-- Create a temporary table variable to hold the output actions.
DECLARE #SummaryOfChanges TABLE(Change VARCHAR(20));
MERGE INTO Sales.SalesReason AS Target
USING (VALUES ('Recommendation','Other'), ('Review', 'Marketing'),
('Internet', 'Promotion'))
AS Source (NewName, NewReasonType)
ON Target.Name = Source.NewName
WHEN MATCHED THEN
UPDATE SET ReasonType = Source.NewReasonType
WHEN NOT MATCHED BY TARGET THEN
INSERT (Name, ReasonType) VALUES (NewName, NewReasonType)
OUTPUT $action INTO #SummaryOfChanges;
-- Query the results of the table variable.
SELECT Change, COUNT(*) AS CountPerChange
FROM #SummaryOfChanges
GROUP BY Change;

IF option in sqlite returning syntax error [duplicate]

I need to create a temp table (which is a copy of Employees table) in a SQLite database, provided the table by the name of 'Employees' exists. There are only 2 columns in Employees table - EmployeeId (integer) and EmployeeName (varchar(100)).
Is there any way to implement the above using SQLite SQL?
Pseudo-code for intended SQL, which does not work in SQlite is as below. I hope there was something as powerful as the pseudo-code below in SQLite.
--if Employees table exists then create a temp table and populate it with all
--rows from Employees table
CREATE TEMP TABLE tempEmployees if exists Employees as select * from Employees;
SQLite has almost no control logic; as an embedded database, it is designed to be used together with a 'real' programming language.
You could just try to copy the data, and ignore the errors:
try:
c.execute("CREATE TEMP TABLE tempEmployees AS SELECT * FROM Employees")
except:
pass
However, this would also suppress any other errors.
A better idea is to check explicitly whether the table exists:
c.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name='Employees'")
if c.fetchone():
c.execute("CREATE TEMP TABLE tempEmployees AS SELECT * FROM Employees")

How To Get Temp Table to Run Multiple Time

I'm using MS SQL Server. I'm currently working on a query for pulling headcount. In this process, I'm creating temp tables, but noticed that I can only run the query once. If I try running it again after making changes, it gives me the 'There is already an object named '#Test1' in the database.'
My SQL looks like this:
SET NOCOUNT ON SET ANSI_WARNINGS OFF
IF OBJECT_ID('Tempdb..#Headcount') IS NOT NULL
Drop Table #Test1
Select Coalesce(Enddate,GETDATE()) as EndDate1,FirstName,LastName,EmployeeID,CostCenter,JobCode, CompanyCode
Into #Test1
from EmployeeDM.dbo.vEmployeeJobReporting EJ
--Group By FirstName,LastName,EmployeeID,CostCenter
Order by 1
IF OBJECT_ID('Tempdb..#Headcount') IS NOT NULL
Drop Table #Final1
Select max(EndDate1) as Date1, FirstName,LastName,EmployeeID,CostCenter,JobCode, CompanyCode
Into #Final1
From #Test1
Group by FirstName,LastName,EmployeeID,CostCenter,JobCode, CompanyCode
Order by 1
SELECT F.CostCenter,F.FirstName,F.LastName, F.Date1, F.CompanyCode, F.JobCode,F.EmployeeID,(t3.Day_of_Month-t2.Day_of_Month+1)*1.0/t4.Day_of_Month as Headcount,
Case
The last Select statement is the start of the non-temp table query. What can I do / write in the code to be able to run multiple times in a row? Also, the error I'm receiving:
Msg 2714, Level 16, State 6, Line 4
There is already an object named '#Test1' in the database.
Thanks!
When you write ...INTO #test1, you are creating the table based on the content of the select statement. You need to either 1) drop the temp tables at the end of your query, 2) check for them existing at the front end and drop if they exist, 3) both.
You are already checking for #headcount, but dropping #final and #test1 at the beginning. I dont see where you are declaring #headcount as a table?

how to update table(new_DB) from old table(old_DB)

What I have:
1 table(table is in both DB's)
2 databases(currently used + archived from last year(old))
"ID" is the primary key for the table.
my issue:
archived database table has rows in it that is not present in the currently used database table. Can anyone tell me how I go about updating the currently used database table from the old database table(i.e. insert * unique rows from old database table into new database table)
It sounds simple enough but wanted some advice before proceeding as I DO NOT want duplicate rows, I just want to throw the rows in the old table(that IS NOT present in the currently used database table) into the new one(copy only is fine).
I hope I explained clearly enough.
Insert rows from the new table only if row with same id not exists in old table:
insert into old_table select * from new_table nt
where not exists (select 1 from old_table
where id = nt.id)
(Specifying columns, both inserted and selected, is nice - but I'm lazy here...)
You can usually address tables from other databases by prefixing the database name: new_db.foo_table or old_db.foo_table. This way you can look for rows in the old table that have no duplicates in the new table:
select *
from old_db.foo_table as old_foo
where not exists (
select 1
from new_db.foo_table as new_foo
where new_foo.key_field = old_foo.key_field
-- add more comparisons as needed
);
Then you can use the insert into new_db.foo_table select ... syntax to put the records into the new table.
Use LEFT JOIN filtering NULLs in target table. I think it will be faster
INSERT INTO NEW_TABLE
SELECT ot.* FROM OLD_TABLE ot
LEFT JOIN NEW_TABLE nt on ot.ID = nt.ID
WHERE nt.ID IS NULL

How to compare two schemas and generate a script that transform one schema to the other? [duplicate]

When releasing database code to non-development databases , I use such approach - I create release sqlplus script that runs multiple create table/view/sequence/package/etc statements in a sequence. I also should create rollback script which performs drop and other statements if would be needed during deployment or further use. But it is quite annoying always to create rollback scripts manually. I.E. - when I put
alter table table_a add column some_column number(5);
into release script. I have to put
alter table table_a drop column some_column;
into the rollback script. And vice-versa.
Is there way to optimize(or semi-optimize) it? Maybe some there are some Java/Python/etc libraries that allow to parse ddl statements into logical parts?
Maybe there are some better approaches for release/rollback pl/sql code?
DBMS_METADATA_DIFF and a few metadata queries can automate this process.
This example demonstrates 6 types of changes: 1) adding a column 2) incrementing a sequence 3) dropping a table 4) creating a table 5) changing a view 6) allocating an extent.
create table user1.add_column(id number);
create table user2.add_column(id number);
alter table user2.add_column add some_column number(5);
create sequence user1.increment_sequence nocache;
select user1.increment_sequence.nextval from dual;
select user1.increment_sequence.nextval from dual;
create sequence user2.increment_sequence nocache;
select user2.increment_sequence.nextval from dual;
create table user1.drop_table(id number);
create table user2.create_table(id number);
create view user1.change_view as select 1 a from dual;
create view user2.change_view as select 2 a from dual;
create table user1.allocate_extent(id number);
create table user2.allocate_extent(id number);
insert into user2.allocate_extent values(1);
rollback;
You are correct that DBMS_METADATA_DIFF does not work for CREATE or DROP. Trying to diff an object that only exists in one schema will generate an error message
like this:
ORA-31603: object "EXTRA_TABLE" of type TABLE not found in schema "USER1"
ORA-06512: at "SYS.DBMS_METADATA", line 7944
ORA-06512: at "SYS.DBMS_METADATA_DIFF", line 712
However, dropping and adding objects may be easy to script with the following:
--Dropped objects
select 'DROP '||object_type||' USER1.'||object_name v_sql
from
(
select object_name, object_type from dba_objects where owner = 'USER1'
minus
select object_name, object_type from dba_objects where owner = 'USER2'
);
V_SQL
-----
DROP TABLE USER1.DROPPED_TABLE
--Added objects
select dbms_metadata.get_ddl(object_type, object_name, 'USER2') v_sql
from
(
select object_name, object_type from dba_objects where owner = 'USER2'
minus
select object_name, object_type from dba_objects where owner = 'USER1'
);
V_SQL
-----
CREATE TABLE "USER2"."CREATED_TABLE"
( "ID" NUMBER
) SEGMENT CREATION DEFERRED
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
NOCOMPRESS LOGGING
TABLESPACE "USERS"
The alters can be handled with a SQL statement like this:
select object_name, object_type, dbms_metadata_diff.compare_alter(
object_type => object_type,
name1 => object_name,
name2 => object_name,
schema1 => 'USER2',
schema2 => 'USER1',
network_link1 => 'MYSELF',
network_link2 => 'MYSELF') difference
from
(
select object_name, object_type from dba_objects where owner = 'USER1'
intersect
select object_name, object_type from dba_objects where owner = 'USER2'
) objects;
OBJECT_NAME OBJECT_TYPE DIFFERENCE
----------- ----------- ----------
ADD_COLUMN TABLE ALTER TABLE "USER2"."ADD_COLUMN" DROP ("SOME_COLUMN")
ALLOCATE_EXTENT TABLE -- ORA-39278: Cannot alter table with segments to segment creation deferred.
CHANGE_VIEW VIEW -- ORA-39308: Cannot alter attribute of view: SUBQUERY
INCREMENT_SEQUENCE SEQUENCE ALTER SEQUENCE "USER2"."INCREMENT_SEQUENCE" RESTART START WITH 3
Some notes about these results:
ADD_COLUMN works as expected.
ALLOCATE_EXTENT is probably a false positive, I doubt you care about deferred segment creation. It is very unlikely to affect your system.
CHANGE_VIEW does not work at all. But as with the previous metadata queries, there should be a relatively easy way to build this script using DBA_VIEWS.
INCREMENT_SEQUENCE works too well. Most of the time an application does not care about the sequence values. But sometimes when things get out of sync you need to change them. This RESTART START WITH syntax can be very helpful. You don't need to drop or re-create the indexes, or mess with the increment by multiple times. This syntax is not in the 12c manual. In fact, I cannot find it anywhere on Google. Looks like this package is using undocumented features.
Some other notes:
The package can be very slow sometimes.
If network links on the server are a problem you will need to run it through a local instance with links to both servers.
There may be false positives. Sometimes it returns a row with just a space in it.
It is possible to fully automate this process. But based on the issues above, and my experience with all such automated tools, you should not trust it 100%.

Resources