I have two databases: db1 and db2 (db2 was completely empty). I was copying all the db1 to db2 but the progress was interrupted and I need to know which tables are still left to copy. How can I compare the count of each table in these two databases to know which tables I still have to transfer?
Basically, you need to loop through the data dictionary and generate some dynamic SQL which executes a count for each table.
I have assumed you're only transferring one schema. If that's not true, or you're not connecting as the target schema, you'll need to use ALL_TABLES instead of USER_TABLES, and include the OWNER column in the driving query and the dynamic query too.
declare
n pls_integer;
stmt varchar2(32767);
begin
for r in ( select table_name from user_tables order by table_name ) loop
stmt := 'select count(*) from ' || r.table_name;
-- uncomment the next line to debug errors
-- dbms_output.put_line(stmt);
execute immediate stmt into n;
-- you may wish to only display empty tables
-- if n = 0 then
dbms_output.put_line(r.table_name || ' = ' || lpad(n, 10));
-- end if;
end loop;
end;
One would hope that your data copying process was clever enough to commit only completed tables. If so you only need to run this on DB2. Otherwise on both.
Related
Our company is in planning to move a Postgres database to SQL Server. This includes all tables, functions and stored procedures etc.
Here is the Postgresql syntax:
CREATE or REPLACE FUNCTION ex.on_update_integrity_reset()
RETURNS trigger
LANGUAGE plpgsql
AS
$$
BEGIN
new."integrity_error_id" := 0 ;
new."requires_processing" := True ;
new."datetime_amended" := now();
return new ;
END;
$$;
I have tried the following conversion BUT no luck I am afraid. I am hoping that the solution is quite straight forward. Any assistance will be gratefully received.
My T-SQL syntax, which isn't working:
CREATE FUNCTION ex.on_update_integrity_reset()
RETURNS TABLE
WITH SCHEMABINDING
AS
BEGIN atomic
new."integrity_error_id" := 0 ;
new."requires_processing" := True ;
new."datetime_amended" := GETDATE();
return new ;
END;
There are a huge amount of differences between Postgres' pgSQL and SQL Server's T-SQL, triggers not the least of them. You must learn the differences from the documentation, rather than dumping in code and expecting it to just "work".
The key things to note about triggers in SQL Server:
Triggers are directly defined on the table, rather than being separate functions which you can call.
Triggers are run per-statement not per-row, and the pseudo-tables inserted and deleted may contain multiple or zero rows.
SET NOCOUNT ON is ideal, due to problems with certain client drivers.
Do not return resultsets from triggers, instead make any updates, inserts or deletes you wish, taking into account the pseudo-tables.
Here is an example for your use case:
CREATE TRIGGER dbo.on_update_integrity_reset ON dbo.YourTable
AFTER INSERT -- what about updates???
AS
SET NOCOUNT ON;
IF NOT EXISTS (SELECT 1 FROM inserted)
RETURN; -- early bail-out
UPDATE t
SET
integrity_error_id = 0,
requires_processing = 1, -- boolean type is not supported
datetime_amended = GETDATE()
FROM dbo.YourTable t
JOIN inserted i ON i.SomePrimaryKey = t.SomePrimaryKey;
Be that as it may, you probably don't actually want a trigger. Instead, you probably need DEFAULT constraints. Although there are other methods for an auto-updating datetime column.
ALTER TABLE dbo.YourTable
ADD DEFAULT 0 FOR integrity_error_id;
ALTER TABLE dbo.YourTable
ADD DEFAULT 1 FOR requires_processing;
ALTER TABLE dbo.YourTable
ADD DEFAULT GETDATE() FOR datetime_amended;
One of our customer have one Oracle 10.2.0.5 RAC (HPUX) and two SQL Server 2012 (Windows server 2008R2). And we are helping them to publish data from Oracle to SQL Server. They also need to know which rows are added, updated and deleted, but they do not want to modify their apps.
The whole thing works like this :
Oracle as publisher -> SQL Server A as distributor -> SQL Server B as
subscriber
Our DBA had configured all the DBs through SSMS(SQL Server Management Studio) like this Create a Publication from an Oracle Database. It worked very well for several days. But the performance of Oracle is getting worse and worse. At last, we have to stop the data publish of Oracle.
It turns out that, SSMS will create one package called "HREPL" in Oracle, which has a procedure called "PollEnd". "PollEnd" will be executed in a very high frequency to delete data in table "HREPL_ARTICLE1LOG_1". But the execution time of "PollEnd" increases through time. At last, the execution time is longer than the time span to execute, and the table is locked, and the performance of Oracle will be very bad.
And we stuck here.
Does anybody have any idea how to fix this? Please help!
The "PollEnd" procedure:
-----------------------------------------------------------------------------------
--
-- Name: PollEnd
-- Purpose: PollEnd request signifies that the change entries identified with the current
-- interval have been successfully entered into the store and forward database
-- and can be deleted from the article log tables.
-- Input:
-- argLSN IN RAW(10) LSN from distributor that was associated
-- with this poll interval
-- Output:
-- Notes: This request causes those entries of the article log tables represented in the
-- Poll Table and having the current pollid to be deleted from both their article log
-- tables and from the Poll Table. The last request value is updated to reflect a
-- PollEnd request.
--
-----------------------------------------------------------------------------------
PROCEDURE PollEnd
(
argLSN IN RAW
)
AS
SQLCommand VARCHAR2(500);
LogTable VARCHAR2(255);
CurrentPollID NUMBER;
TableIDs number_tab;
InstanceIDs number_tab;
IDCount BINARY_INTEGER;
PublisherLSN RAW(10);
BEGIN
-- Put the published tableIDs in a PL/SQL table of IDs
HREPL.GetTableIDs(TableIDs, InstanceIDs);
-- Get the current Poll ID
SELECT Publisher_CurrentPollid INTO CurrentPollID FROM HREPL_Publisher;
IDCount := TableIDs.COUNT;
-- For each table represented in the ID list
FOR id_ind IN 1 .. IDCount
LOOP
LogTable := REPLACE( REPLACE(ArticleLogTemplate, MatchString, TO_CHAR(TableIDs(id_ind))),
MatchStringY, TO_CHAR(InstanceIDs(id_ind)));
BEGIN
-- Generate command to delete from the article log those entries appearing in the
-- Poll Table with the current PollID
SQLCommand := 'DELETE FROM ' || LogTable || ' l ' ||
'WHERE EXISTS (SELECT p.POLL_POLLID FROM HREPL_POLL p ' ||
' WHERE CHARTOROWID(l.ROWID) = p.Poll_ROWID ' ||
' AND p.Poll_PollID = :Pollid)';
HREPL.ExecuteCommandForPollID(SQLCommand, CurrentPollID);
EXCEPTION
WHEN OTHERS THEN NULL;
END;
END LOOP;
FOR POLLID IN (SELECT CurrentPollid FROM DUAL)
LOOP
-- Delete from HREPL_Event those entries appearing in the Poll Table
-- with the current PollID.
DELETE FROM HREPL_Event e
WHERE EXISTS (SELECT p.POLL_POLLID FROM HREPL_POLL p
WHERE CHARTOROWID(e.ROWID) = p.Poll_ROWID
AND p.Poll_PollID = POLLID.CurrentPollID);
-- Delete entries from the Poll Table having the current Pollid
DELETE FROM HREPL_Poll
WHERE Poll_PollID = POLLID.CurrentPollID;
END LOOP;
-- Drop all views associated with articles that are marked as UnPublishPending.
-- Note: We cannot perform these drops in UnPublish table, since UnPublish
-- table can execute concurrently with PollBegin and the querying
-- of published tables by the log reader. PollEnd, however, executes
-- synchronously with respect to these activities, so can be used
-- to cleanup log tables and views that are no longer needed.
HREPL.CleanupLogsandViews;
-- Mark the last request as PollEnd, and update the Publisher LSN
-- to reflect the LSN committed at the publisher.
UPDATE HREPL_Publisher
SET Publisher_PollInProcess = NoPollInProcess,
Publisher_LSN = argLSN;
-- Commit transaction
COMMIT;
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
RAISE;
END PollEnd;
Edit 01:
The full package is here: HREPL
Edit 02:
At last we give up. MS and Oracle blame each other.
We tried to use ogg to copy data from oracle to sql server, which is a mess too.
Now we are trying to use ogg to copy data from oracle to oracle.
Thanks for all the help.
Convert your DELETE ... WHERE EXISTS (...) queries to use multiple table delete syntax.
SQLCommand := 'DELETE l' ||
' FROM HREPL_POLL, ' || LogTable ||
' l WHERE CHARTOROWID(l.ROWID) = p.Poll_ROWID ' ||
' AND p.Poll_PollID = :Pollid)';
Create a function index on each table involved:
CREATE INDEX MYTABLE_CHARTOROWID ON MYTABLE(CHARTOROWID(ROWID));
And then further down:
DELETE e
FROM HREPL_POLL p, HREPL_Event e
WHERE CHARTOROWID(e.ROWID) = p.Poll_ROWID
AND p.Poll_PollID = POLLID.CurrentPollID;
Finally, delete entirely the LOOP over dual - it does absolutely nothing whatsoever; just execute the code inside it using CurrentPollid directly.
Some join conditions seem unreasonable, you might be more lucky with this version - if you try it on production, you do that on your very own risk!
-----------------------------------------------------------------------------------
--
-- Name: PollEnd
-- Purpose: PollEnd request signifies that the change entries identified with the current
-- interval have been successfully entered into the store and forward database
-- and can be deleted from the article log tables.
-- Input:
-- argLSN IN RAW(10) LSN from distributor that was associated
-- with this poll interval
-- Output:
-- Notes: This request causes those entries of the article log tables represented in the
-- Poll Table and having the current pollid to be deleted from both their article log
-- tables and from the Poll Table. The last request value is updated to reflect a
-- PollEnd request.
--
-----------------------------------------------------------------------------------
PROCEDURE PollEnd
(
argLSN IN RAW
)
AS
SQLCommand VARCHAR2(500);
LogTable VARCHAR2(255);
CurrentPollID NUMBER;
TableIDs number_tab;
InstanceIDs number_tab;
IDCount BINARY_INTEGER;
PublisherLSN RAW(10);
BEGIN
-- Put the published tableIDs in a PL/SQL table of IDs
HREPL.GetTableIDs(TableIDs, InstanceIDs);
-- Get the current Poll ID
SELECT Publisher_CurrentPollid INTO CurrentPollID FROM HREPL_Publisher;
IDCount := TableIDs.COUNT;
-- For each table represented in the ID list
FOR id_ind IN 1 .. IDCount
LOOP
LogTable := REPLACE( REPLACE(ArticleLogTemplate, MatchString, TO_CHAR(TableIDs(id_ind))),
MatchStringY, TO_CHAR(InstanceIDs(id_ind)));
BEGIN
-- Generate command to delete from the article log those entries appearing in the
-- Poll Table with the current PollID
SQLCommand := 'DELETE FROM ' || LogTable || ' l ' ||
'WHERE l.ROWID IN (SELECT chartorowid(p.Poll_ROWID) FROM HREPL_POLL p ' ||
' WHERE p.Poll_PollID = :Pollid)';
HREPL.ExecuteCommandForPollID(SQLCommand, CurrentPollID);
EXCEPTION
WHEN OTHERS THEN NULL;
END;
END LOOP;
-- Delete from HREPL_Event those entries appearing in the Poll Table
-- with the current PollID.
DELETE FROM HREPL_Event e
WHERE ROWID in (SELECT chartorowid(p.Poll_ROWID) FROM HREPL_POLL p
WHERE p.Poll_PollID = CurrentPollID);
-- Delete entries from the Poll Table having the current Pollid
DELETE FROM HREPL_Poll
WHERE Poll_PollID = CurrentPollID;
-- Drop all views associated with articles that are marked as UnPublishPending.
-- Note: We cannot perform these drops in UnPublish table, since UnPublish
-- table can execute concurrently with PollBegin and the querying
-- of published tables by the log reader. PollEnd, however, executes
-- synchronously with respect to these activities, so can be used
-- to cleanup log tables and views that are no longer needed.
HREPL.CleanupLogsandViews;
-- Mark the last request as PollEnd, and update the Publisher LSN
-- to reflect the LSN committed at the publisher.
UPDATE HREPL_Publisher
SET Publisher_PollInProcess = NoPollInProcess,
Publisher_LSN = argLSN;
-- Commit transaction
COMMIT;
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
RAISE;
END PollEnd;
My table has several columns that have similar names, such as year_1, year_2, year_3, etc. Is there an easy way to list them all without typing all of them? I cannot use select *, because I don't want the other columns. Thanks.
You'll have to use dynamic nzplsql in a stored procedure. Stored procedures don't easily return tables, though, but you can have it easily output a select statement that builds from _v_relation_column via a cursor. Capture that in bash and feed it to nzsql to select from a table. Either that or you can just return a reftable.
/* Stored procedure header. */
declare sql varchar;
declare col record;
begin_proc
sql := 'select ';
for col in select * from _v_relation_column where name = 'TABLE_NAME' loop
if col.attname like 'year%'
sql := sql || attname || ',';
end if;
sql := substring(sql,1,length(sql)-1); --To strip the last comma. Could probably be more elegant.
sql := sql || ' from table_name;';
raise notice '%',sql;
end_proc;
/* Stored procedure footer. */
I have a trigger, but I need to associate with all tables of the my postgres.
Is there a command like this below?
CREATE TRIGGER delete_data_alldb
BEFORE DELETE
ON ALL DATABASE
FOR EACH ROW
EXECUTE PROCEDURE delete_data();
Well there is no database-wide trigger creation but for all such bulk-admin-operations you could use PostgreSQL system tables to generate queries for you instead of writing them by hand.
In this case you could run:
SELECT
'CREATE TRIGGER '
|| tab_name
|| ' BEFORE DELETE ON ALL DATABASE FOR EACH ROW EXECUTE PROCEDURE delete_data();' AS trigger_creation_query
FROM (
SELECT
quote_ident(table_schema) || '.' || quote_ident(table_name) as tab_name
FROM
information_schema.tables
WHERE
table_schema NOT IN ('pg_catalog', 'information_schema')
AND table_schema NOT LIKE 'pg_toast%'
) tablist;
This will get you set of strings which are SQL commands like:
CREATE TRIGGER schema1.table1 BEFORE DELETE ON ALL DATABASE FOR EACH ROW EXECUTE PROCEDURE delete_data();
CREATE TRIGGER schema1.table2 BEFORE DELETE ON ALL DATABASE FOR EACH ROW EXECUTE PROCEDURE delete_data();
CREATE TRIGGER schema1.table3 BEFORE DELETE ON ALL DATABASE FOR EACH ROW EXECUTE PROCEDURE delete_data();
CREATE TRIGGER schema2.table1 BEFORE DELETE ON ALL DATABASE FOR EACH ROW EXECUTE PROCEDURE delete_data();
CREATE TRIGGER schema2."TABLE2" BEFORE DELETE ON ALL DATABASE FOR EACH ROW EXECUTE PROCEDURE delete_data();
...
etc
You just need to run them at once (either by psql or pgAdmin).
Now some explanation:
I select names of tables in my database using information_schema.tables system table. Because there are data of literally all tables, remember to exclude pg_catalog and information_schema schemas and toast tables from your select.
I use quote_ident(text) function which will put string inside double quote signs ("") if necessary (ie. names with spaces or capital letters require that).
When I have list of tables names I just concatenate them with some static strings to get my SQL commands.
I write that command using sub-query because I want you to get better idea of what's going on here. You may write a single query by putting quote_ident(table_schema) || '.' || quote_ident(table_name) in place of tab_name.
A conveniently encapsulated version of Gabriel's answer. This time I am using the trigger to update a column named update_dt datetime granted to be part of any table in the public schema of the current database.
--
-- function: tg_any_update_datetime_fn
-- when: before insert or update
--
create or replace function tg_any_update_datetime_fn ()
returns trigger
language plpgsql as $$
begin
new.update_dt = now();
return new;
end;
$$;
--
-- function: ddl_create_before_update_trigger_on_all_tables
-- returns: Create a before update trigger on all tables.
--
create or replace procedure ddl_create_before_update_trigger_on_all_tables ()
language plpgsql as $$
declare
_sql varchar;
begin
for _sql in select concat (
'create trigger tg_',
quote_ident(table_name),
'_before_update before update on ',
quote_ident(table_name),
' for each row execute procedure tg_any_update_datetime_fn ();'
)
from
information_schema.tables
where
table_schema not in ('pg_catalog', 'information_schema') and
table_schema not like 'pg_toast%'
loop
execute _sql;
end loop;
end;
$$;
-- create before update trigger on all tables
call ddl_create_before_update_trigger_on_all_tables();
On my DDL scripts I use a large number of such ddl_ functions that have only meaning at DDL time. To remove them from the database use
--
-- function: ddl_drop_ddl_functions
-- returns: Drop all DDL functions.
-- since: 1.1.20
--
create or replace procedure ddl_drop_ddl_functions ()
language plpgsql as $$
declare
r record;
_sql varchar;
begin
for r in
select oid, prokind, proname
from pg_proc
where pronamespace = 'public'::regnamespace
and proname ilike 'ddl_%'
loop
case r.prokind
when 'a' then _sql = 'aggregate';
when 'p' then _sql = 'procedure';
else _sql = 'function';
end case;
_sql = format('drop %s %s', _sql, r.oid::regprocedure);
execute _sql;
end loop;
end
$$;
Quick question about cursors (in particular Oracle cursors).
Let's say I have a table called "my_table" which has two columns, an ID and a name. There are millions of rows, but the name column is always the string 'test'.
I then run this PL/SQL script:
declare
cursor cur is
select t.id, t.name
from my_table t
order by 1;
begin
for cur_row in cur loop
if (cur_row.name = 'test') then
dbms_output.put_line('everything is fine!');
else
dbms_output.put_line('error error error!!!!!');
exit;
end if;
end loop;
end;
/
if I, while this is running, run this SQL:
update my_table
set name = 'error'
where id = <max id>;
commit;
will the cursor in the PL/SQL block pick up that change and print out "error error error" and exit? or will it not pick up the change at all ... or will it even allow the update to my_table?
thanks!
A cursor effectively runs a SELECT and then lets you iterate over the result set, which is kept in a snapshot of the DB state. Because your result set has already been fetched, it won't be affected by the UPDATE statement. (Handling things otherwise would require you to re-run the query every time you advanced your cursor!)
See:
http://www.techonthenet.com/oracle/cursors/declare.php