In Oracle Golden Gate, I'm unable to replicate production sequence to replicate database, since as sequence increased by 1 in production, the count of sequence in target increasing by 2.
Let me elaborate, suppose I have sequence with currval 190, assume after initail load, target sequence also have currval 190.
Now I booked a deal and sequence no get increased by 1 in production, currval is 191 but when i checked in target db, sequence currval showing 192. This creating issue. Need help in resolving this...
Did you follow below procedure for your replicate:
1. running sequence.sql in oracle sqlplus.
2.ALTER TABLE sys.seq$ ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
There are a couple of scenarios when this can happen.
Scenario 1: If the replication setup is a bi-directional replication then sequences are kept at sequnce+1 value on target database. This is done so that just in case a failover or switchover has to happen from source to target database then there will be no need to reset the sequence number to a higher value. Check with your Golden Gate DBA to get more details on how sequences are being maintained.
Scenario 2: In Bi-directional replication with conflict detection and resolution-sequences are maintained so that they can be uniquely identified.
Eg:
Primary site will have sequences which are always ODD and standby site will always have sequences with even numbers. By doing this you will be able to clearly identify on which database sequence has increased.
Related
I faced today strange case when receiving customer database for investigation.
System settings:
Firebird server v 2.5.9.26074
Firebird client v 2.6.5
Database file is accessed directly by the application, i.e., it is NOT registered via aliases.conf.
When I first looked into database, everything seemed to be pretty consistent. However, during the first startup there are two rows added in certain table without any detected SQL execution. I have confirmed with debugger that the application is not adding these rows. I also used Audit and Trace inferface (fbtracemgr) and saw in log file that there are not such rows added to the database.
There is one hint that something is wrong in the original database. The table that contains the problem is using INSERT trigger to set the table row's ID column value from generator. Now the generator value seem to be one too high in the original database. This leads me to think that the "ghost data" has already been entered in the file in some sort of cache as the generator is already increment by one.
The result is that after these the two ghost rows are added, the next real addition to the table leads into exception:
FirebirdSql.Data.FirebirdClient.FbException (0x80004005): violation of
PRIMARY or UNIQUE KEY constraint "INTEG_275" on table "DATALOG" --->
violation of PRIMARY or UNIQUE KEY constraint "INTEG_275" on table
"DATALOG"
as there already exist row with equal ID that the generator suggests.
Is there persistent "unsaved data cache" that could contain row data entered during the previous application runs? What could lead to this situation? Power break during database writing or backuping?
Any thoughts?
Firebird server v 2.5.9.26074
There is no such version released.
Firebird-2.5.8.27089
http://www.firebirdsql.org/en/firebird-2-5/
Basically u seem to use some destabilized FB developers internal build, which can have any number of strange averse effects.
So I would advice to use standard released verison or if using snapshot builds is required for some untold reasons - to ask developers in firebird-support mail list - http://www.firebirdsql.org/en/support/
Though don't hold your breath for much of support over exotic Firebird builds.
UPD. Thanks to Mark, here it is: https://www.firebirdsql.org/en/firebird-2-5-0/
2.5.0 - was the first release after a significant reworking of the engine. Not the most stable, obviously. For example there was an issue with indices right in the next 2.5.1 version.
if the behavior would be repeated on standard 2.5.8 Firebird, then i would suggest exporting all the database (at least all the meta-data, but maybe the data as well) into a long text file, SQL script, and then searching for the said table name in it. For example there might be on-database-connect triggers adding some data. Or stored procedures. Or views made on triggers. Or something yet else. For example - though malpractice - even UDF function may make it's own database connection and do things, though this should be shown in FBTrace.
However, during the first startup there are two rows added in certain table
startup of what ?
will those rows still be added if you use standard tools like iSQL/FlameRobin/IBExpert/etc just to connect and then disconnect from the database?
as there already exist row with equal ID that the generator suggests
Generator can not suggest things like that. It can only suggest that once such a number was reserved for possibly being added to one or another table. It does not mean the row was actually inserted, was inserted into that table, was not deleted later.
You may try to search with indices prohibited, in case index corruption could occur, something like
select id+0, count(*) from tableName group by 1
Also http://www.firebirdfaq.org/faq324/
when receiving customer database for investigation
BTW, how exactly did they created a copy of the database to give you?
Did they made back-up (FBK) ? If not, did they stopped Firebird server before making copies?
We would like to synchronize data (insert, update) from Oracle (11g) to PostgreSQL (10). Our approach was the following:
A trigger on the table in Oracle updates a column with nextval from a sequence before insert and update.
PostgreSQL knows the last sequence number processed and fetches the rows from Oracle > lastSequenceNumberFetched.
We now have the following problem:
Session 1 in Oracle inserts a row, sequence number (let's say 45) is written but no COMMIT is done in Oracle.
Session 2 in Oracle inserts a row, sequence number is written (let's say 49 (because sequences in Oracle can have gaps)) and a COMMIT is done in Oracle.
Session in PostgreSQL fetches rows from Oracle with sequenceNumber > 44 (because the lastSequenceNumberFetched is 44) and gets the row with sequenceNumber 49. So this is the new lastSequenceNumberFetched.
Session 1 in Oracle makes a commit.
Session in PostgreSQL fetches rows from Oracle with sequenceNumber > 49. Problem is that the row with sequenceNumber 45 is never fetched.
Are there any better approaches for our use case avoiding our problem with missing data?
In case you don't have delete operations in your tables and the tables are not very big then I suggest to use Oracle System Change Number (SCN) on the row level which is returned by the pseudo column ORA_ROWSCN (link). This is the commit time presented by number. By default the SCN is tracked for the data block, but you can enable tracking on the row level (keyword rowdependencies). So you have to recreate your table with this keyword. At the sync procedure launch you get the current scn by the function call dbms_flashback.get_system_change_number, then scan all tables where ora_rowscn between _last_scn_value_ and _current_scn_value_. The disadvantage is that this pseudo columns is not indexed, so you will have full table scans, which is slow for big tables.
If you use delete statements then you have to track the records which were deleted. For this purpose you can use one log table having the following columns: table_name, table_id_value, operation (insert/update/delete). Table is filled by the trigger on base tables. So for your case when session 1 commits data in base table - then you have the record in log table to process. And you don't see it until the session commits. So no issues with sequence numbers that you described.
Hope that helps.
Is this purely a data project or do you have some client here. If you do have a middle tier you could use an ORM to abstract some of this and do writes to both. Do you care whether the sequences are the same? It would be possible to do something like collect all the data to synchronize since a particular timestamp (every table would have to have a UTC timestamp) and then take a hash of all the data and compare with what is in Postgres.
It might be useful to have some more of your requirements for the synchronization of data and the reasoning behind this e.g.
Do the keys need to be the same against both environments? Why?
Who views the data, is the same consumer looking at both sources.
Why wouldn't you just use an ORM to target only one db why do you need oracle and postgres?
I have seen a similar setup. An application on Postgres mostly for reporting and other secondary tasks while main app was on Oracle.
Some of the main app tables are cached in Postgres for convenience. But this setup brings in the sync problem.
The compromise solution was a mix of incremental sequence-based sync during daytime and full table copy overnight
Regarding other solutions proposed here:
Postgres fdw is slow for complex queries and it puts extra load on foreign db especially when where clause refer to both local and foreign tables.
The same query will run much faster if foreign table is cached in postgres.
Incremental/differential sync using sequence numbers -tried this and works acceptable for small tables, but the nightmare starts with child relations maybe an orm can help here
The ideal solution in my opinion would probably be to stream Oracle changes to Postgres or intermediary process that replicates changes to Postgres
I have no clue about how to do this as I understood it requires Oracle golden gate app (+ licence)
Most of my databases use IDENTITY columns as primary keys. I am writing a change log/audit trail in the database and want to use the ID BIGINT to keep trap of the changes sequentially.
While BIGINT is pretty big, it will run out of numbers one day and my design will fail to function properly at that point. I have been aware of this problem with my other ID columns and intended to eventually convert to GUIDs/UUIDs as I have used on Postgres in the past.
GUIDs take 16 bytes and BIGINT takes 8. For my current task, I would like to stay with BIGINT for the space savings and the sequencing. Under Postgres, I created a custom sequence with the first two digits as the current year and a fixed number of digits as the sequence within the year. The sequence generator automatically reset the sequence when the year changed.
SQL Server 2008 has no sequence generator. My research has turned up some ideas most of which involve using a table to maintain the sequence number, updating that within a transaction, and then using that to assign to my data in a separate transaction.
I want to write an SP or function that will update the sequence and return me the new value when called from a trigger on the target table before a row is written. There are many ideas but all seem to talk about locking issues and isolation problems.
Does anyone have a suggestion on how to automate this ID assignment, protect the process from assigning duplicates in a concurrent write, and prevent lock latency issues?
The stored procedure is prone to issues like blocking and deadlocks. However, there are ways around that.
For now, why not start the ID off at the bottom of the negative range?
CREATE TABLE FOO
(
ID BIGINT IDENTITY(-9223372036854775808, 1)
)
That gives you a range from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.
Are you really going to eat up 2^63 + 2^63 numbers?
If you are still committed to the other solution, I can give you a piece of working code. However, application locks and Serializable isolation has to be used.
It is still prone to timeouts or blocking depending upon the timeout setting and the server load.
In short, 2012 introduced sequences. That is basically what you want.
When I try to insert a new row in the table, if there is any exception occurs in this transaction then data is rollback.
Now when a new entry is succesfully inserted next time, AutoIncrement id is updated with next value. Means there is Gap between two consequetive Unique Id in the table.
Is there any valid way to overcome this problem?
Thanks in advance
The answer has to be said - no.
The whole idea of IDENTITY columns is to not be meaningful, and to be transaction agnostic - so that the numbers can be dished out without care of other transactions rolling back or not. Imagine a 1000 insert per second system being held up for 10ms for each transaction (insert) to decide whether it will commit! (fyi 10ms * 100 = 1s)
Note: In SQL Server 2012 (latest SP/patch level at time of writing), there is a "feature" noted here on Connect related to identities.
Also even prior to 2012, you don't even need to rollback to consume an IDENTITY value - see here https://stackoverflow.com/a/16156419/573261
This applies to other major RDBMS as well for the same reasons, e.g.
PostgreSQL sequences
Important: To avoid blocking concurrent transactions that obtain numbers from the same sequence, a nextval operation is never rolled back; that is, once a value has been fetched it is considered used, even if the transaction that did the nextval later aborts. This means that aborted transactions might leave unused "holes" in the sequence of assigned values.
(emphasis mine)
I am working on a new system with a SQL Server 2005 database which will soon be going into production. A colleague recently mentioned to me that I should always be specifying the fill factor on my tables. Currently I don't specify fill factor on any of my tables.
My application is OLTP with a mix of reads and writes. A couple of my tables are "reference" tables i.e. read-only but most are read-write. The read-only tables are low volume ( < 50000 rows ).
From what I've read in the SQL Server documentation I should be sticking with the default fill-factor unless the table is read only.
Can anyone comment on this, both for read-only and read-write tables?
No, you shouldn't specify a fill factor on your tables. The fill factor is always ignored by the engine, except for one and only one operation: index build (which includes initial build of an index on a populated table and/or a rebuild of an index). So the fill factor makes sense to be specified only in the ALTER TABLE ... REBUILD and ALTER INDEX ... REBUILD operations.
See also A SQL Server DBA myth a day: (25/30) fill factor.