My setup consists of a master and a slave node connected to a MySQL and Oracle DB, respectively. The slave node already successfully pushes rows added to a table to the master node. However, when I add a column to the source table nothing changes at the target table. So far I figured out that
INSERT INTO `symmetricds`.`sym_table_reload_request` (`target_node_id`, `source_node_id`, `trigger_id`, `router_id`, `create_time`, `create_table`, `delete_first`, `processed`, `last_update_time`)
VALUES ('master', 'client', 'ALL', 'ALL', CURRENT_TIMESTAMP(), '1', '0', '0', CURRENT_TIMESTAMP());
should cause an update of the target schema. However, this only works when I restart the symmetricDS slave node (which sends the data). That is, adding the column, restarting and then performing the insert works and the server's logs confirm that the XML containing the table structure contains the new column. Yet, when I skip the restart the XML shown in the server's logs still misses the new column. Is there a way to make this work without a restart?
With the help of chenson42 I was able to make this work. Let's say you have the folowing trigger for table "my_table"
insert into sym_trigger
(trigger_id,source_catalog_name, source_table_name,channel_id,last_update_time,create_time)
values('my_trigger','my_catalog','my_table','default',current_timestamp,current_timestamp);
Now you add a column:
ALTER TABLE my_catalog.my_table ADD hacky_works varchar(40);
Then, in order to synchronize the chagned table structure with the master note run the following lines:
UPDATE sym_trigger SET last_update_time=CURRENT_TIMESTAMP() where trigger_id="my_trigger";
INSERT INTO sym_table_reload_request (`target_node_id`, `source_node_id`, `trigger_id`, `router_id`, `create_time`, `create_table`, `delete_first`, `processed`, `last_update_time`)
VALUES ('master', 'client', 'ALL', 'ALL', CURRENT_TIMESTAMP(), '1', '0', '0', CURRENT_TIMESTAMP());
Note that in this example 'master' and 'client' are the configured names of the source and target node.
Related
Where can I find breaking changes list from version 2 and/or 3 to version 4 of IdentityServer4?
I am trying to upgrade a project with IdentityServer4 version 2 to version 4.
I made a migration from 2.2 to 4.1.2 version two years ago and I remember I found some breaking changes. Others were imposed by the upgrade of framework I made (net core 2.1 to 3.1). These are some of the changes related only to IdentityServer4:
Database scheme. If you have data in production you'll need to maintain this data in the migration. Automatic migrations will delete and recreate tables mercilessly. There are table and column renames, new columns, deleted columns, changes in the indexes...
Client Cors Origins validation. There is a new validation to force all Url configured in ClientCorsOrigins to complain with an Origin format. If only one of them does not comply with the format, an exception is thrown. You should review your production values to avoid fails.
// format:
<protocol>://<domain>:<port>
// good examples:
http://localhost:5000
http://example.com
https://anotherdomain.com
http://example.com:1234
// bad examples
http://example.com/
https://example.com/mypath
example.com:1234
Some code changes:
AuthorizationRequest.ClientId -> AuthorizationRequest.Client.ClientId.
ResourceValidationResult groups ApiResources and IdentityResources properties in a common property called Resources.
ValidatedTokenRequest changes his property Scopes to RequestedScopes.
GetAllUserConsentsAsync -> GetAllUserGrantsAsync.
In your UI some ModelViews will need to be updated to the new scheme. If you stared with the QuickStart.UI you can compare with the new version to add the new features.
If you have an admin you'll have to adapt it to the new scheme.
Migrations
I created the migrations automatically and then I edited the migration to reorder and to add manual scripts to save the data (For example, create a table before deleting the old one and move the data).
These are the scripts I had to insert manually for the Up migration.
Reorder the code to create ApiResourceScopes table before deleting column ApiResourceId from ApiScopes table.
Insert Into [ApiResourceScopes] ([ApiResourceId], [Scope]) Select [ApiResourceId], [Name] From [ApiScopes]
As ApiScopes has a new field called Enabled and by default takes 0, you'll want to enable for all of them. Run this script just before create the Enabled column:
Update [ApiScopes] set [Enabled] = 1
ApiSecrets must be moved to the new table ApiResourceSecrets. So you should run this script before delete ApiSecrets:
Insert Into [ApiResourceSecrets] ([Description], [Value], [Expiration], [Type], [ApiResourceId], [Created]) Select [Description], [Value], [Expiration], [Type], [ApiResourceId], GetDate() From [ApiSecrets]
The table IdentityClaims is renamed to IdentityResourceClaims. So you'll need to run this script after crate IdentityResourceClaims and before delete IdentityClaims.
Insert Into [IdentityResourceClaims] ([Type], [IdentityResourceId]) Select [Type], [IdentityResourceId] From [IdentityClaims]
For the Down migration you need to do exactly the reverse:
Restore ApiScopes. Move the data from ApiResourceScopes.ApiResourceId by using in the join the Scope and the Name fields respectively.
Update [ApiScopes] Set [ApiScopes].[ApiResourceId] = apir.[ApiResourceId] from [ApiScopes] apis Inner Join [ApiResourceScopes] apir On apis.[Name] = apir.[Scope]
Restore ApiSecrets. Move the data after create ApiSecrets table and before delete ApiResourceSecrets:
Insert Into [ApiSecrets] ([Description], [Value], [Expiration], [Type], [ApiResourceId]) Select [Description], [Value], [Expiration], [Type], [ApiResourceId] From [ApiResourceSecrets]
Restore IdentityClaims. Move the data after create IdentityClaims and before delete IdentityResourceClaims:
Insert Into [IdentityClaims] ([Type], [IdentityResourceId]) Select [Type], [IdentityResourceId] From [IdentityResourceClaims]
Consider the code below in a unit test, where I add a new Tag object in a pre-populated SQLite database.
#Test // Line 1
public void add() {
Tag tagToAdd = new Tag("Tall");
Tag addedTag = this.tagDao.add(tagToAdd);
assertNotNull(addedTag);
assertEquals(3L, addedTag.getId()); // Line 6
assertEquals(tagToAdd.getTag(), addedTag.getTag());
List<Tag> tags = this.tagDao.get();
assertEquals(3, tags.size());
}
On line 6, I expect the ID of the Tag to be 3, because the field is an AUTOINCREMENT and the test is initialized with a database already containing 2 Tags. This works fine every time I run the test and the ID is always 3.
Now, I am integrating flyway to the project. Every time I run the test, the AUTOINCREMENT starts from the value of the last run, so the Tag ID increments by 1 every run, and the test fails.
Any idea on how I can get flyway to always reset the database to a brand new state, and reset the AUTOINCREMENT value ? I could write a query to do it manually, but this is not maintainable.
What I have tried so far ?
Integrate #FlywayTest, as this executes flyway task clean
Defined a FlywayMigrationStrategy bean, which contains flyway.clean()
Set spring.flyway.clean-on-validation-error to true in my application.properties (that said, there was no change in my sql, so not sure if this changed anything)
-- Edit
My 1st migration script contains the below.
DROP TABLE IF EXISTS Tag;
CREATE TABLE Tag(
id INTEGER PRIMARY KEY AUTOINCREMENT,
tag VARCHAR(255) NOT NULL UNIQUE,
createdDate TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
modifiedDate TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
If I understood everything correctly - you have a database and a table in this database which is created once and the same table is used for tests every time - you just delete rows from the table (without removing it) when tests are completed (or before starting next tests) and flyway just inserts two tags into this table every time you run the tests.
If that's right - you can just reset sequence in SQLite to set it back to 1 so next inserted row will be inserted with this id. You can do it by running the following query:
UPDATE `sqlite_sequence` SET `seq` = 1 WHERE `name` = 'tags_table_name';
Alternatively, you can set seq to 0 - this value is incorrect so SQLite will use next available correct value (if there are no rows in the table - it will be one, if there are some values - it will first available number).
Yet another possibility is just to delete your table after tests and recreate it before running next tests - as it is a database and table just for tests - it should work correctly. This way you have your sequence counter set back to value 1 each time. I would actually go this way until you have really good reason not to delete the table.
I have a Spring Boot application and would like to seed the database the first time the application runs, but not every time the application runs, and then only if the data does not already exist.
My application has a data.sql file and I have it inserting the default users:
-- insert the administrator
INSERT INTO users(id, username, password_hash, email, first_name, last_name) VALUES
(1, 'admin', 'comixed', 'email1#domain.com', 'ComixEd', 'Administrator'),
(2, 'user', 'comixeduser', 'email2#domain.com', 'ComixEd', 'User')
;
-- insert the supported roles
INSERT INTO roles(id, name) VALUES
(1, 'Administrator'),
(2, 'User')
;
-- set the administrator roles
INSERT INTO users_roles(user_id, role_id) VALUES
(1, 1),
(1, 2),
(2, 2)
;
But Spring is obviously trying to run this file every time I start the app. And when it does an exception is raised since the users and roles are already in the database.
What's a better way to do this? And, optionally, what's a way to add new seed data if, in future, new features require new roles, etc.?
Flyway is what you are looking for https://flywaydb.org/
It is a database migration tool that can be used to create and modify databases in a project. It creates its own table in your schema and adds the script name and a checksum value into the tables. During application boot it scans the scripts, checks to see if they're in the table and if the checksums match still. This makes sure the files don't change and everything has been migrated.
You have to configure it in the application.properties file.
Add this line and it should work as you wish:
spring.jpa.hibernate.ddl-auto = update
I am attempting to insert a record using the copyFrom('POST') and save() methods of fatfreeframework v3.5. The data from POST does not contain an id field which for this table is set as an autoincrement. The SQL from the logs is
SET IDENTITY_INSERT [xrefs] ON;
INSERT INTO [xrefs] ([status], [supply_id], [description], [unit], [unitcost], [cap], [rev], [buq])
VALUES ('test', 'Htest', 'test', 'test', '1', '1', 1, 1)
As you can see fatfree is adding the set identity insert despite the fact there is no id column included in the insert. Is there a way to tell mapper not to set this flag? Or is there another workaround? I could get the current max ID and then insert +1 but that seems clunky.
I should add this SQL fails because the id column is not included in the columns list.
$this->db->exec(
(preg_match('/mssql|dblib|sqlsrv/',$this->engine) &&
array_intersect(array_keys($pkeys),$ckeys)?
'SET IDENTITY_INSERT '.$this->table.' ON;':'').
'INSERT INTO '.$this->table.' ('.$fields.') '.
'VALUES ('.$values.')',$args
);
This is the code that sets IDENTITY_INSERT in mapper.php function insert.
$this->logger->write( 'xrefs schema:'.
json_encode( $this->tongpodb->schema( 'xrefs' ) ) );
Calling schema on the the db object gives back this array
{"id":{"type":"int","pdo_type":1,"default":null,"nullable":false,"pkey":true},"changed_date":{"type":"datetime","pdo_type":2,"default":null,"nullable":true,"pkey":false},"status":{"type":"varchar","pdo_type":2,"default":null,"nullable":false,"pkey":false},"supply_id":{"type":"varchar","pdo_type":2,"default":null,"nullable":false,"pkey":true},"description":{"type":"varchar","pdo_type":2,"default":null,"nullable":true,"pkey":false},"unit":{"type":"varchar","pdo_type":2,"default":null,"nullable":false,"pkey":false},"hcpcs":{"type":"char","pdo_type":2,"default":null,"nullable":true,"pkey":false},"unitcost":{"type":"decimal","pdo_type":2,"default":null,"nullable":false,"pkey":false},"cap":{"type":"decimal","pdo_type":2,"default":null,"nullable":false,"pkey":false},"rev":{"type":"smallint","pdo_type":1,"default":null,"nullable":false,"pkey":false},"buq":{"type":"smallint","pdo_type":1,"default":null,"nullable":true,"pkey":false},"create_ts":{"type":"datetime","pdo_type":2,"default":null,"nullable":true,"pkey":false},"log_ts":{"type":"int","pdo_type":1,"default":null,"nullable":true,"pkey":false},"filename":{"type":"varchar","pdo_type":2,"default":null,"nullable":true,"pkey":false},"line_no":{"type":"smallint","pdo_type":1,"default":null,"nullable":true,"pkey":false},"file_ts":{"type":"datetime","pdo_type":2,"default":null,"nullable":true,"pkey":false}}
As you can see id has a "pkey":true entry so one could look at the fields from post then look at this and determine if IDENTITY_INSERT needs to set. Perhaps I will implement this. I worry this is above my paygrade.
Updated to the latest version of fatfree fixed this issue.
I want to change name of the system table in my database is it possible? Probably I shouldn't change it but I'm curious.
When I execute sp_rename I get the following error:
Msg 15001, Level 16, State 1, Procedure sp_rename, Line 404
Object 'cdc.[dbo_CdcTest_CT]' does not exist or is not a valid object for this operation.
Edit:
I want to change name of tables created by Change Data Capture because I want to disable CDC mechanism for table and still have data - I know that I can create additional table and move there data from CDC table but it's easier to change name of the CDC and then disable cdc for specified table.
No you cannot change the name of the system tables. However you can refer it with a different name.
You can use synonyms for that:
CREATE SYNONYM [ schema_name_1. ] synonym_name FOR <object>
<object> :: =
{
[ server_name.[ database_name ] . [ schema_name_2 ].| database_name . [ schema_name_2 ].| schema_name_2. ] object_name
}
Also to mention that sp_rename
Changes the name of a user-created object in the current database.
This object can be a table, index, column, alias data type, or
Microsoft .NET Framework common language runtime