I have the following use case:
A database A (Master) and a database B (slave), located on diferent machines.
I want to synchronize the Database A with Database B.
I want to create a java application using SymmetricDS embedded.
As there is no documentation on how to perform this, i want a sample example or a documentation .
Please help me I'm stuck.
this an example how run the Symmetric engine server in embedded mode , and it works perfectley for me :
public class ClientNode {
private ClientSymmetricEngine cEngine;
private File propFile;
public ClientNode(File file) throws FileNotFoundException, IOException {
propFile = file;
Properties propertiesFile = new Properties();
propertiesFile.load(new FileReader(propFile));
cEngine = new ClientSymmetricEngine(propertiesFile, true);
getcEngine().openRegistration("client", "001");// client is the name of the node group and 001 is the ID
getcEngine().setup();
getcEngine().start();
}
public ClientSymmetricEngine getcEngine() {
return cEngine;
}
public void setcEngine(ClientSymmetricEngine cEngine) {
this.cEngine = cEngine;
}
}
Main class :
public static void main(String[] args) {
try {
new ClientNode(new File("client.properties"));
SymmetricWebServer node = new SymmetricWebServer("master.properties");
node.setWebAppDir("Web");
node.setJoin(false);
node.start();
// this will stop the node
//node.stop();
}catch (Exception e) {
e.printStackTrace();
}
}
Properties files :
client.properties :
external.id=001
engine.name=client-001
sync.url=http\://localhost\:31415/sync/client-001
group.id=client
db.url=jdbc\:mysql\://localhost/easyexchangedb_slave
db.driver=com.mysql.jdbc.Driver
db.user=root
registration.url=http\://localhost\:31415/sync/server
db.password=
master.properties :
external.id=server
engine.name=server
sync.url=http\://localhost\:31415/sync/server
group.id=server
db.url=jdbc\:mysql\://localhost/easyexchangedb_master
db.driver=com.mysql.jdbc.Driver
db.user=root
registration.url=http\://localhost\:31415/sync/server
db.password=
auto.registration=true
There's a section in the documentation about embedding symmetricDs engine into Java SE application: http://www.symmetricds.org/doc/3.6/user-guide/html-single/user-guide.html#deployment-options-embedded
Related
I have posted a question few days back- Flink Jdbc sink
Now, I am trying to use the sink provided by flink.
I have written the code and it worked as well. But nothing got saved in DB and no exceptions were there. Using previous sink my code was not finishing(that should happen ideally as its a streaming app) but after the following code I am getting no error and the nothing is getting saved to DB.
public class CompetitorPipeline implements Pipeline {
private final StreamExecutionEnvironment streamEnv;
private final ParameterTool parameter;
private static final Logger LOG = LoggerFactory.getLogger(CompetitorPipeline.class);
public CompetitorPipeline(StreamExecutionEnvironment streamEnv, ParameterTool parameter) {
this.streamEnv = streamEnv;
this.parameter = parameter;
}
#Override
public KeyedStream<CompetitorConfig, String> start(ParameterTool parameter) throws Exception {
CompetitorConfigChanges competitorConfigChanges = new CompetitorConfigChanges();
KeyedStream<CompetitorConfig, String> competitorChangesStream = competitorConfigChanges.run(streamEnv, parameter);
//Add to JBDC Sink
competitorChangesStream.addSink(JdbcSink.sink(
"insert into competitor_config_universe(marketplace_id,merchant_id, competitor_name, comp_gl_product_group_desc," +
"category_code, competitor_type, namespace, qualifier, matching_type," +
"zip_region, zip_code, competitor_state, version_time, compConfigTombstoned, last_updated) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(ps, t) -> {
ps.setInt(1, t.getMarketplaceId());
ps.setLong(2, t.getMerchantId());
ps.setString(3, t.getCompetitorName());
ps.setString(4, t.getCompGlProductGroupDesc());
ps.setString(5, t.getCategoryCode());
ps.setString(6, t.getCompetitorType());
ps.setString(7, t.getNamespace());
ps.setString(8, t.getQualifier());
ps.setString(9, t.getMatchingType());
ps.setString(10, t.getZipRegion());
ps.setString(11, t.getZipCode());
ps.setString(12, t.getCompetitorState());
ps.setTimestamp(13, Timestamp.valueOf(t.getVersionTime()));
ps.setBoolean(14, t.isCompConfigTombstoned());
ps.setTimestamp(15, new Timestamp(System.currentTimeMillis()));
System.out.println("sql"+ps);
},
new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
.withUrl("jdbc:mysql://127.0.0.1:3306/database")
.withDriverName("com.mysql.cj.jdbc.Driver")
.withUsername("xyz")
.withPassword("xyz#")
.build()));
return competitorChangesStream;
}
}
You need enable autocommit mode for jdbc Sink.
new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
.withUrl("jdbc:mysql://127.0.0.1:3306/database;autocommit=true")
It looks like SimpleBatchStatementExecutor only works in auto-commit mode. And if you need to commit and rollback batches, then you have to write your own ** JdbcBatchStatementExecutor **
Have you tried to include the JdbcExecutionOptions ?
dataStream.addSink(JdbcSink.sink(
sql_statement,
(statement, value) -> {
/* Prepared Statement */
},
JdbcExecutionOptions.builder()
.withBatchSize(5000)
.withBatchIntervalMs(200)
.withMaxRetries(2)
.build(),
new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
.withUrl("jdbc:mysql://127.0.0.1:3306/database")
.withDriverName("com.mysql.cj.jdbc.Driver")
.withUsername("xyz")
.withPassword("xyz#")
.build()));
When i do JpaRepository.save(Entity e) the primary key generated with the help of hibernate sequence is saved as any random value generally starting from -43 or -42.
I am spring a spring boot project with JPA.
Below is my property file:
hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect
hibernate.hbm2ddl.auto=validate
hibernate.ejb.naming_strategy=org.hibernate.cfg.ImprovedNamingStrategy
hibernate.show_sql=false
hibernate.format_sql=true
This is my entity on which i am calling save. Sequence name - CPU_Responses_Seq is already present in DB
#Entity
#Table(name="CPU_Responses")
public class CPUResponses extends BaseEnity{
/**
*
*/
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(generator="CPUResponseSeq",strategy=GenerationType.SEQUENCE)
#SequenceGenerator(name="CPUResponseSeq",sequenceName="CPU_Responses_Seq")
#Column(name = "Response_ID", nullable=false,updatable=false)
private long responseId;
This is my persistance config class
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackages= {"package path"})
#PropertySource("classpath:application.properties")
public class PersistanceConfiguration {
#Autowired
private Environment env;
public Environment getEnv() {
return env;
}
public void setEnv(Environment env) {
this.env = env;
}
#Bean
LocalContainerEntityManagerFactoryBean entityManagerFactory() throws NamingException {
LocalContainerEntityManagerFactoryBean entityManagerFactoryBean = new LocalContainerEntityManagerFactoryBean();
entityManagerFactoryBean.setDataSource(dataSource());
entityManagerFactoryBean.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
entityManagerFactoryBean.setPackagesToScan("entity path");
Properties jpaProperties = new Properties();
//Configures the used database dialect. This allows Hibernate to create SQL
//that is optimized for the used database.
jpaProperties.put("hibernate.dialect", env.getRequiredProperty("hibernate.dialect"));
entityManagerFactoryBean.setJpaProperties(jpaProperties);
return entityManagerFactoryBean;
}
#Bean
public DataSource dataSource() throws NamingException {
JndiObjectFactoryBean bean = new JndiObjectFactoryBean();
bean.setJndiName("java:comp/env/jdbc/CPUDB");
bean.setProxyInterface(DataSource.class);
bean.setLookupOnStartup(false);
bean.afterPropertiesSet();
return (DataSource) bean.getObject();
}
#Bean
JpaTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory);
return transactionManager;
}
}
I don't know what is getting wrong. Data is getting saved in DB but with negative primary key. My sequencer in DB have min value as zero so sequencer is correct.
Kindly help
I think this is related to the changes introduced in Hibernate in their sequence generator, try to add
hibernate.id.new_generator_mappings=false
or
spring.jpa.properties.hibernate.id.new_generator_mappings=false
note that the "new generator" is not compatible with previous version so start from a clean database to avoid issues.
I have just started to use FluentMigration for my current project. I wrote my first migration but I have some trouble writing a unit test for it.
Here is some sample code:
private ServiceProvider CreateServiceProvider()
{
return new ServiceCollection()
.AddLogging(lb => lb.AddFluentMigratorConsole())
.AddFluentMigratorCore()
.ConfigureRunner(
builder => builder
.AddSQLite()
.WithGlobalConnectionString("Data Source=:memory:;Version=3;New=True;")
.WithMigrationsIn(typeof(MigrationOne).Assembly))
.BuildServiceProvider();
}
private void PerformMigrateUp(IServiceScope scope)
{
var runner = scope.ServiceProvider.GetRequiredService<IMigrationRunner>();
runner.MigrateUp(1);
}
[Test]
public void ShouldHaveTablesAfterMigrateUp()
{
var provider = this.CreateServiceProvider();
using (var scope = provider.CreateScope())
{
this.PerformMigrateUp(scope);
// here I'd like to test if tables have been created in the database by the migration
}
}
I don't know how (or if it is possible) to access the current database connection so I can perform a query. Any suggestions would be helpful. Thanks.
Ok, I found a solution. I have to use the Process method of the runner's processor to perform my own sql query.
It looks like this:
private ServiceProvider CreateServiceProvider()
{
return new ServiceCollection()
.AddLogging(lb => lb.AddFluentMigratorConsole())
.AddFluentMigratorCore()
.ConfigureRunner(
builder => builder
.AddSQLite()
.WithGlobalConnectionString(#"Data Source=:memory:;Version=3;New=True;")
.WithMigrationsIn(typeof(MigrationDate20181026113000Zero).Assembly))
.BuildServiceProvider();
}
[Test]
public void ShouldHaveNewVersionAfterMigrateUp()
{
var serviceProvider = this.CreateServiceProvider();
var scope = serviceProvider.CreateScope();
var runner = scope.ServiceProvider.GetRequiredService<IMigrationRunner>();
runner.MigrateUp(1);
string sqlStatement = "SELECT Description FROM VersionInfo";
DataSet dataSet = runner.Processor.Read(sqlStatement, string.Empty);
Assert.That(dataSet, Is.Not.Null);
Assert.That(dataSet.Tables[0].Rows[0].ItemArray[0], Is.EqualTo("Migration1"));
}
This is an old question but an important one. I find it strange that I couldnt find any documentation on this.
In any case here is my solution which I find to be a bit better as you dont need to rely on the runner. Since you dont need that the options open up hugely for constructor arguments.
Firstly make sure you install Microsoft.Data.Sqlite or you will get a strange error.
SQLite in memory databases exist for as long as the connection does - and 1 database per connection on first glance. Actually though there is a way to share the database between connections as long as at least 1 connection is open at all times according to my experiments. You just need to name it.
https://learn.microsoft.com/en-us/dotnet/standard/data/sqlite/connection-strings#sharable-in-memory
So to begin with I created a connection that will stay open until the test finishes. It will be named using Guid.NewGuid() so that subsequent connections will work as expected.
var dbName = Guid.NewGuid().ToString();
var connectionString = $"Data Source={dbName};Mode=Memory;Cache=Shared";
var connection = new SqliteConnection(connectionString);
connection.Open();
After that the crux of running the migrations is the same as previously answered but the connection string uses the named database:
var sp = services.AddFluentMigratorCore()
.ConfigureRunner(fluentMigratorBuilder => fluentMigratorBuilder
.AddSQLite()
.WithGlobalConnectionString(connectionString)
.ScanIn(AssemblyWithMigrations).For.Migrations()
)
.BuildServiceProvider();
var runner = sp.GetRequiredService<IMigrationRunner>();
runner.MigrateUp();
Here is a class I use to inject a connection factory everywhere that needs to connect to the database for normal execution:
internal class PostgresConnectionFactory : IConnectionFactory
{
private readonly string connectionString;
public PostgresConnectionFactory(string connectionString)
{
this.connectionString = connectionString;
}
public DbConnection Create()
{
return new NpgsqlConnection(connectionString);
}
}
I just replaced this (all hail dependency inversion) with:
internal class InMemoryConnectionFactory : IConnectionFactory
{
private readonly string connectionstring;
public InMemoryConnectionFactory(string connectionstring)
{
this.connectionstring = connectionstring;
}
public DbConnection Create()
{
return new SqliteConnection(connectionstring);
}
}
where the connection string is the same named one I defined above.
Now you can simply use that connection factory anywhere that needs to connect to the same in memory database, and since we can now connect multiple times possibilities for integration testing open up.
Here is the majority of my implementation:
public static IDisposable CreateInMemoryDatabase(Assembly AssemblyWithMigrations, IServiceCollection services = null)
{
if (services == null)
services = new ServiceCollection();
var connectionString = GetSharedConnectionString();
var connection = GetPersistantConnection(connectionString);
MigrateDb(services, connectionString, AssemblyWithMigrations);
services.AddSingleton<IConnectionFactory>(new InMemoryConnectionFactory(connectionString));
return services.BuildServiceProvider()
.GetRequiredService<IDisposableUnderlyingQueryingTool>();
}
private static string GetSharedConnectionString()
{
var dbName = Guid.NewGuid().ToString();
return $"Data Source={dbName};Mode=Memory;Cache=Shared";
}
private static void MigrateDb(IServiceCollection services, string connectionString, Assembly assemblyWithMigrations)
{
var sp = services.AddFluentMigratorCore()
.ConfigureRunner(fluentMigratorBuilder => fluentMigratorBuilder
.AddSQLite()
.WithGlobalConnectionString(connectionString)
.ScanIn(assemblyWithMigrations).For.Migrations()
)
.BuildServiceProvider();
var runner = sp.GetRequiredService<IMigrationRunner>();
runner.MigrateUp();
}
private static IDbConnection GetPersistantConnection(string connectionString)
{
var connection = new SqliteConnection(connectionString);
connection.Open();
return connection;
}
Then here is a sample test:
public Test : IDisposable {
private readonly IDisposable _holdingConnection;
public Test() {
_holdingConnection = CreateInMemoryDatabase(typeof(MyFirstMigration).Assembly);
}
public void Dispose() {
_holdingConnection.Dispose();
}
}
You may notice that the static factory returns a custom interface. Its just an interface that extends the normal tooling I inject to repositories, but also implements IDisposable.
Untested bonus for integration testing where you will have a service collection created via WebApplicationFactory or TestServer etc:
public void AddInMemoryPostgres(Assembly AssemblyWithMigrations)
{
var lifetime = services.BuildServiceProvider().GetService<IHostApplicationLifetime>();
var holdingConnection= InMemoryDatabaseFactory.CreateInMemoryDapperTools(AssemblyWithMigrations, services);
lifetime.ApplicationStopping.Register(() => {
holdingConnection.Dispose();
});
}
I don't know why I received the error :
org.apache.cayenne.CayenneRuntimeException: [v.4.0.M5 Feb 24 2017 07:47:55] Commit Exception
[...]
Caused by: java.sql.SQLException: Procédure stockée 'auto_pk_for_table' introuvable.
[...]
I'm using Cayenne :
<dependency>
<groupId>org.apache.cayenne</groupId>
<artifactId>cayenne-server</artifactId>
<version>4.0.M5</version>
</dependency>
and JDTS for sql server :
<dependency>
<groupId>net.sourceforge.jtds</groupId>
<artifactId>jtds</artifactId>
<version>1.3.1</version>
</dependency>
The connexion is ok :
avr. 10, 2017 2:36:30 PM org.apache.cayenne.datasource.DriverDataSource getConnection
INFOS: +++ Connecting: SUCCESS.
I'm trying to create a new user (I'm starting by bascis!) so my code is :
(I cut a little bit, it's too long:!)
public abstract class _UserInfo extends CayenneDataObject {
public static final String ADDRESS_PROPERTY = "address";
public void setAddress(String address) {
writeProperty(ADDRESS_PROPERTY, address);
}
public String getAddress() {
return (String)readProperty(ADDRESS_PROPERTY);
}
}
public class UserInfo extends _UserInfo implements Serializable {
private static final long serialVersionUID = 1L;
public String address;
public String getAdress() {
return address;
}
public void setAddress(String address) {
super.setAddress(address);
}
//I have the hashcode and equals too
}
Then, I used vaadin to create my form :
public class UserAddView extends CustomComponent implements View {
private static final long serialVersionUID = 1L;
private TextField address;
private Button save;
public static final String USERVIEW = "user";
public boolean checkValidation() {
if (!checkTextFieldValid(address))
return false;
return true;
}
public boolean checkTextFieldValid(TextField element) {
if (element == null || element.isEmpty()) {
Notification.show(
"You should register a " + element.getDescription(),
Type.WARNING_MESSAGE);
return false;
}
return true;
}
public UserAddView() {
VerticalLayout mainLayout = new VerticalLayout();
mainLayout.setSizeFull();
setCompositionRoot(mainLayout);
final VerticalLayout vlayout = new VerticalLayout();
address = new TextField("Address:");
address.setDescription("Address");
vlayout.addComponent(address);
save = new Button("Save");
vlayout.addComponent(save);
mainLayout.addComponent(new HeaderMenu());
mainLayout.addComponent(vlayout);
addListeners();
}
private void addListeners() {
save.addClickListener(new ClickListener() {
private static final long serialVersionUID = 1L;
#Override
public void buttonClick(ClickEvent event) {
if (checkValidation() == true) {
ServerRuntime cayenneRuntime = ServerRuntime.builder()
.addConfig("cayenne-myapplication.xml").build();
ObjectContext context = cayenneRuntime.newContext();
UserInfo user = context.newObject(UserInfo.class);
user.setAddress(address.getValue());
user.getObjectContext().commitChanges();
Notification.show(
"Has been saved, We will send you your password by email. Your user login is: "
+ email.getValue(), Type.TRAY_NOTIFICATION);
getUI().getNavigator().navigateTo(HomepageView.MAINVIEW);
}
}
});
}
#Override
public void enter(ViewChangeEvent event) {
// TODO Auto-generated method stub
}
}
EDIT, add information : In my user object, I have a userid (primary key), in cayenne I wrote it as primary key too and in smallint. This error seems to be link... https://cayenne.apache.org/docs/3.1/api/org/apache/cayenne/dba/sybase/SybasePkGenerator.html
The error happens when you insert a new object. For each new object Cayenne needs to generate a value of the primary key. There are various strategies to do this. The default strategy depends on the DB that you are using. For SQLServer (and for Sybase, as you've discovered :)) that strategy is to use a special stored procedure.
To create this stored procedure (and other supporting DB objects), go to CayenneModeler, open your project, and select "Tools > Generate Database Schema". In "SQL Options" tab, uncheck all checkboxes except for "Create Primary Key Support". The SQL you will see in the window below the checkboxes is what you need to run on SQL server. Either do it from Cayenne modeler or copy/paste to your favorite DB management tool.
There's also an alternative that does not require a stored procedure - using DB auto-increment feature. For this you will need to go to each DbEntity in the Modeler and under the "Entity" tab select "Database-Generated" in the "Pk Generation Strategy" dropdown. This of course implies that your PK column is indeed an auto-increment in the DB (meaning you may need to adjust your DB schema accordingly).
How to use property files as a object repository in Selenium WebDriver Automation ?
I am seeking for instructions regarding the setup and the steps that need to be done to achieve this.
Create a framework.properties file and store the variables in this way(below are two locators with sample values)
locator1=username
locator2=password
Create a class for loading the properties file. You can use the snippet below:
Note:Path /src/main/resources/com/framework/properties/ is a sample path and may change as per your framework
public class PropertyManager {
private static final Properties PROPERTY = new Properties();
private static final String FRAMEWORKPROPERTIESPATH = "/src/main/resources/com/framework/properties/";
private static final Logger LOGGER = Logg.createLogger();
public static Properties loadPropertyFile(String propertyToLoad) {
try {
PROPERTY.load(new FileInputStream(System.getProperty("user.dir")
+ FRAMEWORKPROPERTIESPATH + propertyToLoad));
} catch (IOException io) {
LOGGER.info(
"IOException in the loadFrameworkPropertyFile() method of the PropertyManager class",
io);
Runtime.getRuntime().halt(0);
}
return PROPERTY;
}
}
When you want to access the variables from the property class, use the snippet below:
private static final Properties LOCATORPROPERTIES = PropertyManager
.loadPropertyFile("framework.properties");
public void click() {
driver.findElement(By.id(LOCATORPROPERTIES.getProperty("locator1")));
}
Create any file & save it with .properties extension
For example - Add new file in eclipse By right click on project > New > File
Add below data in config.properties file and save
Username = Jhon
Password = Qwerty123
Write Below code to access this file
String filepath = "./config.properties" ; // Path of .properties file
File f = new File(filepath);
FileInputStream fs = new FileInputStream(f);
Properties pro = new Properties();
Pro.Load(fs);
pro.getProperty("Username"); // return value "Jhon" return type string
pro.getProperty("Password"); // retun value "Qwerty123" return type string
Also use like -
driver.findelement(By.id("user")).sendKeys(pro.getProperty("Username"));
driver.findelement(By.id("pass")).sendKeys(pro.getProperty("Password"));