Node3 is not getting involved in mutli-master-replication of 3nodes using symmetric-ds 3.10.0 - symmetricds

I am trying to create multi-master replication for 3 nodes (servers) using symmetric ds 3.10.0. After configuring all the 3 nodes, the bi-directional replication is working perfectly between 1st and 2nd nodes. But the bi-directional replication is NOT working between (node 1 and node3) / (node2 and node3).
Node.properties for node1:
engine.name=master-1
group.id=master
external.id=1
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://192.40.40.1:5441/testdb
db.user=user1
db.password=postgres
registration.url=
sync.url=http://192.40.40.1:31415/sync/master-1
cluster.lock.enabled=false
job.purge.period.time.ms=7200000
job.routing.period.time.ms=5000
job.push.period.time.ms=10000
job.pull.period.time.ms=10000
auto.registration=true
initial.load.create.first=true
Node.properties for node2:
engine.name=master-002
group.id=master
external.id=002
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://192.40.40.2:5441/testdb
db.user=user1
db.password=postgres
registration.url=http://192.40.40.1:31415/sync/master-1
sync.url=http://192.40.40.2:31415/sync/master-002
cluster.lock.enabled=false
job.routing.period.time.ms=5000
job.push.period.time.ms=10000
auto.registration=true
job.pull.period.time.ms=10000
Node.properties for node3:
engine.name=master-003
group.id=master
external.id=003
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://192.40.40.3:5441/testdb
db.user=user1
db.password=postgres
registration.url=http://192.40.40.1:31415/sync/master-1
sync.url=http://192.40.40.3:31415/sync/master-003
cluster.lock.enabled=false
job.routing.period.time.ms=5000
job.push.period.time.ms=10000
job.pull.period.time.ms=10000
auto.registration=true
The following sql queries are executed at node1 after creating sym_tables:
INSERT INTO sym_node_group (node_group_id, description, create_time, last_update_by, last_update_time) VALUES ('master', 'The one group in a master to master cluster', NULL, NULL, NULL);
INSERT INTO "public".sym_node_group_link (source_node_group_id, target_node_group_id, data_event_action, sync_config_enabled, is_reversible, create_time, last_update_by, last_update_time) VALUES ('master', 'master', 'P', 1, 0, NULL, NULL, NULL);
INSERT INTO "public".sym_router (router_id, target_catalog_name, target_schema_name, target_table_name, source_node_group_id, target_node_group_id, router_type, router_expression, sync_on_update, sync_on_insert, sync_on_delete, use_source_catalog_schema, create_time, last_update_by, last_update_time, description) VALUES ('master_2_master', NULL, NULL, NULL, 'master', 'master', 'default', NULL, 1, 1, 1, 0, '2022-05-01 11:41:03.738169', 'console', '2022-05-01 11:41:03.738169', NULL);
All required rows are inserted into sym_trigger and sym_trigger_router tables as well.
symmteric.log file in node3:
2022-01-06 16:58:45,903 INFO [master-002] [PushService] [master-002-push-default-3] Push data sent to master:003:003
2022-01-06 16:58:45,912 WARN [master-002] [PushService] [master-002-push-default-3] Registration is needed before communicating with master:003:003
2022-01-06 16:58:45,912 INFO [master-002] [PushService] [master-002-push-default-3] Removing identity because registration is required
2022-01-06 16:58:45,950 ERROR [master-002] [DataExtractorService] [master-002-push-default-4] Failed to extract batch 1-4 StackTraceKey.init [NullPointerException:3940683079]
java.lang.NullPointerException
at org.jumpmind.symmetric.service.impl.DataExtractorService.sendOutgoingBatch(DataExtractorService.java:1414)
at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:769)
at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:559)
at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:206)
at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:167)
at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:519)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2022-01-06 16:58:45,954 INFO [master-002] [PushService] [master-002-push-default-4] Push data sent to master:1:1
2022-01-06 16:58:45,973 INFO [master-002] [PushService] [master-002-push-default-4] Pushed data to node master:1:1. 1 data and 1 batches were processed. (sym_node)
2022-01-06 16:58:45,974 ERROR [master-002] [NodeCommunicationService] [master-002-push-default-4] Failed to execute PUSH for node 1 and channel default StackTraceKey.init [NullPointerException:2481146486]
java.lang.NullPointerException
at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:195)
at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:167)
at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:519)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2022-01-06 16:58:54,351 INFO [master-002] [RegistrationService] [master-002-job-6] This node is unregistered. It will attempt to register using the registration.url
2022-01-06 16:58:54,395 INFO [master-002] [DataLoaderService] [master-002-job-6] Using registration URL of http://192.40.40.1:31415/sync/master-1/registration?nodeGroupId=master&externalId=002&syncURL=http%3A%2F%2F192.40.40.3%3A31415%2Fsync%2Fmaster-002&schemaVersion=%3F&databaseType=PostgreSQL&databaseVersion=9.6&symmetricVersion=3.10.0&deploymentType=server&hostName=MY_PC&ipAddress=192.40.40.3
2022-01-06 16:58:54,505 INFO [master-002] [DataService] [master-002-dataloader-3] Deleted 1 data rows that were on the config channel
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] Grouplets flushed because new grouplet config came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] Load filters flushed because new filter config came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] Channels flushed because new channels came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] About to refresh the cache of conflict settings because new configuration came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] About to refresh the cache of parameters because new configuration came through the data loader
2022-01-06 16:58:55,340 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] About to refresh the cache of node security because new configuration came through the data loader
2022-01-06 16:58:55,341 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-job-6] About to refresh the cache of transformation because new configuration came through the data loader
2022-01-06 16:58:55,341 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-job-6] About to clear the staging area because new transform configuration came through the data loader
2022-01-06 16:58:55,341 INFO [master-002] [StagingManager] [master-002-job-6] Cleaning staging...
2022-01-06 16:58:55,345 INFO [master-002] [StagingManager] [master-002-job-6] Purged 6 staging files, freed 50 KB of disk space.
2022-01-06 16:58:55,345 INFO [master-002] [StagingManager] [master-002-job-6] Finished cleaning staging in 0 seconds.
2022-01-06 16:58:55,345 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-job-6] About to syncTriggers because new configuration came through the data loader
2022-01-06 16:58:55,345 INFO [master-002] [TriggerRouterService] [master-002-job-6] Synchronizing triggers
2022-01-06 16:58:55,913 INFO [master-002] [PushService] [master-002-push-default-6] Push data sent to master:1:1
2022-01-06 16:58:55,918 WARN [master-002] [PushService] [master-002-push-default-6] Registration is needed before communicating with master:1:1
2022-01-06 16:58:55,918 INFO [master-002] [PushService] [master-002-push-default-6] Removing identity because registration is required
2022-01-06 16:58:55,961 ERROR [master-002] [DataExtractorService] [master-002-push-default-5] Failed to extract batch 003-5 StackTraceKey [NullPointerException:3940683079]
2022-01-06 16:58:55,962 INFO [master-002] [PushService] [master-002-push-default-5] Push data sent to master:003:003
2022-01-06 16:58:55,966 WARN [master-002] [PushService] [master-002-push-default-5] Registration is needed before communicating with master:003:003
2022-01-06 16:58:55,966 INFO [master-002] [PushService] [master-002-push-default-5] Removing identity because registration is required
And so on...
......
......

Related

Flink SQL client does not create new files in AWS S3 bucket when using batch execution mode

We are using the Flink SQL client to sink data from various sources to S3 buckets. We are encountering issues when trying to use batch mode to write to S3 bucket. We are using batch mode with checkpointing disabled. When using SELECT INSERT, the files are not created in S3 buckets. The SQL client logs do not provide any additional information.
The same set of SQL queries work fine, when we switch to streaming execution mode. However, we cannot use “INSERT OVERWRITE” in streaming mode.
Any help or feedback would be greatly appreciated. Did anyone encounter similar issues?
Flink release: 1.13
SQL queries:
CREATE TABLE test_emp (
name STRING,
empId STRING,
company STRING
)
WITH (
'connector' = 'filesystem',
'path' = 'file:///home/constgops/Ajay/test',
'format' = 'csv'
);
CREATE TABLE s3_emp (
name STRING,
empId STRING,
company STRING
)
WITH (
'connector' = 'filesystem',
'path' = 's3a://poc-datalake-us-east-1/signaling/test',
'format' = 'parquet'
);
SET execution.runtime-mode = 'batch';
INSERT INTO s3_emp (name, empId, company) SELECT name, empId, company from test_emp;
Logs:
2022-05-16 20:00:44,805 INFO org.apache.flink.streaming.runtime.tasks.StreamTask [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend#63c3188
2022-05-16 20:00:44,805 INFO org.apache.flink.streaming.runtime.tasks.StreamTask [] - Checkpoint storage is set to 'filesystem': (checkpoints "file:/home/constgops/Abhijit/Flink/flink-1.13.1/flink-checkpoints")
2022-05-16 20:00:44,805 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2) switched from DEPLOYING to INITIALIZING.
2022-05-16 20:00:44,838 INFO org.apache.hadoop.conf.Configuration.deprecation [] - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2022-05-16 20:00:44,840 WARN org.apache.flink.metrics.MetricGroup [] - The operator name Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) exceeded the 80 characters length limit and was truncated.
2022-05-16 20:00:44,841 WARN org.apache.flink.runtime.state.StateBackendLoader [] - filesystem state backend has been deprecated. Please use 'hashmap' state backend instead.
2022-05-16 20:00:44,841 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend#19efb7c3
2022-05-16 20:00:45,103 WARN org.apache.flink.runtime.state.StateBackendLoader [] - filesystem state backend has been deprecated. Please use 'hashmap' state backend instead.
2022-05-16 20:00:45,103 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend#5ef29704
2022-05-16 20:00:45,105 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2) switched from INITIALIZING to RUNNING.
2022-05-16 20:00:45,201 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2) switched from RUNNING to FINISHED.
2022-05-16 20:00:45,201 INFO org.apache.flink.runtime.taskmanager.Task [] - Freeing task resources for Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2).
2022-05-16 20:00:45,202 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Un-registering task and sending final execution state FINISHED to JobManager for task Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 7b42630a41eac7ed55a72cdaf1bd29b2.
2022-05-16 20:00:45,991 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Free slot TaskSlot(index:10, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=816.640mb (856309094 bytes), taskOffHeapMemory=102.400mb (107374182 bytes), managedMemory=655.360mb (687194777 bytes), networkMemory=51.200mb (53687091 bytes)}, allocationId: 2daf310333aaed54b7f418886b00abb4, jobId: 6e4c753a14baa10dd3f5bc6553c4eb38).
2022-05-16 20:00:45,992 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Remove job 6e4c753a14baa10dd3f5bc6553c4eb38 from job leader monitoring.
2022-05-16 20:00:45,992 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close JobManager connection for job 6e4c753a14baa10dd3f5bc6553c4eb38.

Liquibase lock error AFTER database creation

I'm currently running a DDL Script using the Liquibase Java API. The whole script and the corresponding changeSet is exceuted successfully. However, after this execution Liquibase throws a LockException.
The ERROR LOG is as follows,
21713 [main] DEBUG liquibase.ext.mssql.database.MSSQLDatabase - Executing Statement: ALTER
TABLE [dbo].[VALIDATIONEXECUTORS] CHECK CONSTRAINT [FK_MSTAPPTYPE_VLDTNEXCUTORS]
21713 [main] INFO liquibase.executor.jvm.JdbcExecutor - ALTER TABLE [dbo].[VALIDATIONEXECUTORS]
CHECK CONSTRAINT [FK_MSTAPPTYPE_VLDTNEXCUTORS]
21715 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - 0 row(s) affected
21715 [main] DEBUG liquibase.ext.mssql.database.MSSQLDatabase - Executing Statement: COMMIT
21715 [main] INFO liquibase.executor.jvm.JdbcExecutor - COMMIT
21735 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - -1 row(s) affected
21735 [main] INFO liquibase.changelog.ChangeSet - SQL in file
E:\\LQBASE\\LiquibaseDemo\\src\\main\\resources\\db\\changelog\\ddl\\DBSchema.sql executed
21737 [main] INFO liquibase.changelog.ChangeSet - ChangeSet
src/main/resources/db/changelog/ddl_changelog.xml::Create_DB::skini ran successfully in 18064ms
21738 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
21739 [main] INFO liquibase.executor.jvm.JdbcExecutor - SELECT MAX(ORDEREXECUTED) FROM
IND_DEV.DATABASECHANGELOG
21742 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
21744 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - Release Database Lock
21745 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
21747 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - UPDATE IND_DEV.DATABASECHANGELOGLOCK
SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1
21749 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
**21751 [main] INFO liquibase.lockservice.StandardLockService - Successfully released change log
lock
21752 [main] ERROR liquibase.Liquibase - Could not release lock
liquibase.exception.LockException: liquibase.exception.DatabaseException: Error executing SQL
UPDATE IND_DEV.DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE
ID = 1: Invalid object name 'IND_DEV.DATABASECHANGELOGLOCK'.**
at liquibase.lockservice.StandardLockService.releaseLock(StandardLockService.java:357)
at liquibase.Liquibase.update(Liquibase.java:206)
at liquibase.Liquibase.update(Liquibase.java:179)
at liquibase.Liquibase.update(Liquibase.java:175)
at liquibase.Liquibase.update(Liquibase.java:168)
at
com.sk.liquibase.LiquibaseDemo.LiquibaseConfig.createManageIDDatabase(LiquibaseConfig.java:34)
at com.sk.liquibase.LiquibaseDemo.App.main(App.java:12)
**Caused by: liquibase.exception.DatabaseException: Error executing SQL UPDATE
IND_DEV.DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1:
Invalid object name 'IND_DEV.DATABASECHANGELOGLOCK'.**
According to the error, IND_DEV (which is the DB username) is somehow being appended to the DATABASECHANGELOGLOCK table. Does anyone have any idea what the issue could be?
Sometimes if the update application is abruptly stopped, then the lock remains stuck. Possibly due to a killed liquibase process not releasing its lock
Then running
UPDATE DATABASECHANGELOGLOCK SET LOCKED=0, LOCKGRANTED=null, LOCKEDBY=null;
against the database helps.
Or you can simply drop the DATABASECHANGELOGLOCK table, it will be recreated. or whatever changeloglock name you have configured.

NiFi connection to SqlServor for ExecuteSQL

I'm trying to import some data from different SqlServer databases using ExecuteSQL in NiFi, but it's returning me an error. I've already imported a lot of other tables from MySQL databases without any problem and I'm trying to use the same workflow structure for the SqlServer dbs.
The structure is as follows:
There's a file .txt with the list of tables to be imported
This file is fetched, splitted and uptaded; so there's a FlowFile for each table of each db that has to be imported,
These FlowFiles are passed into ExecuteSQL which executes their contents
For example:
file.txt
table1
table2
table3
is being updated into 3 different FlowFiles:
FlowFile1
SELECT * FROM table1
FlowFile2
SELECT * FROM table2
FlowFile3
SELECT * FROM table3
which are passed to ExecuteSQL.
Here follows the configuration of ExecuteSQL (identical for SqlServer tables and MySQL ones)
ExecuteSQL
As the only difference with the import from MySQL db is in the connectors, this is how a generic MySQL connector has been configured:
SETTINGSPROPERTIES
Database Connection URL jdbc:mysql://00.00.00.00/DataBase?zeroDateTimeBehavior=convertToNull&autoReconnect=true
Database Driver Class Name com.mysql.jdbc.Driver
Database Driver Location(s) file:///path/mysql-connector-java-5.1.47-bin.jar
Database User user
PasswordSensitive value set
Max Wait Time 500 millis
Max Total Connections 8
Validation query No value set
And this is how a SqlServer connector has been configured:
SETTINGSPROPERTIES
Database Connection URL jdbc:jtds:sqlserver://00.00.00.00/DataBase;useNTLMv2=true;integratedSecurity=true;
Database Driver Class Name net.sourceforge.jtds.jdbc.Driver
Database Driver Location(s) /path/connectors/jtds-1.3.1.jar
Database User user
PasswordSensitive value set
Max Wait Time -1
Max Total Connections 8
Validation query No value set
It has to be noticed that one (only one!) SqlServer connector works and the ExecuteSQL processor imports the data without any problem. The even stranger thing is that the database that is being connected via this connector is located in the same place as other two (the connection URL and user/psw are identical), but only the first one is working.
Notice that I've tried appending ?zeroDateTimeBehavior=convertToNull&autoReconnect=true also to the SqlServer connections, supposing it was a problem of date type, but it didn't give any positive change.
Here is the error that is being returned:
12:02:46 CEST ERROR f1553b83-a173-1c0f-93cb-1c32f0f46d1d
00.00.00.00:0000 ExecuteSQL[id=****] ExecuteSQL[id=****] failed to process session due to null; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodError
Error retrieved from logs:
ERROR [Timer-Driven Process Thread-49] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=****] ExecuteSQL[id=****] failed to process session due to java.lang.AbstractMethodError; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodError
java.lang.AbstractMethodError: null
at net.sourceforge.jtds.jdbc.JtdsConnection.isValid(JtdsConnection.java:2833)
at org.apache.commons.dbcp2.DelegatingConnection.isValid(DelegatingConnection.java:874)
at org.apache.commons.dbcp2.PoolableConnection.validate(PoolableConnection.java:270)
at org.apache.commons.dbcp2.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:389)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2398)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2381)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2110)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
at org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:305)
at org.apache.nifi.dbcp.DBCPService.getConnection(DBCPService.java:49)
at sun.reflect.GeneratedMethodAccessor1696.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
at com.sun.proxy.$Proxy449.getConnection(Unknown Source)
at org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:195)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Apache NiFi :Convert JSONtoSQL (Oracle Database)

I wanted to convert my JSON string to SQL statement by using ConvertJSONtoSQL processor.
example: JSON string -
{"cpuwait":"0.0","servernamee":"mywindows","cpusys":"5.3","cpuidle":"77.6","datee":"29-SEP-2016","timee":"00:01:33","cpucpuno":"CPU01","cpuuser":"17.1"}
Table structure in oracle db -
CREATE TABLE cpu (
datee varchar2(15) DEFAULT NULL,
timee varchar2(10) DEFAULT NULL,
servernamee varchar2(20) DEFAULT NULL,
cpucpuno varchar2(4) DEFAULT NULL,
cpuuser varchar2(5) DEFAULT NULL,
cpusys varchar2(5) DEFAULT NULL,
cpuwait varchar2(5) DEFAULT NULL,
cpuidle varchar2(5) DEFAULT NULL
);
Configuration used for MySQL Database:
Database connection url:jdbc:mysql://localhost:3306/testnifi
Database Driver classname:com.mysql.jdbc.Driver
I was successfully connected to MySQL using(DBCP connection pool) JDBC url,username and password.
ConvertJSONtoSQL processor successfully worked there and I'm getting valid sql insert statement as output.
But when i was trying the same with Oracle Database I'm getting
ERROR [Timer-Driven Process Thread-6] o.a.n.p.standard.ConvertJSONToSQL
java.sql.SQLException: Stream has already been closed
My configuration for Oracle db connection:
I searched for the error in google but I found that this error will occur when Long Datatypes were used in database tables but I'm not using them.
I went through the source code of ConvertJSONtoSQL processor(following stack trace) and tried to implement the same in eclipse where I'm not getting any error ,I can connect to database and make queries.
So is there any mistake in my configuration?
Nifi version - 0.7.0/1.0(i'm getting same error in both)
java version - java8
Oracle DB version - Oracle Database 11g Express Edition
Complete Stack trace:
2016-10-19 07:10:06,557 ERROR [Timer-Driven Process Thread-6] o.a.n.p.standard.ConvertJSONToSQL
java.sql.SQLException: Stream has already been closed
at oracle.jdbc.driver.LongAccessor.getBytesInternal(LongAccessor.java:156) ~[ojdbc6.jar:11.2.0.1.0]
at oracle.jdbc.driver.LongAccessor.getBytes(LongAccessor.java:126) ~[ojdbc6.jar:11.2.0.1.0]
at oracle.jdbc.driver.LongAccessor.getString(LongAccessor.java:201) ~[ojdbc6.jar:11.2.0.1.0]
at oracle.jdbc.driver.T4CLongAccessor.getString(T4CLongAccessor.java:427) ~[ojdbc6.jar:11.2.0.1.0]
at oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:1251) ~[ojdbc6.jar:11.2.0.1.0]
at oracle.jdbc.driver.OracleResultSet.getString(OracleResultSet.java:494) ~[ojdbc6.jar:11.2.0.1.0]
at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:263) ~[na:na]
at org.apache.nifi.processors.standard.ConvertJSONToSQL$ColumnDescription.from(ConvertJSONToSQL.java:677) ~[nifi-standard-processors-0.7.0.jar:0.7.0]
at org.apache.nifi.processors.standard.ConvertJSONToSQL$TableSchema.from(ConvertJSONToSQL.java:621) ~[nifi-standard-processors-0.7.0.jar:0.7.0]
at org.apache.nifi.processors.standard.ConvertJSONToSQL.onTrigger(ConvertJSONToSQL.java:267) ~[nifi-standard-processors-0.7.0.jar:0.7.0]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-0.7.0.jar:0.7.0]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) [nifi-framework-core-0.7.0.jar:0.7.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0.jar:0.7.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0.jar:0.7.0]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-0.7.0.jar:0.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.7.0_40]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.7.0_40]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.7.0_40]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.7.0_40]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_40]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_40]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_40
It seems a bug in Oracle driver. See:
https://blog.jooq.org/2015/12/30/oracle-long-and-long-raw-causing-stream-has-already-been-closed-exception/
Hibernate custom type to avoid 'Caused by: java.sql.SQLException: Stream has already been closed'
The item 2 give me the workaround. Basically add in bootstrap.conf the following argument:
java.arg.xx=-Doracle.jdbc.useFetchSizeWithLongColumn=true

SOLR failed to create index for Cassandra 2.1.1 list which have user defined data type

I m trying to integrate Cassandra 2.1.1 with SOLR, but SOLR failed to
create index with following error message.
16767 [qtp297774990-12] INFO org.apache.solr.handler.dataimport.DataImporter – Loading DIH Configuration: dataconfigCassandra.xml
16779 [qtp297774990-12] INFO org.apache.solr.handler.dataimport.DataImporter – Data Configuration loaded successfully
16788 [Thread-15] INFO org.apache.solr.handler.dataimport.DataImporter – Starting Full Import
16789 [qtp297774990-12] INFO org.apache.solr.core.SolrCore – [Entity_dev] webapp=/solr path=/dataimport params={optimize=false&indent=true&clean=true&commit=true&verbose=false&command=full-import&debug=false&wt=json} status=0 QTime=27
16810 [qtp297774990-12] INFO org.apache.solr.core.SolrCore – [Entity_dev] webapp=/solr path=/dataimport params={indent=true&command=status&_=1416042006354&wt=json} status=0 QTime=0
16831 [Thread-15] INFO org.apache.solr.handler.dataimport.SimplePropertiesWriter – Read dataimport.properties
16917 [Thread-15] INFO org.apache.solr.search.SolrIndexSearcher – Opening Searcher#6214b0dc[Entity_dev] realtime
16945 [Thread-15] INFO org.apache.solr.handler.dataimport.JdbcDataSource – Creating a connection for entity Entity with URL: jdbc:cassandra://10.234.31.153:9160/galaxy_dev
17082 [Thread-15] INFO org.apache.solr.handler.dataimport.JdbcDataSource – Time taken for getConnection(): 136
17429 [Thread-15] ERROR org.apache.solr.handler.dataimport.DocBuilder – Exception while processing: Entity document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from entity Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:283)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:240)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:44)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.cql.jdbc.ListMaker.compose(ListMaker.java:61)
at org.apache.cassandra.cql.jdbc.TypedColumn.<init>(TypedColumn.java:68)
at org.apache.cassandra.cql.jdbc.CassandraResultSet.createColumn(CassandraResultSet.java:1174)
at org.apache.cassandra.cql.jdbc.CassandraResultSet.populateColumns(CassandraResultSet.java:240)
at org.apache.cassandra.cql.jdbc.CassandraResultSet.<init>(CassandraResultSet.java:200)
at org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStatement.java:169)
at org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:205)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:276)
... 12 more
CREATE TABLE dev.entity (
id uuid PRIMARY KEY,
begining int,
domain text,
domain_type text,
template_name text,
****field_values list<frozen<fieldmap>>****
)
User Defined Type :
CREATE TYPE galaxy_dev.fieldmap (
key text,
value text );
Please let me know what driver or Jar use to create SOLR index for latest CASSANDRA.

Resources