Flink SQL client does not create new files in AWS S3 bucket when using batch execution mode - apache-flink

We are using the Flink SQL client to sink data from various sources to S3 buckets. We are encountering issues when trying to use batch mode to write to S3 bucket. We are using batch mode with checkpointing disabled. When using SELECT INSERT, the files are not created in S3 buckets. The SQL client logs do not provide any additional information.
The same set of SQL queries work fine, when we switch to streaming execution mode. However, we cannot use “INSERT OVERWRITE” in streaming mode.
Any help or feedback would be greatly appreciated. Did anyone encounter similar issues?
Flink release: 1.13
SQL queries:
CREATE TABLE test_emp (
name STRING,
empId STRING,
company STRING
)
WITH (
'connector' = 'filesystem',
'path' = 'file:///home/constgops/Ajay/test',
'format' = 'csv'
);
CREATE TABLE s3_emp (
name STRING,
empId STRING,
company STRING
)
WITH (
'connector' = 'filesystem',
'path' = 's3a://poc-datalake-us-east-1/signaling/test',
'format' = 'parquet'
);
SET execution.runtime-mode = 'batch';
INSERT INTO s3_emp (name, empId, company) SELECT name, empId, company from test_emp;
Logs:
2022-05-16 20:00:44,805 INFO org.apache.flink.streaming.runtime.tasks.StreamTask [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend#63c3188
2022-05-16 20:00:44,805 INFO org.apache.flink.streaming.runtime.tasks.StreamTask [] - Checkpoint storage is set to 'filesystem': (checkpoints "file:/home/constgops/Abhijit/Flink/flink-1.13.1/flink-checkpoints")
2022-05-16 20:00:44,805 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2) switched from DEPLOYING to INITIALIZING.
2022-05-16 20:00:44,838 INFO org.apache.hadoop.conf.Configuration.deprecation [] - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2022-05-16 20:00:44,840 WARN org.apache.flink.metrics.MetricGroup [] - The operator name Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) exceeded the 80 characters length limit and was truncated.
2022-05-16 20:00:44,841 WARN org.apache.flink.runtime.state.StateBackendLoader [] - filesystem state backend has been deprecated. Please use 'hashmap' state backend instead.
2022-05-16 20:00:44,841 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend#19efb7c3
2022-05-16 20:00:45,103 WARN org.apache.flink.runtime.state.StateBackendLoader [] - filesystem state backend has been deprecated. Please use 'hashmap' state backend instead.
2022-05-16 20:00:45,103 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend is set to heap memory org.apache.flink.runtime.state.hashmap.HashMapStateBackend#5ef29704
2022-05-16 20:00:45,105 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2) switched from INITIALIZING to RUNNING.
2022-05-16 20:00:45,201 INFO org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2) switched from RUNNING to FINISHED.
2022-05-16 20:00:45,201 INFO org.apache.flink.runtime.taskmanager.Task [] - Freeing task resources for Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 (7b42630a41eac7ed55a72cdaf1bd29b2).
2022-05-16 20:00:45,202 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Un-registering task and sending final execution state FINISHED to JobManager for task Source: TableSourceScan(table=[[default_catalog, default_database, test_emp]], fields=[name, empId, company]) -> Sink: Filesystem (1/1)#0 7b42630a41eac7ed55a72cdaf1bd29b2.
2022-05-16 20:00:45,991 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Free slot TaskSlot(index:10, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=816.640mb (856309094 bytes), taskOffHeapMemory=102.400mb (107374182 bytes), managedMemory=655.360mb (687194777 bytes), networkMemory=51.200mb (53687091 bytes)}, allocationId: 2daf310333aaed54b7f418886b00abb4, jobId: 6e4c753a14baa10dd3f5bc6553c4eb38).
2022-05-16 20:00:45,992 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Remove job 6e4c753a14baa10dd3f5bc6553c4eb38 from job leader monitoring.
2022-05-16 20:00:45,992 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close JobManager connection for job 6e4c753a14baa10dd3f5bc6553c4eb38.

Related

Flink failed to trigger checkpoint when using table API

My flink streaming application (v1.14.4) contain JDBC connector used for initial fetch data from MySQL server
Logic:
JDBC table source -> select.where() -> convert to datastream
Kafka datastream join jdbc table -> further computation
When I run the application locally I can see following exception
14:52:00.401 [Source: TableSourceScan(table=[[default_catalog, default_database, asset, project=[id, status]]], fields=[id, status]) -> Calc(select=[CAST(_UTF-16LE'sergey-test-asset-id':VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS id, status], where=[(id = _UTF-16LE'sergey-test-asset-id')]) -> TableToDataSteam(type=ROW<`id` STRING, `status` STRING> NOT NULL, rowtime=false) -> Sink: Print to Std. Out (4/4)#0] INFO o.a.flink.runtime.taskmanager.Task - Source: TableSourceScan(table=[[default_catalog, default_database, asset, project=[id, status]]], fields=[id, status]) -> Calc(select=[CAST(_UTF-16LE'sergey-test-asset-id':VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS id, status], where=[(id = _UTF-16LE'sergey-test-asset-id')]) -> TableToDataSteam(type=ROW<`id` STRING, `status` STRING> NOT NULL, rowtime=false) -> Sink: Print to Std. Out (4/4)#0 (e8870cf296ac770346384fe2529b325f) switched from RUNNING to FINISHED.
...
14:57:52.963 [Checkpoint Timer] INFO o.a.f.r.c.CheckpointCoordinator - Failed to trigger checkpoint for job 8303c423dd7b9e3f303f0b299d7d37bb because Some tasks of the job have already finished and checkpointing with finished tasks is not enabled. Failure reason: Not all required tasks are currently running.
I do understand that after SQL select statement flink mark jdbc operator to FINISHED state, but I need to continue run the streaming application and have checkpoints during runtime
Do I need to use execution.checkpointing.checkpoints-after-tasks-finish.enabled: true property to fix my issue OR do I need to change my application graph?
Yes, you need to use
execution.checkpointing.checkpoints-after-tasks-finish.enabled: true
so that the job can checkpoint despite the fact that the jdbc source tasks have completed.

Node3 is not getting involved in mutli-master-replication of 3nodes using symmetric-ds 3.10.0

I am trying to create multi-master replication for 3 nodes (servers) using symmetric ds 3.10.0. After configuring all the 3 nodes, the bi-directional replication is working perfectly between 1st and 2nd nodes. But the bi-directional replication is NOT working between (node 1 and node3) / (node2 and node3).
Node.properties for node1:
engine.name=master-1
group.id=master
external.id=1
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://192.40.40.1:5441/testdb
db.user=user1
db.password=postgres
registration.url=
sync.url=http://192.40.40.1:31415/sync/master-1
cluster.lock.enabled=false
job.purge.period.time.ms=7200000
job.routing.period.time.ms=5000
job.push.period.time.ms=10000
job.pull.period.time.ms=10000
auto.registration=true
initial.load.create.first=true
Node.properties for node2:
engine.name=master-002
group.id=master
external.id=002
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://192.40.40.2:5441/testdb
db.user=user1
db.password=postgres
registration.url=http://192.40.40.1:31415/sync/master-1
sync.url=http://192.40.40.2:31415/sync/master-002
cluster.lock.enabled=false
job.routing.period.time.ms=5000
job.push.period.time.ms=10000
auto.registration=true
job.pull.period.time.ms=10000
Node.properties for node3:
engine.name=master-003
group.id=master
external.id=003
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://192.40.40.3:5441/testdb
db.user=user1
db.password=postgres
registration.url=http://192.40.40.1:31415/sync/master-1
sync.url=http://192.40.40.3:31415/sync/master-003
cluster.lock.enabled=false
job.routing.period.time.ms=5000
job.push.period.time.ms=10000
job.pull.period.time.ms=10000
auto.registration=true
The following sql queries are executed at node1 after creating sym_tables:
INSERT INTO sym_node_group (node_group_id, description, create_time, last_update_by, last_update_time) VALUES ('master', 'The one group in a master to master cluster', NULL, NULL, NULL);
INSERT INTO "public".sym_node_group_link (source_node_group_id, target_node_group_id, data_event_action, sync_config_enabled, is_reversible, create_time, last_update_by, last_update_time) VALUES ('master', 'master', 'P', 1, 0, NULL, NULL, NULL);
INSERT INTO "public".sym_router (router_id, target_catalog_name, target_schema_name, target_table_name, source_node_group_id, target_node_group_id, router_type, router_expression, sync_on_update, sync_on_insert, sync_on_delete, use_source_catalog_schema, create_time, last_update_by, last_update_time, description) VALUES ('master_2_master', NULL, NULL, NULL, 'master', 'master', 'default', NULL, 1, 1, 1, 0, '2022-05-01 11:41:03.738169', 'console', '2022-05-01 11:41:03.738169', NULL);
All required rows are inserted into sym_trigger and sym_trigger_router tables as well.
symmteric.log file in node3:
2022-01-06 16:58:45,903 INFO [master-002] [PushService] [master-002-push-default-3] Push data sent to master:003:003
2022-01-06 16:58:45,912 WARN [master-002] [PushService] [master-002-push-default-3] Registration is needed before communicating with master:003:003
2022-01-06 16:58:45,912 INFO [master-002] [PushService] [master-002-push-default-3] Removing identity because registration is required
2022-01-06 16:58:45,950 ERROR [master-002] [DataExtractorService] [master-002-push-default-4] Failed to extract batch 1-4 StackTraceKey.init [NullPointerException:3940683079]
java.lang.NullPointerException
at org.jumpmind.symmetric.service.impl.DataExtractorService.sendOutgoingBatch(DataExtractorService.java:1414)
at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:769)
at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:559)
at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:206)
at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:167)
at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:519)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2022-01-06 16:58:45,954 INFO [master-002] [PushService] [master-002-push-default-4] Push data sent to master:1:1
2022-01-06 16:58:45,973 INFO [master-002] [PushService] [master-002-push-default-4] Pushed data to node master:1:1. 1 data and 1 batches were processed. (sym_node)
2022-01-06 16:58:45,974 ERROR [master-002] [NodeCommunicationService] [master-002-push-default-4] Failed to execute PUSH for node 1 and channel default StackTraceKey.init [NullPointerException:2481146486]
java.lang.NullPointerException
at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:195)
at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:167)
at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:519)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2022-01-06 16:58:54,351 INFO [master-002] [RegistrationService] [master-002-job-6] This node is unregistered. It will attempt to register using the registration.url
2022-01-06 16:58:54,395 INFO [master-002] [DataLoaderService] [master-002-job-6] Using registration URL of http://192.40.40.1:31415/sync/master-1/registration?nodeGroupId=master&externalId=002&syncURL=http%3A%2F%2F192.40.40.3%3A31415%2Fsync%2Fmaster-002&schemaVersion=%3F&databaseType=PostgreSQL&databaseVersion=9.6&symmetricVersion=3.10.0&deploymentType=server&hostName=MY_PC&ipAddress=192.40.40.3
2022-01-06 16:58:54,505 INFO [master-002] [DataService] [master-002-dataloader-3] Deleted 1 data rows that were on the config channel
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] Grouplets flushed because new grouplet config came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] Load filters flushed because new filter config came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] Channels flushed because new channels came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] About to refresh the cache of conflict settings because new configuration came through the data loader
2022-01-06 16:58:55,333 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] About to refresh the cache of parameters because new configuration came through the data loader
2022-01-06 16:58:55,340 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-dataloader-3] About to refresh the cache of node security because new configuration came through the data loader
2022-01-06 16:58:55,341 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-job-6] About to refresh the cache of transformation because new configuration came through the data loader
2022-01-06 16:58:55,341 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-job-6] About to clear the staging area because new transform configuration came through the data loader
2022-01-06 16:58:55,341 INFO [master-002] [StagingManager] [master-002-job-6] Cleaning staging...
2022-01-06 16:58:55,345 INFO [master-002] [StagingManager] [master-002-job-6] Purged 6 staging files, freed 50 KB of disk space.
2022-01-06 16:58:55,345 INFO [master-002] [StagingManager] [master-002-job-6] Finished cleaning staging in 0 seconds.
2022-01-06 16:58:55,345 INFO [master-002] [ConfigurationChangedDatabaseWriterFilter] [master-002-job-6] About to syncTriggers because new configuration came through the data loader
2022-01-06 16:58:55,345 INFO [master-002] [TriggerRouterService] [master-002-job-6] Synchronizing triggers
2022-01-06 16:58:55,913 INFO [master-002] [PushService] [master-002-push-default-6] Push data sent to master:1:1
2022-01-06 16:58:55,918 WARN [master-002] [PushService] [master-002-push-default-6] Registration is needed before communicating with master:1:1
2022-01-06 16:58:55,918 INFO [master-002] [PushService] [master-002-push-default-6] Removing identity because registration is required
2022-01-06 16:58:55,961 ERROR [master-002] [DataExtractorService] [master-002-push-default-5] Failed to extract batch 003-5 StackTraceKey [NullPointerException:3940683079]
2022-01-06 16:58:55,962 INFO [master-002] [PushService] [master-002-push-default-5] Push data sent to master:003:003
2022-01-06 16:58:55,966 WARN [master-002] [PushService] [master-002-push-default-5] Registration is needed before communicating with master:003:003
2022-01-06 16:58:55,966 INFO [master-002] [PushService] [master-002-push-default-5] Removing identity because registration is required
And so on...
......
......

Liquibase lock error AFTER database creation

I'm currently running a DDL Script using the Liquibase Java API. The whole script and the corresponding changeSet is exceuted successfully. However, after this execution Liquibase throws a LockException.
The ERROR LOG is as follows,
21713 [main] DEBUG liquibase.ext.mssql.database.MSSQLDatabase - Executing Statement: ALTER
TABLE [dbo].[VALIDATIONEXECUTORS] CHECK CONSTRAINT [FK_MSTAPPTYPE_VLDTNEXCUTORS]
21713 [main] INFO liquibase.executor.jvm.JdbcExecutor - ALTER TABLE [dbo].[VALIDATIONEXECUTORS]
CHECK CONSTRAINT [FK_MSTAPPTYPE_VLDTNEXCUTORS]
21715 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - 0 row(s) affected
21715 [main] DEBUG liquibase.ext.mssql.database.MSSQLDatabase - Executing Statement: COMMIT
21715 [main] INFO liquibase.executor.jvm.JdbcExecutor - COMMIT
21735 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - -1 row(s) affected
21735 [main] INFO liquibase.changelog.ChangeSet - SQL in file
E:\\LQBASE\\LiquibaseDemo\\src\\main\\resources\\db\\changelog\\ddl\\DBSchema.sql executed
21737 [main] INFO liquibase.changelog.ChangeSet - ChangeSet
src/main/resources/db/changelog/ddl_changelog.xml::Create_DB::skini ran successfully in 18064ms
21738 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
21739 [main] INFO liquibase.executor.jvm.JdbcExecutor - SELECT MAX(ORDEREXECUTED) FROM
IND_DEV.DATABASECHANGELOG
21742 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
21744 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - Release Database Lock
21745 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
21747 [main] DEBUG liquibase.executor.jvm.JdbcExecutor - UPDATE IND_DEV.DATABASECHANGELOGLOCK
SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1
21749 [main] INFO liquibase.executor.jvm.JdbcExecutor - select schema_name()
**21751 [main] INFO liquibase.lockservice.StandardLockService - Successfully released change log
lock
21752 [main] ERROR liquibase.Liquibase - Could not release lock
liquibase.exception.LockException: liquibase.exception.DatabaseException: Error executing SQL
UPDATE IND_DEV.DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE
ID = 1: Invalid object name 'IND_DEV.DATABASECHANGELOGLOCK'.**
at liquibase.lockservice.StandardLockService.releaseLock(StandardLockService.java:357)
at liquibase.Liquibase.update(Liquibase.java:206)
at liquibase.Liquibase.update(Liquibase.java:179)
at liquibase.Liquibase.update(Liquibase.java:175)
at liquibase.Liquibase.update(Liquibase.java:168)
at
com.sk.liquibase.LiquibaseDemo.LiquibaseConfig.createManageIDDatabase(LiquibaseConfig.java:34)
at com.sk.liquibase.LiquibaseDemo.App.main(App.java:12)
**Caused by: liquibase.exception.DatabaseException: Error executing SQL UPDATE
IND_DEV.DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1:
Invalid object name 'IND_DEV.DATABASECHANGELOGLOCK'.**
According to the error, IND_DEV (which is the DB username) is somehow being appended to the DATABASECHANGELOGLOCK table. Does anyone have any idea what the issue could be?
Sometimes if the update application is abruptly stopped, then the lock remains stuck. Possibly due to a killed liquibase process not releasing its lock
Then running
UPDATE DATABASECHANGELOGLOCK SET LOCKED=0, LOCKGRANTED=null, LOCKEDBY=null;
against the database helps.
Or you can simply drop the DATABASECHANGELOGLOCK table, it will be recreated. or whatever changeloglock name you have configured.

SOLR failed to create index for Cassandra 2.1.1 list which have user defined data type

I m trying to integrate Cassandra 2.1.1 with SOLR, but SOLR failed to
create index with following error message.
16767 [qtp297774990-12] INFO org.apache.solr.handler.dataimport.DataImporter – Loading DIH Configuration: dataconfigCassandra.xml
16779 [qtp297774990-12] INFO org.apache.solr.handler.dataimport.DataImporter – Data Configuration loaded successfully
16788 [Thread-15] INFO org.apache.solr.handler.dataimport.DataImporter – Starting Full Import
16789 [qtp297774990-12] INFO org.apache.solr.core.SolrCore – [Entity_dev] webapp=/solr path=/dataimport params={optimize=false&indent=true&clean=true&commit=true&verbose=false&command=full-import&debug=false&wt=json} status=0 QTime=27
16810 [qtp297774990-12] INFO org.apache.solr.core.SolrCore – [Entity_dev] webapp=/solr path=/dataimport params={indent=true&command=status&_=1416042006354&wt=json} status=0 QTime=0
16831 [Thread-15] INFO org.apache.solr.handler.dataimport.SimplePropertiesWriter – Read dataimport.properties
16917 [Thread-15] INFO org.apache.solr.search.SolrIndexSearcher – Opening Searcher#6214b0dc[Entity_dev] realtime
16945 [Thread-15] INFO org.apache.solr.handler.dataimport.JdbcDataSource – Creating a connection for entity Entity with URL: jdbc:cassandra://10.234.31.153:9160/galaxy_dev
17082 [Thread-15] INFO org.apache.solr.handler.dataimport.JdbcDataSource – Time taken for getConnection(): 136
17429 [Thread-15] ERROR org.apache.solr.handler.dataimport.DocBuilder – Exception while processing: Entity document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from entity Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:283)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:240)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:44)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.cql.jdbc.ListMaker.compose(ListMaker.java:61)
at org.apache.cassandra.cql.jdbc.TypedColumn.<init>(TypedColumn.java:68)
at org.apache.cassandra.cql.jdbc.CassandraResultSet.createColumn(CassandraResultSet.java:1174)
at org.apache.cassandra.cql.jdbc.CassandraResultSet.populateColumns(CassandraResultSet.java:240)
at org.apache.cassandra.cql.jdbc.CassandraResultSet.<init>(CassandraResultSet.java:200)
at org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStatement.java:169)
at org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:205)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:276)
... 12 more
CREATE TABLE dev.entity (
id uuid PRIMARY KEY,
begining int,
domain text,
domain_type text,
template_name text,
****field_values list<frozen<fieldmap>>****
)
User Defined Type :
CREATE TYPE galaxy_dev.fieldmap (
key text,
value text );
Please let me know what driver or Jar use to create SOLR index for latest CASSANDRA.

Oracle Identity Federation - RCU OID Schema Creation Failure

I am trying to install OIF - Oracle Identity federation as per the OBE http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/oif/11g/r1/oif_install/oif_install.htm
I have installed the Oracle 11gR2 11.2.0.3 with the charset = AL32UTF8 and db_block size of 8K and nls_length_semantics=CHAR. Created database and listener needed.
Installed weblogic 10.3.6
Started installation of OIM - Oracle identity management, chosen install and configure option and schema creation options.
Installation goes fine, but during configuration it fails. Below is the relevant part of the logs.
I have tried multiple times just to fail again and again. If someone can kindly shed some light as what is going wrong in here. Please let me know, if you need more info on the setup...
_File : ...//oraInventory/logs/install2013-05-30_01-18-31AM.out_
ORA-01450: maximum key length (6398) exceeded
Percent Complete: 62
Repository Creation Utility: Create - Completion Summary
Database details:
Host Name : vccg-rh1.earth.com
Port : 1521
Service Name : OIAMDB
Connected As : sys
Prefix for (non-prefixable) Schema Owners : DEFAULT_PREFIX
RCU Logfile : /data/OIAM/installed_apps/fmw/Oracle_IDM1_IDP33/rcu/log/rcu.log
RCU Checkpoint Object : /data/OIAM/installed_apps/fmw/Oracle_IDM1_IDP33/rcu/log/RCUCheckpointObj
Component schemas created:
Component Status Logfile
Oracle Internet Directory Failed /data/OIAM/installed_apps/fmw/Oracle_IDM1_IDP33/rcu/log/oid.log
Repository Creation Utility - Create : Operation Completed
Repository Creation Utility - Dropping and Cleanup of the failed components
Repository Dropping and Cleanup of the failed components in progress.
Percent Complete: 93
Percent Complete: -117
Percent Complete: 100
RCUUtil createOIDRepository status = 2------------------------------------------------- java.lang.Exception: RCU OID Schema Creation Failed
at oracle.as.idm.install.config.IdMDirectoryServicesManager.doExecute(IdMDirectoryServicesManager.java:792)
at oracle.as.install.engine.modules.configuration.client.ConfigAction.execute(ConfigAction.java:375)
at oracle.as.install.engine.modules.configuration.action.TaskPerformer.run(TaskPerformer.java:88)
at oracle.as.install.engine.modules.configuration.action.TaskPerformer.startConfigAction(TaskPerformer.java:105)
at oracle.as.install.engine.modules.configuration.action.ActionRequest.perform(ActionRequest.java:15)
at oracle.as.install.engine.modules.configuration.action.RequestQueue.perform(RequestQueue.java:96)
at oracle.as.install.engine.modules.configuration.standard.StandardConfigActionManager.start(StandardConfigActionManager.java:186)
at oracle.as.install.engine.modules.configuration.boot.ConfigurationExtension.kickstart(ConfigurationExtension.java:81)
at oracle.as.install.engine.modules.configuration.ConfigurationModule.run(ConfigurationModule.java:86)
at java.lang.Thread.run(Thread.java:662)
_File : ...///fmw/Oracle_IDM1_IDP33/rcu/log/oid.log_
CREATE UNIQUE INDEX rp_dn on ct_dn (parentdn,rdn)
*
ERROR at line 1:
ORA-01450: maximum key length (6398) exceeded
Edited by: 1008964 on May 30, 2013 12:10 PM
Edited by: 1008964 on May 30, 2013 12:12 PM
Update :
I looked at the logs again and tracked which sql statements were leading to the above error…
CREATE BIGFILE TABLESPACE "OLTS_CT_STORE" EXTENT MANAGEMENT LOCAL AUTOALLOCATE SEGMENT SPACE MANAGEMENT AUTO DATAFILE '/data/OIAM/installed_apps/db/oradata/OIAMDB/gcats1_oid.dbf' SIZE 32M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED;
CREATE TABLE ct_dn (
EntryID NUMBER NOT NULL,
RDN varchar2(1024) NOT NULL,
ParentDN varchar2(1024) NOT NULL)
ENABLE ROW MOVEMENT
TABLESPACE OLTS_CT_STORE MONITORING;
*CREATE UNIQUE INDEX rp_dn on ct_dn (parentdn,rdn)
TABLESPACE OLTS_CT_STORE
PARALLEL COMPUTE STATISTICS;*
I ran these statements from sqlplus and I was able to create the index without issues and as per the table space creation statement, autoextend is on. If RCU – repo creation utility runs to create the schemas needed, it fails with the same error as earlier. Any pointers ?
Setting NLS_LENGTH_SEMANTICS=BYTE worked

Resources