Liquibase and distributed transactions across a linked server, SQL Server 2019 - sql-server

A technical question concerning distributed transactions via a SQL Server "linked server" definition, using Liquibase.
background/context:
SQL script is being run by Liquibase, through a changelog file.
SQL Server 2019
Oracle 11g Express on the far side of the db link RPTDB_XE.
MSDTC is turned off on the SQL Server host.
liquibase.properties
classpath=C:\\Program Files\\Microsoft SQL Server\\sqljdbc_10.2\\enu\\mssql-jdbc-10.2.0.jre8.jar
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost:1433;database=SSReporting;Enlist=false
liquibaseSchemaName=dbo
username=deploy_dbbuild
password=deploy_dbbuild
changeLogFile=changelog.xml
liquibase.hub.mode=off
integratedSecurity=false
changelog.xml
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">
<changeSet runOnChange="true" author="smcintosh" id="dw-1042-distributed-trans-error-v2.sql">
<sqlFile path="dw-1042-distributed-trans-error-v2.sql" relativeToChangelogFile="true" endDelimiter="GO"/>
</changeSet>
</databaseChangeLog>
dw-1042-distributed-trans-error-v2.sql
use SSReporting
go
set xact_abort on;
go
set nocount on;
go
begin
-- distributed transaction test
print formatmessage('%s START: %s', convert(varchar(20),getdate(),20), convert(varchar(67),##version));
update RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT set OBJECT_ID=OBJECT_ID+1;
print formatmessage('%s Updated RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT: %d rows', convert(varchar(20),getdate(),20), ##rowcount);
insert into dbo.TASK_DETAIL_LOG (BATCH_ID, LOG_DTM, CATEGORY, SEVERITY, ACTION, TASK, SUBTASK, MSG)
values (0,getdate(),'DISTRIB_TRANS','INFO','LIQUIBASE_MIGRATE','PERFORMANCE',null,
'DISTRIB_TRANSACTION_TEST_SCOTT has been updated.');
print formatmessage('%s Inserted dbo.TASK_DETAIL_LOG : %d rows',convert(varchar(20),getdate(),20), ##rowcount);
print formatmessage('%s END', convert(varchar(20),getdate(),20));
end;
go
Run the script manually to verify that it works, using sqlcmd...
C:\>sqlcmd -Usa -Pxxxx -S tcp:SCOTT-MCINTOSH,1433 -dSSReporting -w 1000 -i dw-1042-distributed-trans-error-v2.sql
Changed database context to 'SSReporting'.
2022-03-23 13:52:08 START: Microsoft SQL Server 2019 (RTM-GDR) (KB4583458) - 15.0.2080.9 (X64)
2022-03-23 13:52:08 Updated RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT: 3 rows
2022-03-23 13:52:08 Inserted dbo.TASK_DETAIL_LOG : 1 rows
2022-03-23 13:52:08 END
Run the script using Liquibase...
C:\>liquibase --defaults-file=liquibase.properties update
####################################################
## _ _ _ _ ##
## | | (_) (_) | ##
## | | _ __ _ _ _ _| |__ __ _ ___ ___ ##
## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ##
## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ##
## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ##
## | | ##
## |_| ##
## ##
## Get documentation at docs.liquibase.com ##
## Get certified courses at learn.liquibase.com ##
## Free schema change activity reports at ##
## https://hub.liquibase.com ##
## ##
####################################################
Starting Liquibase at 13:55:46 (version 4.8.0 #1581 built at 2022-02-18 21:43+0000)
Liquibase Version: 4.8.0
Liquibase Community 4.8.0 by Liquibase
Running Changeset: changelog.xml::dw-1042-distributed-trans-error-v2.sql::smcintosh
Unexpected error running Liquibase: MSDTC on server 'SCOTT-MCINTOSH' is unavailable. [Failed SQL: (8501) begin
-- distributed transaction test
print formatmessage('%s START: %s', convert(varchar(20),getdate(),20), convert(varchar(67),##version));
update RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT set OBJECT_ID=OBJECT_ID+1;
print formatmessage('%s Updated RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT: %d rows', convert(varchar(20),getdate(),20), ##rowcount);
insert into dbo.TASK_DETAIL_LOG (BATCH_ID, LOG_DTM, CATEGORY, SEVERITY, ACTION, TASK, SUBTASK, MSG)
values (0,getdate(),'DISTRIB_TRANS','INFO','LIQUIBASE_MIGRATE','PERFORMANCE',null,
'DISTRIB_TRANSACTION_TEST_SCOTT has been updated.');
print formatmessage('%s Inserted dbo.TASK_DETAIL_LOG : %d rows',convert(varchar(20),getdate(),20), ##rowcount);
print formatmessage('%s END', convert(varchar(20),getdate(),20));
end;]
For more information, please use the --log-level flag
The Question:
Why can't Liquibase run this script, when sqlcmd can? In fact, other database tools like TOAD and DataGrip can also successfully run this script, and they too use a JDBC connection, just like Liquibase.

I can see that you're running the sqlFile change type in your changelog. Liquibase has two different modes of operation. The first splits commands on the delimiter (I see you set it to GO) and executes each statement via a JDBC call to the database. However, in your script the commands are not database commands, they are SQLCMD commands. When those statements get executed on the database they will fail.
However, I found this page on the Liquibase docs site that explains the second mode of operation, which is how to run native SQLCMD scripts via Liquibase.
https://docs.liquibase.com/concepts/changelogs/attributes/using-sqlcmd-integration.html

Related

ODBC connection from Clickhouse to MSSQL databases

Please help with ODBC connection from Clickhouse to SQL Server databases.
I configured ODBC on the Clickhouse server.
Сonnection from clients such as isql, tsql is successful.
But it is not possible to connect from the clickhouse client's.
Operation system – Ubuntu 20.04
Clickhouse Server – version 22.1.3
Clickhouse Client – version 18.16.1
MS SQL Server 2016 on Windows Server.
/etc/freetds/freetds.conf
[TSQL_NE]
host = 10.72.82.72
port = 1433
tds version = 7.4
client charset = UTF-8
/etc/odbcinst.ini
[FreeTDS]
Description=FreeTDS
Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
FileUsage=1
UsageCount=8
/etc/odbc.ini
[TSQL_NE]
Description=FreeTDS NE
Driver=FreeTDS
Server=10.72.82.72
Database=ASU
UID=user
PWD=password
Port=1433
Checking the connection to the MSSQL database via ODBC
root#srv:/# isql -v TSQL_NE "user" "password"
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL> SELECT top 10 v1 from asu.dbo.data
+-------------------------+
| V1 |
+-------------------------+
| 1.45 |
| 1.5062500000000001 |
| 1.385 |
| 1.4237500000000001 |
| 1.3712500000000001 |
| 1.425 |
| 1.39625 |
| 1.6487499999999999 |
| 1.28 |
| 1.2037500000000001 |
+-------------------------+
SQLRowCount returns 10
10 rows fetched
root#srv:/# tsql -v -S TSQL_NE -U user –P password
locale is "C.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> SELECT top 10 v1 from asu.dbo.data
…
10 rows fetched
Connection with clickhouse-client and the error
root#srv:~# clickhouse-client
ClickHouse client version 18.16.1.
Password for user :
Connecting to localhost:9000.
Connected to ClickHouse server version 22.2.2 revision 54455.
b7e1d742cbd0 :) SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;', 'asu', 'dbo.data')
0 rows in set. Elapsed: 0.290 sec.
Received exception from server (version 22.2.2):
> Code: 86. DB::Exception: Received from localhost:9000, 127.0.0.1.
DB::Exception: Received error from remote server /columns_info?connection_string=DSN%3DTSQL_NE%3B%20Uid%3Duser%3BPwd%3Dpassword%3B&table=dbo.data&external_table_functions_use_nulls=true.
HTTP status code: 500 Internal Server Error, body: Error getting columns from ODBC
'Code: 36. DB::Exception: Table dbo.data not found. (BAD_ARGUMENTS) (version 22.2.2.1)'
You command in clickhouse-client should be
SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;Database=asu', 'dbo', 'data')
SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;', '', 'data')
When you remove schema and database you will get success. I saw this method in here

Why does create table operation attach owner as 'yugabyte' to a new table yet the database to which am connected has a different owner?

I have installed yugabytedb in minikube on my laptop and created a database with owner 'Rodgers'.
Then I run the ysqlsh to execute ysql commands from the terminal, one of which is 'CREATE DATABASE ...'.
Problem
When I try connecting to the database using an external Go application by providing the application with user as 'Rodgers' and the set password, it fails to connect.
I have found out that the tables created were attached to owner 'yugabyte', not 'Rodgers'.
But the database to which I have connected and from where am running the CREATE DATABASE command belongs to Rodgers.
What's going on here?
It's best to rehearse all this using "ysqlsh". When everything works there, connecting from any client program (Python, go, ...) etc will work — as long as you have the right driver. The PostgresSQL drivers work with YugabyteDB.
The following is mainly commands for "ysqlsh" — both SQLs and so-called metacommands (the ones starting with backslash). But occasionally, there are commands that you do from the O/S prompt. So you must read the following carefully and then do what it says after each comment — mainly in "ysqlsh" but a couple of times at the O/S prompt. So you can't simply run the script "lights out".
Start with virgin YB single-node cluster (fresh from "yb-create).
$ ysqlsh -h localhost -p 5433 -d yugabyte -U yugabyte
Now follow the script.
-- Shows two "Superuser" users: "postgres" and "yugabyte" (nothing else).
\du
-- Shows two databases: "postgres" and "yugabyte" (nothing else except "system" databases).
-- Both "postgres" and "yugabyte" databases are owned by "postgres".
\l
-- Create a new "ordinary user and connect as that user.
create user rodgers login password 'p';
alter user rodgers createdb;
-- Now connect to database yugabyte as user rodgers
\c yugabyte rodgers
-- Create a new database and check it's there.
create database rog_db owner rodgers;
\l
-- Name | Owner | Encoding | Collate | Ctype | Access privileges
-- -----------------+----------+----------+---------+-------------+-----------------------
...
-- rog_db | rodgers | UTF8 | C | en_US.UTF-8 |
-- ...
-- Now connect to the new "rog_db" database. Works fine.
\c rog_db rodgers
-- Quit "ysqlsh.
\q
Connect again. Works fine.
$ ysqlsh -h localhost -p 5433 -d rog_db -U rodgers
Now carry on with the script.
-- Works fine.
create table t(k int primary key);
-- Inspect it. First "\d", then "\d t".
\d
-- List of relations
-- Schema | Name | Type | Owner
-- --------+------+-------+---------
-- public | t | table | rodgers
\d t
-- Table "public.t"
Column | Type | Collation | Nullable | Default
-- --------+---------+-----------+----------+---------
-- k | integer | | not null |
-- Indexes:
-- "t_pkey" PRIMARY KEY, lsm (k HASH)
-- This is OK for playing. But terrible for real work.
drop table t;
\c rog_db yugabyte
drop schema public;
\c rog_db rodgers
create schema rog_schema authorization rodgers;
-- For future connect commands.
alter user rodgers set search_path = 'rog_schema';
-- for here and now.
set schema 'rog_schema';
create table t(k int primary key);
\d
-- List of relations
-- Schema | Name | Type | Owner
-- ------------+------+-------+---------
-- rog_schema | t | table | rodgers
--------------------------------------------------------------------------------
I just stepped through all of this using "YB-2.2.0.0-b0" on my laptop (macOS Big Sur). It all worked fine.
Please try this in your minikube env and report back.
Regards, Bryn Llewellyn, Technical Product Manager at Yugabyte Inc.

How to synchronize Redmine database pacemaker/pcs/corosync?

Currently I have 3 servers: one master and 2 clients. I installed redmine 3.3.1.stable with postgresql 9.6, and installed pacemaker on 3 servers. To synchronize a database, I follow the documentation. Every thing is working fine until when I stop the active server. The server2 redmine is showing authentication error.
Redmine error when I try to login form client after connect servers.
Completed 500 Internal Server Error in 11ms (ActiveRecord: 3.5ms)
ActiveRecord::StatementInvalid (PG::ReadOnlySqlTransaction: ERROR: cannot execute UPDATE in a read-only transaction
: UPDATE "users" SET "last_login_on" = '2020-08-17 13:05:11.001886' WHERE "users"."type" IN ('User', 'AnonymousUser') AND "users"."id" = $1):
app/models/user.rb:238:in `try_to_login'
app/controllers/account_controller.rb:204:in `password_authentication'
app/controllers/account_controller.rb:199:in `authenticate_user'
app/controllers/account_controller.rb:40:in `login'
lib/redmine/sudo_mode.rb:63:in `sudo_mode'
so far I unterstand redmine only allow server1 to access permission. why redmine can't give access to server2 or server3
Below I give more information about my step so far.
pcs config
pcs config
Cluster Name: mycluster
Corosync Nodes:
server1 server2 server3
Pacemaker Nodes:
server1 server2 server3
Resources:
Resource: MasterVip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=101.226.189.208 nic=lo cidr_netmask=32 iflabel=pgrepvip
Meta Attrs: target-role=Started
Operations: start interval=0s timeout=20s (MasterVip-start-interval-0s)
stop interval=0s timeout=20s (MasterVip-stop-interval-0s)
monitor interval=90s (MasterVip-monitor-interval-90s)
Resource: Apache (class=ocf provider=heartbeat type=apache)
Attributes: configfile=/etc/apache2/apache2.conf statusurl=http://localhost/server-status
Operations: start interval=0s timeout=40s (Apache-start-interval-0s)
stop interval=0s timeout=60s (Apache-stop-interval-0s)
monitor interval=1min (Apache-monitor-interval-1min)
Stonith Devices:
Fencing Levels:
Location Constraints:
Resource: Apache
Enabled on: server1 (score:INFINITY) (role: Started) (id:cli-prefer-Apache)
Ordering Constraints:
Colocation Constraints:
Apache with MasterVip (score:INFINITY) (id:colocation-Apache-MasterVip-INFINITY)
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
migration-threshold: 5
resource-stickiness: 10
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: mycluster
dc-version: 1.1.16-94ff4df
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false
Quorum:
Options:
master postgresql.conf
# Add settings for extensions here
listen_addresses = '*'
wal_level = hot_standby
synchronous_commit = local
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/9.6/main/archive/%f'
max_wal_senders = 10
wal_keep_segments = 30
synchronous_standby_names = 'server2'
synchronous_standby_names = 'server3'
hot_standby = on
master pg_hba.conf
# Localhost
host replication postgres 127.0.0.1/32 md5
# PostgreSQL Master IP address
host replication postgres 101.226.189.205/32 md5
# PostgreSQL SLave IP address
host replication postgres 101.226.189.206/32 md5
ho
st replication postgres 101.226.189.207/32 md5
copy config to client from Master
pg_basebackup -h server1 -U postgres -D /var/lib/postgresql/9.6/main -X stream -P
Database connection status
postgres#oreo:/etc/postgresql/9.6/main$ psql -x -c "select * from pg_stat_replication;"
-[ RECORD 1 ]----+------------------------------
pid | 18174
usesysid | 10
usename | postgres
application_name | server3
client_addr | 101.226.189.207
client_hostname |
client_port | 35236
backend_start | 2020-08-17 15:56:40.687282+02
backend_xmin |
state | streaming
sent_location | 0/7005430
write_location | 0/7005430
flush_location | 0/7005430
replay_location | 0/7005430
sync_priority | 1
sync_state | sync
-[ RECORD 2 ]----+------------------------------
pid | 18175
usesysid | 10
usename | postgres
application_name | server2
client_addr | 101.226.189.206
client_hostname |
client_port | 45862
backend_start | 2020-08-17 15:56:40.717087+02
backend_xmin |
state | streaming
sent_location | 0/7005430
write_location | 0/7005430
flush_location | 0/7005430
replay_location | 0/7005430
sync_priority | 0
sync_state | async
I found answer for my question. I miss one step in pacemaker resources.
The pgsqld defines the properties of a PostgreSQL instance: where it is located, where are its binaries, its configuration files, how to montor it, and so on.
The pgsql-ha resource controls all the PostgreSQL instances pgsqld in your cluster, decides where the primary is promoted and where the standbys are started.
pcs resource create pgsqld ocf:heartbeat:pgsqlms \
bindir="/usr/lib/postgresql/9.6/bin" \
pgdata="/etc/postgresql/9.6/main" \
datadir="/var/lib/postgresql/9.6/main" \
pghost="/var/run/postgresql" \
recovery_template="/etc/postgresql/9.6/main/recovery.conf.pcmk" \
op start timeout=60s \
op stop timeout=60s \
op promote timeout=30s \
op demote timeout=120s \
op monitor interval=15s timeout=10s role="Master" \
op monitor interval=16s timeout=10s role="Slave" \
op notify timeout=60s
pcs resource master pgsql-ha pgsqld notify=true
pcs resource cleanup
pcs status

sqoop EXPORT - There is no column found in the target table

I wrote a simple script to create a user (TestV100), create a table (Xy100) in that schema and export a tabl delimited flat file from hadoop to this oracle table.
This is the shell script: - ExportOracleTestV100.sh
#!/bin/bash
# Testing connectivity to Oracle DB
#sqoop eval --connect jdbc:oracle:thin:#hostname:1521:orcl -username test -password password --query "SELECT count(*) as bob FROM \"TestV1\".\"Test\"" --verbose
HOST=$1
USER=$2
PASS=$3
SCHEMA=$4
PORT=$5
SID=$6
SQOOP=/usr/bin/sqoop
JDBC="jdbc:oracle:thin:#$1:$5:$6"
SQOOP_EVAL="$SQOOP eval --connect $JDBC --username $USER --password $PASS --query"
#Create Schema and Tables;
${SQOOP_EVAL} "CREATE USER \"TestV100\" identified by \"password\""
${SQOOP_EVAL} "GRANT CONNECT TO \"TestV100\""
${SQOOP_EVAL} "ALTER USER \"TestV100\" QUOTA UNLIMITED ON USERS"
${SQOOP_EVAL} "DROP TABLE \"TestV100\".\"Xy100\""
${SQOOP_EVAL} "CREATE TABLE \"TestV100\".\"Xy100\"( \"a\" NVARCHAR2(255) DEFAULT NULL, \"x\" NUMBER(10,0) DEFAULT NULL, \"y\" NUMBER(10,0) DEFAULT NULL )"
############################
## Load Data into tables; ##
############################
SQOOP_EXPORT="/usr/bin/sudo -u hdfs $SQOOP export --connect ${JDBC} --username $USER --password $PASS --export-dir"
${SQOOP_EXPORT} "/tmp/rv/TestV100/xy100.txt" --table "\"\"$SCHEMA\".\"Xy100\"\"" --fields-terminated-by "\t" --input-null-string null -m 1
And this is the input file: - cat /tmp/rv/TestV100/Xy100.txt
c 8 3
a 1 4
c 6 1
c 2 0
a 7 7
c 4 2
c 7 5
a 0 0
c 5 6
a 2 2
a 5 5
a 3 6
c 9 7
a 4 1
c 3 4
a 6 3
b 6 5
b 8 7
b 5 1
b 7 3
b 2 4
b 1 0
b 4 6
b 3 2
This is how the shell script is called:
sh ./ExportOracleTestV100.sh oracle11 test password TestV100 1521 orcl --verbose
Note: 'test' user has full access on TestV100 schema.
Output:
[root#abc-repo-app1 rv]# sh ./ExportOracleTestV100.sh oracle11 test password TestV100 1521 orcl --verbose
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/11/02 12:40:07 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.4.1
15/11/02 12:40:07 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/11/02 12:40:07 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
15/11/02 12:40:07 INFO manager.SqlManager: Using default fetchSize of 1000
15/11/02 12:40:07 INFO tool.CodeGenTool: Beginning code generation
15/11/02 12:40:08 INFO manager.OracleManager: Time zone has been set to GMT
15/11/02 12:40:08 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "TestV100"."xy100" t WHERE 1=0
15/11/02 12:40:08 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: There is no column found in the target table "TestV100"."xy100". Please ensure that your table name is correct.
java.lang.IllegalArgumentException: There is no column found in the target table "TestV100"."xy100". Please ensure that your table name is correct.
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1658)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
As you can see above, Sqoop version is 1.4.5-cdh5.4.1
I somehow managed to get it this far. I see a lot of posting online for sqoop import and this error, and solution is to change the table name to UPPERCASE in the command. But, I am running export. Also, the oracle table HAS to be created with mixed case.
I hope I gave all required information here. Can someone please help or point to some place which can help me get past this error?
table name in uppercase worked.
sqoop export --connect jdbc:oracle:thin:#xyzx:1569:xyz --username xyz --password xyz --table CIPHADOOPCUSTOMERREPORT --export-dir /apps/hive/warehouse/ciprpt.db/dtd_customer_report --input-fields-terminated-by "\t" --input-lines-terminated-by "\n" --verbose -m 8 --input-null-string '\N' --input-null-non-string '\N'
Supply the table name in upper case in the --table argument.
Sounds silly but yeah - works with table name in upper caps.
I had this problem because target table for sqoop export was one column short.
Solution:
specify list of columns with --columns parameter; or regenerate target table to match schema of the input table.

Fail to export from voltdb to kafka

VoltDB is enterprise 5.1.2
Kafka is 2.9.2-0.8.1.1, also tried 2.10-0.8.2.1
VoltDB is at 192.168.56.101
Kafka is at 192.168.56.102
Here is my deployment configuration for VoltDB:
<deployment>
<cluster hostcount="1" sitesperhost="4" kfactor="0" />
<commandlog enabled="true">
<frequency time="400" transactions="1000" />
</commandlog>
<export>
<configuration enabled="true" type="kafka" stream="archive">
<property name="metadata.broker.list">192.168.56.102:9092</property>
<property name="producer.type">sync</property>
<property name="batch.mode">true</property>
</configuration>
</export>
</deployment>
The schema is defined as:
drop table person if exists;
create table person (
ic varchar(9) not null,
first_name varchar(20) not null,
last_name varchar(20) not null,
middle_name varchar(20),
gender tinyint not null,
dob timestamp not null,
date_created timestamp default now
);
partition table person on column ic;
export table person to stream archive;
And for server.properties of Kafka, I only added this line
auto.create.topics.enable=true
I first started Kafka as following:
bin/zookeeper-server-startsh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
When I start VoltDB, I encounter this exception:
david#u14voltdb:~$ voltdb create catalog.jar --deployment=config.xml
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=64m; support was removed in 8.0
Initializing VoltDB...
_ __ ____ ____ ____
| | / /___ / / /_/ __ \/ __ )
| | / / __ \/ / __/ / / / __ |
| |/ / /_/ / / /_/ /_/ / /_/ /
|___/\____/_/\__/_____/_____/
--------------------------------
Build: 5.1.2 voltdb-5.1.2-0-g6d05c33-local Enterprise Edition
Connecting to VoltDB cluster as the leader...
Host id of this node is: 0
Starting VoltDB with trial license. License expires on May 31, 2015.
Initializing the database and command logs. This may take a moment...
WARN: Failed to send producer request with correlation id 2 to broker 0 with data for partitions [voltdbexportPERSON,0]
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:72)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
at scala.collection.Iterator$class.foreach(Iterator.scala:631)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:42)
at org.voltdb.exportclient.KafkaExportClient$KafkaExportDecoder.onBlockCompletion(KafkaExportClient.java:217)
at org.voltdb.export.processors.GuestProcessor$2.run(GuestProcessor.java:223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.java:735)
at java.lang.Thread.run(Thread.java:745)
At the Kafka side, I keep getting this:
[2015-05-14 00:40:08,197] INFO Closing socket connection to /192.168.56.101. (kafka.network.Processor)
Any suggestions?
This is purely an issue related to Kafka setting. In the setting, there is a commented setting:
advertised.host.name=something
just need to replace "something" to the IP address of the server in which Kafka is running. This is found at Kafka - Unable to send a message to a remote server using Java

Resources