Flink to Nifi the Magic Header was not present - apache-flink

I am trying to use this example to connect Nifi to Flink:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();
SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);
DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
#Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});
dataStream.print();
env.execute();
I am running Nifi as a standalone server with default properties, except these properties:
nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true
The call fails each time, with following log in Nifi:
[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection
Nifi version: 1.7.1, Flink version: 1.7.1

After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.
The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:
Unable to refresh Remote Group's peers due to Remote instance of NiFi
is not configured to allow RAW Socket site-to-site communications
That's why I thought it was a port issue
So now with the default config of Nifi, this works as expected:
SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8080/nifi")
.portName("portNameAsInNifi")
.transportProtocol(SiteToSiteTransportProtocol.HTTP)
.requestBatchCount(1)
.buildConfig();

Related

ESP-32 can't connect to MQTT broker: mqtt_client: Error network response

I am trying to connect my ESP32 which runs using the ESP-IDF framework to MQTT. I have imported this MQTT library successfully and have set up the configuration to look like this:
static void mqtt_app_start(void)
{
const esp_mqtt_client_config_t mqtt_cfg = {
// .host = "m15.cloudmqtt.com",
.uri = "mqtt://rxarkckf:smNb81Ppfe7T#m15.cloudmqtt.com:10793", // uri in the format (username:password#domain:port)
// .host = "m15.cloudmqtt.com", // config with host, port, user, password seperated
// .port = 10793,
// .username = "rxarkckf",
// .password = "smNb81Ppfe7T",
.event_handle = mqtt_event_handler,
// .user_context = (void *)your_context
};
esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
esp_mqtt_client_start(client);
}
I call mqtt_app_start(); in my app_main function. After uploading the code my ESP-32 doesn't connect to the MQTT broker and outputs this:
␛[0;32mI (12633410) MQTT_CLIENT: Sending MQTT CONNECT message, type: 1, id: 0000␛[0m
␛[0;31mE (12633710) MQTT_CLIENT: Error network response␛[0m
␛[0;32mI (12633710) MQTT_CLIENT: Error MQTT Connected␛[0m
␛[0;32mI (12633710) MQTT_CLIENT: Reconnect after 10000 ms␛[0m
␛[0;32mI (12633710) MQTT_SAMPLE: MQTT_EVENT_DISCONNECTED␛[0m
I have double checked that the values for the host, username, password, and port are all correct. When I look at the logs on the web interface hosted at cloudmqtt.com, I can see this output:
2018-11-17 03:50:53: New connection from 73.94.66.49 on port 10793.
2018-11-17 03:50:53: Invalid protocol "MQIs�" in CONNECT from 73.94.66.49.
2018-11-17 03:50:53: Socket error on client <unknown>, disconnecting.
2018-11-17 03:51:20: New connection from 73.94.66.49 on port 10793.
I had similar experience using mosquitto.
Adding this line to mqtt_config.h made my mqtt working.
#define CONFIG_MQTT_PROTOCOL_311
I think the more correct way to set this configuration is in sdkconfig.h, either manually or using "make menuconfig"
The problem is very simple. The library you are using implements the MQTT 3.1 protocol. The server you are trying to connect to implements the MQTT 3.1.1 protocol or higher.
As specified in the document (https://www.oasis-open.org/committees/download.php/55095/mqtt-diffs-v1.0-wd01.doc):
4.1 Protocol Name
The Protocol Name is present in the variable header of a MQTT CONNECT control packet. The Protocol Name is a UTF-8 encoded
string. In MQTT 3.1 the protocol name is "MQISDP". In MQTT 3.1.1 the
protocol name is represented as "MQTT".
For technical info:
https://mqtt.org/mqtt-specification/

read string datastream in Flink from socket without using netcat server

I have a case scenario in which I have a stream generator client which is generating multiple streams, merging them and sending it to socket and I want Flink program to listen to it as the server. As we know that server has to be turned up first, so that it can listen to client requests. I tried to do the same by using code given below
public static void main(String[] args) throws Exception {
//setting the envrionment variable as StreamExecutionEnvironment
StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
environment.setParallelism(1);
DataStream<String> stream1 = environment.socketTextStream("localhost", 9000);
stream1.print();
//start the execution
environment.execute(" Started the execution ");
}// main
The code for stream generator acting as client is given below
DataStream<Event> stream1 = envrionment
.addSource(new EventGenerator(2,60,1,1,100, 200 ))
.name("stream 1")
.setParallelism(parallelism_for_stream_rr);
DataStream<Event> stream2 = envrionment
.addSource(new EventGenerator(3,60,1,2,10, 20 ))
.name("stream 2")
.setParallelism(parallelism_for_stream_rr);
DataStream<Event> stream3 = envrionment
.addSource(new EventGenerator(5,60,1,3,30, 40 ))
.name("stream 3")
.setParallelism(parallelism_for_stream_rr);
DataStream<Event> merged = stream1.union(stream2,stream3);
merged.print();
// sending data to Mobile Cep via socket
merged.map(new MapFunction<Event, String>() {
#Override
public String map(Event event) throws Exception {
String tuple = event.toString();
return tuple + "\n";
}
}).writeToSocket("localhost", 9000, new SimpleStringSchema() );
Issue # 1: The issue is that client code works only when I start a Netcat server, but then Netcat server doesn't forwards the data streams.If Netcat server is not up, client code says it cant make a connection
Issue # 2: Flink program doesn't execute if Netcat server is not up
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
I know that one possible solution for this is to generate the streams within the Flink program, but I want to receive the streams via socket.
Thanks in Advance ~
Neither Flink's socket source nor its sink starts a TCP server and waits for incoming connections. They are both clients which connect against an already started TCP server. That's also why you have to start netcat before launching the jobs. If you want to write to and read from a socket, then you have to write a TCP server which can buffer the incoming data and forwards them once a client connects to it.

How to perform EJB DB RBAC with WildFly?

I'm trying to create a rich client that performs EJB RMI to interact with a server/DB. Previously, I had communications working with the remoting system to authenticate a user. Then I tacked on HTTPS communications using a keystore and clustering to the environment. Everything worked at that point.
The file-based authentication was an interim step in moving towards database authentication & authorization. I may still have configurations from that lingering and effecting this new step, I'm not certain.
Below is the failure message when trying to authenticate via the client:
Jan 19, 2017 12:51:52 PM org.jboss.ejb.client.EJBClient <clinit>
INFO: JBoss EJB Client version 2.1.4.Final
Jan 19, 2017 12:51:52 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.4.0.Final
Jan 19, 2017 12:51:52 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.4.0.Final
Jan 19, 2017 12:51:52 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 4.0.21.Final
Jan 19, 2017 12:51:53 PM org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector setupEJBReceivers
WARN: Could not register a EJB receiver for connection to 10.0.0.1:8443
javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
JBOSS-LOCAL-USER: javax.security.sasl.SaslException: Failed to read server challenge [Caused by java.io.FileNotFoundException: /home/appsrv/wildfly-10.1.0.Final/domain/tmp/auth/local4807198060994958453.challenge (No such file or directory)]
DIGEST-MD5: Server rejected authentication
at org.jboss.remoting3.remote.ClientConnectionOpenListener.allMechanismsFailed(ClientConnectionOpenListener.java:114)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:389)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:241)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:198)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:112)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:446)
at org.xnio.ssl.JsseStreamConduit.readReady(JsseStreamConduit.java:547)
at org.xnio.ssl.JsseStreamConduit$2.readReady(JsseStreamConduit.java:319)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:430)
at org.jboss.ejb.client.remoting.EndpointPool$PooledEndpoint.connect(EndpointPool.java:192)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:153)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:133)
at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:78)
at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:161)
at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:118)
at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:47)
at org.jboss.ejb.client.EJBClientContext.getCurrent(EJBClientContext.java:281)
at org.jboss.ejb.client.EJBClientContext.requireCurrent(EJBClientContext.java:291)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:178)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy6.getVer(Unknown Source)
at com.test.clientapp.TestClient.authenticate(TestClient.java:208)
at com.test.clientapp.BackgroundServiceEngine.run(BackgroundServiceEngine.java:136)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
In my WildFly domain configuration, I added the following authentication module to the "ha" profile (the one assigned to my host controllers) in -> Security -> MySecurityDomain via the GUI:
name: testds01
code: Database
flag: required
module options:
dsJndiName = java:/TestDS01
principalsQuery = SELECT password FROM users WHERE username=?
password-stacking = useFirstPass
hashAlgorithm = MD5
hashEncoding = BASE64
hashCharset = utf-8
I also added the following authorization module to the same area:
name: testds01
code: Delegating
flag: required
module options:
dsJndiName = java:/TestDS01
rolesQuery = SELECT role, 'Roles' FROM roles INNER JOIN users ON users.role_id = roles.role_id WHERE users.username =?
Honestly, I don't know what the flag "required" means. Nor the code "Delegating". I just found these in a book I read.
My WildFly setup includes: 1x Domain Controller, 2x Host Controllers w/1 server each, 2x SQL databases. All 5 of these are separate VMs. So, In addition to the testds01 modules added above, I have testds02 modules added pointing to "java:/TestDS02".
Let me know if additional information is needed. I'm not sure I covered everything.
Update: It's probably useful to have the client properties I'm using to setup & perform RMI:
// Set TLS Properties
System.setProperty("javax.net.ssl.keyStore", "test.keystore");
System.setProperty("javax.net.ssl.trustStore", "test.truststore");
System.setProperty("javax.net.ssl.keyStorePassword", "testpass1");
System.setProperty("javax.net.ssl.trustStorePassword", "testpass2");
// Set Application Server Properties
properties = new Properties();
properties.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "true");
properties.put("remote.connections", "hcl01,hcl02");
// Host Controller
properties.put("remote.connection.hcl01.port", "8443");
properties.put("remote.connection.hcl01.host", "10.0.0.1");
properties.put("remote.connection.hcl01.protocol", "https-remoting");
properties.put("remote.connection.hcl01.connect.options.org.xnio.Options.SSL_STARTTLS", "true");
properties.put("remote.connection.hrl01.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
properties.put("remote.connection.hcl01.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "true");
properties.put("remote.connection.hrl01.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
properties.put("remote.connection.hcl02.port", "8443");
properties.put("remote.connection.hcl02.host", "10.0.0.2");
properties.put("remote.connection.hcl02.protocol", "https-remoting");
properties.put("remote.connection.hcl02.connect.options.org.xnio.Options.SSL_STARTTLS", "true");
properties.put("remote.connection.hrl02.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
properties.put("remote.connection.hcl02.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "true");
properties.put("remote.connection.hrl02.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
// Build SLSB Lookup String
String appName = "/"; //name of ear containg ejb
String moduleName = "testapp/"; //name of ejb jar w/o extension
String distinctName = "/"; //any distinct name set within jboss for this deployment
String beanName = Login.class.getSimpleName(); //name of the bean we're looking up
String viewClassName = LoginRemote.class.getName(); //name of the bean interface
System.out.println("beanName=" + beanName + " viewClassName=" + viewClassName);
lookupSLSB = "ejb:" + appName + moduleName + distinctName + beanName + "!" + viewClassName;
// Configure EJB Lookup
Properties props = new Properties();
props.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
context = new InitialContext(props);
properties.put("remote.connection.hcl01.username", au.getUsername());
properties.put("remote.connection.hcl01.password", au.getPassword());
properties.put("remote.connection.hcl02.username", au.getUsername());
properties.put("remote.connection.hcl02.password", au.getPassword());
// JBoss Cluster Setup (using properties above)
EJBClientConfiguration cc = new PropertiesBasedEJBClientConfiguration(properties);
ContextSelector<EJBClientContext> selector = new ConfigBasedEJBClientContextSelector(cc);
EJBClientContext.setSelector(selector);
LoginRemote bean = (LoginRemote)context.lookup(lookupSLSB);
System.out.println("NIC [From bean]: Class=\"" + bean.getStr() + "\"");
I resolved this issue. After dumping traffic it appeared that the queries were being sent to the database. I enabled query logging on the database and found that they were being received, but there was a permission issue with the database user. After granting privileges to the tables being queried, the communication was successful.

Apache Camel MQXAQueueConnectionFactory

So well, I am trying to get a MQXAQueueConnectionFactory to work, I have created a extended class from the JmsComponent to handle username and password when sending data to the queue.
It does get/put messages on the queue, but in my case I've created a router to test the XA such as
from("wmq:queue:incomingQueue")
.process(new Processor(){
... Thread.sleep(20000)
})
.to("wmq:queue:outgoingQueue")
while being in sleep, I shut down the queuemanager. However when trying to get uncommited messages from the queue
DISPLAY QSTATUS('qChainQueue') i get CURDEPTH(0), while it should be 1 as I understand the XA part.
Am I doing this totally wrong?
How can it be tested?
HelpClass to handle WMQ:
public class WMQComponent extends JmsComponent {
private final String username;
private final String password;
public WMQComponent(String hostname, int port, String username, String password,
String queueManager, String channel) throws JMSException {
super();
this.username = username;
this.password = password;
MQXAQueueConnectionFactory connectionFactory = new MQXAQueueConnectionFactory();
connectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
connectionFactory.setFailIfQuiesce(1);
connectionFactory.setHostName(hostname);
connectionFactory.setPort(port);
connectionFactory.setQueueManager(queueManager);
connectionFactory.setChannel(channel);
setConnectionFactory(connectionFactory);
}
#Override
public Endpoint createEndpoint(String uri) throws Exception {
if (uri.contains("username") || uri.contains("password")) {
throw new IllegalStateException("Username and password is set by the component");
}
if (uri.contains("?")) {
return super.createEndpoint(uri + "&username=" + username + "&password=" + password);
} else {
return super.createEndpoint(uri + "?username=" + username + "&password=" + password);
}
}
}
With the following errors:
2015-03-25 14:01:12,077 [ #2 - Multicast] INFO dest_chain_ldap - org.springframework.jms.IllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QMBATCHESB' with connection mode 'Client' and host name 'hostname.com'.; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QMBATCHESB' with connection mode 'Client' and host name 'hostname.com'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE').
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:228)
at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:431)
at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:385)
at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:153)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:163)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:163)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
at org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:732)
at org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:82)
at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:303)
at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:288)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QMBATCHESB' with connection mode 'Client' and host name 'hostname.com'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:496)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:430)
at com.ibm.msg.client.wmq.internal.WMQXAConnection.<init>(WMQXAConnection.java:70)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:190)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6210)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:278)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6155)
at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:144)
at com.ibm.mq.jms.MQQueueConnectionFactory.createConnection(MQQueueConnectionFactory.java:223)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.doCreateConnection(UserCredentialsConnectionFactoryAdapter.java:175)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.createConnection(UserCredentialsConnectionFactoryAdapter.java:150)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
... 29 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
... 41 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host 'hostname.com(1514)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9213: A communications error for occurred. [1=java.net.ConnectException[Connection refused: connect],3=hostname.com]],3=hostname.com(1514),5=RemoteTCPConnection.connnectUsingLocalAddress]
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1831)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:345)
... 40 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9213: A communications error for occurred. [1=java.net.ConnectException[Connection refused: connect],3=hostname.com]
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:612)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:940)
at com.ibm.mq.jmqi.remote.internal.system.RemoteConnection.connect(RemoteConnection.java:1097)
at com.ibm.mq.jmqi.remote.internal.system.RemoteConnectionPool.getConnection(RemoteConnectionPool.java:348)
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1503)
... 41 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:69)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection$2.run(RemoteTCPConnection.java:597)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:588)
... 45 more
The reason code 2059 and various errors stating that the connection was refused suggest either a mechanical issue (i.e. Listener not running) or an auths issue.
If I were attempting to debug this, the first thing I'd do is to enable authorization events, channel events, and any others you would normally enable. If you use MQ Explorer, also install the MS0P Plugin which will allow you to view the event messages in human-readable text.
Next, I would use the MQ sample programs to test. Since I always install the full client rather than grabbing the jar files, I have amqsputc available. However, the Java classes have IVT (initial verification test) programs. These ensure that the listener is running, the channel is configured and available, etc. As of v7.1 this also ensures that the CHLAUTH rules are set to allow the access. As of v8.0, or if you had the Capitalware exit installed, this also lets us test the user ID and password authentication.
The queue manager's error log and the event messages should provide good diagnostics, assuming the connection request makes it to MQ. Be sure to look both in the QMgr-specific error logs and the installation-global error logs.
Once I had confirmed that basic connectivity is in place, I'd reconcile my client-side configuration parameters for host, port, channel and if it is specified [shudder!] the QMgr name. Assuming these are correct and having proven basic connectivity works, it is now possible to test the app with some confidence.
The same method applies. First make sure the app's connection request makes it to the QMgr. If it does and is refused, the event messages and error logs will note this and why. If there is no indication of a failure in these places, the app isn't getting to the QMgr. The 2059 can indicate that the socket was refused, that the listener is up but the QMgr is not, that the channel instances have maxed out, or that after provisionally starting the channel it was closed by the QMgr, often due to a CHLAUTH rule. In any case, the event messages and error logs will have a detailed explanation as to why.
So I had done this a bit wrong, it was not enough to use the MQXAConnectionFactory but I had to create the JmsComponent as transacted.
Have tried to stop the queue manager while running the application and stop the application while handling a message and it seems to do the rollback as expected.
Ended up with
public static JmsComponent mqXAComponentTransacted(String hostname, int port, String username, String password,
String queueManager, String channel) throws JMSException {
MQXAQueueConnectionFactory connectionFactory = new MQXAQueueConnectionFactory();
connectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
connectionFactory.setFailIfQuiesce(1);
connectionFactory.setHostName(hostname);
connectionFactory.setPort(port);
connectionFactory.setQueueManager(queueManager);
connectionFactory.setChannel(channel);
UserCredentialsConnectionFactoryAdapter connectionFactoryAdapter=new UserCredentialsConnectionFactoryAdapter();
connectionFactoryAdapter.setTargetConnectionFactory(connectionFactory);
connectionFactoryAdapter.setUsername(username);
connectionFactoryAdapter.setPassword(password);
return JmsComponent.jmsComponentTransacted(connectionFactoryAdapter);
}
Also using the UserCredentialsConnectionFactoryAdapter, I didn't want to use Spring components but since the Jms package is already dependent of it, it was easier to use it than my previous solution to handle credentials.

Kafka: can I put a path after the zookeeper port number?

The kafka server is configured with a path following the port number (from the server.properties)
zookeeper.connect=xxxxx007:2181/kafka
The java producer code:
Properties props = new Properties();
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("metadata.broker.list", "xxxxx007:9092");
The producer populates the topic if the broker omits /kafka
The producer gets numberFormatException when the broker contains /kafa
Properties props = new Properties();
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("metadata.broker.list", "xxxxx007:9092/kafka");
java.lang.NumberFormatException: For input string: "9092/kafka"
The java consumer hangs (returns no data) if the zookeeper connection contains /kafka
Properties props = new Properties();
props.put("zookeeper.connect", "xxxxx007:2181/kafka");
The java consumer gets exception if the zookeeper connection omits /kakfa
Properties props = new Properties();
props.put("zookeeper.connect", "xxxxx007:2181");
Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: group1_BFTSLBHW0000RGU-1397591737558-f75b6658 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:428)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:718)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:209)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
at kafka.examples.TitaniumConsumer.main(TitaniumConsumer.java:73)
The intention behind specifying this zookeeper path is so that for a particular cluster all data available in kafka appear under this particular path.
Note that you must create this path yourself prior to starting the broker and consumers must use the same connection string.

Resources