I'm trying to create a rich client that performs EJB RMI to interact with a server/DB. Previously, I had communications working with the remoting system to authenticate a user. Then I tacked on HTTPS communications using a keystore and clustering to the environment. Everything worked at that point.
The file-based authentication was an interim step in moving towards database authentication & authorization. I may still have configurations from that lingering and effecting this new step, I'm not certain.
Below is the failure message when trying to authenticate via the client:
Jan 19, 2017 12:51:52 PM org.jboss.ejb.client.EJBClient <clinit>
INFO: JBoss EJB Client version 2.1.4.Final
Jan 19, 2017 12:51:52 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.4.0.Final
Jan 19, 2017 12:51:52 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.4.0.Final
Jan 19, 2017 12:51:52 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 4.0.21.Final
Jan 19, 2017 12:51:53 PM org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector setupEJBReceivers
WARN: Could not register a EJB receiver for connection to 10.0.0.1:8443
javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
JBOSS-LOCAL-USER: javax.security.sasl.SaslException: Failed to read server challenge [Caused by java.io.FileNotFoundException: /home/appsrv/wildfly-10.1.0.Final/domain/tmp/auth/local4807198060994958453.challenge (No such file or directory)]
DIGEST-MD5: Server rejected authentication
at org.jboss.remoting3.remote.ClientConnectionOpenListener.allMechanismsFailed(ClientConnectionOpenListener.java:114)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:389)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:241)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:198)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:112)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:446)
at org.xnio.ssl.JsseStreamConduit.readReady(JsseStreamConduit.java:547)
at org.xnio.ssl.JsseStreamConduit$2.readReady(JsseStreamConduit.java:319)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:430)
at org.jboss.ejb.client.remoting.EndpointPool$PooledEndpoint.connect(EndpointPool.java:192)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:153)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:133)
at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:78)
at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:161)
at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:118)
at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.getCurrent(ConfigBasedEJBClientContextSelector.java:47)
at org.jboss.ejb.client.EJBClientContext.getCurrent(EJBClientContext.java:281)
at org.jboss.ejb.client.EJBClientContext.requireCurrent(EJBClientContext.java:291)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:178)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy6.getVer(Unknown Source)
at com.test.clientapp.TestClient.authenticate(TestClient.java:208)
at com.test.clientapp.BackgroundServiceEngine.run(BackgroundServiceEngine.java:136)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
In my WildFly domain configuration, I added the following authentication module to the "ha" profile (the one assigned to my host controllers) in -> Security -> MySecurityDomain via the GUI:
name: testds01
code: Database
flag: required
module options:
dsJndiName = java:/TestDS01
principalsQuery = SELECT password FROM users WHERE username=?
password-stacking = useFirstPass
hashAlgorithm = MD5
hashEncoding = BASE64
hashCharset = utf-8
I also added the following authorization module to the same area:
name: testds01
code: Delegating
flag: required
module options:
dsJndiName = java:/TestDS01
rolesQuery = SELECT role, 'Roles' FROM roles INNER JOIN users ON users.role_id = roles.role_id WHERE users.username =?
Honestly, I don't know what the flag "required" means. Nor the code "Delegating". I just found these in a book I read.
My WildFly setup includes: 1x Domain Controller, 2x Host Controllers w/1 server each, 2x SQL databases. All 5 of these are separate VMs. So, In addition to the testds01 modules added above, I have testds02 modules added pointing to "java:/TestDS02".
Let me know if additional information is needed. I'm not sure I covered everything.
Update: It's probably useful to have the client properties I'm using to setup & perform RMI:
// Set TLS Properties
System.setProperty("javax.net.ssl.keyStore", "test.keystore");
System.setProperty("javax.net.ssl.trustStore", "test.truststore");
System.setProperty("javax.net.ssl.keyStorePassword", "testpass1");
System.setProperty("javax.net.ssl.trustStorePassword", "testpass2");
// Set Application Server Properties
properties = new Properties();
properties.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "true");
properties.put("remote.connections", "hcl01,hcl02");
// Host Controller
properties.put("remote.connection.hcl01.port", "8443");
properties.put("remote.connection.hcl01.host", "10.0.0.1");
properties.put("remote.connection.hcl01.protocol", "https-remoting");
properties.put("remote.connection.hcl01.connect.options.org.xnio.Options.SSL_STARTTLS", "true");
properties.put("remote.connection.hrl01.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
properties.put("remote.connection.hcl01.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "true");
properties.put("remote.connection.hrl01.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
properties.put("remote.connection.hcl02.port", "8443");
properties.put("remote.connection.hcl02.host", "10.0.0.2");
properties.put("remote.connection.hcl02.protocol", "https-remoting");
properties.put("remote.connection.hcl02.connect.options.org.xnio.Options.SSL_STARTTLS", "true");
properties.put("remote.connection.hrl02.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
properties.put("remote.connection.hcl02.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "true");
properties.put("remote.connection.hrl02.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
// Build SLSB Lookup String
String appName = "/"; //name of ear containg ejb
String moduleName = "testapp/"; //name of ejb jar w/o extension
String distinctName = "/"; //any distinct name set within jboss for this deployment
String beanName = Login.class.getSimpleName(); //name of the bean we're looking up
String viewClassName = LoginRemote.class.getName(); //name of the bean interface
System.out.println("beanName=" + beanName + " viewClassName=" + viewClassName);
lookupSLSB = "ejb:" + appName + moduleName + distinctName + beanName + "!" + viewClassName;
// Configure EJB Lookup
Properties props = new Properties();
props.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
context = new InitialContext(props);
properties.put("remote.connection.hcl01.username", au.getUsername());
properties.put("remote.connection.hcl01.password", au.getPassword());
properties.put("remote.connection.hcl02.username", au.getUsername());
properties.put("remote.connection.hcl02.password", au.getPassword());
// JBoss Cluster Setup (using properties above)
EJBClientConfiguration cc = new PropertiesBasedEJBClientConfiguration(properties);
ContextSelector<EJBClientContext> selector = new ConfigBasedEJBClientContextSelector(cc);
EJBClientContext.setSelector(selector);
LoginRemote bean = (LoginRemote)context.lookup(lookupSLSB);
System.out.println("NIC [From bean]: Class=\"" + bean.getStr() + "\"");
I resolved this issue. After dumping traffic it appeared that the queries were being sent to the database. I enabled query logging on the database and found that they were being received, but there was a permission issue with the database user. After granting privileges to the tables being queried, the communication was successful.
Related
I am trying to connect to SQL Server in Windows Authentication mode and here is my database configuration:
database.name= DatabaseName spring.datasource.url=jdbc:sqlserver://1.2.3.4:11111;DatabaseName=${database.name};integratedSecurity = true;
spring.datasource.username=user
spring.datasource.password=passw
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
and I got these errors:
Negative matches:
AVSSimulatorServerApplication:
Did not match:
- #ConditionalOnProperty (spring.application.name=avs-simulator-service) found different value in property 'spring.application.name' (OnPropertyCondition)
ActiveMQAutoConfiguration:
Did not match:
- #ConditionalOnClass did not find required classes 'javax.jms.ConnectionFactory', 'org.apache.activemq.ActiveMQConnectionFactory' (OnClassCondition)
AopAutoConfiguration.JdkDynamicAutoProxyConfiguration:
Did not match:
- #ConditionalOnProperty (spring.aop.proxy-target-class=false) found different value in property 'proxy-target-class' (OnPropertyCondition)
ArtemisAutoConfiguration:
Did not match:
- #ConditionalOnClass did not find required classes 'javax.jms.ConnectionFactory', 'org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory' (OnClassCondition)
AsyncCustomAutoConfiguration:
Did not match:
- #ConditionalOnBean (types: org.springframework.scheduling.annotation.AsyncConfigurer; SearchStrategy: all) did not find any beans (OnBeanCondition)
Matched:
- #ConditionalOnProperty (spring.sleuth.async.enabled) matched (OnPropertyCondition)
.......
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'user'. ClientConnectionId:ce912574-1f20-4843-9b43-01820e67cf54
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'user'. ClientConnectionId:ce912574-1f20-4843-9b43-01820e67cf54
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.util.HashMap]:
................
Does anybody know what should I do in this situation?
I found the solution and it was only to use
integratedSecurity = false;
instead of
integratedSecurity = true;
so in this case connection string will be:
database.name= DatabaseName spring.datasource.url=jdbc:sqlserver://1.2.3.4:11111;DatabaseName=${database.name};**integratedSecurity = false**;.
Why to use integratedSecurity = false?
The reason is here:
Integrated Security When false, User ID and Password are specified in the connection. When true, the current Windows account credentials are used for authentication.
I'm using doctrine in a project (not symfony). In this project I also use phpstan, i installed both phpstan/phpstan-doctrine and phpstan/extension-installer.
My phpstan.neon is like this:
parameters:
level: 8
paths:
- src/
doctrine:
objectManagerLoader: tests/object-manager.php
Inside tests/object-manager.php it return the result of a call to a function that return the entity manager.
Here is the code that create the entity manager
$database_url = $_ENV['DATABASE_URL'];
$isDevMode = $this->isDevMode();
$proxyDir = null;
$cache = null;
$useSimpleAnnotationReader = false;
$config = Setup::createAnnotationMetadataConfiguration(
[$this->getProjectDirectory() . '/src'],
$isDevMode,
$proxyDir,
$cache,
$useSimpleAnnotationReader
);
// database configuration parameters
$conn = [
'url' => $database_url,
];
// obtaining the entity manager
$entityManager = EntityManager::create($conn, $config);
When i run vendor/bin/phpstan analyze i get this error:
Internal error: An exception occurred in the driver: SQLSTATE[08006] [7] could not translate host name "postgres_db" to address: nodename nor servname provided, or not known
This appear because i'm using docker and my database url is postgres://user:password#postgres_db/database postgres_db is the name of my database container so the hostname is known inside the docker container.
When i run phpstan inside the container i do not have the error.
So is there a way to run phpstan outside docker ? Because i'm pretty sure that when i'll push my code the github workflow will fail because of this
Do phpstan need to try to reach the database ?
I opened an issue on the github of phpstan-doctrine and i had an answer by #jlherren that explained :
The problem is that Doctrine needs to know what version of the DB server it is working with in order to instantiate the correct AbstractPlatform implementation, of which there are several available for the same DB vendor (e.g. PostgreSQL94Platform or PostgreSQL100Platform for postgres, and similarly for other DB drivers). To auto-detect this information, it will simply connect to the DB and query the version.
I just changed my database url from:
DATABASE_URL=postgres://user:password#database_ip/database
To:
DATABASE_URL=postgres://user:password#database_ip/database?serverVersion=14.2
I created the stub classes using CXF wsdl2java tool.
I am using Apache CXF library, with JCIFS. I validated the WSDL file itself through couple tools, it is good. Here is the code. It looks like some setting I must do.
//JCIFS Authentication related code
jcifs.Config.setProperty("jcifs.smb.client.domain", "NTS");
jcifs.Config.setProperty("jcifs.netbios.wins", "ecmchat.mark.gov");
jcifs.Config.setProperty("jcifs.smb.client.soTimeout", "300000"); // 5 minutes
jcifs.Config.setProperty("jcifs.netbios.cachePolicy", "1200"); // 20 minutes
jcifs.Config.setProperty("jcifs.smb.client.username", "user");
jcifs.Config.setProperty("jcifs.smb.client.password", "password");
//Register the jcifs URL handler to enable NTLM
jcifs.Config.registerSmbURLHandler();
//WSDL and Client settings
URL wsdlURL = BF.WSDL_LOCATION;
if (args.length > 0 && args[0] != null && !"".equals(args[0])) {
File wsdlFile = new File(args[0]);
try {
if (wsdlFile.exists()) {
wsdlURL = wsdlFile.toURI().toURL();
} else {
wsdlURL = new URL(args[0]);
}
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
BF ss = new BF(wsdlURL, SERVICE_NAME);
BFSoap port = ss.getBFSoap12();
Client client = ClientProxy.getClient(port);
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(36000);
httpClientPolicy.setAllowChunking(false);
httpClientPolicy.setReceiveTimeout(32000);
http.setClient(httpClientPolicy);
// Calling the method
System.out.println("Invoking testMethod...");
String _testMethod__return = port.testMethod();
System.out.println("testMethod.result=" + _testMethod__return);
I am getting the following exception
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </span>; expected </br>.
at [row,col,system-id]: [59,22,"https://ecmchat.mark.gov/BF/BF.asmx"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1361)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1255)
at org.apache.cxf.staxutils.StaxUtils.read(StaxUtils.java:1183)
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:235)
... 9 more
If I comment out the JCIFS NTLM authentication code, I get a HTTP 401 error. Therefore, I believe, at least it is passing some kind of authorization step.
And, if I use local WSDL in place of remote URL WSDL, then I get a different error like "method not implemented" on the call to the method. May be this is due to me not using the local WSDL correctly. I do not even know if we can use the local WSDL reference for remote service.
Then, I created a SoapUI dummy service with this WSDL, and the same code (but without the JCIFS authentication code) works good, and successfully calls the methods.
It appears to me that I must add some more appropriate settings in the configuration related code.
Am I right, and are you aware of any, for NTLM authentication and Apache CXF?
But parsing error is confusing???
I do not know if this is related.
My original WSDL URL that I gave was this.
https://ecmchat.mark.gov/BF/BF.asmx
I added a ?wsdl like below
https://ecmchat.mark.gov/BF/BF.asmx?wsdl
Then I am getting a different error.
I wonder why it is working if I access my local SoapUI version of the same WSDL service, but not for the remote one.
Invoking testMethod...
Jan 07, 2020 10:47:25 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
WARNING: Interceptor for {https://ecmchat.mark.gov}BF#{https://ecmchat.mark.gov}testMethod has thrown exception, unwinding now
java.lang.UnsupportedOperationException: Method not implemented.
at java.net.URLStreamHandler.openConnection(URLStreamHandler.java:96)
at java.net.URL.openConnection(URL.java:1028)
at org.apache.cxf.transport.https.HttpsURLConnectionFactory.createConnection(HttpsURLConnectionFactory.java:92)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.createConnection(URLConnectionHTTPConduit.java:121)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.setupConnection(URLConnectionHTTPConduit.java:125)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:505)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
at com.sun.proxy.$Proxy33.testMethod(Unknown Source)
at edison.learn.BFSoap_BFSoap12_Client.main(BFSoap_BFSoap12_Client.java:90)
Exception in thread "main" javax.xml.ws.soap.SOAPFaultException: Method not implemented.
at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:195)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at com.sun.proxy.$Proxy33.testMethod(Unknown Source)
at edison.learn.BFSoap_BFSoap12_Client.main(BFSoap_BFSoap12_Client.java:90)
Caused by: java.lang.UnsupportedOperationException: Method not implemented.
at java.net.URLStreamHandler.openConnection(URLStreamHandler.java:96)
at java.net.URL.openConnection(URL.java:1028)
at org.apache.cxf.transport.https.HttpsURLConnectionFactory.createConnection(HttpsURLConnectionFactory.java:92)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.createConnection(URLConnectionHTTPConduit.java:121)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.setupConnection(URLConnectionHTTPConduit.java:125)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:505)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
... 2 more
I am trying to use this example to connect Nifi to Flink:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();
SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);
DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
#Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});
dataStream.print();
env.execute();
I am running Nifi as a standalone server with default properties, except these properties:
nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true
The call fails each time, with following log in Nifi:
[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection
Nifi version: 1.7.1, Flink version: 1.7.1
After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.
The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:
Unable to refresh Remote Group's peers due to Remote instance of NiFi
is not configured to allow RAW Socket site-to-site communications
That's why I thought it was a port issue
So now with the default config of Nifi, this works as expected:
SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8080/nifi")
.portName("portNameAsInNifi")
.transportProtocol(SiteToSiteTransportProtocol.HTTP)
.requestBatchCount(1)
.buildConfig();
So well, I am trying to get a MQXAQueueConnectionFactory to work, I have created a extended class from the JmsComponent to handle username and password when sending data to the queue.
It does get/put messages on the queue, but in my case I've created a router to test the XA such as
from("wmq:queue:incomingQueue")
.process(new Processor(){
... Thread.sleep(20000)
})
.to("wmq:queue:outgoingQueue")
while being in sleep, I shut down the queuemanager. However when trying to get uncommited messages from the queue
DISPLAY QSTATUS('qChainQueue') i get CURDEPTH(0), while it should be 1 as I understand the XA part.
Am I doing this totally wrong?
How can it be tested?
HelpClass to handle WMQ:
public class WMQComponent extends JmsComponent {
private final String username;
private final String password;
public WMQComponent(String hostname, int port, String username, String password,
String queueManager, String channel) throws JMSException {
super();
this.username = username;
this.password = password;
MQXAQueueConnectionFactory connectionFactory = new MQXAQueueConnectionFactory();
connectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
connectionFactory.setFailIfQuiesce(1);
connectionFactory.setHostName(hostname);
connectionFactory.setPort(port);
connectionFactory.setQueueManager(queueManager);
connectionFactory.setChannel(channel);
setConnectionFactory(connectionFactory);
}
#Override
public Endpoint createEndpoint(String uri) throws Exception {
if (uri.contains("username") || uri.contains("password")) {
throw new IllegalStateException("Username and password is set by the component");
}
if (uri.contains("?")) {
return super.createEndpoint(uri + "&username=" + username + "&password=" + password);
} else {
return super.createEndpoint(uri + "?username=" + username + "&password=" + password);
}
}
}
With the following errors:
2015-03-25 14:01:12,077 [ #2 - Multicast] INFO dest_chain_ldap - org.springframework.jms.IllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QMBATCHESB' with connection mode 'Client' and host name 'hostname.com'.; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QMBATCHESB' with connection mode 'Client' and host name 'hostname.com'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE').
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:228)
at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:431)
at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:385)
at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:153)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:163)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:163)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
at org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:732)
at org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:82)
at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:303)
at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:288)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QMBATCHESB' with connection mode 'Client' and host name 'hostname.com'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:496)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:430)
at com.ibm.msg.client.wmq.internal.WMQXAConnection.<init>(WMQXAConnection.java:70)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:190)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6210)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:278)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6155)
at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:144)
at com.ibm.mq.jms.MQQueueConnectionFactory.createConnection(MQQueueConnectionFactory.java:223)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.doCreateConnection(UserCredentialsConnectionFactoryAdapter.java:175)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.createConnection(UserCredentialsConnectionFactoryAdapter.java:150)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
... 29 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
... 41 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host 'hostname.com(1514)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9213: A communications error for occurred. [1=java.net.ConnectException[Connection refused: connect],3=hostname.com]],3=hostname.com(1514),5=RemoteTCPConnection.connnectUsingLocalAddress]
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1831)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:345)
... 40 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9213: A communications error for occurred. [1=java.net.ConnectException[Connection refused: connect],3=hostname.com]
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:612)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:940)
at com.ibm.mq.jmqi.remote.internal.system.RemoteConnection.connect(RemoteConnection.java:1097)
at com.ibm.mq.jmqi.remote.internal.system.RemoteConnectionPool.getConnection(RemoteConnectionPool.java:348)
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1503)
... 41 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:69)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection$2.run(RemoteTCPConnection.java:597)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection.connnectUsingLocalAddress(RemoteTCPConnection.java:588)
... 45 more
The reason code 2059 and various errors stating that the connection was refused suggest either a mechanical issue (i.e. Listener not running) or an auths issue.
If I were attempting to debug this, the first thing I'd do is to enable authorization events, channel events, and any others you would normally enable. If you use MQ Explorer, also install the MS0P Plugin which will allow you to view the event messages in human-readable text.
Next, I would use the MQ sample programs to test. Since I always install the full client rather than grabbing the jar files, I have amqsputc available. However, the Java classes have IVT (initial verification test) programs. These ensure that the listener is running, the channel is configured and available, etc. As of v7.1 this also ensures that the CHLAUTH rules are set to allow the access. As of v8.0, or if you had the Capitalware exit installed, this also lets us test the user ID and password authentication.
The queue manager's error log and the event messages should provide good diagnostics, assuming the connection request makes it to MQ. Be sure to look both in the QMgr-specific error logs and the installation-global error logs.
Once I had confirmed that basic connectivity is in place, I'd reconcile my client-side configuration parameters for host, port, channel and if it is specified [shudder!] the QMgr name. Assuming these are correct and having proven basic connectivity works, it is now possible to test the app with some confidence.
The same method applies. First make sure the app's connection request makes it to the QMgr. If it does and is refused, the event messages and error logs will note this and why. If there is no indication of a failure in these places, the app isn't getting to the QMgr. The 2059 can indicate that the socket was refused, that the listener is up but the QMgr is not, that the channel instances have maxed out, or that after provisionally starting the channel it was closed by the QMgr, often due to a CHLAUTH rule. In any case, the event messages and error logs will have a detailed explanation as to why.
So I had done this a bit wrong, it was not enough to use the MQXAConnectionFactory but I had to create the JmsComponent as transacted.
Have tried to stop the queue manager while running the application and stop the application while handling a message and it seems to do the rollback as expected.
Ended up with
public static JmsComponent mqXAComponentTransacted(String hostname, int port, String username, String password,
String queueManager, String channel) throws JMSException {
MQXAQueueConnectionFactory connectionFactory = new MQXAQueueConnectionFactory();
connectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
connectionFactory.setFailIfQuiesce(1);
connectionFactory.setHostName(hostname);
connectionFactory.setPort(port);
connectionFactory.setQueueManager(queueManager);
connectionFactory.setChannel(channel);
UserCredentialsConnectionFactoryAdapter connectionFactoryAdapter=new UserCredentialsConnectionFactoryAdapter();
connectionFactoryAdapter.setTargetConnectionFactory(connectionFactory);
connectionFactoryAdapter.setUsername(username);
connectionFactoryAdapter.setPassword(password);
return JmsComponent.jmsComponentTransacted(connectionFactoryAdapter);
}
Also using the UserCredentialsConnectionFactoryAdapter, I didn't want to use Spring components but since the Jms package is already dependent of it, it was easier to use it than my previous solution to handle credentials.