GremlinPython add connection and request timeout to DriverRemoteConnection with aiohttp - aiohttp

I am upgrading the gremlinpython package from 3.4 to 3.5
As part of the upgrade, tornado has been removed and only aiohttp is supported.
Before, to create a DriverRemoteConnection with a connection and request timeout, I used the following code
from tornado import httpclient
req = httpclient.HTTPRequest(
connection_str,
connect_timeout=gremlin_connect_timeout_secs,
request_timeout=gremlin_request_timeout_secs
)
driver_remote_connection = DriverRemoteConnection(
req, "g", pool_size=pool_size, max_workers=max_workers
)
g = traversal().withRemote(driver_remote_connection)
Now that tornado isn't supported anymore, what would be the equivalent in 3.5+?
My connection is to AWS Neptune.
I've looked into the aiogremlin package and the aiohttp library but it requires me to create asynchronous client while I don't need for the connection to be asynchronous.
I can create the DriverRemoteConnection with simply this
driver_remote_connection = DriverRemoteConnection(
self.conn, pool_size=self.pool_size, max_workers=self.max_workers
)
But then I'm not able to pass a connection/request timeout.

Found the answer by digging through the source code of gremlinpython and aiohttp.
The kwargs passed to DriverRemoteConnection are passed to AiohttpTransport.
So the proper way to pass connection and request timeout parameters is as such:
driver_remote_connection = DriverRemoteConnection(
self.conn,
pool_size=self.pool_size,
max_workers=self.max_workers,
timeout=gremlin_connect_timeout_secs,
read_timeout=gremlin_request_timeout_secs,
)

Related

com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </span>; expected </br>

I created the stub classes using CXF wsdl2java tool.
I am using Apache CXF library, with JCIFS. I validated the WSDL file itself through couple tools, it is good. Here is the code. It looks like some setting I must do.
//JCIFS Authentication related code
jcifs.Config.setProperty("jcifs.smb.client.domain", "NTS");
jcifs.Config.setProperty("jcifs.netbios.wins", "ecmchat.mark.gov");
jcifs.Config.setProperty("jcifs.smb.client.soTimeout", "300000"); // 5 minutes
jcifs.Config.setProperty("jcifs.netbios.cachePolicy", "1200"); // 20 minutes
jcifs.Config.setProperty("jcifs.smb.client.username", "user");
jcifs.Config.setProperty("jcifs.smb.client.password", "password");
//Register the jcifs URL handler to enable NTLM
jcifs.Config.registerSmbURLHandler();
//WSDL and Client settings
URL wsdlURL = BF.WSDL_LOCATION;
if (args.length > 0 && args[0] != null && !"".equals(args[0])) {
File wsdlFile = new File(args[0]);
try {
if (wsdlFile.exists()) {
wsdlURL = wsdlFile.toURI().toURL();
} else {
wsdlURL = new URL(args[0]);
}
} catch (MalformedURLException e) {
e.printStackTrace();
}
}
BF ss = new BF(wsdlURL, SERVICE_NAME);
BFSoap port = ss.getBFSoap12();
Client client = ClientProxy.getClient(port);
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(36000);
httpClientPolicy.setAllowChunking(false);
httpClientPolicy.setReceiveTimeout(32000);
http.setClient(httpClientPolicy);
// Calling the method
System.out.println("Invoking testMethod...");
String _testMethod__return = port.testMethod();
System.out.println("testMethod.result=" + _testMethod__return);
I am getting the following exception
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </span>; expected </br>.
at [row,col,system-id]: [59,22,"https://ecmchat.mark.gov/BF/BF.asmx"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1361)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1255)
at org.apache.cxf.staxutils.StaxUtils.read(StaxUtils.java:1183)
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:235)
... 9 more
If I comment out the JCIFS NTLM authentication code, I get a HTTP 401 error. Therefore, I believe, at least it is passing some kind of authorization step.
And, if I use local WSDL in place of remote URL WSDL, then I get a different error like "method not implemented" on the call to the method. May be this is due to me not using the local WSDL correctly. I do not even know if we can use the local WSDL reference for remote service.
Then, I created a SoapUI dummy service with this WSDL, and the same code (but without the JCIFS authentication code) works good, and successfully calls the methods.
It appears to me that I must add some more appropriate settings in the configuration related code.
Am I right, and are you aware of any, for NTLM authentication and Apache CXF?
But parsing error is confusing???
I do not know if this is related.
My original WSDL URL that I gave was this.
https://ecmchat.mark.gov/BF/BF.asmx
I added a ?wsdl like below
https://ecmchat.mark.gov/BF/BF.asmx?wsdl
Then I am getting a different error.
I wonder why it is working if I access my local SoapUI version of the same WSDL service, but not for the remote one.
Invoking testMethod...
Jan 07, 2020 10:47:25 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
WARNING: Interceptor for {https://ecmchat.mark.gov}BF#{https://ecmchat.mark.gov}testMethod has thrown exception, unwinding now
java.lang.UnsupportedOperationException: Method not implemented.
at java.net.URLStreamHandler.openConnection(URLStreamHandler.java:96)
at java.net.URL.openConnection(URL.java:1028)
at org.apache.cxf.transport.https.HttpsURLConnectionFactory.createConnection(HttpsURLConnectionFactory.java:92)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.createConnection(URLConnectionHTTPConduit.java:121)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.setupConnection(URLConnectionHTTPConduit.java:125)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:505)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
at com.sun.proxy.$Proxy33.testMethod(Unknown Source)
at edison.learn.BFSoap_BFSoap12_Client.main(BFSoap_BFSoap12_Client.java:90)
Exception in thread "main" javax.xml.ws.soap.SOAPFaultException: Method not implemented.
at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:195)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at com.sun.proxy.$Proxy33.testMethod(Unknown Source)
at edison.learn.BFSoap_BFSoap12_Client.main(BFSoap_BFSoap12_Client.java:90)
Caused by: java.lang.UnsupportedOperationException: Method not implemented.
at java.net.URLStreamHandler.openConnection(URLStreamHandler.java:96)
at java.net.URL.openConnection(URL.java:1028)
at org.apache.cxf.transport.https.HttpsURLConnectionFactory.createConnection(HttpsURLConnectionFactory.java:92)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.createConnection(URLConnectionHTTPConduit.java:121)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit.setupConnection(URLConnectionHTTPConduit.java:125)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:505)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
... 2 more

Kong: Client Closing keep-alive connections

I am writing a custom plugin for kong. The plugin will transform request/response in accordance with my server. I am getting [info] 27#0: *588 client <x> closed keepalive connection.
After some debugging, I found that the error occurs whenever I set the
ngx.arg[1] with my transformed response. I have followed the existing response-transformer plugin provided by kong.
This is the body of the kong body_filter function:
local ctx = ngx.ctx
local chunk, eof = ngx.arg[1], ngx.arg[2]
ctx.rt_body_chunks = ctx.rt_body_chunks or {}
ctx.rt_body_chunk_number = ctx.rt_body_chunk_number or 1
if eof then
local someChunks = concat(ctx.rt_body_chunks)
local aBody = responseTransformer.transform(theConf, someChunks)
aBody = unEscapeString(aBody)
ngx.arg[1] = aBody or someChunks
else
ctx.rt_body_chunks[ctx.rt_body_chunk_number] = chunk
ctx.rt_body_chunk_number = ctx.rt_body_chunk_number + 1
ngx.arg[1] = nil
end
I ran the same plugin with a local dummy server. It worked properly. But when I proxied to my actual server, I got the closed keepalive connection error.
From kong logs, I can see that the response got transformed properly.
Using curl, I get about half the body of the response.
Found the cause. The server was sending Content-Length header. During rewrite of body this remained unchanged. So, connection was being closed before the full content was being delivered.
To solve this, I had to clear the Content-Length header in the header_filter function:
kong.response.clear_header("Content-Length")

Flink to Nifi the Magic Header was not present

I am trying to use this example to connect Nifi to Flink:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8090/nifi")
.portName("Data for Flink")
.requestBatchCount(5)
.buildConfig();
SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);
DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource).setParallelism(2);
DataStream<String> dataStream = streamSource.map(new MapFunction<NiFiDataPacket, String>() {
#Override
public String map(NiFiDataPacket value) throws Exception {
return new String(value.getContent(), Charset.defaultCharset());
}
});
dataStream.print();
env.execute();
I am running Nifi as a standalone server with default properties, except these properties:
nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8090
nifi.remote.input.http.enabled=true
The call fails each time, with following log in Nifi:
[Site-to-Site Worker Thread-24] o.a.nifi.remote.SocketRemoteSiteListener
Unable to communicate with remote instance null due to
org.apache.nifi.remote.exception.HandshakeException: Handshake
with nifi://localhost:61680 failed because the Magic Header
was not present; closing connection
Nifi version: 1.7.1, Flink version: 1.7.1
After using the nifi-toolkit I removed the custom value of nifi.remote.input.socket.port and then added transportProtocol(SiteToSiteTransportProtocol.HTTP) to my SiteToSiteClientConfig and http://localhost:8080/nifi as the URL.
The reason why I changed the port in the first place is that without specifying the protocol HTTP it will use RAW by default.
And when using the RAW protocol from Flink side, the client cannot create Transaction and prints the following warning:
Unable to refresh Remote Group's peers due to Remote instance of NiFi
is not configured to allow RAW Socket site-to-site communications
That's why I thought it was a port issue
So now with the default config of Nifi, this works as expected:
SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
.url("http://localhost:8080/nifi")
.portName("portNameAsInNifi")
.transportProtocol(SiteToSiteTransportProtocol.HTTP)
.requestBatchCount(1)
.buildConfig();

X-Appengine-Inbound-Appid header not set

I have two AppEngine modules, a default module running Python and "java" module running Java. I'm accessing the Java module from the default module using urlfetch. According to the AppEngine docs (cloud.google.com/appengine/docs/java/appidentity), I can verify in the Java module that the request originates from a module in the same app by checking the X-Appengine-Inbound-Appid header.
However, this header is not being set (in a production deployment). I use urlfetch in the Python module as follows:
hostname = modules.get_hostname(module="java")
hostname = hostname.replace('.', '-dot-', 2)
url = "http://%s/%s" % (hostname, "_ah/api/...")
result = urlfetch.fetch(url=url, follow_redirects=False, method=urlfetch.GET)
Note that I'm using the notation:
<version>-dot-<module>-dot-<app>.appspot.com
rather than the notation:
<version>.<module>.<app>.appspot.com
which for some reason results in a 404 response.
In the Java module I'm running a servlet filter which looks at all the request headers as follows:
Enumeration<String> headerNames = httpRequest.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
String headerValue = httpRequest.getHeader(headerName);
mLog.info("Header: " + headerName + " = " + headerValue);
}
AppEngine does set some headers, e.g. X-AppEngine-Country. But the X-Appengine-Inbound-Appid header is not set.
Why am I'm not seeing the documented behaviour? Any suggestions would be much appreciated.
Have a look at what I've been answered on Google groups, which led to an issue opened on the public issue tracker.
As suggested in the answer I received you can follow, for any update, the issue over there.

CXF - catching/handling HTTP exceptions

This is rather a basic cxf usage question. How/where can we catch the actual HTTP exception/error. I kind of followed the Interceptor/MessageObserver concept but could not capture the HTTP error using them.
I see this error in the log4j log file.
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response
'401: Unauthorized' when communicating with http://10.107.172.79/test/_vti_bin/lists.asmx
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1502)
at org.apache.cxf.transpot.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1448)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1356)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:614)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 9 more
Only javax.xml.ws.WebServiceException with "Could not send Message."
message is thrown while calling the service
try{
GetListCollectionResult result = port.getListCollection();
}catch (javax.xml.ws.WebServiceException excep){
}
This is how we call the service.
To provide NTLM credentials:
Authenticator.setDefault( extended class of Authenticator);
Create the service.
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setServiceClass(ListsSoap.class);
factory.setAddress(list_url);
ListsSoap port = (ListsSoap) factory.create();
Update the conduit.
..
Client client = ClientProxy.getClient(port);
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(36000);
httpClientPolicy.setAllowChunking(false);
http.setClient(httpClientPolicy);
Call service and get the result.
GetListCollectionResult result = port.getListCollection();
Nevermind, I found the answer in the CXF mailing list.
excep.getCause()
gives access to the underlying exception, in my case it is the HTTP Transport exception.

Resources