I am using talend-ESB and want to parse EDI message to XML using smooks & I am getting null in body. The code looks as below.
from(
"file://D:/cimt/InvoiceEDI_Mapping/" + "?noop=true"
+ "&autoCreate=true" + "&flatten=false"
+ "&fileName=InDev_EDI_Msg.txt" + "&bufferSize=128")
.routeId("TestSmooksConfig_cFile_1")
.log(org.apache.camel.LoggingLevel.WARN,
"TestSmooksConfig.cLog_1", "${body}")
.id("TestSmooksConfig_cLog_1")
.to("smooks://EDI_Config.xml")
.to("log:TestSmooksConfig.cLog_2" + "?level=WARN")
.id("TestSmooksConfig_cLog_2");
}
My Talend route looks as below.
I used following set of external dependencies.
milyn-commons-1.7.0.jar
milyn-smooks-camel-1.7.0.jar
milyn-smooks-edi-1.7.0.jar
milyn-smooks-core-1.7.0.jar
jaxen-1.1.6.jar
milyn-edisax-parser-1.4.jar
Also, I see a strange behavior that, upon execution, I still see "starting" prior to cJavaDSLProcessor, which initially made me wonder if at all it gets executed. But later, when I intentionally made a mistake in EDI-Mapping, then the route was throwing errors, which kind of convinced me that it does parse the EDI message.
I did also search before posting this question here, and found a similar problem in this link
And I tried to lower my revision of org.milyn.* jars to 1.4.0, and got an exception that the route could not register smooks component. So I continued using 1.7.0 version of org.milyn.* jars.
For the benefit of others who might bump into similar issue, I 'assume' that the output of the smooks gets written into an Object of type StringResult.class. However, in my initial implementation, there was no such option and hence the output body was null.
Later, I tried alternative approach from http://smooks.org/guide where they used processor endpoint.Actually they had even made a statement that the data could be retrieved through exports element. The below code snippet helped to fix issue.
Smooks smooks = new Smooks("edi-to-xml-smooks-config.xml");
ExecutionContext context = smooks.createExecutionContext();
smooks.setExports(new Exports(StringResult.class));
SmooksProcessor processor = new SmooksProcessor(smooks, context);
from("file://input?noop=true")
.process(processor)
.to("mock:result");
My website has a search form where someone can search a URL beginning with http:// like this:
https://www.google.com
which should then be encoded and appended as a query parameter value like this:
localhost:4000/api/https%3A%2F%2Fwww.google.com
When I run it (above) locally, it works, but when deployed (below):
https://api.mysite.com/search/api/https%3A%2F%2Fwww.google.com%2F
=> returns 404.
If I type this in:
http://localhost:4000/api/https://www.google.com
I get this error:
Phoenix.Router.NoRouteError at GET /api/v1/https://www.google.com
no route found for GET /api/v1/https:/www.google.com (ExternalPing.Router)
I'm not sure if these are related. What is the correct way to append a url as a query parameter value?
I have already tried encoding with URI.encode and URI.encode_www_form but they didn't resolve this
Now you haven't posted your server code, so I am just going to assume here.
I think the problem is that you didn't encode the second string, since it contains / in the url you have problems.
The url is:
http://localhost:4000/api/https://www.google.com
The server will interpret it wrong. So you are asking for a route called:
/api/https:/
With a parameter called /www.google.com
You need to encode the query string.
But again this is guessing since I have no idea how your server looks.
I just tried calling an endpoint at my iis server with a unencoded url as a parameter, and this is what it gave me back:
<Error>
<Message>The request is invalid.</Message>
</Error>
I come here because I am searching (like the title mentionned) to do a query from geotools (through geoserver) to get feature from a solr index.
To be more precise :
I saw on geoserver user manual that i can do query on solr like this in http :
http://localhost:8080/geoserver/wfs?service=WFS&version=1.1.0&request=GetFeature
&typeName=mySolrLayer
&format="xxx"
&viewparams=q:"mySolrQuery"
The important part on this URL is the viewparams that I want to use directly from geotools.
I have already test this case (this is a part of my code):
url = new URL(
"http://localhost:8080/geoserver/wfs?request=GetCapabilities&VERSION=1.1.0";
);
Map<String, String> param = new HashMap();
params.put(WFSDataStoreFactory.URL.key, url);
param.put("viewparams","q:myquery");
Hints hints = new Hints();
hints.put(Hints.VIRTUAL_TABLE_PARAMETERS, viewParams);
query.setHints(hints);
...
featureSource.getFeatures(query);
But here, it seems to doesn't work, the url send to geoserver is a normal "GET FEATURE" request without the viewparams parameter.
I tried this with geotools-12.2 ; geotools-13.2 and geotools-15-SNAPSHOT but I didn't succeed to pass the query, geoserver send me all the feature in my database and doesn't take "viewparams" as a param.
I need to do it like this because actually the query come from another program and I would easily communicate this query to another part of the project...
If someone can help me ?
There doesn't currently seem to be a way to do this in the GeoTool's WFSDatastore implementations as the GetFeature request is constructed from the URL provided by the getCapabilities document. This is as the standard requires but it may be worth making a feature enhancement request to allow clients to override this string (as QGIS does for example) which would let you specify the additional parameter in your base URL which would then be passed to the server as you need.
Unfortunately the WFS module lives in Unsupported land at present so unless you have resources to work on this issue yourself and can provide a PR to implement it there is not a great chance of it being implemented.
When uploading to GCS (Google Cloud Storage) using the BlobStore's createUploadURL function, I can provide a callback together with header data that will be POSTed to the callback URL.
There doesn't seem to be a way to do that with GCS's signed URL's
I know there is Object Change Notification but that won't allow the user to provide upload specific information in the header of a POST, the way it is possible with createUploadURL's callback.
My feeling is, if createUploadURL can do it, there must be a way to do it with signed URL's, but I can't find any documentation on it. I was wondering if anyone may know how createUploadURL achieves that callback calling behavior.
PS: I'm trying to move away from createUploadURL because of the __BlobInfo__ entities it creates, which for my specific use case I do not need, and somehow seem to be indelible and are wasting storage space.
Update: It worked! Here is how:
Short Answer: It cannot be done with PUT, but can be done with POST
Long Answer:
If you look at the signed-URL page, in front of HTTP_Verb, under Description, there is a subtle note that this page is only relevant to GET, HEAD, PUT, and DELETE, but POST is a completely different game. I had missed this, but it turned out to be very important.
There is a whole page of HTTP Headers that does not list an important header that can be used with POST; that header is success_action_redirect, as voscausa correctly answered.
In the POST page Google "strongly recommends" using PUT, unless dealing with form data. However, POST has a few nice features that PUT does not have. They may worry that POST gives us too many strings to hang ourselves with.
But I'd say it is totally worth dropping createUploadURL, and writing your own code to redirect to a callback. Here is how:
Code:
If you are working in Python voscausa's code is very helpful.
I'm using apejs to write javascript in a Java app, so my code looks like this:
var exp = new Date()
exp.setTime(exp.getTime() + 1000 * 60 * 100); //100 minutes
json['GoogleAccessId'] = String(appIdentity.getServiceAccountName())
json['key'] = keyGenerator()
json['bucket'] = bucket
json['Expires'] = exp.toISOString();
json['success_action_redirect'] = "https://" + request.getServerName() + "/test2/";
json['uri'] = 'https://' + bucket + '.storage.googleapis.com/';
var policy = {'expiration': json.Expires
, 'conditions': [
["starts-with", "$key", json.key],
{'Expires': json.Expires},
{'bucket': json.bucket},
{"success_action_redirect": json.success_action_redirect}
]
};
var plain = StringToBytes(JSON.stringify(policy))
json['policy'] = String(Base64.encodeBase64String(plain))
var result = appIdentity.signForApp(Base64.encodeBase64(plain, false));
json['signature'] = String(Base64.encodeBase64String(result.getSignature()))
The code above first provides the relevant fields.
Then creates a policy object. Then it stringify's the object and converts it into a byte array (you can use .getBytes in Java. I had to write a function for javascript).
A base64 encoded version of this array, populates the policy field.
Then it is signed using the appidentity package. Finally the signature is base64 encoded, and we are done.
On the client side, all members of the json object will be added to the Form, except the uri which is the form's address.
var formData = new FormData(document.forms.namedItem('upload'));
var blob = new Blob([thedata], {type: 'application/json'})
var keys = ['GoogleAccessId', 'key', 'bucket', 'Expires', 'success_action_redirect', 'policy', 'signature']
for(field in keys)
formData.append(keys[field], url[keys[field]])
formData.append('file', blob)
var rest = new XMLHttpRequest();
rest.open('POST', url.uri)
rest.onload = callback_function
rest.send(formData)
If you do not provide a redirect, the response status will be 204 for success. But if you do redirect, the status will be 200. If you got 403 or 400 something about the signature or policy maybe wrong. Look at the responseText. If is often helpful.
A few things to note:
Both POST and PUT have a signature field, but these mean slightly different things. In case of POST, this is a signature of the policy.
PUT has a baseurl which contains the key (object name), but the URL used for POST may only include bucket name
PUT requires expiration as seconds from UNIX epoch, but POST wants it as an ISO string.
A PUT signature should be URL encoded (Java: by wrapping it with a URLEncoder.encode call). But for POST, Base64 encoding suffices.
By extension, for POST do Base64.encodeBase64String(result.getSignature()), and do not use the Base64.encodeBase64URLSafeString function
You cannot pass extra headers with the POST; only those listed in the POST page are allowed.
If you provide a URL for success_action_redirect, it will receive a GET with the key, bucket and eTag.
The other benefit of using POST is you can provide size limits. With PUT however, if a file breached your size restriction, you can only delete it after it was fully uploaded, even if it is multiple-tera-bytes.
What is wrong with createUploadURL?
The method above is a manual createUploadURL.
But:
You don't get those __BlobInfo__ objects which create many indexes and are indelible. This irritates me as it wastes a lot of space (which reminds me of a separate issue: issue 4231. Please go give it a star)
You can provide your own object name, which helps create folders in your bucket.
You can provide different expiration dates for each link.
For the very very few javascript app-engineers:
function StringToBytes(sz) {
map = function(x) {return x.charCodeAt(0)}
return sz.split('').map(map)
}
You can include succes_action_redirect in a policy document when you use GCS post object.
Docs here: Docs: https://cloud.google.com/storage/docs/xml-api/post-object
Python example here: https://github.com/voscausa/appengine-gcs-upload
Example callback result:
def ok(self):
""" GCS upload success callback """
logging.debug('GCS upload result : %s' % self.request.query_string)
bucket = self.request.get('bucket', default_value='')
key = self.request.get('key', default_value='')
key_parts = key.rsplit('/', 1)
folder = key_parts[0] if len(key_parts) > 1 else None
A solution I am using is to turn on Object Changed Notifications. Any time an object is added, a Post is sent to a URL - in my case - a servlet in my project.
In the doPost() I get all info of objected added to GCS and from there, I can do whatever.
This worked great in my App Engine project.
I am using the GAE Blobstore with Jersey REST on ther server side. I send a GET request to the server via Android and include a query parameter called logindx. My server side code snippet looks like this:
#Path("/getuploadurl")
#GET
#Produces(MediaType.TEXT_PLAIN)
public Response getUploadUrl(#QueryParam("logindx") Long logIndx ) {
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
String uurl = blobstoreService.createUploadUrl("/logblobkey");
logger.severe("urltest: " + uurl);
return Response.ok(uurl).build();
}
The problem is that the result String I get back at Android (and which is also logged) is:
urltest: http://bardroid123.appspot.com/_ah/upload/?logindx=-43803902306520/AMmfu6b2Ubvf17gD_5uheZeDhTIsr8nm582oaNi0_SDPWfuxqHmYgtkWqVVP52QbBwnnNbWyJf_lDdf9GDmFKtdHU_eUn5gjjtrOSAB32HSu3HiVgLovO5pYeYDkapBPfu7uuo460Ez0/ALBNUaYAAAAAUeuzYniVLlTqyYCjIkfK7-n0ARv5yoo1/
The part ?logindx=-43803902306520/ in the above upload URL should surely not be there? Ho does the createUploadUrl function even know how to get hold of the HttpRequest object to extract the query parameter?
The problem is when I try to use the above uri in my android app like so:
HttpPost postRequest = new HttpPost(uri);
I get the following error:
java.lang.IllegalArgumentException: Illegal character in query at index 253: http://bardroid123.appspot.com/_ah/upload/?logindx=-43803902306520/AMmfu6ZDQr7WenGd0N3ZkbI3zfSl0xPcY56XS5p_VQiS_MWxtTwtc1xm8NbhdrhK-PxopCIolsWci_06DQ3EsUJXSlbiavtJKX9JXT7RU3vTnwj-H0yY5DZKv9hbYR0brfOezaVwob1k/ALBNUaYAAAAAUevBZWOmVC0m1tipSR7Lk9WcwePsXBzf/
Even more confusing is that I don't get the ?logindx=-43803902306520/ part when I do the get request on my local server (from Eclipse provided by App Engine):
http://localhost:8888/res/logs/getuploadurl?logindx=1234567.
In that case the browser returns something like:
http://localhost:8888/_ah/upload/agtiYXJkcm9pZDEyM3IbCxIVX19CbG9iVXBsb2FkU2Vzc2lvbl9fGDIM
Clearly it has got nothing to do with Android and I can't see how this can be Jersey specific either.
Any help would be greatly appreciated.
Thanks - from Africa.
EDIT:
I got it right now by simply dropping the last slash (/) in the uri and the Illegal character in query error went away. The uri was working perfectly with the Blobstore with the ?logindx=-43803902306520/ part included. Don't matter now, but still wondering why it is included in the upload uri?