I have a simple Restlet service hosted on AppEngine. This performs basic CRUD operations with strings and is working well with all sorts of UTF-8 characters when I test it with curl (for all the verbs).
This is consumed by a simple restlet client hosted in a servlet on another AppEngine app:
// set response type
resp.setContentType("application/json");
// Create the client resource
ClientResource resource = new ClientResource(Messages.SERVICE_URL + "myentity/id");
// Customize the referrer property
resource.setReferrerRef("myapp");
// Write the response
resource.get().write(resp.getWriter());
The above is pretty much all I have in the servlet. Very plain.
The servlet is invoked via jquery ajax, and the json that I get back is well formed and everything, but the problem is that UTF-8 encoded strings are coming back scrambled, for example:
Université de Montréal becomes Universit?? de Montr??al.
I tried adding this line in the servlet (before everything else):
resp.setCharacterEncoding("UTF-8");
But the only diference is that instead of getting ?? I get Universitￃﾩ de Montrￃﾩal (I don't even know what kind of characters those are, asian I suppose).
I am 100% sure the restlet service is OK, because other than debugging it line by line I am able to test it from cmd line with curl and it's returning well formed strings.
By looking at the http header of the response from firefox (when calling the servlet via javascript) I can see the encoding is indeed UTF-8, as expected. After hours of struggling reading every possible related article I came across this restlet discussion and noticed that indeed I do have Transfer-Encoding: chunked on the http header of the response. I tried the proposed solutions (override ClientResource.toRepresentation, didn't do any good so I tried restlet 2.1 as susggested with ClientResource.setRequestEntityBuffering(true), no luck there either) but I am not convinced my issue is related to Transfer-Encoding: chunked at all.
At this point I am out of ideas, and I would really appreciate any suggestions! O_o
UPDATE:
I tried doing a manual GET with a classic UrlConnection and the string is coming back alright:
URL url = new URL(Messages.SERVICE_URL + "myentity/id");
URLConnection conn = url.openConnection();
InputStream is = conn.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
resp.getWriter().print(writer.toString());
So much for being all RESTful and fancy ...but still I have no clue why the original version doesn't work! :/
I tried doing a manual GET with a classic UrlConnection and the string is coming back alright:
URL url = new URL(Messages.SERVICE_URL + "myentity/id");
URLConnection conn = url.openConnection();
InputStream is = conn.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
resp.getWriter().print(writer.toString());
So much for being all RESTful and fancy ...but still I have no clue why the original version doesn't work! :/
Does your response contain the appropriate "Content-Type" header? It should be something like "Content-Type: application/json; charset=UTF-8" (note the charset).
Try starting your development server and retrieving your resource from the command line using cURL and inspecting the headers, e.g. curl -i http://localhost:8080/myentity/id. In theory browsers should assume UTF-8 for JSON, but I wouldn't trust on that.
Related
I am trying to create a signed URL and upload files from my PC to google cloud storage using it.
I am using Advanced REST Client(ARC) as the client side application. On the server side, I have a jersey based server running on Appengine.
I first send a GET request from ARC, on receiving which the app engine generates a signed URL and returns it back in the response.
After that I do a PUT request with the file I want to upload in the body and the request URL set to what was received in the response to GET.
The code snippet to create signed URL:
String encodedUrl = null;
String contentMD5 = "";
String contentType = "";
String httpVerb;
httpVerb = "PUT";
Calendar calendar = Calendar.getInstance();
calendar.add(Calendar.MINUTE, 10);
long expiration = calendar.getTimeInMillis() / 1000L;
String canonicalizedResource = "/" + bucketName + "/" + objectName;
String baseURL = "https://storage.googleapis.com" + canonicalizedResource;
String stringToSign =
httpVerb + "\n" + contentMD5 + "\n" + contentType + "\n" + expiration + "\n"
+ canonicalizedResource;
AppIdentityService service = AppIdentityServiceFactory.getAppIdentityService();
String googleAccessId = service.getServiceAccountName();
SigningResult signingResult = service.signForApp(stringToSign.getBytes());
String encodedSignature = null;
try {
encodedSignature =
new String(Base64.encodeBase64(signingResult.getSignature(), false), "UTF-8");
} catch (UnsupportedEncodingException e) {
throw new InternalServerErrorException();
}
String signature = null;
try {
signature = URLEncoder.encode(encodedSignature, "UTF-8").toString();
} catch (UnsupportedEncodingException e) {
throw new InternalServerErrorException();
}
encodedUrl =
baseURL + "?GoogleAccessId=" + googleAccessId + "&Expires=" + expiration
+ "&Signature=" + signature;
System.out.println("Signed URL is: "+encodedUrl);
However I observe the following issue:
Whenever I send the PUT request with any file type, I get the following error:
Error - 403
Code - SignatureDoesNotMatch
Message - The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method
Please note that in my code, I am setting the content Type to "" while creating the string to sign. Also while creating the PUT request I don't include any Content-type header.
As far as I understand, if I don't include the contentType in the stringToSign while creating the signed URL and also not add it as a header while sending PUT request it should be fine. So what could be the reason for the error?
After that I changed by code and added the contentType while creating the stringToSign in the code and also gave the corresponding Content-Type header while sending the PUT request.
In this case I am able to upload the file, however the uploaded file is modified/corrupted.I tried with text/plain and image/jpeg.
The problem is that the following text is added at the beginning of the file:
------WebKitFormBoundaryZX8rPPhnm1WXPrUf
Content-Disposition: form-data; name="fileUpload5"; filename="blob"
Content-Type: text/plain
I can see this in the text file and on opening the .jpg file in the hex editor. The .jpg does not open in standard image application since the file has been corrupted by the text in the beginning
Am I missing something here? Is this any issue in the Advanced REST Client?
Actually whenever I send a PUT request with some file in the body, I get a message in the ARC saying that :
The content-type header will be finally changed to multipart/form-data while sending the request
However, I saved exported all the messages to a file from ARC and I didn't find any message with Content-type header set to multipart/form-data.
So why does this message come and is it actually an issue?
URL-signing code is tricky and notoriously hard to debug. Fortunately, Google's google-cloud library has a signUrl function that takes care of this for you. I highly encourage you to use it instead of rewriting it yourself. Here's the documentation.
Now, if you want to debug it yourself, checking the error message is super useful. It will include a complete copy of the string the server checked the signature of. Print out your stringToSign variable and see how it's different. That'll tell you what's wrong.
Now, on to your specific problem: it sounds like you are generating an acceptable signed URL, but then your client is attempting to upload to GCS as if it were doing a multipart, form upload. The text you're looking at is part of an HTTP multipart request, and the "multipart/form-data" warning also points in that direction. See if the app you're using has some sort of "Form" mode/option that you are perhaps accidentally using?
I´m using CXF to consume a WebService and, as the responses are quite large, I´m requesting with a gzip "Accept-Encoding" and using GZIPInInterceptor to handle the gziped response. Also my WSDL is very large (360kb) and it takes a long time(+10 seconds) to create the stub, because it has to read and parse the WSDL, so I´m creating the stub once and reusing it.
The problem is, whenever I try to use two different methods the second request gives me an error saying it is expecting the previous request.
To illustrate my problem I created a simple example with this public WebService:
http://www.webservicex.net/BibleWebservice.asmx?WSDL
Without the GZip compression it works fine:
BibleWebserviceSoap bibleService = new BibleWebservice().getBibleWebserviceSoap();
String title = bibleService.getBookTitles();
response.getWriter().write(title);
String johnResponse = bibleService.getBibleWordsbyKeyWord("John");
response.getWriter().write(johnResponse);
I´m able to receive both responses.
Enabling Gzip compression:
BibleWebserviceSoap bibleService = new BibleWebservice().getBibleWebserviceSoap();
//GZIP compression on bibleService
Client client = ClientProxy.getClient(bibleService);
client.getInInterceptors().add(new GZIPInInterceptor());
client.getInFaultInterceptors().add(new GZIPInInterceptor());
// Creating HTTP headers
Map<String, List<String>> headers = new HashMap<String, List<String>>();
headers.put("Accept-Encoding", Arrays.asList("gzip"));
// Add HTTP headers to the web service request
client.getRequestContext().put(Message.PROTOCOL_HEADERS, headers);
String title = bibleService.getBookTitles();
response.getWriter().write(title);
String johnResponse = bibleService.getBibleWordsbyKeyWord("John");
response.getWriter().write(johnResponse);
When I try to receive the second response I´m getting this exception:
org.apache.cxf.interceptor.Fault: Unexpected wrapper element {http://www.webserviceX.NET}GetBookTitlesResponse found. Expected {http://www.webserviceX.NET}GetBibleWordsbyKeyWordResponse.
On my real application I´m getting an exception with the request:
org.apache.cxf.binding.soap.SoapFault: OperationFormatter encountered an invalid Message body. Expected to find node type 'Element' with name 'GetAvailabilityRequest' and namespace 'http://schemas.navitaire.com/WebServices/ServiceContracts/BookingService'. Found node type 'Element' with name 'ns4:PriceItineraryRequest' and namespace 'http://schemas.navitaire.com/WebServices/ServiceContracts/BookingService'
My sample project can be downloaded here:
http://www.sendspace.com/file/plt0m4
Thank you
Instead of setting the protocol headers directly like that, use CXF's GZIPOutInterceptor to handle that.
Either that or reset the PROTOCOL headers for each request. When set like that, the headers map gets updated as the request goes through the chain. In this case, the soapaction gets set. This then gets resent on the second request.
I am using the GAE Blobstore with Jersey REST on ther server side. I send a GET request to the server via Android and include a query parameter called logindx. My server side code snippet looks like this:
#Path("/getuploadurl")
#GET
#Produces(MediaType.TEXT_PLAIN)
public Response getUploadUrl(#QueryParam("logindx") Long logIndx ) {
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
String uurl = blobstoreService.createUploadUrl("/logblobkey");
logger.severe("urltest: " + uurl);
return Response.ok(uurl).build();
}
The problem is that the result String I get back at Android (and which is also logged) is:
urltest: http://bardroid123.appspot.com/_ah/upload/?logindx=-43803902306520/AMmfu6b2Ubvf17gD_5uheZeDhTIsr8nm582oaNi0_SDPWfuxqHmYgtkWqVVP52QbBwnnNbWyJf_lDdf9GDmFKtdHU_eUn5gjjtrOSAB32HSu3HiVgLovO5pYeYDkapBPfu7uuo460Ez0/ALBNUaYAAAAAUeuzYniVLlTqyYCjIkfK7-n0ARv5yoo1/
The part ?logindx=-43803902306520/ in the above upload URL should surely not be there? Ho does the createUploadUrl function even know how to get hold of the HttpRequest object to extract the query parameter?
The problem is when I try to use the above uri in my android app like so:
HttpPost postRequest = new HttpPost(uri);
I get the following error:
java.lang.IllegalArgumentException: Illegal character in query at index 253: http://bardroid123.appspot.com/_ah/upload/?logindx=-43803902306520/AMmfu6ZDQr7WenGd0N3ZkbI3zfSl0xPcY56XS5p_VQiS_MWxtTwtc1xm8NbhdrhK-PxopCIolsWci_06DQ3EsUJXSlbiavtJKX9JXT7RU3vTnwj-H0yY5DZKv9hbYR0brfOezaVwob1k/ALBNUaYAAAAAUevBZWOmVC0m1tipSR7Lk9WcwePsXBzf/
Even more confusing is that I don't get the ?logindx=-43803902306520/ part when I do the get request on my local server (from Eclipse provided by App Engine):
http://localhost:8888/res/logs/getuploadurl?logindx=1234567.
In that case the browser returns something like:
http://localhost:8888/_ah/upload/agtiYXJkcm9pZDEyM3IbCxIVX19CbG9iVXBsb2FkU2Vzc2lvbl9fGDIM
Clearly it has got nothing to do with Android and I can't see how this can be Jersey specific either.
Any help would be greatly appreciated.
Thanks - from Africa.
EDIT:
I got it right now by simply dropping the last slash (/) in the uri and the Illegal character in query error went away. The uri was working perfectly with the Blobstore with the ?logindx=-43803902306520/ part included. Don't matter now, but still wondering why it is included in the upload uri?
Well I am wondering how I can achieve to post a multipart in chunked mode. I have 3 parts, and the files which can be big so must be sent in chunks.
Here what I do :
MultipartEntity multipartEntity = new MultipartEntity() {
#Override
public boolean isChunked() {
return true;
}
};
multipartEntity.addPart("theText", new StringBody("some text", Charset.forName("UTF-8")));
FileBody fileBody1 = new FileBody(file1);
multipartEntity.addPart("theFile1", fileBody1);
FileBody fileBody2 = new FileBody(file2);
multipartEntity.addPart("theFile2", fileBody2);
httppost.setEntity(multipartEntity);
HttpParams params = new BasicHttpParams();
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpClient httpClient = new DefaultHttpClient(params);
HttpResponse httpResponse = httpClient.execute(httppost);
On the server side, I do receive the 3 parts but the files for example are not chunked, they are received as one piece... basically total I see 4 boundaries appearing only : 3 --xxx, 1 at the end --xxx-- .
I thought the override of isChunked would do the trick but no... ;(
Is what I am trying to do feasible ? How could I make that work ?
Thanks a lot.
Fab
To generate a multipart body chunked, one of the part must have it size unavailable. Like a part that is streaming.
For example let assume your file2 is a really big video. You could replace the part of your code:
FileBody fileBody2 = new FileBody(file2);
multipartEntity.addPart("theFile2", fileBody2);
wtih that code:
final InputStreamBody binVideo = new InputStreamBody(new FileInputStream(file2), "video/mp4", file2.getName());
multipartEntity.addPart("video", binVideo);
since now the third part is an InputStream instead of File, your multipart HTTP request will have the header Transfer-Encoding: chunked.
Usually any decent server-side HTTP framework (such as Java EE Servlet API) would hide transport details such as transfer coding from the application code. just because you are not seeing chunk delimiters by reading from the content stream does not mean the chunk coding was not used by the underlying HTTP transport.
You can see exactly what kind of HTTP packets HttpClient generates by activating the wire logging as described here:
http://hc.apache.org/httpcomponents-client-ga/logging.html
I'm downloading (from server to client) a file using a WebClient object:
WebClient wc = new WebClient();
wc.OpenReadCompleted += Load_TransferCompleted;
wc.OpenReadAsync(uriAddress, Filename);
I would like to know the file size before starting the download operation. Is there a way to do this in SL4?
thanks for your help.
Gilad.
Here is some air code for you to play with (I haven't tested it myself)
WebRequest req = WebRequestCreator.ClientHttp.Create(yourUri);
req.Method = "HEAD";
req.BeginGetResponse(ar =>
{
WebResponse resp = req.EndGetResponse(ar);
int length = resp.ContentLength;
// Do stuff with length
}, null);
By using the ClientHttp stack you can use "HEAD" request which will return the same set of headers as a "GET" but not the actual entity body.
There is at least one thing to look out for though, none of the existing cookies for the uri will be sent in the request. If the response is sensitive to cookies (for example because it needs a session id) then things will get a whole lot more complicated.
The only way I see to make this possible is by publishing the size. It could be coded but also obtainable through a web service.