Intermitent Base64 Task Conversion Errors - google-app-engine

Question

I'm experiencing a really weird situation when passing on a POJO java object within the payload of a Pull Queue task using Gson. Without changing the code or the POJO being set within the payload of a task, this will randomly succeed or fail.
This is the code I'm using:
PullQueueTaskPayLoad tqp = new PullQueueTaskPayLoad("id","name");
tqp.uploadURL = taskPayLoad.uploadURL;
tqp.urls = taskPayLoad.urls;
tqp.sliceQueryParameter = taskPayLoad.sliceQueryParameter;
TaskOptions task = TaskOptions.Builder.withMethod(TaskOptions.Method.PULL);
task.payload(new Gson().toJson(tqp));
q.add(task);
Using an external queue consumer I then retrieve the POJO as follows:
Type GSON_TYPE = new TypeToken<PullQueueTaskPayLoad>() {}.getType();
byte[] b = new Base64().decodeBase64(leasedTask.getPayloadBase64().getBytes());
String payload = new String(b);
logger.info("About to convert payload: "+payload);
PullQueueTaskPayLoad taskpayload = new Gson().fromJson(payload, GSON_TYPE);
So from the debugging I did, the problem seems to be happening when I'm decoding the payload bytes. While encoding the same POJO (with different Ids) I randomly get 2 different decoded payload Strings as follows:
Correct decoding:
{"id":"1786024566","sliceQueryParameter": {"queryId":786024566,"sliceStart":-1,"sliceNumber":1,"params":{"DefaultAnnotation":{"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints_SIGN":["\u003d"],"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints":["0"],"http://www.slicepedia.org/ontology#hasNumberOfTokens":["80"],"http://www.slicepedia.org/ontology#hasNumberOfTokens_SIGN":["\u003e"]},"VG":{"http://www.slicepedia.org/ontology#hastense":["?"],"http://www.slicepedia.org/ontology#hasroot":["?"]}}},"uploadURL":"http://3.linguabox0412.appspot.com/_ah/upload/AMmfu6YRjxX23Ks-yh-9AZs4-3I1p6hxrFd6d4ptxSQegUkQHN7y4hNZwX6u7PufIHJbwtsHLXFZJ5P-vs90mslZEOMw0T-amN2qhEOAj_6YdwuY50FXMi8/ALBNUaYAAAAAT7Towgs4M00M5RLI8xnEOMdIxouZzuGu/","status":"IN_PROGRESS","action":"SLICE_SEARCH_AND_CREATE"}
Incorrect decoding:
{"id":"1-1968382407","sliceQueryParameter":{"queryId":-1968382407,"sliceStart":-1,"sliceNumber":1,"params":{"DefaultAnnotation":{"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints_SIGN":["\u003d"],"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints":["0"],"http://www.slicepedia.org/ontology#hasNumberOfTokens":["80"],"http://www.slicepedia.org/ontology#hasNumberOfTokens_SIGN":["\u003e"]},"VG":{"http://www.slicepedia.org/ontology#hastense":["?K??????˜?X?\YXK?ܙ?????H?\ܛ????Ȃ%?????'W??EU$?#?&?GG???2?Ɩ?wV&??C"?7?B?6??????W??B???gSe????'u?U'd?D??6?S??4UV?D?e7?%U?&%F%f?D?$???$&vu6?fF$????EG?v??6?6դvt?D???G??&D?fdֵ6%?甦??GD????F???$?V?CuF?$?F?F֤֧f?D??u?wt?4?C$?W?"?'7FGW2#?$???$?u$U52"?&7F???#?%4Ĕ4U?4T$4???E?5$TDR'
So the second string obviously fails when using Gson to convert it back to a POJO. But I dont' understand why this happens in only some cases and not others. For what I've seen, it seems to always happen after a ["?"] character string. I tried replacing and ? with other strings but it didn't change anything.

Answer 1

I think what is happening here is that the payload is webSafe-base64 encoded. In practice, this means swapping + and / and = for - and _ and .. Most base64 libraries have native support for decoding websafe base 64 strings.
Probably you are meeting one of these chars at a certain point, and that kills the decoding.
Here is some info on WebSafeBase64
Word of warning, though: the taskqueue implementation is actually sending padding equals (=) that you will have to convert manually before parsing.