This may be related to Google File Service closeFinally() timeout.
My Java GAE app is running on SDK 1.8.2 and is hitting a translateException when calling closeFinally() on a FileWriteChannel. I'll be updating the SDK and migrating to the new Cloud Storage API but need to resolve this exception asap for a customer so am wondering if anyone has some insight into the issue for a quick fix.
Here's a simplification of the code that's causing the issue:
AppEngineFile writableFile = fileService.createNewGSFile(optionsBuilder.build());
String gcsWritePath = writableFile.getFullPath();
FileWriteChannel writeChannel = fileService.openWriteChannel(writableFile, true);
for(conditions) {
Do stuff and write to gcsWritePath
if(recordsWritten > threshold) {
Kick off new task with a startAtparam to avoid OOM and timeouts
break;
}
}
if(all records written) {
writeChannel.closeFinally(); // Exception happens here
}
And the exception details
java.io.IOException
at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:621)
at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:594)
at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:565)
at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:457)
at com.google.appengine.api.files.FileWriteChannelImpl.closeFinally(FileWriteChannelImpl.java:97)
The exception only appears to occur for longer running tasks (e.g. 5+ minutes) so it looks like this might be a timeout. I've tried adjusting the threshold without much luck though. I also tried looking through the app engine sdk source for FileServiceImpl.java and ApiProxy.java but didn't find any good clues.
Does anyone have any ideas as to what might be causing this?
Related
The specific action we are trying to performe is to create a charge request with Stripe:
Charge charge = Charge.create(params);
Using Stripe's Java implementation (version 5.35.1) we encounter a APIConnectionException when running the application on the deployed App Engine server. Interestingly, the issue does not occur using App Engine's local dev server.
We contacted Stripe and they said everything is fine on their side. They explained that "this error indicates that your server is not able to contact our API and that it ends up timing out waiting for an answer. Usually, this is due to something misconfigured on your server such as a DNS not redirecting to Stripe properly."
We were wondering if there are some configuration settings on App Engine which cause the problem. However, since there is no specific error message we cound not figure out what might cause the connection problem.
Similar problems which did not provide enough help to resolve this issue: https://issuetracker.google.com/35901039
Thanks for your help!
The problem seems to have solved itself. It occured for a couple of days and since a few hours the exact same code is working fine. Just like in https://issuetracker.google.com/issues/35901039 the problem is suddenly gone for some reason.
I'm using Drive API on Google app engine.
Application servers 7k request per day.
During last day number of errors from api increased to nearly 60%.
{ "code" : 500, "message" : null }
I use this code to initialize drive:
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
AppIdentityCredential credential =
new AppIdentityCredential.Builder(Arrays.asList(DriveScopes.DRIVE)).build();
GoogleClientRequestInitializer keyInitializer =
new CommonGoogleClientRequestInitializer(Settings.API_KEY);
Drive service = new Drive.Builder(httpTransport, jsonFactory, null)
.setHttpRequestInitializer(credential)
.setApplicationName(APP_NAME)
.setGoogleClientRequestInitializer(keyInitializer)
.build();
return service;
Does any one have same situation?
Are there any solutions?
Thanks!
UPDATE:
Have started working without any changes from my side.
There are many bugs in Drive that can cause hard 500 errors, and also many transient internal scenarios (esp. timeouts) that can cause them. It's important that you do as much research as possible so you can distinguish between the two, since some are permanent whereas others may succeed after a backoff and retry.
In your case, I suspect you are tripping over the infrastructure issues that Google have confirmed have been affecting Drive (and perhaps other services) over the last few days. See https://plus.google.com/106160348960403302854/posts/CwH9SEDTQ4C
The quickstart app started acting strangely for me. I can't auth any users. I'm getting the below error. This works fine locally though. Anyone know how to fix this or get around it?
Here's where it's happening in the code (AuthServlet:55)
LOG.info("Got a code. Attempting to exchange for access token.");
AuthorizationCodeFlow flow = AuthUtil.newAuthorizationCodeFlow();
TokenResponse tokenResponse =
flow.newTokenRequest(req.getParameter("code"))
.setRedirectUri(WebUtil.buildUrl(req, "/oauth2callback")).execute();
Uncaught exception from servlet
java.net.SocketTimeoutException: Timeout while fetching URL:
https://accounts.google.com/o/oauth2/token
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:141)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
at com.google.api.client.extensions.appengine.http.UrlFetchRequest.execute(UrlFetchRequest.java:75)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:980)
at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:299)
at com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeTokenRequest.execute(GoogleAuthorizationCodeTokenRequest.java:175)
at com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeTokenRequest.execute(GoogleAuthorizationCodeTokenRequest.java:78)
at com.google.glassware.AuthServlet.doGet(AuthServlet.java:55)
Thanks in advance
This looks like a bug in App Engine and the Google OAuth endpoints. It's being tracked in the Glass issue tracker. You can star that issue to receive updates as the investigation proceeds.
In the mean time, check out the .NET and PHP quick starts. They're working great.
Google has had some intermittent connectivity issues over the last two days. It appears to be working fine now.
See reports on the Google App Engine downtime notification group
https://groups.google.com/forum/?fromgroups#!topic/google-appengine-downtime-notify/TqKVL9TNq2A
I have a production issue with a GAE application. The problem seems to be related only to some users.
Is there a possibility to connect the local debugger to the prod version ? I'd like to avoid to have to copy the prod data to the dev server.
Thanks !
This situation has improved as of a few days ago with the intoduction of Google Cloud Debugger
Earlier this week at Google Cloud Platform Live, we launched the beta release of Cloud Debugger which makes it easier to troubleshoot applications in production. Now you can simply pick a line of code, set a watchpoint and the debugger will return local variables and a full stack trace from the next request that executes that line on any replica of your service. There is zero setup time, no complex configurations and no performance impact noticeable to your users.
Here's the blog post
Here's the info
Not possible, you can only debug using the logs.
I managed to solve the issue. The problem was that one program was throwing an exception and I did not catch it properly. The code was
catch (Exception e) {
e.printStackTrace();
}
On the development platform the exception was thrown at the console. The problem in production is that it does not appear in the GAE console. This has two very bad outcomes :
--> 1) The end user does not know that the transaction did not end successfully
--> 2) The administrator has no clue that a problem has occurred and no way to debug
==>
I have therefore replaced the code by
catch (Exception e) {
final Logger log = Logger.getLogger(InternalServiceImpl.class.getName());
log.severe("Problem while updating XXX record : " + e.getLocalizedMessage());
updatedRecord.setRPCResult(DB_PROBLEM); // This is specific to my application to notify the end user
}
I did modify the code relaunched the application and found out that the problem was a resource contention in a transaction.
Was really a stupid beginner mistake.
Recently I experienced two problems in uploading files to my Java gae app.
I'm using the tecnique described in the blobstore doc.
With regular files, occasionally (let's say 15% of times) the client receives a "503 Service Unavailable".
With high resolution images (example 7000x10000) the client always receives a "400 Bad Request".
On both cases on the server there are no error messages logged, the blobs are written correctly, but the successPath url (the callback of createUploadUrl) is never called. It seems that the GAE endpoint handling the upload crashes for some reasons.
My client is a js XMLHttpRequest, wrapped in GWT:
public native void uploadWithXMLHttpRequest(UploadForm uploadForm) /*-{
var fd = new FormData();
var files = uploadForm.#mypackage.UploadForm::getFiles()();
for (var i = 0; i < files.length; i++) {
fd.append("uploadFile"+i, files[i]);
}
var xhr = new XMLHttpRequest();
//xhr.upload.addEventListeners... omitted
xhr.open("POST", uploadForm.#mypackage.UploadForm::getUploadUrl()());
xhr.send(fd);
}
Any ideas for possible causes and solutions/workarounds?
Thx.
This issue is being discussed in a GAE ticket opened by another user having the same problem: https://code.google.com/p/googleappengine/issues/detail?id=7619 (btw, the bug tracker system has a "start" feature, which allows you to vote for the ticket and receive notifications)
Possible reason:
1 You uploading large file (> 1MB) and writing it all. You should write it portinal: 1 write = 1MB.
2 Your request takes longer than 30 sec - use backend.
In this case the 503 is caused by errors when we write the upload info into your datastore. As you are using M/S datastore then transient errors are expected from time to time. I suggest you convert your app to HRD to minimize the chances of there being errors related to writing the upload information to your datastore.
The 400 error was generated by your app & is in your application logs.
Try to use Google Cloud Storage, since the blob store service has lot of problem, so google is trying to migrate uses from Blob to GCS support
I guess the image resolution can't exceed 8000 in the app engine blob store that is the reason it caused.