What causes Memcache operation failed, giving up exception? - google-app-engine

I have an app engine java project and am using objectify. I get a stack trace sporadically in the "stack driver error reporting" view of the app engine web console related to putting an item into memcache. This is the code:
try {
TestItem t = new TestItem(...);
ofy().save().entity(t).now();
} catch (Exception e) {
}
and this is the error I'll see sporadically:
com.googlecode.objectify.cache.MemcacheServiceRetryProxy invoke: Memcache operation failed, giving up
java.lang.reflect.InvocationTargetException
at com.google.appengine.runtime.Request.process-i4dx9s2kED3CVcPe(Request.java)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:44)
at com.googlecode.objectify.cache.MemcacheServiceRetryProxy.invoke(MemcacheServiceRetryProxy.java:68)
at com.sun.proxy.$Proxy9.putAll(Unknown Source)
at com.googlecode.objectify.cache.KeyMemcacheService.putAll(KeyMemcacheService.java:91)
at com.googlecode.objectify.cache.EntityMemcache.empty(EntityMemcache.java:319)
at com.googlecode.objectify.cache.CachingAsyncDatastoreService$5.trigger(CachingAsyncDatastoreService.java:445)
at com.googlecode.objectify.cache.TriggerFuture.isDone(TriggerFuture.java:87)
at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:102)
at com.googlecode.objectify.impl.ResultAdapter.now(ResultAdapter.java:34)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:22)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:10)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:22)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:10)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.me.test.Test.putSomethinInMemcache(Test.java:13)
...
Caused by: com.google.appengine.api.memcache.MemcacheServiceException: Memcache putAll: Unknown exception setting 1 keys
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.handleApiProxyException(MemcacheServiceApiHelper.java:69)
at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$RpcResponseHandlerForPut.handleApiProxyException(AsyncMemcacheServiceImpl.java:349)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentException(MemcacheServiceApiHelper.java:111)
at com.google.appengine.api.utils.FutureWrapper.handleParentException(FutureWrapper.java:52)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:91)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:26)
at com.google.appengine.api.memcache.MemcacheServiceImpl.putAll(MemcacheServiceImpl.java:115)
... 52 more
It doesn't appear to be caught in the try-statement. I just see it in that admin console mentioned earlier.
Does anyone know what this means, or how I can catch it? My main worry is that there could be an old copy of the object stuck in memcache after this operation fails.
Using objectify 5.1.10.
Thanks

This is a get() operation. If memcache is unavailable during a get() operation, Objectify just reads from the datastore. The error is logged and performance suffers somewhat but the app marches on.
It is technically possible to have errors during write operations (any save() clears the cache entry; reads will repopulate the cache). This could in theory leave stale info in the cache. There's nothing that can really be done about this - if you can't clear the cache entry, it's going to be stuck there. My advice is that if you have sensitive data but want it cached, put a reasonable timeout on the cache entry (#Cache(expirationSeconds=60) or whatnot).

Related

Flink1.5.4 exception: Corrupt stream, found tag: 105

My program wants to join two streams without Flink Window.
I connect two streams and define a class A extends RichCoFlatMapFunction to handle them.
In class A, I use a Guava cache to hold all the data from flatmap1/2 method, and join them by a tag from streams.
Then Guava cache has a remove listener to collect joined&expired data to next Flink Function.
private synchronized void collect(ReqFeatures features) {
feaCollector.collect(features);
}
Each time at the beginning, it runs well, but a few hours later, it's always dead because of this exception.
java.io.IOException: Corrupt stream, found tag: 105
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:220)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:49)
at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:106)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:172)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:104)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
at java.lang.Thread.run(Thread.java:748)
And sometimes there's another error log:
java.lang.IllegalStateException: When there are multiple buffers, an unfinished bufferConsumer can not be at the head of the buffers queue.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.runtime.io.network.partition.PipelinedSubpartition.pollBuffer(PipelinedSubpartition.java:158)
at org.apache.flink.runtime.io.network.partition.PipelinedSubpartitionView.getNextBuffer(PipelinedSubpartitionView.java:51)
at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.getNextBuffer(LocalInputChannel.java:186)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:551)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:508)
at org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:94)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:209)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:104)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
at java.lang.Thread.run(Thread.java:748)
If I use Flink Window Function instead, this exception doesn't occur.
Why does this exception occur, and how can I resolve it?
I can confirm this also happens in Flink 1.9.1 (albeit for us, it happens when we run flink stop <job-id>)
I fixed the same problem with getting checkpointing lock while collecting output. The users flatMap function already hold the checkpointing lock, so if u collect output in flatMap function could also fix this problem.
in flink's code:
synchronized (checkpointingLock) {
numRecordsIn.inc();
streamOperator.setKeyContextElement1(record);
streamOperator.processElement(record);
}

Objectify save without hitting Memcache

My entity is #Cached and that's fine.
But, for a cron job, I don't want ofy().save.entities() to touch Memcache. The reason is, I save thousands of objects, and I don't need them hot for retrieval.
Also, I keep getting weird exceptions, such as:
java.lang.reflect.InvocationTargetException
at com.google.appengine.runtime.Request.process-326a59f97d0f0252 (Request.java)
at sun.reflect.GeneratedMethodAccessor21.invoke (Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:42)
at com.googlecode.objectify.cache.MemcacheServiceRetryProxy.invoke (MemcacheServiceRetryProxy.java:68)
at com.sun.proxy.$Proxy9.putAll (Unknown Source)
at com.googlecode.objectify.cache.KeyMemcacheService.putAll (KeyMemcacheService.java:91)
at com.googlecode.objectify.cache.EntityMemcache.empty (EntityMemcache.java:323)
at com.googlecode.objectify.cache.CachingAsyncDatastoreService$5.trigger (CachingAsyncDatastoreService.java:445)
at com.googlecode.objectify.cache.TriggerFuture.isDone (TriggerFuture.java:87)
at com.googlecode.objectify.cache.TriggerFuture.get (TriggerFuture.java:102)
at com.googlecode.objectify.cache.PendingFutures.completeAllPendingFutures (PendingFutures.java:57)
at com.googlecode.objectify.ObjectifyService$2.close (ObjectifyService.java:120)
at com.googlecode.objectify.ObjectifyFilter.doFilter (ObjectifyFilter.java:49)
and:
com.google.appengine.api.memcache.MemcacheServiceException: Memcache putAll: Unknown exception setting 2 keys
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.handleApiProxyException (MemcacheServiceApiHelper.java:69)
at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$RpcResponseHandlerForPut.handleApiProxyException (AsyncMemcacheServiceImpl.java:349)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentException (MemcacheServiceApiHelper.java:111)
at com.google.appengine.api.utils.FutureWrapper.handleParentException (FutureWrapper.java:52)
at com.google.appengine.api.utils.FutureWrapper.get (FutureWrapper.java:91)
at com.google.appengine.api.utils.FutureWrapper.get (FutureWrapper.java:89)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet (MemcacheServiceImpl.java:26)
at com.google.appengine.api.memcache.MemcacheServiceImpl.putAll (MemcacheServiceImpl.java:115)
at com.googlecode.objectify.cache.KeyMemcacheService.putAll (KeyMemcacheService.java:91)
at com.googlecode.objectify.cache.EntityMemcache.getAll (EntityMemcache.java:242)
at com.googlecode.objectify.cache.CachingAsyncDatastoreService.get (CachingAsyncDatastoreService.java:252)
and:
com.google.appengine.api.memcache.MemcacheServiceException: Memcache getAll: exception getting multiple keys
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.handleApiProxyException (MemcacheServiceApiHelper.java:69)
at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentException (MemcacheServiceApiHelper.java:111)
at com.google.appengine.api.utils.FutureWrapper.handleParentException (FutureWrapper.java:52)
at com.google.appengine.api.utils.FutureWrapper.get (FutureWrapper.java:91)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet (MemcacheServiceImpl.java:26)
at com.google.appengine.api.memcache.MemcacheServiceImpl.getAll (MemcacheServiceImpl.java:64)
at com.googlecode.objectify.cache.KeyMemcacheService.getAll (KeyMemcacheService.java:83)
at com.googlecode.objectify.cache.EntityMemcache.cacheGetAll (EntityMemcache.java:365)
at com.googlecode.objectify.cache.EntityMemcache.putAll (EntityMemcache.java:296)
at com.googlecode.objectify.cache.CachingAsyncDatastoreService$3.success (CachingAsyncDatastoreService.java:279)
at com.googlecode.objectify.cache.CachingAsyncDatastoreService$3.success (CachingAsyncDatastoreService.java:268)
So, my question is, how do I save using Objectify without hitting Memcache?
And/or, how do I prevent those exceptions from happening?
You can use ofy().cache(false) but you have to be careful about this.
On save, Objectify clears cache entries. On load, Objectify checks memcache and if the value is not present, saves it in memcache for later. Generally you want to maintain this clearing behavior on save otherwise you risk leaving stale entries in the cache. When loading seldom-accessed entities in bulk, use ofy().cache(false).load()... and memcache will remain unpolluted.
Or just remove #Cache from the entity class.
I don't know what to say about your exception; it comes from the bowels of GAE and not Objectify.
You can disable the global cache for an operation:
ofy().cache(false).save()...
More details here

AppEngine RemoteAPI SocketTimeoutException

I'm using RemoteAPI to fetch entities from GAE Datastore, 300 at a time.
I'm doing something along the lines of:
while(!(emails = getEmails()).isEmpty()) {
Filter filter = new FilterPredicate("email", FilterOperator.IN, emails)
Query query = new Query("MyEntity").setFilter(filter);
QueryResultIterable<Entity> result = ds.prepare(query).asQueryResultIterable();
for (Entity entity : result) {
System.out.println(entity.getProperty("name"));
}
}
I'm processing something like 50k emails. The first time I ran this code it got to maybe 3/4 of the way, then it threw the following exception. Now it throws it after a single loop iteration is run.
com.google.appengine.tools.remoteapi.RemoteApiException: remote API call: I/O error
at com.google.appengine.tools.remoteapi.RemoteRpc.makeException(RemoteRpc.java:160)
at com.google.appengine.tools.remoteapi.RemoteRpc.callImpl(RemoteRpc.java:104)
at com.google.appengine.tools.remoteapi.RemoteRpc.call(RemoteRpc.java:50)
at com.google.appengine.tools.remoteapi.RemoteDatastore.runQuery(RemoteDatastore.java:156)
at com.google.appengine.tools.remoteapi.RemoteDatastore.handleRunQuery(RemoteDatastore.java:115)
at com.google.appengine.tools.remoteapi.RemoteDatastore.handleDatastoreCall(RemoteDatastore.java:93)
at com.google.appengine.tools.remoteapi.RemoteApiDelegate.makeDefaultSyncCall(RemoteApiDelegate.java:57)
at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate.makeSyncCall(StandaloneRemoteApiDelegate.java:47)
at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate$1.call(StandaloneRemoteApiDelegate.java:58)
at com.google.appengine.tools.remoteapi.StandaloneRemoteApiDelegate$1.call(StandaloneRemoteApiDelegate.java:54)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at com.google.appengine.repackaged.com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
at com.google.appengine.repackaged.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94)
at com.google.appengine.repackaged.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
at com.google.appengine.tools.remoteapi.OAuthClient.post(OAuthClient.java:54)
at com.google.appengine.tools.remoteapi.RemoteRpc.callImpl(RemoteRpc.java:102)
... 12 more
I can't figure out what the problem is, but the code seems to be evaluating the for() condition before throwing the exception.
Could this be a quota problem? The quota details screen doesn't show any problems and I can't find any relevant information in the documentation.
For future readers of this question, if you see occurrences of RemoteApiException: remote API call: I/O error which are happening consistently and not intermittently, this could be related to a disruption in network connectivity or possibly a remote issue on the App Engine side.
If the first possibility is ruled out, the best course of action is to report the issue on the Google App Engine issue tracker.
To fix this, first, check your Internet connection. Then clean all artifacts and build them again by (with IntelliJ):
Go to Build => Build Artifacts...
Focus on All Artifacts => Clean
Focus on All Artifacts => Build

Is there a list of applicationError codes & descriptions for the Google AppEngine ApplicationException?

Hello everyone and thanks up front for your time,
I am working on a java-based GAE web application and now and then I get ApiProxy.ApplicationExceptions.
In the current case they appear randomly and come with the applicationError 108 when I open a write channel to a blob using the (yes I know, still experimental) FileStore API. Although the API is still in an experimental state, I'd like to handle the thrown exception correctly. Thus my question:
Where can I find a list of possible application errors including their descriptions?
As of right now it is not possible for me to figure out where the problem resides since the thrown exception does not contain something like a message, hint or reason phrase but only the error ID 108:
Caused by: com.google.apphosting.api.ApiProxy$ApplicationException: ApplicationError: 108:
at java.lang.Thread.getStackTrace(Thread.java:1495)
at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:240)
at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:66)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:183)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:180)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:180)
at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:66)
at com.googlecode.objectify.cache.TriggerFutureHook.makeSyncCall(TriggerFutureHook.java:154)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:107)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:56)
at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:584)
... 65 more
Also, the corresponding javadoc is quite conservative with giving information: https://developers.google.com/appengine/docs/java/javadoc/com/google/apphosting/api/ApiProxy.ApplicationException
Currently I bluntly cancel these requests with a 500, but since I am not sure what has happened I should probably do something else/more.
Thanksalot!
the best information I could get is from the Python source code :
http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/api/files/file_service_pb.py

java.lang.NullPointerException at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)

Every now and then I come across this error. I usually end up cleaning all the database. what could be the cause?
This is on Win XP SP3, GAE 1.6.5, Objectify 3.0, GWT 2.4.
QueryResultIterator<Key<x>> it = query.fetchKeys().iterator();
ArrayList<Key<x>> keys = new ArrayList<Key<x>> (num);
while (it != null && it.hasNext()) { <== error
keys.add(it.next());
}
Here is the error -
Caused by: java.lang.NullPointerException
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72)
at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
at com.google.appengine.api.datastore.QueryResultsSourceImpl.peekQueryResultAndIfFirstRecordIndexList(QueryResultsSourceImpl.java:175)
at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:111)
at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:98)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:164)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:66)
at com.googlecode.objectify.util.TranslatingIterator.hasNext(TranslatingIterator.java:29)
It is a bug in GAEJ. I got this all the time, just wait for a day and it will work. Do not send any hit to the server. I know it is kind of bad. Even worse, I am a paying customer.
I hit the "Report production issue" every time it occurs and pray that someone will look into it.

Resources