We are currently migrating our App Engine from Java 8 to Java 11 using the bundled services (https://cloud.google.com/blog/products/serverless/support-for-app-engine-services-in-second-generation-runtimes).
We made it as far as that the application seems to be running as expected - BE, FE and Datastore all seem to work in our staging GCP project.
The following errors are showing up in our logs every 10 seconds, however:
Failed to query GCE metadata service
and then
java.io.IOException: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:70)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:609)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:488)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.readResponse(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.queryMetadataAttribute(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.getProjectId(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GcpHubClient.registerDebuggee(Unknown Source)
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:844)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:117)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:54)
... 6 more
Edit:
The Cloud Debugger says "The debugger could not find a debug target for the application". The versions still relying on Java 8 work though, so what extra step is missing for the migration to set up the debugger correctly? As we're using the App Engine standard environment, that should be enabled by default (https://cloud.google.com/debugger/docs/setup/java#gae-standard).
Edit 2:
Looks like this could be a Google Cloud Debugger bug:
https://github.com/GoogleCloudPlatform/cloud-debug-java/issues/18
As per the App Engine documentation regarding issuing HTTP requests, if you use URL Fetch, it will cause requests to cloud client libraries (and that includes the Google Cloud Debugger Client for Java) to fail.
As wer are using the bundled services, we are still using an appengine-web.xml file.
We had the following line in there:
<url-stream-handler>urlfetch</url-stream-handler>
Removing that line fixed the issue for us.
Related
I have a Google Appengine app (Java) and the datastore was recently auto-migrated into "Cloud Firestore in Datastore mode" (completed on 6/6/2022) that has been running for years.
It has now started throwing these exceptions in new deployments without any relevant code change to trigger them:
com.google.appengine.api.datastore.DatastoreFailureException: Missing or invalid authentication.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:69)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:127)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:68)
at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
at com.google.appengine.api.datastore.BaseQueryResultsSource.getIndexList(BaseQueryResultsSource.java:154)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:187)
at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:166)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:146)
at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:64)
at com.googlecode.objectify.impl.KeysOnlyIterator.hasNext(KeysOnlyIterator.java:29)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:48)
at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:20)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:52)
at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1105)
at com.googlecode.objectify.impl.ChunkingIterator.hasNext(ChunkingIterator.java:51)
at com.google.common.collect.Iterators.addAll(Iterators.java:366)
at com.google.common.collect.Lists.newArrayList(Lists.java:163)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:34)
at com.sun.proxy.$Proxy68.isEmpty(Unknown Source)
My code. After loading a list of entities, the exception is thrown on the isEmpty() call on the returned list:
List<SystemSetting> applicationSettings = ofy().load().type(SystemSetting.class).filter("name", name).list();
if (applicationSettings.isEmpty()) { // Exception thrown here
It's worth also noting that on previous versions, datastore reads and writes are still happening without any issues. It is only new deployments that exhibit this issue. However, redeploying previous versions from previous commits using the same CI service to deploy them are failing.
Is this caused by the datastore migration?
Slight update: these errors are only happening on newly deployed versions... In fact, I deployed a new version (from CI, bitbucket pipeline) that worked, but when I redeployed it with no changes it no longer works. Same app engine version id, code and users etc.
Things we've tried without success:
Re-deploying last known version
Clearing memcache
Deploying from local dev
Stopping instances
Trying alternative service account key
Expanding service account permissions
Clearing BitBucket pipeline caches (the CI builder)
Have recreated the issue with datastore api directly (no objectify)
Have redeployed with different user account - no success
Have redeployed with new service account - no success
Update 1:
If I run the following code on a local java application (pointing at live datastore using same service account) it works locally, but fails if I deploy to app engine with my project-id.
DatastoreOptions datastoreOptions = DatastoreOptions.getDefaultInstance();
Datastore datastore = datastoreOptions.getService();
// A known entity in my live datastore
Key key = datastore.newKeyFactory()
.setKind("MyEntityType")
.newKey(123123123l);
Entity retrieved = datastore.get(key);
When it fails, I can get slightly more information:
java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
full stack:
com.google.cloud.datastore.DatastoreException: I/O error
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:434)
at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:431)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:430)
...
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:171)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:117)
at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:93)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
... 74 more
Caused by: java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token
at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:206)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
... 76 more
Update 2:
I've since paid for Google support and they have now confirmed it "is linked to a known issue related to the GAE standard Service Agent and access token which is affecting some users"
I am developing an application which uses MWS (Amazon market web services). What I have done import java libraries given by Amazon Configuration application with respect seller or Merchant credential. But when I run application the following error occur at run time. I put my best to get rid off but never find any helpful answer.
AndroidRuntime:
FATAL EXCEPTION: main Process: com.example.aun.sample java.lang.NoSuchMethodError: No direct method (Ljava/lang/String;ILorg/apache/http/conn/scheme/SchemeSocketFactory;)V in class Lorg/apache/http/conn/scheme/Scheme; or its super classes (declaration of 'org.apache.http.conn.scheme.Scheme' appears in /system/framework/ext.jar) org.apache.http.impl.conn.SchemeRegistryFactory.createDefault(SchemeRegistryFactory.java:47) org.apache.http.impl.conn.PoolingClientConnectionManager.(PoolingClientConnectionManager.java:93) com.example.aun.sample.MwsConnection1.getConnectionManager(MwsConnection1.java:260) com.example.aun.sample.MwsConnection1.freeze(MwsConnection1.java:227) com.example.aun.sample.MwsConnection1.newCall(MwsConnection1.java com.example.aun.sample.MwsConnection1.call(MwsConnection1.java:398) com.example.aun.sample.MarketplaceWebServiceProductsClient1.getServiceStatus(MarketplaceWebServiceProductsClient1.java:168) com.example.aun.sample.GetServiceStatus.invokeGetServiceStatus(GetServiceStatus.java:22) com.example.aun.sample.MainActivity.onCreate(MainActivity.java:44) android.app.Activity.performCreate(Activity.java:6092) android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1112) android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2468) android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2595) android.app.ActivityThread.access$800(ActivityThread.java:178) android.app.ActivityThread$H.handleMessage(ActivityThread.java:14 android.os.Handler.dispatchMessage(Handler.java:111) android.os.Looper.loop(Looper.java:194) android.app.ActivityThread.main(ActivityThread.java:5631) java.lang.reflect.Method.invoke(Native Method) java.lang.reflect.Method.invoke(Method.java:372)
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteIni) com.android.internal.os.ZygoteInit.main(ZygoteInit)
Your log is missing the name of the missing method. I'm guessing it is "init", but anyways, it seems your build process is broken.
See this almost identical error: NoSuchMethodError: org.apache.http.conn.scheme.Scheme.<init>(Ljava/lang/String;ILorg/apache/http/conn/scheme/SchemeSocketFactory;)V
I am running into an issue similar to this one. I have a GAE app and a GCE app that seem to work fine once in the cloud but I am having trouble getting my local environment working so that both of them access the same datastore.
I have setup the local datastore as described in the link above except my code looks like this (I had to build it this way in order to get it working in the cloud):
print("Connecting to datastore - " + datasetId);
// Get credentials form GCE if not local.
Credential computeEngineCredential = DatastoreHelper.getComputeEngineCredential();
if(computeEngineCredential != null)
{
print("Compute Engine Credetianls are not null! Access token: " + computeEngineCredential.getAccessToken());
}
DatastoreOptions options = DatastoreHelper.getOptionsfromEnv().credential(DatastoreHelper.getComputeEngineCredential()).dataset(datasetId).build();
print("Connection Host: " + options.getHost());
print("Connection Dataset: " + options.getDataset());
datastore = DatastoreFactory.get().create(options);
When I run the GCE app locally and try to connect to the running GAE datastore I get the following (I have replaced the actual data set id with "myDatasetId" in the output below):
Connecting to datastore - "myDatasetId"
Connection Host: http://localhost:8888
Connection Dataset: "myDatasetId"
com.google.api.services.datastore.client.DatastoreFactory makeClient
WARNING: Not using any credentials
There was a problem running query: Error:
runQuery Error 404 Error
404
com.google.api.services.datastore.client.DatastoreException: Error 404 Error
404
at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:114)
at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:80)
at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109)
My GAE console prints this out (I can access the admin console in the 8888 port just fine):
com.google.appengine.tools.development.AbstractModule startup
INFO: The admin console is running at http://localhost:8888/_ah/admin
com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for:
/datastore/v1beta2/datasets/"myDatasetId"/runQuery
I have verified the dataset ID in the GCE app and the GAE app match. I have been able to successfully run each app localy on their own and they both are able to properly connect to the local Datastore (this is while using the gcd.cmd tool for the GCE app). Based on the answer in the link above, it sounds like this is possible, am I doing something wrong?
Update
Not sure if this is related but I am getting the following error when starting up the GCD Tool:
SEVERE: Unable to load the App Engine dev agent. Security restrictions
will not be completely emulated. java.lang.RuntimeException:
Unexpected exception during cast.
at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:385)
at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:252)
at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:263)
at com.google.appengine.tools.development.agent.AppEngineDevAgent.premain(AppEngineDevAgent.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown
Source)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown
Source) Caused by: java.lang.IllegalAccessException: Class
com.google.apphosting.utils.clearcast.ClearCast$CasterImpl can not ac
cess a member of class
com.google.appengine.tools.development.agent.$Proxy0 with modifiers
"public"
at sun.reflect.Reflection.ensureMemberAccess(Unknown Source)
at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(Unknown
Source)
at java.lang.reflect.AccessibleObject.checkAccess(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:383)
... 9 more
If this initialized properly, could I somehow connect my GAE App to the GCD Tool datastore? So confused.
There's no officially supported way to share Datastore data between the Java Development Server (dev_appserver.sh) and the local Cloud Datastore tool (gcd.sh).
However, if your app is written in Java, you may be able to use the workaround described here and point dev_appserver.sh to the data file generated by gcd.sh by specifying the -Ddatastore.backing_store=<project dir>/WEB-INF/appengine-generated/local_db.bin option.
Thanks for replying this was very useful. I was able to point my local GAE to the local_db.bin that the gcd tool uses through eclipse by providing the "-Ddatastore.backing_store" as a VM argument as you suggested.
However, they still seem to have a different views of the datastore. The admin viewer for the GAE app running on the default 8888 port only shows the data added by the GAE app, on the other hand, the gcd one running on the gcd tool port (8080) shows the data added by the GCE app.
I assumed this was just a visibility issue on the admin site so I tried to access the GAE data through my GCE app but the query is unsuccessful, it doesn't seem to find the Entity kind and thus returns no results. I was able to verify that once deployed GCE is able to access the data the GAE app wrote to the datastore with the same query, due to this I am assuming it is not a namespace issue but more of an issue on where the data is held. Even though they are both pointing to the same local_db.bin file it seems the data is still split somewhere.
Am I supposed to run the dev_appserver.cmd directly form the command line maybe? If so, How do I do this for an EAR project (currently on eclipse).
I created one google app engine project. There I configured DWR(DirectWebRemoting).
I have created one ajax functionality which will checks the username and password of the user. But it does give me any output rather it gives Error:
dwr is not defined
Source File: http://localhost:8081/dwr.jsp
Line: 16
Where as it works fine in Tomcat web application.
So My question is that does Google app engine not support DWR configuration?
DWR is compatible with GAE according to "Will it play in Java" (broken link. possibly: a list of GAE-compatible java libraries) page.
Several links to threads with proof of success and potential limitations:
thread about "DWR 3 RC1" from "users#dwr.java.net"
StackOverflow question "DWR sometimes die on the GAE server"
Most useful post in "DWR 3 RC1" thread is this one (about potential problems).
Do you require reverse ajax? (Current implementation uses a thread to clean up expired sessions. Current implementation uses javax.swing.event.EventListenerList which is blacklisted by GAE/J.)
Do you require file upload? (Can potentially write to disk)
Do you require file download? (Current implementation uses a thread to clean up expired downloads)
If you don't need anything from the above, then you could replace problematic classes with dummy implementation.
I don't think this is supported, but the same functionality is offered by channels :
https://developers.google.com/appengine/docs/java/channel/
I'm trying to make http requests from my Google App Engine webapp, and discovered I have to use URLConnection since it's the only whitelisted class. The corresponding Clojure library is clojure.contrib.http.agent, and my code is as follows:
(defroutes example
(GET "/" [] (http/string (http/http-agent "http://www.example.com")))
(route/not-found "Page not found"))
This works fine in my development environment- the browser displays the text for example.com. But when I test it out with Google's development app server:
phrygian:example wei$ dev_appserver.sh war
2010-09-28 14:53:36.120 java[43845:903] [Java CocoaComponent compatibility mode]: Enabled
...
INFO: The server is running at http://localhost:8080/
It just hangs when I load the page. No error, or anything. Any idea what might be going on?
http-agent creates threads so that might be why it does not work.
From the API documentation:
Creates (and immediately returns) an Agent representing an HTTP
request running in a new thread.
You could try http-connection, which is a wrapper around HttpURLConnection, so this should work.
Another alternative is to try clj-http. The API seems to be a bit more high-level, but it uses Apache HttpComponents which might be blacklisted.
I am guessing http.async.client is a definite no-go due to its strong asynchronous approach.
You might want to try appengine.urlfetch/fetch from appengine-clj (http://github.com/r0man/appengine-clj, also in clojars)