If I start my local google app engine dev server by using the App Engine maven plugin :
mvn appengine:devserver
and I try to launch a mapreduce job (using the appengine-mapreduce library), I get a java.security.AccessControlException: access denied ("javax.security.auth.AuthPermission" "getSubject") because the code try to invoke javax.security.auth.Subject.getSubject() method (see below for the full stack trace)
It's very weird because :
1) the javax.security.auth.Subject.getSubject() is in the white list of gae and can be invoked
2) If I start the server from intellij, I don't get this exception and the map reduce job is launched successfully
I tried with GAE 1.8.1, 1.8.2 and the latest 1.8.3 and respectively the same version for the maven google appengine plugin.
What is going wrong ?
The full stack trace :
[INFO] java.security.AccessControlException: access denied ("javax.security.auth.AuthPermission" "getSubject")
[INFO] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:366)
[INFO] at java.security.AccessController.checkPermission(AccessController.java:560)
[INFO] at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[INFO] at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:383)
[INFO] at javax.security.auth.Subject.getSubject(Subject.java:287)
[INFO] at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:467)
[INFO] at org.apache.hadoop.mapreduce.JobContext.(JobContext.java:80)
[INFO] at com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java:123)
[INFO] at com.google.appengine.tools.mapreduce.AppEngineJobContext.createContextForNewJob(AppEngineJobContext.java:132)
[INFO] at com.google.appengine.tools.mapreduce.v2.impl.handlers.Controller.handleStart(Controller.java:85)
[INFO] at com.google.appengine.tools.mapreduce.v2.impl.handlers.Status.handleStartJob(Status.java:81)
[INFO] at com.google.appengine.tools.mapreduce.v2.impl.handlers.Status.handleCommand(Status.java:182)
[INFO] at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost(MapReduceServlet.java:203)
[INFO] at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
[INFO] at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[INFO] at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
Related
During first-deploy on google app engine (flexible) I get the following error:
[INFO] GCLOUD: If this is your first deployment, this may take a while...
[INFO] GCLOUD:
[INFO] GCLOUD: If this is your first deployment, this may take a while...done.
[INFO] GCLOUD:
[INFO] GCLOUD: Beginning deployment of service [default]...
[INFO] GCLOUD: WARNING: Deployment of App Engine Flexible Environment apps is currently in Beta
[INFO] GCLOUD: ERROR: (gcloud.app.deploy) Unable to determine the root directory where GAE runtimes are stored. Please define the CLOUDSDK_APP_RUNTIME_ROOT environment variable.
[INFO] ------------------------------------------------------------------------
I haven't found anything related to CLOUDSDK_APP_RUNTIME_ROOT,
Is there any documentation that describes this parameter?
I'm using the following (standard) runtime:
runtime: java
vm: true
runtime_config: # Optional
jdk: openjdk8
The command I'm using is:
mvn appengine:deploy
I'm trying to deploy and run Google-pubsub sample code in Google app engine by this read me file. While running the deploy command I encounter below build error.
E:\java\cloud-pubsub-samples-java-master>mvn gcloud:deploy -Dgcloud.version=1 -DskipTests=true
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] pubsub-pull-sample
[INFO] Example for the Google Cloud Pub/Sub on Google App Engine.
[INFO] Cloud Dataflow Examples
[INFO] cloud-pubsub-samples-java
Downloading: https://repo.maven.apache.org/maven2/org/eclipse/m2e/lifecycle-mapping/1.0.0/lifecycle-mapping-1.0.0.pom
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available
Downloading: https://repo.maven.apache.org/maven2/org/eclipse/m2e/lifecycle-mapping/1.0.0/lifecycle-mapping-1.0.0.jar
[WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: Could not find artifact org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in central (https://repo.maven.apache.org/maven2)
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml (13 KB at 9.8 KB/sec)
Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml (20 KB at 8.9 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] pubsub-pull-sample ................................. SKIPPED
[INFO] Example for the Google Cloud Pub/Sub on Google App Engine. SKIPPED
[INFO] Cloud Dataflow Examples ............................ SKIPPED
[INFO] cloud-pubsub-samples-java .......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.167 s
[INFO] Finished at: 2016-08-21T22:20:45+05:30
[INFO] Final Memory: 13M/245M
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'gcloud' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (C:\Users\JAISON.G\.m2\repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException
This means that you haven't included the gcloud maven plugin in pom.xml. But you should no longer use gcloud, rather using appengine. See the documentation here.
I use jetty9-compat + java8 configuration and receive the following exception. Probably because of this I am not able to store objects in session, so in the end I cannot log in to my service.
[INFO] cze 28, 2016 11:17:56 AM com.google.apphosting.vmruntime.VmMetadataCache getMetadata
[INFO]
[INFO] INFO: Meta-data 'attributes/gae_affinity' path retrieval error: metadata
[INFO]
[INFO] cze 28, 2016 11:17:56 AM com.google.apphosting.vmruntime.VmApiProxyDelegate runSyncCall
[INFO]
[INFO] INFO: HTTP ApiProxy I/O error for memcache.Get: The target server failed to respond
[INFO]
[INFO] cze 28, 2016 11:17:56 AM com.google.appengine.api.memcache.LogAndContinueErrorHandler handleServiceError
[INFO]
[INFO] INFO: Service error in memcache
[INFO]
[INFO] com.google.appengine.api.memcache.MemcacheServiceException: RCP Failure for API call: memcache Get
[INFO]
[INFO] at com.google.apphosting.vmruntime.VmApiProxyDelegate.constructApiException(VmApiProxyDelegate.java:232)
[INFO]
[INFO] at com.google.apphosting.vmruntime.VmApiProxyDelegate.runSyncCall(VmApiProxyDelegate.java:195)
[INFO]
[INFO] at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeApiCall(VmApiProxyDelegate.java:154)
[INFO]
[INFO] at com.google.apphosting.vmruntime.VmApiProxyDelegate.access$000(VmApiProxyDelegate.java:60)
[INFO]
[INFO] at com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:436)
[INFO]
[INFO] at com.google.apphosting.vmruntime.VmApiProxyDelegate$MakeSyncCall.call(VmApiProxyDelegate.java:412)
[INFO]
[INFO] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[INFO]
[INFO] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[INFO]
[INFO] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[INFO]
[INFO] at java.lang.Thread.run(Thread.java:745)
[INFO]
[INFO]
[INFO]
[INFO] DEBUG 2016-06-28 11:17:56,493 api_server.py:277] Handled datastore_v3.Get in 0.0000
[INFO] cze 28, 2016 11:17:59 AM com.mysql.jdbc.log.Slf4JLogger logInfo
This issue is reproduced only locally, I face it too. I assume that it is caused by incorrect configuration of local memcached which is run by dev server.
I'll try to explain my point. In my case first request is always successful.
Because in case of first request Apache HttpClient opens a new connection and puts it into the pool.
There is source code https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/blob/master/appengine-managed-runtime/src/main/java/com/google/apphosting/vmruntime/VmApiProxyDelegate.java
All further requests are successful too, if they are executed with interval up to 10 seconds. But if there were no request within 10 seconds, next request fails.
It happens because memcached service closes connection from its side, but HttpClient doesn't know about it.
In VmApiProxyDelegate HttpClient is configured to close idle connections after 60 seconds. So, if I wait more than 1 minute instead of 10 seconds, my next request doesn't fail. Because in this case HttpClient opens new connection which is not closed yet.
To avoid this issue it would be correct to configure connection timeout from memcached stub side. But the documentation says that your requests to memcached can fail and you must handle these errors. So, I offer you to add error handler and some retry mechanism.
Of course all of the above fits your case, if everything is ok in your configuration. Taking into account that you did not provide examples of configuration and code, I assume that there everything is ok.
I had a java GAE application running using gcloud-java 0.1.4 for some time and wanted to upgrade to gcloud-java 0.1.7/0.2.0, as we are experiences several 404s etc. (another problem). We are using gcloud as we connect to a datastore instance in another GAE application.
But after upgrading the package, our authentication fails. From what I read in release notes, we had to re-enable the datastore-api on the "host" application, so we did but without any results. When "downgrading" again, the application works as intended but still with some stability issues.
Are there other undocumented fixes/methods to get it to work?
My connection code/function looks like (stripped):
fis = new FileInputStream(Constants.PATH_TO_JSON_KEY);
sac = AuthCredentials.createForJson(fis);
DatastoreOptions.Builder builder = DatastoreOptions.builder();
builder.projectId(Constants.DATASTORE_PROJECT_ID);
builder.authCredentials(sac);
DatastoreOptions options = builder.build();
return options.service();
EDIT:
Here are the some of the stack traces that I get.
[INFO] apr 17, 2016 6:19:33 PM com.google.datastore.v1beta3.client.DatastoreFactory makeClient
[INFO] com.google.cloud.datastore.DatastoreException: Not authorized.
[INFO] at com.google.cloud.datastore.spi.DefaultDatastoreRpc.translate(DefaultDatastoreRpc.java:102)
[INFO] at com.google.cloud.datastore.spi.DefaultDatastoreRpc.runQuery(DefaultDatastoreRpc.java:157)
[INFO] at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:92)
[INFO] at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:89)
[INFO] at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:181)
[INFO] at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:247)
[INFO] at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:237)
[INFO] at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:88)
[INFO] at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:73)
[INFO] at com.google.cloud.datastore.QueryResultsImpl.<init>(QueryResultsImpl.java:57)
[INFO] at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:82)
[INFO] at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:73)
.....
[INFO] Caused by: com.google.datastore.v1beta3.client.DatastoreException: Not authorized., code=PERMISSION_DENIED
[INFO] at com.google.datastore.v1beta3.client.RemoteRpc.makeException(RemoteRpc.java:126)
[INFO] at com.google.datastore.v1beta3.client.RemoteRpc.makeException(RemoteRpc.java:169)
[INFO] at com.google.datastore.v1beta3.client.RemoteRpc.call(RemoteRpc.java:89)
[INFO] at com.google.datastore.v1beta3.client.Datastore.runQuery(Datastore.java:108)
[INFO] at com.google.cloud.datastore.spi.DefaultDatastoreRpc.runQuery(DefaultDatastoreRpc.java:155)
[INFO] ... 56 more
Thanks for reporting this issue. I have a couple of follow up questions so that we can get to the bottom of this:
Does the project ID you set (Constants.DATASTORE_PROJECT_ID) contain any prefixes with a tilde, perhaps "s~" or "e~"? Datastore v1beta3 no longer accepts those prefixes; instead you should provide everything after the tilde as your project ID. For example, if your project ID is s~my-project-id-123, you should use just my-project-id-123. This might be the issue if you see an error message similar to:
com.google.datastore.v1beta3.client.DatastoreFactory makeClient: Not using any credentials
com.google.cloud.datastore.DatastoreException: The project s~my-project-id-123 does not exist.
If that isn't the issue, do you mind providing a stack trace and a copy of the exception text (removing confidential information as necessary). Thanks!
The service account we were using wasnt set setup with the appropriate permissions in the destination application. So we had (from advice from google) re-add the service account as a viewer here: https://console.cloud.google.com/permissions/projectpermissions
I hope this can help others in the future!
I am getting the below error in the line 5 while executing the app engine code in localhost and in cloud.
1 HttpClient client = new HttpClient();
2 GetMethod getMethod =new GetMethod(url);
3 client.getHttpConnectionManager().getParams()
4 .setConnectionTimeout(1000);
5 int response = client.executeMethod(getMethod);
I am seeing this error while running the job in localhost and in app engine.
Please find the error logs below:
INFO] 2016-03-07 11:04:03 DEBUG HttpConnection:1215 - enter HttpConnection.closeSockedAndStreams()
[INFO] 2016-03-07 11:04:03 INFO HttpMethodDirector:439 - I/O exception (java.net.SocketException) caught when processing request: Permission denied: Not allowe
d to issue a socket bind: permission denied.
[INFO] 2016-03-07 11:04:03 DEBUG HttpMethodDirector:443 - Permission denied: Not allowed to issue a socket bind: permission denied.
[INFO] java.net.SocketException: Permission denied: Not allowed to issue a socket bind: permission denied.
[INFO] at com.google.appengine.api.socket.SocketApiHelper.translateError(SocketApiHelper.java:95)
[INFO] at com.google.appengine.api.socket.SocketApiHelper.translateError(SocketApiHelper.java:106)
[INFO] at com.google.appengine.api.socket.SocketApiHelper.makeSyncCall(SocketApiHelper.java:74)
[INFO] at com.google.appengine.api.socket.AppEngineSocketImpl.createSocket(AppEngineSocketImpl.java:470)
[INFO] at com.google.appengine.api.socket.AppEngineSocketImpl.bind(AppEngineSocketImpl.java:486)
[INFO] at java.net.Socket.bind(Socket.java:631)
[INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[INFO] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.lang.reflect.Method.invoke(Method.java:606)
[INFO] at com.google.appengine.tools.development.agent.runtime.Runtime.invoke(Runtime.java:130)
[INFO] at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
[INFO] at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
[INFO] at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
[INFO] at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
[INFO] at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
[INFO] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
[INFO] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
[INFO] at com.disney.unityads.UnityAdsStats.downloadZip(UnityAdsStats.java:169)
[INFO] at com.disney.unityads.UnityAdsStats.run(UnityAdsStats.java:72)
[INFO] at com.disney.unityads.UnityAdsChannelJobWorker.doPost(UnityAdsChannelJobWorker.java:46)
[INFO] at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
[INFO] at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[INFO] at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
[INFO] at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
Check the documentation for how to make HTTP requests. AppEngine is a very limited environment, for example if you try to use an alternative HTTP client your app gets the same treatment as one that tries to use sockets directly.
https://cloud.google.com/appengine/docs/java/urlfetch/