I just stumbled on the following issue in App Engine Standard with a Python 2.7 enviroment
So I deployed to my test environment yesterday and today I had the idea of updating one of my applications. I do my normal "gcloud deploy ... " and it says updating 3 files ... While I actually changed a bunch of files. Basically my deploy command says the files are not changed.
After some searching around I found that files are being uploaded to a staging area and checked with a hash. Is it safe to actually clear this staging area, or does the gcloud command have some secret force option to actually force the files to be renewed.
The gcloud command has not given any errors what so ever, nor was it aborted at some point of deployment or something. So I have no errors, but my files aren't uploaded at all. I also tried modifying alot of files, and nothing changed
I never use the promote option for these rare cases that a deploy might fail
So anyone encountered this before, or has a solution to this issue ?
I also was encountering this and the only solution I could find was to deploy to a new bucket. To do this:
go to https://console.cloud.google.com/storage/browser and create a new bucket
redeploy using gcloud app deploy --bucket gs://your-new-bucket. (Change your-new-bucket to the actual bucket name)
This uploaded all the files again and created a new version in App Engine.
You can Go to https://console.cloud.google.com/storage/browser and delete your application bucket, on the next deploy it will be recreated. Additionally you can use the parameter --verbosity=info to check which files are being uploaded.
Related
I'm using GCP App Engine with auto scaling.
If I deploy a new version of the app code (Python 3 Flask app) with a simple change for control and test purposes, lets say I add a comment to one of the .js files I am not seeing that change to the file in the browser after it has been deployed.
I have 100% of traffic being served by the new version of the app. When I look at the source code for the version I can see the comment in there, but when I clear my browser cache and visit the page I only ever get the old version of the page (without the comment in the .js).
I have tried using the --promote and --no-cache values in the app deploy command, but no use. I have added:
default_expiration: "0d 0h 0m 0s"
To the app.yaml
I have also turned on object versioning of the storage account which app engine uses to try to ensure that only a single version of the file is available to be served - still no use.
The command I'm using to deploy is:
run: "gcloud app deploy app.yaml --quiet --promote --no-cache"
I can't understand why it should be so difficult to simply deploy a new version of the app and have the app engine serve the latest files; I must be doing something wrong but cannot see what.
Would appreciate any pointers.
The files are cached (even if for a short while and sometimes it takes time to clear it).
The trick is to make the urls (for the static files) unique for each deployment. This way, the browser is loading a 'distinct' url after each deployment. For example, you could append the environment variable, CURRENT_VERSION_ID to the url for all static elements. This means having something like (assuming Python/Jinja2)
src="/static/js/my_js_file.js?{{CURRENT_VERSION_ID}}"
os.environ['CURRENT_VERSION_ID'] changes for each deployment. There's a possibility this attribute is not available in newer runtimes. If so, just dump the environment variables and look for an attribute that is always present but the value changes (e.g. GAE_INSTANCE).
You could also just generate a random number each time your App is deployed and use that instead i.e.
src="/static/js/my_js_file.js?{{RANDOM_NUMBER}}"
Recently tried to update my Gaelyk project (yes, it's old, but it works well and I still use it), but Google App Engine will no longer accept the update. The error message returned is "Deployments using appcfg are no longer supported. See https://cloud.google.com/appengine/docs/deprecations". The thing is, I never used appcfg to deploy my application; I used Gaelyk and Gradle. But obviously Gaelyk must have used appcfg under the covers.
I did download the replacement Google Cloud SDK, but this new tool is not similar at all to how Gaelyk and Gradle worked. Is there anything I can do to get Gaelyk to work anymore? Or is Gaelyk just dead and I need to rewrite my application (like in Node.js or something instead of Groovy).
This will be hard, however I will try to help you as possible. I think you may try to migrate it somehow to app.yaml configuration of GAE.
I am not sure what plugins are used in the project. From Gaelyk temple project I can see that it's using appengine-geb which, according to the documentation, behind the scenes, is using gradle-appengine-plugin (there is wrong link on this doc, but proper is bellow).
On the github of gradle-appengine-plugin I have found following.
There is a note:
NOTE: All App Engine users are encouraged to transition to the new
gradle plugin for their projects.
And in FAQ part there is following information:
How do I deploy with gcloud?
If you're using gcloud to deploy your application, the newest version of app deploy > doesn't support war
directories, you will need to provide it with an app.yaml OR you can
use the appengineStage task to create a directory that is deployable
in /build/staged-app
$ ./gradlew appengineStage
$ gcloud app deploy build/staged-app/app.yaml --project [app id]
--version [some version]
NOTES:
You must explicitly define all config files your want to upload
(cron.yaml, etc)
This does not work with EAR formatted projects.
I think the best option will be to migrate to new appenine plugin or if not possible try to implement is with gcloud app deploy command crating the config files manually (at least app.yaml). And for this migration I can provide you this document.
I hope you will manage somehow...
I can confirm that Serge's answer on the Gaelyk Groups site works; the same procedure that he figured out also worked for me. To summarize:
Run gradlew appengineRun as run previously with Gaelyk.
Copy all jar files inside the build\exploded-app\WEB-INF\lib folder into a \src\main\webapp\web-inf\lib folder (for me the new lib folder did not exist previously).
To deploy, use the new required gcloud tool, and instead of running gradlew appengineUpdate (which fails now), instead run
gcloud app deploy appengine-web.xml where that XML file can be found in your webapp/WEB-INF directory. I navigated to that directory to run the gcloud command, but you can use a relative path there if your working directory is elsewhere. (There are a number of optional flags associated with the gcloud app deploy command, but I didn't need any of them.)
Serge needed to use these instructions to convert datastore-indexes.xml to index.yaml and run gcloud app deploy index.yaml, however, I didn't need to do this because I had no datastores.
Have a Meteor application that I’m deploying to a custom flex environment. Deploying same built folder to multiple Google Projects. Usually a .tgz file is created in my local temp folder and then uploaded to the projects default Google bucket and extracted from there to create an App Engine version.
Usually isn’t working for me in one of the projects and instead the gcloud app deploy command is uploading 57K individual files from node_modules. This makes a process go from minutes to multiple hours (ran it over night and still not done).‘
I’ve tried reinitializing the gcloud configuration, updating gcloud components, changing the default bucket, but not working. It’s doing some sort of check to see what’s been uploaded because it’ll skip uploaded files if I kill and start again.
An option for you to achieve that, it's to use the .gcloudignore to indicate which files you want to be uploaded, deployed, etc., or not.
As per the official documentation gcloud topic gcloudignore:
Several commands in gcloud involve uploading the contents of a directory to Google Cloud Platform to host or build. In many cases, you will not want to upload certain files (i.e., "ignore" them).
If there is a file called .gcloudignore in the top-level directory to upload, the files that it specifies will be ignored.
This means that you can use this file to decide which files you want to ignore. This way, you can, for example, ignore a whole directory or more than one, so this thousand of files of your don't get uploaded.
In this below post, there are examples of how to set this file for your own usage.
How to ignore files when running gcloud app deploy?
Let me know if the information helped you!
While I was deploying my google app engine project (Java) using gradle appengineDeploy my internet connection was interrupted.
I re-deployed the project. Although the console said BUILD SUCCESSFUL, the app engine instance no longer works. No matter how many times I re-deploy or update my application, the logs show nothing but errors:
java.lang.NoClassDefFoundError: com/google/appengine/api/ThreadManager
at
com.google.api.control.extensions.appengine.GoogleAppEngineControlFilter.createClient
(GoogleAppEngineControlFilter.java:61) at
com.google.api.control.ControlFilter.init (ControlFilter.java:141) at
org.eclipse.jetty.servlet.FilterHolder.initialize
(FilterHolder.java:139) at
org.eclipse.jetty.servlet.ServletHandler.initialize
(ServletHandler.java:873) at
org.eclipse.jetty.servlet.ServletContextHandler.startContext
(ServletContextHandler.java:349) at
org.eclipse.jetty.webapp.WebAppContext.startWebapp
(WebAppContext.java:1406) at
com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.startWebapp
(AppEngineWebAppContext.java:175) at
org.eclipse.jetty.webapp.WebAppContext.startContext
(WebAppContext.java:1368) at
org.eclipse.jetty.server.handler.ContextHandler.doStart
(ContextHandler.java:778) at
org.eclipse.jetty.servlet.ServletContextHandler.doStart
(ServletContextHandler.java:262) at
org.eclipse.jetty.webapp.WebAppContext.doStart
(WebAppContext.java:522) at
com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doStart
(AppEngineWebAppContext.java:120) at
org.eclipse.jetty.util.component.AbstractLifeCycle.start
(AbstractLifeCycle.java:68) at
com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.createHandler
(AppVersionHandlerMap.java:240) at
com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.getHandler
(AppVersionHandlerMap.java:178) at
com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest
(JettyServletEngineAdapter.java:120) at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest
(JavaRuntime.java:747) at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest
(JavaRuntime.java:710) at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run
(JavaRuntime.java:680) at
com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run
(JavaRuntime.java:872) at
com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run
(ThreadGroupPool.java:270) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
com.google.appengine.api.ThreadManager at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
com.google.apphosting.runtime.ApplicationClassLoader.findClass
(ApplicationClassLoader.java:135) at java.lang.ClassLoader.loadClass
(ClassLoader.java:424) at java.lang.ClassLoader.loadClass
(ClassLoader.java:357)
Is there some way for me to clear out the partial build (or whatever there may be) in google app engine?
I tried deleting all versions and instances (you cannot delete them all, it won't let you delete the serving one).
I tried increasing the version number in appengine-web.xml.
I tried gradle clean.
I tried disabling and enabling the application in the google cloud console.
No luck.
One idea I have is maybe if I can somehow force all the files for the app to be uploaded again? Because with gradle appengineDeploy only the changed files get uploaded.
EDIT:
I managed to fix one part of this by upgrading classpath 'com.google.cloud.tools:appengine-gradle-plugin:2.2.0' in my gradle to the latest version. (Felt kind of hackish to do that and not reliable in the future if this happens again.) Maybe this flushed whatever was in app engine or something by doing that. I still have one problem remaining:
I am using the firebase admin SDK to send firebase cloud messages (authenticated with a .json credentials file) like this:
FirebaseMessaging.getInstance().send(msg);
When I do I am getting this error:
com.google.firebase.messaging.FirebaseMessagingException: Unexpected
HTTP response with status: 401; body: null
Which is strange because other parts of the firebase admin SDK are working (like writing/reading from firestore).
So there is still something weird going on and I think it has to do with the fact that as I was uploading my google app engine project the connection was interrupted.
EDIT 2:
Here is something interesting: When I deploy my application using gcloud app deploy appengine-web.xml the application again will not work. Deploying with gradle appengineDeploy does though. I also noticed that the size of the application is shown as much smaller in the GCP console when deploying with gcloud app deploy appengine-web.xml. So something is messed up here. I tried looking up some sort of gcloud command to clear cache? Or something like that but no luck.
EDIT 3:
Additional Info:
My app was already deployed and working before the failed attempt. I changed one small piece of code in a function and upon uploading the app, the connection was interrupted because the internet went out.
I am on app engine standard environment.
I am deploying my application from the macOS terminal and using android studio to develop.
I have tried the stopPreviousVersion promote and version configs in gradle (actually the first two are true by default and version gets auto-generated if you do not set it).
Running gcloud app deploy appengine-web.xml --verbosity=debug shows a lot of sensitive information but one thing I am seeing is all the files in WEB-INF are being skipped:
DEBUG: Skipping upload of [WEB-INF/....
INFO: Incremental upload skipped 100.0% of data
DEBUG: Uploading 0 files to Google Cloud Storage
DEBUG: Using [16] threads
So perhaps files are not all being uploaded? This SO post raises a similar problem but has no solution: stackoverflow.com/q/42137452/3075340
It's weird but when I do gcloud app deploy, the app won't work at all. There are run-time errors all over the place. Doing gradle appengineDeploy fixes that but I still am having the firebase-admin issue.
To actually answer your question, GAE uses a staging bucket to cache files.
You can delete this bucket, and it will get recreated next time you try to deploy.
The bucket is always named staging.[PROJECT-ID].appspot.com. It should show on the buckets overview page
Just to be clear, I'm not anywhere near sure that this will actually resolve your issue, but this will most definitely clear whatever files GAE cached
Here there was a discussion on a, more or less, similar issue on App Engine from last year.
You can use it as a source of inspiration to fix your current issue, because I think they are really similar as a concept. From there I think it may worth trying to put the appengine-api-1.0-sdk-1.9.63.jar ( or whatever your version is ) file under WEB-INF/lib.
It the same post there were some guys who found out the there was a problem with gcloud v194 and switching to another version fixed the issue.
Here there is a similar issue which was resolved by upgrading the gcloud tool.
Anyhow, this behaviour is not normal and things should not work like this. You can try to find a workaround, but my recommendation is to report the issue and let an App Engine engineer taking a look over it. There are good chances to be something internal.
I cannot get my server code to update. I'm running a PHP instance on GAE and no matter what I do, the files won't update. In the source code view, I can see the files have updated, but when I attempt to access the updated file, I'm still viewing the old version. I've also attempted disconnecting my Bitbucket repo and using the appcfg.py update project-name command, but the files aren't refreshing when I attempt to access them. I'm not sure what to do to force the changes to take place.
My app.yaml contains the following code
- url: /(.+\.php)$
script: \1
secure: always
So the files should be getting read, right?
I was able to figure out what went wrong. I downloaded my code using appcfg.py download_app -A <your_app_id> -V <your_app_version> <output-dir> and noticed that I was downloading the old versions of the files (and wasn't downloading the new files). Turns out using source control within GAE will upload new code, but won't deploy it. I attempted to use appcfg.py update project-name one more time, but it didn't work. Turns out I didn't disconnect my Bitbucket account (could have sworn that I did...). Once disconnected, I was able to update the project using appcfg.py update project-name. While I was figuring this out, I reached out to Google support and received this message:
To use the feature of push to deploy you need to spin-up the Jenkins
Instance on GCE (Google Compute Engine) and then it will take the
updated code and execute it in the environment. Go through [1] for how
to enable the Jenkins instance and its configuration according to
different run time.
In your issue, you just mirrored the code from Bit Bucket to Cloud
Repository, as it is just doing the version control for the
application not executing the application. So basically you have have
the option of using Jenkins instance as I described above to test the
different version of the code or using the appcfg.py update command
from your local repository.
I haven't attempted to install and use Jenkins since I fixed it after disconnecting my Bitbucket account), but it may help others who have run into this problem.