We've deployed a Vue SPA to Google App Engine and it's served completely by the static handlers.
The issue that we are facing is that if a user is active on our site mid-deploy, then their old Webpack chunk manifest becomes invalid (since some chunks' hashes are overwritten). If they now try to route to a new page and that page tries to fetch a chunk that got overwritten, we get the following error:
ChunkLoadError: Loading chunk Conversations failed.
(error: https://example.com/js/Conversations.71762189.js)
Ideally, we'd like to keep N (2-3?) previous versions of the app's static files.
Is our only option to push all the assets to a Cloud Storage Bucket? If so, how would we go about pruning older versions?
Here is my app.yaml for reference:
runtime: nodejs10
instance_class: F4
automatic_scaling:
min_instances: 2
max_instances: 10
default_expiration: "30d"
error_handlers:
- file: default_error.html
handlers:
- url: /api/*
secure: always
redirect_http_response_code: 301
script: auto
- url: /js/*
secure: always
redirect_http_response_code: 301
static_dir: dist/js
- url: /css/*
secure: always
redirect_http_response_code: 301
static_dir: dist/css
- url: /img/*
secure: always
redirect_http_response_code: 301
static_dir: dist/img
- url: /(.*\.(json|js|txt))$
secure: always
redirect_http_response_code: 301
static_files: dist/\1
upload: dist/.*\.(json|js|txt)$
expiration: "10m"
- url: /.*
secure: always
redirect_http_response_code: 301
static_files: dist/index.html
upload: dist/index.html
expiration: "2m"
The issue typically happens when the deployment overwrites an existing service version which receives traffic (i.e. the service version is not changed). From Deploying an app:
Note: If you deploy a version that specifies the same version ID as a version that already exists on App Engine, the files that you
deploy will overwrite the existing version. This can be problematic if
the version is serving traffic because traffic to your application
might be disrupted. You can avoid disrupting traffic if you deploy
your new version with a different version ID and then move traffic to
that version.
As long as a service version is deployed and not deleted or overwritten its respective static assets remain accessible.
To prevent the issue always deploy using a fresh service version, then (gradually) migrate traffic to the newly deployed version. Keeping the latest N service versions around will give you those N sets of static assets you desire.
In general, this deployment practice is good/recommended for a few other reasons:
avoids potential outages, see Continuous integration/deployment/delivery on Google App Engine, too risky?
avoids potential traffic loss while GAE spins up enough new version instances to handle the traffic load, see 2nd half of GAE shutdown or restart all the active instances of a service/app
Potentially of interest: Google Frontend Retention between deployments
Deploy using the --no-promote flag, and utilize the Traffic Migration feature in the Standard Environment to gradually migrate traffic over to the new version so that all users don't experience an immediate switchover the moment the new version goes live. App Engine will host both the old and new versions (or, "blue" and "green") for a period of time until all traffic points to the new version, and then the old version will be shut down.
See also:
Testing on App Engine
Migrating and Splitting Traffic with the Admin API
Blue-Green Deployment Pattern
Related
If part of my app.yaml file looks like this:
handlers:
- url: /favicon\.ico
static_files: favicon.ico
upload: favicon\.ico
- url: /static
static_dir: public
- url: /.*
secure: always
redirect_http_response_code: 301
script: auto
automatic_scaling:
min_idle_instances: automatic
max_idle_instances: automatic
min_pending_latency: automatic
max_pending_latency: automatic
max_concurrent_requests: 1
min_instances: 1
max_instances: 10
Then is my static content also affected by the scaling parameters for the app? Example, it would run with the same max_concurrent_requests restriction per node, or not?
My assumption is that serving /static would be a completely different layer independent from the instances running for your app in GAE. I was trying to find an architecture diagram confirming this kind of decoupling (maybe a diagram with nginx running with a LB to the GAE Application Instance nodes).
Ideally, a clear answer would be qualified with a reference to Google Cloud documentation material.
Closest related doc I found was this, but it does not clearly answer my question:
Storing and Serving Static Files
Your understanding of the static file serving architecture is correct. App Engine will handle the static file request directly without letting the requests get to the language runtime.
Because of that, these requests will not be affected by the scaling settings the same way as the "regular" requests would. The max_concurrent_requests is a good example of that.
I have requested an update to the documentation page you referenced to add this information there.
I have a flask + react application that is deployed on Google App Engine. Recently, I discovered that each time I deployed a new version to the GAE, my site would go down for a few hours, and several web pages cannot load correctly. I checked the console, the web application is trying to get the static files from the last version, which resulted in a 404 Error. Can anyone help me to find what the problem is?
Here is my app.yaml file:
runtime: python37
env: standard
default_expiration: "5m"
entrypoint: gunicorn -b :$PORT main:app --timeout 150
instance_class: F4
automatic_scaling:
max_instances: 5
min_instances: 1
min_pending_latency: "5s"
target_cpu_utilization: 0.75
inbound_services:
- warmup
handlers:
- url: /static/js/(.*)
static_files: build/static/js/\1
upload: build/static/js/(.*)
- url: /static/css/(.*)
static_files: build/static/css/\1
upload: build/static/css/(.*)
- url: /static/media/(.*)
static_files: build/static/media/\1
upload: build/static/media/(.*)
- url: /(.*\.(json|ico))$
static_files: build/\1
upload: build/.*\.(json|ico)$
- url: /
static_files: build/index.html
upload: build/index.html
- url: /.*
script: auto
I am here to answer my own question. I seem to find the problem and how to solve it.
The main problem seems to be a caching issue. For the app.yaml settings, although the default expiration time is set to 5m, the url with path don't have the expiration set. For example, page www.example.com/about will have a different caching time than the js package. This means when a new build folder is deployed, the js packages have been changed, but the www.example.com/about page generated by your backend application is still the old version, and it will try to request the js package from the previous build foler. Thus, causing the 404 error.
The way to solve this is to set the expiration time for your response generated by your backend application. I am using the Flask environment, so the code for that is (credited to this answer)
response["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response["Pragma"] = "no-cache" # HTTP 1.0.
response["Expires"] = "0" # Proxies.
the web application is trying to get the static files from the last version
So are these files that were removed in your new version that you just deployed?
In general, it sounds like your problem has to do with stale browser caches. I wouldn't remove static assets from your deployed app right away specifically for this reason.
I see you're using ReactJS. Are you using the features that combine all the js and css into a single file whose name contains a hash? This should help with cache-busting.
The part that's confusing is that you said it would go down for a few hours. You have default_expiration: "5m" in your app.yaml so a few hours sounds a bit extreme. Are you not doing a hard reload (or even a full reload) when you are trying to check out your changes in your browser?
I am still working to get my old style appengines to work under at least go111 (go112 would not work due to dependencies on memcache). I am now stumbling over app.yaml configuration issues with my static files, I used a completely static directory layout before and just specified a few dynamic handlers in the root like this:
runtime: go111
handlers:
- url: /_ah/.*
script: auto
login: admin
secure: always
- url: /dynamic
script: auto
secure: always
- url: /admin/.*
script: auto
login: admin
secure: always
- url: (.*)/
static_files: html\1/index.html
upload: html/index.html
secure: always
- url: /(.*\.map)
mime_type: application/json
static_files: html/\1
upload: html/(.*\.map)
secure: always
- url: /
static_dir: html
secure: always
dev_appserver.py will never call my dynamic entry point. In production this does work, but I am still working on the conversion and would like to test locally. Any hints how to convince dev_appserver.py to let me do this? By the way my gcloud tools are updated as of today.
Your question is why your application works well when you deploy it to production but doesn't work when you use dev_appserver.py to run it locally, and how could you run it with dev_appserver.py. The answer to that is:
You won't be able to run it locally properly using dev_appserver.py, since it does not support runtime Go 1.11. Look at the Local Development Server Options documentation, there's only a link for "Go 1.9".
(as you can see, the links for "Go 1.11" and "Go 1.12" are disabled, which translates to: not supported.)
Documentation for Go 1.11 on App Engine Standard states that in order to test your application locally you would have to use "go run" command (notice how it doesn't mention dev_appserver.py tool). The command would be something like this:
go run [build flags] [-exec xprog] package [arguments...]
For more information about the command go here.
I'm sure you might have already read this but, to know more about the migrations process from Go1.9 to Go1.11 read this documentation.
You have stated that "go run" command wouldn't work for your case. So, a workaround for that would be to test your application directly into App Engine but without migrating the traffic.
When deploying your test version use:
gcloud app deploy --no-promote
To access to it go to:
http://VERSION_ID.default.YOUR_PROJECT_ID.appspot.com
If everything turned out great, you can migrate the traffic on the Cloud Console UI selecting the version you just deployed and clicking "Migrate traffic".
I am creating an offline-first app on Google App Engine, with PouchDB as my local DB, and CouchDB as my remote DB. I have enabled CouchDB on Google AppEngine, and tried to go to the following URL:
https://[my-app-id].appspot.com:5984/_utils/
When I do that, I get the following:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
So I tried enabling https access in the firewall settings.
**Firewalls**
[ ] Allow HTTP traffic
[Y] Allow HTTPS traffic
Still getting the error above.
I searched the documentation but cannot find anything helpful about how to access Fauxton (or Futon) on Google AppEngine. (The instructions only tell you how to access Fauxton on your local machine.)
I have generated a private and public key and logged in to the server via command line.
I have also followed the instructions about configuring the firewall to allow remote access, and have given it to my PC only.
None of this has enabled me to access https://[my-app-id].appspot.com:5984/_utils/
How do I access Fauxton on the Google AppEngine platform?
Update: according to the development tools in my browser, my PouchDB application has successfully created a database to sync to, but it isn't on the server:
app.yaml file
application: [app-name]
version: 4
runtime: python27
api_version: 1
threadsafe: false
handlers:
- url: /
script: main.py
- url: /(favicon)\.ico$
static_files: \1.ico
upload: /(favicon)\.ico
application_readable: true
- url: /(package)\.json$
static_files: \1.json
upload: /(package)\.json
application_readable: true
# Serve images as static resources #
- url: /(.+\.(gif|png|jpg|json|ico))$
static_files: \1
upload: .+\.(gif|png|jpg|json|ico)$
application_readable: true
- url: /index.html
static_files: index.html
upload: index.html
- url: /licence.html
static_files: licence.html
upload: licence.html
- url: /privacy.html
static_files: privacy.html
upload: privacy.html
- url: /pouchnotes.manifest
static_files: pouchnotes.manifest
upload: pouchnotes.manifest
- url: /manifest.json
static_files: manifest.json
upload: manifest.json
# static directories #
- url: /img
static_dir: img
- url: /js
static_dir: js
- url: /css
static_dir: css
libraries:
- name: webapp2
version: "2.5.2"
EDIT: I posted this question in the Bitnami community forum (they provide CouchDB on Google App Engine)
FWIW, one of the references in your post points to Google Compute Engine (GCE), which is an IaaS, not a PaaS like Google App Engine (GAE), you might be looking at the wrong product.
The app.yaml file indicates you are using the standard environment, which doesn't offer ways to configure a listening port. And it also doesn't allow listening sockets. From Limitations and restrictions:
Although App Engine supports sockets, there are certain limitations
and behaviors you need to be aware of when using sockets :
You cannot create a listen socket; you can only create outbound sockets.
The GAE flexible environment might be an alternative as it drops many of the standard environment restrictions, but it's a significantly different solution (which I didn't use yet). The remainder of the answer assumes the flexible environment and it's based solely on the documentation.
Not 100% certain, but you might need to teach your app to listen to port 8080 instead. From Listen to port 8080:
The App Engine front end will route incoming requests to the
appropriate module on port 8080. You must be sure that your
application code is listening on 8080.
Unless you can use the forwarded ports network config (again, not 100% certain, I didn't use flex env). From Port forwarding:
Port forwarding allows for direct connections to the Docker container
on your instances. This traffic can travel over any protocol. Port
forwarding is intended to help with situations where you might need to
attach a debugger or profiler.
By default, incoming traffic from outside your network is not allowed
through the Google Cloud Platform firewalls. After you have
specified port forwarding in your app.yaml file, you must add a
firewall rule that allows traffic from the ports you want opened.
You can specify a firewall rule in the Networking Firewall Rules page
in the Google Cloud Platform Console or using gcloud
commands.
For example, if you want to forward TCP traffic from port 2222:
Modify the app.yaml to include:
entrypoint: gunicorn -b :$PORT -b :2222 main:app
In the network settings of your app.yaml, include:
network:
forwarded_ports:
- 2222/tcp
Specify a firewall rule in the Cloud Platform Console or using gcloud compute firewall-rules create to allow traffic from any
source (0.0.0.0/0) and from tcp:2222.
I'm getting some help with this from the Bitnami Community Forum.
Answers so far...
(1) set up the firewall rules - make sure you have a permanent IP address for this.
(2) set up SSH keys to access the server via command line
(3) sudo /opt/bitnami/couchdb/scripts/ctl.sh stop couchdb
(4) edit local.ini to point to 0.0.0.0 instead of 127.0.0.1 - but note that you will need to type cd /opt/bitnami/couchdb/etc/, press enter, and then sudo vi local.ini (rather than vi local.ini as the instructions suggest).
(5) Log in to the external IP address. (Log in as admin and prefix commands with sudo)
(NB: you don't need to run this in the GAE flexible environment)
I tried to use the documented way of restricting access to urls marked as static by way of login: required rules in the app.yaml file. My intention is to have access to script urls handled by the go programming language by xmlhttprequests, but the first step of authenticating the user before she can load the file dist/index.html fails.
Surprisingly for me the user is not prompted to login, instead receives the dist/index.html file and all other files it asks for from the static folder as if no restricting rule were present.
This is my app.yaml file:
application: helloworld
version: 1
runtime: go
api_version: go1
handlers:
- url: /
static_files: dist/index.html
upload: dist/index.html
secure: always
login: required - this is what fails as far as I'm concerned
- url: /(.*\.(txt|html|json|png|js|log|md|css|ico))
static_files: dist/\1
upload: dist/(.*\.(txt|html|json|png|js|log|md|css|ico))
secure: always
login: required
- url: /.*
script: _go_app
secure: always
login: required
The folder that I uploaded to appengine looks like this:
app.yaml
index.yaml
xhr_responses.go - this is the intended future non static AJAX part
dist/
index.html
loads of other stuff that is static
The 'login:' handler options in the .yaml config files rely on Google's authentication, which can be persisted using cookies and survive a browser restart.
To properly test the authentication you need to either use a fresh incognito browser session or go to one of the Google sites and ensure you're not logged in (explicitly log out from all Google accounts if needed) before testing.
Apparently I was signed in when trying stuff on the live google app engine, which I just forgot is the way it knows not to redirect access to a new login prompt.