I am using Google App Engine Standard Environment for a node.js app.
I have a folder named public for my static files. This contains files such as chunk-XIAWB55V.js. I want these files to be heavily cached as the hashed filename will change if the contents change.
I added a static file handler to my app.yaml to redirect requests starting /static to my public folder, and I confirmed it worked by checking the logs: requests to e.g. https://<redacted>.com/static/javascript/build/prod/chunk-Z4M5HAC7.js are in the request log and are not hitting my app. The docs imply that a default cache of 10 minutes should be applied. However the browser devtools show that the actual headers are cache-control: no-cache, must-revalidate and a request (which gets a 304) is sent every time.
I have tried adding a expiration param to my app.yaml but it did not make a difference. I also tried setting the headers in app.yaml but it did not make a difference. My current file looks like:
handlers:
- url: /static
static_dir: public
secure: always
expiration: "1d"
http_headers:
Cache-Control: public
Vary: Accept-Encoding
- url: /.*
secure: always
script: auto
I am still getting responses like:
I am using Identity Aware Proxy in front of App Engine and I know from e.g. this question that IAP and caching can mix badly but that seems to refer simply to possible IAP bypass. My static files are not sensitive and I am happy to accept that risk.
GCP support have suggested this is an edge case due to the use of IAP. A workaround I have implemented which I can recommend for anyone else in this situation is to remove the static route and serve the files from your app instead. While this increases load on the app, the high max age means they will not be requested often.
There's some documentation of this at : This page describes best practices for using Identity-Aware Proxy (IAP) https://cloud.google.com/iap/docs/concepts-best-practices -- TL;DR, App Engine does some caching for static_files that interacts poorly with IAP. That page has some instructions you can apply if you want to protect your static_files.
Related
How do we stop App Engine from using caches of previously loaded files?
According to Google's documents here under "App Caching", they suggest that App Engine caches files that are loaded. They seem to suggest that it is related to the presence of their main() function. However, we removed all references to main() and find that App Engine is still using cached files.
We also tried this HTML below per this answer, but are still getting cached results in the browser. (When we return to the home page - the tables which load JSON are not getting the new file results - they are receiving the old versions of the files.)
<meta http-equiv="cache-control" content="no-cache, no-store, must-revalidate">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="0">
Here is our app.yaml
application: OurApp
version: 0.5
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /stylesheets
static_dir: stylesheets
handlers:
- url: /scripts
static_dir: scripts
- url: /.*
script: main.app
Any suggestions for how to prevent this file load caching behavior in App Engine?
If the files you are trying to update are static files, such as those inside the /stylesheets or /scripts folders from your project, you should take a look at this doc for static cache expiration for GAE projects in the standard environment for Pyhton. According to it, "files are likely to be cached by the user's browser, as well as by intermediate caching proxy servers such as Internet Service Providers". But I've found a way to flush static files cached by your app on Google Cloud.
Head to your Google Cloud Console and open your project. Under the left hamburger menu, head to Storage -> Browser. There you should find at least one Bucket: your-project-name.appspot.com. Under the Lifecycle column, click on the link with respect to your-project-name.appspot.com. Delete any existing rules, since they may conflict with the one you will create now.
Create a new rule by clicking on the 'Add rule' button. For the object conditions, choose only the 'Newer version' option and set it to 1. Don't forget to click on the 'Continue' button. For the action, select 'Delete' and click on the 'Continue' button. Save your new rule.
This new rule will take up to 24 hours to take effect, but at least for my project it took only a few minutes. Once it is up and running, the version of the files being served by your app under your-project-name.appspot.com will always be the latest deployed, solving the problem. Also, if you are routinely editing your static files, you should remove any expiration element from handlers related to those static files and the default_expiration element from the app.yaml file, which will help avoid unintended caching by other servers.
The section of the doc that you cite has to do with how python files on the server side are handled. It's a server-side optimization, and is entirely separate from how a browser caches HTML from your application.
You can specify additional HTTP headers (e.g., Cache-Control) in app.yaml. See the Static File Handlers section in the doc. You can also add headers dynamically via the self.response.header. This is discussed, with a small example, in the Responses section of the doc.
Long story short:
I'm building a static web application which seems to work fine locally when served with the Node http-server, but when deployed to Google App Engine, has issues. I've traced the problem to the content-type header: when serving locally, some of my files (from an Emscripten library) are served with application/octet-stream; charset=utf-8 which works. When served from App Engine, they're served with simply application/octet-stream, which does not work. I've been able to verify this by adding a mime_type: application/octet-stream; charset=utf-8 line to one of my app.yaml handlers, but there are dozens or hundreds of files intermingled with different MIME types. I don't want to mark them all as octet-stream if I don't have to.
So is there a way in app.yaml to simply specify a default mimetype besides application/octet-stream? If not, I'll have to get much more creative with my handler matchers.
You can't set a default global mime type, but you can (as you mention) get creative. This is a good example of an app.yaml file for a static website, it's not unusual to define cases for each different file type, in order to have fine control over placement/mime.
In your case, you might wish to extend on the following idea:
- url: /(.*\.(svg|svgz))
mime_type: images/svg+xml
static_files: static/\1
upload: static/(.*\.(svg|svgz))
Using this pattern you can match multiple file types with the same mime type. Might be the quickest (& dirtiest?) path to solve your problem.
Here's another app.yaml example for further reference.
I'm using Google App Engine 1.7.2 / Python 2.7 and am trying to add client caching of static files.
When I specify static_dir in my app.yaml, the cache-control headers do not get set.
- url: /static/images
static_dir: static/images
expiration: "7d 0h"
However, when I switch the specification over to static_files, like the following, it does get set.
- url: /static/images(.*)
static_files: static/images/\1
upload: static/(.*)
expiration: "7d 0h"
Is anyone else seeing this? Am I missing something? I was under the impression that static_dir and static_files were equivalent if written like the above.
Thanks!
I realise your question was asked a while ago, but I came across it while searching for a similar issue and thought I'd answer it for others' benefit.
You've specified the static file handlers correctly.
The issue with caching may have been due to you being logged in as administrator in the browser (logged in to appengine.google.com). Running your application, without being logged in as administrator, should show caching working as expected. Here's a link to the 'issue'.
Is it possible in the app.yaml to make sure that all requests to my app on http get redirected to https without having to specify secure: always for every url endpoint in my app.
Currently I am doing this:
url: /users/login
script: users_handler.app
secure: always
url: /signin
script: authentication.app
secure: always
url: /users/logout
script: users_handler.app
secure: always
But as new urls are added, its risky as a developer might forget to specify secure always. I would prefer to just specify a global setting that applies to all urls in my app.
If you don not want to use secure in your app.yaml you can accomplish this with webapp2.
https://webapp2.readthedocs.io/en/latest/guide/routing.html#restricting-uri-schemes
And here is a working code eaxmple: How to use WSGI to reroute a user from http to https
I feel like archaeologist, but nonetheless I'll share my findings.
All it has to be made is to add this into app.yaml:
handlers:
- url: /.*
secure: always
script: auto
There is even documentation example with that case. It's exactly the same as mine, but I removed redirection.
I don't believe that is possible. However, I think you can mitigate the problem by restructuring the URLs you have defined in your app.yaml.
The URLs is matched against handlers in your app.yaml starting from the top. So you should specify your one-off URLs and then at the bottom have a catch all setting that routes all URLs that didn't match the other settings to your default handler. You application should then display 404 pages etc. for URLs that don't exist in your application.
Something like this would work:
- url: /signin
script: authentication.app
secure: always
- url: /.*
script: users_handler.app
secure: always
This way you only have to specify a couple handlers for your application and you are much less likely to miss setting secure:always when adding a new URL or setting.
Hope that helps!
Google Page Speed says.
The following publicly cacheable, compressible resources should have a "Vary: Accept-Encoding" header:
for my css and js files.
How can I set google app engine to do this?
From the docs, just add the http_headers for the handlers you need in app.yaml (for Python):
- url: /static
static_dir: static
http_headers:
Vary: Accept-Encoding
Google App Engine tries to serve compressed content when it can. So you shouldn't have to do anything as long as the client follows the guidelines at the link.
The return headers for static content can be set while using the Python language in GAE. Details here: Static Directory Handlers