Long story short:
I'm building a static web application which seems to work fine locally when served with the Node http-server, but when deployed to Google App Engine, has issues. I've traced the problem to the content-type header: when serving locally, some of my files (from an Emscripten library) are served with application/octet-stream; charset=utf-8 which works. When served from App Engine, they're served with simply application/octet-stream, which does not work. I've been able to verify this by adding a mime_type: application/octet-stream; charset=utf-8 line to one of my app.yaml handlers, but there are dozens or hundreds of files intermingled with different MIME types. I don't want to mark them all as octet-stream if I don't have to.
So is there a way in app.yaml to simply specify a default mimetype besides application/octet-stream? If not, I'll have to get much more creative with my handler matchers.
You can't set a default global mime type, but you can (as you mention) get creative. This is a good example of an app.yaml file for a static website, it's not unusual to define cases for each different file type, in order to have fine control over placement/mime.
In your case, you might wish to extend on the following idea:
- url: /(.*\.(svg|svgz))
mime_type: images/svg+xml
static_files: static/\1
upload: static/(.*\.(svg|svgz))
Using this pattern you can match multiple file types with the same mime type. Might be the quickest (& dirtiest?) path to solve your problem.
Here's another app.yaml example for further reference.
Related
I am using Google App Engine Standard Environment for a node.js app.
I have a folder named public for my static files. This contains files such as chunk-XIAWB55V.js. I want these files to be heavily cached as the hashed filename will change if the contents change.
I added a static file handler to my app.yaml to redirect requests starting /static to my public folder, and I confirmed it worked by checking the logs: requests to e.g. https://<redacted>.com/static/javascript/build/prod/chunk-Z4M5HAC7.js are in the request log and are not hitting my app. The docs imply that a default cache of 10 minutes should be applied. However the browser devtools show that the actual headers are cache-control: no-cache, must-revalidate and a request (which gets a 304) is sent every time.
I have tried adding a expiration param to my app.yaml but it did not make a difference. I also tried setting the headers in app.yaml but it did not make a difference. My current file looks like:
handlers:
- url: /static
static_dir: public
secure: always
expiration: "1d"
http_headers:
Cache-Control: public
Vary: Accept-Encoding
- url: /.*
secure: always
script: auto
I am still getting responses like:
I am using Identity Aware Proxy in front of App Engine and I know from e.g. this question that IAP and caching can mix badly but that seems to refer simply to possible IAP bypass. My static files are not sensitive and I am happy to accept that risk.
GCP support have suggested this is an edge case due to the use of IAP. A workaround I have implemented which I can recommend for anyone else in this situation is to remove the static route and serve the files from your app instead. While this increases load on the app, the high max age means they will not be requested often.
There's some documentation of this at : This page describes best practices for using Identity-Aware Proxy (IAP) https://cloud.google.com/iap/docs/concepts-best-practices -- TL;DR, App Engine does some caching for static_files that interacts poorly with IAP. That page has some instructions you can apply if you want to protect your static_files.
I've been working on a new site to be hosted on Google App Engine. I've been deploying my app to a development project and it is hosted here:
Development: https://herdboss-dev.appspot.com/
It works fine.
But when I take the exact same code and deploy it to my other project which going to be the real, production website, it is serving some of the css as application/octet-stream instead of text/css and so those files aren't being parsed by the browser, so almost all of my css is not working on the production site:
Production: https://herdboss-prod.appspot.com/
Even weirder is that SOME of the css is being served correctly. /css/normalize.css is being served as text/css but /css/site.css is being served as application/octet-stream.
https://herdboss-prod.appspot.com/css/normalize.css
https://herdboss-prod.appspot.com/css/site.css
https://herdboss-prod.appspot.com/js/jquery-ui-1.12.1.custom-theme/jquery-ui.min.css
My app.yaml has a static handler for css files:
- url: /css
static_dir: css
secure: always
I tried adding a mime_type as well but that didn't change anything:
- url: /css
static_dir: css
mime_type: 'text/css'
secure: always
EDIT:
While experimenting, I cut my site.css in half, deployed, and then it started serving correctly. Then I reinstated the full size site.css, deployed... and it's still serving properly now.
But my jquery-ui.min.css is still serving as octet-stream. This is crazy.
EDIT2:
And it's serving my svg's with the wrong mime type as well.
Is mime-typing just utterly broken in the GAE? If so, why is it working on my dev gae?
My solution was:
duplicate the offending file and rename it
change one byte (otherwise it will not be uploaded/updated)
verify that it downloads as stylesheet
use the renamed, changed css file in your html
Wait for the original to get served correctly: after about 24h the problem fixed itself
There is an issue on the GAE issue tracker: https://issuetracker.google.com/issues/186012458
Initially, I tested a workaround but this workaround broke after a day: I modified my HTML to load the stylesheets from a version server basically cross-site as (hxxps://VERSION-dot-SERVER.appspot.com/static/stylesheet.css). The version server served the correct mime-type ... but only for some time.
I suspect the problem is a wrong mime-type due to an extremely rare race condition (cache access/refresh during upload?). It is simply not possible to trick the Google Frontend to refresh the file. Even appending .../styles.css?x=1 or switching back to an older code version will not trigger a refresh.
Files and their mime-types seem to be stored based on their hashed-values, so re-upload of a file with the same content has no effect even if it is simply renamed. You will see an "uploaded 0 files" message if you deploy a version with a duplicate file. It has to be changed and renamed.
I did check both of your sites: https://herdboss-prod.appspot.com/, https://herdboss-prod.appspot.com/
I can see that you have several CSS files: normalize.css, jquery-ui.min.css, base-min.css, grids-min.css, grids-responsive-custom.css, css?family=Open+Sans,
site.css, ie-is-terrible.css, css_compiled.css.
All of them served correctly as Type: stylesheet
If you encounter this problem I recommend performing following test:
Check content type in Developer Console > Network with the option Disable Cache > And reload page. If it helps: clean cache of the browser.
Check content of the page in different browser, as content can be interpreted differently depending on the rendering engine.
If 1 and 2 does not work, and you are using nginx, it is most probably configuration problem. Change the file /etc/nginx/conf.d/default.conf. Put /etc/nginx/mime.types in the 'http' section:
http / {
include /etc/nginx/mime.types;
default_type application/octet-stream;
....
}
if it is still not working, move it to location section
location / {
include /etc/nginx/mime.types;
...
}
I have a weird issue on GAE standard (running node), more precisely with my app.yaml below.
runtime: nodejs8
# Environment variables
env_variables:
GOOGLE_CLOUD_PROJECT: '...'
# Static directories and files
handlers:
- url: /static
static_dir: public
Without the handlers part, everything works like a charm: my app is deployed and works. Note that the size is 4.8 MB (version 1-0-43).
However, with the handlers part, my app doesn't work anymore: 4xx error and an app size of 324 KB (version 1-0-43).
The sole purpose of the handlers part is to define a directory to serve static assets (CSS, JS, images...).
Any clue? Thanks.
When you add your static_dir handler definition the entire content of your public subdirectory is no longer (by default) uploaded together with your app code (most likely explaining the app size difference). Instead it's uploaded to a different location to be served directly by the GAE infra, see How to serve static files in AppEngine Standard and nodejs
It sounds that your app may need some of those files as well. In such case the easiest solution would be to add the application_readable flag to the static definition, causing that directory to be uploaded both in the static content location and together with your app code, see GAE: file_get_contents() and static files.
To speedup your deployments could also try to separate the static content in 2: one portion not needed by your app code, deployed without application_readable (thus not also uploaded with your app code, deployed faster) and one with the flag. If it's not too much trouble.
Could you try changing static_dir from 'public' to 'static'. Because the url and static_dir are usually located at the same place. Please try it and let me know. Cheers!
# Static directories and files
handlers:
- url: /static
static_dir: static
I'm using google app engine with Python and have a couple static .png image files but they are all being served with an "image/x-png" content-type. This is a problem when I use a browser like chrome and try to view these images as the content-type is unrecognized, which forces chrome to download it as binary, rather than displaying the image.
How can I make App Engine serve these with the proper "image/png" mime type?
Assuming you are using Java, this is specified normally in the mime-mapping section in the web.xml file. See eg here or here.
In your case, I'd try
<mime-mapping>
<extension>png</extension>
<mime-type>image/png</mime-type>
</mime-mapping>
In Python, it seems you should add some handler to your app.yaml with the appropiate mime_type, for example (replace with your own url and static_dir) :
handlers:
- url: /static/*.png
static_dir: static
mime_type: image/png
I'm using Google App Engine 1.7.2 / Python 2.7 and am trying to add client caching of static files.
When I specify static_dir in my app.yaml, the cache-control headers do not get set.
- url: /static/images
static_dir: static/images
expiration: "7d 0h"
However, when I switch the specification over to static_files, like the following, it does get set.
- url: /static/images(.*)
static_files: static/images/\1
upload: static/(.*)
expiration: "7d 0h"
Is anyone else seeing this? Am I missing something? I was under the impression that static_dir and static_files were equivalent if written like the above.
Thanks!
I realise your question was asked a while ago, but I came across it while searching for a similar issue and thought I'd answer it for others' benefit.
You've specified the static file handlers correctly.
The issue with caching may have been due to you being logged in as administrator in the browser (logged in to appengine.google.com). Running your application, without being logged in as administrator, should show caching working as expected. Here's a link to the 'issue'.