We want all our static requests to an CDN, and the CDN is a Google Storage Object.
expamle.come/static/file.js => cdnurl.com/example-bucket/file.js
And since we're using Storage Objects from the same platform ( Google obviously), we thought this should be a piece of cake, but I couldn't find any doc in Google.
here's our current dispatch.yaml.
dispatch:
# Default service serves simple hostname request.
- url: "*/example.com"
service: default
# Send all api traffic to the api backend.
- url: "*/api/*"
service: backend
We want this :
# Send all api traffic to the static CDN.
- url: "*/static/*"
ourbound: https://cdn/our-bucket/static/*
On a side note, is it a good idea to have another service that uses an NGINX in front of our app that does this?
The purpose of the dispatch.yaml file is to route requests to different services rather than to define url to serve the static files. You can nonetheless define which service will take care of serving the static files. The, to define where and how the static files are retrieved you should modify the app.yaml of the service that will handle the requests, in this case the app.yaml for the default service.
One option would be to serve the static files directly from the application. For this you just need to put the files in the ./public folder and set your handlers as explained in the following part of the documentation.
Second option would be to let Google Cloud Storage serve the static files. This would require creating a bucket, putting the assets therein and allowing public access to the files. This might be done through the urls provided by Cloud Storage as could be seen here.
Regarding your question of load balancing, I don't think it makes sense to put another service for GAE service for load balancing as it is not the intended use for such product.
Related
I have a Create React App front end, served from an EC2 express instance behind an nginx.
I'd like to know the best way/ pros and cons between my two approaches to providing config files such json settings or even static image assets.
These assets are stored in S3 (staticAssets/subdomain/config.js) and made publicly available via URL.
I can use window.location on app load, parse the subdomain and append that to a basic fetch request to retrieve the appropriate config file in s3.
It seems that I could use nginx config to proxy requests from the client to the s3 bucket and retrieve the configs this way.
I'm a bit stumped on if there is a significant difference to either approach. Implementation wise, the url determination from the FE seems much simpler then dealing with the nginx conf.
What are the differences and pros/cons of either approach?
Is there a better architecture/pattern then either of these approaches?
I could also benefit from some clarification from the accepted answer on this post and specifically what "associating your domain name with s3 bucket" means/enhances. https://serverfault.com/questions/414440/nginx-static-content-with-amazon-s3
How do I direct the www. subdomain to just domain.tld without www? I'm used to firebase doing this automatically. Should I look into configuring the app.yaml, dispatch.yaml, or another method?
What you're describing is called a "naked domain", and this is described in the documentation on Custom Domains. The documentation provides the steps for mapping a custom domain to your app and updating the DNS records at your domain registrar once your service has already been mapped to your custom domain in App Engine.
To redirect your requests, you can use wildcard mappings with services in App Engine by using the dispatch.yaml file. You can find instructions on how to do that here. If you would like to know more about routing requests, you can take a look at this documentation as well which also highlights creating a dispatch file. Handlers are limited to handle URLs by executing application code, or by serving static files uploaded with the code, such as images, CSS, or JavaScript. Therefore, they cannot directly redirect one URL to another.
You would need to handle your URL by running a script that executes code that will redirect your URL.
The comment shows a complete example as the script runs main.py which then redirects the URL
I'd like to be able to access the files in a Cloud Storage Bucket from my App Engine App without making the objects or the bucket itself Publicly Readable. While I'm aware of a bunch of options out there that allow access to bucket objects with authentication (client libraries, signed urls, etc.), the complicating factor is that I'd like to be able to access the files with path that is similar to the folder structure of the bucket in question.
For example, if I make my bucket publicly readable, I can access objects with the public link: https://storage.googleapis.com/MY_BUCKET/FOLDER_IN_MY_BUCKET/FILE_IN_FOLER.txt. This url mimics the internal folder structure of the bucket. However, there doesn't appear to be a comparable url if the bucket is not publicly readable. My App Engine App service account has been added as a storage admin for the bucket I need but I'm not sure if there's a url that I can use to access the buckets objects. An object's mediaLink won't work because generation information is appended to the end, and selfLink results in a 404 error.
The need for a url like this is because the bucket contains several thousand objects. Downloading them using a client library to the the App Engine's persistent storage kind of defeats the purpose of using cloud storage in my case. Obtaining signed urls for all of them when a request is made would be time consuming and then I'd have to manage thousands of signed urls somewhere.
Is there a way to read from the cloud storage bucket with a predictable url, like the public url, while also still authenticating the request?
Rather than trying to vend thousands of signed URLs in the response you can create a 'redirect' endpoint in your app engine app.
e.g. user does a 'GET' against www.myapp.com/fetch/<bucket>/<object>
Your app engine code handling this endpoint authorizes the user to make sure they should have access, pulls the bucket/object out of the URL, then generates a signed URL granting access to the resource and returns a 302 redirect to that URL.
The URL you mention can be accessed without making the bucket or file public given that your browser is authenticated with an account having access to those resources:
https://storage.cloud.google.com/MY_BUCKET/FOLDER_IN_MY_BUCKET/FILE_IN_FOLDER
Regarding the access to the file through a different application (for example App Engine), you can always use the client libraries for your preferred language. You can test how the API works in the documentation, just by defining the bucket parameter as MY_BUCKET and the object parameter as FOLDER_IN_MY_BUCKET/FILE_IN_FOLDER. You should use this same structure when applying it to the client library of your choice.
I currently have an Google App Engine Flexible project with four services. And when I map my custom domain to my project using the documentation https://cloud.google.com/appengine/docs/standard/python/mapping-custom-domains, it automatically points to the default service which is not the frontend application. How do I map it to a different service.
The answer from #dan isn't up to date anymore:
The naming of the dispatch.yaml file changed from 'module' to 'service', like this:
dispatch:
- url: "sub1.yourdomain.com/*"
service: web-app
You deploy the stand-alone-file via this command (it hasn't to be in a project folder):
gcloud app deploy dispatch.yaml
Reference: https://cloud.google.com/appengine/docs/standard/python/config/dispatchref
You cannot map a certain (sub)domain to a certain service in the app-level custom domain mapping, mapping is done only at the app level (as a whole).
To direct a certain (sub)domain to a certain service inside your app you'll need to use a dispatch file, for example:
dispatch:
- url: "example.com/*"
module: <frontend-service-name>
Side note: you may want to revisit the decision of handling the frontend in a non-default service: the frontend is IMHO best suited to handle any garbage request coming in (which would typically not match any routing rule and would thus be directed towards the default service). If your default service does something more sensitive than the frontend it might not like that spam coming in.
I'm developing a Single Page Application on Google App Engine.
The backend will be in Go and the frontend in React.
For the backend I would like to use Google Endpoints.
This can't be used with a custom domain so I will use CORS:
https://code.google.com/p/googleappengine/issues/detail?id=9384
But now the question is how should I host the frontend. These are just static files. Should I use a separate GAE project for this? Is there a better solution?
GAE can easily serve static files, just mark them as static in your app.yaml.
https://cloud.google.com/appengine/docs/go/config/appconfig#Go_app_yaml_Static_file_pattern_handlers
For efficiency, App Engine stores and serves static files separately
from application files. Static files are not available in the
application's file system.
Example:
handlers:
# All URLs ending in .gif .png or .jpg are treated as paths to static files in
# the static/ directory. The URL pattern is a regexp, with a grouping that is
# inserted into the path to the file.
- url: /(.*\.(gif|png|jpg))$
static_files: static/\1
upload: static/.*\.(gif|png|jpg)$
I believe they are served from Google's general infrastructure, from a datacenter near to the end user. So it seems like a good idea to do it like this.
In fact for a SPA you will find instances will not spin up if you just serve static files :)
CORS support details also available on that link.