I am using App Engine Modules in my python project. (https://developers.google.com/appengine/docs/python/modules/#Python_Background_threads)
I am also receiving email in m project:
https://developers.google.com/appengine/docs/python/mail/receivingmail
I want to direct the emails to my worker module and not the default module. To that end my worker.yaml has the following settings
worker.yaml
api_version: 1
application: integrate
module: worker
version: 1-0-0
runtime: python27
threadsafe: true
inbound_services:
- mail
builtins:
- deferred: on
handlers:
- url: /admin/.+
script: src.worker.main.app
login: admin
- url: /_ah/mail/.+
script: src.worker.main.app
login: admin
- url: /.*
script: src.worker.main.app
app.yaml
api_version: 1
application: integrate
version: 1-0-0
runtime: python27
threadsafe: true
builtins:
- deferred: on
handlers:
- url: /admin/.+
script: src.default.main.app
login: admin
- url: /.*
script: src.default.main.app
I even tried adding a dispatch.yaml
application: integrate
dispatch:
- url: "*/_ah/mail/.+"
module: worker
But no matter what I do the emails which reach my app are handled by the default module. Any idea what I am missing here? I see the emails coming in but no matter what I do they only go to the default module.
Inbound services could be used only within default module and that is expected behavior. The fact that it works for you locally in devserver is a bug, actually.
Just some additional info for the answer which may help folks in a similar situation.
I noticed in the DevServer log:
"Skipping dispatch.yaml rules because /_ah/mail/[EMAIL_ADDRESS_FOR_APP] is not a dispatchable path."
This is no doubt due to local config, however.
Regardless, the workaround I have now using Tasks is:
Dispatch or directly handle Inbound Mail in the default module
Provide a script handler that creates a Task, taking the relevant MailMessage data as the payload
Set the TaskQueue in queue.yaml to target the module you wish to process the payload data, e.g. a 'worker' module
Related
I have two application that I want to deploy with Google Cloud App Engine.
One of them is react front end, and I want to serve this through www.videoo.io
Second one is back-end, which will be served via api.videoo.io
Frontend yaml file react.yaml :
runtime: nodejs16
env: standard
handlers:
- url: /static
static_dir: static
secure: always
- url: www.videoo.io/*
service: frontend
script: auto
secure: always%
API yaml file, api.yaml :
runtime: python37
entrypoint: gunicorn -b :$PORT videoo.wsgi
service: "videoo-api"
env: standard
handlers:
- url: api.videoo.io/*
service: backend
script: auto
secure: always%
Is this the correct way to achieve this ?
What is the best strategy to serve these two separate applications that will interactively communicate (Frontend will make calls to API to get object information that is stored Django app) ?
Here is also my domain name information in Google App Engine settings :
You are on the right path. You are using the microservices architecture which is basically deploying individual apps as parts (services) under a single project.
Your frontend service seems to be your default so you don't need a service name for it. Every GAE App needs a default service.
Rename react.yaml to app.yaml (since it will be your default service) and update the contents to
runtime: nodejs16
env: standard
handlers:
- url: /static
static_dir: static
secure: always
- url: /.*
script: auto
secure: always
Also rename your api.yaml to backend.yaml since that is what you called your service (not sure if this is required but I do that to easily track of what is controlling my service). Update the contents of the file to
service: backend
runtime: python37
entrypoint: gunicorn -b :$PORT videoo.wsgi
env: standard
handlers:
- url: /.*
script: auto
secure: always
You'll need a dispatch.yaml file to route traffic to the different services. Something like
dispatch:
# Send all api traffic to the backend service.
- url: "api.videoo.io/*"
service: backend
# Send all other traffic to the default (frontend).
- url: "*/*"
service: default
Final step is that during your deploy, you will deploy the 2 services in addition to your dispatch.yaml file. The dispatch.yaml file has to be in your project root folder
gcloud app deploy app.yaml dispatch.yaml <path_to_backend.yaml>
I'm running a static site on GAE and using a custom domain (let's call it example.com) with SSL certificates enabled. I'd like to canonicalize URLs to https://www.example.com/. That means catching any requests to myproject.appspot.com, plain HTTP, and/or the naked domain, and redirecting to www over HTTPS.
I understand that it's not possible to put redirect logic in app.yaml, but ideally I'd like to keep the static file serving logic there, and only have app code for the redirect. (As opposed to doing the static serving in app code as well.)
Here's what I have so far:
Contents of the file app.yaml:
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /
static_files: www/index.html
upload: www/index.html
- url: /(.*)
static_files: www/\1
upload: www/(.*)
Contents of the file dispatch.yaml:
dispatch:
- url: "myproject.appspot.com/*"
module: canonicalizer
Contents of the file canonicalizer.yaml:
module: canonicalizer
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: canonicalizer.app
Contents of the file canonicalizer.py:
import webapp2
def get_redirect_uri(handler, *args, **kwargs):
return 'https://www.example.com/' + kwargs.get('path')
app = webapp2.WSGIApplication([
webapp2.Route('/<path:.*>',
webapp2.RedirectHandler,
defaults={'_uri': get_redirect_uri, '_code': 302}),
], debug=True)
As you can see, I've only attempted to implement redirecting myproject.appspot.com so far. I haven't been able to get it working; myproject.appspot.com still serves content rather than redirecting to the custom domain.
I saw a similar SO question and used it as a basis for my code above. I followed it fairly closely, so I'm not sure if it's outdated or missing details.
I'm not very familiar with webapp2. Also open to solutions in a different framework or even different programming language.
As sllopis said in their answer, an HTTP to HTTPS redirect can be implemented via a secure: always element.
The rest of what I wanted to do needed to be done in app code. The code in my answer was on the right track, but I had some confusion about how services work in GAE and about dispatch.yaml. Here's my final code:
<application root>/app.yaml
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /
static_files: www/index.html
upload: www/index.html
secure: always
redirect_http_response_code: 301
- url: /(.*)
static_files: www/\1
upload: www/(.*)
secure: always
redirect_http_response_code: 301
<application root>/dispatch.yaml
dispatch:
- url: "*.appspot.com/*"
service: canonicalizer
- url: "example.com/*"
service: canonicalizer
<application root>/canonicalizer/app.yaml
service: canonicalizer
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: canonicalizer.app
<application root>/canonicalizer/canonicalizer.py
import webapp2
def get_redirect_uri(handler, *args, **kwargs):
return 'https://www.justinforcentral.com/' + kwargs.get('path')
app = webapp2.WSGIApplication([
webapp2.Route('/<path:.*>',
webapp2.RedirectHandler,
defaults={'_uri': get_redirect_uri, '_code': 301}),
], debug=False)
This allows all the redirects to be done while still maintaining the ability to route the static site via static_files handlers.
As an aside, I also didn't realize that simply doing gcloud app deploy . from the application root only deploys the default service. To deploy this whole thing I had to run gcloud app deploy . dispatch.yaml canonicalizer.
Mapping Custom Domains to GAE
App Engine allows applications to be served via a custom domain, such as example.com, instead of the default appspot.com address. You can create a domain mapping for your App Engine app so that it uses a custom domain.
You will need to do the following:
Verify that you are the owner of your domain through Webmaster Central
Ensure that your domain has been verified.
Delegate the ownership of your domain to other users or service accounts, if needed.
Map your domain to your App Engine app.
Fill out the form with the listed resource records, including their type and canonical name (CNAME).
Add this information to the DNS configuration of your domain registrar.
Securing Custom Domains with SSL
By default, when you map your custom domain to your app, App Engine issues a managed certificate for SSL for HTTPS connections. Securing your custom domains with SSL offers more information about this.
Handling URL requests that do not use HTTPS
Any URL handler can use the secure setting, including script handlers and static file handlers. If secure is set to always, requests for a URL that match this handler that do not use HTTPS are automatically redirected to the HTTPS URL with the same path. Query parameters are preserved for the redirect.
Example in the app.yaml file:
handlers:
- url: /youraccount/.*
secure: always
script: auto
Conclusion
As a result, after following these steps, you should have a custom domain properly mapped to your App Engine site that uses SSL certificates to secure the custom domain.
Moreover, by adding the secure:always handler in your app.yaml file, any URL requests made against your App Engine site will be automatically redirected to the HTTPS URL with the same path.
Update - Redirect all URLs with Google App Engine
Credits to How to redirect all URLs with Google App Engine:
app.yaml
handlers:
- url: /.*
script: main.py
main.py
import webapp2
class MainPage(webapp2.RequestHandler):
def get(self):
self.redirect("https://example.com", True)
app = webapp2.WSGIApplication([
('/', MainPage),
], debug=True)
Then, you can adjust this code to your needs.
I'm refactoring an existing codebase. I switched from using the appcfg.py to using the gcloud command, which seemed to go fine. Our entire codebase was running on one default frontend instance, which I'm now trying to break into services. To start, I created one "worker" backend service, and I'm using a cron job to test.
I can see the worker in the console, but no instance is started. The logs for that service are rapidly flooded with 404's to /_ah/start. I've tried manual and basic scaling. The documentation states that it's okay not to have a startup script, and that a 404 at that endpoint is considered success. However, the instance is not starting.
Logs
worker.yaml
service: worker
runtime: python27
api_version: 1
instance_class: B2
manual_scaling:
instances: 1
threadsafe: false
handlers:
- url: /work/.*
script: worker.app
secure: always
login: admin
worker.py
import webapp2
import handlers
config = {
#...
}
app = webapp2.WSGIApplication([
webapp2.Route(
'/work/test<:/?>',
handlers.Test,
methods=['GET'],
),
], debug=True, config=config)
dispatch.yaml
dispatch:
- url: "*/work/*"
module: worker
I want to create very standard setup for GAE (php runtime): 2 modules with specific URLs (routings):
module-api for REST API
module-oli for backend static process ..
regarding to doc (https://cloud.google.com/appengine/docs/php/modules/ and https://cloud.google.com/appengine/docs/php/modules/routing) I've created 3 .yaml config files:
dispatch.yaml
application: ABC
dispatch:
- url: "*/oli/*"
module: module-oli
- url: "*/"
module: module-api
app.yaml
application: ABC
version: v1
module: module-api
runtime: php55
api_version: 1
handlers:
- url: /.*
script: public/api.php
module-oli.yaml
application: ABC
version: v1
module: module-oli
runtime: php55
api_version: 1
manual_scaling:
instances: 1
handlers:
- url: /.*
script: public/oli.php
I also tried many changes in URL handling, but the error I always get is "Duplicate module: module-api".
Can you help me please? Thank you in advance ..
I have no idea why, but (on my computer) it does not work just from googleAppEngineLauncher.app (OSX 10.11.3, googleAppEngineLauncher version 1.9.32), but it works if I use command line tools:
dev_appserver.py app.yaml module-oli.yaml
and only if I do not use param --skip_sdk_update_check (does not matter value)
more about naming modules (and using default module or not) .. it all works even if I name default module with specific name (of course, it causes error when not all request are dispatched to the module).
more about dispatching to modules; it works (as it's documented here). of course, it does not work at local development environment, as every single module runs at a different port (than the dispatch does not work, and if there is a module with manual_scaling, the _ah/start can not be handled and as err500 it stops the start)
here are all setup files, and how to run it:
app.yaml
application: <APPLICATION-ID>
module: default
version: v1
runtime: php55
api_version: 1
threadsafe: yes
handlers:
- url: /favicon\.ico
static_files: favicon.ico
upload: favicon\.ico
- url: .*
script: blablabla.php
module-oli.yaml
application: <APPLICATION-ID>
version: v1
module: module-oli
runtime: php55
api_version: 1
threadsafe: yes
instance_class: B1
manual_scaling:
instances: 1
handlers:
- url: /oli/.*
script: blebleble-oli.php
dispatch.yaml
dispatch:
- url: "*/oli/*"
module: module-oli
- url: "*/*"
module: default
to run on local development environment: dev_appserver.py app.yaml module-oli.yaml .. (no automatic routing, instances are running on different ports)
one everything is deployed to gcloud and dispatching is updated (appcfg.py -A wellfedcat-1221 update_dispatch .), it works like this:
APPLICATION-ID.appspot.com/* : served by default module ..
APPLICATION-ID.appspot.com/oli/* : served by module-oli ..
the dispatching is necessary when you want to use our own domain to map to gcloud ..
thank you #Tom for help!
Problem fixed via Google Cloud support:
I removed the /_ah/spi/* handler from my endpoints yaml file, and the
- url: /.*
script: api.app
did not instantiate the endpoint.
It used to work before since the API was deployed for the previous version, but with the new version, there was nothing explicit to deploy the endopoints. So handlers need to be
handlers:
- url: /_ah/spi/.*
script: api.app
- url: /.*
script: api.app
Keeping the issue below for reference purposes
I've just deployed a new version of my application, and calls to http://app.appspot.com/_ah/api/app/version/method now return a 404. It worked perfectly before the update.
However, there's no trace at all in the logs, and no instance launched when I call/ping those URIs.
While if I call /_ah/whatever/app/version/method, I still have a 404, but it appears in my module logs, and it has the following message
{"state": "APPLICATION_ERROR", "error_message": "Not Found"}
The app is configured using modules, my app.yaml is defined with
application: appname
version: 2015-04-07
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /_ah/spi/.*
script: api.api.app
- url: /.*
script: www.www.app
secure: always
libraries:
- name: pycrypto
version: latest
- name: endpoints
version: 1.0
- name: webapp2
version: latest
- name: jinja2
version: latest
And, in api/api.yaml
application: appname
module: default
version: 2015-04-07
runtime: python27
api_version: 1
threadsafe: true
inbound_services:
- warmup
handlers:
- url: /.*
script: api.app
libraries:
- name: pycrypto
version: latest
- name: endpoints
version: 1.0
- name: ssl
version: latest
I've updated the app to serve this new version in the admin console, and all other modules work perfectly.
Also, I can't see my API in the API explorer, https://appname.appspot.com/_ah/api/explorer returns an empty list (while I see it when running the dev server on localhost).
Update: I've just noticed, looking at #bossylobster reply in GAE cloud endpoints - Api not updating after deploy, that I do not have "Successfully updated API configuration" in my logs after the "Completed update of a new alternate version". However, I have "API deletion serving" at about the time everything started to be 404. Yet, I have no idea why there's this API deletion query in my logs. Any idea of what can be wrong?
That's an app in production and so the mobile version is down at the moment. I'm happy to send the app ID to a devrel in PM if that helps.