I have an AppEngine app with 2 services, where Service A is queueing tasks for Service B using the task (push) queue. How do I test this using the development server? When running multiple services with the development server, each service gets a unique port number, and the task queue can't resolve the URL because the target URL is actually running on another port, i.e. Service A is on port 8080 and Service B is on port 8081. This all works great in production where everything is on the same port, but how do I go about testing this locally?
The push queue configuration allows for specifying the target service by name, which the development server understands. From Syntax:
target (push queues)
Optional. A string naming a service/version, a frontend version, or a
backend, on which to execute all of the tasks enqueued onto this
queue.
The string is prepended to the domain name of your app when
constructing the HTTP request for a task. For example, if your app ID
is my-app and you set the target to my-version.my-service, the
URL hostname will be set to
my-version.my-service.my-app.appspot.com.
If target is unspecified, then tasks are invoked on the same version
of the application where they were enqueued. So, if you enqueued a
task from the default application version without specifying a target
on the queue, the task is invoked in the default application version.
Note that if the default application version changes between the time
that the task is enqueued and the time that it executes, then the task
will run in the new default version.
If you are using services along with a dispatch file, your task's
HTTP request might be intercepted and re-routed to another service.
For example a basic queue.yaml would be along these lines:
queue:
- name: service_a
target: service_a
- name: service_b
target: service_b
I'm not 100% certain if this alone is sufficient, personally I'm also using a dispatch.yaml file as I need to route requests other than tasks. But for that you need to have a well-defined pattern in the URLs as host-name based patterns aren't supported in the development server. For example if the Service A requests use /service_a/... paths and Service B use /service_b/... paths then these would do the trick:
dispatch:
- url: "*/service_a/*"
service: service_a
- url: "*/service_b/*"
service: service_b
In your case it might be possible to achieve what you want with just a dispatch file - i.e. still using the default queue. Give it a try.
Related
Instead of generating a build every time I make a change, I want to use the service worker while developing. I've already managed to use the https protocol with a valid certificate, but the service worker doesn't install it. I imagine it is related to the following error "No matching service worker detected. You may need to reload the page, or check that the service worker for the current page also controls the start of the URL from the manifest."
Basically what the title says. The API and client docs state that a retry can be passed to create_task:
retry (Optional[google.api_core.retry.Retry]): A retry object used
to retry requests. If ``None`` is specified, requests will
be retried using a default configuration.
But this simply doesn't work. Passing a Retry instance does nothing and the queue-level settings are still used. For example:
from google.api_core.retry import Retry
from google.cloud.tasks_v2 import CloudTasksClient
client = CloudTasksClient()
retry = Retry(predicate=lambda _: False)
client.create_task('/foo', retry=retry)
This should create a task that is not retry. I've tried all sorts of different configurations and every time it just uses whatever settings are set on the queue.
You can pass a custom predicate to retry on different exceptions. There is no formal indication that this parameter prevents retrying. You may check the Retry page for details.
Google Cloud Support has confirmed that task-level retries are not currently supported. The documentation for this client library is incorrect. A feature request exists here https://issuetracker.google.com/issues/141314105.
Task-level retry parameters are available in the Google App Engine bundled service for task queuing, Task Queues. If your app is on GAE, which I'm guessing it is since your question is tagged with google-app-engine, you could switch from Cloud Tasks to GAE Task Queues.
Of course, if your app relies on something that is exclusive to Cloud Tasks like the beta HTTP endpoints, the bundled service won't work (see the list of new features, and don't worry about the "List Queues command" since you can always see that in the configuration you would use in the bundled service). Barring that, here are some things to consider before switching to Task Queues.
Considerations
Supplier preference - Google seems to be preferring Cloud Tasks. From the push queues migration guide intro: "Cloud Tasks is now the preferred way of working with App Engine push queues"
Lock in - even if your app is on GAE, moving your queue solution to the GAE bundled one increases your "lock in" to GAE hosting (i.e. it makes it even harder for you to leave GAE if you ever want to change where you run your app, because you'll lose your task queue solution and have to deal with that in addition to dealing with new hosting)
Queues by retry - the GAE Task Queues to Cloud Tasks migration guide section Retrying failed tasks suggests creating a dedicated queue for each set of retry parameters, and then enqueuing tasks accordingly. This might be a suitable way to continue using Cloud Tasks
in init() function for GAE's golang app, how can I set-up initial values for my application ?
How to read from Cloud Datastore in the init() function or immediately after applications start-up ? As I understand, server cannot write to the local filesystem and Cloud Datastore is the only option ?
I need some global variables and slices of data..
Using static files
On AppEngine you don't have access to the file system of the host operating system, but you can access files of your web application (you have read-only permission, you can't change them and you can't create new files in the app's folder).
So the question is: can your application's code change the data that you want to read and use for initialization? Or is it fine if it is deployed "statically" with your app's code?
If you don't need to change it (or only when you redeploy your app), easiest is to store it as a "static" file as part of your webapp. You may refer to files of your app using relative paths, where the current or working directory is your app's root. E.g. if your app contains a data folder in its root (where app.yaml resides), and there is an init_values.txt file inside the data folder, you can refer to it with the path: data/init_values.txt.
One important note: not every file is readable by code, this depends on the app configuration. Quoting from Configuring with app.yaml / Static file handlers:
If you have data files that need to be read by the application code, the data files must be application files, and must not be matched by a static file pattern.
Using the Datastore
You can't use AppEngine services that require a Context outside of handlers (because the creation of a Context requires an *http.Request value). This by nature means you can't use them in package init() functions either.
Note that you can use them from cron jobs and tasks added to task queues, because tasks and cron jobs are executed by issuing HTTP GET requests.
You have to restructure your code so that your initialization (e.g. reading from the Datastore) gets called from a handler.
Example of achieving this with Once.Do():
var once = sync.Once{}
func MainHandler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
once.Do(func() { mysetup(ctx) })
// do your regular stuff here
}
func mysetup(ctx appengine.Context) {
// This function is executed only once.
// Read from Datastore and initialize your vars here.
}
"Utilizing" warmup requests
Yes, this may cause first requests to take considerably longer to serve. For this purpose (to avoid this) I recommend you to utilize Warmup requests. A warmup request is issued to a new instance before it goes "live", before it starts serving user requests. In your app.yaml config file you can enable warmup requests by adding -warmup to the inbound_services directive:
inbound_services:
- warmup
This will cause the App Engine infrastructure to first issue a GET request to /_ah/warmup. You can register a handler to this URL and perform initialization tasks. As with any other request, you will have an http.Request in the warmup handler.
But please note that:
..you may encounter loading requests, even if warmup requests are enabled in your app.
Which means that in rare cases it may happen a new instance will not receive a warmup request, so its best to check initialization state in user handlers too.
Related questions:
How do I store the private key of my server in google app engine?
Fetching a URL From the init() func in Go on AppEngine
Environment variables specified on app.yaml but it's not fetching on main.go
I'm trying to set up two Google App Engine modules where one of the modules is configured with Basic scaling so it can handle long-running computation. The front-end module interacts with the user and enqueues tasks.
I need for the front-end module to be able to enqueue a task for the back-end module to pick up the task and execute it. I've gotten it mostly to work except when I enqueue the task, it gets assigned to run in the front-end module rather than the back-end module.
The problem is in the development server environment. On production App Engine it seems clear how to do it by simply stating in the header with the "Host" parameter:
Queue queue = QueueFactory.getDefaultQueue();
TaskOptions taskOptions = TaskOptions.Builder.withUrl("/longtest").param("content", content).header("Host", "nbsocialmetrics-backend");
log.info("SignGuestbookServlet taskOption " + taskOptions);
queue.add(taskOptions);
But in the development server, modules are addressed by port numbers rather than by module name. I don't think using the <target> parameter will work either because it also addresses the module by name rather than by port number.
I've found that the approach with the Host header DOES work on the development server with App Engine SDK version 1.9.15.
Also note this was also posted as an issue on code.google.com: Task queues don't work with multiple modules on development server
Is it possible to start a Deferred frontend task from a Deferred backend on App Engine/Java. Deferred Tasks are started on the backend using a specific host with code like:
queue.add(withPayload(new MyDeferredTask()).header("Host",
BackendServiceFactory.getBackendService().getBackendAddress("backend1", 1));
And this works well. If a Deferred task is started from this backend then the task also runs on the backend. Is there a specific host to be used, or another means of explicitly starting a Deferred task on the frontend?
Update
I missed out a bit of important info in the original question: I'm talking about Deferred Tasks, where a payload is passed in. Starting a Deferred task from a backend starts the new Deferred task in the same backend. What I want to know is if its possible to explicitly start a Deferred task in the frontend, when its started from a backend. The original question above has been modified to reflect this.
To answer my old question, specifically for deferred tasks - if you have a task running on a backend and want to start a task on the frontend, you should explicitly specify the host of the frontend instance i.e. myapp.appspot.com. If you don't specify a host then the task will run on the same instance as the starting code.
To explcitly start on the frontend, regardless of the instance the caller is running on, do something like:
Queue queue = QueueFactory.getQueue("my-queue");
TaskOptions taskOptions = TaskOptions.Builder.withPayload(new MyDeferredTask());
taskOptions.header("Host", "myappid.appspot.com");
queue.add(taskOptions);
Just call the URL of your front-end "servlet you want to run" with the URL-fetch service or add a task to one queue with the servlet's url.
Front-ends just handle all the HTTP calls to your application and send it to the servlet configured in the web.xml file