I can't get my Cloud Tasks queue to stop overriding the app engine service routing. I created the queue through the GCP dashboard, not using queue.yml or anything else.
Description of the queue:
$ gcloud tasks queues describe import
appEngineRoutingOverride:
host: my-project-id.uc.r.appspot.com
name: projects/my-project-id/locations/us-central1/queues/import
purgeTime: '2021-03-31T00:45:22.941195Z'
rateLimits:
maxBurstSize: 100
maxConcurrentDispatches: 1000
maxDispatchesPerSecond: 500.0
retryConfig:
maxAttempts: 100
maxBackoff: 3600s
maxDoublings: 16
minBackoff: 0.100s
state: PAUSED
Attempting to clear the routing override flag with --clear-routing-override appears to be successful but does not actually clear the flag
$ gcloud tasks queues update import --clear-routing-override
WARNING: You are managing queues with gcloud, do not use queue.yaml or queue.xml in the future. More details at: https://cloud.google.com/tasks/docs/queue-yaml.
Updated queue [import].
$ gcloud tasks queues describe import
appEngineRoutingOverride:
host: my-project-id.uc.r.appspot.com
name: projects/my-project-id/locations/us-central1/queues/import
purgeTime: '2021-03-31T01:15:16.727288Z'
rateLimits:
maxBurstSize: 20
maxConcurrentDispatches: 1000
maxDispatchesPerSecond: 100.0
retryConfig:
maxAttempts: 100
maxBackoff: 3600s
maxDoublings: 16
minBackoff: 0.100s
state: PAUSED
If I change the routing to another service with --routing-override the change is successful, but that means I can only have tasks pointing to one specific service in this queue. Additionally, if I change the routing, then click on Edit Queue on the Queue details panel the app engine routing override fields will be blank, not populated with the service I specified earlier. If I click save the appEngineRoutingOverride flag will reset to the default service host. Even if I change the override options through the Edit Queue panel, refresh, then open it back up the service will still not be populated, meaning if I change the service override once and somebody on my team needs to change another setting such as rate limits then the routing will be reset and things will break.
imgur vid to clearly demonstrate the issue
EDIT: This has been reported on the GCP issue tracker a few times
https://issuetracker.google.com/issues/179701687
https://issuetracker.google.com/issues/177651361
https://issuetracker.google.com/issues/176221773
Did some digging, DO NOT use the UI on could.google.com to create or update queues.
New ones will always have a host pointing to defaults.
And updated ones will get a host pointing to default.. and then you have silently messed up your queue.
Use the cli command to create and update queues
gcloud tasks queues update mey-new-queue --max-dispatches-per-second=10 --max-concurrent-dispatches=3
adjust the flags to what you need. This command works as an upsert
(just realized how I messed up my queue, I accidentally updated it through the web UI)
Related
Basically what the title says. The API and client docs state that a retry can be passed to create_task:
retry (Optional[google.api_core.retry.Retry]): A retry object used
to retry requests. If ``None`` is specified, requests will
be retried using a default configuration.
But this simply doesn't work. Passing a Retry instance does nothing and the queue-level settings are still used. For example:
from google.api_core.retry import Retry
from google.cloud.tasks_v2 import CloudTasksClient
client = CloudTasksClient()
retry = Retry(predicate=lambda _: False)
client.create_task('/foo', retry=retry)
This should create a task that is not retry. I've tried all sorts of different configurations and every time it just uses whatever settings are set on the queue.
You can pass a custom predicate to retry on different exceptions. There is no formal indication that this parameter prevents retrying. You may check the Retry page for details.
Google Cloud Support has confirmed that task-level retries are not currently supported. The documentation for this client library is incorrect. A feature request exists here https://issuetracker.google.com/issues/141314105.
Task-level retry parameters are available in the Google App Engine bundled service for task queuing, Task Queues. If your app is on GAE, which I'm guessing it is since your question is tagged with google-app-engine, you could switch from Cloud Tasks to GAE Task Queues.
Of course, if your app relies on something that is exclusive to Cloud Tasks like the beta HTTP endpoints, the bundled service won't work (see the list of new features, and don't worry about the "List Queues command" since you can always see that in the configuration you would use in the bundled service). Barring that, here are some things to consider before switching to Task Queues.
Considerations
Supplier preference - Google seems to be preferring Cloud Tasks. From the push queues migration guide intro: "Cloud Tasks is now the preferred way of working with App Engine push queues"
Lock in - even if your app is on GAE, moving your queue solution to the GAE bundled one increases your "lock in" to GAE hosting (i.e. it makes it even harder for you to leave GAE if you ever want to change where you run your app, because you'll lose your task queue solution and have to deal with that in addition to dealing with new hosting)
Queues by retry - the GAE Task Queues to Cloud Tasks migration guide section Retrying failed tasks suggests creating a dedicated queue for each set of retry parameters, and then enqueuing tasks accordingly. This might be a suitable way to continue using Cloud Tasks
I want to send a particular HTTP request (or otherwise communicate a message) to every (dynamic/autoscaled) instance which is currently running for a particular App Engine application.
My goal is to trigger each instance to discard some locally cached data (because I have just modified the underlying data and want them to reload it).
One possible solution is to store a value in Memcache, and have instances check this each time they handle a request to see if they should flush their cache. But this adds latency to every request.
Another possible solution would be to somehow stop all running instances. No fixed overhead, but some impact while instances are restarted.
An even less desirable solution would be to redeploy the application code in order to cause all instances to be stopped. This now adds additional delay on my end as a deployment takes some time.
You could use the management API to list instances for a given version, but I'd suggest that you'd probably want to use something like the PubSub API to create a subscription on each of your App Engine instances. Since each instance has its own subscription, any messages sent to the monitored queue will be received by all instances.
You can create the subscription at startup (the /_ah/start endpoint may be useful), and then delete it at shutdown (using the /_ah/stop endpoint).
It looks like I can create a push-queue that will start backends to process tasks and that I can limit the number of workers to 1. However, is there a way to do this with a pull-queue?
Can App-Engine auto-start a named backend when a pull-queue has tasks and then let it expire when idle and the queue is empty?
It looks like I just need some way to call an arbitrary URL to "notify" it that there are tasks to process but I'm unable to find any documentation on how this can be done.
Use a cron task or a push queue to periodically start the backend. The backend can loop through the tasks (if any) in the pull queue, and then expire.
There isn't a notification system for pull queues, only inspection through queue statistics and empty/non-empty lease results.
First of all you need to decide scalability type you want to use for your module. I think that you should take a look to Basic Scaling (https://developers.google.com/appengine/docs/java/modules/)
Next, to process tasks from pull queue you can use Cron to check queues every several minutes. It will be important to request not basic scaling module, but frontend module, cause cron will start instances. The problem is that you will still need to pay for at least one instance of frontend cause your cron job will not allow it to shutdown.
So the solution could be the following:
Start cron every 1 or 5 minutes and call frontend
Check queue in frontend and issue URLFetch request to basic scaling module if there are tasks in pull queue
Process tasks in queue using basic scaling module
If you use F1 instances for frontend and b2 or greate for other modules it could save you some money.
I am unable to update my frontends nor my backends. I get the error message 'Version is not ready'. This bug has persisted for coming up to 24 hours now. I have a task perpetually running in a queue. My best guess is that this task is stopping the update. I am unable to delete the task as it is perpetually running, nor can I delete the queue as I am unable to upload a new queue.yaml definition. The same task previously failed due to a maximum recursion error as I had a synchronous RPC within an asynchronous tasklet.
I'm pretty sure the fix will require someone from the GAE side forcibly resetting the task queue. Thus, this question would be more suitably directed to the GAE team with details about my app in a less public forum. Though, from what I can see, they do not allow direct support questions and suggest posting the question here. My follow up question, then, is when you have a GAE issue that requires action from the GAE team - how do you get hold of them (other than paying US$500/month for a premium support account)?
EDIT:
The task is/was meant to be running on a backend instance. I intended to shutdown all backend and frontend instances via the console assuming that they would cancel the task and restart themselves. But I found that only one frontend instance was running - no backends. After shutting down that frontend instance, the dashboard has reported that I have 0 instances running, yet the website is still serving and the task remains perpetually running.
EDIT:
Disabling the app stopped the task from running. After reenabling the app, I was able to update it. Though I am left with a ghost task in my queue.
If you have a stuck task queue job, I'd try disabling the queue and killing the instance running that job. If that doesn't work, I'd try disabling the app temporarily.
I have configured a sticky session set up with a load balancer (Apache) and three app nodes running Jboss 4.2.2 .
the load balancer uses mod_jk and settings as mentioned in the tutorial here.
http://community.jboss.org/wiki/UsingModjk12WithJBoss;jsessionid=1569CBFB7C3096C59C977CD3F7159A32
I have the jumRoute set as node1 ,node2 and node3 for the three nodes and my workerlist property for load balancer is set as
node1,node2,node3
The tutorial has been followed till the last point but I did not configure the useJK parameters under.the value is still set to false.
The sticky sessions are holding up but I seem to loose session and get this error in my mod_jk log file
[error] ajp_get_reply::jk_ajp_common.c (1926): (node1) Timeout with waiting reply from tomcat. Tomcat is down, stopped or network problems (errno=110)
I personally checked the user logged in on node1 and then moved to node2.
Does Apache redirect to another node when it fails to get reply from node1, How does useJk help in this situation.
---edit 01---
I changed the UseJK value to true but still few users still experience sudden log out which I know due to change in the server node catering the users request.
I also wanted to know whether traffic on the nodes has any affect on sticky session and how to counter it.( I am experiencing high load on all the servers for a few days)
----edit 02 ----
I would also like to know about controlling the number of connections per worker.
controlling the number of ajp connector/connections.
relation between the number of connections of apache load balancer and number of
ajp connections in JBoss worker nodes.
what would be the best configuration between Apache 2.2.3 and JBoss 4.2.2 worker nodes with Tomcat 5.5 connectors.
---- edit03-----
http://community.jboss.org/wiki/OptimalModjk12Configuration
using the above article i just wanted to know the best values for Apache
MaxClients
ThreadPerChild
I found the following note in this article interesting. I haven't tried this, but perhaps could be useful for someone experiencing the same problem.
If you are using mod_jk and have turned sticky sessions on, but your sessions are failing to stick, you have probably failed to set the domain, or you have failed to set the jvmRoute, or you are using a non-standard cookie name to implement the stickyness!
I think in your worker.properties file the workerlist should have loadBalancer worker not the node1,node2 & node3.It should be like this
worker.list=loadmanager
worker.loadmanager.balance_workers=node1,node2,node3
I hope u must have these correct.
Also you have to set UserJK arttribute to set as true for load balancing with sticky session combined with JvmRoute. If set to true, it will insert a JvmRouteFilter
to intercept every request and replace the JvmRoute if it detects a failover.
<attribute name="UseJK">true</attribute>
in deploy/jboss-web.deployer/META-INF/jboss-service.xml