I want to set up a push queue with max_concurrent_requests set to 1. So I created a queue.yaml like this:
queue:
- name: myqueue
max_concurrent_requests: 1
When running in the dev server, I get the error:
root: WARNING: Refill rate must be specified for push-based queue. Please check queue.yaml file.
Doing a Google search for "refill rate" and queue.yaml doesn't give any relevant hits except for the taskqueue stub, which doesn't help me.
Changing queue.yaml to this:
queue:
- name: myqueue
max_concurrent_requests: 1
rate: 10/s
Gets rid of the error in the dev server. Can anyone confirm that this will actually create a queue with a max of 1 concurrent request? (ok, that it is also limited to 10 per second) I'm suspicious because the queue.yaml documentation doesn't address this.
Although not specified in the documentation, you must specify a "rate" when creating a queue. To achieve 1 maximum concurrent request, you can simply set a high rate, and the rate will essentially be ignored. My tasks take about 0.25 seconds (i.e., 4/s) so a rate of 10/s makes sure that the rate does not impact task execution.
Related
I am using Pysolr to add data to solr. I add 100 documents at once.But i am getting the below error.
Solr responded with an error (HTTP 500): [Reason: Task queue processing has stalled for 20121 ms with 0 remaining elements to process.]
is solr has queue internally and is it filled due to high number of hits? Can I increase the size of the queue(I mean limit)
It sounds like you can control this time via the solr.cloud.client.stallTime system property.
Reference: https://issues.apache.org/jira/browse/SOLR-13975?focusedCommentId=16988216&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16988216
I'm using sharded counters (https://cloud.google.com/appengine/articles/sharding_counters) in my GAE application for performance reasons, but I'm having some trouble understanding why it's so slow and how I can speed things up.
Background
I have an API that grabs a set of 20 objects at a time and for each object, it gets a total from a counter to include in the response.
Metrics
With Appstats turned on and a clear cache, I notice that getting the totals for 20 counters makes 120 RPCs by datastore_v3.Get which takes 2500ms.
Thoughts
This seems like quite a lot of RPC calls and quite a bit of time for reading just 20 counters. I assumed this would be faster and maybe that's where I'm wrong. Is it supposed to be faster than this?
Further Inspection
I dug into the stats a bit more, looking at these two lines in the get_count method:
all_keys = GeneralCounterShardConfig.all_keys(name)
for counter in ndb.get_multi(all_keys):
If I comment out the get_multi line, I see that there are 20 RPC calls by datastore_v3.Get totaling 185ms.
As expected, this leaves get_multi to be the culprit for 100 RPC calls by datastore_v3. Get taking upwards of 2500 ms. I verified this, but this is where I'm confused. Why does calling get_multi with 20 keys cause 100 RPC calls?
Update #1
I checked out Traces in the GAE console and saw some additional information. They show a breakdown of the RPC calls there as well - but in the sights they say to "Batch the gets to reduce the number of remote procedure calls." Not sure how to do that outside of using get_multi. Thought that did the job. Any advice here?
Update #2
Here are some screen shots that show the stats I'm looking at. The first one is my base line - the function without any counter operations. The second one is after a call to get_count for just one counter. This shows a difference of 6 datastore_v3.Get RPCs.
Base Line
After Calling get_count On One Counter
Update #3
Based on Patrick's request, I'm adding a screenshot of info from the console Trace tool.
Try splitting up the for loop that goes through each item and the actual get_multi call itself. So something like:
all_values = ndb.get_multi(all_keys)
for counter in all_values:
# Insert amazeballs codes here
I have a feeling it's one of these:
The generator pattern (yield from for loop) is causing something funky with get_multi execution paths
Perhaps the number of items you are expecting doesn't match actual result counts, which could reveal a problem with GeneralCounterShardConfig.all_keys(name)
The number of shards is set too high. I've realized that anything over 10 shards causes performance issues.
When I've dug into similar issues, one thing I've learned is that get_multi can cause multiple RPCs to be sent from your application. It looks like the default in the SDK is set to 1000 keys per get, but the batch size I've observed in production apps is much smaller: something more like 10 (going from memory).
I suspect the reason it does this is that at some batch size, it actually is better to use multiple RPCs: there is more RPC overhead for your app, but there is more Datastore parallelism. In other words: this is still probably the best way to read a lot of datastore objects.
However, if you don't need to read the absolute most current value, you can try setting the db.EVENTUAL_CONSISTENCY option, but that seems to only be available in the older db library and not in ndb. (Although it also appears to be available via the Cloud Datastore API).
Details
If you look at the Python code in the App Engine SDK, specifically the file google/appengine/datastore/datastore_rpc.py, you will see the following lines:
max_count = (Configuration.max_get_keys(config, self.__config) or
self.MAX_GET_KEYS)
...
if is_read_current and txn is None:
max_egs_per_rpc = self.__get_max_entity_groups_per_rpc(config)
else:
max_egs_per_rpc = None
...
pbsgen = self._generate_pb_lists(indexed_keys_by_entity_group,
base_req.ByteSize(), max_count,
max_egs_per_rpc, config)
rpcs = []
for pbs, indexes in pbsgen:
rpcs.append(make_get_call(base_req, pbs,
self.__create_result_index_pairs(indexes)))
My understanding of this:
Set max_count from the configuration object, or 1000 as a default
If the request must read the current value, set max_gcs_per_rpc from the configuration, or 10 as a default
Split the input keys into individual RPCs, using both max_count and max_gcs_per_rpc as limits.
So, this is being done by the Python Datastore library.
Curious if anyone has any ideas on how I might create a jmeter script that will loop the scenario while increasing throughput and load until an error is received.
Any guidance would be appreciated.
You can go the following way:
Set your test to run "Forever" on Thread Group level
Set "Action to be taken on a Sampler error" to "Stop Test"
Add a Constant Throughput Timer to your test plan with very low initial value like 60 requests per minute (= 1 request per second)
Despite "Constant" word in its name, Constant Throughput Timer value can be changed on-the-fly see official documentation for example.
I am using Java mapreduce module of appengine
I get the following info message
Out of mapper quota. Aborting request until quota is replenished. Consider increasing mapreduce.mapper.inputprocessingrate (default 1000) if you would like your mapper job to complete faster.
Task parameters.
queue name = default
rate = 1/s
bucketsize = 1
I have about 2000 entities of the KIND, and I am just doing the logging in the map() call
What mapreduce/task parameters needs to be provided to get rid of that info message.
-Aswath
I believe that this is a special quota in mapreduce implemented by the framework itself. It's designed to limit the speed that it can consume resources, so that
mapreduce does not run through available app engine quota too quickly. It looks like it denotes the maximum overall rate of map() calls/second.
Try increasing the mapreduce.mapper.inputprocessingrate property in the configuration for your map job.
Or, just to test, you can change the default, defined in mapreduce/AppEngineJobContext.java:
public static final int DEFAULT_MAP_INPUT_PROCESSING_RATE = 1000;
I want to do some heavy processing in the map() call of the mapper.
I was going through the source file MapReduceServlet.java:
// Amount of time to spend on actual map() calls per task execution.
public static final int PROCESSING_TIME_PER_TASK_MS = 10000;
Does it mean, the map call can last only for 10secs. What happens after 10sec?
Can I increase this to large number like 1min or 10min.
-Aswath
MapReduce operations are executed in tasks using Push Queues, and as said in the documentation the task deadline is currently 10 minutes (limit after which you will get a DeadlineExceededException).
If the task failed to execute, by default App Engine retries it until it succeed. If you need longer deadline that 10 minutes, you can use Backend for executing your tasks.
Looking at the actual usage of PROCESSING_TIME_PER_TASK_MS in Worker.java, this value is used to limit the number of map call done in a single task.
After each map call has been executed if more than 10s has elapsed since the beginning of the task, it will spawn a new task to handle the rest of the map calls.
Worker.scheduleWorker spawns a new Task for a each given shard
Each task will call Worker.processMapper
processMapper execute 1 map call
if less than PROCESSING_TIME_PER_TASK_MS have elapsed since 2. go back to 3.
else if processing is not finished reschedule a new worker task
In the worst case scenario the default task request deadline (10 minutes) should apply to each of your individual map call.