I want to implement few second sleep (delay) to express delaying processing on mulesoft flow designer.
If the above doesn't work, can mulesoft anypoint platform do it?
client → (request) → server(mulesoft)
..few second sleep..
client ← (response) ← server(mulesoft)
I really don't recommend to use any kind of sleep in variants Mule applications unless you have very clear the impact it may have and how it affects resource usage. I have written a blog article explaining the problems it can cause: https://medium.com/#adobni/sleep-will-make-your-mule-runtime-slow-b7b80303c78
WARNING: this should be done only for very simple tests and never ever be used in a production application, nor for benchmarks and load tests. Failing to follow this warning can cause outages or the case of tests it can cause unrealistic results.
You can use the wait() function in a Transform script to cause an artificial delay.
Note that the bigger the delay is, the easiest to run out of resources to process requests.
Update wait time Based on your Requirement
%dw 2.0
import * from dw::Runtime
output application/json
---
{ "user" : 1 } wait 2000
Related
I wanted to try out integration of Selenium Jmeter and StormRunner. My end goal is to do Load testing with 'n' number of users on StormRunner
What ? - For e.g. I have Selenium Script, convert it in to Jmeter (I can get this information from many sources)
Then my Jmeter script should get ready
Then upload Jmeter script in to StormRunner and pass the necessary parameter through Jenkins and run the load test.
I really want the opinion here about feasibility and whether it is in right direction or not.
Idea here is that Automated Load/Performance test
Selenium is a browser automation framework and JMeter acts on HTTP protocol level so your "Automated" requirement might not be fulfilled especially if your tests are relying on client-side checks like sorting or waiting for element to appear.
Theoretically given you properly configure JMeter it can behave like a real browser, but it still not be executing client-side JavaScript.
If you're fine with this constraint - your approach is valid, if not and the "automated functional test" requirement is the must - consider migrating to TruClient Protocol instead
Why wouldn't you covert your script to a native Loadrunner/Stormrunner form of virtual user?
You should look at the value of what you are trying to achieve. The end value of a performance test is in analysis. Analysis simply takes the timing records and the resource measurements produced during the test, bringing them together on a common timestamp, and then allowing you to analyze what resource "X" is being impinged when timing record "Y" is too long. This then points to some configuration or code which locks up on resource, "X."
What is your path to value in your model? You speak about converting a functional test script to a performance one. Realistically, you should already know that your code, "works for one," before you get to asking, "Does it work for many?" There is a change in script definitions which typically accompanies this understanding.
Where are your collection of resources noted? Which Resources? On which Hosts? This is on the "path to value" problem where you need to have the resource measurements to diagnose root cause of poor performance.
I am doing a project in apache flink where I need to call multiple APIs so as to achieve my goal. The result of each API is required for the next API to work. Also as I am doing it on a KeyedStream, the same flow will be applicable to multiple data at once.
Below dig. can explain the scenario
/------API1---API2----
KeyedStream ----|------API1---API2----
\------API1---API2----
As I am doing all this, I am getting an exception saying "Buffer pool destroyed" after the job runs for sometime. Is it something related to API call, do I need to make use of Asynchronous function?? Please suggest. Thanks in advance.
a few things that are typically needed to help answer questions about Flink...
What version are you running?
How are you running it (from IDE, YARN cluster, stand-alone, etc)?
What's the complete stack trace for the exception?
(often) Can you share your code?
But at a high level, the "buffer pool destroyed" message you mentioned is not the root cause of failover, it's just a byproduct of Flink trying to kill off the workflow after an error has happened. So you need to dig deeper in the logs (typically Task Manager logs are where you'd look first).
Google Cloud has a powerful tracing tool for analyzing latency of requests and RPCs. But it seems to just pick some requests that it finds deserving of traces. Sometimes that's good enough, you can just browse through existing traces. But if you are working on a performance enhancement, you want the trace on your particular query right now, you don't want to wait until it is deemed interesting.
Questions are
What rules intervene in deciding which queries are traced ?
Is there a way to ask for traces to be captured for a given URI ?
Either from within developer console, or by calling some API from within our application ? Or through some app.yaml configuration ? Or do we have to just wait and pray for the great algorithm to chose our request ?
You can force tracing of a HTTP request by setting the cloud trace context header properly:
$ curl -H "X-Cloud-Trace-Context: 01234567890123456789012345678901;o=1" http://<your-app>.appspot.com/<path>
01234567890123456789012345678901 (32 hex characters) is the trace id. You want to use a different one each time.
o=1 enables tracing.
Use the following URL to view the trace (last part is the trace id):
http://console.developer.google.com/traces/details/01234567890123456789012345678901
Since you're interested in particular request, why don't you use appstats?
https://cloud.google.com/appengine/docs/python/tools/appstats?hl=en you can make performance enchancement, turn appstats on and deploy it to different version and have some control from appengine_config.py
I use cloud trace to get aggregate analysis, to get into more detail per request basis, I always use appstats as it contains more information.
What rules intervene in deciding which queries are traced ?
Currently there is a sampling rate that guides which request are traced. The requests are sampled at a small number of requests per second per instance.
Is there a way to ask for traces to be captured for a given URI ?
The following could possibly help depending on your scenario.
You could add a Trace Context to force the request to force tracing on the request. Trace Context is essentially a HTTP header (X-Cloud-Trace-Context)
Here is an pointer to help inject a Trace Context: https://github.com/liqianluo/gcloud-trace-java/blob/master/cloud-trace-sdk-java-core/src/main/java/com/google/cloud/trace/sdk/TraceContext.java
To my understanding, we should do database calls async since it will block IOLoop, and I also found some library/tools that working on this.
But recently I just found lots of posts about this question on stackoverflow
Tornado Asynchronous Handler
How does async work in Tornado?
and something from wiki https://github.com/tornadoweb/tornado/wiki/Threading-and-concurrency
Do it synchronously and block the IOLoop. This is most appropriate for
things like memcache and database queries that are under your control
and should always be fast. If it's not fast, make it fast by adding
the appropriate indexes to the database, etc.
So should we do database calls async?
and a related question: as I know, there are two methods to do database calls async
For sync database(mysqldb), we can
executor = ThreadPoolExecutor(4)
result = yield executor.submit(mysqldb_operation)
For async database(motor), we can use it directly.
what's the different of them based on tornado?
I had a similar doubt and went through some research and digging and finally have some grip over this.
Short Answer - Yes, Do it Asynchronously.
Long Answer :
Database calls are generally blocking calls. Tornado is best suited as a framework when we do async/Non-blocking I/O calls to completely utilize its full potential.
Now, You can do Database Calls in Tornado in the following ways, I am giving the How and Why for each method.
Method 1 : Synchronous Database Calls - Start a number of Tornado instances behind a load balancer like for example nginx, and make synchronous database calls(as mentioned in the wiki). In this case, the process with the particular request making the blocking database call will block. So nginx will load balance the other requests by directing them to the other running tornado instances to handle the requests.
NOTE : That wiki is old, and hasn't been updated so that method mentioned is prior to the addition of coroutines in the newer versions of Tornado as mentioned by Ben Darnell, one of the lead developers of Tornado.
Update : The wiki has been updated by Ben recently and you can check it now.
Method 2 : By using multiprocessing or multithreading, concurrent.futures, similar to what you have mentioned:
For sync database(mysqldb), we can
executor = ThreadPoolExecutor(4)
result = yield executor.submit(mysqldb_operation)
Method 3 : By using Async libraries, like for instance if you are using MySql, then you can use libraries like TorMySql, Tornado-MySQL, other links given here-Tornado 3rd-party libraries, go to the MySql section. These libraries are to be used with Tornado's async decorators, tornado.gen.coroutine or tornado.web.asynchronous. Similarly, you need to use other async libraries for different databases.
NOTE : You cannot use blocking libraries otherwise its all the same as Method 1.
CONCLUSION : Method 3 is what I mentioned as my Short Answer. Method 2 is the next best solution, followed by Method 1. This is in accordance with your performance requirements. If you requirement is for not-so-heavy(few to moderate) request processing per second or per minute, than going for the easy and general Method 1 will suffice.
I had converted some tasks to run on a dynamic backend.
The tasks are failing silently [no logged error, no retry, nothing] ~20% of the time (min:10%, max:60%, sample:large, long term). Switching the task away from the backend restores retries and gets the failure rate back to ~0%.
Any ideas?
Converting it to a backend exacerbated the problem but wasn't the problem.
I had specified a task_retry_limit and the queue was a push queue. With a backend the number of instances is specified. (I believe you can replicate this issue on the frontend by ramping up requests rapidly, to a big number).
Tasks were failing 503: Instance Unavailable until they hit the task_retry_limit. This is visible temporarily in Task Queues, but will not show up in Logs.
I should be using pull queues. Even if my use case was stupid I'd probably +1 a task dying due to multiple 503: Instance Unavailable logging something so it doesn't appear like a phantom task.
Which runtime are you using on the backend?
Try running the backend for a bit without dynamic set to true and exercise the failing component.
On my project, I have seen tasks that target a static backend disappear on occasion, but no where near the rate you are seeing.