Scenario:
After login into the application, system check number of pending files and if any pending file present then get assign to the user. The maximum files that can be assigned to user is two.
Once user process the first files, again system check for the pending files and if any pending file present then get assign to the user.
For this I used Loop Controller, but it not working.
Thread Group
HTTP Request - Login
JDBC Connection Configuration
JDBC Request
JSR223 PostProcessor
List<String> fileIDListresultSet = vars.getObject("File_ID")
vars.put("fileIDListSize", String.valueOf(fileIDListresultSet.size()))
Loop Controller
${fileIDListSize}
HTTP Request - 1 Lock File
JSR223 PreProcessor
def counterVal = vars.get("fileIDCounter") as int
def fileIDListresultSet = vars.getObject("File_ID").get(counterVal).get("FileId")
vars.put("fileId",fileIDListresultSet.toString())
HTTP Request - 2 Process File
JDBC Request
JSR223 PostProcessor
List<String> fileIDListresultSet = vars.getObject("File_ID")
vars.put("fileIDListSize", String.valueOf(fileIDListresultSet.size()))
Counter
From the first glance your Counter placement is incorrect, it needs to be a child of the Loop Controller, see JMeter Scoping Rules - The Ultimate Guide article for more details
You don't even need the counter as the Loop Controller exposes a special variable holding current iteration number - ${__jm__Loop Controller__idx}
In any case we don't have any possibility to help you unless you show the values of these File_ID, fileIDListSize, fileIDCounter, FileId and fileId for each iteration (can be obtained using Debug Sampler) along with jmeter.log file contents
Related
Currently, I have 2 cases in the following detail
I run an API with a file upload, after API is run successfully. There are 2 statuses it can be displayed
status = waiting
status = ready_to_process (this is the correct status if the system not have a problem)
NOTE: The status change like that is cause be system sometime can't change at the time file is uploaded, but it can be recorded to DB and run successfully
And then in the Database, data change is also different
Changed to the role: unknow
Changed to the role: pass
How can I write steps if..else in Cucumber use MAF framework like:
If API run then it returns status 1 => verify the result 1 in DB
Else
verify result 2 in DB
You can't write conditional code in Cucumber features. What you should be doing is writing a scenario for each condition. So you should write something like
Scenario: Run ends with waiting status
...
Scenario: Run ends with ready to process status
...
I'm trying to schedule a PySpark Job. I followed the GCP documentation and ended up deploying a little python script to App Engine which does the following :
authenticate using a service account
submit a job to a cluster
The problem is, I need the cluster to be up and running otherwise the job won't be sent (duh !) but I don't want the cluster to always be up and running, especially since my job needs to run once a month.
I wanted to add the creation of a cluster in my python script but the call is asynchronous (it makes an HTTP request) and thus my job is submitted after the cluster creation call but before the cluster is really up and running.
How could I do ?
I'd like something cleaner than just waiting for a few minutes in my script !
Thanks
EDIT : Here's what my code looks like so far :
To launch the job
class EnqueueTaskHandler(webapp2.RequestHandler):
def get(self):
task = taskqueue.add(
url='/run',
target='worker')
self.response.write(
'Task {} enqueued, ETA {}.'.format(task.name, task.eta))
app = webapp2.WSGIApplication([('/launch', EnqueueTaskHandler)], debug=True)
The job
class CronEventHandler(webapp2.RequestHandler):
def create_cluster(self, dataproc, project, zone, region, cluster_name):
zone_uri = 'https://www.googleapis.com/compute/v1/projects/{}/zones/{}'.format(project, zone)
cluster_data = {...}
dataproc.projects().regions().clusters().create(
projectId=project,
region=region,
body=cluster_data).execute()
def wait_for_cluster(self, dataproc, project, region, clustername):
print('Waiting for cluster to run...')
while True:
result = dataproc.projects().regions().clusters().get(
projectId=project,
region=region,
clusterName=clustername).execute()
# Handle exceptions
if result['status']['state'] != 'RUNNING':
time.sleep(60)
else:
return result
def wait_for_job(self, dataproc, project, region, job_id):
print('Waiting for job to finish...')
while True:
result = dataproc.projects().regions().jobs().get(
projectId=project,
region=region,
jobId=job_id).execute()
# Handle exceptions
print(result['status']['state'])
if result['status']['state'] == 'ERROR' or result['status']['state'] == 'DONE':
return result
else:
time.sleep(60)
def submit_job(self, dataproc, project, region, clusterName):
job = {...}
result = dataproc.projects().regions().jobs().submit(projectId=project,region=region,body=job).execute()
return result['reference']['jobId']
def post(self):
dataproc = googleapiclient.discovery.build('dataproc', 'v1')
project = '...'
region = "..."
zone = "..."
clusterName = '...'
self.create_cluster(dataproc, project, zone, region, clusterName)
self.wait_for_cluster(dataproc, project, region, clusterName)
job_id = self.submit_job(dataproc,project,region,clusterName)
self.wait_for_job(dataproc,project,region,job_id)
dataproc.projects().regions().clusters().delete(projectId=project, region=region, clusterName=clusterName).execute()
self.response.write("JOB SENT")
app = webapp2.WSGIApplication([('/run', CronEventHandler)], debug=True)
Everything works until the deletion of the cluster. At this point I get a "DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded." Any idea ?
In addition to general polling either through list or get requests on the Cluster or the Operation returned with the CreateCluster request, for single-use clusters like this you can also consider using the Dataproc Workflows API and possibly its InstantiateInline interface if you don't want to use full-fledged workflow templates; in this API you use a single request to specify cluster settings along with jobs to submit, and the jobs will automatically run as soon as the cluster is ready to take it, after which the cluster will be deleted automatically.
You can use the Google Cloud Dataproc API to create, delete and list clusters.
The list operation can be (repeatedly) performed after create and delete operations to confirm that they completed successfully, since it provides the ClusterStatus of the clusters in the results with the relevant State information:
UNKNOWN The cluster state is unknown.
CREATING The cluster is being created and set up. It is not ready for use.
RUNNING The cluster is currently running and healthy. It is ready for use.
ERROR The cluster encountered an error. It is not ready for use.
DELETING The cluster is being deleted. It cannot be used.
UPDATING The cluster is being updated. It continues to accept and process jobs.
To prevent plain waiting between the (repeated) list invocations (in general not a good thing to do on GAE) you can enqueue delayed tasks in a push task queue (with the relevant context information) allowing you to perform such list operations at a later time. For example, in python, see taskqueue.add():
countdown -- Time in seconds into the future that this task should run or be leased. Defaults to zero. Do not specify this argument if
you specified an eta.
eta -- A datetime.datetime that specifies the absolute earliest time at which the task should run. You cannot specify this argument if
the countdown argument is specified. This argument can be time
zone-aware or time zone-naive, or set to a time in the past. If the
argument is set to None, the default value is now. For pull tasks, no
worker can lease the task before the time indicated by the eta
argument.
If at the task execution time the result indicates the operation of interest is still in progress simply enqueue another such delayed task - effectively polling but without an actual wait/sleep.
I'm using the following code in Gatling:
.exec(ws("Open WS connection")
.open("/${session_id}/socket?device=other"))
.pause(2)
.exec(ws("Get client browser id")
.sendText("[]")
.check(wsListen.within(10).until(1).jsonPath("$.[2]").saveAs("clientID")))
It does not report any failure. I assume it means that the JSON value was stored in the clientID variable successfully.
When I add
.exec{
session =>
println("clientID: " + session("clientID").as[String])
session
}
I get error
[ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-1' crashed with 'java.util.NoSuchElementException: key not found: clientID', forwarding to the next one
This call works in JMeter.
Please help.
I guess you have to reconciliate ws branch and main branch:
https://gatling.io/docs/2.3/http/websocket/#reconciliate
As stated in the ref doc:
As a consequence, main HTTP branch and a WebSocket branch can exist in a Gatling scenario in a dissociated way, in parallel. When doing so, each flow branch has it’s own state, so a user might have to reconcile them, for example when capturing data from a WebSocket check and wanting this data to be available to the HTTP branch.
I am using the deferred task queues library with GAE. Every day I need to send a piece of text to all users connected to a certain page in my app. My app has multiple pages connected, so for each page, I want to go over all users, and send them a daily message. I am using a cursor to iterate over the table of Users in batches of 800. If there are more than 800 users, I want to remember where the cursor left off, and start another task with the other users.
I just want to make sure that with my algorithm I am going to send all users only one message. I want to make sure I won't miss any users, and that no user will receive the same message twice.
Does this look like the proper algorithm to handle my situation?
def send_news(page_cursor=None, page_batch_size=1,
user_cursor=None, user_batch_size=800):
p_query = PageProfile.query(PageProfile.subscribed==True)
all_pages, next_page_cursor, page_more = p_query.fetch_page(page_batch_size,
start_cursor=page_cursor)
for page in all_pages:
if page.page_news_url and page.subscribed:
query = User.query(User.subscribed==True, User.page_id == page.page_id)
all_users, next_user_cursor, user_more = query.fetch_page(user_batch_size, start_cursor=user_cursor)
for user in all_users:
user.sendNews()
# If there are more users on this page, remember the cursor
# and get the next 800 users on this same page
if user_more:
deferred.defer(send_news, page_cursor=page_cursor, user_cursor=next_user_cursor)
# If there are more pages left, use another deferred queue to
# send the daily news to users in that page
if page_more:
deferred.defer(send_news, page_cursor=next_page_cursor)
return "OK"
You could wrap your user.sendNews() into another deferred task with specific name which will ensure that it's created only once.
interval = int(time.time()) / (60 * 60 * 24)
args = ('positional_arguments_for_object')
kwargs = {'param': 'value'}
task_name = '_'.join([
'user_name',
'page_name'
str(interval_num)
])
# with interval presented in the name we are sure that the task name for the same page and same user will stay same for 24 hours
try:
deferred.defer(obj, _name=task_name, _queue='my-queue', _url='/_ah/queue/deferred', *args, **kwargs)
except (taskqueue.TaskAlreadyExistsError):
pass
# task with such name already exists, likely wasn't executed yet
except (taskqueue.TombstonedTaskError)
pass
# task with such name was created not long time ago and this name isn't available to use
# this should reset once a week or so
Note that as far as I remember App Engine does not guarantee that the task will be executed only once, in some edge cases it could be executed twice or more times and ideally they should be idempotent. If such edge cases are important for you – you could transactionally read/write some flag in the datastore for each task, and before executing the task you check if that entity is there to cancel the execution.
I am writing an apache module output filter that needs to consume a couple of internal-only response headers. These response headers are set by a perl based application running in the backend. The APR function I am using in my output filter is:
apr_table_get(r->headers_out, "x-my-response-header");
However, what seems to happen is that in my output filter I do not see the above response header set, up until the third or fourth bucket brigade - which is unfortunately already too late - I actually need to use the value of x-my-response-header to compute a new response header and set that in the response to the browser.
I insert my output filter this way:
ap_hook_insert_filter(insertOutputFilterHook, NULL, NULL, APR_HOOK_FIRST);
ap_register_output_filter(myFiltersName, myOutputFilter, NULL, AP_FTYPE_CONTENT_SET);
What I have verified:
The internal-only headers do appear on the HTTP response on my browser (haven't unset them yet)
The first two bucket brigade's buckets contain html page text
Questions:
What could be the reasons for why the internal-only response header not be set/visible in the first call to my output filter / first bucket brigade?
Is it possible to instead accumulate the first few bucket brigades and then start flushing them out once the internal only response header's value known?