I am using Jmeter to test API's. I am often using queries to access the DB (JCDB connection)
So far so good. However as i am using more and more queries it seems i am copying data.
For instance:
Thread 1:
HTTP request 1
Query A
Query B
Query C
Thead 2:
HTTP request 2
Query D
Query A
Thead 3:
HTTP request 3
Query A
Query C
As you can see. I have the same query duplicate it often. Not only on 1 jmx file but i have a lot of jmx files where i use Query API
So i am looking for a way to have to write query A once. I would think to create a new jmx file and just include the jmx file and call to that. Is this a good way to appraoch this? Also how do i call Query A from any thread? I would need to pass (and return) parameters.
Help would be appreciated
It appears you're looking for the Module Controller, you can define a "module" per query
and build your test using the "modules" instead of copying and pasting the real JDBC Request samplers
If you're going to store the "modules" as external .jmx files consider using Test Fragments
Related
I'm currently getting data from an Impala query into Javascript/Angular, within a Zeppelin note, by using a %angular paragraph that does an AJAX call to the Zeppelin Notebook REST API (to run a paragraph synchronously, doc: https://zeppelin.apache.org/docs/0.8.2/usage/rest_api/notebook.html#run-a-paragraph-synchronously ) - this runs another paragraph in the same note, which is set to %impala and has the query. Doing this via the API means I get the data back into Javascript as the REST response to the API call. (That API link seems to be the only call in the whole API that will run a paragraph and return data from it).
However, it's been proving a bit unreliable in our big corporate setting for various reasons (network and policy related etc). My question is are there ANY other ways, within Zeppelin, that I could use to get data from a query (on a huge database), into Javascript. The data is often a few 10's of KB but could potentially by multi-MB.
For example, I know I can run pyspark .sql("My query here") type queries in a %pyspark paragraph but how do I then get the data over to JS running in the %angular para? Is the only option the same API call? I know of variable binding using the z. context but not sure that can cope with larger data sizes?
Any suggestions very welcome! Thanks
I have two entities in data-config.xml file. I want to run a cron job for each entity at different frequencies. Is it possible to have different times for each entity? Or will it always make two requests when there are two entities when import is done using cron job?
You could do it using
DIH commands are sent to Solr via an HTTP request. The following operations are supported.
Only the SqlEntityProcessor supports delta imports.
For example: http://localhost:8983/solr/dih/dataimport?command=delta-import.
This command supports the same clean, commit, optimize and debug parameters as full-import command described below.
entity
The name of an entity directly under the tag in the configuration file. Use this to execute one or more entities selectively.
Multiple "entity" parameters can be passed on to run multiple entities at once. If nothing is passed, all entities are executed.
For example:
http://localhost:8983/solr/dih/dataimport?command=delta-import&entity=YOUR_ENTIY_NAME
Read more
https://lucene.apache.org/solr/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html
I can see similar problems in different variations but haven't managed to find a definite answer.
Here is the usecase:
SFTP server that I want to poll from every hour
on top of that, I want to expose a REST endpoint that the user can hit do force an ad-hoc retrieval from that same SFTP. I'm happy with the schedule on the polling to remain as-is, i.e. if I polled, 20 mins later the user forces refresh, the next poll can be 40 mins later.
Both these should be idempotent in that a file that was downloaded using the polling mechanism should not be downloaded again in ad-hoc pull and vice-versa. Both ways of accessing should download ALL the files available that were not yet downloaded (there will likely be more than one new file - I saw a similar question here for on-demand fetch but it was for a single file).
I would like to avoid hammering the SFTP via pollEnrich - my understanding is that each pollEnrich would request a fresh list of files from SFTP, so doing pollEnrich in a loop until all files are retrieved would be calling the SFTP multiple times.
I was thinking of creating a route that will start/stop a separate route for the ad-hoc fetch, but I'm not sure that this would allow for the idempotent behaviour between routes to be maintained.
So, smart Camel brains out there, what is the most elegant way of fulfilling such requirements?
Not a smart camel brain, but I would give a try as per my understanding.
Hope, you already went through:
http://camel.apache.org/file2.html
http://camel.apache.org/ftp2.html
I would have created a filter, separate routes for consumer and producer.
And for file options, I would have used: idempotent, delay, initialDelay, useFixedDelay=true, maxMessagesPerPoll=1, eagerMaxMessagesPerPoll as true, readLock=idempotent, idempotent=true, idempotentKey=${file:onlyname}, idempotentRepository, recursive=false
- For consuming.
No files will be read again! You can use a diversity of options as documented and try which suits you the best, like delay option. If yo
"I would like to avoid hammering the SFTP via pollEnrich - my understanding is that each pollEnrich would request a fresh list of files from SFTP, so doing pollEnrich in a loop until all files are retrieved would be calling the SFTP multiple times." - > Unless you use the option disconnect=true, the connection will not be terminated and you can either consume or produce files continously, check ftp options for disconnect and disconnectOnBatchComplete.
Hope this helps!
GoodDay, starting experimentation on Node against MsSql on Azure for solutions very data-centric.
What would be best practice for this short of architecture ?
(all communication to the DB to be made trough Store Procedures, obviously)
So far, we are experimenting with https://www.npmjs.com/package/mssql and found that output parameters last a while sometimes until they become ready...
so we have thought into:
SP call parameters as separate input-Parameters (easier to treat in SQL)
Data to node application as a single (sometimes composed) JSON, as a recordset
SP execution code trough RETURN (202 ok, rest of codes as w3c web codes)
Return message, in case of error, trough an output parameter.
Any indications ?
I found very interesting this binomia (Node + MsSQL), strength from one side, flexibility and speed from the other one ...
If you are planning to return single JSON object as recordset, are you planning to just execute select query with FOR JSON clause? MSSQL supports JSON outputs, but even without this feature maybe you can use pipe method to stream results into output stream (I have not tried it).
Here is a sample app that uses tedious and just stream FOR JSON output to Http response output:
https://github.com/Microsoft/sql-server-samples/blob/master/samples/features/json/todo-app/nodejs-express4-rest-api/routes/todo.js and just stream results generated by FOR JSON to Http output. It does not uses MSSQL but you can easily rewrite it to use MSSQL.
FOR JSON might be handy if you want to return chunked data to clients since Azure SQL Database spits large JSON responses into 2KB chunks, but MSSQL enables you to parse entire response as one big object.
Regarding error handling, it might be a little bit unusual to return HTTP error codes directly from stored proc as output paramters. Maybe you can use standard throw/raiserror statements and map error code to Http error codes. There are around 80 useful Http error codes in the 4xx and 5xx range so even error state (tinyint) would be enough.
I'm profiling my rails 3.2 app with miniprofiler, and it shows me a
SELECT 1
query at the beginning of each page load or ajax call. It only takes 0.4ms, but it is still a seemingly unnecessary database query.
Anyone know why this query is happening or how to get rid of it?
SELECT 1 is like a ping - the cheapest query to test whether the session is alive and kicking. Various clients use it for that purpose. It may be useless in your case ...
For Postgres you can find it in this line on Github.
Or, if you using MySQL, you can see the solution in this groupon engineering blog.