I am trying to execute a curl for 300 at a same time and using array. I do not know how to bring the content of my file in array. The code I write is bellow.
array=();
for i in {1..300}; do
array+=( file.txt ) ;
done;
curl "${array[#]}";
The file.text include the following code
--next 'https://d16.server.com/easy/api/OmsOrder' -H 'Connection: keep-
alive' - H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Accept:
application/json,
text/plain, */*' -H 'Sec-Fetch-Dest: empty' -H 'User-Agent: Mozilla/5.0
(Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/80.0.3987.132 Safari/537.36' -H 'Content-Type: application/json' -H
'Origin: https://d.server.com' -H 'Sec-Fetch-Site: same-site' -H
'Sec-Fetch-Mode: cors' -H 'Referer: https://d.server.com/' -H 'Accept-
Language: en-US,en;q=0.9,fa;q=0.8' --data-binary
'{"isin":"IRO3TPEZ0001","financeId":1,"quantity":50000,"price":5400}' --
compressed"
array=();
for i in {1..300}; do
array+=( $(cat file.txt|head -$i | tail -1) );
done;
curl "${array[#]}";
You have a file with shell formatted words that you are trying to repeat over and over in a command.
Since the words are shell formatted, you'll need to interpret them using e.g. eval:
contents=$(< file.txt)
eval "words=( $contents )"
arguments=()
for i in {1..300}
do
arguments+=( "${words[#]}" )
done
curl "${arguments[#]}"
A more robust design would be to not use shell quoting and instead format one argument per line:
--next
https://d16.server.com/easy/api/OmsOrder
-H
Connection: keep-alive
-H
Pragma: no-cache
You can then use the above code and replace the eval line with:
mapfile -t words < file.txt
The answer to this question should have been "put each request into a file, one option per line, and use -K/--config to include the file into the command line." That certainly should allow for 300 requests in a single curl command without exceeding the limit on the size of a shell command. (By "request" here, I mean "a URL with associated options". If you only want to use 300 URLs without modifying any other option, you can easily do that by just listing the URLs, on the command line if they aren't too long or otherwise in a file.)
Unfortunately, it doesn't work. I believe that it is supposed to work, and the fact that it doesn't is a bug. If you specify multiple -K options and each of them refers to a file which includes one request and the --next option, then curl will execute only the first and last file. If you instead put the --next options on the command-line in between the -K options, all the request options will be merged, and in addition curl will complain about a missing URL.
However, you can use the -K option by concatenating all 300 requests and passing them through stdin, using -K - to read from stdin. To test that, I created the file containing a single request:
$ cat post-req
--next
-H "Connection: keep-alive"
-H "Pragma: no-cache"
-H "Cache-Control: no-cache"
-H "Accept: application/json, text/plain, */*"
-H "Sec-Fetch-Dest: empty"
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36"
-H "Content-Type: application/json"
-H "Origin: https://d.server.com"
-H "Sec-Fetch-Site: same-site"
-H "Sec-Fetch-Mode: cors"
-H "Referer: https://d.server.com/"
-H "Accept-Language: en-US,en;q=0.9,fa;q=0.8"
--data-binary "{\"isin\":\"IRO3TPEZ0001\",\"financeId\":1,\"quantity\":50000,\"price\":5400}"
--compressed
--url "http://localhost/foo"
and then set up a little webserver that just returns the requested path, and invoked curl with:
for i in $(seq 300); do cat post-req; done | curl -K -
Indeed, all three hundred requests are passed through.
For what it's worth, I reported the bug as https://github.com/curl/curl/issues/5120, and many thanks to Daniel Stenberg for being incredibly responsive by committing a fix in less than two days. So probably the issue will be resolved in the next curl release.
Related
Distributed Services Got Started Successfully:
[2021-10-17 18:04:29,693] INFO Started o.e.j.s.ServletContextHandler#1422ac7f{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:916)
[2021-10-17 18:04:29,693] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:319)
[2021-10-17 18:04:29,693] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
But not able to make schema registry:
curl -X PUT -H "Content-Type:application/json" --data '{"name": "file-stream-demo-distributed","config":{"connector.class":"com.snowflake.kafka.connector.SnowflakeSinkConnector","topic":"demo-2-distributed","file":"/home/ramakrishnakonda/kafka_2.13-2.8.0/config/connect-distributed.properties"}}' http://localhost:8083/connectors
{"error_code":405,"message":"HTTP 405 Method Not Allowed"}
Please help
Use POST rather than PUT, as in this example:
$ curl -X POST -H "Content-Type: application/json" --data '{"name": "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector", "tasks.max":"1", "file":"test.sink.txt", "topics":"connect-test" }}' http://localhost:8083/connectors
curl -X POST "http://localhost:8083/connectors" -H "Content- type:application/json" --data
"{
"name":"file-stream-demo-distributed",
"config":{
"connector.class":"com.snowflake.kafka.connector.SnowflakeSinkConnector",
"tasks.max":"1",
"topics":"demo-2-distributed",
"buffer.count.records":"10000",
"buffer.flush.time":"60",
"buffer.size.bytes":"5000000",
"snowflake.url.name":"XXXXXXXX.XXXXXXXXX.snowflakecomputing.com:443",
"snowflake.user.name":"kafka_connector_user_1",
"snowflake.private.key":"XXXXXXXXXXXXXXXXXXX+XXXXXXXXXXXXXXXXXXXXXXXXXXXXX,
"snowflake.database.name":"KAFKA_DB",
"snowflake.schema.name":"KAFKA_SCHEMA",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable":"true", "value.converter":"com.snowflake.kafka.connector.records.SnowflakeJsonConverter",
"value.converter:schemas.enable":"true",
"value.converter.schema.registry.url":"http://localhost:8081"
}
}"
I am trying to call POST request POST /sql_jobs (dashdbAPI_v2), but I can’t figure out what should be in stop_on_error. It requires string and always ends with error "Parameter stop_on_error is required".
Could you help me solve this issue?
The only valid values for stop_on_error are yes and no. If the value doesn't match one of them, it returns the error message you're seeing.
This is a working sample:
curl \
-X POST "https://host.bluemix.net/dashdb-api/v2/sql_jobs" \
-H "accept: application/json" \
-H "Authorization: Bearer eyJhbGciOiJSUzA ..." \
-H "content-type: application/json" \
-d '{ "commands": "SELECT 1", "limit": 0, "stop_on_error": "yes", "separator": ";" }'
I create batch file to sent message to line notify.
I want new line in my message. I try to use \n, %0a but it can't use.
curl -X POST -H "Authorization: Bearer mytoken" -F "message=Computer 1 sent message!" https://notify-api.line.me/api/notify
in message=Computer 1 sent message! I want output follow:
Computer 1
sent message!
Try like below, that should work
curl -X "POST" "https://notify-api.line.me/api/notify" \
-H "Authorization: Bearer mytoken" \
-H "Content-Type: multipart/form-data" \
-d $'Computer 1 sent message!
Line1
Line2
Line3'
I am trying to send a HTTP POST custom request header (X-Privet-Token) with value set to "" using cURL on a Linux machine. Following is my command:
curl -k -sI POST -H "X-Privet-Token:" -H "Content-Type: application/json" --trace-ascii --data "action=start&user=xyz#gmail.com" https://127.0.0.1/privet/register
I would like to send an empty string to the server.
Am I missing something?
You can use the following syntax to set a custom header with an empty value
curl -H "X-custom-header;" <some host>
From the man page:
man curl
...
-H, --header <header>
...
If you send the custom header with no-value then its header must be termi-
nated with a semicolon, such as -H "X-Custom-Header;" to send "X-Custom-Header:".
...
I know it is an old question, but I think it is worth to answer it.
I faced a similar issue when calling the privet/info API call in a Java client app, I realized the header was not sent when it was empty. If you want to set an empty header, try to use \"\" (that's what privet documentation states for /privet/info API call):
curl -k -sI POST -H "x-privet-token: \"\"" -H "Content-Type: application/json" --trace-ascii --data "action=start&user=xyz#gmail.com" https://127.0.0.1/privet/register
On the other hand, the privet/register API call requires a valid x-privet-token. Leaving the x-privet-token header empty, 'curl' may be ommiting it.
Having said that, the error you are getting occurs when the endpoint is not advertising /privet/register API, probably because it's already registered. You can check the APIs that the device is exposing examining the /privet/info response.
Please forgive me for the potentially basic question but I am a z/OS person trying to learn cURL and Cloudant. I have gotten the following example to work to add a record to a database (using DOS from Windows) :
curl -X POST -b /tmp/cloudant.cookie -H "Content-Type: application/json" -d "{\"_id\":\"2\",\"empName\":\"John Doe\",\"phone\":\"646-598-4133\",\"age\":\"28\"}" --url https://xxxxxxxxxx-bluemix.cloudant.com/rcdb
Now I would like to add a _attachment image1.jpg dile to that record...
Could anyone please tell me what the syntax on windows would be...trying a few combinations but nothing so far works.
To add an attachment follow the instructions in the Cloudant documentation at https://docs.cloudant.com/attachments.html
Example:
Assuming you have already created a document with ID "2" and revision number "1-954695fb9642f02975d76b959d0b5e98" in database rcdb, run the following command:
curl -X PUT -H "Content-Type: image/jpeg" --data-binary "#image1.jpg" --url https://xxxxxxxxxx-bluemix.cloudant.com/$DATABASE/$DOCUMENT_ID/$ATTACHMENT?rev=$REV
replacing $DATABASE with rcdb, $DOCUMENT_ID with 2, $REV with 1-954695fb9642f02975d76b959d0b5e98 and $ATTACHMENT with the desired attachment property name, e.g. mypic.