I am looking for the unix command(s) to calculate md5-content header to be used with IBM Cloud Object Storage API for deletion of multiple objects. I tried echo “request body….” | md5 | base64, however API response is - `
The Content-MD5 you specified was an invalid.
Curl CMD:
curl \
-H "Content-Type: text/plain;charset=utf-8" \
-H "Content-MD5: 75ff06f81643655397a5911ddc195ce8" \
-H "Authorization: $AuthToken" \
"https://<cos-endpoint-name>/<bucket-name>?delete" \
-d 'xml body...'
Error Response:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
<Code>InvalidDigest</Code>
<Message>The Content-MD5 you specified was an invalid.</Message>
<Resource>/ghhsa-bucket-etl-dev/</Resource>
<RequestId>aed25243-22a1-477d-9ab8-b87780625a61</RequestId>
<httpStatusCode>400</httpStatusCode>
</Error>
Appreciate any pointers on this.
The md5 builtin is kinda weak, it's a bit more straightforward using openssl for encryption if possible. Using an example from the docs:
echo -n '<?xml version="1.0" encoding="UTF-8"?><Delete><Object><Key>pasture/cow-one</Key></Object><Object><Key>pasture/cow-two</Key></Object></Delete>' | openssl dgst -md5 -binary | openssl enc -base64
This returns /Gx4aOgplRXMRI2qXXqXiQ== which is what we'd expect.
Related
How might i take the output from a pipe and use curl to post that as a file?
E.g. the following workds
curl -F 'file=#data/test.csv' -F 'filename=test.csv' https://mydomain#apikey=secret
I'd like to get the file contents from a pipe instead but I can't quite figure out how to specify it as a file input. My first guess is -F 'file=#-' but that's not quite right.
cat data/test.csv | curl -F 'file=#-' -F 'filename=test.csv' https://mydomain#apikey=secret
(Here cat is just a substitute for a more complex sequence of events that would get the data)
Update
The following works:
cat test/data/test.csv | curl -XPOST -H 'Content-Type:multipart/form-data' --form 'file=#-;filename=test.csv' $url
If you add --trace-ascii - to the command line you'll see that curl already uses that Content-Type by default (and -XPOST doesn't help either). It was rather your fixed -F option that did the trick!
I am trying to execute a curl for 300 at a same time and using array. I do not know how to bring the content of my file in array. The code I write is bellow.
array=();
for i in {1..300}; do
array+=( file.txt ) ;
done;
curl "${array[#]}";
The file.text include the following code
--next 'https://d16.server.com/easy/api/OmsOrder' -H 'Connection: keep-
alive' - H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Accept:
application/json,
text/plain, */*' -H 'Sec-Fetch-Dest: empty' -H 'User-Agent: Mozilla/5.0
(Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/80.0.3987.132 Safari/537.36' -H 'Content-Type: application/json' -H
'Origin: https://d.server.com' -H 'Sec-Fetch-Site: same-site' -H
'Sec-Fetch-Mode: cors' -H 'Referer: https://d.server.com/' -H 'Accept-
Language: en-US,en;q=0.9,fa;q=0.8' --data-binary
'{"isin":"IRO3TPEZ0001","financeId":1,"quantity":50000,"price":5400}' --
compressed"
array=();
for i in {1..300}; do
array+=( $(cat file.txt|head -$i | tail -1) );
done;
curl "${array[#]}";
You have a file with shell formatted words that you are trying to repeat over and over in a command.
Since the words are shell formatted, you'll need to interpret them using e.g. eval:
contents=$(< file.txt)
eval "words=( $contents )"
arguments=()
for i in {1..300}
do
arguments+=( "${words[#]}" )
done
curl "${arguments[#]}"
A more robust design would be to not use shell quoting and instead format one argument per line:
--next
https://d16.server.com/easy/api/OmsOrder
-H
Connection: keep-alive
-H
Pragma: no-cache
You can then use the above code and replace the eval line with:
mapfile -t words < file.txt
The answer to this question should have been "put each request into a file, one option per line, and use -K/--config to include the file into the command line." That certainly should allow for 300 requests in a single curl command without exceeding the limit on the size of a shell command. (By "request" here, I mean "a URL with associated options". If you only want to use 300 URLs without modifying any other option, you can easily do that by just listing the URLs, on the command line if they aren't too long or otherwise in a file.)
Unfortunately, it doesn't work. I believe that it is supposed to work, and the fact that it doesn't is a bug. If you specify multiple -K options and each of them refers to a file which includes one request and the --next option, then curl will execute only the first and last file. If you instead put the --next options on the command-line in between the -K options, all the request options will be merged, and in addition curl will complain about a missing URL.
However, you can use the -K option by concatenating all 300 requests and passing them through stdin, using -K - to read from stdin. To test that, I created the file containing a single request:
$ cat post-req
--next
-H "Connection: keep-alive"
-H "Pragma: no-cache"
-H "Cache-Control: no-cache"
-H "Accept: application/json, text/plain, */*"
-H "Sec-Fetch-Dest: empty"
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36"
-H "Content-Type: application/json"
-H "Origin: https://d.server.com"
-H "Sec-Fetch-Site: same-site"
-H "Sec-Fetch-Mode: cors"
-H "Referer: https://d.server.com/"
-H "Accept-Language: en-US,en;q=0.9,fa;q=0.8"
--data-binary "{\"isin\":\"IRO3TPEZ0001\",\"financeId\":1,\"quantity\":50000,\"price\":5400}"
--compressed
--url "http://localhost/foo"
and then set up a little webserver that just returns the requested path, and invoked curl with:
for i in $(seq 300); do cat post-req; done | curl -K -
Indeed, all three hundred requests are passed through.
For what it's worth, I reported the bug as https://github.com/curl/curl/issues/5120, and many thanks to Daniel Stenberg for being incredibly responsive by committing a fix in less than two days. So probably the issue will be resolved in the next curl release.
I am trying to send a HTTP POST custom request header (X-Privet-Token) with value set to "" using cURL on a Linux machine. Following is my command:
curl -k -sI POST -H "X-Privet-Token:" -H "Content-Type: application/json" --trace-ascii --data "action=start&user=xyz#gmail.com" https://127.0.0.1/privet/register
I would like to send an empty string to the server.
Am I missing something?
You can use the following syntax to set a custom header with an empty value
curl -H "X-custom-header;" <some host>
From the man page:
man curl
...
-H, --header <header>
...
If you send the custom header with no-value then its header must be termi-
nated with a semicolon, such as -H "X-Custom-Header;" to send "X-Custom-Header:".
...
I know it is an old question, but I think it is worth to answer it.
I faced a similar issue when calling the privet/info API call in a Java client app, I realized the header was not sent when it was empty. If you want to set an empty header, try to use \"\" (that's what privet documentation states for /privet/info API call):
curl -k -sI POST -H "x-privet-token: \"\"" -H "Content-Type: application/json" --trace-ascii --data "action=start&user=xyz#gmail.com" https://127.0.0.1/privet/register
On the other hand, the privet/register API call requires a valid x-privet-token. Leaving the x-privet-token header empty, 'curl' may be ommiting it.
Having said that, the error you are getting occurs when the endpoint is not advertising /privet/register API, probably because it's already registered. You can check the APIs that the device is exposing examining the /privet/info response.
Please forgive me for the potentially basic question but I am a z/OS person trying to learn cURL and Cloudant. I have gotten the following example to work to add a record to a database (using DOS from Windows) :
curl -X POST -b /tmp/cloudant.cookie -H "Content-Type: application/json" -d "{\"_id\":\"2\",\"empName\":\"John Doe\",\"phone\":\"646-598-4133\",\"age\":\"28\"}" --url https://xxxxxxxxxx-bluemix.cloudant.com/rcdb
Now I would like to add a _attachment image1.jpg dile to that record...
Could anyone please tell me what the syntax on windows would be...trying a few combinations but nothing so far works.
To add an attachment follow the instructions in the Cloudant documentation at https://docs.cloudant.com/attachments.html
Example:
Assuming you have already created a document with ID "2" and revision number "1-954695fb9642f02975d76b959d0b5e98" in database rcdb, run the following command:
curl -X PUT -H "Content-Type: image/jpeg" --data-binary "#image1.jpg" --url https://xxxxxxxxxx-bluemix.cloudant.com/$DATABASE/$DOCUMENT_ID/$ATTACHMENT?rev=$REV
replacing $DATABASE with rcdb, $DOCUMENT_ID with 2, $REV with 1-954695fb9642f02975d76b959d0b5e98 and $ATTACHMENT with the desired attachment property name, e.g. mypic.
How to post 5000 files to Solr server?
While posting by using command "java -jar post.jar dir/*.xml", command tool tells Argument list is too long.
The quickest solution would be using a bash script like the following:
for i in $( ls *.xml); do
cat $i | curl -X POST -H 'Content-Type: text/xml' -d #- http://localhost:8080/solr/update
echo item: $i
done
which adds to Solr, using curl, all the xml files within the current directory.
Otherwise you can write a Java main similar to the one included in post.jar, which adds all the xml files within a directory instead of having to pass all of them as arguments.