I am trying to do HTTPS POST using curl api in C. I am posting binary data by converting it into base64. However the base64 converted data is getting modified by the curl library. The + symbols in the parameter are getting replaced with spaces. I tried setting the charset as utf-8 and the content type as json. That did not help. Is there a reason why this could be happening? I am on macOS.
Is there a way/option to tell curl to not touch/change my data?
Related
I'm trying to use Environments in Paw to help with testing a web API that uses a multipart/form-data submission of an image file. Paw will do this fine if I have a Body part name with a Value of the "File" type and drag my image file in.
file in multipart form
This produces a request part like:
--Emm5HQuI6PlP2Jo1k3KW284fY8yeluRO
Content-Disposition: form-data; name="persfront"; filename="sample_front.jpg"
Content-Type: image/jpeg
ÿØÿá ExifMM*i&;...
But when I try to abstract that file out of the normal Multipart Body window and into an Environment variable and then use that, it still puts the file contents in the request, but leaves the file information out of the part header.
This:
environment variables for file in form
produces this request:
--2gEAMPgSTNfB0dsLImFDHYMRLjqag7Hu
Content-Disposition: form-data; name="persfront"
ÿØÿá ExifMM*i&;...
The lack of file header, normally supplied in there, causes my request to fail.
It's as if the environment variable is being evaluated prior to it being used and so the multipart form is just seeing a bunch of binary instead of a file. Is this a bug or am I doing something incorrectly?
Is there any other way to use the utility of the Environments in this situation?
Thanks!
Probably very late for an answer, but yes, Paw has a special case where if the content of a multipart request is a file, it will include the filename in the multpart headers. Though, if you abstract the file in an environment variable, this special case isn't honoured anymore...
I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later stage this shall be used in a deployed script at the user end without any type of user-visible authentication.
So far I generate a signed url like this:
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/testfile
which will generate something like
https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount#developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D
The next step (at the user end) would be curl uploading the file with a PUT request. Like so:
curl -X PUT --data-binary #testfile 'https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount#developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D'
I can get this to work with an existing file in the bucket and a GET request (for downloading), but it does not seem to work for uploading. curl throws the server's response with error messages like this at me:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
Check your Google secret key and signing method.</Message>
<StringToSign>PUT
application/x-www-form-urlencoded
1430963040
/mybucket/testfile</StringToSign>
</Error>
And this makes sense to me, as obviously I am not just making a bare PUT request, but one for a particular file of a specific size, whereas the signature computed by 'gsutil signurl' would not know about these details at the time it is computed.
Somehow I was under the impression (e.g., based on the last usage case described in gsutil signurl documentation and also in the post How to allow anonymous uploads to cloud storage) that it should be possible to generate a generic signed url for uploading purposes and then use it later. Am I just mistaken about this point or is there a way to fix the curl request?
Any thoughts about this are appreciated. However, I'd like this to work with "minimal tools", i.e., ideally shell and curl only, but no other programming languages.
EDIT:
Organising one's thoughts by formulating the exact problem is the first step towards the solution. I realise now that
curl -X PUT -T - [request-url] < testfile
does actually solve the immediate problem. However, this means multiple users would write to the same file if they use the same signed url. The documentation suggests you can omit the object name in the creation of the signed url, i.e., use
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/
This, supposedly, would allow anyone with the resulting signed url to put any object of any type into my bucket. Only I do not get this work, as I don't see how you can then tell GCS which object you are actually writing to.
This was driving me mad too. Turns out it was the binary-file part of the curl command. Try this instead:
curl -X PUT --upload-file me.jpeg $SIGNED_URL
If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:
curl -X PUT -T - [request-url]?name=[object-name] < testfile
This surely works with storage/v1, although I have not tried myself with a signed URL yet.
I encountered the similar problem(403 forbidden).
It turned out that my json library, which I use it to marshal each response, would replace & by \u0026 for security concern. So the url may be correct in the program but invalid in client side
So I guess that there might be some string encoding bug inside the Signature query string of your url since the signature string is harder to detect error in comparison with my \u0026.
Need a syntax example for sending input file using wget command.
the input file has an URL and a string argument.
Examples will be helpful.
Thanks in advance.
It is possible to do this with wget only. At least with version 1.13.4 and maybe others. The --post-file option allows you to specify a file to send, so long as the postdata file is constructed properly.
I have also tested this with binary files and it works as expected. You do NOT need to base64 encode the file, but you do need to ensure your file does not contain the boundary.
The minimum command required to make this work would be:
wget --header="Content-type: multipart/form-data boundary=FILEUPLOAD" --post-file postfile http://domain/uploadform
and the postdata file would need to contain something like:
--FILEUPLOAD Content-Disposition: form-data; name="comment"
I love uploading files!
--FILEUPLOAD Content-Disposition: form-data; name="uploadFile"; filename="myfile.bin"; Content-Type: application/octet-stream Media
Type: application/octet-stream
Give me some automated file upload action!
--FILEUPLOAD--
A number of details are important here:
Lines in the post data file are terminated with \r\n. The only exception is data inside the file context.
Every BOUNDARY attribute in the postdata must match the BOUNDARY value in the call to wget. (FILEUPLOAD in the example)
All boundaries are prefixed with two hyphens "--" and terminated with \r\n
The last boundary is suffixed with two extra hyphens "--" and terminated with \r\n
Each piece of data, file content or parameter value, is surrounded by an empty line "\r\n"
I am trying to send camera settings file through .cgi with wget. I have code
wget --http-user=admin --http-password=aaa --post-file=file.bin
http://192.168.1.54/restore.cgi
but answer is : http request sent awaiting response... no data received. retrying.
do you have any ideas where can be problem?
Thanks
You are showing a file named 'file.bin', which would suggest it is a raw binary file. wget can only send URL-encoded data. You will need to reverse-engineer the form fields typically sent to restore.cgi and duplicate that format in 'file.bin'.
From wget(1):
In particular, --post-file is not for transmitting files as form
attachments: those must appear as "key=value" data (with appropriate
percent-coding) just like everything else. Wget does not currently
support "multipart/form-data" for transmitting POST data; only
application/x-www-form-urlencoded".
Wonder if anyone has any experience posting image files with CURL in C..?
I am writing a program to post to a facebook Type web service, and everything is going fine, except when I attempt to post image files...
There's a special format that the server needs or it will not accept the post...
something like this:
---webformkitXXXXXXXX\r\n
filename"somefile.jpg"\r\n
JPEG or IMAGE FILE HERE (in binary)
---webformkitXXXXXXXX\r\n
END----
So when I am finally able to to memcpy together the different pieces I need,
I can save it to file, and it looks just fine, but I can see from the packet captures, that CURL doesn't like taking the binary, it appears that it's truncating the buffer at the first sign of a '\0' because, it only sends like 300 bytes, when it should be sending 80K...
I've been using this: curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
Thank You!
You need to use CURLOPT_WRITEDATA and plug in a function you have written to write the POST data.