I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later stage this shall be used in a deployed script at the user end without any type of user-visible authentication.
So far I generate a signed url like this:
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/testfile
which will generate something like
https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount#developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D
The next step (at the user end) would be curl uploading the file with a PUT request. Like so:
curl -X PUT --data-binary #testfile 'https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount#developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D'
I can get this to work with an existing file in the bucket and a GET request (for downloading), but it does not seem to work for uploading. curl throws the server's response with error messages like this at me:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
Check your Google secret key and signing method.</Message>
<StringToSign>PUT
application/x-www-form-urlencoded
1430963040
/mybucket/testfile</StringToSign>
</Error>
And this makes sense to me, as obviously I am not just making a bare PUT request, but one for a particular file of a specific size, whereas the signature computed by 'gsutil signurl' would not know about these details at the time it is computed.
Somehow I was under the impression (e.g., based on the last usage case described in gsutil signurl documentation and also in the post How to allow anonymous uploads to cloud storage) that it should be possible to generate a generic signed url for uploading purposes and then use it later. Am I just mistaken about this point or is there a way to fix the curl request?
Any thoughts about this are appreciated. However, I'd like this to work with "minimal tools", i.e., ideally shell and curl only, but no other programming languages.
EDIT:
Organising one's thoughts by formulating the exact problem is the first step towards the solution. I realise now that
curl -X PUT -T - [request-url] < testfile
does actually solve the immediate problem. However, this means multiple users would write to the same file if they use the same signed url. The documentation suggests you can omit the object name in the creation of the signed url, i.e., use
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/
This, supposedly, would allow anyone with the resulting signed url to put any object of any type into my bucket. Only I do not get this work, as I don't see how you can then tell GCS which object you are actually writing to.
This was driving me mad too. Turns out it was the binary-file part of the curl command. Try this instead:
curl -X PUT --upload-file me.jpeg $SIGNED_URL
If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:
curl -X PUT -T - [request-url]?name=[object-name] < testfile
This surely works with storage/v1, although I have not tried myself with a signed URL yet.
I encountered the similar problem(403 forbidden).
It turned out that my json library, which I use it to marshal each response, would replace & by \u0026 for security concern. So the url may be correct in the program but invalid in client side
So I guess that there might be some string encoding bug inside the Signature query string of your url since the signature string is harder to detect error in comparison with my \u0026.
Related
I am trying to do HTTPS POST using curl api in C. I am posting binary data by converting it into base64. However the base64 converted data is getting modified by the curl library. The + symbols in the parameter are getting replaced with spaces. I tried setting the charset as utf-8 and the content type as json. That did not help. Is there a reason why this could be happening? I am on macOS.
Is there a way/option to tell curl to not touch/change my data?
I am trying to write a C program that takes in two arguments, either [-url | -phone | -email] and a text file that the user will download from a website.
After the user inputs the flag and the name of the text file, the program is supposed to extract and display the contents based on the regular expression I have developed.
For example, for URL the regex is
/^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w \.-]*)*\/?$/
I am having a hard time figuring out how I can implement such a task. Do I need to use fork()? How exactly can I read the data from the text file and display back results based on the regex?
Here is the example OUTPUT
$ gcc –o minor1 minor1.c
$ ./minor1
Usage:
./minor1 [-url | -email | -phone] input_file
URL CASE SCENARIO:
$./minor1 -url index.html
https://www.web.edu/
...
http://webpreview.web.edu/
...
httpL//policy.web.edu/
Based on the flag and the input file, this is what it is supposed to return
You may use curl to download the file from the web.
How to download a file from a URL in C, as a browser would?
C program for downloading files with curl
Then you can iterate and parse the data to extract the regex pattern of either url, email or phone.
Try to come out with some code yourself and if there is any problem, post what you did, snippet of the code that failed, and explain your own thoughts on why do you think it failed.
A Batch file is executed and it gives few parameters as output.I need to write those output parameters in virtual table server (VTS), as i need to pass them to my LR Script.
Output of Batch file
USING API KEY : Android
Base URL = /user/authorization
HEADERS :
accessKey = 45k907its35dooeo182dm0guy8k0dv8o
signature = Tdo0ZBfZazTvYd8UwmHT+haq2vM=
timestamp = 1455397355435enter
The major concern is, the output of batch-file is valid only for few minutes(HMACSHA1), i cant wait to copy it in csv file and then upload it in VTS. Any ideas how I can directly write it on VTS. Thanks in advance :)
Have you considered having your batch automatically output the CSV and then also automatically upload it to VTS. Otherwise, the API in use is a set of LoadRunner libraries.
An alternate path that I often use is with RabbitMQ. With the management plugin there is an HTTP interface available that you could push data with CURL and then pull it using a standard HTTP call from within your virtual user. I use Rabbit MQ for it's open back end to use stuff outside of a LoadRunner virtual user to seed a queue and also because the HTTP interface is simpler than the VTS API set.
Wonder if anyone has any experience posting image files with CURL in C..?
I am writing a program to post to a facebook Type web service, and everything is going fine, except when I attempt to post image files...
There's a special format that the server needs or it will not accept the post...
something like this:
---webformkitXXXXXXXX\r\n
filename"somefile.jpg"\r\n
JPEG or IMAGE FILE HERE (in binary)
---webformkitXXXXXXXX\r\n
END----
So when I am finally able to to memcpy together the different pieces I need,
I can save it to file, and it looks just fine, but I can see from the packet captures, that CURL doesn't like taking the binary, it appears that it's truncating the buffer at the first sign of a '\0' because, it only sends like 300 bytes, when it should be sending 80K...
I've been using this: curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
Thank You!
You need to use CURLOPT_WRITEDATA and plug in a function you have written to write the POST data.
This works:
curl --url http://someurl.tld --form "apikey=39485730"
This does not:
curl --url http://someurl.tld --form "apikey=<keyfile"
Error:
Invalid API keys
The cURL manual (http://curl.haxx.se/docs/manpage.html) explicitly states
-F/--form (HTTP) This lets curl emulate a filled-in form in which a user has pressed the submit button. [...] To force the 'content' part to be a file, prefix the file name with an # sign. To just get the content part from a file, prefix the file name with the symbol <. The difference between # and < is then that # makes a file get attached in the post as a file upload, while the < makes a text field and just get the contents for that text field from a file.
FWIW:
If I use --form "apikey=#keyfile" I get the error: "API key is required".
This confirms that # is definately wrong (which I am okay with).
But why does < not work, and what to do about it?
It looks as if the content of the file is either not passed on, or wrong. Thus I've made quadruple sure, that only the api key (here: 39485730) and nothing else is in the file.
If it's important why I am trying to do this:
I need this curl-command in scripts, and don't want to put the API key there. Instead, it should be in the HOME of the user who runs this script, only readable by him, and nobody else.
Thank you in advance for any insight... :-)
I think you might have a trailing newline in keyfile. You can check this with:
xxd keyfile
There should be no 0a at the end. If there is, you can re-create the keyfile without the trailing newline like this:
echo -n 39485730 > keyfile
and try again with the new keyfile.