Perform HTTP Get with curl inside a page - file

I have this download link
It downloads a zip file when link is clicked oR pasted in browser address bar. If pasted in the browser, the browser performs an http GET and the response forces the browser to download the file.
I want to perform http get from within a php script and force the browser to download the file. How can i achieve this with cURL in php.
I have tried below code but it prints binary response on the browser.
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_URL => 'http://windows.php.net/downloads/pecl/releases/pthreads/3.1.0/php_pthreads-3.1.0-7.0-ts-vc14-x64.zip'
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);

The one thing you can try is simply dump that content instead of capturing it with the right headers. This way your script/server serves as an intermediary instead of a user, as it is right now.
Thus:
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_URL => 'http://windows.php.net/downloads/pecl/releases/pthreads/3.1.0/php_pthreads-3.1.0-7.0-ts-vc14-x64.zip'
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
header('Content-Description: File Transfer');
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="php_pthreads-3.1.0-7.0-ts-vc14-x64.zip"');
header('Expires: 0');
header('Cache-Control: must-revalidate');
header('Pragma: public');
header('Content-Length: ' . strlen($resp));
exit($resp);

Related

Fill a form with a captcha solving script using requests

So basically the script running perfectly, I get the feedback that my Captcha has been solved. But the problem lies when the script enters the rest of the form's input.
Any idea where it is coming from?
import requests
from random import randint
from time import sleep
# Add these values
API_KEY = 'ApiKey' # Your 2captcha API KEY
site_key = 'SiteKey' # site-key, read the 2captcha docs on how to get this
url = 'https://site' # example url
proxy = 'proxy' # example proxy
proxy = {'http': 'http://' + proxy, 'https': 'https://' + proxy}
s = requests.Session()
# here we post site key to 2captcha to get captcha ID (and we parse it here too)
captcha_id = s.post("http://2captcha.com/in.php?key={}&method=userrecaptcha&googlekey={}&pageurl={}".format(API_KEY, site_key, url), proxies=proxy).text.split('|')[1]
# then we parse gresponse from 2captcha response
recaptcha_answer = s.get("http://2captcha.com/res.php?key={}&action=get&id={}".format(API_KEY, captcha_id), proxies=proxy).text
print("solving ref captcha...")
while 'CAPCHA_NOT_READY' in recaptcha_answer:
sleep(5)
recaptcha_answer = s.get("http://2captcha.com/res.php?key={}&action=get&id={}".format(API_KEY, captcha_id), proxies=proxy).text
recaptcha_answer = recaptcha_answer.split('|')[1]
# we make the payload for the post data here, use something like mitmproxy or fiddler to see what is needed
payload = {
'username' : 'username',
'password' : 'password' ,
'password_again' : 'password' ,
'email' : 'email#gmail.com' ,
'key': 'value',
'gresponse': recaptcha_answer # This is the response from 2captcha, which is needed for the post request to go through.
}
# then send the post request to the url
response = s.post('https://site',payload, proxies=proxy)
You only need to change 'gresponse' to 'g-recaptcha-response' and it's working perfectly

Getting error: Authentication with the CRM provider failed, while creating sfdc instance through cloudelements

I am trying to create an element instance in cloudelements. Have retrieved the code parameter from the url query string, have following codebase:
$ch = curl_init();
$code = $_REQUEST['code'];
$headers = array(
'Content-Type: application/json',
'Accept: application/json',
'Authorization: User xxxxxxxxyyyyyyyyyzzzzzzzzzzz=, Organization ccccccccccccccccccccccccc'
);
$url = 'https://api.cloud-elements.com/elements/api-v2/elements/23/instances';
$curl_post_data = array(
"providerData" => array(
"code" => $code
),
"name" => "salesforce_instance_".date("Y-m-d-H:i:s"),
"configuration" =>
array(
"base.url" => "https://login.salesforce.com",
"filter.response.nulls" => "true"
)
);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($curl_post_data));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$curl_response = curl_exec($ch);
But, i am getting following error:
Authentication with the CRM provider failed. Please ensure valid authentication values were provided.
This might be an issue with the header part. Please check your 'Authorisation' field in the header. It is a formatted string. So the spaces and commas are countable. It is better to copy-paste from their UI itself.
You can get the authorisation header from their 'api try it out' page. To enter to that screen first login to cloud elements. Then press the 'API Docs' button. It will direct you to the page - https://my.cloudelements.io/api-docs/platform . Then select some option from left side. For example 'Elements'. Then there you can see a set of endpoints over the right side. Select one from there, say /elements/{keyOrId}/instances. Then press on the 'Try it out ' button. On the authorisation field , you can get the header value. Copy- paste it in your code.
Thank you

Unable to download a document from google cloud storage

I am able to upload a document and download the document from google cloud storage for signed url using httpclient in java.But,when i put the same signed url in browser i am unable to download document for the link.I am getting following error
The request signature we calculated does not match the signature you
provided. Check your Google secret key and signing method.`
But when i mark check shared publicly check box in storage browser i am able to download from the generated signed url.But i want to allow a user to download a document from the browser without marking it as shared publicly.
.
I want to get confirm on some confusing part like
For document to get accessible by user who does not have google account after creating a signed url also i have to check shared publicly check box in storage browser?
But i think if the url is signed then it should not be check for shared publicly checkbox and user who does not have google account can access the document?But in my case it is not happening .According to link
https://developers.google.com/storage/docs/accesscontrol#About-CanonicalExtensionHeaders
it talks about Canonicalized_Extension_Headers .So i put in my request header
request.addHeader("x-goog-acl","public-read");
This is my code
// construct URL
String url = "https://storage.googleapis.com/" + bucket + filename +
"?GoogleAccessId=" + GOOGLE_ACCESS_ID +
"&Expires=" + expiration +
"&Signature=" + URLEncoder.encode(signature, "UTF-8");
System.out.println(url);
HttpClient client = new DefaultHttpClient();
HttpPut request = new HttpPut(url);
request.addHeader("Content-Type", contentType);
request.addHeader("x-goog-acl","public-read");// when i put this i get error
request.addHeader("Authorization","OAuth 1/zVNpoQNsOSxZKqOZgckhpQ");
request.setEntity(new ByteArrayEntity(data));
HttpResponse response = client.execute(request);
When i put request.addHeader("x-goog-acl","public-read");i get error
HTTP/1.1 403 Forbidden error .
.But when i remove this line it is uploaded successfully .It seems like i need to set
request.addHeader("x-goog-acl","public-read") to make it publicly accessible but on putting this on my code i am getting error.
.Any suggestion Please?
Finally Solved it.
To run singed url from browser you have to set HTTP header . In https://developers.google.com/storage/docs/accesscontrol#Construct-the-String
Content_Type Optional. If you provide this value the client (browser) must provide this HTTP header set to the same value.There is a word most.
So if you are providing Content_Type for sign string you must provide same Content_Type in browser http header.When i set Content_Type in browser header this error finally solved
this works for me:
set_include_path("../src/" . PATH_SEPARATOR . get_include_path());
require_once 'Google/Client.php';
function signed_storageURL($filename, $bucket, $p12_certificate_path, $access_id, $method = 'GET', $duration = 3600 )
{
$expires = time( ) + $duration*60;
$content_type = ($method == 'PUT') ? 'application/x-www-form-urlencoded' : '';
$to_sign = ($method."\n"."\n".$content_type."\n".$expires."\n".'/'.$bucket.'/'.$filename);
$signature = '';
$signer = new Google_Signer_P12(file_get_contents($p12_certificate_path), 'notasecret');
$signature = $signer->sign($to_sign);
$signature = urlencode( base64_encode( $signature ) );
return ('https://'.$bucket.'.commondatastorage.googleapis.com/'.$filename.'?GoogleAccessId='.$access_id.'&Expires='.$expires.'&Signature='.$signature);
}
$url = signed_storageURL(rawurlencode("áéíóú espaço & test - =.jpg"),'mybucket', 'mykey.p12','myaccount#developer.gserviceaccount.com');
echo ''.$url.'';

Batch fetching messages performance

I need to get the last 100 messages in the INBOX (headers only). For that I'm currently using the IMAP extension to search and then fetch the messages. This is done with two requests (SEARCH and then UID FETCH).
What's the Gmail API equivalent to fetching multiple messages in one request?
All I could find is a batch API, which seems way more cumbersome (composing a long list of messages:get requests wrapped in plain HTTP code).
It's pretty much the same in the Gmail API as in IMAP. Two requests: first is messages.list to get the message ids. Then a (batched) message.get to retrieve the ones you want. Depending on what language you're using the client libraries may help with the batch request construction.
A batch request is a single standard HTTP request containing multiple Google Cloud Storage JSON API calls, using the multipart/mixed content type. Within that main HTTP request, each of the parts contains a nested HTTP request.
From: https://developers.google.com/storage/docs/json_api/v1/how-tos/batch
It's really not that hard, took me about an hour to figure it out in python even without the python client libraries (just using httplib and mimelib).
Here's a partial code snippet of doing it, again with direct python. Hopefully it makes it clear that's there's not too much involved:
msg_ids = [msg['id'] for msg in body['messages']]
headers['Content-Type'] = 'multipart/mixed; boundary=%s' % self.BOUNDARY
post_body = []
for msg_id in msg_ids:
post_body.append(
"--%s\n"
"Content-Type: application/http\n\n"
"GET /gmail/v1/users/me/messages/%s?format=raw\n"
% (self.BOUNDARY, msg_id))
post_body.append("--%s--\n" % self.BOUNDARY)
post = '\n'.join(post_body)
(headers, body) = _conn.request(
SERVER_URL + '/batch',
method='POST', body=post, headers=headers)
Great reply!
If somebody wants to use a raw function in php to make batch requests for fetching emails corresponding to message ids, please feel free to use mine.
function perform_batch_operation($auth_token, $gmail_api_key, $email_id, $message_ids, $BOUNDARY = "gmail_data_boundary"){
$post_body = "";
foreach ($message_ids as $message_id) {
$post_body .= "--$BOUNDARY\n";
$post_body .= "Content-Type: application/http\n\n";
$post_body .= 'GET https://www.googleapis.com/gmail/v1/users/'.$email_id.
'/messages/'.$message_id.'?metadataHeaders=From&metadataHeaders=Date&format=metadata&key='.urlencode($gmail_api_key)."\n\n";
}
$post_body .= "--$BOUNDARY--\n";
$headers = [ 'Content-type: multipart/mixed; boundary='.$BOUNDARY, 'Authorization: OAuth '.$auth_token ];
$curl = curl_init();
curl_setopt($curl,CURLOPT_URL, 'https://www.googleapis.com/batch' );
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl,CURLOPT_CONNECTTIMEOUT , 60 ) ;
curl_setopt($curl, CURLOPT_TIMEOUT, 60 ) ;
curl_setopt($curl,CURLOPT_POSTFIELDS , $post_body);
curl_setopt($curl, CURLOPT_RETURNTRANSFER,TRUE);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER,0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
$tmp_response = curl_exec($curl);
curl_close($curl);
return $tmp_response;
}
FYI the above function gets just the headers for the emails, in particular the From and Date fields, please adjust according to the api documentation https://developers.google.com/gmail/api/v1/reference/users/messages/get
In addition to MaK you can perform multiple batch requests using the google-api-php-client and Google_Http_Batch()
$optParams = [];
$optParams['maxResults'] = 5;
$optParams['labelIds'] = 'INBOX'; // Only show messages in Inbox
$optParams['q'] = 'subject:hello'; // search for hello in subject
$messages = $service->users_messages->listUsersMessages($email_id,$optParams);
$list = $messages->getMessages();
$client->setUseBatch(true);
$batch = new Google_Http_Batch($client);
foreach($list as $message_data){
$message_id = $message_data->getId();
$optParams = array('format' => 'full');
$request = $service->users_messages->get($email_id,$message_id,$optParams);
$batch->add($request, $message_id);
}
$results = $batch->execute();
here is the python version, using the official google api client. Note that I did not use the callback here, because I need to handle the responses in a synchronous way.
from apiclient.http import BatchHttpRequest
import json
batch = BatchHttpRequest()
#assume we got messages from Gmail query API
for message in messages:
batch.add(service.users().messages().get(userId='me', id=message['id'],
format='raw'))
batch.execute()
for request_id in batch._order:
resp, content = batch._responses[request_id]
message = json.loads(content)
#handle your message here, like a regular email object
solution from Walty Yeung is worked partially for my use case.
if you guys tried the code and nothing happens use this batch
batch = service.new_batch_http_request()

Basic Authentication in CakePHP

I am trying to setup Basic Authentication for my CakePHP app so I can use it as an API for an upcoming mobile application. However If I pass the following:
cameron:password#dev.driz.co.uk/basic/locked/
Where cameron is the username, password is the password, and the rest is the domain and application. locked is a method that requires authentication. (obviously the password is wrong in this example)
(Q1) I will be requested for a username and password in a prompt... but the username and password are in fact correct as if I then type them into the prompt they work... Why would this happen? Haven't I just passed the username and password?
I can't see anything wrong with the way I have set this up in CakePHP.
I set Basic Auth in AppController as:
public $components = array('Auth');
function beforeFilter()
{
parent::beforeFilter();
$this->Auth->authorize = array('Controller');
$this->Auth->authenticate = array('Basic');
$this->Auth->sessionKey = false;
$this->Auth->unauthorizedRedirect = false;
}
(Q2) Even so I have set both sessions to be false and the redirect to false, if the user cancels the prompt then they are redirected to the login page? Any ideas on how to stop this from happening? Ideally I want to send back a JSON response or status code of 401 (depending if it's an AJAX request or not).
So something like:
if ($this->request->is('ajax')) {
$response = json_encode(
array(
'meta'=>array(
'code'=>$this->response->statusCode(401),
'in'=>round(microtime(true) - TIME_START, 4)
),
'response'=>array(
'status'=>'error',
'message'=>'401 Not Authorized'
)
)
);
// Handle JSONP
if(isset($_GET['callback'])) {
$response = $_GET['callback'] . '(' . $response . ')';
}
// Return JSON
$this->autoRender = false;
$this->response->type('json');
$this->response->body($response);
} else {
header('HTTP/1.0 401 Unauthorized');
}
But where would this go in the application logic to show this? It needs to happen for ALL requested methods that require authentication and user fails or cancels the authentication.
(Q3) If you enter incorrect details you are just shown the prompt again until you get the username/password correct or hit cancel. How can I make it show an error?
Any ideas for these three issues (marked as sub questions numbers).
Update: This is how I send the headers to the API:
"use strict";jQuery.base64=(function($){var _PADCHAR="=",_ALPHA="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",_VERSION="1.0";function _getbyte64(s,i){var idx=_ALPHA.indexOf(s.charAt(i));if(idx===-1){throw"Cannot decode base64"}return idx}function _decode(s){var pads=0,i,b10,imax=s.length,x=[];s=String(s);if(imax===0){return s}if(imax%4!==0){throw"Cannot decode base64"}if(s.charAt(imax-1)===_PADCHAR){pads=1;if(s.charAt(imax-2)===_PADCHAR){pads=2}imax-=4}for(i=0;i<imax;i+=4){b10=(_getbyte64(s,i)<<18)|(_getbyte64(s,i+1)<<12)|(_getbyte64(s,i+2)<<6)|_getbyte64(s,i+3);x.push(String.fromCharCode(b10>>16,(b10>>8)&255,b10&255))}switch(pads){case 1:b10=(_getbyte64(s,i)<<18)|(_getbyte64(s,i+1)<<12)|(_getbyte64(s,i+2)<<6);x.push(String.fromCharCode(b10>>16,(b10>>8)&255));break;case 2:b10=(_getbyte64(s,i)<<18)|(_getbyte64(s,i+1)<<12);x.push(String.fromCharCode(b10>>16));break}return x.join("")}function _getbyte(s,i){var x=s.charCodeAt(i);if(x>255){throw"INVALID_CHARACTER_ERR: DOM Exception 5"}return x}function _encode(s){if(arguments.length!==1){throw"SyntaxError: exactly one argument required"}s=String(s);var i,b10,x=[],imax=s.length-s.length%3;if(s.length===0){return s}for(i=0;i<imax;i+=3){b10=(_getbyte(s,i)<<16)|(_getbyte(s,i+1)<<8)|_getbyte(s,i+2);x.push(_ALPHA.charAt(b10>>18));x.push(_ALPHA.charAt((b10>>12)&63));x.push(_ALPHA.charAt((b10>>6)&63));x.push(_ALPHA.charAt(b10&63))}switch(s.length-imax){case 1:b10=_getbyte(s,i)<<16;x.push(_ALPHA.charAt(b10>>18)+_ALPHA.charAt((b10>>12)&63)+_PADCHAR+_PADCHAR);break;case 2:b10=(_getbyte(s,i)<<16)|(_getbyte(s,i+1)<<8);x.push(_ALPHA.charAt(b10>>18)+_ALPHA.charAt((b10>>12)&63)+_ALPHA.charAt((b10>>6)&63)+_PADCHAR);break}return x.join("")}return{decode:_decode,encode:_encode,VERSION:_VERSION}}(jQuery));
$(document).ready(function(){
var username = 'cameron';
var password = 'password';
$.ajax({
type: 'GET',
url: 'http://dev.driz.co.uk/basic/locked',
beforeSend : function(xhr) {
var base64 = $.base64.encode(username + ':' + password);
xhr.setRequestHeader("Authorization", "Basic " + base64);
},
dataType: 'jsonp',
success: function(data) {
console.log(data);
},
error: function(a,b,c) {
//console.log(a,b,c);
}
});
});
Q1
You don't specify how you visit the protected URL (dev.driz.co.uk/basic/locked). Are you sure that the way you are doing it you are setting up the request headers properly? You need to Base64 encode the username/password.
When your first request fails the browser jumps in with the prompt and to be succeeding means that the browser does it properly for you the second time.
Have a look at you request headers to see what you send the first time and what the browser sends the second.
Q2
When basic auth fails your server sends a 401 with a header WWW-Authenticate:Basic which is picked up from the browser and you are presented with the prompt. That is build in normal behavior for all browsers since ages, you can't change that.
About your issue with canceling and being redirected to login, Auth had some API changes after 2.4 that are highlighted in the book. Before version 2.4 you are always redirected to loginAction.
Finally, let Auth do the work for you by setting it up properly and don't attempt to hardwire the responses yourself like in the code you suggest. You also shouldn't ever be using php's header() in cakephp, use CakeRequest::header() instead.
Q3
Answered in Q2, you can't have Basic and 401 not trigger the prompt. Either change the required authentication header (by perhaps setting a name like Basic-x instead of Basic) or don't send the response code 401 on failure but send i.e. 200 or 400 and add an error message explaining the situation.

Resources