I'm trying to play mp3 files from server-side to client-side. Where the client access the server passing some ID and the server return the file.
Right now, how this is working?
Well, using Laravel (server-side) and AngularJS (client-side) on distinct urls, i'm able to play the song.
But, if I get the request response I'm able to download the song.
So, what would be the good way to work so that information wouldn't be visible to the user?
I would write some sort of file proxy.
You have to move your files out of the publicly accessible area. F.a one level above the page root. So it is not possible to get the data directly.
Then you need a server side script, that gets the data and returns it with the headers you need.
Here is an example (plain PHP):
/**
* #param string $file_name
* #param string $mime
* #param bool $download
*/
public function fileProxyAction($file_name, $mime, $download = false) {
if(basename($file_name) != $file_name) return 'Filename not valid!';
$path = '... your path goes here';
$file = $path.$file_name;
if (!(file_exists($file) && is_readable($file))) return 'The file "'.$file_name.'" could not be found!';
ob_clean();
if($download === false) {
header('Content-type: '.$mime);
header('Content-length: '.filesize($file));
$open = # fopen($file, 'rb');
if ($open) {
fpassthru($open);
exit;
}
} else {
// download
$path_parts = pathinfo($file);
header("Content-Disposition: attachment; filename=\"".$path_parts["basename"]."\"");
header("Content-type: application/octet-stream");
header("Content-length: " . filesize($file));
header("Content-Disposition: filename=\"".$path_parts["basename"]."\"");
header("Cache-control: private"); // open files directly
readfile($file);
die;
}
}
Laravel has an excellent Built-In-Filesystem. Check it out. I'm sure you can optimize my method with it.
EDIT
If you need to check a token or something, you shouldn't call the fileProxyAction directly by the router. Instead let your router call a Method which checks the token or what ever you're using ;)
Example (pseudo code):
Route::get('/mp3/{id}/{token}', function($id, $token) {
if($token !== Session::get('token')) return App::abort(401);
$name = Mp3::findOrFail($id)->name;
$mime = Mp3::findOrFail($id)->mime;
return $this->fileProxyAction($name, $mime);
});
Related
I'm working on my cakephp project
and I am currently upgrading from 3.3.16 to 3.4.0
The project uses the cakephp-upload plugin to save an image.
The Upload plugin needs an existing entity to attach a file to it. A modification of the request is done to grab the avatar, before unsetting it to save the user.
I know this is not a good practice to modify a request, but the code was made this way.
With immutable objects in version 3.4.0, it is just not possible anymore. But i dont know how to do it properly.
Here is the error message given by my unit-test,
ran by vendor/bin/phpunit --filter testAdd tests/TestCase/Controller/Api/V1/UsersControllerTest.php:
There was 1 failure:
1) App\Test\TestCase\Controller\Api\V1\UsersControllerTest::testAdd
Failed asserting that file "/home/comptoir/Comptoir-srv/webroot/img/files/Users/photo/5/avatar/correctAvatarLogo.jpg" exists.
/home/comptoir/Comptoir-srv/tests/TestCase/Controller/Api/V1/UsersControllerTest.php:208
Here is the actual code:
public function add()
{
if (!empty($this->request->data)) {
$user = $this->Users->newEntity($this->request->data);
} else {
$user = $this->Users->newEntity();
}
$message = "";
// Get the avatar before unset it to save the user.
// The Upload plugin need an existing entity to attach a file to it.
if ($this->request->is('post')) {
if (isset($this->request->data['photo']) && !$user->errors()) {
$avatar = $this->request->data['photo'];
$this->request->data['photo'] = "";
}
$user = $this->Users->patchEntity($user, $this->request->data);
if ($this->Users->save($user)) {
$user = $this->Users->get($user->id, ['contain' => []]);
isset($avatar) ? $this->request->data['photo'] = $avatar : null;
$user = $this->Users->patchEntity($user, $this->request->data);
if ($this->Users->save($user)) {
$message = "Success";
$this->Flash->success(__d("Forms", "Your are registred on the Comptoir du Libre, welcome !"));
if (!$this->request->is('json')) {
$this->Auth->setUser($this->Auth->identify());
$this->redirect([
"prefix" => false,
"controller" => "Pages",
"language" => $this->request->param("language")
]);
}
} else {
$message = "Error";
}
} else {
$message = "Error";
$this->Flash->error(__d("Forms", "Your registration failed, please follow rules in red."));
}
$message == "Error" ? $this->set('errors', $user->errors()) : null;
}
$this->ValidationRules->config('tableRegistry', "Users");
$rules = $this->ValidationRules->get();
$userTypes = $this->Users->UserTypes->find('list', ['limit' => 200]);
$this->set(compact('user', 'userTypes', 'rules', 'message'));
$this->set('_serialize', ['user', 'userTypes', 'rules', 'message', 'errors']);
}
Does anyone know how to do that respecting the immutable rule ?
Your premise is wrong.
The Upload plugin needs an existing entity to attach a file to it
That's actually not correct, uploading files alongside creating new records works fine. There's no need for this stuff in your controller, it should be possible to handle this with a single basic save, ie you should investigate the problem that you're having with that, and fix it.
However looking at your test, it should fail anyways, because the file data that you're passing is invalid, it's neither an actual uploaded file for which is_uploaded_file() would return true, nor is it acceptable for user data to be able to define the temporary file path, and the error code, ie you're not properly validating the data if that test passes as is. Accepting such data is a security vulnerability, it could allow all sorts of attacks, from path traversal to arbitrary file injections!
Ideally your whole upload validation and writing functionality would support \Psr\Http\Message\UploadedFileInterface objects, that would allow for very simply testing by being able to pass instances of that class into the test data, that might be something worth suggesting for the plugin. Without such functionality, your second best bet would probably be something like modifying the table's validation rules before issuing the test request, so that is_uploaded_file() is being skipped, or you're switching to integration tests over HTTP, instead of the simulation in CakePHP.
How can I test a file upload function with a controller test case in CakePHP 3?
I keep running into the problem that PHP thinks the file was not actually uploaded. The validation rules that works for a browser test, but not for a test case:
->add('file', [
'is_uploaded_file' => [
'rule' => ['uploadedFile', ['optional' => false]],
'message' => 'File is no valid uploaded file'
],
I quickly found out that the is_uploaded_file and move_uploaded_file are impossible to fool in a unit test.
However, most topics on this are old and/or not about CakePHP specifically, so I figured I'd post a new question.
You don't necessarily need to modify validation rules, what you can do alternatively is using an object that implements \Psr\Http\Message\UploadedFileInterface. CakePHP's default uploaded file validation supports such objects.
CakePHP requires zendframework/zend-diactoros, so you can use \Zend\Diactoros\UploadedFile and do something like this in your tests:
$data = [
// ...
'file' => new \Zend\Diactoros\UploadedFile([
'/path/to/the/temporary/file/on/disk',
1234, // filesize in bytes
\UPLOAD_ERR_OK, // upload (error) status
'filename.jpg', // upload filename
'image/jpeg' // upload mime type
])
];
The uploadedFile rule will automatically treat such an object as an uploaded file.
Of course your code that handles the file upload must support that interface too, but it's not that complicated, you just need to make sure that the regular file upload arrays are being converted into UploadedFileInterface implementations so that your upload handler can make that a requirement.
It could of course be done in the upload handler itself, so that validation will use regular file upload arrays as well as UploadedFile objects. Another way would be to convert them earlier when creating entities, using the beforeMarshal handler/event, something along the lines of this:
public function beforeMarshal(\Cake\Event\Event $event, \ArrayObject $data, \ArrayObject $options)
{
$file = \Cake\Utility\Hash::get($data, 'file');
if ($file === null) {
return;
}
if (!($file instanceof \Psr\Http\Message\UploadedFileInterface)) {
if (!is_uploaded_file(\Cake\Utility\Hash::get($file, 'tmp_name'))) {
$file = new \Zend\Diactoros\UploadedFile(
null,
0,
UPLOAD_ERR_NO_FILE,
null,
null
);
} else {
$file = new \Zend\Diactoros\UploadedFile(
\Cake\Utility\Hash::get($file, 'tmp_name'),
\Cake\Utility\Hash::get($file, 'size'),
\Cake\Utility\Hash::get($file, 'error'),
\Cake\Utility\Hash::get($file, 'name'),
\Cake\Utility\Hash::get($file, 'type')
);
}
$data['file'] = $file;
}
}
This will convert the data into an UploadedFile object in case it's an actually uploaded file. This extra check is added because CakePHP's behavior of merging file data with the POST data, making it impossible (unless one can access the request object, or the $_FILES superglobal) to determine whether a user posted that data, or whether PHP generated that data for an actual file upload.
If you then use \Psr\Http\Message\UploadedFileInterface::moveTo() to move the file, it will work in SAPI (browser based) as well as non-SAPI (CLI) environments:
try {
$file->moveTo($targetPath);
} catch (\Exception $exception) {
$entity->setError(
'file', [__('The file could not be moved to its destination.')]
);
}
See also
PSR-7: HTTP message interfaces > Uploaded Files
API > \Cake\Validatiom\Validation::uploadedFile()
I actually figured it out almost immediately after I'd posted.
The solution is based on https://pierrerambaud.com/blog/php/2012-12-29-testing-upload-file-with-php
So the only way to get around the problem is overriding both the built-in functions: is_uploaded_file and move_uploaded_file.
The uploadedFile validation rule lives inside Cake\Validation, and I'm using the move function in a table event, so inside App\Model\Table.
I added the following to the top of the controller test case:
<?php
namespace Cake\Validation;
function is_uploaded_file($filename)
{
return true;
}
namespace App\Model\Table;
function move_uploaded_file($filename, $destination)
{
return copy($filename, $destination);
}
namespace App\Test\TestCase\Controller;
use App\Controller\CarsController;
use Cake\TestSuite\IntegrationTestTrait;
use Cake\TestSuite\TestCase;
use Cake\Core\Configure;
/**
* App\Controller\CarsController Test Case
*/
class CarsControllerTest extends BaseTestCase
{
use IntegrationTestTrait;
// ...
And it works!
I'm trying to integrate PayPal's IPN code into CakePHP 3.
namespace App\Controller;
use PayPal\Api\PaypalIPN;
class IpnController extends AppController
{
public function index()
{
$this->autoRender = false;
$ipn = new PayPalIPN();
// Use the sandbox endpoint during testing.
$ipn->useSandbox();
$verified = $ipn->verifyIPN();
if ($verified) {
/*
* Process IPN
* A list of variables is available here:
* https://developer.paypal.com/webapps/developer/docs/classic/ipn/integration-guide/IPNandPDTVariables/
*/
}
// Reply with an empty 200 response to indicate to paypal the IPN was received correctly.
header("HTTP/1.1 200 OK");
}
}
This is failing to validate on PayPal's end and I'm suspecting it has to do with setting the headers in the controller view.
Is there a way to set the header properly in CakePHP's controller.
I had this code running stand alone (in just a php file) and it seemed to work just fine.
You should not output any data in your controller action - that means you should not use echo, header() or any function or construct that would return anything to browser. If you do, you will encounter a "headers already sent" error.
If you want to set headers, you should use withHeader() or withAddedHeader() methods of Cake\Http\Response.
For status codes, you also have withStatus() method:
$response = $this->response;
$response = $response->withStatus(200,"OK");
return $response; // returning response will stop controller from rendering a view.
More about setting headers can be found in docs:
Setting response headers in CakePHP 3
Cake\Http\Response::withStatus()
Maybe that's not very Cakish, but actually one can send headers this way - it just have to be followed by die; or exit; to prevent app from further response processing.
Anyway, for sure your problem is not associated with headers. IPN seems to doesn't work properly with Paypal Sandbox. Maybe you should try it other way with ApiContext class?
A server side application requires authorization on file download links. This means a normal <a ng-href="{{createLinkToFile()}}"> is no longer sufficient to get enough parameters passed to the server.
When trying to use a programmatic call to the file download, I get the response data back to Dart client application. Using a simple http GET:
var url = "http://example.com/file";
headers.putIfAbsent("Authorization", () => "bearer " + token;
_http.get(url: url, headers : headers);
The future returned by the GET will hold the data, but how do I instruct the browser to download it as a file, instead of just trying to keep it in memory?
Or is there a way to just do it in a normal link?
After downloading the data from the server like shown in Using Dart to Download a PNG File (Binary File) and displaying it not working you can create a download link like shown at http://blog.butlermatt.me/2014/03/dynamically-generating-download-files/
import 'dart:html';
void main() {
List body = [ 'Some test data ...\n'];
// Create a new blob from the data.
Blob blob = new Blob(body, 'text/plain', 'native');
// Create a data:url which points to that data.
String url = Url.createObjectUrlFromBlob(blob);
// Create a link to navigate to that data and download it.
AnchorElement link = new AnchorElement()
..href = url
..download = 'random_file.txt'
..text = 'Download Now!';
// Insert the link into the DOM.
var p = querySelector('#text');
p.append(link);
}
The code of Seth solves indeed part of the problem. To make it a bit more complete, I'm now using the following:
void doPdfFileRequest(String url) {
var request = new HttpRequest();
request.open('GET', url);
request.responseType = "blob";
request.withCredentials = false;
request.setRequestHeader("Accept", _httpAcceptHeader);
request.setRequestHeader("Authorization", "bearer " + token);
request.onReadyStateChange
.listen((r) => onData(request, "filename.pdf"));
request.send();
}
void onData(HttpRequest request, String filename) {
if (request.readyState == HttpRequest.DONE && request.status == 200) {
if (!isIE()) {
var contentType = request.getResponseHeader("content-type");
AnchorElement downloadLink = new AnchorElement(
href: Url.createObjectUrlFromBlob(request.response));
downloadLink.rel = contentType;
downloadLink.download = filename;
var event = new MouseEvent("click", view: window, cancelable: false);
downloadLink.dispatchEvent(event);
} else {
var href = Url.createObjectUrlFromBlob(request.response);
window.open(href, "_self");
}
}
}
A few things to notice. Instead of using the downloadLink.click(), a mouse event is constructed to ensure that it works on Firefox as well as on Safari and Chrome. Firefox seems not to handle the click() otherwise. Binding it to the DOM as is done in the code of Seth isn't necessary.
Internet Explorer doesn't understand the download attribute, so nothing will happen, therefore a window.open is used to at least have it work (though not ideal) on IE, it's redirecting to self to avoid being hit by the pop up blocker.
There are solutions that convert the result download result to Base64 first and put it in a data:mimetype href, using the blob this isn't necessary.
A nice way to set the filename on the file to download would be through the content disposition header, but this header is marked as unsafe, so cannot be used. The filename is now set in the code.
Another note, notice that a HttpRequest is used instead http.get(), The HttpRequest allows you to set the responseType, in this case blob, which can be transformed into a object url.
I am creating a Google App Engine web app to "transform" files of 10K~50M
Scenario:
User opens http://fixdeck.appspot.com in web browser
User clicks on "Browse", select file, submits
Servlet loads file as an InputStream
Servlet transforms file
Servlet saves file as an OutputStream
The user's browser receives the transformed file and asks where to save it, directly as a response to the request in step 2
(For now I did not implement step 4, the servlet sends the file back without transforming it.)
Problem: It works for 15MB files but not for a 40MB file, saying: "Error: Request Entity Too Large. Your client issued a request that was too large."
Is there any workaround against this?
Source code: https://github.com/nicolas-raoul/transdeck
Rationale: http://code.google.com/p/ankidroid/issues/detail?id=697
GAE has a hard limits of 32MB for HTTP requests and HTTP responses. That will limit the size of uploads/downloads directly to/from a GAE app.
Revised Answer (Using Blobstore API.)
Google provides to the Blobstore API for handling larger files in GAE (up to 2GB). The overview documentation provides complete sample code. Your web form will upload the file to blobstore. The blobstore API then rewrites the POST back to your servlet where you can do your transformation and save the transformed data back in to the blobstore (as a new blob).
Original Answer (Didn't Consider Blobstore as an option.)
For downloading, I think GAE only workaround would be to break the file up in to multiple parts on the server, and then reassemble after downloading. That's probably not doable using a straight browser implementation though.
(As an alternative design, perhaps you could send the transformed file from GAE to an external download location (such as S3) where it could be downloaded by the browser without the GAE limit restrictions. I don't believe GAE initiated connections have same request/response size limitations, but I'm not positive. Regardless, you would still be restricted by the 30 second maximum request time. To get around that, you'd have to look in to GAE Backend instances and come up with some sort of asynchronous download strategy.)
For uploading larger files, I've read about the possibility of using HTML5 File APIs to slice the file in to multiple chunks for uploading, and then reconstructing on the server. Example: http://www.html5rocks.com/en/tutorials/file/dndfiles/#toc-slicing-files . However, I don't how practical a solution that really is due to changing specifications and browser capabilities.
You can use the blobstore to upload files as large as 2 gigabytes.
When uploading larger files, you can consider the file to be chunked into small sets of requests (should be less than 32MB which is the current limit) that Google App Engine supports.
Check this package with examples - https://github.com/pionl/laravel-chunk-upload
Following is a working code which uses the above package.
View
<div id="resumable-drop" style="display: none">
<p><button id="resumable-browse" class="btn btn-outline-primary" data-url="{{route('AddAttachments', Crypt::encrypt($rpt->DRAFT_ID))}}" style="width: 100%;
height: 91px;">Browse Report File..</button>
</div>
Javascript
<script>
var $fileUpload = $('#resumable-browse');
var $fileUploadDrop = $('#resumable-drop');
var $uploadList = $("#file-upload-list");
if ($fileUpload.length > 0 && $fileUploadDrop.length > 0) {
var resumable = new Resumable({
// Use chunk size that is smaller than your maximum limit due a resumable issue
// https://github.com/23/resumable.js/issues/51
chunkSize: 1 * 1024 * 1024, // 1MB
simultaneousUploads: 3,
testChunks: false,
throttleProgressCallbacks: 1,
// Get the url from data-url tag
target: $fileUpload.data('url'),
// Append token to the request - required for web routes
query:{_token : $('input[name=_token]').val()}
});
// Resumable.js isn't supported, fall back on a different method
if (!resumable.support) {
$('#resumable-error').show();
} else {
// Show a place for dropping/selecting files
$fileUploadDrop.show();
resumable.assignDrop($fileUpload[0]);
resumable.assignBrowse($fileUploadDrop[0]);
// Handle file add event
resumable.on('fileAdded', function (file) {
$("#resumable-browse").hide();
// Show progress pabr
$uploadList.show();
// Show pause, hide resume
$('.resumable-progress .progress-resume-link').hide();
$('.resumable-progress .progress-pause-link').show();
// Add the file to the list
$uploadList.append('<li class="resumable-file-' + file.uniqueIdentifier + '">Uploading <span class="resumable-file-name"></span> <span class="resumable-file-progress"></span>');
$('.resumable-file-' + file.uniqueIdentifier + ' .resumable-file-name').html(file.fileName);
// Actually start the upload
resumable.upload();
});
resumable.on('fileSuccess', function (file, message) {
// Reflect that the file upload has completed
location.reload();
});
resumable.on('fileError', function (file, message) {
$("#resumable-browse").show();
// Reflect that the file upload has resulted in error
$('.resumable-file-' + file.uniqueIdentifier + ' .resumable-file-progress').html('(file could not be uploaded: ' + message + ')');
});
resumable.on('fileProgress', function (file) {
// Handle progress for both the file and the overall upload
$('.resumable-file-' + file.uniqueIdentifier + ' .resumable-file-progress').html(Math.floor(file.progress() * 100) + '%');
$('.progress-bar').css({width: Math.floor(resumable.progress() * 100) + '%'});
});
}
}
</script>
Controller
public function uploadAttachmentAsChunck(Request $request, $id) {
// create the file receiver
$receiver = new FileReceiver("file", $request, HandlerFactory::classFromRequest($request));
// check if the upload is success, throw exception or return response you need
if ($receiver->isUploaded() === false) {
throw new UploadMissingFileException();
}
// receive the file
$save = $receiver->receive();
// check if the upload has finished (in chunk mode it will send smaller files)
if ($save->isFinished()) {
// save the file and return any response you need, current example uses `move` function. If you are
// not using move, you need to manually delete the file by unlink($save->getFile()->getPathname())
$file = $save->getFile();
$fileName = $this->createFilename($file);
// Group files by mime type
$mime = str_replace('/', '-', $file->getMimeType());
// Group files by the date (week
$dateFolder = date("Y-m-W");
$disk = Storage::disk('gcs');
$gurl = $disk->put($fileName, $file);
$draft = DB::table('draft')->where('DRAFT_ID','=', Crypt::decrypt($id))->get()->first();
$prvAttachments = DB::table('attachments')->where('ATTACHMENT_ID','=', $draft->ATT_ID)->get();
$seqId = sizeof($prvAttachments) + 1;
//Save Submission Info
DB::table('attachments')->insert(
[ 'ATTACHMENT_ID' => $draft->ATT_ID,
'SEQ_ID' => $seqId,
'ATT_TITLE' => $fileName,
'ATT_DESCRIPTION' => $fileName,
'ATT_FILE' => $gurl
]
);
return response()->json([
'path' => 'gc',
'name' => $fileName,
'mime_type' => $mime,
'ff' => $gurl
]);
}
// we are in chunk mode, lets send the current progress
/** #var AbstractHandler $handler */
$handler = $save->handler();
return response()->json([
"done" => $handler->getPercentageDone(),
]);
}
/**
* Create unique filename for uploaded file
* #param UploadedFile $file
* #return string
*/
protected function createFilename(UploadedFile $file)
{
$extension = $file->getClientOriginalExtension();
$filename = str_replace(".".$extension, "", $file->getClientOriginalName()); // Filename without extension
// Add timestamp hash to name of the file
$filename .= "_" . md5(time()) . "." . $extension;
return $filename;
}
You can also use blobstore api to directly upload to cloud storage. Blow is the link
https://cloud.google.com/appengine/docs/python/blobstore/#Python_Using_the_Blobstore_API_with_Google_Cloud_Storage
upload_url = blobstore.create_upload_url(
'/upload_handler',
gs_bucket_name = YOUR.BUCKET_NAME)
template_values = { 'upload_url': upload_url }
_jinjaEnvironment = jinjaEnvironment.JinjaClass.getJinjaEnvironemtVariable()
if _jinjaEnvironment:
template = _jinjaEnvironment.get_template('import.html')
Then in index.html:
<form action="{{ upload_url }}"
method="POST"
enctype="multipart/form-data">
Upload File:
<input type="file" name="file">
</form>