I need to serve files to authenticated users and recognise that using PHP will include a performance penalty, however what I've experienced so far seems to be unworkable.
I have a very simple controller action which sends the file:
public function view($id = null) {
$id = $id | $this->params->named['id'];
if (!$this->Attachment->exists($id)) {
throw new NotFoundException(__('Invalid attachment'));
}
$this->autoRender = false;
$this->Attachment->recursive = -1;
$file = $this->Attachment->findById($id);
$this->response->file(APP . DS . $file['Attachment']['dir']);
return $this->response;
}
A small (55 KB) PNG file takes 8 seconds to load using this method, where as if I move the file to the webroot directory and load it directly it takes less than 2.5 seconds. From looking at Chrome Dev Tools, the 'Receiving' part of the response is taking > 7s (compared with 1.5s direct).
A medium sized PDF file (2.5MB) takes over 2 minutes through CakeResponse, compared to ~4s directly. Surely I must be missing something in my controller action as this must be unworkable for anyone?
EDIT: CakePHP version is 2.4.1.
Thanks to the suggestion to use Xdebug I was able to quickly track down the problem.
In CakeResponse, there is the following function:
/**
* Flushes the contents of the output buffer
*
* #return void
*/
protected function _flushBuffer() {
//#codingStandardsIgnoreStart
#flush();
#ob_flush();
//#codingStandardsIgnoreEnd
}
Obviously with the error suppression operator the calls to flush and ob_flush would not normally cause a problem.
However, I also have Sentry as a remote debugging tool installed. This ignores the error suppression operator, reports that there is no buffer to flush (because ob_start has not been called) and in doing so outputs the contents of the file to the log file!
Related
I wrote the following method:
/**
* Downloads an arbitrary file to the cache asynchronously, if the current
* platform has a cache path, or to the app home; if the file was previously
* downloaded and if it's still available on the cache, it calls the
* onSuccess callback immediatly.More info:
* https://www.codenameone.com/blog/cache-sorted-properties-preferences-listener.html
*
* #param url The URL to download.
* #param extension you can leave it empty or null, however iOS cannot play
* videos without extension (https://stackoverflow.com/q/49919858)
* #param onSuccess Callback invoked on successful completion (on EDT by
* callSerially).
* #param onFail Callback invoked on failure (on EDT by callSerially).
*/
public static void downloadFileToCache(String url, String extension, SuccessCallback<String> onSuccess, Runnable onFail) {
FileSystemStorage fs = FileSystemStorage.getInstance();
if (extension == null) {
extension = "";
}
if (extension.startsWith(".")) {
extension = extension.substring(1);
}
String name = "cache_" + HashUtilities.sha256hash(url);
if (!extension.isEmpty()) {
name += "." + extension;
}
String filePath;
if (fs.hasCachesDir()) {
// this is supported by Android, iPhone and Javascript
filePath = fs.getCachesDir() + fs.getFileSystemSeparator() + name;
} else {
// The current platform doesn't have a cache path (for example the Simulator)
String homePath = fs.getAppHomePath();
filePath = homePath + fs.getFileSystemSeparator() + name;
}
// Was the file previously downloaded?
if (fs.exists(filePath)) {
CN.callSerially(() -> onSuccess.onSucess(filePath));
} else {
Util.downloadUrlToFileSystemInBackground(url, filePath, (evt) -> {
if (fs.exists(filePath)) {
CN.callSerially(() -> onSuccess.onSucess(filePath));
} else {
CN.callSerially(onFail);
}
});
}
}
It works. It's similar to some methods provided by the Util class, but with two main differences: the first is that the Util class provides methods only to download images to the cache, while I want to download arbitrary files; the second is that I can assume that the same url always returns the same file, so I don't need to download it again if it's still in the cache (while the Util methods always download the files when invoked).
However, I have some doubts.
My first question is about how caching works: currently I'm using this method to download images and videos to cache (in a chatting app), assuming that I don't need to care about when the files will be not more necessary, because the OS will delete them automatically. Is it so, right? Is it possible that the OS deletes files while I'm using them (for example immediately after storing them to the cache), or Android and iOS delete only older files?
I wrote this method to store arbitrary files. Is there any reasonable limit in MB to the file size that we can store in the cache?
Finally, I have a doubt about the callSerially that I used in the method. Previously I didn't use that, but I got odd results: my callbacks do UI manipulations and frequently (but not always) something went wrong. I solved all my callbacks problems adding the callSerially, so callSerially is the solution. But... why? The odd fact is that the ActionListener of Util.downloadUrlToFileSystemInBackground is called under the hood by the addResponseListener(callback) of a ConnectionRequest instance, so the callback is already invoked in the EDT (according to the javadoc). To be sure, I tested CN.isEdt() in the callbacks without adding the callSerially, and it returned true, so in theory callSerially is not necessary, but in practice it is. What's wrong in my reasoning?
Thank you for the explanations.
As far as I know the cache directory is just a directory that the OS is allowed to delete if it needs space. I don't think it will delete it for an active foreground application but that might vary.
There are no limits other than storage. But you still need to consider that the OS won't just clean that directory for you. It will only flush it when storage is very low and even then not always. So you still need to store data responsibly.
I think only the first callSeially has an impact. It defers the result to the next EDT loop instead of continuing in the existing thread.
In my code I am prompting the user to load a json file.
I am then attempting to copy this file into an sqlite database.
Once I have the data I am then able to manipulate it as needed - but I need to get it there in the first place.
So step 1 is to get the data in.
I have progressed as far as prompting the user to navigate to the file they want - but when I try and read the file I get this error ..
ERROR: resources must reside in the root directory thus must start with a '/' character in Codename One! Invalid resource: file:///tmp/temp3257201851214246357..json
So I think that I need to copy this file to the root directory
I cannot find a link that shows me how to do this.
Here is my code so far ...
case "Import Script":
try
{
JSONParser json = new JSONParser();
if (FileChooser.isAvailable()) {
FileChooser.showOpenDialog(".json", e2-> {
String file = (String)e2.getSource();
if (file == null) {
home.add("No file was selected");
home.revalidate();
} else {
home.add("Please wait - busy importing");
home.revalidate();
String extension = null;
if (file.lastIndexOf(".") > 0) {
extension = file.substring(file.lastIndexOf(".")+1);
}
if ("json".equals(extension)) {
FileSystemStorage fs = FileSystemStorage.getInstance();
try {
InputStream fis = fs.openInputStream(file);
try(Reader r = new InputStreamReader(Display.getInstance().getResourceAsStream(getClass(), file), "UTF-8"))
{
Map<String, Object> data = json.parseJSON(r);
Result result = Result.fromContent(data);
...... I progress from here
The error is occurring on this line ...
try(Reader r = new InputStreamReader(Display.getInstance().getResourceAsStream(getClass(), file), "UTF-8"))
If I hard code a filename and manually place it in the /src folder it works ... like this ...
try(Reader r = new InputStreamReader(Display.getInstance().getResourceAsStream(getClass(), '/test.json'), "UTF-8"))
But that defeats the purpose of them selecting a file
Any help would be appreciated
Thanks
I suggest watching this video.
It explains the different ways data is stored. One of the core sources of confusion is the 3 different ways to store files:
Resources
File System
Storage
getResourceAsStream returns a read only path that's physically embedded in the jar. It's flat so all paths to getResourceAsStream must start with / and must have only one of those. I would suggest avoiding more than one . as well although this should work in theory.
The sqlite database must be stored in file system which is encapsulated as FileSystemStorage and that's really the OS native file system. But you can't store it anywhere you want you need to give the DB name to the system and it notifies you where the file is stored and that's whats explained in the code above.
My app features a button to record audio (and another to play it back when the recording is over). I send the recorded audio files on a server. On Android the files is recorded as .amr (mime type audio/amr) and can be played back.
On iOS however the file can neither be played back on the device (iPhone 4 or 4S) nor on a computer. ffmpeg -i reports
[mov,mp4,m4a,3gp,3g2,mj2 # 0x2fac120] moov atom not found
9gMjOnnmsj9JJZR3.m4a: Invalid data found when processing input
Please note that VLC cannot play it either.
I give the m4a extension because Voice recorder uses it (along with aac codec).
Here is the code I use (mostly based on https://github.com/codenameone/CodenameOne/blob/master/Ports/iOSPort/src/com/codename1/impl/ios/IOSImplementation.java#L2768-L2794 ) :
audioMemoPath = ParametresGeneraux.getWRITABLE_DIR() + "AudioMemo-"
+ System.currentTimeMillis() +
(Display.getInstance().getPlatformName().equals("and")
? ".amr"
: ".m4a");
audioMemoMimeType = MediaManager.getAvailableRecordingMimeTypes()[0];
audioMemoRecorder = MediaManager.createMediaRecorder(audioMemoPath, audioMemoMimeType);
// If the permission audio has not been granted
// the audiomemorecoreder will be null
if (audioMemoRecorder != null) {
audioMemoRecorder.play();
boolean b = Dialog.show("Recording", "", "Save", "Cancel");
audioMemoRecorder.pause();
audioMemoRecorder.cleanup();
...
}
Moreover if I display the available mime types on iOS, it yields "audio/amr" which I doubt according to all the posts I could read that tell you amr is not supported on iOS. Looking at the source it appears amr is the by default mime type because it is always returned :
/**
* Gets the available recording MimeTypes
*/
public String [] getAvailableRecordingMimeTypes(){
return new String[]{"audio/amr"};
}
So my question : is it possible to record audio on iOS, and if it is, how can it be done ?
Any help appreciated,
Have you looked at the Capture class? That seems to be more straightforward.
https://www.codenameone.com/javadoc/index.html
Ok I got it working by overloading some methods of the MediaManager, namely getAvailableRecordingMimeTypes() and also createMediaRecorder() to prevent it from using its getAvailableRecordingMimeTypes method.
Here is the code for getAvailableRecordingMimeTypes():
/**
* Normally the method returns amr even for ios. So we overload the original
* method to return aac on ios.
* #return
*/
public static String[] getAvailableRecordingMimeTypes() {
if (Display.getInstance().getPlatformName().equals("ios")) {
return new String[]{"audio/aac"};
} else {
return new String[]{"audio/amr"};
}
}
createMediaRecorder() is left as is (copied without changes).
Now it is possible to record audio in iOS and play it back in both iOS and Android!
everyone!
I have a trouble. I tried to save excel file in jsf web application.
I generated file by my utils and trying to get "save" window, but I failed.
Here is my code:
<div>
<h:commandButton value="Apply" actionListener="#{hornPhonesBean.generateReport}"/>
</div>
and:
public void generateReport(ActionEvent event) {
System.out.println("GENERATE REPORT FROM = " + this.dateFrom + "; TO = " + this.dateTo);
try {
XSSFWorkbook workbook = (XSSFWorkbook) HornReportGenerator.getWorkbook(null, null);
String fileName = "1.xlsx";
FacesContext fc = FacesContext.getCurrentInstance();
ExternalContext ec = fc.getExternalContext();
// Some JSF component library or some Filter might have set some headers in the buffer beforehand. We want to get rid of them, else it may collide.
ec.responseReset();
// Check http://www.w3schools.com/media/media_mimeref.asp for all types. Use if necessary ExternalContext#getMimeType() for auto-detection based on filename.
ec.setResponseContentType("application/vnd.ms-excel");
// Set it with the file size. This header is optional. It will work if it's omitted, but the download progress will be unknown.
//ec.setResponseContentLength(contentLength);
// The Save As popup magic is done here. You can give it any file name you want, this only won't work in MSIE, it will use current request URL as file name instead.
ec.setResponseHeader("Content-Disposition", "attachment; filename=\"" + fileName + "\"");
OutputStream output = ec.getResponseOutputStream();
workbook.write(output);
output.flush();
output.close();
fc.responseComplete(); // Important! Otherwise JSF will attempt to render the response which obviously will fail since it's already written with a file and closed.
System.out.println("END");
} catch (Exception e) {
e.printStackTrace();
}
}
I read suggestions here and from another forums - everyone says I shouldnt use , but I didn't use it at all.
Then I thought that the problem could be in the
<ice:form>,
where I kept the
<h:commandButton>,
and I changed to
<h:form>,
but it didn't help.
Maybe the problem in the request - it has header Faces-Request partial/ajax. But I am not sure.
Please give me some ideas - I already spent 4 hours for this crazy jsf download issue)
Maybe the problem in the request - it has header Faces-Request partial/ajax. But I am not sure.
This suggests that the request is an ajax request. You can't download files by ajax. Ajax requests are processed by JavaScript which has for obvious security reasons no facilities to programmatically pop a Save As dialogue nor to access/manipulate client's disk file system.
Your code snippet does however not show that you're using ajax. Perhaps you oversimplified it too much or you're using ICEfaces which silently auto-enables ajax on all standard JSF command components.
In any case, you need to make sure that it's not sending an ajax request.
See also:
How to provide a file download from a JSF backing bean?
ICEfaces libary in classpath prevents Save As dialog from popping up on file download
Here is some background about my app:
I am developing an Android app that will display a random quote or verse to the user. For this I am using an SQLite database. The size of the DB would be approximately 5K to 10K records, possibly increasing to upto 1M in later versions as new quotes and verses are added. Thus the user would need to update the DB as and when newer versions are of the app or DB are released.
After reading through some forums online, there seem to be two feasible ways I could provide the DB:
1. Bundle it along with the .APK file of the app, or
2. Upload it to my app's website from where users will have to download it
I want to know which method would be better (if there is yet another approach other than these, please do let me know).
After pondering this problem for some time, I have these thoughts regarding the above approaches:
Approach 1:
Users will obtain the DB along with the app, and won't have to download it separately. Installation would thereby be easier. But, users will have to reinstall the app every time there is a new version of the DB. Also, if the DB is large, it will make the installable too cumbersome.
Approach 2:
Users will have to download the full DB from the website (although I can provide a small, sample version of the DB via Approach 1). But, the installer will be simpler and smaller in size. Also, I would be able to provide future versions of the DB easily for those who might not want newer versions of the app.
Could you please tell me from a technical and an administrative standpoint which approach would be the better one and why?
If there is a third or fourth approach better than either of these, please let me know.
Thank you!
Andruid
I built a similar app for Android which gets periodic updates with data from a government agency. It's fairly easy to build an Android compatible db off the device using perl or similar and download it to the phone from a website; and this works rather well, plus the user gets current data whenever they download the app. It's also supposed to be possible to throw the data onto the sdcard if you want to avoid using primary data storage space, which is a bigger concern for my app which has a ~6Mb database.
In order to make Android happy with the DB, I believe you have to do the following (I build my DB using perl).
$st = $db->prepare( "CREATE TABLE \"android_metadata\" (\"locale\" TEXT DEFAULT 'en_US')");
$st->execute();
$st = $db->prepare( "INSERT INTO \"android_metadata\" VALUES ('en_US')");
$st->execute();
I have an update activity which checks weather updates are available and if so presents an "update now" screen. The download process looks like this and lives in a DatabaseHelperClass.
public void downloadUpdate(final Handler handler, final UpdateActivity updateActivity) {
URL url;
try {
close();
File f = new File(getDatabasePath());
if (f.exists()) {
f.delete();
}
getReadableDatabase();
close();
url = new URL("http://yourserver.com/" + currentDbVersion + ".sqlite");
URLConnection urlconn = url.openConnection();
final int contentLength = urlconn.getContentLength();
Log.i(TAG, String.format("Download size %d", contentLength));
handler.post(new Runnable() {
public void run() {
updateActivity.setProgressMax(contentLength);
}
});
InputStream is = urlconn.getInputStream();
// Open the empty db as the output stream
OutputStream os = new FileOutputStream(f);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024 * 1000];
int written = 0;
int length = 0;
while (written < contentLength) {
length = is.read(buffer);
os.write(buffer, 0, length);
written += length;
final int currentprogress = written;
handler.post(new Runnable() {
public void run() {
Log.i(TAG, String.format("progress %d", currentprogress));
updateActivity.setCurrentProgress(currentprogress);
}
});
}
// Close the streams
os.flush();
os.close();
is.close();
Log.i(TAG, "Download complete");
openDatabase();
} catch (Exception e) {
Log.e(TAG, "bad things", e);
}
handler.post(new Runnable() {
public void run() {
updateActivity.refreshState(true);
}
});
}
Also note that I keep a version number in the filename of the db files, and a pointer to the current one in a text file on the server.
It sounds like your app and your db are tightly bound -- that is, the db is useless without the database and the database is useless without the app, so I'd say go ahead and put them both in the same .apk.
That being said, if you expect the db to change very slowly over time, but the app to change quicker, and you don't want your users to have to download the db with each new app revision, then you might want to unbundle them. To make this work, you can do one of two things:
Install them as separate applications, but make sure they share the same userID using the sharedUserId tag in the AndroidManifest.xml file.
Install them as separate applications, and create a ContentProvider for the database. This way other apps could make use of your database as well (if that is useful).
If you are going to store the db on your website then I would recommend that you just make rpc calls to your webserver and get data that way, so the device will never have to deal with a local database. Using a cache manager to avoid multiple lookups will help as well so pages will not have to lookup data each time a page reloads. Also if you need to update the data you do not have to send out a new app every time. Using HttpClient is pretty straight forward, if you need any examples please let me know