I am trying to add offline usage to an app. I simply need all the work done by NSURLRequest / NSURLCache, while being able to choose exactly the disk storage location, so I can put it in "/Library/Application Support/whatever" where it won't ever be deleted (without forgetting the flag so it's not synced on iCloud / iTunes).
I feel like I have to do all the work myself and I run in a first issue. NSURLCache is keeping in memory a dictionary where the keys are the NSURLRequest and the values are the associated NSCachedURLResponse. I'm doing the same but then, I can't write this dictionary on the disk as it isn't made of basic types.
Do you have an idea on how to write on disk such a dictionary?
I am in a similar situation, I need a cache that can be used when the app is offline or untile the app parses new data.
AFAIK everyone would recommend you: https://github.com/steipete/SDURLCache.
But theoretically in iOS6 the NSURLConnection writes the cache to disk and you can use that cache as offline cache, but I still have to found out how.
So partial answer, will try to find out more and update the answer. :)
Related
1. The setup
I'm currently initiating a GET request to an S3 bucket (not important) to download a very large file using the browser fetch(). This file is, in it's stored form, raw and unusable binary data, not structured.
2. The task and problem
There are a few things I want to do on the client-side with this data:
I need to process this data as it streams into the client to perform transformations on it (decryption, for example).
Once the data is processed and downloaded, it might still not be of any immediate use to the user outside the context of the web UI. Maybe the data should stay stored within the web app's sandbox disk space unless a user explicitly exports it?
3. The question
Where can I store this blob of unstructured data in both or either of the use cases listed above? There appear to be many options but none that fit this use case precisely. Any thoughts?
EDIT:
I feel like an idiot. I totally forgot about the FileSystem API. I'll take a look and answer my own question with a pseudo-implementation of this works.
EDIT 2:
I feel the need to reiterate what I stated in 2.2 above:
within the web app's sandbox disk space
I don't care about accessing the user's whole file system. I just want a space I can work with large files in on disk, similar to the app space directories provided to mobile applications by Android and iOS.
If you want to save and process a file at client level, and Blob is not an option, you may consider File System Access API (https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API#writing_to_files), even if this will introduce an interaction with the user.
Another option would be to take the advantages of PWAs client-side storage (https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Client-side_storage), this is also about your application architecture.
Before to check if to process your file at client level can be done as you need with the existing technologies, check if you really need to do that because it is only option, or, instead, if you are able to move such logic at server level, depending on your use cases.
I am working on an electron project to keep inventory of a warehouse but I want to store the data on the client-side (on the client's desktop/laptop) and not on a cloud database. How do I do this? Is using an xlsx file a good idea to store the data. As it will come with an added bonus as the user can read the data outside the app if they want to in an excel sheet.
P. S: even if xslx is a way I would like to know other possible ways so I can choose which is more comfortable for me. Thank you.
Edit: sorry I forgot to mention that I might also have to store images in the data.
You have plenty of option. You can store json file and read it when application boot up. As this is node js related thing I would suggest you to use electron store
And xlsx is a good choice but that may be overkill if the thing you are storing is too simple. On windows you can store some settings in registry too. But I prefer the config version.
I have also used sqlite3 database for some app. In Android I believe many app uses sqlite approach to store local database.
i have a multitenant app with a zipped package for each tenant/client which contains the templates and handlers for the public site for each of them. right now i have under 50 tenants and its fine to keep the imported apps in memory after the first request to that specific clients domain.
this approach works well but i have to redeploy the app with the new clients zipped package every time i make changes and/or a new client gets added.
now im working to make it possible to upload those packages via web upload and store them into the blobstore.
my concerns now are:
getting the packages from the blobstore is of course slower than importing a zipped package in the filesystem.
but this is not the biggest issue.
how do i load/import a module that is not in the filesystem and has no path?
if every clients package is around 1mb its not a problem as long as the client base is low but what if it raises
to 1k or even more? obviously there i dont have enough memory to store a few GB of data in memory.
what is the best way to deal with this?
if i use the instance memory to store the previously tenant package in memory how would
invalidate the data in memory if there would be a newly uploaded package?
i would appreciate some thougts about how to deal this kind of situation.
i agree with nick. there should be no python code in the tenant specific zip. to solve the memory issue i would cache most of the pages in the datastore. to serve them you don't need to have all tenants loaded in your instances. you might also wanna look in pre generating html views on save rather then on request.
In our Silverlight business application, we need to cache very large files (100's MBs) into isolated storage. We distribute the files separately to be downloaded by users and then they can import those files into Isolated Storage through the application.
However, the Isolated Storage API seem to be very slow and it takes an hour to import about 500MB of data.
Given, that we're in a corporate environment where users trust us, I would like users to be able to copy the files directly into the physical location on their file system where Silverlight store files when using the API.
The location varies per OS, but that's ok. The problem however, is that Silverlight seems to store files in a somewhat cryptic way. If I go to my AppData\LocalLow\Microsoft\Silverlight\is, I can see some weirdly named folder that look like long Guid.
My question: is it possible to copy files directly in there, or is it going to upset Silverlight ?
From what I've been testing it will make stuff fail/act weird. We had some stuff we had to clear and even though we did delete the files to test how it worked the usedspace didn't drop. So there is some sort of register of which files are in IS and how big they are.
I think it would be paramount you find out why IS is so slow. Can you confirm its like that on all clients? Test some others. This should be brought up to Microsoft if it is the case. Possibly you can change your serailization schema and save smaller files? I would not advise trying to figure out Microsoft's temporary and volatile IS storage location.
planning to launch a comic site which serves comic strips (images).
I have little prior experience to serving/caching images.
so these are my 2 methods i'm considering:
1. Using LinkProperty
class Comic(db.Model)
image_link = db.LinkProperty()
timestamp = db.DateTimeProperty(auto_now=True)
Advantages:
The images are get-ed from the disk space itself ( and disk space is cheap i take it?)
I can easily set up app.yaml with an expiration date to cache the content in user's browser
I can set up memcache to retrieve the entities faster (for high traffic)
2. Using BlobProperty
I used this tutorial , it worked pretty neat. http://code.google.com/appengine/articles/images.html
Side question: Can I say that using BlobProperty sort of "protects" my images from outside linkage? That means people can't just link directly to the comic strips
I have a few worries for method 2.
I can obviously memcache these entities for faster reads.
But then:
Is memcaching images a good thing? My images are large (100-200kb per image). I think memcache allows only up to 4 GB of cached data? Or is it 1 Mb per memcached entity, with unlimited entities...
What if appengine's memcache fails? -> Solution: I'd have to go back to the datastore.
How do I cache these images in the user's browser? If I was doing method no. 1, I could just easily add to my app.yaml the expiration date for the content, and pictures get cached user side.
would like to hear your thoughts.
Should I use method 1 or 2? method 1 sounds dead simple and straightforward, should I be wary of it?
[EDITED]
How do solve this dilemma?
Dilemma: The last thing I want to do is to prevent people from getting the direct link to the image and putting it up on bit.ly because the user will automatically get directed to only the image on my server
( and not the advertising/content around it if the user had accessed it from the main page itself )
You're going to be using a lot of bandwidth to transfer all these images from the server to the clients (browsers). Remember appengine has a maximum number of files you can upload, I think it is 1000 but it may have increased recently. And if you want to control access to the files I do not think you can use option #1.
Option #2 is good, but your bandwidth and storage costs are going to be high if you have a lot of content. To solve this problem people usually turn to Content Delivery Networks (CDNs). Amazon S3 and edgecast.com are two such CDNs that support token based access urls. Meaning, you can generate a token in your appengine app that that is good for either the IP address, time, geography and some other criteria and then give your cdn url with this token to the requestor. The CDN serves your images and does the access checks based on the token. This will help you control access, but remember if there is a will, there is a way and you can't 100% secure anything - but you probably get reasonably close.
So instead of storing the content in appengine, you would store it on the cdn, and use appengine to create urls with tokens pointing to the content on the cdn.
Here are some links about the signed urls. I've used both of these :
http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#signed-urls
http://www.edgecast.com/edgecast_difference.htm - look at 'Content Security'
In terms of solving your dilemma, I think that there are a couple of alternatives:
you could cause the images to be
rendered in a Flash object that would
download the images from your server
in some kind of encrypted format that
it would know how to decode. This would
involve quite a bit of up-front work.
you could have a valid-one-time link
for the image. Each time that you
generated the surrounding web page,
the link to the image would be
generated randomly, and the
image-serving code would invalidate
that link after allowing it one time. If you
have a high-traffic web-site, this would be a very
resource-intensive scheme.
Really, though, you want to consider just how much work it is worth to force people to see ads, especially when a goodly number of them will be coming to your site via Firefox, and there's almost nothing that you can do to circumvent AdBlock.
In terms of choosing between your two methods, there are a couple of things to think about. With option one, where are are storing the images as static files, you will only be able to add new images by doing an appcfg.py update. Since AppEngine application do not allow you to write to the filesystem, you will need to add new images to your development code and do a code deployment. This might be difficult from a site management perspective. Also, serving the images form memcache would likely not offer you an improvement performance over having them served as static files.
Your second option, putting the images in the datastore does protect your images from linking only to the extent that you have some power to control through logic if they are served or not. The problem that you will encounter is that making that decision is difficult. Remember that HTTP is stateless, so finding a way to distinguish a request from a link that is external to your application and one that is internal to your application is going to require trickery.
My personal feeling is that jumping through hoops to make sure that people can't see your comics with seeing ads is solving the prolbem the wrong way. If the content that you are publishing is worth protecting, people will flock to your website to enjoy it anyway. Through high volumes of traffic, you will more than make up for anyone who directly links to your image, thus circumventing a few ad serves. Don't try to outsmart your consumers. Deliver outstanding content, and you will make plenty of money.
Your method #1 isn't practical: You'd need to upload a new version of your app for each new comic strip.
Your method #2 should work fine. It doesn't automatically "protect" your images from being hotlinked - they're still served up on a URL like any other image - but you can write whatever code you want in the image serving handler to try and prevent abuse.
A third option, and a variant of #2, is to use the new Blob API. Instead of storing the image itself in the datastore, you can store the blob key, and your image handler just instructs the blobstore infrastructure what image to serve.