1. The setup
I'm currently initiating a GET request to an S3 bucket (not important) to download a very large file using the browser fetch(). This file is, in it's stored form, raw and unusable binary data, not structured.
2. The task and problem
There are a few things I want to do on the client-side with this data:
I need to process this data as it streams into the client to perform transformations on it (decryption, for example).
Once the data is processed and downloaded, it might still not be of any immediate use to the user outside the context of the web UI. Maybe the data should stay stored within the web app's sandbox disk space unless a user explicitly exports it?
3. The question
Where can I store this blob of unstructured data in both or either of the use cases listed above? There appear to be many options but none that fit this use case precisely. Any thoughts?
EDIT:
I feel like an idiot. I totally forgot about the FileSystem API. I'll take a look and answer my own question with a pseudo-implementation of this works.
EDIT 2:
I feel the need to reiterate what I stated in 2.2 above:
within the web app's sandbox disk space
I don't care about accessing the user's whole file system. I just want a space I can work with large files in on disk, similar to the app space directories provided to mobile applications by Android and iOS.
If you want to save and process a file at client level, and Blob is not an option, you may consider File System Access API (https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API#writing_to_files), even if this will introduce an interaction with the user.
Another option would be to take the advantages of PWAs client-side storage (https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Client-side_storage), this is also about your application architecture.
Before to check if to process your file at client level can be done as you need with the existing technologies, check if you really need to do that because it is only option, or, instead, if you are able to move such logic at server level, depending on your use cases.
I have successfully implemented video sharing in my app using react native and firebase, but I want to ensure videos being stored are no more than 1080x1080 (maybe 720 depending how it looks).
Videos are max 8 seconds long, I am trying my best to keep them under 5MB each if possible. I was able to do some compressing on the client side (crop to square/trim), but I am hoping to be able to compress the videos even more without losing that quality via cloud functions (storage trigger).
After doing some looking around, it looks like Moviepy is a good option, but it use's python and I am not sure how I can use this script inside of a cloud function storage trigger.
Here is what that looks like:
//Not sure how this will import
import moviepy.editor as mp
//Can I get the video here from the bucket path in a cloud function?
clip = mp.VideoFileClip("video-stored.mp4")
clip_resized = clip.resize(height=1080) # make the height 1080px ( According to moviePy documenation The width is then computed so that the width/height ratio is conserved.)
//resize video, then we need to store it in the same location (same file path)
clip_resized.write_videofile("video-stored-resized.mp4")
I would love to hear some suggestions regarding video compressing via a cloud function and thoughts on using the above script/module with cloud functions.
At this point, firebase functions does not support languages other than node.js.
Thus, there are 2 solutions.
if you would like to keep using moviepy.
Writing a part to call moviepy-related apis when firebase storage is triggered in node.js and having the python api in any preferred environments. (I guess you should use pubsub provided by gcp to call the python apis)
writing all parts in node.js
There is a great module called fluent-ffmepg in node.js too but I know adding a watermark with the module is not as easy as moviepy is...
By the way, when I tried to combine vids in firebase functions, I was not able to make it maybe because of the limitation of the environment.
So, I personally recommend the very first solution: ofc it depends on your situation such as how big your video files are.
in short: i have some audio files in my local sqlitedatabse and want to play with with the native media player from android.
reffered to the documentation i can play an audio either by placing my audio file on the sd card or by streaming from server or by an URI.
There is no way to play an audio file by giving an byte array to the media player.
so my solution would be to build an CONTENTPROVIDER which lets my media player access the audio file in the database via an URI. I came up with that idea through this tutorial
http://www.vogella.com/articles/AndroidSQLite/article.html
Is this possible? Are there better ways to implement my issue?
If I understand your question correctly, you are proposing creating an artificial URL that you are then passing to the android system to be read, and you supply the data backing the URL by reading it from SQLite. That will probably work, and it solves some issues with content decoding, but it requires the creation of awkward URLs and sockets within the system. All that might not be the most reliable or efficient way to go.
The alternative is to decode the data yourself into PCM, and supply it to Android via an Audio Track. This would be superior because you aren't trying to create a hacked (and artificial) URL just to pipe a stream of data through, but it may require you to parse the data yourself and convert it to PCM. Depending on what formats you have in your database, this may be difficult and not worth it because Android libraries for these kinds of conversions are not as accessible. So, which way is best depends on your goals, but I think those are your two options.
I have to use BinaryResponseParser in my application.I dont know what it is the use of BinaryResponseParser. As I searched in web I got one information that "A BinaryResponseParser that sends callback events rather then build a large response"
Here what is the call back events in the response.Can any one clearly explain what is the call back event and how it was used in Solr.
If am not using BinaryResponseParser in my application means , what will be the effect ?
BinaryResponseParser is a parser for Solr responses serialized in a binary format as opposed to json or xml. The class has one method that can be of use to you (v. 3.6.x):
public NamedList<Object> processResponse(InputStream body, String encoding).
The advantage of using binary serialization is that your response size will be much smaller. This can be critical for the performance of live IR systems like Lucene/Solr - just imagine an auto-suggest service, which has to provide a list of suggestions for a user for every key-stroke. The response must be dellivered to the user bellow 100ms. If you don't use binary encoding, your response will be larger and consequently will take a bit longer to transfer over HTTP.
I suggest you take a look at SolrJ - a Java client for Solr that will possibly solve most of your problems.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I hear people writing these programs all the time and I know what they do, but how do they actually do it? I'm looking for general concepts.
Technically, screenscraping is any program that grabs the display data of another program and ingests it for it's own use.
Quite often, screenscaping refers to a web client that parses the HTML pages of targeted website to extract formatted data. This is done when a website does not offer an RSS feed or a REST API for accessing the data in a programmatic way.
One example of a library used for this purpose is Hpricot for Ruby, which is one of the better-architected HTML parsers used for screen scraping.
Lots of accurate answers here.
What nobody's said is don't do it!
Screen scraping is what you do when nobody's provided you with a reasonable machine-readable interface. It's hard to write, and brittle.
As an example, consider an RSS aggregator, then consider code that gets the same information by working through a normal human-oriented blog interface. Which one breaks when the blogger decides to change their layout?
Of course, sometimes you have no choice :(
In general a screen scraper is a program that captures output from a server program by mimicing the actions of a person sitting in front of the workstation using a browser or terminal access program. at certain key points the program would interpret the output and then take an action or extract certain amounts of information from the output.
Originally this was done with character/terminal outputs from mainframes for extracting data or updating systems that were archaic or not directly accessible to the end user. in modern terms it usually means parsing the output from an HTTP request to extract data or to take some other action. with the advent of web services this sort of thing should have died away, but not all apps provide a nice api to interact with.
A screen scraper downloads the html page, and pulls out the data interested either by searching for known tokens or parsing it as XML or some such.
In the early days of PC's, screen scrapers would emulate a terminal (e.g. IBM 3270) and pretend to be a user in order to interactively extract, update information on the mainframe. In more recent times, the concept is applied to any application that provides an interface via web pages.
With emergence of SOA, screenscraping is a convenient way in which to services enable applications that aren't. In those cases, the web page scraping is the more common approach taken.
Here's a tiny bit of screen scraping implemented in Javascript, using jQuery (not a common choice, mind you, since scraping is usually a client-server activity):
//Show My SO Reputation Score
var repval = $('span.reputation-score:first'); alert('StackOverflow User "' + repval.prev().attr('href').split('/').pop() + '" has (' + repval.html() + ') Reputation Points.');
If you run Firebug, copy the above code and paste it into the Console and see it in action right here on this Question page.
If SO changes the DOM structure / element class names / URI path conventions, all bets are off and it may not work any longer - that's the usual risk in screen scraping endeavors where there is no contract/understanding between parties (the scraper and the scrapee [yes I just invented a word]).
Technically, screenscraping is any program that grabs the display data of another program and ingests it for it's own use.In the early days of PC's, screen scrapers would emulate a terminal (e.g. IBM 3270) and pretend to be a user in order to interactively extract, update information on the mainframe. In more recent times, the concept is applied to any application that provides an interface via web pages.
With emergence of SOA, screenscraping is a convenient way in which to services enable applications that aren't. In those cases, the web page scraping is the more common approach taken.
Quite often, screenscaping refers to a web client that parses the HTML pages of targeted website to extract formatted data. This is done when a website does not offer an RSS feed or a REST API for accessing the data in a programmatic way.
Typically You have an HTML page that contains some data you want. What you do is you write a program that will fetch that web page and attempt to extract that data. This can be done with XML parsers, but for simple applications I prefer to use regular expressions to match a specific spot in the HTML and extract the necessary data. Sometimes it can be tricky to create a good regular expression, though, because the surrounding HTML appears multiple times in the document. You always want to match a unique item as close as you can to the data you need.
Screen scraping is what you do when nobody's provided you with a reasonable machine-readable interface. It's hard to write, and brittle.
As an example, consider an RSS aggregator, then consider code that gets the same information by working through a normal human-oriented blog interface. Which one breaks when the blogger decides to change their layout.
One example of a library used for this purpose is Hpricot for Ruby, which is one of the better-architected HTML parsers used for screen scraping.
You have an HTML page that contains some data you want. What you do is you write a program that will fetch that web page and attempt to extract that data. This can be done with XML parsers, but for simple applications I prefer to use regular expressions to match a specific spot in the HTML and extract the necessary data. Sometimes it can be tricky to create a good regular expression, though, because the surrounding HTML appears multiple times in the document. You always want to match a unique item as close as you can to the data you need.
Screen scraping is what you do when nobody's provided you with a reasonable machine-readable interface. It's hard to write, and brittle.
Not quite true. I don't think I'm exaggerating when I say that most developers do not have enough experience to write decents APIs. I've worked with screen scraping companies and often the APIs are so problematic (ranging from cryptic errors to bad results) and often don't give the full functionality that the website provides that it can be better to screen scrape (web scrape if you will). The extranet/website portals are used my more customers/brokers than API clients and thus are better supported. In big companies changes to extranet portals etc.. are infrequent, usually because it was originally outsourced and now its just maintained. I refer more to screen scraping where the output is tailored, e.g. a flight on particular route and time, an insurance quote, a shipping quote etc..
In terms of doing it, it can be as simple as web client to pull the page contents into a string and using a series of regular expressions to extract the information you want.
string pageContents = new WebClient("www.stackoverflow.com").DownloadString();
int numberOfPosts = // regex match
Obviously in a large scale environment you'd be writing more robust code than the above.
A screen scraper downloads the html
page, and pulls out the data
interested either by searching for
known tokens or parsing it as XML or
some such.
That is cleaner approach than regex... in theory.., however in practice its not quite as easy, given that most documents will need normalized to XHTML before you can XPath through it, in the end we found the fine tuned regular expressions were more practical.