We have been successfully using the Realtime API in production for a few months, and now we want to change file formats. Our initial file format was, for example,
{
version: 1,
originalKey: customCollaborativeObject
}
and now we want to update our file format to
{
version: 2,
newKey: differentCustomCollaborativeObject
}
This is a difficult challenge because we don't have access to the files until users open them. To make matters worse, a user may simultaneously have two versions of our code open at the same time.
Does the Google Realtime team, or anyone else, have any best practices published for a change in format like this?
Related
In the past, I have built a custom component (REACT based) for the Streamlit-Framework which lets the user record audio inside a web browser of choice. Please have a look at the current version of streamlit-audio-recorder here.
As a mediocre web developer, I did not succeed in converting audio data stored in the browser's cache (audio-blob object) so that I can return it to the Streamlit backend.
What I have tried so far & my thought process:
There exist various scripts that enable saving audio to a local disk. However, none of these solutions work in an online-deployed scenario. (the program would save to the server's disk instead of the user's). This is why I came to the conclusion that this issue requires a solution that uses the audio data which is stored in the user's browser cache after being recorded.
The data stored in this cache via the audio-blob format, can not directly be passed back to python as a return variable and needs to be converted to an "environment agnostic datatype" (I tried binary base64). This conversion's complexity scales exponentially with the length of the audio data. Therefore I considered splitting the audio-blob into slices which can then be converted, aggregated and returned to Python. However, this process of splitting and concatenating WAV-audio blobs was not possible for me to implement due to the data structure/metadata inside the wav-file and the lack of libraries that would enable audio-blob slicing etc.
Does somebody know of a more elegant and performant solution? This would enable to finalize the audio recorder component and provide immense value to the Streamlit community which currently lacks comparable functionality.
I have a python script that fetches data twice a day from a server of mine. The script returns around 40 JSON files containing various data. The files aren't particularly big and the combined size of all the files is around 250KB.
Alongside my script I am developing a dashboard in React that renders the data from each file into a table, allowing me a visual representation of the data.
I have been looking at what would be the best way to store these files, something that allows me to upload and fetch them twice a day.
Someone mentioned to me about using MongoDB to store the files, but after some research I feel like Mongo is better at storing the contents of the file rather than the file itself. I tried to develop a solution but I couldn't figure out how it could be done when each object is stored as a document with no clear way (to me) which document came from which file.
Other options I have considered are:
Storing the files on the server that is hosting my React project and rendering them locally as I am doing now during development
Storing the files using a provider such as AWS/Firebase
Storing them in a different database (I see SQL now support the storing of JSON files)
Are there any other solutions that you think would work best for this scenario? If so, why?
Hello,
Check about use of FTP server.
We have clients that send us data every 10 min via FTP that is inside XML files, then I have NodeJS back-end which read these files.
You can use it for your scenario with JSON files.
I am working on an electron project to keep inventory of a warehouse but I want to store the data on the client-side (on the client's desktop/laptop) and not on a cloud database. How do I do this? Is using an xlsx file a good idea to store the data. As it will come with an added bonus as the user can read the data outside the app if they want to in an excel sheet.
P. S: even if xslx is a way I would like to know other possible ways so I can choose which is more comfortable for me. Thank you.
Edit: sorry I forgot to mention that I might also have to store images in the data.
You have plenty of option. You can store json file and read it when application boot up. As this is node js related thing I would suggest you to use electron store
And xlsx is a good choice but that may be overkill if the thing you are storing is too simple. On windows you can store some settings in registry too. But I prefer the config version.
I have also used sqlite3 database for some app. In Android I believe many app uses sqlite approach to store local database.
I am trying to create a site where users can upload images, videos and other types of files.
I did some research and people seem to suggest that saving the files as BLOB in database is a Bad idea; instead, save the file paths in database.
My questions are, if I save the file paths in a database:
1. How do I generate the file names?
I thought about computing the MD5 value of the file name, but what if two files have the same name? Adding the username and time-stamp etc. to file name? Does it even make sense?
2. What is the best directory structure?
If a user uploads images at 12/17/2013, 12/18/2018, can I just put it in user_ABC/images/, then create time-stamped sub-directories 20131217, 20131218 etc. ? What is the best structure for all these stuff?
3. How do all these come together?
It seems like maintaining this system is such a pain, because the file system manipulation scripts are tightly coupled with the database operations(may also need the worry about database transactions? Say in one transaction I updated the database but failed to modify the file system so I need to roll back my database?).
And I think this system doesn't scale (what if my machine runs out of hard disk so I need to upload the files to a second machine? What if my contents are on a cluster?)
I think my real question is:
4. Is there any existing framework/design pattern/db that handles this problem?
What is the standard way of handling this kind of problems?
Thanks in advance for your answers.
I've actually asked this same question when I was designing a social website for food chefs. I decided to store the url of the image in a MySQL database along with recipe. If you plan on storing multiple images for one recipe, in my example, maybe having a comma separated value would work. When the recipe loaded on the page, I would fetch the image associated with that recipe onto the screen.
Since it was a hackathon and wasn't meant for production purposes, I didn't encode the file name into something unique. However, if I were developing for productional purposes, I would append the time-stamp to the media file name when storing it into the server and database/backend.
I believe what I've proposed is the best data structure of handling this scenario. Storing the image onto the server is not only faster, but it should also take less space. I have found that when converting a standard jpg file of reasonable resolution to base64 encoding, the encoded text file representation took 30% more space. There is also the time of encoding the file and decoding the file for storage and resolving when using some BLOB type of data format instead of straight up storing the file on the server.
Using some sort of backend server scripting like PHP, you'll be able to do some pretty neat stuff with the information you have available. Fetch the result from the database, and load it in from the page using HTML.
As far as I know, there isn't a standard way of fetching media from a database yet. Perhaps there will be one day.
There is not standard way to do that, it is different to the different application. The idea is you need generate a different Path+FileName for every upload, here is a way:
HashId = sha1(microsecond + random(1,1000000));
Path = /[user_id]/[HashId{0,2}]/[HashId{-2}];
FileName = HashId
A call to all Win32 developers... I'm developing an application in C using plain Win32. I wanted to ask about Windows development standards regarding these things:
Is there a standard Windows error log api? For example if my client uses my app and it crashes, I would like them to send me the error log and I would prefer this being a standard location so they can maybe access it with a standard Windows log utility.
My app needs to store settings information. I think the registry is the standard utility for this task. Is that right?
My app needs to store and retrieve files that it downloaded from the internet - images, executables etc. Is Application Data/myapp the standard location to store this type of information?
My app needs a very straight-forward database - I'm using CSV for this. I basically need to store and retrieve this type of data so I'm just serializing a .csv file from Application Data/myapp. Is there a better Windows standard way of doing this?
That's all for now :). Thanks!
Is there a standard Windows error log api?
There is the Windows Event Log, but I don't think you want a typical user having to go into it to extract your logged information.
You probably don't want to log by default, unless you're shipping questionable pre-release code. When a user is experiencing problems, then you have them turn logging on. In this case, I recommend placing the file somewhere that typical users have experience with, like My Documents.
By the way, if you're writing a standalone application and want the best possible information in the event of a crash, look into minidumps. Here is a Codeproject sample.
My app needs to store settings information
Yep, registry.
My app needs to store and retrieve files
Yes, App Data. Just be sure to use SHGetFolderPath and CSIDL_APPDATA.
My app needs a very straight-forward database
There's nothing wrong with CSV for simple data. You could store the data in XML and use MSXML to process it, if you prefer. I've used SQlite in the past when I needed fast, lightweight storage of more complicated data.