I've been working on project where useres are able to load some data from the CSV, but each client have CSV with different structure. I can't ask them to change structure of the file because they are using those somewhere else. All of those files contains data that i need and I want to ask once at first upload which columns contains data i expect and then save this information so i don't have to ask them again.
Do you know how to ask client to somehow point those columns for me ? And how i should store those informations to use it next time ?
Related
My basic workflow is this: I check an FTP server for a specific file. If the file exist, I pick up the file and sends it to a Blob Storage. My problem is this: I want to filter the file content, eg. remove first and last row since they dont contain any real data before I send it to the blob. The first row consist of a time stamp and the last row contains a "row count". The file contains comma separated fields. How do I accomplish this? Is it even possible?
Thanks
Ausgar
There is no simple solution for this problem. You can try converting csv to json, delete unnecessary data from it, and create a blob based on that json, but this is sounds harder than it should be.
Consider using Azure functions:
Azure Functions allows you to run small pieces of code (called
"functions") without worrying about application infrastructure. With
Azure Functions, the cloud infrastructure provides all the up-to-date
servers you need to keep your application running at scale.
It will be much easier to do such file manipulation there.
If I have a website, and every user can upload his or her file, after I get this file, how can I deal with this file.
Should I store it to a specific folder(like upload), and store the file path in a specific database(for example table upload_file)?
Or should I just store this file in the database, if I can, how can I apply this to reality??
I think the first way is much better, but can I store all upload files in the same folder, or should I make a folder for each user?
Maybe I can encrypt the file name with md5 or anothers?
I do not know in real website development, how can you deal with the file, and keep the relationship with the user?
Thanks for your help !!
If they are small, just store it in the database;
Otherwise store they in a floder
I work for a small publishing company with an internal website that displays a static HTML table of our published products.
We have a need to be able to list and sort published products (about 1-2 items are published per day) dynamically that is being fed from an Excel spreadsheet. The Excel spreadsheet is what we are currently using to maintain the data. The Excel spreadsheet is on a shared network drive available to the company.
I am familiar with AngularJS, ReactJS, and VueJS2 for front-end development and was wondering if I would be able to use one of those tools to consume a Excel file, parse it to JSON, and then display it dynamically on the client side.
Is something like this is possible?
When a user finishes editing the Excel sheet and saves it to the shared network drive, is there a script that would automatically save the data as JSON? I assume we would then simply have our Javascript framework reference and consume the saved JSON to populate its published products list.
Note: We are unable to use a relational database at this time (ie MySQL).
Part 1 - generating json from excel...
front-end technologies are not the way to go. You need to run a service that watches folder for change (like nodejs or python). Saving as csv instead of xls might make things easier as you may not need extra libraries to make sense of your xls file
Part 2, displaying json data...
Your browser, by default, cannot load a local json file. So you may need to run a server (again nodejs and python make this relatively easy) to host your json file.
there are many ways of presenting data these days, but without knowing some of your particular and based on the information you did share, looks like you've got a steep learning curve to get something like this going.
I am trying to create a site where users can upload images, videos and other types of files.
I did some research and people seem to suggest that saving the files as BLOB in database is a Bad idea; instead, save the file paths in database.
My questions are, if I save the file paths in a database:
1. How do I generate the file names?
I thought about computing the MD5 value of the file name, but what if two files have the same name? Adding the username and time-stamp etc. to file name? Does it even make sense?
2. What is the best directory structure?
If a user uploads images at 12/17/2013, 12/18/2018, can I just put it in user_ABC/images/, then create time-stamped sub-directories 20131217, 20131218 etc. ? What is the best structure for all these stuff?
3. How do all these come together?
It seems like maintaining this system is such a pain, because the file system manipulation scripts are tightly coupled with the database operations(may also need the worry about database transactions? Say in one transaction I updated the database but failed to modify the file system so I need to roll back my database?).
And I think this system doesn't scale (what if my machine runs out of hard disk so I need to upload the files to a second machine? What if my contents are on a cluster?)
I think my real question is:
4. Is there any existing framework/design pattern/db that handles this problem?
What is the standard way of handling this kind of problems?
Thanks in advance for your answers.
I've actually asked this same question when I was designing a social website for food chefs. I decided to store the url of the image in a MySQL database along with recipe. If you plan on storing multiple images for one recipe, in my example, maybe having a comma separated value would work. When the recipe loaded on the page, I would fetch the image associated with that recipe onto the screen.
Since it was a hackathon and wasn't meant for production purposes, I didn't encode the file name into something unique. However, if I were developing for productional purposes, I would append the time-stamp to the media file name when storing it into the server and database/backend.
I believe what I've proposed is the best data structure of handling this scenario. Storing the image onto the server is not only faster, but it should also take less space. I have found that when converting a standard jpg file of reasonable resolution to base64 encoding, the encoded text file representation took 30% more space. There is also the time of encoding the file and decoding the file for storage and resolving when using some BLOB type of data format instead of straight up storing the file on the server.
Using some sort of backend server scripting like PHP, you'll be able to do some pretty neat stuff with the information you have available. Fetch the result from the database, and load it in from the page using HTML.
As far as I know, there isn't a standard way of fetching media from a database yet. Perhaps there will be one day.
There is not standard way to do that, it is different to the different application. The idea is you need generate a different Path+FileName for every upload, here is a way:
HashId = sha1(microsecond + random(1,1000000));
Path = /[user_id]/[HashId{0,2}]/[HashId{-2}];
FileName = HashId
I want to save my form fields without DB and display this value in Grid.
I tried to output as JSON format. but its temporary. I want save as permanent value either in PHP or JSON structure.
You could save it to a text file on the server side. Just write server side processing to write data to a text file. I don't know why you would though. There are plenty of free open source SQL and non-sql databases out there that you can use. You could use couchdb or something similar to store your JSON data directly.