Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
In this post I am looking for some advice/clearification regarding performance for external db (or api) vs local storage, specifically for a VueJS/React application (but that doesn't matter).
I am developing a website where the user can list, sort and filter around 200 products. During development I have stored the data in vuex using object litteral format:
{
name: "Productname",
description: "Description of product",
price: 20,
student: true
}
My plan was to put the data in a noSQL database on production. The reasoing being it would give better performance and remove one thousand lines of code from the store. However, using (e.g.) mongoDB I would probably fetch all products on mount and put them in the store anyway.
If I assume all users will see all 200 products, is it a bad idea to fetch all 200 products (from api or db) to vuex/redux on mount? I mean would not risk loads of fetch calls (which could become costly) and data would be preloaded for the user. For this specific example we are talking under 10000 rows of ascii json (but I am also curious in general).
Lets say a user wants to filter for student-products, then could there be any benefits of doing that remotely? Why not do products.filter(p => p.student) in the store or a component (we already fetched all products)?
If 1 and 2 are true then why use an extarnal db? Is it mostly for maintaining the data stored in these places, for example adding/removing products, that we use them? Can this statement be made: "Yes, of course if you have X products then external storage is not needed" and if so then what is X?
It is considered a bad idea performance wise and network wise. The advantage of using an API in this case, is that you can limit the data you send and paginate it. Most of the times a user doesn't need to see 200 items at once and can happen that the item he wants to see/update/delete is in the first ones returned, that means you sent a lot of data that didn't need to be sent. That is why you have pagination or infinite scroll (when you get to the bottom of the page and it loads more data).
You could first filter for the data that is already fetched and if it doesn't return anything you then could do a call to an end route that you defined in your backend and query your db there to return the data the user is searching for.
A user can delete is localStorage and all the items go bye bye unless they are hard coded, in which case why even use localStorage, if your data is in a db and you took all the precautions to make it secure and build the API whithout security faults, then you could make sure that your users would always have data available to them. It doesn't really matter how much X is suppose to be, what really matters is: Would various users have access to the same data that needs to be the same for all of them? Can the users alter the data in any way?
This is really what I've learned and you need to think more about really what your application will do. Your state manager in the frontend should be considered more of a, well, state manager. It will manage the data you fetched so you can guarantee one source of truth for your application.
I hope this somewhat helps, and I would also appreciate if someone with more experience could explain it better or tell me why I'm wrong.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 12 months ago.
Improve this question
There are some blog-articles in project I work, and,
I guess, its text field should be limited somehow
(probably it would be a JSON field one day)
There are no limitations in sense of domain -
user can write as much as he wants, but,
just to prevent DB harm by hack-attacks with uncommonly huge amounts of text, guess, some limit is needed.
As SO Q/A says:
PostgreSQL limit text field with 1GB Is there a maximum length when storing into PostgreSQL TEXT
http POST limits depend on browser (2GB - 4GB) https://serverfault.com/questions/151090/
By rumors, Nginx's default client_max_body_size is 1MB
So, how to deal with all of that?
Probably, there are some practice like:
"Just limit it with million chars in app-lvl and don't worry"?
This is an interesting question. If I understand correctly you are working on an application where the DB entry will essentially be a blog post (typically 500-1000 words based on most blogs I read).
You are storing the blog post as a text field in your database. You are quite reasonably worried about what happens with large blobs of data.
I fully advocate for you having a limit on the amount of data the user can enter. Without fully understanding the architecture of your system it's impossible to say what is the theoretical max size based on the technologies used.
However, it is better to look at this from a user perspective. What is the maximum reasonable amount of text for you to have to store, then let's add a bit more say 10% because let's face it users will do the unexpected. You can then add an error condition when someone tried to enter more data.
My reason for proposing this approach is a simple one, once you have defined a maximum post size you can use boundary value analysis (testing just either side of the limit) to prove that your product behaves correctly both just below and at the limit. This way you will know and can explain the product behaviour to the users etc.
If you choose to let the architecture define the limit then you will have undefined behaviour. You will need to analyze each component in turn to work out the maximum size they will accept and the behaviour they exhibit when that size is exceeded.
Typically (in my experience) developers don't put this effort in and let the users do the testing for them. This of course is generally worse because the user reports a strange error message and then the debugging is ultimately time consuming and expensive.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I am working on an online tool that serves to a number of merchants(e.g. lets say retail merchants).This application takes data from different merchants and provides some data on their retail shop. The solution that I am trying to incorporate here is that any merchant can signup for the tool, send (may be upload through excel or my application can input a json object) their transaction and inventory data and in turn return the result to merchant.
My application consist of domain that is intrinsic to the application and contain all the datapoints that can be used by merchants, e.g
Product {
productId,
productName,
...
}
But the problem that I am facing is that, each merchant will have their own way of representing data, for e.g. merchant x may call product as prod or merchant y may call product
as proddt.
Now I would need to way to convert data represented in merchant format to a way that application understand, i.e each time there is a request from merchant x, application should map prod to product e.t.c e.t.c.
Firstly I was thinking of coding these mappers but then this is not a viable solution as I can't really code these mappings for 1000's of merchants that may join my application.
Another solution I was think was to enable the merchant to map a field from their domain to application domain through UI. And then save this somewhere in DB and on each request from merchant first find the mapping from db and then apply it over any incoming request.(Though I am still confused how this can be done).
Does anyone has faced similar design issue before and know of the better way of solving this problem.
if you can find the order of fields then you can easily map data send by your client and you can return result. for example in Excel you client can mention data in this format:
product | name | quantity | cost
condition: your ALL client should send data in this format.
then it will be easy for you to map these field and access then with correct DTO and later save and process data.
I appreciate this "language" concern, and -in fact- multi-lingual applications do it the way you describe. You need to standardize your terminology at your end, so that each term has only one meaning and only one word/term to describe it. You could even use mnemonics for that, e.g. for "favourite product" you use "Fav_Prod" in your app and in your DB. Then, when you present data to you customer, your app looks-up their preferred term for it in a look-up-table, and uses "favourite product" for customer one, and perhaps the admin, and then "favr prod" for customer two, etc...
Look at SQL and DB design, you'll find that this is a form of normalization.
Are you dealing with legacy systems and/or APIs at the customer end? If so, someone will indeed have to type in the data.
If you have 1000s of customers, but there are only 10..50 terms, it may best to let the customer, not you, set the terms.
You might be lucky, and be able to cluster customers together who use similar or close enough terminology. For new customers you could offer them a menu of terms that they can choose from.
If merchants were required to input their mapping with their data, your tool would not require a DB. In JSON, the input could be like the following:
input = {mapping: {ID: "productId", name: "productName"}, data: {productId: 0, productName: ""}}
Then, you could convert data represented in any merchant's format to your tool's format as follows:
ID = input.data[input.mapping.ID]
name = input.data[input.mapping.name]
To reacp:
You have an application
You want to load client data (merchants in this case) into your application
Your clients “own” and manage this data
There are N ways in which such client data can be managed, where N <= the number of possible clients
You will run out of money and your business will close before you can build support for all N models
Unless you are Microsoft, or Amazon, or Facebook, or otherwise have access to significant resources (time, people, money)
This may seem harsh, but it is pragmatic. You should assume NOTHING about how potential clients will be storing their data. Get anything wrong, your product will process and return bad results, and you will lose that client. Unless your clients are using the same data management tools—and possibly even then—their data structures and formats will differ, and could differ significantly.
Based on my not entirely limited experience, I see three possible ways to handle this.
1) Define your own way of modeling data. Require your customers to provide data in this format. Accept that this will limit your customer base.
2) Identify the most likely ways (models) your potential clients will be storing data (e.g. most common existing software systems they might be using for this.) Build import structures, formats to suppor these models. This, too, will limit your customer base.
3) Start with either of the above. Then, as part of your business model, agree to build out your system to support clients who sign up. If you already support their data model, great! If not, you will have to build it out. Maybe you can work the expense of this into what you charge them, maybe not. Your customer base will be limited by how efficiently you can add new and functional means of loading data to your system.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
We are moving a logging table from DB2 to Oracle, in here we log exceptions and warnings from many applications. With the move I want to accomplish 2 main things among others: less space consumption (to add more rows because tablespaces are kinda small) while not increasing the server processing usage too much (cpu usage increases our bill).
In DB2 we basically have a table that holds text strings.
In Oracle I am taking the approach of normalizing the tables for columns with duplicated data (event_type, machine, assemblies, versno). I have a procedure that receives multiple parameters and I query the reference tables to get the IDs.
This is the Oracle table description.
One of the feedback I have so far from a co-worker is that I will not necessary reduce table space since indexes take space and my solution might end up using more than what saving all string uses. We don't know if this is true, does anyone have more information on this?
I am taking the right approach?
Will this approach help me accomplish my 2 main goals?
Additional feedback is welcome and appreciated.
The approach of using surrogate keys (numerical ID) and dimension tables (containing the ID key and the description) is popular in both OLPT and data warehouse. IMO the use for logging is a bit strange.
The problem is, the the logging component should not have much assumption about the data to be logged - it is vital to be able to log the exeptional cases.
So in case that you can't map the host name to ID (it was misspelled or simple not configured), it would be not much helpfull to log unknownhost or to suppress the logging at all.
Concerned the storage you can indeed save a lot storing IDs istead of long strings, but (dependent on data) you may get similar effect using table compression.
I think the most important thing about logging is that it works all the time. It must be bullet-proof, because if the logger fails you lose the information you need to diagnose problems in your system. Even worse would be to fail business transactions because the logger abended.
There is a strong inverse relationship between the complexity of a logging implementation and its reliability. The more moving parts it has, the more things there are to go wrong. So, while normalization is a good thing for business data it introduces unwelcome risk. Also, look-ups increase the overhead of writing a log message and that is also undesirable.
" tablespaces are kinda small"
Increasing the chance of failure is not a good trade-off. Ask for more space.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
How to make such a SCANNER by making use of Access Database Table?
A lot of stock market websites give us the ability for scanning the stocks which fulfill particular criteria / conditions etc. based on different parameters. Just as shown in these snapshots –
http://finviz.com/screener.ashx?v=351&f=cap_smallover,ta_pattern_channelup2,ta_rsi_nob50,ta_sma20_pb&ft=4
[IMG]http://i58.tinypic.com/517x5i.png[/IMG]
[IMG]http://i62.tinypic.com/27ymp6u.png[/IMG]
I want to make something similar for my personal offline use. I have already got the required data for all the fields in my access database in one single table itself.
What are the various options available for creating such a Scanner, either inside access itself or in any other 3rd party platform. Because if it is not possible to design something similar inside Access, then I am also open to the idea of using access data table as a back end and using any 3 rd party app as a front end if that can do this work very efficiently.
I hope that it will give me the ability to apply 10+ filters easily at one go, just by using the drop down type of feature, without having to switch back into the query design mode again and again for changing the filter parameters.
My current method of using a simple query in which I apply all these filters, is taking me a lot of time and effort, for changing the filter values again and again. So the current method is extremely inefficient. Therefor I am looking for an easier way to get the final query output data, which shows the filtered results and in which I can very easily change the filter conditions quickly.
Any ideas in this regard are welcome.
Thanks a lot
You need to create a dynamic form and query. The best and easiest way is to create a form with all of your dropdowns on it. Then, in your query, in the Condition section for each field, you set it equal to
Forms!MyForm!MyDropdown1
Obviously you have to change the reference name to the actual name of your form and combo, but you get the idea. This makes your query completely dynamic and you'll never have to go in it and edit the conditions (aka filter values).
I'm trying to come up with a clean database design for a new project I'm working on. One of the data items I need to store in the database will come from an HTML form:
Q1: "Anticoagulated patient?" [YES, NO]
JavaScript;
(If yes is selected, an additional question is displayed):
Q2: "Type of Anticoagulant" [Warfari, Coumarin, Clopidogrel]
My question is, is it necessary to store the first question's response in the database? To me the data seems redundant. If the type is specified, then it can be assumed that the patient is "anticoagulated".
Once the form is submitted, the form will be accessed at a later point so the data can be ammended and the interface will need to reflect the state of the database. I should still be able to do this without needing to record the first question:
JavaScript;(If Q2 has a value, then the default option should be set to Yes
otherwise it should be set to No)
Q1: "Anticoagulated patient?" [Yes, No]
JavaScript;(Only display if Q1 is set to Yes):
Q2: "Type of Anticoagulant" [Warfari, Coumarin, Clopidogrel]
What are your thoughts on this?
I would say the extra space required to store the response to question 1 will be far outweighed by the amount of extra logic involved in marking Q2 as implying Q1. Keep it simple!
I think the two questions should indeed be handled on the client-side, as you suggested. This will provide a better user experience.
As for the database, don't use two fields, one is more than enough. Note that even if you don't show the second question dynamically, you can still store the answer somewhere else (such as the web server session) and not the back-end database.
I think , you have make right decision on this, but just think about the future and extending the application logic , maybe you need to add another question later then you will have to store the question in db,
Briefly for current situation it is adequate to store the answer.