In a WPF application, there is a data set which is being used almost every screen through the application.
That data set is being created dynamically by background process like that
The Background process will fetch the data in regular interval and make the updated data available to access from every screen.
What I planning is to develop a Global Caching mechanism using
System.Runtime.Caching
I have 3 Data set which is about 2 thousand rows with multiple columns for each data set.
Just wondering whether I might fetch any problem in future for the large data number?
Just wondering whether I might fetch any problem in future for the large data number?
Whether this is an issue depends on how much memory there is and will be available on your client machines. If you target Any CPU or x64, the managed heap will grow as needed to basically consume all available address space on a 64-bit machine. Please refer to the docs for more information about the limits.
Storing 3 x 2000 objects in a desktop application shouldn't generally be an issue, unless the objects are huge in size.
Related
Let's say I had a web application where users could store up to a gigabyte of their own data. The pages and code are stored on an application server, and the data is stored on a database server. The user requests a page from the application server, and the application server gets the data for the page from the database server.
Simple enough, but what happens when the database server gets full but the amount of users my application has keeps growing? Say the database server had a terabyte hard drive, and 1000 users were using all their available space. The physical disk space left is dwindling fast and new users have nowhere to put their data.
I imagine this is a fairly common occurrence for nearly every web application ever, so how is this remedied? How do designers of highly scalable web applications overcome the requirement for massive data storage?
How can you remedy a full disk? You can stop service, or delete data or provision more storage.
This situation is very undesirable because there is no good alternative at the moment this happens. Stopping service is very expensive and not a solution at all. Deleting data should have happened when there was some headroom left. It should not happen at the last possible moment. Provisioning more storage can take a long time. It should have happened before the disk became full.
The way this is dealt with is to not let the disk become full in the first place. Monitor storage utilization and always keep enough of it available.
If it turns out that storing all data is too expensive you have a problem with the business model. This is not a technical issue. Decide what data is profitable to keep and automatically remove the rest.
I imagine this is a fairly common occurrence for nearly every web application ever
It is common to have to decide what to do with data. It is very uncommon, and a failure of operations, to let a disk become full.
How do designers of highly scalable web applications overcome the requirement for massive data storage?
They either do not overcome the requirement and store everything, or they decide what to delete.
My company maintains a number of large time series databases of process data. We implement a replica of a subset at a pseudo-central location. I access the data from my laptop. The data access over our internal WAN even to the pseudo-central server is fairly expensive (time).
I would like to cache data requests locally on my laptop so that when I access it for a second time I actually pull from a local db.
There is a fairly ugly client side DAO that I can wrap to maintain the cache but I'm unsure how I can get the "official" client applications to talk to the cache easily. I have the freedom to write my own "client" graphing/plotting system, and already have a custom application that does some data mining already implemented. The custom application dumps data into .csv files that are manually moved around on a very ad-hoc basis.
What is the best approach to this sort of caching/synchonization? What tools could implement the cache?
For further info, the raw data set I have estimated at approx 5-8Tb of RAW time series data per year, with at least half of the data being very compressible. I only want to cache say a few hundred Mb locally. When ad-hoc queries are made on the data it tends to be very repetitive over very small chunks of the data.
I am working on a financial database that I need to develop caching for. I have a MySQL database with a lot of raw, realtime data. This data is then provided over a HTTP API using Flask (Python).
Before the raw data is returned it is manipulated by my python code. This manipulation can involve a lot of data, therefore a caching system is in order.
The cached data never changes. For example, if someone queries for data for a time range of 2000-01-01 till now, the data will get manipulated, returned and stored in the cache as being the specifically manipulated data from 2000-01-01 till now. If the same manipulated data is queried again later, the cache will retrieve the values from 2000-01-01 till the last time it was queried, elimination the need for manipulation for that entire period. Then, it will manipulate the new data from that point till now, and add that to the cache too.
The data size shouldn't be enormous (under 5GB I would say at max).
I need to be able to retrieve from the cache using date ranges.
Which DB should I be looking it? MongoDB? Redis? CouchDB?
Thanks!
Using BigData solution for such a small data set seems like a waste and might still not yell the required latency.
It seems like what you need is not one of the BigData solution like MongoDB or CouchDB but a distributed Caching (or In Memory Data Grid).
One of the leading solution which (which I'm one of its contributors) seems like a perfect match for you needs is XAP Elastic Caching.
For more details see: http://www.gigaspaces.com/datagrid
And you can find a post describing exactly this case on how you can use DataGrid to scale MySQL: "Scaling MySQL" - http://www.gigaspaces.com/mysql
I have a small-scale WPF application using VB.net as the code behind and I want to add certain features, but i'm concerned about performance. I REALLY appreciate any responses especially if you could include beginner-friendly articles regarding this, but please help me so I can be at ease...
1) My app interacts with a third party database to display "realtime" data to the user. My proposed method is to create a background worker to query a database every 30 seconds and display the data. I query about 2,000 records all long integer type, store them in a dataset, and then use LINQ to create subsets of observable collections which the WPF controls are bound to.
Is this too intensive? how much memory am i using for 2,000 records of long int? Is the background worker querying every 30 seconds too taxing? will it crash eventually? Will it interfere with the users other daily work (excel, email, etc)?
2) If an application is constantly reading/writing from text files can that somehow be a detriment to the user if they are doing day to day work? I want the app to read/write text files, but I don't want it to somehow interfere with something else the person is doing since this app will be more of a "run in the background check it when I need to" app.
3) Is there a way to quantify how taxing a certain block of code, variable storage, or data storage will be to the end user? What is acceptable?
4) I have several list(of t) that I use as "global" lists where I can hit them from any window in my application to display data. Is there a way to quantify how much memory these lists take up? The lists range from lists of integers to lists of variables with dozens of properties. Can I somehow quantify how taxing this is on the app or the end user?
Thank you for any help and I will continue to search for articles to answer my questions
IF you really want/need to get into the details of memory usage of an application you should use a memory profiler:
http://memprofiler.com/ (commercial)
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/ (commercial)
http://www.jetbrains.com/profiler/ (commercial)
http://www.microsoft.com/download/en/details.aspx?id=16273 (free)
http://www.scitech.se/blog/ (commercial)
Your other questions are hard to answer since all relevant aspects are rather unknown:
what DB is used ?
how powerful is the machine running the DB server ?
how many users ?
etc.
On some things a performance profiler can help - for example the above mentioned memory profilers (esp. from RedGate / JetBrains etc.) usually are available in a packaged together with a performance profiler...
I will just try a few. A byte integer uses a byte of memory. An int32 uses 4 bytes. So 2000 Int32 would use 8 kb. If you have a query you need to run a lot and it takes 5-10 seconds you need to look close at that query and add any missing indexes. If this is dynamic data then with (nolock) may be OK and faster with less (no) locking. If the query is returning the same data for all users then I hope you don't have all users running the same query. You should have a two tier application where the server runs the query every x seconds and sends that answer to the multiple clients that request it. As for size of an object just add it up - a byte is a byte. You can put you app in debug and get a feel for which statements are fast and slow.
I am about to port a Windows Form app to WPF or Silverlight. The current application uses a cache to store SQL responses temporaily as well as for later use in order not to have to run the queries again. The local cache should be able to handle 1 to 4 GB.
1) Is the Internal Storage capable to handle this amount of data? A search has not given me a clear answer so far, many talk about a 1MB limit, some say storage is of size long.
2) SQLite has C# managed code port, but I am not sure if that is stable enough to use in a professional application. Any experience or opinion?
3) Is it possible to use the SQLite ADO.Net provider for the Isolated storage or would it be an idea to run a local server that is responsible for the cache only? Or any way to achieve this through the COM access?
4) Any file based db system that you can recommend as a substitute for SQLite in case nothing else does work?
Any other ideas welcome, I need the local cache. If not, I need to do the application in Silverlight AND WPF and I would like to avoid that.
Thanks!
Regarding your 1 question:
Is the Internal Storage capable to handle this amount of data? A
search has not given me a clear answer so far, many talk about a 1MB
limit, some say storage is of size long.
Basically, by default Silverlight apps are granted 1 Mb of storage but they can request an increase in its storage quota (see here and here for more details).
Hope this helps