My problem is I'm making too many API requests, which I want to cut down if possible. Below I'll describe the situation:
I have three pages, all linked using ngRoute. Like this:
Page A: Teams (list of teams)
URL: "/teams"
Page B: Team Details (list of players)
URL: "/teams/team-details"
Page C: Player Details (list of player stats)
URL: "/teams/team-details/player-details"
Page A is populated by pulling an array of the teams from an API very easily using a simple $resource.query() request, and using ng-repeat to iterate through them.
Page B is populated by calling an html template and populating specific fields with values from a separate API request to the /team-details endpoint, taking the team_id value from the clicked element on Page A.
Page C (as with page B) takes a player_id from the clicked player on Page B and calls the /player-details endpoint using that value. This is yet another separate request.
This all works fine, but as you can imagine, a single user could quite easily rack up in excess of 100 API requests within an hour.
I have a request limit of 1000/hour, so if a mere 10 users are online at the same time, it could easily exceed my limit and shut down my API.
If I could access the API as one single master endpoint that outputted all data and subdata in one set, then that would solve my problem, but since I need to request separate endpoints I can't see how to do this.
Is there a better way to approach this? Or are these excessive API requests the only way?
Any help would be appreciated.
As far as I can see, Your model looks suitable for the application and meets how an API-driven application should work...
However, One potential cut-down you could make is to cache some of the results locally. i.e. store a local version of some of the data that is unlikely to change within a session. For example, If the number of teams is unlikely to change, then store the results of 1 API request locally and use that instead of recalling data from your API.
Following on from this route, you could choose to only update certain data after a certain time period. So, if a user has looked at some team-details then refuse to update this data for the next 10-20minutes. However, this does again depend how time-sensitive your data is.
Related
I have a react-redux application which:
Loads N records from the database depending on a "limit" query parameter (by default 20 records) on first application load (initialization)
Every 10 seconds app requests same (or newer) records from the database to update data in real time
If a user changes filters - app requests new records from the database according to the filter and re-renders app (+ changes interval to load data according to the filters)
If users scrolls down, the app automatically loads more records.
The problem is that if a user for and instance tries to filter something out and at this same time interval is loading more data, 2 requests can clash and overwrite each other. How in react-redux app I can be sure in a request sequence. Maybe there is a common approach on how to properly queue requests?
Thanks in advance!
I am not sure what you mean by 'clash'. My understanding is that the following will happen:
Assuming that both requests are successful, then data is retrieved for each of them, the redux state will be updated twice, and the component which renders the updated state will render twice (and the time passed between the two renders might be very short, which might not be very pleasant to the user)
If you want only one of these two requests to refresh the component, then a possible solution may be the following:
Each request starts, before retrieval of data from the database, by creating a 'RETRIEVAL_START' action. 'RETRIEVAL_START' will set a redux state variable 'retrievalInProgress'
If you want, in such a case, to get results only from the 1st of the two requests, you can check, before calling the action creator from the component, if 'retrievalInProgress' is on. If it is, don't call the action creator (in other words, do not request data when a request is in progress). 'retrievalInProgress' will be cleared upon successful or failed retrieval of data.
If you want to get results only from the 2nd of the two requests, then make 'retrievalInProgress' a counter, instead of a boolean. In the 'retrievalSuccess' action of the reducer, if this counter is higher than 1, it means that a new request already started. In this case, do not update the state, but decrement the counter.
I hope that this makes sense. I cannot be 100% sure that this works before I test it, which I am not going to do :), but this is the approach I would take.
. Requesting the page(on HTTP or WebPage), it is very slow or even crash unless i load my JSON with fewer data. I really need to solve this since sooner or later i will be using large amount of data frequently. Here are my JSON data. --->>>
Notes:
1. The JSON loads only String and Integer.
2. I used to view my JSON in JSONView more like treeview using plugin
from GoogleChrome.
I am using angular and nodejs. tq
A quick resume of all the things that comes to my mind :
I had a similar issue once. My solutions may make the UI change.
Pagination
I doubt you can display that much data at one time, so the strategy should be divising your data in small amounts and then only load more when the client ask for it.
This way, the whole data is no longer stored in RAM as it is currently. This is how forums works (only 20 topics at a time).
Just imagine if StackOverflow make you load the whole historic of questions in the main page, how much GB would your navigator need just for that ?
You can use pagination in a classic way (button with page number, like google), or in an infinite scroll way, as you want.
For that you need to adapt your api and keep track of the index of the pages you already loaded at every moment in your Front. There are plenty of examples in AngularJS.
Only show the beginning of the data
When you look at Facebook comments, you may have a "show more" button. In their case, maybe it's to not break the UI, but it can also be used to not overload data.
You can display only the main lines of your datae (titles or somewhat) and add a button so the user can load more details if they want.
In your data model, the cost seems to be on the second level of "C". Just load data untill this second level, and download the remaining part (for this object) only if the user asks for.
Once again, no need to overload, your client's RAM will be thankfull, and your client's mobile 3G too.
Optimize your data stucture
If this is still not enough :
As StefanArya said in comment, indeed remove the "I" attribute, which is redundant with the JSON key.
Remove the "I" as you can use Object.keys() to get key name.
You also may don't need that much precision on your floats.
If I see any other ideas, I'll edit this post later.
Many examples on the net show you how to use ng-repeat with in-memory data, but in my case I have long table with infinite scroll that gets data by sending requests to a REST API (scroll down - fetch some data, scroll down again - fetch some more data, etc.). It does work, but I'm wondering how can I integrate that with filters?
Right now I have to call a specific method of API service that makes a request based on text in "search" input box and then controller updates $scope.data.
Is it possible to build a custom filter that would do that? And then my view would be utterly decoupled from the service and I could declaratively tell it how to group and order and filter data, regardless if it's in-memory or comes from a remote server, server that can serve only limited records at a time.
Also later I'm gonna need grouping and ordering as well, I'm so tempted to download the entire dataset and lock parts of the app responsible for grouping, searching and ordering (until all data is on the client), but:
a) that dataset is huge (hundred thousands of records)
b) nobody wants to deal with cache invalidation headaches
c) doing so feels so damn wrong, you don't really expect me to 'keep' all that data in-memory, right?
Can you guys point me to maybe some open-source examples where I can steal some ideas from?
Basically I need to build a service and filters that let me to work with my "pageable" data that comes from api, like it's in memory-data.
Regardless of how you choose to solve it (there are many ways to infinite-scroll with angular, here is one: http://binarymuse.github.io/ngInfiniteScroll/), at its latest current beta version, ng-repeat works really bad with large amount of data - so do filters. The reason is obvious - pulling so much information for changes is a tuff job. Moreover, ng-repeat by default will re-draw your complete list every time something changes.
There are many solutions you can explore in this area, here are the ones I found productive:
http://kamilkp.github.io/angular-vs-repeat/#?tab=8
http://www.williambrownstreet.net/blog/2013/07/angularjs-my-solution-to-the-ng-repeat-performance-problem/
https://github.com/allaud/quick-ng-repeat
You should also consider the following, which really helps with large amounts of data.
https://github.com/Pasvaz/bindonce
Updated
I guess you can't really control your server output, because filtering and ordering large amount of data are better off done on the server side.
I was pointing out the links above since even if you write your own filters (and order-bys), which is quite simple to do - http://jsfiddle.net/gdefpfqL/ - (filter by some company name and then click the "Add More" button - to add more items). ordering by is virtually impossible if you can't control the data coming for the server - the only option is getting it all, ordering and then lazy load from the client's memory. So if each of your list items doesn't have many binding by it self (as in the example I've added) - the list item is a fairly simple one (for instance: you simply present the results as a plain text in a <li>{{item.name}}</li> then angular ng-repeat might work for you. In this case, filters will work as expected - say you filter by searched text:
<li ng-repeat="item in items | filter:searchedText"></li>
even for new items added after the user has searched a text, it will still works because the magic of binding.
I am trying to pass an object from one zul page to another. where :
Both zul pages have own composers.
I want to set value of object in 1st zul's composer.
And Want to get it in 2nd zul's composer.
I have tried executions.sendredirect(), but it clears the value of object, forward() thrwos an exception saying that "use sendRedirect instead to process user request".
Your problem is scope.
In ZK, like other web application frameworks, you have access to a number of different scopes: webapp, desktop, page, session, request, etc. If you have two different pages served from two different URLs, those will have distinct request scopes.
When moving from one request to another, you can pass information on the request URL:
Executions.sendRedirect("page2.zul?myId=1234")
Then, in the composer on page2, you can retrieve this value from the Execution:
Execution execution = Executions.getCurrent();
execution.getParameter("myId");
This is standard HTTP query string business so you're limited to text and numbers here. For passing things like database ids though, this can be quite convenient.
A more robust solution might be to leverage some of ZK's other scopes. For example, you could put your object in the user's Session scope. Refer to my answer to ZK session variable with a menu for implementation details. Note that when using the Session, you are no longer limited to text but can store an actual Object.
I have WCF server and silverlight client. The client call the server to get list of items.
There is some cases that the item list is very big and I want to have the ability to get the items in more then one call -
Call1 => get the items 0-100
Call2 ( if the user click on 'more' button ) => get the item 101-200
.
.
Call N => get the 100*n - 100*(n+1) items.
How can I do it ?
Is there some 'easy' pattern to do it ?
Thanks.
If you have a standard page size of 100 then have the client pass the page they want to the service. Or get the client to tell the service how big their pages are and which page they want
You could hold in memory on the service which page the client has and then have then say "Next" but holding in-memory state in the service on behalf of the client reduces scalability and increases fragility (if that state is lost then the client has to start paging again.
making the client explicitly say what they want is a more robust and scalable solution and has an easy LINQ implementation with Skip and Take
As Richard mentions, paging is a common option. Also, returning the results as a stream (and not a buffered byte[] array but an actual stream -- WCF has some caveats around use of stream) is going to generally be most efficient. Also as marc_s noted, Silverlight local storage isn't huge, so keep that handicap in mind.
The chance of a user 'consuming' more than 100 items in one go is very small, even if the items have very little detail, maybe add navigation (categories etc) as filters to the data so the user will only get the 20 or so items they are actually interested in. Tree views can be quite handy to break lists up into smaller lists that are more relevant to users, but there are many ways of doing this...