I am working on one application that produces timeline of user on every site of Stack Exchange. We have some users on SharePoint.SE.
Since I want list of all the sites, I am trying to get it using API 2.2
GET /2.2/sites/
As shown here. But it is getting only partial sites. The response is missing sites like Beer.SE, SharePoint.SE.
How do I retrieve them too?
You need to pass additional parameters. Beer.SE appears on page 3 with a pagesize of 100:
https://api.stackexchange.com/docs/sites#page=3&pagesize=100&filter=default&run=true
With no parameters, the default pagesize is 30 results. You can tell if there are more items than what is returned in your call by checking the has_more parameter in the common wrapper.
In this instance, with a pagesize of 3, there are 3 total pages and Beer.SE happens to be on the last page.
After rereading the sites documentation, it says
The pagesize parameter for this method is unbounded, in acknowledgement that for many applications repeatedly fetching from /sites would complicate start-up tasks needlessly.
This means that you can set your page size to something like 1000, and Beer.SE will appear in that results. At this point, there are not 1,000 Stack Exchange sites, so that will work.
Related
. Requesting the page(on HTTP or WebPage), it is very slow or even crash unless i load my JSON with fewer data. I really need to solve this since sooner or later i will be using large amount of data frequently. Here are my JSON data. --->>>
Notes:
1. The JSON loads only String and Integer.
2. I used to view my JSON in JSONView more like treeview using plugin
from GoogleChrome.
I am using angular and nodejs. tq
A quick resume of all the things that comes to my mind :
I had a similar issue once. My solutions may make the UI change.
Pagination
I doubt you can display that much data at one time, so the strategy should be divising your data in small amounts and then only load more when the client ask for it.
This way, the whole data is no longer stored in RAM as it is currently. This is how forums works (only 20 topics at a time).
Just imagine if StackOverflow make you load the whole historic of questions in the main page, how much GB would your navigator need just for that ?
You can use pagination in a classic way (button with page number, like google), or in an infinite scroll way, as you want.
For that you need to adapt your api and keep track of the index of the pages you already loaded at every moment in your Front. There are plenty of examples in AngularJS.
Only show the beginning of the data
When you look at Facebook comments, you may have a "show more" button. In their case, maybe it's to not break the UI, but it can also be used to not overload data.
You can display only the main lines of your datae (titles or somewhat) and add a button so the user can load more details if they want.
In your data model, the cost seems to be on the second level of "C". Just load data untill this second level, and download the remaining part (for this object) only if the user asks for.
Once again, no need to overload, your client's RAM will be thankfull, and your client's mobile 3G too.
Optimize your data stucture
If this is still not enough :
As StefanArya said in comment, indeed remove the "I" attribute, which is redundant with the JSON key.
Remove the "I" as you can use Object.keys() to get key name.
You also may don't need that much precision on your floats.
If I see any other ideas, I'll edit this post later.
My problem is I'm making too many API requests, which I want to cut down if possible. Below I'll describe the situation:
I have three pages, all linked using ngRoute. Like this:
Page A: Teams (list of teams)
URL: "/teams"
Page B: Team Details (list of players)
URL: "/teams/team-details"
Page C: Player Details (list of player stats)
URL: "/teams/team-details/player-details"
Page A is populated by pulling an array of the teams from an API very easily using a simple $resource.query() request, and using ng-repeat to iterate through them.
Page B is populated by calling an html template and populating specific fields with values from a separate API request to the /team-details endpoint, taking the team_id value from the clicked element on Page A.
Page C (as with page B) takes a player_id from the clicked player on Page B and calls the /player-details endpoint using that value. This is yet another separate request.
This all works fine, but as you can imagine, a single user could quite easily rack up in excess of 100 API requests within an hour.
I have a request limit of 1000/hour, so if a mere 10 users are online at the same time, it could easily exceed my limit and shut down my API.
If I could access the API as one single master endpoint that outputted all data and subdata in one set, then that would solve my problem, but since I need to request separate endpoints I can't see how to do this.
Is there a better way to approach this? Or are these excessive API requests the only way?
Any help would be appreciated.
As far as I can see, Your model looks suitable for the application and meets how an API-driven application should work...
However, One potential cut-down you could make is to cache some of the results locally. i.e. store a local version of some of the data that is unlikely to change within a session. For example, If the number of teams is unlikely to change, then store the results of 1 API request locally and use that instead of recalling data from your API.
Following on from this route, you could choose to only update certain data after a certain time period. So, if a user has looked at some team-details then refuse to update this data for the next 10-20minutes. However, this does again depend how time-sensitive your data is.
I want to implement a paging grid of people, similar to Gmail contacts, where the grid loads a maximum of N people's names, the paging bar says something like 'Allen - Baxter', and you can page through the alphabetical list. The main differences with the stock ExtJS pager is:
(a) This custom pager doesn't use a page number from beginning, but rather it uses an offset to begin the query (e.g. Baxter). On the server side, it actually queries names > Baxter.
(b)The pager won't know total count of people or pages, because the server doesn't check this. It just queries users > Baxter up to N people.
I guess you can sort of call it "infinite paging".
Is there a simpler approach to this problem other than writing a totally custom pager class (and possibly making changes to grid, store, and/or proxy classes)?
I came up with a working solution by creating my own toolbar class which is basically a copy-paste of Ext.toolbar.Paging that changes almost every method. The two client parameters I used to specify the type of request is: (1) direction, and (2) cursor. Those two parameters together can identify every type of paging request: (a) first, (b) next, (c) previous, (d) last. Other than the limit parameter, the class ignores the existing paging behavior built into ExtJs; it's an entirely different paradigm. And the server also includes a response parameter called hasMore, tells the client whether there are more items in the direction it is paging. So the server then always queries one more item than the limit to determine whether there is more.
adjusted bounce is defined as 'bounces excluding the single page visits that have finished some interactions on the page'; for instance, single page visits that stay longer than 15 seconds, or trigger an ajax event; I want these visits to be excluded from bounce rate calculation
I know in GA it can be adjusted quite easily by customizing the GATC, but I haven't found anything about Sitecatalyst, does anyone have any idea?
thanks!
In SiteCatalyst, a "bounce" is considered a single image request (no other calls, not even custom links). Based on what you're looking for, I'd anticipate this is the segment you're looking for:
Include all visits where time spent on page is greater than 15 seconds
Exclude all visits where path length is greater than 1
Keep in mind this cannot include single page visits that did not send a custom link image request, nor can it track visits where more than one s.t() call was fired. It only includes visits where a single s.t() was fired, and at least one custom link.
I have WCF server and silverlight client. The client call the server to get list of items.
There is some cases that the item list is very big and I want to have the ability to get the items in more then one call -
Call1 => get the items 0-100
Call2 ( if the user click on 'more' button ) => get the item 101-200
.
.
Call N => get the 100*n - 100*(n+1) items.
How can I do it ?
Is there some 'easy' pattern to do it ?
Thanks.
If you have a standard page size of 100 then have the client pass the page they want to the service. Or get the client to tell the service how big their pages are and which page they want
You could hold in memory on the service which page the client has and then have then say "Next" but holding in-memory state in the service on behalf of the client reduces scalability and increases fragility (if that state is lost then the client has to start paging again.
making the client explicitly say what they want is a more robust and scalable solution and has an easy LINQ implementation with Skip and Take
As Richard mentions, paging is a common option. Also, returning the results as a stream (and not a buffered byte[] array but an actual stream -- WCF has some caveats around use of stream) is going to generally be most efficient. Also as marc_s noted, Silverlight local storage isn't huge, so keep that handicap in mind.
The chance of a user 'consuming' more than 100 items in one go is very small, even if the items have very little detail, maybe add navigation (categories etc) as filters to the data so the user will only get the 20 or so items they are actually interested in. Tree views can be quite handy to break lists up into smaller lists that are more relevant to users, but there are many ways of doing this...