So yes, apparently it is possible to have long grid with lots of rows built with angular. But then a problem comes with data updates.
You see if I just get all (let's say 10 000 rows) and render them in my grid - that works. It just takes a few seconds initially. And
a) I don't have all the date up front
b) I need the grid to be responsive immediately.
I can just do that with throwing let's say only 100 rows at the beginning, and then slowly update data as it becomes available. And that turns out to be the problem. Everytime you push new rows into $scope.data - it blocks UI. So I need to be smart about these updates.
Maybe I should set an interval and update the data only every few seconds? That seems to be not working
Maybe I should somehow watch for mouse movements and once it stops moving - start/resume adding rows, once mouse-movement detected seize adding rows and wait for another chance? - What if user never stops moving the mouse? (say some sort of a psycho)
Experimenting with _.throtle and _.debounce didn't get me anywhere.
You guys have any ideas?
UPD: here's a crazy one: what if? instead of waiting till angular updates the DOM, create entire DOM structure in memory, right before the digest cycle (with no data) and then insert that HTML chunk (that would be faster, right?) And after that let angular do its magic, data should appear. Would that work?
You are going to run into performance issues when something changes even if you can get all those rows rendered to the DOM. And your user probably isn't going to scroll through 10000 rows anyway. I would just use pagination. e.g.:
<div ng-repeat="item in items | startFrom:currentPage*itemsPerPage | limitTo:itemsPerPage"></div>
If you really want to have everything on one page you could load the rows as the user scrolls. If you are interested in that solution checkout http://binarymuse.github.io/ngInfiniteScroll/
One of the things I've noticed that I stupidly used to ignore - the container has to have fixed height. That significantly makes updates faster. Although technically it doesn't solve the problem entirely
Related
So I'm running into an issue on Internet Explorer(works on all other browsers) where it seems like a directive's HTML is taking 20+ seconds to retrieve from the server. The page is set up like this:
<div class="groups-page">
...
<test-directive></test-directive>
<div class="groups">
<h1 ng-repeat="x in records">{{x}}</h1>
</div>
...
</div>
Records is array with about 5000 items. Looking at the networks tab, the groups-page HTML is loaded in several ms. Now since we have 5000 records, I'm assuming it takes a lot of time to render. As a result of this, I see that the test-directive HTML just sits there in the network tab and finally is "downloaded" after 20 seconds. Am I right in saying that I need to limit the amount of data bindings here and that's what is causing the test-directive HTML to take so long to download? If I set a limitTo on the ng-repeat, it makes it much faster the more I limit it.
The issue is most likely due to the sheer amount of work Angular has to do to generate and bind 5,000 copies of your template. What you will see in the browser is that your client downloads the data, and then as soon as it is loaded, it hangs for a while while it generates the necessary HTML to add to the DOM. Once it finishes, the page becomes responsive again and the elements are drawn on the screen. At very large amounts of data, your browser starts to slow down with the number of DOM elements it is keeping track of, even after it is painted.
You have several options to mitigate this, including the following:
Reduce the number of data-bindings in your template
Page your data so that you aren't showing so many at a time
Use "infinite scrolling" (performance gets worse as items are added to the DOM while you scroll)
Look into implementing virtual scrolling for your list (essentially having the DOM only render a portion of the actual results that shifts as you scroll).
I ran into the same issue myself, and ultimately went with option 4. It is probably the most work out of options above, but the solution is handling an array of 200,000 items quite smoothly. If you don't want to create your own virtual scroll, there are a handful of open source implementations out there now that are worth looking into.
I'm working on a single-page app where some parts are really slow. They're slow because I'm displaying 400 complex things in a repeater for the user to scroll through. Each thing is generated by a fairly complex directive that does a ton of data binding and expression evaluation, adds one or two click handlers, and displays a couple of images. In some cases, I also need a grayscale CSS filter on those images, but that really seems way too slow.
I have of course already turned most of my data binding into one-time data binding, but simply generating the 400 things for the first time is still slow. It's initially hidden through ng-if, which speeds it up when I'm not showing it, but once I do need to show it, everything waits 10 seconds for that to happen. I would like to load it in advance, but using ng-show instead of ng-if means the loading of the entire app has to wait for this.
What I would like, is to load the rest of the app, and then, while we wait for user input, start creating these 400 things so they're ready once I need to show them. I don't want the user to notice how slow this is.
Problem is, I have no idea how to do this. Any ideas?
Edit: Having thought about this (and discussed this with my wife), I'm seeing two options (that I conceptually understand, at least):
The easy, quick, boring and cowardly solution is to simply not show the 400 things at the same time, but cut them in pieces and show 40 at a time. That should make it a lot quicker, but also less nice, as the user needs to click around to access all the data.
The interesting solution is to write a new ng-repeat that generates the 400 transcluded copies of the template each in their own asynchronous event, so they don't block user interaction. I really like this idea, but there's one big downside: it's an ambitious idea with deep Angular magic, and I don't have much time available.
OK, not seeing your code structure, through Q&A, I'm trying to get clarification. If I understand you correctly, I believe the solution is to process your images asynchronously and remove reliance of creating/processing them on the fly when the view is visible (i.e. via clicking on a button/tab to 'show' the array 'view' and thus triggering the ng-repeat). BTW, this solution assumes the delays are because the images are being processed rather than because they are being shown.
METHOD 1 (less preferred)
To do this, it's best to create an 'ImageDataService' service, where it get's kicked off at application start, and proceeds with building this array and taking whatever time it needs asynchronously without caring what view is showing or not. This service will be injected into the proper view or directive controller--perhaps where the current ng-repeat scope is.
Meanwhile, you also need to change the directives inside your ng-repeat to get the pre-built data from the array provided by ImageDataService rather than actually doing the calculation at view time. If the data for that directive is not ready, the directive will show something like 'loading...' otherwise, the directive will simply show the prebuilt images. This way, it doesn't matter when ng-repeat is triggered (i.e. its view is showing), because it will just show what are processed and ready at that point and the display the rest as 'loading...'.
METHOD 2 (I prefer this)
Alternatively, and maybe better, you can forego creating a pre-processed array in a service as in METHOD 1 and instead modify your directives to process their data asynchronously by invoking a service method that returns an asynchronous promise like this:
(code inside a controller)
var promise = ImageDataService.processImage(directiveData);
promise.then(function() {...set the directive image attributes..})
Needless to say, the ImageDataService.processImage() method needs to return a promise when it is done processing the data. The directive, as usual, will show 'loading...' until then. Also, although ImageDataService no longer needs to pre-populate its array mentioned in METHOD 1, it might be a good idea to save the processed images to a similar array anyway to serve as cache and not reprocess them needlessly. NOTE, in this case you don't need to have processImage() inside a service--but it is really good 'separation of concerns' practice to reserve the work of asynchronous data processing (ajax, etc) within a service which can be injected app-wide rather than within a controller.
Either of these 2 general tacks should work.
When I started working on my current project I was given quite an arduous task - to build something that in essence suppose to replace big spreadsheet people use internally in my company.
That's why we I thought a paginated table would never work, and quite honestly I think pagination is stupid. Displaying dynamically changing data on a paginated table is lame. Say an item on page #2 with next data update can land on page whatever.
So we needed to build a grid with nice infinite scroll. Don't get me wrong, I've tried many different solutions. First, I built vanilla ng-repeat thing and tried using ng-infinite-scroll, and then ng-scroll from UI.Utils. That quickly get me to the point where scrolling became painfully slow, and I haven't even had used any crazy stuff like complicated cell templates, ng-ifs or filters. Very soon performance became my biggest pain. When I started adding stuff like resizable columns and custom cell templates, no browser could handle all those bindings anymore.
Then I tried ng-grid, and at first I kinda liked it - easy to use, it has a few nice features I needed, but soon I realized - ng-grid is awful. Current version stuffed with bugs, all contributors stopped fixing those and switched to work on a next version. And only God knows when that will be ready to use. ng-grid turned out to be pretty much worse than even vanilla ng-repeat.
I kept trying to find something better. trNgGrid looked good, but way too simplistic and doesn't offer features I was looking for out of the box.
ng-table didn't look much different from ng-grid, probably it would've caused me same performance issues.
And of course I needed to find a way to optimize bindings. Tried bind-once - wasn't satisfied, grid was still laggy. (upd: angular 1.3 offers {{::foo}} syntax for one-time binding)
Then I tried React. Initial experiment looked promising, but in order to build something more complicated I need to learn React specifics, besides that thing feels kinda non-anguleresque and who knows how to test directives built with angular+react. All my efforts to build nice automated testing failed - I couldn't find a way to make React and PhanthomJS to like each other (which is probably more Phantom's problem. is there better headless browser) Also React doesn't solve "appending to DOM" problem - when you push new elements into the data array, for a few milliseconds browser blocks the UI thread. That of course is completely different type of problem.
My colleague (who's working on server-side of things) after seeing my struggles, grumbled to me that I already spent too much, trying to solve performance problems. He made me to try SlickGrid, telling me stories how this is freakin zee best grid widget. I honestly tried it, and quickly wanted to burn my computer. That thing completely depends on jQuery and bunch of jQueryUI plugins and I refuse to suddenly drop to medieval times of web-development and lose all angular goodness. No, thank you.
Then I came by ux-angularjs-datagrid, and I really, really, really liked it. It uses some smart bad-ass algorithm to keep things very responsive. Project is young, yet looks very promising. I was able to build some basic grid with lots of rows (I mean huge number of rows) without straying too much from the way of angular zen and scrolling still smooth. Unfortunately it's not a complete grid widget solution - you won't have resizable columns and other things out of the box, documentation is somewhat lacking, etc.
Also I found this article, and had mixed feelings about it, these guys applied a few non-documented hacks to angular and most probably those would breaks with feature versions of angular.
Of course there are at least couple of paid options like Wijmo and Kendo UI. Those are compatible with angular, however examples shown are quite simple paginated tables and I'm not sure if it is worth even trying them. I might end-up having same performance issues. Also you can't selectively pay just for the grid widget, you have to buy entire suite - full of shit I probably never use.
So, finally to my question - is there good, guaranteed, less painful way to have nice grid with infinite scrolling? Can someone point to good examples, projects or web-pages? Is it safe to use ux-angularjs-datagrid or better to build my own thing using angular and react? Anybody ever tried Kendo or Wijmo grids?
Please! Don't vote for closing this question, I know there are a lot of similar questions on stackoverflow, and I read through almost every single one of them, yet the question remains open.
Maybe the problem is not with the existing widgets but more with the way you use it.
You have to understand that over 2000 bindings angular digest cycles can take too long for the UI to render smoothly. In the same idea the more html nodes you have on your page, the more memory you will use and you might reach the browser capacity to render so many nodes in a smooth way. This is one of the reason why people use this "lame" pagination.
At the end what you need to achieve to get something "smooth" is to limit the amount of displayed data on the page. To make it transparent you can do pagination on scroll.
This plunker shows you the idea, with smart-table. When scrolling down, the next page is loaded (you will have to implement the previous page when scrolling up). And at any time the maximum amount of rows is 40.
function getData(tableState) {
//here you could create a query string from tableState
//fake ajax call
$scope.isLoading = true;
$timeout(function () {
//if we reset (like after a search or an order)
if (tableState.pagination.start === 0) {
$scope.rowCollection = getAPage();
} else {
//we load more
$scope.rowCollection = $scope.rowCollection.concat(getAPage());
//remove first nodes if needed
if (lastStart < tableState.pagination.start && $scope.rowCollection.length > maxNodes) {
//remove the first nodes
$scope.rowCollection.splice(0, 20);
}
}
lastStart = tableState.pagination.start;
$scope.isLoading = false;
}, 1000);
}
This function is called whenever the user scroll down and reach a threshold (with throttle of course for performance reason)
but the important part is where you remove the first entries in the model if you have loaded more than a given amount of data.
I'd like to bring your attention towards Angular Grid. I had the exactly same problems as you said, so ended up writing (and sharing) my own grid widget. It can handle very large datasets and it has excellent scrolling.
Consider the following Plunker:
http://plnkr.co/edit/10Rs92cRulf9VtI3Bav6
It appears to work correctly, but if you look at the console, you can see that something is wrong. With just that little table, Angular has already hit the Digest loop limit of 10!
it isn't a problem in a small dataset like this one, but the actual production page has 10 different filters and can span 30+ days. As the data grows, the $digest looping is getting way, way out of hand.
What can I do to stop it? I have been pouring over the source code trying to figure out why the Table keeps getting digested over and over and over again, even though the data isn't changing at all. Am I missing something?
Filters are really for massaging data for the view.
The filter on your ng-repeat that loops and creates the date range is causing the issue. ng-repeat is calling that filter every. time. it. 'repeats' :) and from what I can gather you only need to create the range once at start-up (or when data changes). If you move the date range creation code out of a filter it should be more performant. You might even consider moving the range creation server side.
Forked your plunkr http://plnkr.co/edit/XTn7hjbQgV1DxcBlOYVJ.
Our Utilty has one single table, and it has 10 million to 50 million rows, There may be a case we need to show 50 million rows in a single page html client page, To show the rows in browser we use jQuery in UI.
To retrieve rows we use Hibernate and use Spring for MVC. I am looking for best practice in retrieving the rows and showing in UI. Should I retrieve a bulk of thousands rows or two thousand rows in Hibernate and buffer to Web Client or a best practice is there ?
The best practice is not to do this. It will explode the browser memory and rendering engine, and will take too much time to load.
Add a search form to your webapp, make the end user search for what he's interested about, and only display the N first search results, just like Google does.
Nobody is able to do anything meaningful with 50 million rows without searching anyway.
i think you should use scroll pagination (when user reaches to almost bottom of page makes ajax call and load data).
Just for example quick google example & demo
and if your data is tabular then you can use jQGrid
Handling a larger quantity of data in an application must be done via virtualization. While it's true that the user will be overwhelmed by millions of records, it's not exactly true that they can't do stuff with it, nor that such quantities of data are unfathomable.
In practice and depending on what you're doing you'll note that this limit crops up on you with just thousands of records. Which frankly is very little data. Data centric apps just need a different approach, altogether, if they are going to work in a browser and perform well.
The way we do this is quite simple but not all that straightforward.
It helps to decide on a fixed height, because you will need to know the max height of a scrollable container. You then render into this container the subset of records that can be visible at any given moment and position them accordingly (based on scroll events). There are more or less efficient ways of doing this.
The end goal remains the same. You basically need to cull everything which isn't directly visible on screen in such a way that the browser isn't paying the cost of memory and layout logic for the app to be responsive. This is common practice in game development, only the world that is visible right now on screen is ever present at any given moment. So that's what you need to do to get large quantities of stuff to behave well.
In the context of a browser, anything which attributes to memory use and layout/render cost needs to go away if it's isn't absolutely vital.
You can also stagger and smear recalculations so that you don't incur the cost of whatever is causing the app to degrade on every small update. The user can afford to wait 1 second, if the apps remains responsive.