I am trying to use ui-scroll to provide dynamic scrolling through a large dataset.
The get method calls a back end function which queries a database. The query is "slow" so I'd like to disable queries while the scroll thumb is moving.
For example, suppose the viewport shows 10 rows and there are 100,000 rows in the database and we fetch 100 rows at a time.
If you quickly drag the thumb from the top to the bottom, ui-scroll currently makes lots and lots of requests for data. I'd like to skip all of the intermediate requests and only fetch data when the thumb stops moving.
I've fiddled with isLoading on the adapter but couldn't find a way to make this work.
Suggestions? Angular 1.4.1.
$scope.datasource = { };
$scope.datasource.get = fetchData
$scope.datasource.minIndex = 0;
$scope.datasource.maxIndex = 10553;
$scope.scrollAdaptor = {};
var loadCount = 0;
function fetchData(desc, successCb)
{
var start = desc.index;
var end = desc.index + desc.count - 1;
var url = "scrollTest.php?start=" + start + "&end=" + end;
$http.get(url)
.then(function(goodResp)
{
console.log("got data, loadCount", loadCount);
successCb(goodResp.data);
},
function(badResp)
{
});
};
There is a discussion on GitHub related to the question. In short, angular-ui-scroll can't skip intermediate requests out-of-box, but following workaround might help.
Set min/max indexes on start to make the viewport relevant to
100,000 rows
$scope.datasource.minIndex = 1
$scope.datasource.maxIndex = 10000
Bind additional scroll event to intercept user scrolling over the viewport
Check scroll position delta between two events, and if it is more than some fixed value (say, 500px), rebuild the viewport
Rebuild means adapter.reload(indexToReload) method call and setting min/max indexes
The last thing is indexToReload, it should be calculated by current scroll position and item height:
indexToReload = Math.floor(scrollTop/itemHeight)
Related
I've written a CasperJS script that works very well except that it takes a (very very) long time to scrape pages.
In a nutshell, here's the pseudo code:
my functions to scrape the elements
my casper.start() to start the navigation and log in
casper.then() where I loop through an array and store my links
casper.thenOpen() to open each link and call my functions to scrap.
It works perfectly (and fast enough) for scraping a bunch of links. But when it comes to thousands (right now I'm running the script with an array of 100K links), the execution time is endless: the first 10K links have been scrapped in 3h54m10s and the following 10K in 2h18m27s.
I can explain a little bit the difference between the two 10K batches : the first includes the looping & storage of the array with the 100K links. From this point, the scripts only open pages to scrap them. However, I noticed the array was ready to go after roughly 30 minutes so it doesn't explain exactly the time gap.
I've placed my casper.thenOpen() in the for loop hoping that after each new link built and stored in the array, the scrapping will happen. Now, I'm sure I've failed this but will it change anything in terms of performance ?
That's the only lead I have in mind right now and I'd be very thankful if anyone is willing to share his/her best practices to reduce significantly the running time of the script's execution (shouldn't be hard!).
EDIT #1
Here's my code below:
var casper = require('casper').create();
var fs = require('fs');
// This array maintains a list of links to each HOL profile
// Example of a valid URL: https://myurl.com/list/74832
var root = 'https://myurl.com/list/';
var end = 0;
var limit = 100000;
var scrapedRows = [];
// Returns the selector element property if the selector exists but otherwise returns defaultValue
function querySelectorGet(selector, property, defaultValue) {
var item = document.querySelector(selector);
item = item ? item[property] : defaultValue;
return item;
}
// Scraping function
function scrapDetails(querySelectorGet) {
var info1 = querySelectorGet("div.classA h1", 'innerHTML', 'N/A').trim()
var info2 = querySelectorGet("a.classB span", 'innerHTML', 'N/A').trim()
var info3 = querySelectorGet("a.classC span", 'innerHTML', 'N/A').trim()
//For scraping different texts of the same kind (i.e: comments from users)
var commentsTags = document.querySelectorAll('div.classComments');
var comments = Array.prototype.map.call(commentsTags, function(e) {
return e.innerText;
})
// Return all the rest of the information as a JSON string
return {
info1: info1,
info2: info2,
info3: info3,
// There is no fixed number of comments & answers so we join them with a semicolon
comments : comments.join(' ; ')
};
}
casper.start('http://myurl.com/login', function() {
this.sendKeys('#username', 'username', {keepFocus: true});
this.sendKeys('#password', 'password', {keepFocus: true});
this.sendKeys('#password', casper.page.event.key.Enter, {keepFocus: true});
// Logged In
this.wait(3000,function(){
//Verify connection by printing welcome page's title
this.echo( 'Opened main site titled: ' + this.getTitle());
});
});
casper.then( function() {
//Quick summary
this.echo('# of links : ' + limit);
this.echo('scraping links ...')
for (var i = 0; i < limit; i++) {
// Building the urls to visit
var link = root + end;
// Visiting pages...
casper.thenOpen(link).then(function() {
// We pass the querySelectorGet method to use it within the webpage context
var row = this.evaluate(scrapDetails, querySelectorGet);
scrapedRows.push(row);
// Stats display
this.echo('Scraped row ' + scrapedRows.length + ' of ' + limit);
});
end++;
}
});
casper.then(function() {
fs.write('infos.json', JSON.stringify(scrapedRows), 'w')
});
casper.run( function() {
casper.exit();
});
At this point I probably have more questions than answers but let's try.
Is there a particular reason why you're using CasperJS and not Curl for example ? I can understand the need for CasperJS if you are going to scrape a site that uses Javascript for example. Or you want to take screenshots. Otherwise I would probably use Curl along with a scripting language like PHP or Python and take advantage of the built-in DOM parsing functions.
And you can of course use dedicated scraping tools like Scrapy. There are quite a few tools available.
Then the 'obvious' question: do you really need to have arrays that large ? What you are trying to achieve is not clear, I am assuming you will want to store the extracted links to a database or something. Isn't it possible to split the process in small batches ?
One thing that should help is to allocate sufficient memory by declaring a fixed-size array ie:
var theArray = new Array(1000);
Resizing the array constantly is bound to cause performance issues. Every time new items are added to the array, expensive memory allocation operations must take place in the background, and are repeated as the loop is being run.
Since you are not showing any code, so we cannot suggest meaningful improvements, just generalities.
I'm looking for a way to change the value (or format) for an individual series label when hovering it in anycharts.
Currently I'm only able to access the entire axis and I can find no getter method for individual labels so as to attach a listener.
xAxis.labels().listen('mouseOver', function(e) {
console.log(this, e.target);
});
This jsfiddle is as far as I got (see console log), this as well as the event.target reference the entire axis but not the label:
https://jsfiddle.net/robstarbuck/pbhd4b7L/9/
Indeed, there was a little bug with cache and format() function, our dev team made the fix, so please check the working sample:
var labelIndex = e.labelIndex;
var label = this.getLabel(labelIndex);
var value = xAxis.scale().ticks().get()[labelIndex];
label.format(value * 2);
https://jsfiddle.net/pbhd4b7L/13/ – it also shows how to work with tick values:
Currently it takes the js from branch, but this fix will be included in the upcoming release – 7.14.0 version (ETA: May 2017)
Our API is a little bit complicated here, but we're working hard to improve it. Does this what you're looking for?
var labelIndex = e.labelIndex;
var label = this.getLabel(labelIndex);
label.fontColor('red');
label.draw();
https://jsfiddle.net/pbhd4b7L/10/
This issue was fixed in the 7.14.0 release, use this code:
xAxis.labels().listen('mouseOver', function(e) {
var labelIndex = e.labelIndex;
var label = this.getLabel(labelIndex);
var value = xAxis.scale().ticks().get()[labelIndex];
label.format(value * 2);
label.fontColor('red');
label.draw();
});
with the latest version: https://jsfiddle.net/2t08ahkg/3/
I'm using AngularJS 1.5.0 and Microsoft Edge browser screen doesn't reflect the DOM.
I would like some suggestion how this can be fixed.
I can't really apply a fix for each element as the application is somewhat big with dynamic user content including Angular equations.
Also the app include a lot of dynamic bindings linked to input boxes.
Selecting the text with the mouse turn the 0 into a 2 which is the right value in the example below.
Also changing the position style back and forth seem to force Edge to redraw the element but it's somewhat an ugly fix that I don't like very much and it need to be trigger at so many places (Ajax request, input changes and so on...)
The page start with a value of 0. Then an Ajax call is made and it go fetch the real data. After some experimentation the bug only appears if the new data is 1 character (ex: 2 or 9). And it happens every time. If it's a 2 digit number (ex: 26) then the good number appears.
Any help on this matter would be gladly appreciated.
We were facing the same problem and it stopped occurring after removing the text-transform: uppercase style from the elements that are not updating.
#Sampson, it looks like this is a bug in Edge.
I was able to make a temporary fix with the use of Mutation Observer.
Little script that I made that works for AngularJS elements created server side.
Currently those are the only one causing trouble in my app.
<script>
$(document).ready(function(){
var ua = navigator.userAgent;
var re = new RegExp("Edge/");
if (re.exec(ua) != null){
console.log('edgeMutationObserver ON!!')
var edgeMutationConfig = { characterData: true, subtree: true };
var edgeMutationObserver = new MutationObserver(function(mutations) {
var rgb_p = /rgb\((\d+), (\d+), (\d+)\)/g;
var rgba_p = /rgba\((\d+), (\d+), (\d+), (\d.?\d*)\)/g;
mutations.forEach(function(mutation){
if(mutation.target.nodeType == 3){
var that = mutation.target.parentNode;
// Save the background color
var bgc = that.style.backgroundColor;
// Get applied style from style or css
var bgc_css = bgc || $(that).css('backgroundColor');
var bgc_temp,match;
if(match = rgb_p.exec(bgc_css)){
// rgb transformed in rgba triggers update
bgc_temp = 'rgba('+match[1]+','+match[2]+','+match[3]+',1)';
}else if(match = rgba_p.exec(bgc_css)){
// Slightly modify transparency
var alpha = match[4] == 0 ? 0.01 : match[4] - 0.01 ;
bgc_temp = 'rgba('+match[1]+','+match[2]+','+match[3]+','+alpha+')';
}else{
// If either inline style or css is already equal to transparent the redraw is not made so i choose an alternate color with 0 opacity
bgc_temp = bgc_css != 'transparent' ? 'transparent' : 'rgba(0,0,0,0)';
}
// Change background color to force redraw
that.style.backgroundColor = bgc_temp;
setTimeout(function(){
// Apply back previous style
// Doesn't redraw without a timeout in Edge
that.style.backgroundColor = bgc;
},0);
}
});
});
$('.ng-binding').each(function(){
edgeMutationObserver.observe(this, edgeMutationConfig);
});
}
});
</script>
I have been trying to get the scroll position of a tree grid in and ext.js application using sencha architect. Originally I wanted to keep the scroll position after a reload so when a user deletes a post it will scroll back to that same spot after reload. I have attempted to use various methods I have searched for but nothing is working. I'm currently testing by having a java script timeout function give me the scroll position after a few seconds, long enough for me to scroll down and console.log the position but it gives me nothing and I'm not showing anything but zero when I go into the dom and look at the tree view object. I have already tried the following
1st: var newcompheight = comboboxeample.getHeight();
2nd: getScroll: function() {
var me = this,
dom = me.dom,
doc = document,
body = doc.body,
docElement = doc.documentElement,
left, top;
if (dom === doc || dom === body) {
// the scrollLeft/scrollTop may be either on the body or documentElement,
// depending on browser. It is possible to use window.pageXOffset/pageYOffset
// in most modern browsers but this complicates things when in rtl mode because
// pageXOffset does not always behave the same as scrollLeft when direction is
// rtl. (e.g. pageXOffset can be an offset from the right, while scrollLeft
// is offset from the left, one can be positive and the other negative, etc.)
// To avoid adding an extra layer of feature detection in rtl mode to deal with
// these differences, it's best just to always use scrollLeft/scrollTop
left = docElement.scrollLeft || (body ? body.scrollLeft : 0);
top = docElement.scrollTop || (body ? body.scrollTop : 0);
} else {
left = dom.scrollLeft;
top = dom.scrollTop;
}
return {
left: left,
top: top
};
3rd: var test = window.pageYOffset;
I have also attempted to setting the keepscroll position property in Sencha architect to true and other things I have googled but nothing is working. Any help would be appreciated.
Use this:
treeGrid.getEl().getScrollTop()
I need is to dynamically add the points I’ve received from a forex provider to s Shield UI Chart. According to the documentation, there isn’t the possibility for dynamically adding of points At least there is no such method, something like: AddPoint or similar.
How can I still achieve a web page using Shield UI Chart, which to constantly show a couple of exchange rates?
You are right, that there is no addPoints method of the Shield UI Chart. However we can add the incoming data values to an array instead. You might find useful following code:
We need some arrays- as many as we need to show.
var EURUSD = new Array();
var USDCAD = new Array();
var GBPUSD = new Array();
In the body of the function, that will actually display the data we will have the following code:
EURUSD[EURUSD.length] = parseFloat(data.ticks.EURUSD);
USDCAD[USDCAD.length] = parseFloat(data.ticks.USDCAD);
GBPUSD[GBPUSD.length] = parseFloat(data.ticks.GBPUSD);
it will actually put the new data to the designated arrays. You can note, that each time data is received, it has been added to the last index of each array:
EURUSD.length
Since we don’t want our arrays to grow too large, it is good to specify how many points we need to keep. Once that limit is reached, we remove the oldest point:
if (EURUSD.length > 50)
EURUSD = EURUSD.splice(1, 49);
if (USDCAD.length > 50)
USDCAD = USDCAD.splice(1, 49);
if (GBPUSD.length > 50)
GBPUSD = GBPUSD.splice(1, 49);
At the end we need to recreate the chart, referencing the appropriate container:
var containter = $("#EURUSDChart").swidget();
containter.destroy();
and than create the chart again.