I have a scheduled job that loops through all pages of a certain type and creates a block for each page and puts it in a ContentArea.
if (productPageClone.GeneralContentArea == null)
{
productPageClone.GeneralContentArea = new ContentArea();
}
var newBlockForArea = _contentRepository.GetDefault<CrossLinkContainerBlock>
(assetsFolderForPage.ContentLink, productPageClone.Language);
(newBlockForArea as IContent).Name = "newCrossLinkContainer";
var blockReference = _contentRepository.Save((newBlockForArea as IContent), SaveAction.Publish,
AccessLevel.NoAccess);
var newItem = new ContentAreaItem();
newItem.ContentLink = blockReference;
productPageClone.GeneralContentArea.Items.Add(newItem);
When the block is created it is published.
When the page is updated it is either saved or published depending on earlier status.
_contentRepository.Save(productPageClone, SaveAction.ForceCurrentVersion | SaveAction.Publish,
AccessLevel.NoAccess);`
Later when inspecting the page, the block is in the page's assets folder and the block is in the correct ContentArea and it renders correctly. The only problem is that when I edit the block, it says "This item is not used anywhere."
However, then I republish the page the block is in, and then edit the block, it says "Changes made here will affect at least 1 item" as it should.
I am using Episerver 11.11.2.0
I have run the scheduled job manually each time I've tested this.
Has anyone any idea why this is happening?
I found the solution after reading this page:
https://gregwiechec.com/2015/10/reindexing-soft-links/
After page that has the new block has been published, get the page's softLinks and re-index them:
var links = _contentSoftLinkIndexer.GetLinks(productPageClone);
_softLinkRepository.Save(productPageClone.ContentLink.ToReferenceWithoutVersion(),
productPageClone.Language, links, false);
Softlink-tools are imported like this:
private IContentSoftLinkRepository _softLinkRepository =
ServiceLocator.Current.GetInstance<IContentSoftLinkRepository>();
private ContentSoftLinkIndexer _contentSoftLinkIndexer =
ServiceLocator.Current.GetInstance<ContentSoftLinkIndexer>();
This will occur if your content area is null
Try the following
// Before adding the ContentAreaItem
if(productPageClone.GeneralContentArea == null)
{
productPageClone.GeneralContentArea = new ContentArea();
}
productPageClone.GeneralContentArea.Items.Add(newItem);
Related
I have a form with customer info that needs to be processed one transaction per page. I'm using the binding navigator to manage my pagination.
It works in all but some cases. In the cases where it doesn't work, I have to open a different window to look up information and return it to the main form. Here is the code for that:
// save current work
updateDataTable();
// Open a window and get new customer info
// CurrentCustomer is returned from the opened window
using (SqlConnection cx = new SqlConnection(GetConnectionString()))
{
DataRowView dataRow = (DataRowView)procBindingSource.Current;
dataRow.BeginEdit();
dataRow["CUSTOMER"] = CurrentCustomer;
dataRow.EndEdit();
updateDataItems();
SqlCommand cmd = new SqlCommand(
#" select acct_no from cust_processing where id = #id ", cx);
cmd.Parameters.AddWithValue("#id", (int)dataRow["ID"]);
cx.Open();
var results = cmd.ExecuteScalar();
if (results != null)
{
dataRow.BeginEdit();
dataRow["ACCT_NO"] = results.ToString();
dataRow.EndEdit();
updateDataItems(); <------ CONCURRENCY ERROR
}
}
The error I am getting is a concurrency error. I think that I have more than one version of the row possibly ? I thought I was making sure that I was on the most recent version of the row by calling updateDataTable(). I am the only user so I know I am creating the problem myself.
Here is my update method which is called when I change pages or save and exit or want to write the commit the data:
void updateDataItems()
{
this.procBindingSource.EndEdit();
this.procTableAdapter.Update(xyzDataSet);
xyzDataSet.AcceptChanges();
}
I have tried executing updateDataItems from various places such as after I assign dataRow["ACCT_NO"] = results.ToString() or before and after assigning that.
I'm pretty much down to guess and check so any thoughts, help and advice will be appreciated and +1.
Okay -- so the problem was that I was trying to update the current row from the program and also using the binding navigator. They were not working together properly.
The solution was to add a text box to the form in the forms designer and set visible = false and bind it to ACCT_NO. Once I got the results from my other form, I just needed to set the .text property of the ACCT_NO textbox to the new value and the binding navigator managed all my updates for me correctly.
txtAcct_No.text = results.ToString();
I've written a CasperJS script that works very well except that it takes a (very very) long time to scrape pages.
In a nutshell, here's the pseudo code:
my functions to scrape the elements
my casper.start() to start the navigation and log in
casper.then() where I loop through an array and store my links
casper.thenOpen() to open each link and call my functions to scrap.
It works perfectly (and fast enough) for scraping a bunch of links. But when it comes to thousands (right now I'm running the script with an array of 100K links), the execution time is endless: the first 10K links have been scrapped in 3h54m10s and the following 10K in 2h18m27s.
I can explain a little bit the difference between the two 10K batches : the first includes the looping & storage of the array with the 100K links. From this point, the scripts only open pages to scrap them. However, I noticed the array was ready to go after roughly 30 minutes so it doesn't explain exactly the time gap.
I've placed my casper.thenOpen() in the for loop hoping that after each new link built and stored in the array, the scrapping will happen. Now, I'm sure I've failed this but will it change anything in terms of performance ?
That's the only lead I have in mind right now and I'd be very thankful if anyone is willing to share his/her best practices to reduce significantly the running time of the script's execution (shouldn't be hard!).
EDIT #1
Here's my code below:
var casper = require('casper').create();
var fs = require('fs');
// This array maintains a list of links to each HOL profile
// Example of a valid URL: https://myurl.com/list/74832
var root = 'https://myurl.com/list/';
var end = 0;
var limit = 100000;
var scrapedRows = [];
// Returns the selector element property if the selector exists but otherwise returns defaultValue
function querySelectorGet(selector, property, defaultValue) {
var item = document.querySelector(selector);
item = item ? item[property] : defaultValue;
return item;
}
// Scraping function
function scrapDetails(querySelectorGet) {
var info1 = querySelectorGet("div.classA h1", 'innerHTML', 'N/A').trim()
var info2 = querySelectorGet("a.classB span", 'innerHTML', 'N/A').trim()
var info3 = querySelectorGet("a.classC span", 'innerHTML', 'N/A').trim()
//For scraping different texts of the same kind (i.e: comments from users)
var commentsTags = document.querySelectorAll('div.classComments');
var comments = Array.prototype.map.call(commentsTags, function(e) {
return e.innerText;
})
// Return all the rest of the information as a JSON string
return {
info1: info1,
info2: info2,
info3: info3,
// There is no fixed number of comments & answers so we join them with a semicolon
comments : comments.join(' ; ')
};
}
casper.start('http://myurl.com/login', function() {
this.sendKeys('#username', 'username', {keepFocus: true});
this.sendKeys('#password', 'password', {keepFocus: true});
this.sendKeys('#password', casper.page.event.key.Enter, {keepFocus: true});
// Logged In
this.wait(3000,function(){
//Verify connection by printing welcome page's title
this.echo( 'Opened main site titled: ' + this.getTitle());
});
});
casper.then( function() {
//Quick summary
this.echo('# of links : ' + limit);
this.echo('scraping links ...')
for (var i = 0; i < limit; i++) {
// Building the urls to visit
var link = root + end;
// Visiting pages...
casper.thenOpen(link).then(function() {
// We pass the querySelectorGet method to use it within the webpage context
var row = this.evaluate(scrapDetails, querySelectorGet);
scrapedRows.push(row);
// Stats display
this.echo('Scraped row ' + scrapedRows.length + ' of ' + limit);
});
end++;
}
});
casper.then(function() {
fs.write('infos.json', JSON.stringify(scrapedRows), 'w')
});
casper.run( function() {
casper.exit();
});
At this point I probably have more questions than answers but let's try.
Is there a particular reason why you're using CasperJS and not Curl for example ? I can understand the need for CasperJS if you are going to scrape a site that uses Javascript for example. Or you want to take screenshots. Otherwise I would probably use Curl along with a scripting language like PHP or Python and take advantage of the built-in DOM parsing functions.
And you can of course use dedicated scraping tools like Scrapy. There are quite a few tools available.
Then the 'obvious' question: do you really need to have arrays that large ? What you are trying to achieve is not clear, I am assuming you will want to store the extracted links to a database or something. Isn't it possible to split the process in small batches ?
One thing that should help is to allocate sufficient memory by declaring a fixed-size array ie:
var theArray = new Array(1000);
Resizing the array constantly is bound to cause performance issues. Every time new items are added to the array, expensive memory allocation operations must take place in the background, and are repeated as the loop is being run.
Since you are not showing any code, so we cannot suggest meaningful improvements, just generalities.
I am using Selenium webdriver.
I have to read a xpath of link from a file and search whether the link is present on webpage, If it is present then click on it.
Thats it!!
Heres the file for the links
link1 //a[contains(text(), 'Volunteer Registration')]/#href
link2 //a[contains(text(), 'Sign Up')]/#href
link3 //a[contains(text(), 'Register/sign Up')]/#href
Like wise I have one file from where I'll read one link and its associated xpath and based on that Xpath I'll search whether the link is present on webpage or not.
The Code I have written for that is :
Reading data from text file into Hashtable -
public HashMap<String, String> readDataFromFile(String fileName) {
try {
FileReader fr = new FileReader(fileName);
BufferedReader br = new BufferedReader(fr);
String strLine = null;
String[] prop = null;
while ((strLine = br.readLine()) != null) {
prop = strLine.split("\t");
recruiters.put(prop[0], prop[1]);
}
br.close();
fr.close();
} catch (Exception exception) {
System.out.println("Unable to read data from recruiter file: " + exception.getMessage());
}
return recruiters;
}
Method to return xpath value from hashtable based on the key
public String findValue(String Name){
for (String s: HashTable.entrySet()) {
Map.Entry entry = (Map.Entry) s;
if(entry.getKey().equale(Name))
{
String value= entry.getValue();
return value;
}
}
return null;
}
Now I want to write a method to just search whether the xpath related link is present on webpage or not..
Please help me with that..
The logic is something like
public void Search&ClickLink()
{
List<WebElement> links = driver.findElements(By.tagName("a"));
System.out.println(links.size());
for (WebElement myElement : links){
String link = myElement.getText();
System.out.println(link);
myElement.click();
driver.navigate().back();
}
But I am not sure about it.
Please let me know if the approch is correct also if this function is approprite.
Please suggest the better way to implement the code.
Thanks!!
Well...there are a couple of problems with that last set of code.
The first is that you are going to get StaleElementReferences. When you find an element (or a list of elements), they are pointing to a element to a page. If you refresh the page or leave and come back, they will not be valid, and you have to re-find all of your elements.
Also, many times a link doesn't navigate you to a new page. If this is the case with any of your links, you will suddenly find yourself clicking links on the wrong page (because you navigated back)
Finally, you aren't actually doing anything on the page. For all you know, the link could go to a 500 Error, and Selenium would have no idea.
However, since you have all of the links in a file, why not just read the file, store it in an array, and then do a simple for loop:
for (String linkName: allLinks){
driver.get(urlWithLinks);
driver.findElement(By.linkText(linkName)).click();
...validate the page...
}
Lastly...I personally believe that clicking on all of the links on a page is a terrible test. A much better test would be to go to the link, and the do stuff on the page. That way you are actually testing the functionality of the website.
So, I have a LoaderMax instance loading images from various URLs. I want to add all loaded images to an array.
Here's my code:
var photosArray:Array = new Array(5);
var imageLoadingQueue:LoaderMax = new LoaderMax({name:"mainQueue", onComplete:completeHandler});
for (var g:uint=0; g<5; g++)
{
imageLoadingQueue.append(new ImageLoader("/img" + g + ".jpg", {name:"photo", container:photosArray[g], noCache:false, smoothing:true, width:126, height:126, scaleMode:"proportionalOutside"}));
}
imageLoadingQueue.load();
private function completeHandler(e:LoaderEvent):void
{
trace("finished loading pictures!");
//the next two lines will return an error (saying that photosArray[1] is null)
stage.addChild(photosArray[1]);
photosArray[1].x = 250;
}
A few problems:
If I set the container of the image being loaded to the Array, it won't work. I'm not being able to access the image inside the array because it says it's null.
If I set the container of the image being loaded to "this" (using the container property when appending a new ImageLoader) and, on the completeHandler, set my array equal to event.target.content, it kinda works (but it's not the ideal). The problem is that, by doing so, the images are appearing on the stage as they are loaded, and I do no want them to do so.
Any help would be heavily appreciated.
Thanks!!
David is correct, but I also wanted to mention that the LoaderMax's "content" is actually an array of all of its children's content, so you could just use that for simplicity. Keep in mind that ImageLoaders automatically create a Sprite (technically called a "ContentDisplay") to drop the image into so you probably don't need to create ANOTHER Sprite (a container for the container).
var photos:Array = imageLoadingQueue.content;
stage.addChild(photos[1]);
The other nice thing is that it creates the ContentDisplay Sprites immediately, even before any content is loaded into them, so you can place them and size them however you want while (or before or after) loading occurs.
The container needs to be a DisplayObjectContainer. ImageLoader will try to add the image to the container using addChild(), so obviously this won't work with an empty array. Create a new Sprite for each image and add it into the array first:
for (var g:uint=0; g<5; g++)
{
photosArray[g] = new Sprite();
imageLoadingQueue.append(new ImageLoader("/img" + g + ".jpg", {name:"photo", container:photosArray[g], noCache:false, smoothing:true, width:126, height:126, scaleMode:"proportionalOutside"}));
}
I am trying to manipulate a requested document in the WPF WebBrowser-control. I already managed it to invoke JavaScript on loaded document, but I am not able to change the shown HTML-code in the control itself.
My (very simplified) code in the OnNavigating-Handler looks like this:
mshtml.HTMLDocument doc = (mshtml.HTMLDocument)View.browser.Document;
HTMLTableClass table = doc.getElementById("someTable") as HTMLTableClass;
if (table != null)
{
table.appendChild((IHTMLDOMNode)(doc.createElement("<tr>") as IHTMLElement));
}
doc.close();
The -element doesn't get appended to displayed document in the control.
Any hints are very appreciated!
I finally got it. Its only possible to change the content of the table by adding rows and cells which i wanted to avoid in first place. My approach was to directly change the content of the -tag, which didnt work.
mshtml.IHTMLTableRow row = table.IHTMLTable_insertRow(-1) as mshtml.IHTMLTableRow;
mshtml.IHTMLElement c = (mshtml.IHTMLElement)row.insertCell(0);
c.innerText = "some";
mshtml.IHTMLElement c1 = (mshtml.IHTMLElement)row.insertCell(1);
c1.innerText = "text";