I'm using Selenium's Python bindings for a quick scraping job, but have found that for whatever reason, the Firefox WebDriver becomes unresponsive after precisely nine iterations.
Basic operation consists of loading a page, selecting a state from a dropdown menu of all fifty, clicking through to the results page and then returning to select another state. Irrespective of where I start in the list of states, after iterating through nine pages, the Firefox WebDriver becomes unresponsive, though no errors are thrown.
Code in question below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
def set_up(url):
driver = webdriver.Firefox()
driver.get(url)
return(driver)
def search(driver):
for i in range(1, 49):
element = driver.find_element_by_id("StateList")
all_options = element.find_elements_by_tag_name("option")
print("Acquiring data for" + str(all_options[i].get_attribute("label")))
all_options[i].click()
driver.find_element_by_id("SearchButton").click()
#scrape page components here
driver.back()
url = 'http://www.example.com/'
driver = set_up(url)
search(driver)
Any thoughts?
Related
I am starting to use Selenium Webdriver (Chrome) and while pages open automatically keys are not being sent for some reason. For example, I open google.com, and I use a "BY.ID" to aim at the search box by it's ID and send "Hi" but for some reason, it doesn't send the keys, I tried looking at the Selenium documentation but It kinda left me more confused, anybody knows why keys aren't being sent? Thanks in advance!
from selenium import webdriver
from selenium.webdriver.common.by import By
browser = webdriver.Chrome('/Users/thras/Desktop/Chromedriver/chromedriver')
browser.get('https://www.youtube.com')
title = browser.title
browser.implicitly_wait(3)
text_box = browser.find_element(By.ID,"search").send_keys('Hi')
from selenium import webdriver
from selenium.webdriver.common.by import By
browser = webdriver.Chrome('/Users/thras/Desktop/Chromedriver/chromedriver')
browser.get('https://www.google.com')
title = browser.title
browser.implicitly_wait(3)
text_box = browser.find_element(By.NAME, value="q")
text_box.send_keys('Hiii')
fixed, it was syntax issue it seems, I needed to add "value" before the actual ID
Scenario: Unique application number(Zee1106) using to enroll students and running testng for Multiple browsers(parallel) like chrome, firefox in selenium webdriver.
In the above scenario, I have run the test suite,first browser(chrome) enrolled successfully and the next browser(firefox) is not enrolled. Because already enrolled alert was coming. In this scenario, How can i enroll students using unique application number for multiple browser in webdriver.
Thanks,
Vairamuthu
There are quite a few ways to achieve this. One easiest way is to have the "Application Numbers" stored as a comma separated values in your test data sheet and use each for the respective browsers. Example:
//Assume applicationNo is stored as a comma separated value in test data. Something like this
String applicationNo="zee1106, zee1107, zee1108"; //please read these data from test data sheet
String[] unquieAppNo=applicationNo.split(",");
//get the browserName
Capabilities caps = ((RemoteWebDriver) driver).getCapabilities();
String browserName = caps.getBrowserName();
//if the browser name is chrome then use one of the application id and so on for each browser.
if(browserName.equalsIgnoreCase("chrome")){
driver.findElement(By.id("<employee app number>")).sendKeys(unquieAppNo[0]);
}else if (browserName.equalsIgnoreCase("firefox")) {
driver.findElement(By.id("<employee app number>")).sendKeys(unquieAppNo[1]);
}else{ //any other browser
driver.findElement(By.id("<employee app number>")).sendKeys(unquieAppNo[2]);
}
It is hard to tell with no code posted at all. Which Language you are working on?Please post the way you instantiate you Webdriver. From your description, I'm guessing that you are using something like this (static):
public static WebDriver driver;
Whilst you need a different WebDriver instance everytime:
public WebDriver driver;
I have written a selenium application using webdriver. I wish to run it on a remote server. When I do that by logging into the server via putty (along with Xming), the selenium tries opening the browser on the server only and load the pages through the external display. However in doing that, it takes a lot of time than if I would have been able to get the browser open on my localhost only (and not the server). Is it possible for such thing to happen or opening on the server only is the only option (which is painfully slow). Kindly tell me if I am missing something as well.
Thanks in advance.
Try using Selenium Grid, instead of Putty, to run your Selenium application on a remote server. The Selenium website has an excellent Quick Start guide for using the Selenium Grid: http://code.google.com/p/selenium/wiki/Grid2.
You can run Selenium with a"headless" driver, HtmlUnitDriver, that does not actually open a browser:
http://code.google.com/p/selenium/wiki/HtmlUnitDriver
Note: HtmlUnitDriver will accept an argument, so that it can emulate a specific driver.
#Lori
I implemented the code but it still tries opening it from putty so takes a lot of time to get the work done. The code is as follows: 'code'
import sys
from scrapy.spider import BaseSpider
from scrapy.http import FormRequest
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item
from scrapy.http import Request
from selenium import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
class DmozSpider(BaseSpider):
name = "linkedin_crawler"
#defence news
global company
global query
companyFilename = '<filename>'
f=open(companyFilename,"r")
f.seek(0)
company = f.readline().strip()
f.close()
queryFilename = '/var/www/Symantec/recon/recon/' +company+ '/Spider/LinkedIn/query.txt'
f = open(queryFilename)
f.seek(0)
query=f.readline().strip()
f.close()
start_urls = ['https://www.linkedin.com/uas/login'];
def __init__(self):
BaseSpider.__init__(self)
capabilities = webdriver.DesiredCapabilities()
self.selenium = webdriver.Remote(command_executor = 'http://localhost:5000/wd/hub', desired_capabilities = capabilities.FIREFOX)
def __del__(self):
self.selenium.quit()
def parse(self, response):
sel= self.selenium
sel.get(response.url)
global query
elem1 = sel.find_element_by_name("session_key")
elem2 = sel.find_element_by_name("session_password")
elem1.send_keys("myemailid")
elem2.send_keys("mypassword")
elem2.send_keys(Keys.RETURN)
return Request(query, callback=self.page_parse)
def page_parse(self,response):
global query
global company
sel= self.selenium
sel.get(query)
for i in xrange(10):
#for i in xrange(5):
nameFilename = ''
#print hxs
nlist = sel.find_elements_by_xpath('//ol[#class="search-results"]/li/div/h3/a')
fh = open(nameFilename,"a")
for j in xrange(len(nlist)):
url = nlist[j].get_attribute("href").encode('utf-8')
name = nlist[j].text.encode('utf-8')
fh.write(name)
fh.write("<next>")
fh.write(url)
fh.write('\n')
fh.close()
next = sel.find_elements_by_xpath('//a[#class="page-link"]')
next[0].click()
time.sleep(5)
To tun this script on server, I am using putty to fire the command. But then it again uses Xming to open the browser which makes the process slow again. So, how to run the script without opening the browser on my local machine via Xming so that this does not become the bottleneck. Thanks
I am new to selenium. Actually I am working on some cookie validation project, which requires me to manually check the cookies present before and after clicking on some consent link in multiple browsers (Firefox, ie, chrome, safari).
Previously in the phase 1 project I ran a qtp script to treat the firefox as a window object and capture screenshots, but that is quite troublesome if the resolution changes or any minor look-n-feel changes. Also it is quite difficult to manage and it works on firefox only and I needed to write the same script again for chrome and safari. Apart from this since QTP is licensed product and currently we are using seat license so I can't run it on multiple machines to speed up execution.
So I thought moving to Selenium. As of now my requirement is:
1. open the page - take the screenshot once page loaded.
2. check the cookies using firebug or any other way - take the screenshot
3. click the link to close the consent - take screenshot once consent closed.
4. refresh the page and again check the cookies using firebug - take screenshot
So I done some research on selenium and found that I can validate the cookies using verifyCookie but still I need screenshot of firebug window for cookies. So I got stuck here.
please help me out here..
I found some possible way to do this on Firefox but now I was looking forward for something similar for Chrome if that possible. Thanks
Selenium cannot interact with firefox extensions, or the browser in the way you want it to.
What you can do is collect a list of cookies on the page by doing:
driver.manage().getCookies()
This will give you a list of all cookies that are visible to Selenium. Please note that this is the same as the cookies that are visible in the JavaScript console (Not all cookies are visible via JavaScript, for example cookies set with the HTTPOnly attribute) using:
document.cookie
I would suggest you use getCookies() to programatically validate the cookies.
In selenium IDE if you want to take screenshot of the page use captureEntirePageScreenshot command
captureEntirePageScreenshot | D:\\test.png |
D:\\test.png - path of file where you want to save the file
Got some solution
public class Selenium1st {
/**
* #param args
*/
public static void main(String[] args) throws IOException, AWTException{
// TODO Auto-generated method stub
System.setProperty("webdriver.firefox.bin","C:\\Program Files (x86)\\Mozilla Firefox\\Firefox.exe");
FirefoxProfile firefoxProfile = new FirefoxProfile();
String domain = "extensions.firebug.";
firefoxProfile.setPreference("app.update.enabled", false);
firefoxProfile.addExtension(new File("E:\\softs\\selenium-2.29.0\\firebug\\firebug-1.11.2-fx.xpi"));
firefoxProfile.setPreference(domain + "currentVersion", "1.11.2");
firefoxProfile.setPreference("extensions.firebug.cookies.enableSites", true);
firefoxProfile.setPreference("extensions.firebug.allPagesActivation", "on");
firefoxProfile.setPreference(domain + "framePosition", "bottom");
firefoxProfile.setPreference(domain + "defaultPanelName", "cookies");
WebDriver driver = new FirefoxDriver(firefoxProfile);
driver.get("http://www.google.com/webhp?complete=1&hl=en");
WebElement query = driver.findElement(By.name("q"));
query.sendKeys("Cheese");
query.sendKeys("\n");
Robot robot = new Robot();
BufferedImage img = robot.createScreenCapture(new Rectangle(new Dimension(1024, 768)));
File path = new File("E:\\abc");//Path to your file
if(path.getName().indexOf(".jpg") == -1){
path = new File(path.getPath() + ".jpg");
}
ImageIO.write(img, "jpg", path);
}
}
might be useful.
I am seeing a problem with either Robot Framework or Selenium Webdriver in cases where a link or element is clicked that results in a page transition. The script hangs & stops running as if it's trying & failing to click the requested element/link even though the window successfully processed the click. Manually refreshing the Webdriver window to reload the page kick-starts the script and it resumes from there.
The only thing I can think is there is a delay between when Selenium or Robot executes the command and when it's able to listen for an HTTP response from the browser, and the page is loading before Selenium is ready to listen for it. This is running on an intranet and so the page load times are pretty quick. I've never seen the issue happen when running the same script on a SauceLabs VM since the tunnel between us & them adds a lot of latency.
Assuming my theory is correct, what do I do about it (apart from the obvious running over a slower connection)? Setting a delay in Selenium only slows down the execution and doesn't really affect the problem.
You can try fluent wait...
public static WebElement fluentWait(final By locator, RemoteWebDriver rwd){
Wait<WebDriver> wait = new FluentWait<WebDriver>(rwd)
.withTimeout(30, TimeUnit.SECONDS)
.pollingEvery(2, TimeUnit.SECONDS)
.ignoring(NoSuchElementException.class);
WebElement foo = wait.until(
new Function<WebDriver, WebElement>() {
public WebElement apply(WebDriver driver) {
return driver.findElement(locator);
}
}
);
return foo;
};
This will poll every 2 seconds and wait a maximum of 30 seconds
in your Test you then wait for an element eg fluentWait(By.className("home"), driver);
before you can click or verify text etc
Try:
Set Selenium Implicit Wait 60
Set Browser Implicit Wait 60
The number 60 is the seconds to wait by default for both selenium/browser.