I started playing with Seleniumlibrary tests (run with robotframework) and as our websites have ads and metrics and so on, every time I run a test those URLs get loaded.
Is there a way to tell selenium/robotframework to not load certain types of URLs or prevent it to load external resources (i.e. everything that is not from localhost).
You can do that with browsermob-proxy. Once you have that installed, you simply setup the proxy and set the blacklist.
ProxyServer server = new ProxyServer(9000)
server.start();
final DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, server.seleniumProxy());
//Send a 200 for all requests to the facebook cdn
server.blacklistRequests("http://.*\\.fbcdn.net/.*", 200);
//Finish setting up your driver
WebDriver driver = new SomeDriverImpl(capabilities);
I believe the following will work with this python wrapper (the regex might be slightly different):
from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()
proxy.blacklist('http://.*\\.fbcdn.net/.*', 200)
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
...
proxy.stop()
driver.quit()
This can now be done in two lines in Chrome:
driver.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["www.baidu.com"]})
driver.execute_cdp_cmd('Network.enable', {})
Related
I'm trying to do automated tests on a localhost site using the following settings:
Python 3.8.10
selenium 3.141.0
Firefox 90.0
Burp Suite Community Edition v2021.6.2
I'm using Burp proxy with the address 127.0.0.1:8080.
I tested several examples available here. The below code is the one that has worked best so far.
from selenium import webdriver
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
PROXY = "127.0.0.1:8080"
firefox_capabilities['proxy'] = {
"proxyType": "MANUAL",
"httpProxy": PROXY,
"sslProxy": PROXY
}
driver = webdriver.Firefox(capabilities=firefox_capabilities)
driver.get("http://127.0.0.1")
This code works fine when the url in driver.get("URL here") is not localhost. When I enter the url http://127.0.0.1, the access made by selenium does not appear in Burp Suite's HTTP history. Instead of the accessed url, "http://detectportal.firefox.com" appears.
Is this a problem in the code or some configuration that I need to do?
Burp HTTP History
Look into the Firefox proxy settings, there you can find a statement that localhost connections are never directed to a proxy.
But this can be changed by opening about:config and set the option network.proxy.allow_hijacking_localhost to true.
For Burp to track links on localhost just add ("network.proxy.allow_hijacking_localhost", True) to FirefoxProfile().
from selenium import webdriver
fp = webdriver.FirefoxProfile()
fp.set_preference("network.proxy.type", 1)
fp.set_preference("network.proxy.http", "127.0.0.1")
fp.set_preference("network.proxy.http_port", 8080)
fp.set_preference("network.proxy.allow_hijacking_localhost", True)
fp.update_preferences()
driver = webdriver.Firefox(firefox_profile=fp)
driver.get('http://127.0.0.1')
I got this exception for all browsers. For example, I create a remote webdriver on chrome like this:
caps = DesiredCapabilities.chrome();
ChromeOptions options = new ChromeOptions();
options.addArguments("disable-infobars");
caps.setCapability(ChromeOptions.CAPABILITY, options);
webDriver = new RemoteWebDriver(new URL("http://myIP:5555/wd/hub"), caps);
And I got UnreachableBrowserException as follow:
org.openqa.selenium.remote.DesiredCapabilities chrome
INFO: Using `new ChromeOptions()` is preferred to `DesiredCapabilities.chrome()`
org.openqa.selenium.remote.UnreachableBrowserException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
But I check my selenium hub at http://myIP:4444/grid/console, everything is fine, the node is stil registered. I then check my node at http://myIP:5555/wd/hub/static/resource/hub.html, I still can click "Create Session" to create a session for all browsers.
I just got this exception today, it still worked few days ago. I am using Selenium 3.11.0, IntelliJ 2017.3, all drivers and browsers are latest versions.
I googled here, but I can't find a solution for this while my gird is still running. Any help much appreciated.
The error says it all :
INFO: Using `new ChromeOptions()` is preferred to `DesiredCapabilities.chrome()`
The current implementation of Selenium while invoking RemoteWebDriver supports the ChromeOptions and you can use the following code block :
ChromeOptions options = new ChromeOptions();
options.addArguments("disable-infobars");
webDriver = new RemoteWebDriver(new URL("http://myIP:5555/wd/hub"), options);
Update
As per your comment update the documentation at seleniumhq-documentation is yet to be updated. Here are the relevant bytes from the Selenium Release Notes :
Selenium v3.5.0 :
* Start making *Option classes instances of Capabilities. This allows
the user to do:
`WebDriver driver = new RemoteWebDriver(new InternetExplorerOptions());`
Selenium v3.6.0 :
* All `*Option` classes now extend `MutableCapbilities`
`new RemoteWebDriver(new ChromeOptions());`
Selenium v3.7.0 :
* Migrated from using `DesiredCapabilities` to either
`MutableCapabilities` or (preferably) `ImmutableCapabilities`.
We are running cucumber tests against our Single Page App using Selenium WebDriver and PhantomJS. We recently implemented an appcache manifest and now our cukes fail. In our troubleshooting it seems to be failing because phantomjs does not support cache manifests. In looking through the phantomjsdriver.log we that the negotiated capabilities contained:
"applicationCacheEnabled":false
We then tried to set the desired capabilities to enable it with:
var webdriver = require('selenium-webdriver');
var hooks = function() {
this.Before(function(callback) {
var pBrowser = webdriver.Capabilities.phantomjs();
pBrowser.set('applicationCacheEnabled', true);
this.driver = new webdriver.Builder().withCapabilities(pBrowser).build();
});
The desired vs negotiated capabilities in the log file show that it's trying to be set but isn't succeeding (and our cukes are still failing).
SessionManagerReqHand - _postNewSessionCommand - New Session Created: c6f61520-b603-11e3-83b0-2b905be08819
GhostDriver - Main - running on port 63911
CONSTRUCTOR - Desired Capabilities: {"browserName":"phantomjs","applicationCacheEnabled":true}
CONSTRUCTOR - Negotiated Capabilities: {"browserName":"phantomjs","version":"1.9.1","driverName":"ghostdriver","driverVersion":"1.0.3","platform":"mac-unknown-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
Does anyone know if our scenario isn't supported or if we're implementing it the wrong way?
According to line #47 of the source code at http://bit.ly/1mrRvaV, GhostDriver does not support the feature of desired vs. negotiated capabilities.
I am facing issue with the proxy prompts for firefox browser with version 17.
I tried the steps mentioned in this link. But still am getting the prompts.
Also referred selenium document for using proxies, but unable to succeed.
I need to use FF17 only as per my project req and am using selenium 2.28.
My company proxy addr is proxy.comp_name.com port: 8080.
I am not getting any proxy prompts while doing manually.
While running multiple tests prompt will be displayed arbitrarily.
Any updates on the below issue?
code used to launch the driver with the default firefox profile
FirefoxProfile profile = new ProfilesIni().getProfile("default");
DesiredCapabilities dCap = DesiredCapabilities.firefox();
dCap.setCapability(FirefoxDriver.PROFILE, profile);
driver = new FirefoxDriver(dCap);
code used to launch the driver with the new profile:
String PROXY = "proxy.abc.com:8080";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(PROXY)
.setFtpProxy(PROXY)
.setSslProxy(PROXY);
DesiredCapabilities cap = new DesiredCapabailities();
cap.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(cap);
Also I tried by setting Preferences to the firefox profile, but still am getting the proxy prompts..
FirefoxProfile firefoxProfile = new FirefoxProfile();
firefoxProfile.setPreference("network.proxy.type", ProxyType.SYSTEM.ordinal());
firefoxProfile.setPreference("signon.autologin.proxy" , true );
firefoxProfile.setEnableNativeEvents(false);
desiredCapabilities.setCapability(FirefoxDriver.PROFILE, firefoxProfile);
return new FirefoxDriver(desiredCapabilities);
Proxy proxy = new Proxy();
proxy.setProxyAutoconfigUrl("");
// We use firefox as an example here.
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(CapabilityType.PROXY, proxy);
// You could use any webdriver implementation here
WebDriver driver = new FirefoxDriver(capabilities);
I am using selenium web driver (Not selenium RC). I need to download a xml file by clicking a link. I have did some Google search, and I have found in some answers an using AutoIT in order to handle OS related dialog boxes.
But is there any other option using selenium to handle this without using AutoIT tool.
Kindly suggest some ideas.
Can you be a bit more specific which browser you are using. If it's firefox you can have a greater control on the file download. Any other browser including firefox you can use robot class. This can be used to perform the click on ok button for downloads. If its chorme then file download happens automatically without any intervention.
For the newest version of Firefox (as of writing), these are the arguments that I needed to avoid the download box. Note that you need to specify a directory that you can write to as seen in the third statement:
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference( "browser.download.folderList", 2 );
profile.setPreference( "browser.download.dir", <YOUR DOWNLOAD PATH> );
profile.setPreference( "plugin.disable_full_page_plugin_for_types", "application/pdf" );
profile.setPreference(
"browser.helperApps.neverAsk.saveToDisk",
"application/csv,text/csv,application/pdfss, application/excel" );
profile.setPreference( "browser.download.manager.showWhenStarting", false );
profile.setPreference( "pdfjs.disabled", true );
Note that the last line with pdfjs is required for the newer versions of Firefox, where it wasn't before. More info Here
I face Authentication Proxy pop-up issue in my project. So I tried below solution and it is working fine.
When we run Script from Selenium Web driver on Security Environment following Setup needs to be done to handle Authentication Proxy.
First you need to know below details,
network.proxy.autoconfig_url (Example: "http://example.com/abc.pac")
network.proxy.http (Example: abc-proxy.com)
network.proxy.http_port (Example: 8080)
private static WebDriver initFirefoxDriver(String appURL)
{
System.out.println("Launching Firefox browser..");
FirefoxProfile firefoxProfile = new FirefoxProfile();
firefoxProfile.setPreference("network.proxy.type", 1);
firefoxProfile.setPreference("network.proxy.autoconfig_url", "http://example.com/abc.pac");
firefoxProfile.setPreference("network.proxy.http", " abc-proxy.com");
firefoxProfile.setPreference("network.proxy.http_port", 8080);
WebDriver driver = new FirefoxDriver(firefoxProfile);
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driver.manage().window().maximize();
driver.navigate().to(appURL);
//driver.get(appURL);
return driver;
}