can not invoke chromedriver on Linux (debian) machine - selenium-webdriver

Resources: VM -Debian Linux , Selenium 4 , Chrome V78, ChromeDriver , Jenkins , Maven, JAVA 8
**Issue:** I can not invoke chromedriver on Linux (debian) machine.
I can see message "Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code."
on Jenkins.
Maven command : clean install test
Log Trace:::
[JENKINS REMOTING CAPACITY]===>channel started
Executing Maven: -B -f /var/lib/jenkins/workspace/Maven/pom.xml clean install test
[INFO] Scanning for projects...
[WARNING]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # Land ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 8 source files to /var/lib/jenkins/workspace/Maven/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) # Land ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
[INFO ] 2019-09-18 21:51:26.644 [main] BaseClass - [32mINFO[m - Creating Screenshot folder for this current execution.
directory is : /var/lib/jenkins/workspace/Maven/screenshots
directory is exist /var/lib/jenkins/workspace/Maven/logs
directory is exist /var/lib/jenkins/workspace/Maven/test-output
[INFO ] 2019-09-18 21:51:26.655 [main] SetUpTest - [32mINFO[m - ============================================================
[INFO ] 2019-09-18 21:51:26.655 [main] SetUpTest - [32mINFO[m - ====== Initializing Driver, Starting Browser Session =========
[INFO ] 2019-09-18 21:51:26.655 [main] SetUpTest - [32mINFO[m - ============================================================
projectPath is: /var/lib/jenkins/workspace/Maven
[INFO ] 2019-09-18 21:51:26.656 [main] BaseClass - [32mINFO[m - projectPath is: /var/lib/jenkins/workspace/Maven
[INFO ] 2019-09-18 21:51:26.656 [main] BaseClass - [32mINFO[m - Driver value is : null
[INFO ] 2019-09-18 21:51:26.656 [main] BaseClass - [32mINFO[m - chrome driver is here
Starting ChromeDriver 78.0.3904.11 (eaaae9de6b8999773fa33f92ce1e1bbe294437cf-refs/branch-heads/3904#{#86}) on port 4878
Only local connections are allowed.
**Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.**
[INFO ] 2019-09-18 21:51:27.930 [main] SetUpTest -
[INFO ] 2019-09-18 21:51:27.930 [main] SetUpTest - ============ createAccount Test case is Started. ==============
[INFO ] 2019-09-18 21:51:27.931 [main] SetUpTest -
[INFO ] 2019-09-18 21:51:27.932 [main] TestListener -
[INFO ] 2019-09-18 21:51:27.933 [main] TestListener - ===== createAccount - Test case is Skipped =======
[INFO ] 2019-09-18 21:51:27.934 [main] TestListener -
[INFO ] 2019-09-18 21:51:27.934 [main] TestListener - ===== createAccount - Test case is Failed =======
[INFO ] 2019-09-18 21:51:27.941 [main] BaseClass - ******** Disposing Browser Driver **********
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[ERROR] There are test failures. This is the final result
----------------------------------------------------------------
**My code :**
This is my Selenium code:
if (browserName.equalsIgnoreCase("chrome")) {
System.setProperty("webdriver.chrome.driver", "/usr/bin/chromedriver");
ChromeOptions options = new ChromeOptions();
options.addArguments("--verbose");
options.addArguments("--whitelisted-ips=''");
options.addArguments("--disable-extensions");
options.addArguments("--disable-notifications");
options.addArguments("disable-infobars");
options.addArguments("--incognito");
options.addArguments("--disable-gpu");
options.addArguments("--no-sandbox");
options.addArguments("--disable-gpu --disable-software-rasterizer");
options.addArguments("--disable-gpu-sandbox");
options.addArguments("--disable-features=VizDisplayCompositor");
These are options I have tried on chromedriver so far
driver = new ChromeDriver(options); // This line is not being invoked successfully.
------------------------------------------
What I tried so far:
1. I tried firefox browser - which is not launching as well
2. I tried to "chmod 777" files, drivers and folders as per requirement
3. I set up few chrome "options.addArguments" to "disbale GPU" , "headless" and "no sandbox" as per google
4. I looked in VM that if there are already multiple instance opened for Chromedriver via "htop" and other commands to kill the older sessions but I did not see any chrome because it's never been launched
5. https://stgconsulting.com/running-automated-selenium-tests-from-jenkins-on-linux/
6. https://tecadmin.net/setup-selenium-chromedriver-on-ubuntu

Resources: Maven, Chrome,ChromeDriver, Linux(Debian), Java, Testng, jenkins
Link I followed for basic Linux (Debian) set up : https://stgconsulting.com/running-automated-selenium-tests-from-jenkins-on-linux/
Before solution we have been seeing this below error:
[INFO ] 2019-09-20 09:41:19.341 [main] BaseClass - browserName value is : chrome
[INFO ] 2019-09-20 09:41:19.341 [main] BaseClass - chrome driver is here
Starting ChromeDriver 78.0.3904.11 (eaaae9de6b8999773fa33f92ce1e1bbe294437cf-refs/branch-heads/3904#{#86}) on port 29688
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[INFO ] 2019-09-20 09:41:23.904 [main] SetUpTest - ==========================================================
[INFO ] 2019-09-20 09:41:23.904 [main] SetUpTest - =========== createAccount Test case is Started.
After solution I am not seeing failure/error and I am able to execute lest on Linux (Debian):
[INFO ] 2019-09-20 08:46:30.675 [main] BaseClass - chrome driver is here
Starting ChromeDriver 75.0.3770.140 (2d9f97485c7b07dc18a74666574f19176731995c-refs/branch-heads/3770#{#1155}) on port 9770
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1568994392.814][WARNING]: This version of ChromeDriver has not been tested with Chrome version 76.
Sep 20, 2019 8:46:33 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2019-09-20 08:46:33.227 [main] BaseClass - Browser is Chrome
solution steps and information:
(1) You can have latest Chrome Linux Browser on your box
(2) Go here https://chromedriver.chromium.org/downloads and do bruteforce of ChromeDriver versions
(3) I am using Chrome linux 77 (as of now 9/20/2019) and chromedriver 76.0.3809.25 , worked for me
(4) "Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code" - got nothing with firewall
(5) Check your Chmod 777 permission for chromeDriver
It's a BROWSER AND CHROME DRIVER MISMTACH , ALL YOU NEED TO DO IS TO CHECK THE COMPATIBILITY

Related

How to configure REST properties in Java-only Camel Main?

I'm working on a simple "hello-world" Camel3 (Camel version 3.4.3) REST API example with the following setup:
REST API is described in OpenAPI specification (Swagger v2.0)
The API spec is converted to Java DSL with camel-restdsl-swagger-plugin
Using Camel Main
100% Java (i.e. no XML)
The example below works as expected but I fail to set jetty port. Therefore on every run the web server runs on a random port. How do I set the port (e.g. to 8081) in the Java code ? It also seems to be possible to set the port in application.properties file. Example of that option is also appreciated.
// https://camel.apache.org/components/latest/others/main.html
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.main.Main;
import org.apache.camel.main.RestConfigurationProperties;
import rest.dsl.generated.Api101;
public class App {
public static void main(String[] args) throws Exception {
Main main = new Main();
// How do I set the rest properties ?
// The code below doesn't seem to have effect.
RestConfigurationProperties p = main.configure().rest()
.withComponent("jetty")
.withHost("localhost")
.withPort(8081)
;
main.configure().addConfiguration(p);
// The route generated by camel-restdsl-swagger-plugin
main.configure().addRoutesBuilder(
new Api101()
);
// The actual route.
main.configure().addRoutesBuilder(
new RouteBuilder() {
public void configure() {
from("direct:rest1")
.routeId("Rest1Route")
.log("START:")
.setBody(constant("{hello: \"Hello, World!\"}"))
.log("END:")
;
}
}
);
main.run(args);
}
}
Generated Api101 code:
package rest.dsl.generated;
import javax.annotation.Generated;
import org.apache.camel.builder.RouteBuilder;
/**
* Generated from Swagger specification by Camel REST DSL generator.
*/
#Generated("org.apache.camel.generator.swagger.PathGenerator")
public final class Api101 extends RouteBuilder {
/**
* Defines Apache Camel routes using REST DSL fluent API.
*/
public void configure() {
rest("/api101")
.get("/hello")
.description("Basic Hello World")
.to("direct:rest1");
}
}
Compilation & run example:
/tmp/openapi$ mvn clean compile exec:java
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.stackoverflow:openapi >----------------------
[INFO] Building openapi 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # openapi ---
[INFO] Deleting /tmp/openapi/target
[INFO]
[INFO] --- camel-restdsl-swagger-plugin:3.4.3:generate-with-dto (default) # openapi ---
[INFO] reading from /tmp/openapi/swagger.yaml
[INFO] Generating Camel DSL source in directory: /tmp/openapi/target/generated-sources/restdsl-swagger
[INFO] Generating DTO classes using io.swagger:swagger-codegen-maven-plugin:2.4.12
[INFO] reading from /tmp/openapi/swagger.yaml
[WARNING] Output directory does not exist, or is inaccessible. No file (.swagger-codegen-ignore) will be evaluated.
[WARNING] 'host' not defined in the spec. Default to 'localhost'.
[INFO] writing file /tmp/openapi/target/generated-sources/swagger/src/main/java/io/swagger/client/model/InlineResponse200.java
[WARNING] 'host' not defined in the spec. Default to 'localhost'.
[INFO]
[INFO] --- build-helper-maven-plugin:3.2.0:add-source (default) # openapi ---
[INFO] Source directory: /tmp/openapi/target/generated-sources/restdsl-swagger added.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # openapi ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) # openapi ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /tmp/openapi/target/classes
[INFO]
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) # openapi ---
[ App.main()] BaseMainSupport INFO Using properties from: classpath:application.properties;optional=true
[ App.main()] DefaultRoutesCollector INFO No additional Camel XML routes discovered from: classpath:camel/*.xml
[ App.main()] DefaultRoutesCollector INFO No additional Camel XML rests discovered from: classpath:camel-rest/*.xml
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) is starting
[ App.main()] AbstractCamelContext INFO StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[ App.main()] JettyHttpComponent WARN JMX disabled in CamelContext. Jetty JMX extensions will remain disabled.
[ App.main()] log INFO Logging initialized #8774ms to org.eclipse.jetty.util.log.Slf4jLog
[ App.main()] Server INFO jetty-9.4.29.v20200521; built: 2020-05-21T17:20:40.598Z; git: 77c232aed8a45c818fd27232278d9f95a021095e; jvm 11.0.8+10-post-Ubuntu-0ubuntu120.04
[ App.main()] ContextHandler INFO Started o.e.j.s.ServletContextHandler#4d9f9a55{/,null,AVAILABLE}
[ App.main()] AbstractConnector INFO Started ServerConnector#157aa401{HTTP/1.1, (http/1.1)}{0.0.0.0:44267}
[ App.main()] Server INFO Started #8938ms
[ App.main()] InternalRouteStartupManager INFO Route: route1 started and consuming from: jetty:http://0.0.0.0:0/api101/hello
[ App.main()] InternalRouteStartupManager INFO Route: Rest1Route started and consuming from: direct://rest1
[ App.main()] AbstractCamelContext INFO Total 2 routes, of which 2 are started
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) started in 0.225 seconds
[ qtp1580225589-17] Rest1Route INFO START:
[ qtp1580225589-17] Rest1Route INFO END:
^C[ad #0 - CamelHangupInterceptor] DefaultMainShutdownStrategy INFO Received hang up - stopping the main instance.
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) is shutting down
[ad #0 - CamelHangupInterceptor] DefaultMainShutdownStrategy INFO Waiting for CamelContext to graceful shutdown, elapsed: 0.000 seconds
[ App.main()] DefaultShutdownStrategy INFO Starting to graceful shutdown 2 routes (timeout 45 seconds)
[el-1) thread #1 - ShutdownTask] DefaultShutdownStrategy INFO Route: Rest1Route shutdown complete, was consuming from: direct://rest1
[el-1) thread #1 - ShutdownTask] AbstractConnector INFO Stopped ServerConnector#157aa401{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
[el-1) thread #1 - ShutdownTask] ContextHandler INFO Stopped o.e.j.s.ServletContextHandler#4d9f9a55{/,null,UNAVAILABLE}
[el-1) thread #1 - ShutdownTask] DefaultShutdownStrategy INFO Route: route1 shutdown complete, was consuming from: rest://get:/api101:/hello
[ App.main()] DefaultShutdownStrategy INFO Graceful shutdown of 2 routes completed in 0 seconds
[ App.main()] MainLifecycleStrategy INFO CamelContext: camel-1 has been shutdown, triggering shutdown of the JVM.
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) uptime 44.386 seconds
[ App.main()] AbstractCamelContext INFO Apache Camel 3.4.3 (camel-1) is shutdown in 0.037 seconds
/tmp/openapi$
As you can see from the above the web server is available in port 44267:
Started ServerConnector#157aa401{HTTP/1.1, (http/1.1)}{0.0.0.0:44267}
And can be successfully accessed:
$ sudo netstat -tulpn | grep /java
tcp6 0 0 :::44267 :::* LISTEN 16926/java
$ curl http://0.0.0.0:44267/api101/hello
{hello: "Hello, World!"}
$
I found an answer to my secondary question: this is the way to define the port with the properties:
$ cat src/main/resources/application.properties
camel.rest.port=8081
$
This is also the method how to set all the other configuration options mention in the documentation. Might be obvious for Java veterans, but it was not obvious for me.

appcfg.py upload_data won't ask for authentication locally

In the local development environment, the upload_data command will not launch a browser for OAuth. Why is that?!?
The code works perfectly fine on App Engine but not the local development environment. Is there a trick to use the remote API for the dev environment.
Here's how I use the command...
appcfg.py upload_data --config_file=bulkloader.yaml --filename=./stops.txt --kind=StopLocationLoader --url=http://localhost:8082/_ah/remote_api
10:39 PM Uploading data records.
[INFO ] Logging to bulkloader-log-20161017.223916
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 401: --- begin server output ---
You must be logged in as an administrator to access this.
--- end server output ---
This is a bug: https://code.google.com/p/googleappengine/issues/detail?id=12445
It links to a workaround posted for another question:
gcloud auth login
gcloud auth print-access-token
appcfg.py upload_data --oauth2_access_token=<oauth2_access_token> --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<you#gmail.com>

Trouble Downloading Data from GAE Datastore

I'm having issues downloading the deployed datastore down to my local directory.
The documentation seemed pretty straightforward
https://developers.google.com/appengine/docs/python/tools/uploadingdata#Downloading_and_Uploading_All_Data
I'm a bit new to GAE... am I missing something obvious here?
I run the following command
appcfg.py download_data --url=http://myapp.appspot.com/_ah/remote_api --filename=ds_copy
I'm not even asked to authenticate,
and this is the output:
12:33 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20130507.123333
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 500: --- begin server output ---
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>If the problem persists, please report your problem and mention this error message and the query that caused it.</h2>
<h2></h2>
</body></html>
--- end server output ---
Any help is appreciated!
Have you configured the remote_api to be enabled in your app.yaml ?
https://developers.google.com/appengine/docs/python/tools/remoteapi
You simply need to enable the remote api
builtins:
- remote_api: on
Update your application then run the following commands:
appcfg.py download_data -A appName --url=http://appName.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data <directory>
this taken from https://stackoverflow.com/a/7944641/2954800

Broken Pipe error on appcfg upload_data

I have successfully download my data from datastore on my Mac OS X 10.8 (SDK 1.6). However when I'm using appcfg upload_data I get the following error; any idea ?
appcfg.py upload_data --url=http://localhost:8081/_ah/remote_api --filename=db_backup_2012_Nov_3
ERROR:
[INFO ] Connecting to localhost:8081/_ah/remote_api [INFO ] Starting import; maximum 10 entities per post ...........[INFO ] Unexpected thread death: WorkerThread-2 [INFO ] An error occurred. Shutting down... ........[ERROR ] Error in WorkerThread-2: [ERROR ] Error in WorkerThread-7:
[INFO ] 350 entities total, 0 previously transferred [INFO ] 190 entities (179958 bytes) transferred in 15.2 seconds [INFO ] Some entities not successfully transferred
I solved the problem by limiting the number of threads to 1 when runnign appcfg upload_data:
appcfg.py upload_data --url=http://localhost:8081/_ah/remote_api --filename=db_backup_2012_Nov_3 --num_threads=1
The dev app server seems to be dropping threads. Perhaps because of having "threadsafe: no" in app.yaml!

Google App Engine download_data authentication error

I have read the numerous questions on this, but found no solution that works :(
$ appcfg.py download_data --url=http://THING.appspot.com/_ah/remote_api --filename=backup1 .
08:47 PM Application: THING
08:47 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20120910.204726
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20120910.204726.sql3
[INFO ] Opening database: bulkloader-results-20120910.204726.sql3
[INFO ] Connecting to THING.appspot.com/_ah/remote_api
[INFO ] Authentication Failed
So I have several questions about what's going on:
Why does this not ask me for my password, when in almost every other question I've seen it does? Is it because I've already uploaded a new version of my app and signed in?
Why do some people have to put application='s~THING' in their command line (does not help me).
I'm using a gmail address as my admin, so presumably that means its not related to any of the OpenID bugs given as answers to other similar questions?
I have builtins: - remote_api: on in my app.yaml (which is in this directory - hence the ".", right?), do I need to put a handler in?
The request for /_ah/remote_api goes to my main ("/.*") handler! Is that the cause of the problem?
How can I fix any of these things?
Edit:
Sebastian kindly pointed me in the right direction, but I now have this error:
$ appcfg.py download_data --application='s~THING' --url=http://THING.appspot.com/_ah/remote_api --filename=backup1 --kind=Article .
09:47 PM Application: s~THING (was: THING)
09:47 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20120910.214744
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20120910.214744.sql3
[INFO ] Opening database: bulkloader-results-20120910.214744.sql3
[INFO ] Connecting to THING.appspot.com/_ah/remote_api
[INFO ] Downloading kinds: ['Article']
.[ERROR ] [WorkerThread-1] WorkerThread:
Traceback (most recent call last):
File "/home/me/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 176, in WorkOnItems
status, instruction = item.PerformWork(self.__thread_pool)
File "/home/me/google_appengine/google/appengine/tools/bulkloader.py", line 764, in PerformWork
transfer_time = self._TransferItem(thread_pool)
File "/home/me/google_appengine/google/appengine/tools/bulkloader.py", line 1170, in _TransferItem
self, retry_parallel=self.first)
File "/home/me/google_appengine/google/appengine/tools/bulkloader.py", line 1471, in GetEntities
results = self._QueryForPbs(query)
File "/home/me/google_appengine/google/appengine/tools/bulkloader.py", line 1442, in _QueryForPbs
raise datastore._ToDatastoreError(e)
Error: API error 4 (datastore_v3: NEED_INDEX): no matching index found.
[INFO ] An error occurred. Shutting down...
[ERROR ] Error in WorkerThread-1: API error 4 (datastore_v3: NEED_INDEX): no matching index found.
[INFO ] Have 10 entities, 0 previously transferred
[INFO ] 10 entities (12985 bytes) transferred in 1.6 seconds
I am still left with errors (see above) but the basics appear to be covered by https://developers.google.com/appengine/docs/go/tools/uploadingdata
as recommended in a comment. Thanks for that.
If I fix the other errors I will update this.
There's a small bug in the Go remote_api support. To work around it you can either add the relevant index, or use a dummy Python version to download the data. It should be fixed in a future release.

Resources