so today i was trying to web scrape via apache-airflow but it is giving this error
File "/home/siva/.local/lib/python3.10/site-packages/selenium/webdriver/chromium/webdriver.py", line 89, in __init__
self.service.start()
File "/home/siva/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 105, in start
raise WebDriverException("Can not connect to the Service %s" % self.path)
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service /d/apache-airflow/dags/chromedriver.exe
what should I do to connect the service so the web scrapping can be done I run my airflow test bench in ubuntu wsl so if there is any solution please provide it to get its work done in airflow
or if there are other ways to scrape in airflow do suggest.
Can not connect to the Service /d/apache-airflow/dags/chromedriver.exe
I don't have this set up to test with Airflow to be sure, but I have successfully run Chrome with Selenium under WSL2.
It sounds like you might be following some old instructions that were applicable for WSL1. Under WSL1, my understanding is that it is possible to use the Windows Chrome executable/webdriver.
You might want to try switching to WSL1, but I don't know for sure that Airflow will run there. It's very likely that it will.
However, if you do need to use WSL2, you'll have to use the Linux binaries.
This means that you'll need to install Google Chrome inside the WSL distribution and use the corresponding chromedriver_linux64.zip.
You'll also need to either ...
... be running WSL with the ability to run graphical applications - If you have Windows 11, this is automatic. If not, I recommend Xrdp as the next easiest path.
... or run Chrome in headless mode. I'm not sure off the top of my head how to do this with Airflow, unfortunately.
Related
I am new to the node world. I created a node js rest API. When I run npm start in my local machine or in the terminal for the first time, I can see console.log() in my terminal. Now, I am running the same application on an AWS Ec2 instance with Ubuntu as os. I run npm start and serve my app on port 80. I do this via ssh and after running my server I close the ssh connection. But when I reconnect via ssh, I want to see those console.log() messages in my terminal for some purpose.
I completely understand that logging messages in the terminal is not a good idea and there can be so many alternatives. Just want to know how to access the same terminal window/result as we see when I start my application.
if you are using pm2, you can try "pm2 logs"
So Nodemon won't work well in a production server or in any instance where you need to have the app going by itself.
Nodemon is a dev tool to enable you to restart your server during development. In a "real" vps you need to place the process in the background or it will be automatically killed when the connection times out.
Check out this YT series for a correct deploy architecture in an NGINX server with the use of pm2 and NGINX on a Red Hat server, I've personally used it more than once:
https://www.youtube.com/playlist?list=PLQlWzK5tU-gDyxC1JTpyC2avvJlt3hrIh
Does anybody work with Tosca and Jenkins?
I have problems to run my batch script from Jenkins. It does not open browser, but job is finished successfully. Batch script runs normally when I try to run in on Windows, without Jenkins.
Anybody has an experience using these two tools together?
In order to use Tosca CI with Jenkins you need to:
Configure your Jenkins build project
Add an additional build step (Execute Windows batch command)
Call ToscaCIClient.exe/ToscaCIClient.jar with parameters
e.g. "PathToTosca\ToscaCI\Client\ToscaCIClient.exe" -t junit -r PathToResults\result.xml
Further information can be found here: https://support.tricentis.com/community/article.do?number=KB0012411
The issue your facing is because your running Jenkins as windows service, in this case headless execution happens technically your script is executed for other user account.
Solution
Instead of installing Jenkins host it on Apache Tomcat server
Download Jenkins war file
Copy it inside webapps folder of Tomcat
Run startup.bat
Refer-
https://www.tutorialspoint.com/jenkins/jenkins_tomcat_setup.htm
I had faced the same issue.
I am guessing your slave machine running the test has jenkins services installed as services.
Uninstall the service. (.jenkins/jenkins-slave.exe uninstall)
Use the .jnlp or slave.jar to connect the machine as a slave.
Your test should run fine after that.
If it does just create a batch file to connect the machine as slave whenever it is logged in.
I tried that. But in any case, Jenkins does not open the browser, so Tests from Execution List and I do not know why.
We use Tosca 10.
I followed this tutorial
to get a Bigtable client up and running in Google Managed VMs. But is there a way to run this locally? Reason is that deploying the code remotely in development is a pain.
Normally I can use dev_appserver.sh to run GAE app locally. But when I run it, I'm getting this error:
Caused by: java.lang.IllegalStateException: Jetty ALPN has not been
properly configured.
Which means we need to include ALPN library? Since our codebase is in Java 7, I used this ALPN version: 7.1.3.v20150130.
I then tried again with this:
dev_appserver.sh --jvm_flag=-Xbootclasspath/p:/Users/shouguoli/tmp/alpn-boot-7.1.3.v20150130.jar
still getting this error:
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException:
The API package 'urlfetch' or call 'Fetch()' was not found.
How do you get it to work locally?
The sample was updated last week. It's based on the java 8 compat runtime, which means that you have access to most of the App Engine API's including Users, Task Queues, and Datastore.
There is a new Netty TCNative module that uses Boring SSL.
To use it with the pom.xml in the sample, do:
mvn clean -Pmac jetty:run -Dbigtable.projectID=<your-project> -Dbigtable.clusterID=<your-cluster> -Dbigtable.zone=<your-zone>
To use on Windows, use -Pwindows instead of -Pmac. For linux, omit the Profile -P as it's the default.
To deploy:
mvn clean gcloud:deploy -Dbigtable.projectID=<your-project> -Dbigtable.clusterID=<your-cluster> -Dbigtable.zone=<your-zone>
NOTE - it is advisable to do the clean between running locally and running remotely as the TCNative module is currently specific to the platform the code runs on.
We are in the process of updating all of our samples to use TCNative, we hope to have this by 3/10/16.
I am trying to set up an environment for end-to-end testing on a droplet running Ubuntu server 12.04.3 on digital ocean.
What I am trying to achieve in the end is for my jenkins (installed on the one droplet) to be able to run my end-to-end tests. Now, the server is ofcourse headless and the end-to-end tests need to run through a browser (I am using protractor with the selenium standalone server with chromedriver).
My question is: how do I spawn a browser on that machine? I have installed xorg and if I do startx on the server, log out and ssh -X to it, I can manually run the end-to-end tests (a browser pops up on my local machine). But I can get it to work without ssh -X to it, and since jenkins is on the same droplet where the tests are to be run. Well I dont get a browser to spawn.
NOTE: I know I might be missing something really trivial here since I don't fully understand the configuration nor the xorg.
Any hints or a complete answer is very much appreciated, this is giving me gray hair.
Edit: After a little digging I think i got the xorg stuff a bit wrong, i am guessing the purpose of X is to be able to spawn a window on a remote machine ( ie my local machine). And what i am after is more along the lines of a virtual frame buffer such as Xvfb...
There is PhantomJS but with Protractor is buggy and a dead-end.
You can still use Chrome & Firefox headless through docker-selenium or, if you don't like Docker you can do it yourself with ubuntu-headless sample. Both solutions provide Chrome & Firefox by using Xvfb even though there is no real DISPLAY.
UPDATE 2 Seems to be possible to run Xvfb in OSX: http://xquartz.macosforge.org/landing/
UPDATE 1 Mac OSX selenium headless solution:
Enable multi-user remote desktop access to OSX machine
So can test selenium headless on mac. Not headless really but as another user so it doesn't interfere with your current user display.
To do this you need kickstart: http://support.apple.com/en-us/HT201710
Begin using the kickstart utility
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -restart -agent
Activate Remote Desktop Sharing, enable access privileges for all users and restart ARD Agent:
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -restart -agent -privs -all
Apple Remote Desktop 3.2 or later only
Allow access for all users and give all users full access
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -allowAccessFor -allUsers -privs -all
Kickstart help command
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -help
A lot of angular apps use Travis CI to perform Protractor based end-to-end integration tests on headless vms all the time. I do not know the details of exactly how they do it but I do know that they use a linux service called xvfb which is a headless x windows implementation. Looking at a typical Travis configuration file, it appears that all they do before firing up their web server, selenium server and kicking off Protractor is to call sh -e /etc/init.d/xvfb start to start this service.
I use windows 7, apache 2.2.22 at port 80 and geoserver 2.1.3 at port 8080.
I download and run kaazing 3.5 msi x64 installer to install it locally on my laptop. I followed the official guidelines from the site. The msi succesfully installed the gateway.
But gets installed in C:\Applications Files\etc. not in C:\Program Filesx68\etc.
Anyhow, I tried to start the services, I ran the demo-services.start.bat and a notification came from windows saying that the Windows Fire Wall has blocked some of the features of java. So I hit "Allow" and wait. Command line says something like
"Sending data to ucd://localhost/50505, ucd://localhost/50506"
for over an hour, nothing happens. So I thought something went wrong with windows and java. I uninstall the kaazing, edit the Windows Fire Wall settings to allow java and re-install kaazing. The notification is not showing up now, but when I ran the demo-services.start.bat, still does nothing, just says the same thing. When I visit
http://localhost/8000
gives an 404 error.
I tried everything, chanching ports, uninstall and re-install a couple of times, installing while not connected to the internet, checking the windows fire wall settings, manually running demo-services.start.bat and gateway.start.bat . The gateway.start.bat actually runs ok and says that the gateway started, but still an 404 error when I visit localhost/8000. Installation through msi is always completed with no errors. But the Gateway does not work. Is it the fire wall, the demos bat file, I dont get it...
Tips/hints?
The first thing to do is get the Gateway running successfully first. So don't worry about running demo-services.start.bat yet.
From the Windows Services application, start Kaazing WebSocket Gateway. Then go to C:\Program Files\Kaazing, locate your installation and look in the log directory. Open error.log using a text editor and verify there are no errors.
If there are no errors, you should be able to open http://localhost:8001 from a browser. (Note, you had http://localhost/8000 in your example, but that last slash should be a colon.) You can use either port 8000 or 8001, but 8001 is where the samples are.
If you are using a firewall or something else that is intercepting ports, then you'll need to make sure ports 8000 and 8001 are accessible.
If you're not sure, start a different server process on port 8000 or 8001 (e.g. configure Apache to listen on port 8000 or 8001) and see if the browser can connect.
The msi succesfully installed the gateway. But gets installed in
C:\Applications Files\etc. not in C:\Program Filesx68\etc.
The Gateway is not an executable itself, but runs in a JVM. Therefore there is no 32bit code which constrains the application to be installed into C:\Program Files (x86). Thus C:\Program Files made the most sense.
You could use a 32-bit JVM which would reside in C:\Program Files (x86), but the Kaazing files are abstracted from that via Java, so C:\Program Files is a reasonable location for the Gateway.
BTW There is a forum on the Kaazing website for Kaazing questions.