How to use Selenium with Firefox browser when I have no display - selenium-webdriver

I am newly assigned to this shared hosting (ssh access only). Have no sudo or yum facility. I am trying to scrape some data from a dynamically loaded website [So, can't use scrapy or bs]. When I am using Selenium, it gives an error:
/home/sanelywr/voyager-bot/app.py:28: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Firefox(options=options, executable_path=gecko_path)
Traceback (most recent call last):
File "/home/sanelywr/voyager-bot/app.py", line 28, in <module>
driver = webdriver.Firefox(options=options, executable_path=gecko_path)
File "/home/sanelywr/virtualenv/voyager-bot/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/webdriver.py", line 180, in __init__
RemoteWebDriver.__init__(
File "/home/sanelywr/virtualenv/voyager-bot/3.9/lib/python3.9/site- packages/selenium/webdriver/remote/webdriver.py", line 275, in __init__
self.start_session(capabilities, browser_profile)
File "/home/sanelywr/virtualenv/voyager-bot/3.9/lib/python3.9/site- packages/selenium/webdriver/remote/webdriver.py", line 365, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/sanelywr/virtualenv/voyager-bot/3.9/lib/python3.9/site- packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
self.error_handler.check_response(response)
File "/home/sanelywr/virtualenv/voyager-bot/3.9/lib/python3.9/site- packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status 1
When checking the geckodriver.log it says,
1653393333249 geckodriver INFO Listening on 127.0.0.1:33457
1653393333256 mozrunner::runner INFO Running command: "/home/sanelywr/voyager-bot/firefox/firefox" "--marionette" "--headless" "--no-sandbox" "-headless" "--disable-dev-shm-usage" "--remote-debugging-port" "37442" "--remote-allow-hosts" "localhost" "-no-remote" "-profile" "/tmp/rust_mozprofilei8tj67"
(firefox:28332): Gtk-WARNING **: 07:55:33.285: Locale not supported by C library.
Using the fallback 'C' locale.
**Error: no display specified**
Apparently, I have no screen on this server as I have checked by the following command:
$ echo $DISPLAY
It displays nothing.
I have already set the browser option to headless mode [within the python code].
And, I have also done the following:
$ export MOZ_HEADLESS = 1
And yes, I cannot use the virtual screen like xvfb because even when I am able to pip install the related wrapper, I still cannot install xvfb because I don't have sudo or yum [and cannot find a .tar.bz2 or tar.gz file].
How can I run the selenium browser without having to open the browser on any screen altogether? [Tried PhantomJS() too, even that's not working]
My python script :
import os
from credentials import API_KEY, API_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET
#from pyvirtualdisplay.xephyr import XephyrDisplay
#from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
import tweepy
auth = tweepy.OAuthHandler(API_KEY, API_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
api = tweepy.API(auth)
options = Options()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
options.headless= True
#options = webdriver.ChromeOptions()
#options.add_argument('--no-sandbox')
options.add_argument("--disable-dev-shm-usage")
#options.binary_location = '/home/sanelywr/voyager-bot'
#driver = webdriver.Chrome(executable_path='./chromedriver', options=options)
#driver = webdriver.PhantomJS()
gecko_path = '/home/sanelywr/voyager-bot/geckodriver'
options.binary=FirefoxBinary('/home/sanelywr/voyager-bot/firefox/firefox')
#display = XephyrDisplay()
#display = Display(visible=0, size=(800, 600))
#display.start()
driver = webdriver.Firefox(options=options, executable_path=gecko_path)
url=""
driver.get(url)
#<-----------Some code for extracting some elements from the page----------------->
#driver.quit()
#display.stop()
My script folder:
drwxr-xr-x 4 sanelywr sanelywr 4.0K Jan 24 2016 phantomjs-2.1.1-linux-x86_64
-rw-rw-r-- 1 sanelywr sanelywr 23M Jan 24 2016 phantomjs-2.1.1-linux-x86_64.tar.bz2
drwxr-xr-x 7 sanelywr sanelywr 4.0K Apr 7 2016 firefox
-rw-rw-r-- 1 sanelywr sanelywr 50M Apr 11 2016 firefox-45.0.2.tar.bz2
-rwxr-xr-x 1 sanelywr sanelywr 8.3M Apr 6 11:54 geckodriver
drwxr-xr-x 2 sanelywr sanelywr 4.0K May 23 02:09 public
drwxr-xr-x 2 sanelywr sanelywr 4.0K May 23 02:09 tmp
-rw-r--r-- 1 sanelywr sanelywr 145 May 23 02:09 passenger_wsgi.py
-rw-r--r-- 1 sanelywr sanelywr 326 May 23 02:30 credentials.py
-rw-r--r-- 1 sanelywr sanelywr 0 May 23 02:31 temp.txt
-rw-r--r-- 1 sanelywr sanelywr 16 May 23 02:36 requirements.txt
drwxrwxr-x 2 sanelywr sanelywr 4.0K May 23 06:29 __pycache__
-rw-r--r-- 1 sanelywr sanelywr 8.3M May 24 05:26 geckodriver-v0.31.0-linux64.tar
-rw-r--r-- 1 sanelywr sanelywr 1.9K May 24 07:24 app.py
-rw-rw-r-- 1 sanelywr sanelywr 4.8K May 24 07:55 geckodriver.log

Related

How can I query/interrogate a USB device which is not listed in vfs /sys/class/hwmon*?

I have a Corsair HX1000i PSU, but looks like I can't quite get information for it in the vfs /sys/class/hwmon*; this is my output:
ema#scv:~$ ls -la /sys/class/hwmon/
total 0
drwxr-xr-x 2 root root 0 Jan 21 20:22 .
drwxr-xr-x 85 root root 0 Jan 21 20:22 ..
lrwxrwxrwx 1 root root 0 Jan 21 20:22 hwmon0 -> ../../devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:02.0/0000:03:00.0/nvme/nvme0/hwmon0
lrwxrwxrwx 1 root root 0 Jan 21 20:22 hwmon1 -> ../../devices/pci0000:00/0000:00:01.3/0000:0d:00.0/nvme/nvme1/hwmon1
lrwxrwxrwx 1 root root 0 Jan 21 20:22 hwmon2 -> ../../devices/pci0000:00/0000:00:18.3/hwmon/hwmon2
lrwxrwxrwx 1 root root 0 Jan 21 20:22 hwmon3 -> ../../devices/virtual/thermal/thermal_zone0/hwmon3
My device is revision 2, id: 1b1c:1c1e (I can see it in lsusb):
...
Bus 001 Device 003: ID 1b1c:1c1e Corsair HX1000i Power Supply
...
And I'm running on Ubuntu 22.04 - which is using kernel 5.15.0.
Looking at the kernel sources seems like rev 2 of my PSU is only supported from kernel 6.1 forward in terms of appearing in hwmon* vfs.
Am I correct in expecting this adaptor to appear in this vfs once my kernel gets updated to 6.1+?
What shall I do if I want to query this device myself now? Shall I just use the hid part of the driver to communicate with the low level USB interface?
Is there any library to facilitate USB I/O on Linux (happy to write - or borrow - c code)?
I guess libusb should do whilst I wait Ubuntu to update the kernel?
Thanks!

How to change user/group uid/gid in LXC container and affecting the used files/directorys

In a fresh Debian 11 container I installed urbackup.
In the container the uid/gid from urbackup is 107/115.
I needed to bind mounted a zfs dataset to store the actual backups..
Therefore I changed in lxc the urbackup user/group.
root#urBackup:~# usermod -u 1005 urbackup
root#urBackup:~# groupmod -g 1005 urbackup
It seems though that some directory's stil had old uid/gid
root#urBackup:~# ls -al /var/urbackup
total 108012
drwxr-xr-x 3 nobody 115 4096 Apr 19 05:54 .
drwxr-xr-x 12 root root 4096 Apr 19 05:46 ..
-rwxr-x--- 1 nobody 115 35044777 Apr 19 05:47 UrBackupUpdate.exe
-rwxr-x--- 1 nobody 115 40 Apr 19 05:47 UrBackupUpdate.sig
-rwxr-x--- 1 nobody 115 102 Apr 19 05:47 UrBackupUpdata.sig2
Is there a easy way to change also the directory's and files belonging to a certain user/group.
Thanks for reading
Guy

certbort commands return ModuleNotFoundError: No module named '_cffi_backend'

I followed a guide to get my python flask app running and I am at the last step where I change http into https with certbot. But when I run my certbot command sudo certbot --nginx -d domainname -d www.domainname I get ModuleNotFoundError: No module named '_cffi_backend'
The whole error is:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in <module>
load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2859, in load_entry_point
return ep.load()
File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2450, in load
return self.resolve()
File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2456, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 10, in <module>
import josepy as jose
File "/usr/lib/python3/dist-packages/josepy/__init__.py", line 44, in <module>
from josepy.interfaces import JSONDeSerializable
File "/usr/lib/python3/dist-packages/josepy/interfaces.py", line 8, in <module>
from josepy import errors, util
File "/usr/lib/python3/dist-packages/josepy/util.py", line 4, in <module>
import OpenSSL
File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import crypto, SSL
File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 12, in <module>
from cryptography import x509
File "/usr/lib/python3/dist-packages/cryptography/x509/__init__.py", line 8, in <module>
from cryptography.x509.base import (
File "/usr/lib/python3/dist-packages/cryptography/x509/base.py", line 16, in <module>
from cryptography.x509.extensions import Extension, ExtensionType
File "/usr/lib/python3/dist-packages/cryptography/x509/extensions.py", line 18, in <module>
from cryptography.hazmat.primitives import constant_time, serialization
File "/usr/lib/python3/dist-packages/cryptography/hazmat/primitives/constant_time.py", line 9, in <module>
from cryptography.hazmat.bindings._constant_time import lib
ModuleNotFoundError: No module named '_cffi_backend'
I hope someone can help as I have found a lot of people asking questions about this mysterious _cffi_backend thing.
Some more info:
If I do python3 -m pip install cffi it says requirement already satisfied.
I have also gotten this error when installing other things and trying different peoples solutions.
'ModuleNotFoundError: No module named 'apt_pkg'
that seems to be fixed by doing sudo apt-get install python3-apt --reinstall but now I get: ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import) (/usr/lib/python3/dist-packages/gi/__init__.py)
I also made a symbolic link with /usr/lib/python3/dist-packages/apt_pkg.so -> apt_pkg.cpython-36m-x86_64-linux-gnu.so
something other people said would work. I am running python 3.8 but probably have 3.6 on the server as well. If I do python it opens the 3.8 shell. Also I am running ubuntu 18.04.4.
ls -al /usr/bin | grep python
gives :
-rwxr-xr-x 1 root root 1056 Apr 16 2018 dh_python2
lrwxrwxrwx 1 root root 23 Nov 7 10:07 pdb2.7 -> ../lib/python2.7/pdb.py
lrwxrwxrwx 1 root root 23 Nov 7 10:44 pdb3.6 -> ../lib/python3.6/pdb.py
lrwxrwxrwx 1 root root 23 Nov 7 10:50 pdb3.7 -> ../lib/python3.7/pdb.py
lrwxrwxrwx 1 root root 23 Oct 28 16:14 pdb3.8 -> ../lib/python3.8/pdb.py
lrwxrwxrwx 1 root root 31 Oct 25 2018 py3versions -> ../share/python3/py3versions.py
lrwxrwxrwx 1 root root 24 Jun 19 2019 python -> /etc/alternatives/python
lrwxrwxrwx 1 root root 16 Apr 16 2018 python-config -> python2.7-config
lrwxrwxrwx 1 root root 9 Apr 16 2018 python2 -> python2.7
lrwxrwxrwx 1 root root 16 Apr 16 2018 python2-config -> python2.7-config
-rwxr-xr-x 1 root root 3637096 Nov 7 10:07 python2.7
lrwxrwxrwx 1 root root 33 Nov 7 10:07 python2.7-config -> x86_64-linux-gnu-python2.7-config
lrwxrwxrwx 1 root root 25 Jan 5 10:38 python3 -> /etc/alternatives/python3
-rwxr-xr-x 1 root root 384 Feb 5 2018 python3-futurize
-rwxr-xr-x 1 root root 388 Feb 5 2018 python3-pasteurize
-rwxr-xr-x 1 root root 152 Nov 11 2017 python3-pbr
-rwxr-xr-x 2 root root 4526456 Nov 7 10:44 python3.6
-rwxr-xr-x 2 root root 4526456 Nov 7 10:44 python3.6m
-rwxr-xr-x 2 root root 4873376 Nov 7 10:50 python3.7
-rwxr-xr-x 2 root root 4873376 Nov 7 10:50 python3.7m
-rwxr-xr-x 1 root root 5203488 Oct 28 16:14 python3.8
lrwxrwxrwx 1 root root 10 Oct 25 2018 python3m -> python3.6m
lrwxrwxrwx 1 root root 29 Apr 16 2018 pyversions -> ../share/python/pyversions.py
lrwxrwxrwx 1 root root 10 Sep 27 2018 uwsgi_python36 -> uwsgi-core
lrwxrwxrwx 1 root root 33 Apr 16 2018 x86_64-linux-gnu-python-config -> x86_64-linux-gnu-python2.7-config
-rwxr-xr-x 1 root root 2971 Nov 7 10:07 x86_64-linux-gnu-python2.7-config
The files do exist on my system because:
dpkg -l python3-cffi-backend python3-cryptography
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii python3-cffi-backend 1.11.5-1 amd64 Foreign Function Interface for Python 3 calling C code - runtime
ii python3-cryptography 2.1.4-1ubuntu1.3 amd64 Python library exposing cryptographic recipes and primitives (Python 3)
This fixes the problem:
pip install -U cffi
The solution ended up being to install sudo apt install python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools but not in any virtual environment and then to install cffl with pip3 also not in any virtual environment.
On an Ubuntu 18.04 system there were multiple version of python3 installed, with the default python3 (3.9) being different to the (apt managed) system version (3.6). Using update-alternatives to set the default to the system version allowed certbot to run as expected. E.g.:
update-alternatives --config python3
There are 3 choices for the alternative python3 (providing /usr/bin/python3).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/bin/python3.9 3 auto mode
1 /usr/bin/python3.6 1 manual mode
2 /usr/bin/python3.8 2 manual mode
3 /usr/bin/python3.9 3 manual mode
Press <enter> to keep the current choice[*], or type selection number: 1
Then switch it back when done.
You can try following solution, if the solution above doesn't work for you..
pip3 -vvv install --upgrade --force-reinstall cffi

How to install Aerospike on Mac?

I tried following: https://www.aerospike.com/docs/operations/install/vagrant/mac/
and did following:
mkdir aerospike-vm
cd aerospike-vm
vagrant init aerospike/aerospike-ce
This creates Vagrantfile in the same directory.
Next I try:
vagrant up
Getting error:
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'aerospike/aerospike-ce' could not be found. Attempting to find and install...
default: Box Provider: virtualbox
default: Box Version: >= 0
The box 'aerospike/aerospike-ce' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/aerospike/aerospike-ce"]
Error: The requested URL returned error: 404 Not Found
What is the proper procedure to install Aerospike on Mac?
Thanks
I recorded this few months ago along with two other videos in this set. See if this youtube video helps you out: https://www.youtube.com/watch?v=qm42c0juam4&list=PLGo1-Ya-AEQDa32hFggyB0yIIOldxUFwv&index=3
Here is the output from my run:
Administrators-MacBook-Pro-4:aerospike-vm piyush$ ls -al
total 8
drwxr-xr-x 4 piyush staff 128 Apr 9 2018 .
drwxr-xr-x+ 54 piyush staff 1728 Jan 18 10:56 ..
drwxr-xr-x 3 piyush staff 96 Apr 9 2018 .vagrant
-rw-r--r-- 1 piyush staff 3029 Apr 9 2018 Vagrantfile
Administrators-MacBook-Pro-4:aerospike-vm piyush$ mv Vagrantfile Vagrantfile_old
Administrators-MacBook-Pro-4:aerospike-vm piyush$ vagrant init aerospike/aerospike-ce
==> vagrant: A new version of Vagrant is available: 2.2.3!
==> vagrant: To upgrade visit: https://www.vagrantup.com/downloads.html
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
Administrators-MacBook-Pro-4:aerospike-vm piyush$ ls -al
total 16
drwxr-xr-x 5 piyush staff 160 Jan 31 14:13 .
drwxr-xr-x+ 54 piyush staff 1728 Jan 18 10:56 ..
drwxr-xr-x 3 piyush staff 96 Apr 9 2018 .vagrant
-rw-r--r-- 1 piyush staff 3029 Jan 31 14:13 Vagrantfile
-rw-r--r-- 1 piyush staff 3029 Apr 9 2018 Vagrantfile_old
Administrators-MacBook-Pro-4:aerospike-vm piyush$

Why is goapp test is looking for files in /tmp?

I'm trying to get tests working on my local dev machine Cloud SDK version is: 159.0.0
Everything I've read says that I should no change GOROOT so I'm not sure how to fix this.
$ /Users/bryan/google-cloud-sdk/platform/google_appengine/goroot/bin/goapp test
go: cannot find GOROOT directory: /tmp/go_sdk887571938/appengine/go_appengine/goroot
bryan#Bryans-MacBook Thu Jun 15 10:22:37 ~/go/src/skincarereview
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/bryan/go/"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GO15VENDOREXPERIMENT="1"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"
bryan#Bryans-MacBook Thu Jun 15 10:22:57 ~/go/src/skincarereview
$ ls ~/google-cloud-sdk
total 408
drwxr-xr-x 30 bryan staff 1020 Jun 14 20:31 .install
-rw-r--r-- 1 bryan staff 980 Jun 14 20:30 LICENSE
-rw-r--r-- 1 bryan staff 673 Jun 14 20:30 README
-rw-r--r-- 1 bryan staff 162673 Jun 14 20:30 RELEASE_NOTES
-rw-r--r-- 1 bryan staff 8 Jun 14 20:30 VERSION
drwxr-xr-x 10 bryan staff 340 Jun 14 20:30 bin
-rw-r--r-- 1 bryan staff 2734 Jun 14 20:30 completion.bash.inc
-rw-r--r-- 1 bryan staff 2083 Jun 14 20:30 completion.zsh.inc
drwxr-xr-x 3 bryan staff 102 Jun 14 20:30 help
-rwxr-xr-x 1 bryan staff 1581 Jun 14 20:30 install.bat
-rwxr-xr-x 1 bryan staff 3471 Jun 14 20:30 install.sh
drwxr-xr-x 10 bryan staff 340 Jun 14 20:30 lib
-rw-r--r-- 1 bryan staff 308 Jun 14 20:30 path.bash.inc
-rw-r--r-- 1 bryan staff 1210 Jun 14 20:30 path.fish.inc
-rw-r--r-- 1 bryan staff 31 Jun 14 20:30 path.zsh.inc
drwxr-xr-x 6 bryan staff 204 Jun 14 20:30 platform
-rw-r--r-- 1 bryan staff 40 Jun 14 20:30 properties
bryan#Bryans-MacBook Thu Jun 15 10:24:22 ~/go/src/skincarereview
$ find / -name goroot 2>/dev/null
/Users/bryan/google-cloud-sdk/platform/google_appengine/goroot
bryan#Bryans-MacBook Thu Jun 15 10:28:43 ~/go/src/skincarereview
$ echo $PATH
/Users/bryan/google-cloud-sdk/bin:/Users/bryan/go/src/:/Users/bryan/google-cloud-sdk/platform/google_appengine/goroot/bin/:/Users/bryan/google-cloud-sdk/bin/:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin
Looking at main.go line 155, where I think the error is being thrown, I don't see where "goroot" is being set .
if fi, err := os.Stat(goroot); err != nil || !fi.IsDir() {
fmt.Fprintf(os.Stderr, "go: cannot find GOROOT directory: %v\n", goroot)
os.Exit(2)
}
You can try and follow the recommendation of this answer:
If your install is messed up beyond reason (happened to me once), just remove the cloud SDK and any references to it in your $PATH. Also completely uninstall the regular Go installation.
Then start from scratch. Install Go, unpack google-cloud-sdk, run installer (add to $PATH if needed), gcloud components install app-engine-go.
Voila.
As mentioned in that same answer, you don't set GOROOT anywhere, it is set for you.
You might want to read up on how go run/test works. The program basically builds into a temporary location and then runs, so that might be why it's complaining about your goroot.
This question could possibly help you: Golang: tests and working directory
I believe you are calling wrong goapp. You should be calling one that is directly in google_appengine folder as follows:
/Users/bryan/google-cloud-sdk/platform/google_appengine/goapp test
If you don't have goapp in google_appengine for some reason, I'll be quite surprised, but then following should work too. (Still the above is the correct way of calling it. goapp resolves appengine's goroot as a relative path from it's own location, so it's important where goapp itself is located.)
GOROOT=/Users/bryan/google-cloud-sdk/platform/google_appengine/goroot /Users/bryan/google-cloud-sdk/platform/google_appengine/goapp test

Resources