I'm writing a paper in RMarkdown and for better reproducibility, I want to containerize all required software in a singularity container. Unfortunately, when I try to install TinyTeX (which is recommended for Rmarkdown and I would prefer over TeXLive to not inflate the container more than needed), it fails with the following error message (the full build log is pasted here):
Can't locate TeXLive/TLConfig.pm in #INC (you may need to install the TeXLive::TLConfig module) (#INC contains: /~/.TinyTeX/texmf-dist/scripts/texlive /~/.TinyTeX/tlpkg /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at ~/.TinyTeX/bin/x86_64-linux/tlmgr line 100.
BEGIN failed--compilation aborted at ~/.TinyTeX/bin/x86_64-linux/tlmgr line 100.
This is the build definition file, basically it uses a very slimmed down ubuntu 18.04 and then executes the %post section to install software
BootStrap: library
From: ubuntu:18.04
%post
# Add universe repository
echo "deb http://us.archive.ubuntu.com/ubuntu bionic universe" >> /etc/apt/sources.list
apt -y update
# Install utilites
apt install -y wget
# Install R
apt install -y r-base-core
## Install RMarkdown and TinyTeX
R --slave -e 'install.packages(c("rmarkdown","tinytex")); tinytex::install_tinytex()'
# Clean
apt-get clean
%environment
export LC_ALL="en_US.UTF-8"
%labels
Author DP
I have also tried tinytex::install_tinytex(dir="/opt/tinytex") but that didn't seem to change anything. Does anyone have an idea what's wrong?
That error message is complaining that your image (or, more likely, your path) is missing the TeXLive::TLConfig perl module.
My guess is that the path contents are not being rehashed with the installed modules after the install. The simplest solution is to break it into two commands:
R --slave -e 'install.packages(c("rmarkdown","tinytex"))'
R --slave -e 'tinytex::install_tinytex()'
Installation succeeds when I try that locally.
A potentially useful alternative, if the image is just for document generation, could be converting a docker image with rmarkdown and tex (e.g. https://hub.docker.com/r/rocker/verse) to a singularity one.
With singularity pull docker://rocker/verse you can do that for the latest version, or for a specific version with verse:version_number.
Related
The issue
I have a web scraper running in AWS lambda but in a few weeks AWS lambda will stop supporting Ruby 2.7. I built my scraper last year using this tutorial.
I need to find a version of chrome driver & headless chrome that is compatible with Ruby 2.7, But I don't know exactly where to start.
I have looked at the ChromeDriver's downloads portal But I don't see any indication there that Chrome driver will work for ruby 2.7 or any other specific version of ruby for that matter.
The code I have works by accessing the ChromeDriver binary and starting it inside a specific folder
I downloaded the specific binaries I am using by running these commands:
# serverless chrome
wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
unzip stable-headless-chromium-amazonlinux-2017-03.zip -d bin/
rm stable-headless-chromium-amazonlinux-2017-03.zip
# chromedriver
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip -d bin/
rm chromedriver_linux64.zip
Solution
I found the solution to this problem. Ruby 2.7 that Lambda offers by default runs on top of Amazon Linux 2 (which lacks many important libraries & dependencies), unfortunately, there's nothing you can do to change that.
However, Amazon offers you the ability to run your code in a custom docker image that can be up to 10GB in size.
I fixed this problem by creating my own image using the following Dockerfile
FROM public.ecr.aws/lambda/ruby:2.7
# Install dependencies needed to run MySQL & Chrome
RUN yum -y install libX11
RUN yum -y install dejavu-sans-fonts
RUN yum -y install procps
RUN yum -y install mysql-devel
RUN yum -y install tree
RUN mkdir /var/task/lib
RUN cp /usr/lib64/mysql/libmysqlclient.so.18 /var/task/lib
RUN gem install bundler
RUN yum -y install wget
RUN yum -y groupinstall 'Development Tools'
# Ruby Gems
ADD Gemfile ${LAMBDA_TASK_ROOT}/
ADD Gemfile.lock ${LAMBDA_TASK_ROOT}/
RUN bundle config set path 'vendor/bundle' && \
bundle install
# Install chromedriver & chromium
RUN mkdir ${LAMBDA_TASK_ROOT}/bin
# Chromium
RUN wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
RUN unzip stable-headless-chromium-amazonlinux-2017-03.zip -d ${LAMBDA_TASK_ROOT}/bin/
RUN rm stable-headless-chromium-amazonlinux-2017-03.zip
# Chromedriver
RUN wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip -d ${LAMBDA_TASK_ROOT}/bin/
RUN rm chromedriver_linux64.zip
# Copy function code
COPY app.rb ${LAMBDA_TASK_ROOT}
WORKDIR ${LAMBDA_TASK_ROOT}
RUN tree
RUN ls ${LAMBDA_TASK_ROOT}/bin
# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "app.handle" ]
Notes
If your code was previously deployed using a zip file you will have to either destroy the previous function or create a second function with the code update, it all comes down to how you want to handle deployment.
It is possible to automate the deployment process using the serverless framework
I would like salt-cloud to install the salt-minion using the same sources.list.d/saltstack.list file that is on the saltmaster server, but it uses the Ubuntu APT repository instead.
What I get on a new minion in /etc/apt/sources.list.d/saltstack.list:
deb https://repo.saltstack.com/apt/ubuntu/18.04/amd64/latest bionic main
What I want:
deb https://repo.saltstack.com/py3/ubuntu/18.04/amd64/latest bionic main
I installed salt-master and salt-cloud from the Saltstack repository by doing the following on my saltmaster server:
wget -O - https://repo.saltstack.com/py3/ubuntu/18.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
echo "deb https://repo.saltstack.com/py3/ubuntu/18.04/amd64/latest bionic main" > etc/apt/sources.list.d/saltstack.list
apt update
apt install python3-pip salt-master salt-minion salt-cloud
I spin up a server using salt-cloud and the /etc/apt/sources.list.d/saltstack.list without fail uses the apt repository, not the py3 repository.
Is there an option I can set in the saltmaster configuration file that defines which repository salt-cloud should use? Is there a similar option in a salt-cloud profile that can be passed to the salt bootstrap script?
I see in the bootstrap script that salt-cloud runs (found at bootstrap.saltstack.com) a command line option called _CUSTOM_REPO_URL, but I don't know how to pass options to that script when creating a server with salt-cloud -p
Happy to ask elsewhere (please indicate where) if more appropriate.
I can provide applicable parts of my salt-cloud profile if needed.
I found https://docs.saltstack.com/en/latest/topics/cloud/deploy.html#deploy-script-arguments
That led me to add the following to my cloud server profile used with the salt-cloud -p command:
script: bootstrap-salt
script_args: -x python3
Then the correct repository was installed.
Im running Ubuntu 18.04 and trying to install ROS for the first time on my machine. I have tried to follow the instructions at: http://wiki.ros.org/melodic/Installation/Ubuntu
I get stuck at this part.
sudo apt-get install ros-melodic-desktop-full
My attempts:
sudo apt -f install
sudo apt update --fix-missing
sudo dpkg --configure -a
sudo dpkg --remove --force-remove--reinstreq
sudo apt clean
sudo apt update
sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
Nothing has worked so far.
Terminal commands:
$sudo apt-get install ros-melodic-desktop-full
Reading package lists… Finished
Builds dependency trees
Reading state information ... Finished
Some packages could not be installed. It may mean that you have requested
an impossible situation or, if you use the unstable distribution
that some necessary packages have not yet been created or moved
from "Incoming".
The following information may help to resolve the situation:
The following packages have dependencies that cannot be satisfied:
rose-melodic-desktop-full: Dependent on: rose-melodic-desktop but it will not be installed
Depending on: rose-melodic-perception but it will not be installed
Depending on: rose-melodic-simulators but it will not be installed
Depending on: ros-melodic-urdf-sim-tutorial but it will not be installed
E: Could not correct the problems, you have withheld broken packages.
Please help me install ROS
1 - After following ROS melodic Installation first steps (see link below), Try to install it with the link "click here" below the "sudo apt install ros-melodic-desktop-full" line
ROS melodic Installation page
If it's doesn't work
2 - check, "Software & Update" parameters
(if you cannot find it: launch "Ubuntu Software Center" and in the menu bar, on top, select "Software & Update")
In "Other Sofware" tab, the link
"http://pacckage.ros.org/ros/ubuntu bionic main"
is printed, and in "Autentication" tab a key from Open Robotics is printed
3 - Try to do 1- again, if it doesn't work, follow 4-
4 - In "Updates" tab, check "recommanded Updates", launch "Update Manager" and install updates (I don't do it properly on my PC)
5 - And finally you need to install manually rosdep with:
"sudo apt-get install python-rosdep"
Ubuntu (apt) does NOT allow packages with the same name but different versions to be installed.
Remove other versions first. For example: apt remove ros-desktop*
first of all install program on ubuntu.
sudo apt-get update
sudo apt-get upgrade
after update packet try follow the instructions at: enter link description here again
I got it working like so:
Install deps:
sudo apt install gazebo9-common libgazebo9-dev gazebo9 ros-melodic-gazebo-ros-pkgs ros-melodic-gazebo-dev ros-melodic-gazebo-ros-control ros-melodic-gazebo-ros ros-melodic-urdf-sim-tutorial ros-melodic-simulators
Install ros melodic
sudo apt-get install ros-melodic-desktop-full
I tried to install generator-angularjs using Yo (Yoeman) without sudo:
npm install -g generator-angular
I get:
Error: EACCES, mkdir '/usr/lib/node_modules/generator-angular'
When I type in sudo yo, yo tells me that I should not use sudo (which is perfectly understandable).
I have a ~/node_modules directory - why doesn't yo install its packages there?
Generators are designed to be installed globally. Otherwise, you always have to install the generator you're about to use in each project, which is unnecessarily painful. Also, you don't get to see the lovely yo menu which lists you all the available generators (unless of course, you install them all locally):
Setting up npm for global installation
So, how do we get npm to install packages globally? As you correctly said, you should never, ever run yo with sudo. There are lots of different solutions to this problem and you can spend hours discussing their pros and cons religiously.
I personally dislike installing my user packages into the global /usr/ folder. /usr/ is for software that is shared across all users on the computer. Even if it's only using the machine, there are still good reasons to respect the way the Unix file system hierarchy is designed. For example if you decide at one point to wipe your whole node installation.
My preferred way of enabling npm to install packages globally without breaking out of $HOME is to set a local node prefix. This is as easy as running
echo 'prefix = ~/.node' >> ~/.npmrc
in your local shell. After that, you want to adjust your $PATH, to point to the new installation destination for global node executables by adjusting your favorite shell's config. E.g. by adding
export PATH="$PATH:$HOME/.node/bin"
to your ~/.bashrc. After that, you can happily run npm install -g generator-angular without sudo, without running into permission conflicts and if something is completely broken and you want to start from scratch, all you need to do is remove your ~/.node directory.
Thanks to #passy I managed to finally get this working on ubuntu 13.04 (in case anyone is having similar set up issues) with the following :
sudo apt-get update
sudo apt-get install python-software-properties python g++ make
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
trying to run:
npm install -g yo
resulted in
Error: EACCES, mkdir '/usr/lib/node_modules/yo'
Fixed using:
echo prefix = ~/.node >> ~/.npmrc
echo 'export PATH=$HOME/.node/bin:$PATH' >> ~/.bashrc
. ~/.bashrc
Running:
yo webapp
resulted in:
Error: EACCES, permission denied '/home/username/.config/configstore/update-notifier-yo.yml'
Fixed using:
sudo chown yourusername:yourusername /home/yourusername/.config/configstore/update-notifier-yo.yml
hi in my case (on ubuntu 12.04), the prefix addition in ~/.npmrc did not changed anything.
if so, build the node package by yourself and install it in /opt/node or /home/user/.node.
I had an almost identical error involving a rogue .yo-rc.json file in my root directory from a project I installed earlier. Yeoman was switching cwd from the installation dir to root dir half way through the installation, but was only outputting the EACCESS permissions error without any details that the installation directory was /. It took ages to figure out why this was, and involved debugging through the Yeoman source, but I eventually learned that Yeoman will look up through the directory tree until it finds a .yo-rc.json, and generate the code there by calling chdir to the new location.
Yeoman should maybe check that the user has write permissions for the directory. Alternatively, it could mention in the output either that the cwd has changed, or print the name of the installation directory if where it finds .yo-rc.json is different than cwd.
The command for finding rogue .yo-rc.json files
sudo find / -name .yo-rc.json
From yoeman getting started page appears the command:
yo doctor
In my case, $NODE_PATH (which in my case, Ubuntu 14.04, is defined in /etc/profile.d) isn't the same than npm root. Adding in npm root in $NODE_PATH solve the problem.
I have been trying to get yeoman to play nice with my vagrant box and this is what I had to do to install npm packages globally without sudo on ubuntu:
1. Create the directory to store global packages
$ mkdir "${HOME}/.npm-packages"
2. Tell npm where to put any packages installed globally
Insert this snippet into your ~/.npmrc file:
prefix=${HOME}/.npm-packages
3. Make sure that npm can locate installed binaries et cetera
Insert this snippet into your .bashrc/.zshrc:
NPM_PACKAGES="${HOME}/.npm-packages"
PATH="$NPM_PACKAGES/bin:$PATH"
// `unset` `manpath` to allow inheritance from `/etc/manpath` with
// the `manpath` command
unset MANPATH // remove this line if you have previously modified `manpath`
export MANPATH="$NPM_PACKAGES/share/man:$(manpath)"
4. Run the following or restart terminal
$ source ~/.bashrc
Hope this helps anyone who finds themselves in a similar situation.
Whenever I try to build DBD::Sybase to connect to MSSQL I get an error,
$ sudo cpanp install DBD::Sybase
Installing DBD::Sybase (1.15)
Running [/usr/bin/perl /usr/bin/cpanp-run-perl /home/ecarroll/.cpanplus/5.14.2/build/DBD-Sybase-1.15/Makefile.PL INSTALLDIRS=site]...
Can't find any Sybase libraries in /etc/lib or /etc/lib64 at /home/ecarroll/.cpanplus/5.14.2/build/DBD-Sybase-1.15/Makefile.PL line 155, <IN> line 44.
BEGIN failed--compilation aborted at /usr/bin/cpanp-run-perl line 11, <IN> line 44.
[ERROR] Could not run '/usr/bin/perl Makefile.PL': Can't find any Sybase libraries in /etc/lib or /etc/lib64 at /home/ecarroll/.cpanplus/5.14.2/build/DBD-Sybase-1.15/Makefile.PL line 155, <IN> line 44.
BEGIN failed--compilation aborted at /usr/bin/cpanp-run-perl line 11, <IN> line 44.
-- cannot continue
[ERROR] Unable to create a new distribution object for 'DBD::Sybase' -- cannot continue
*** Install log written to:
/home/ecarroll/.cpanplus/install-logs/DBD-Sybase-1.15-1374605483.log
Error installing 'DBD::Sybase'
Problem installing one or more modules
I've also gotten this error on other Debian systems.
There are two ways to do that,
(a) with the freetds that the distro provides
or, (b) installing the vanilla freetds upstream and building against that.
The second option (b) is always possible, but then your system may have two different versions of freetds.
The first option can not be done without some hacking and the author will not fix it. He is simply hard headed and wants to fix internal structures to match the OS he uses rather than making it accepting of other configurations.
Internally DBD::Sybase expects there to be a directory, and a $libdir (a subdirectory with lib or lib64). The directories DBD::Sybase requires to build properly are not provided by the Debian package freetds-dev; the Debian package installs to /usr/include which doesn't have a lib or a lib64 subdirectory. You can get around this by fooling make and recreating that structure, first make sure you have freetds-dev installed,
sudo apt-get install freetds-dev
Then link it to create a pseudo-package. On my 64 bit machine, it looks something like this.
mkdir /tmp/freetds
ln -s /usr/lib/x86_64-linux-gnu/ /tmp/freetds/lib64
ln -s /usr/include /tmp/freetds/include/freetds
Now, it should work and you can build DBD::Sybase against system libraries.
sudo SYBASE=/tmp/freetds cpanp install DBD::Sybase
Viola.
To install modules for the system perl, you can install the packages from the Ubuntu repositories. While these may be out of date, library dependencies are resolved aoutomatically. In this case, a
$ sudo apt-get install libdbd-sybase-perl
should do the trick.
I believe you have to make the include subdirectory, otherwise the second ln will fail:
mkdir /tmp/freetds/include