Submitting an R job on a linux cluster - package

In submitting my R job to a cluster using the Rscript myscript.R command, I keep getting an error message saying the packages I have asked to load do not exist. However I have installed these packages onto a library on my profile. Do I have to set the working directory into the same pathway as the where the R packages are contained? I am genuinely puzzled by how to fix this error.
Thanks
Nikhail

try to add directly the install.packages command in your script. Like that, you're sure it will install properly:
for example add that at the beginning of your script (of course change "pacman;" for your real package and add one line per package):
if (!require("pacman")) install.packages("pacman")
I had this kind of problem before when the shell didn't call the good R for me (other version than the one I thought or 32 bits vs 64 bits). Don't know if it's your problem however...

Related

VOLTTRON install on rasbian buster

Can I get a tip for installing on rasp buster? Im hung up on the install directions to check the status of the rabbitMQ server. Traceback of bash console:
(volttron) pi#raspberry:~/Desktop/volttron $ echo 'export RABBITMQ_HOME=$HOME/rabbitmq_server/rabbitmq_server-3.7.7'|sudo tee --append ~/.bashrc
export RABBITMQ_HOME=$HOME/rabbitmq_server/rabbitmq_server-3.7.7
(volttron) pi#raspberry:~/Desktop/volttron $ source ~/.bashrc
pi#raspberry:~/Desktop/volttron $ RABBITMQ_HOME/sbin/rabbitmqctl status
bash: RABBITMQ_HOME/sbin/rabbitmqctl: No such file or directory
There are a few tracebacks earlier on the installation...
If it makes a difference or not here is the entire bash console process. The git gist link I just created the name install.py even though its just bash commands copied pasted per install directions...
`pi#raspberry:~/Desktop $ git clone https://github.com/VOLTTRON/volttron --branch releases/7.x`
It looks like there are a couple of different issues going on here:
The issue you quote above (RABBITMQ_HOME/sbin/rabbitmqctl: No such file or directory) is that your shell isn't finding the rabbitmqctl command. It looks like you added the RABBITMQ_HOME environment variable to your .bashrc, but used the string RABBITMQ_HOME instead of the variable expansion $RABBITMQ_HOME when you tried to run the command. Try running it as $RABBITMQ_HOME/sbin/rabbitmqctl status instead.
The rabbitmqctl status command will check the status of the rabbitmq application, but I don't think you've done anything to start it yet (that happens when you bootstrap the platform and/or start the platform configured to use the RMQ broker)
I think that the traces earlier in the installation process are problematic (appears to be the same error hit two different ways), but you just haven't run into them yet. I haven't seen any issues building gevent on the RPi 4 with buster (though it is pretty slow), but the ctypes error makes me wonder if there's an issue with the underlying c library it is trying to build on top of. I did notice that you're getting amd64 erlang packages, are you running Raspbian on an x86 processor? (if so this isn't a permutation we've tried and you may be hitting some package compatibility edge case we haven't seen)
One thing to try is to manually install cython into your virtualenvironment and then try running the bootstrap script again with the virtualenvironment activated. You could also try and pip install gevent==20.6.1 directly in that virtualenvironment (this is what the bootstrap script was doing at the failure point). VOLTTRON depends on gevent, so if that isn't installing the platform won't be able to run.

KNIME Command Line Execution - ClassNotFoundException

I'd like to schedule a KNIME workflow. The workflow does its job very good as long as I start it from the KNIME GUI application. When I execute the same workflow via command line, java complains that com.microsoft.sqlserver.jdbc.SQLServerDriver
could not be found (ClassNotFoundException).
I invoke it via:
"D:\Progamme\KNIME\knime.exe" -nosplash -application -consoleLog org.knime.product.KNIME_BATCH_APPLICATION -preferences="absolutepathto\preferences.epf" -workflowDir="absolutepathto\workflow"
Since the error message signals missing content in the java CLASSPATH I also tried to add the parameters
-vmargs -classpath .;"absolutepathto/sqljdbc42.jar"
But still I earn a java slap, pointing to the same error...
I also tried to run the command from within the knime.exe's directory and I also tried to add the JAR file to Preferences -> Java -> Build Path -> Classpath Variable / User Libraries (referenced via the -preference argument). But that had no effect.
Did anybody face the same problems? Maybe with other third party JARs?
It is all about a Database connector that is configured like this:
Does the integrated security maybe force a misleading error?
System spec: KNIME 3.2.2 on Windows Server 2008 R2
Update - extract from preferences file
/configuration/org.eclipse.core.net/org.eclipse.core.net.hasMigrated=true
/configuration/org.eclipse.ui.ide/MAX_RECENT_WORKSPACES=10
/configuration/org.eclipse.ui.ide/RECENT_WORKSPACES=<list of some workspaces>
/configuration/org.eclipse.ui.ide/RECENT_WORKSPACES_PROTOCOL=3
/configuration/org.eclipse.ui.ide/SHOW_RECENT_WORKSPACES=false
/configuration/org.eclipse.ui.ide/SHOW_WORKSPACE_SELECTION_DIALOG=true
Is there maybe a problem due to the fact that it is a shared KNIME instance among several users and the command line execution does not know which workspace has to be chosen? Is the workspace somehow needed and why?
Partial Solution:
I finally managed it but I don't know exactly why it works now. What I did was to load a fresh portable version of KNIME and ran the same commands only changing the executable path to the new portable version. Before that I started the portable version once to set the workspace directory and register the database driver in preferences dialog and .ini file, nothing else, same configuration so far as the shared KNIME instance. What I am really wondering abpout is that from now on the commands are also working with the shared KNIME instance. I really don't know what caused the change that let KNIME find the driver class.
Info
Because I encountered a few more problems within shared environment in KNIME command line mode, that led to undeterministic execution results, I wrote a little .NET library. This gives me more flexibility/control over the workflow execution (which returncodes and error messages occured and so on). You can find it here if you're interested: KnimeNet
I took a very minimal approach:
cd "C:\Program Files\KNIME"
.\knime -nosplash -noexit -consoleLog -reset -application org.knime.product.KNIME_BATCH_APPLICATION -workflowFile="D:\Work\Knime Workflows\Output\CMD_Test.knwf" -preferences="D:\Work\Knime Workflows\Output\CMD_Test.epf"

MacPorts Build of Gimp ends prematurely

I wanted to install The Gimp on my mac. It's not in the HomeBrew tree, so I decided to install it with MacPorts, which I've never used before, and installed today just to build The Gimp. I used the command recommended at http://alemani.com/installing-the-gimp-with-macports/ namely
sudo port -v install gimp +quartz +animation +python27 -x11
After more than 5 hours, I got this:
---> Activating gimp #2.8.14_0+quartz
x ./
x ./+COMMENT
x ./+CONTENTS
x ./+DESC
x ./+PORTFILE
x ./+STATE
x ./opt/
x ./opt/local/
x ./opt/local/share/
x ./opt/local/share/gimp/
x ./opt/local/share/gimp/2.0/
x ./opt/local/share/gimp/2.0/themes/
x ./opt/local/share/gimp/2.0/themes/Nodoka/
x ./opt/local/share/gimp/2.0/themes/Nodoka/gtkrc
---> Cleaning gimp
---> Removing work directory for gimp
Error: Port + not found
and the job stopped. Now, The Gimp is installed, and it seems to be working, although I haven't done any real testing. But the verification that the website above indicates will happen never took place.
What does the error message mean? Is there a way I can use MacPorts to audit if the build was successful, and perhaps to repair it? Or, can I fix whatever was wrong and resume the build? (I'm not willing to wait another 5 and half hours to get to this point, though.) Or can I say to hell with it, and uninstall The Gimp and MacPorts? How do I do this? (I know there are pre-built binaries out there, and I would have grabbed one if I'd known the build was going to take so long.)
I really don't want to invest any time in learning MacPorts, since I'm happy with HoneBrew, so if someone can lead me by the hand, I'd be grateful.
The error message indicates you had a space between + and one of the variant names in the command line you invoked. That likely means gimp was not installed with the variants you chose – you can check which variants were used by running port installed gimp.
Anyway, if GIMP is running as you expect it to, you don't need to do anything at this point. You can force the check you were missing by running sudo port rev-upgrade (and it will also look a little different from the output mentioned on the website), but it's really just a sanity check that should not find any problems.

"Your GStreamer installation is missing a plug-in." (GstURIDecodeBin)

I have: gstreamer-sdk, gstreamer-ffmpeg, gstreamer-plugins-good, bad, and ugly. I googled everywhere for this error and have found nothing relevant. I'm going a little nuts trying to figure out this error:
Error received from element decodebin20: Your GStreamer installation is missing a plug-in.
Debugging information: gstdecodebin2.c(3576): gst_decode_bin_expose (): /GstPlayBin2:playbin20/GstURIDecodeBin:uridecodebin0/GstDecodeBin2:decodebin20:
no suitable plugins found
It throws when I run my gstreamer program. Any ideas on why?
You may not be missing any plugins at all.
This error can be a result of just an unlinked pipeline.
Playbin2(decodebin2) got some changes that made it unable to automatically link up some pipelines that formally worked, for example transcoding a decoder to an encoder. In my case, explicitly adding the ffdec_h264 that it used to add automatically fixed it.
Relying on the Playbin2 can be very frustrating when it does not work. Using the setup below, you can create a .png diagram of the pipeline in various phases of construction. It's very helpful in finding why it isn't linking up.
export GST_DEBUG_DUMP_DOT_DIR=~/gstdump
for f in $GST_DEBUG_DUMP_DOT_DIR/*.dot ; do dot -T png $f >$f.png; done
This tool also lets you learn from it how to link up pipelines, and replace them with explicit ones that are easier to debug and less likely to break.
In Fedora, I resolved this issue removing gstreamer1-vaapi.x86_64:
sudo yum remove gstreamer1-vaapi.x86_64
uridecodebin is part of the "base" plugin set, so make sure you have gstreamer-plugins-base.
Another thing to look into is your LD_LIBRARY_PATH and GST_PLUGIN_PATH. If they point to a different GStreamer installation, it could cause problems like this. Also, if you didn't install GStreamer with a package manager, you may need to set your LD_LIBRARY_PATH to point to it (or better yet, install it with a package manager).
Pleas try to use gst-inspect command to find out if environment is correctly setup.
use gst-launch -v playbin2 uri = "your_uri_here" to find more information to trace this issue.

Install Packages (for dummies)

I know there's a lot of information on here about installing python packages, but I'm quite new to python and I think I need a more "for dummies" level of help.
I was trying to install openpyxl for which as far as I can tell I need easy_install, for which, as far as I can tell, I need setuptools. I tried running the code provided here https://pypi.python.org/pypi/setuptools which is supposed to download and install setuptools (and according to some sites, easy_install aswell?) - it runs successfully, but help(modules) doesn't show setuptools or easy_install as modules, I have no idea whats installed and what isn't, or how I'm supposed to install any of it!
Essentially I'm very confused, very frustrated and really need someone to talk me through (in idiot-speak) what I'm supposed to do.
Thankyou!
We all start somewhere, I was there two weeks ago.
I'll assume you're using Python2. I believe Distribute and Pip are recommended for Python3 (which I will be using as examples). I will also assume you are on Windows.
First, python needs to be registered to Path. To check if this has been done automatically, open a command prompt (start -> programs -> accessories), and type 'python', then enter. If it returns the version number, etc, skip down a bit. If it throws an error, you need to add Python to Path.
Adding Python to Path
To add Python to Path on a Windows computer, go to:
Control Panel -> System -> Advanced Settings -> Environment Settings -> System Variables
Scroll down to select path, then click edit. Copy the entire line to a text document, and add your install directory for Python.exe (and the scripts folder) using ';' as a delimiter between different directories. Copy this back to Path and save. (Additionally close your command prompt window to reset it.)
For my Windows 7 machine, I added:
;C:\Python33;C:\Python33\scripts;
Take care when editing this file. There are many videos out there describing this in detail if you feel unsure about changing this.
Installing Modules (Such as setup_tools)
Once Python is registered in the Path file, download and unzip setup_tools to a folder within your Python install directory called 'modules'. I use ExtractNow to unzip, as it will unzip twice (as required) automatically.
Open a command prompt window again, and direct it to change directories by typing
cd [directory for module you want to install]
On my computer, this would be
cd C:\Python33\modules\distribute-0.6.40
Again, I use distribute, rather than setup_tools as it sounds like you need would for Python2. Simply use the appropriate directory. Press enter to change the directory.
Once you've entered this and it shows a changed directory, type:
python setup.py install
This indicates that you want to use the program python to use the setup.py file in the specified folder to install the module. This should be successful, and will write many lines of code.
If you want to install other modules, you would install them in a similar way. Python would automatically use setup_tools on your computer to finish each install.
Remember to import at the start of your script when using them to code:
import [module]
Happy Programming!

Resources