The project I'm working in uses fabric for many build steps and requires a offline build as fallback.
I'm currently stuck at installing python packages provided in tarballs.
The thing is I have trouble getting into the newly extracted directory and running setup.py install in there.
#task
def deploy_artifacts():
"""Installs dependencies from local path, useful for offline builds"""
#TODO: Handle downloading files and do something like this bellow
tmpdir = tempfile.mkdtemp()
artifacts_path = ''
if not 'http' in env.artifacts_path:
artifacts_path = env.artifacts_path
with lcd(artifacts_path):
for f in os.listdir(artifacts_path):
if 'gz' in f:
put(f, tmpdir)
tar = os.path.join(tmpdir, f)
target_dir = os.path.join(tempfile.gettempdir(), normalize(f))
if not files.exists(target_dir):
run('mkdir %s' % target_dir)
else:
run('rm -rf %s' %target_dir)
run('mkdir %s' % target_dir)
run('tar xf %s -C %s' % (tar, target_dir))
run('rm %s' % tar)
with cd(target_dir):
sudo('python setup.py install')
I come from reading the tar man page for the bazillion time and I got nowhere near to getting what I want.
Did some of you face a situation like this? is there some other (read: better) approach to this scenario?
There's nothing wrong (in principle) with what you're trying do. Maybe just take smaller steps getting there. Rather than using temporary directories, it might make debugging easier if everything was put in a systematic location that has known permissions that nothing else writes to by convention. At least that would let you use some combination of fabric and manual intervention to check what is going wrong.
In the longer term, there are a few alternatives that I see. For simplicity you want the online and offline versions to work the same way, and that means fetching packages using easy_install / pip for both cases.
One way to do this is to build a mirror of PyPi. The right way to do this if you've got plenty of storage space (30Gb) is to use software that implements PEP381 (Mirroring Infrastructure for PyPI), there is already a client that does this (pep381client). A number of other projects are available that do similar things (basketweaver, djangopypi2, chishop).
An alternative is to consider a lighter weight proxying scheme. I've been looking a pip2pi and pipli. I'm unsure if they will work directly with easy_install, but it would be worth a try.
It's also worth noting that if you were using pip, you could have installed directly from the tarballs.
Related
I am new to scripting. I was trying to create a chocolatey package that would automatically do a custom(not typical) install. For example with MariaDB installations, I would like to specify which parts of the server to install and the username and password for the database.
I was trying to practice on Libreoffice where the package chooses Custom install and intalls only libre Writer. But the following script does the default installations what am I missing here? thanks.
chocolateinstall.ps1
e$ErrorActionPreference = 'Stop'; # stop on all errors
$toolsDir = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
$fileLocation = ".\LibreOffice_7.2.7_Win_x64.msi"
$pp= Get-PackageParameters
if (!$pp['SetupType']){$pp['SetupType']='Custom'}
if (!$pp['InstallOption']){$pp['InstallOption']='LibreWriter'}
$packageArgs = #{
packageName = $env:ChocolateyPackageName
unzipLocation = $toolsDir
fileType = 'msi'
file = $fileLocation
softwareName = 'libre*'
checksum = '...'
checksumType = 'sha256'
silentArgs = "/qn /norestart /l*v `"$($env:TEMP)\$($packageName).$($env:chocolateyPackageVersion).MsiInstall.log`"" # ALLUSERS=1 DISABLEDESKTOPSHORTCUT=1 ADDDESKTOPICON=0 ADDSTARTMENU=0
validExitCodes= #(0, 3010, 1641)
}
Install-ChocolateyPackage #packageArgs
This is a pretty huge question, to one extent, and quite easy in another!
Short answer:
Your example above doesn't pass any of the package arguments you're crafting (e.g. $pp['InstallOption']) to the actual installer. They're being stored in the variable ($pp) and never used.
The values you want to use should be passed in to Install-ChocolateyPackage using the silentArgs parameter.
However, I don't think the arguments you have there are going to work, even if you pass them in (though I may be mistaken).
Longer answer:
MSIs don't just accept random arguments.
Accepted arguments vary hugely by installer, and by software, and there's no guarantee you can do what you want silently from the commandline.
You can use something like Orca to find out what arguments an MSI may support, or search for documentation (or other folk having done the work before), or create an MST file to apply.
You could also use the Chocolatey for Business Package Builder, which scans the file and tries to identify useful arguments you can pass - though this requires a paid Business license for Chocolatey.
gitolite info didn't work, adding keys turned them into a no access key and did NOT create a corresponding entry in auth-keys file.
To fix this run gitolite setup on gitolite server
Question: what could have landed me in that mess?
And what does gitolite setup do when invoked for the n-th time (it's no longer setting things up, according to the docs it fixes hooks, but I wonder what the use case would be and which was mine)?
More details on gitolite info
gitolite info command is invoked like so:
> ssh git-user#ser-git
PTY allocation request failed on channel 0
hello git-admin, this is ...#... running gitolite3 3.6.7-2 (Debian) on git 2.17.1
R W some-repository
R W gitolite-admin
R W testing
Connection to ser-git closed.
Bad output is: FATAL: unknown git/gitolite command: 'info'
More details: keys without access.
gitolite sshkeys-lint was showing keys with (no access), now those keys have access as I set them (now meaning after gitolite setup).
ssh-keygen -lf /home/repo/.ssh/authorized_keys | wc -l (or without piped part, regardless) number of keys and their names indicated I didn't have the newest one added.
Similar question that did not work for me: keydir entries not propagating to authorized_keys
Docs pretty much had the answer once I dug deeper, I guess. Which is fairly nice of #sitaramc.
Without options, 'gitolite setup' is a general "fix up everything" command
(for example, if you brought in repos from outside, or someone messed
around with the hooks, or you made an rc file change that affects access
rules, etc.)
Symptoms keys stopped propagating and error FATAL: unknown git/gitolite command: 'info' on ssh git-user#ser-git. Fix was to run gitolite setup. So onto first question, the title one:
what does gitolite setup fix?
gitolite setup is implemented here
my Perl is rather weak, but there's a setup function in line 56. It calls args (which parses options, so here it had nothing to parse), then unless h_only (hooks only arg for setup), which wasn't used, so we skip compile and POST_COMPILE trigger and go for the hooks.
sub setup {
my ( $admin, $pubkey, $h_only, $message ) = args();
unless ($h_only) {
setup_glrc();
setup_gladmin( $admin, $pubkey, $message );
_system("gitolite compile");
_system("gitolite trigger POST_COMPILE");
}
hook_repos(); # all of them, just to be sure
}
package Gitolite::conf::store has hook_repos(), line 228: we change the dir to repo base dir (as per config file), and for each phy_repo we do hook_1(phy_repo). What is a phy_repo? a physical one.
same package, different method and line: hook_1($repo) in line 354.
Method hook_1($repo)
It's quite literally about fixing all the hooks.
Recreates dirs for common and admin hooks.
Rewrites update_hook (common) and post_update_hook (admin).
Sets 755 permissions for both common and admin hooks.
Then using ln_sf it symlinks the folders for common/admin hooks.
ln_sf is in common module, in line 162
I have an application that uses libnl. It can use either versions (1 or 3), and during configure it tries first to use ibnl3 and fallback to libnl-1 if libnl3 was not found.
My app uses another library that also uses libnl.
The problem is that I only have libnl1-dev on my machine so my app must use it.
But the library that I use uses libnl3 (was installed with yum i guess it's static linked)
so i have both version and my application crashes!!
here are some prints
ldd myapp.so|grep libnl
libnl.so.1 => /lib64/libnl.so.1 (0x00007fda33eb5000)
libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00007fda32a3d000)
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007fda3281b000)
yum list|grep libnl
libnl.x86_64 1.1.4-3.el7
libnl-devel.x86_64 1.1.4-3.el7
libnl3.x86_64 3.2.28-2.el7
libnl3-cli.x86_64 3.2.28-2.el7
libnl.i686 1.1.4-3.el7
libnl-devel.i686 1.1.4-3.el7
libnl3.i686 3.2.28-2.el7
libnl3-cli.i686 3.2.28-2.el7
if in install libnl3-dev it fixes the issue
is there another solution?
if I install libnl3-dev it fixes the issue is there another solution?
There are other solutions, but the bottom line is that you can only have libnl.so.1 or libnl-3.so.200, but not both.
Fixing this by "going all in on libnl-3" is the simplest solution.
The alternative is to "go all on on libnl-1", which means rebuilding anything that requires libnl-3 from source (against libnl-1). This is assuming that your other dependencies can be built against libnl-1 at all (which is by no means guaranteed).
Running chef-solo (Installing Chef Omnibus (12.3)) on centos6.6
My recipe has the following simple code:
package 'cloud-init' do
action :install
end
log 'rpm-qi' do
message `rpm -qi cloud-init`
level :warn
end
log 'yum list' do
message `yum list cloud-init`
level :warn
end
But it outputs the following:
- install version 0.7.5-10.el6.centos.2 of package cloud-init
* log[rpm-qi] action write[2015-07-16T16:46:35+00:00] WARN: package cloud-init is not installed
[2015-07-16T16:46:35+00:00] WARN: Loaded plugins: fastestmirror, presto
Available Packages
cloud-init.x86_64 0.7.5-10.el6.centos.2 extras
I am at a loss as to why rpm/yum and actually rpmquery don't see the package as installed.
EDIT: To clarify I am specifically looking for the following string post package install to then apply a change to the file (I understand this is not a very chef way to do something I am happy to accept suggestions):
rpmquery -l cloud-init | grep 'distros/__init__.py$'
I have found that by using the following:
install_report = shell_out('yum install -y cloud-init').stdout
cloudinit_source = shell_out("rpmquery -l cloud-init | grep 'distros/__init__.py$'").stdout
I can then get the file I am looking for and perform
Chef::Util::FileEdit.new(cloudinit_source.chomp(''))
The file moves based on the distribution but I need to edit that file specifically with in place changes.
Untested code, just to give the idea:
package 'cloud-init' do
action :install
notifies :run,"ruby_block[update_cloud_init]"
end
ruby_block 'update_cloud_init' do
block do
cloudinit_source = shell_out("rpmquery -l cloud-init | grep 'distros/__init__.py$'").stdout
rc = Chef::Util::FileEdit.new(cloudinit_source.chomp(''))
rc.search_file_replace_line(/^what to find$/,
"replacement datas for the line")
rc.write_file
end
end
ruby_block example taken and adapted from here
I would better go using a template to manage the whole file, what I don't understand is why you don't know where it will be at first...
Previous answer
I assume it's a compile vs converge problem. at the time the message is stored (and so your command is executed) the package is not already installed.
Chef run in two phase, compile then converge.
At compile time it build a collection of resources and at converge time it execute code for the resource to get them in the described state.
When your log resource is compiled, the ugly back-ticks are evaluated, at this time there's a package resource in the collection but the resource has not been executed, so the output is correct.
I don't understand what you want to achieve with those log resources at all.
If you want to test your node state after chef-run use a handler maybe calling ServerSpec as in Test-Kitchen.
This is a cross post from Perl Monks and Mahalo answers, where I have not received a satisfactory response yet. Thanks for your time and spirit:
Why do I get this error message from perl:
Can't call method "scaleY" on an undefined value at C:/strawberry/perl +/site/lib/ Image/Seek.pm line 137?
I am getting the error in the title when calling the Image::Seek module from my script. My script is basically a rehash of the module's suggested code.
Here's the error again:
Can't call method "scaleY" on an undefined value at C:/strawberry/perl +/site/lib/ Image/Seek.pm line 137.
Here's my code:
#!/usr/local/bin/perl
use Imager;
use Image::Seek qw(loaddb add_image query_id savedb);
loaddb("haar.db");
my $img = Imager->new("photo-1.jpg")
or die Imager->errstr;
# my $img = Imager->new();
# $img->open(file => "photo-1.jpg")or die Imager->errstr;
add_image($img, 1);
savedb("haar.db");
Here's the section of the Image::Seek module causing the issue:
sub add_image_imager {
my ($img, $id) = #_;
my ($reds, $blues, $greens);
require Imager;
my $thumb = $img->scaleX(pixels => 128)->scaleY(pixels => 128);
for my $y (0..127) {
my #cols = $thumb->getscanline(y => $y);
for (#cols) {
my ($r, $g, $b) = $_->rgba;
$reds .= chr($r); $blues .= chr($b); $greens .= chr($g);
}
}
addImage($id, $reds, $greens, $blues); }
Line 137 is:
my $thumb = $img->scaleX(pixels => 128)->scaleY(pixels => 128);
If I remove
->scaleY(pixels => 128)
then line 129:
my #cols = $thumb->getscanline(y => $y);
gives me essentially the same error.
At this point I'm just trying to add one image to the database. There is an image in the directory where I'm running the script to add the image, named "photo-216.jpg". If I change the name to "photo-1.jpg" or "photo-0.jpg" and change the corresponding "add_image" and "query_id" to respectively 1 or 0, it's the same result.
I do have a database that is 385 KB big that comes from running makedb.pl below, but it is filled with null characters. I renamed this "haar.db". This is the database that gives me the error. If I recreate the haar.db file as an empty one, then the script hangs and after a couple of minutes, it give this different message:
"This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information."
If there is no "haar.db" the file still gives me the error in this post's title and unlike running makedb.pl, gives me no database named "haar.db".
By the way I get multiple examples of this post's title error also when trying to run this database filling script: http://www.drk7.jp/pub/imgseek/t/makedb.pl.txt/, which I was alluding to before. I obviously removed the .txt extension before trying it. The makedb.pl script is from this Japanese site: http://www.drk7.jp/MT/archives/001258.html.
If I run makedb.pl in a directory of 2423 scanned collectible postage stamps images, I get 362 instances of the error. The 2423 stamps is the number I have after removing the "small" thumbnail versions which I orignally thought might be causing the issue.
Could it be, that some of the images are less than 128 pixels and that is the issue? However if this is true why does the database get filled with null characters?...Unless they are not really null even though the editor I'm using, Notebook++, says they are.
Also note my images are of stamps which are only sometimes perfect squares. Otherwise, sometimes they are "landscape" sometimes "portrait". Maybe the issue is when the "landscape" scaled images get an X axis of 128 pixels and then their Y axis ends up less or much less. Could this be?
Thanks much
Update: Answer completely re-organized.
Image::Seek is not checking if
scaleX returned error. In your case, for some images, scaleX is failing.
You seem to know for which images scaleX is failing. So, leave your current
code aside, and put together a short test script:
#!/usr/bin/perl
use strict;
use warnings;
use Imager;
die "Specify image file name\n" unless #ARGV;
my ($imgfile) = #ARGV;
my $img = Imager->new;
$img->read( file => $imgfile )
or die "Cannot read '$imgfile': ", $img->errstr;
my $x_scaled = $img->scaleX( pixels => 128 )
or die 'scaleX failed: ', $img->errstr;
my $thumb = $x_scaled->scaleY( pixels => 128 )
or die 'scaleY failed: ', $x_scaled->errstr;
__END__
Running this test script, you got the error message:
Cannot read 'photo-1.jpg': format 'jpeg' not supported - formats bmp,
ico, pnm, raw, sgi, tga available for reading
indicating the underlying problem: When you installed Imager via Strawberry
Perl's cpan, the libraries for png, jpg etc were not installed. One
solution is to build those libraries with the gcc compiler provided with
Strawberry Perl.
First, you will need zlib.
C:\Temp\zlib-1.2.3> copy win32\Makefile.gcc Makefile
Set prefix = /strawberry/c/local in the Makefile. Compile. You may have to
manually copy the files zlib.h and zconf.h to
C:\strawberry\c\local\include and zlib1.dll, libz.a and libzdll.a to
C:\strawberry\c\local\lib (I don't know because I do not use Strawberry Perl very often and my Strawberry environment is very neglected.)
Then, get libpng. I used the source archive without config script.
C:\Temp\libpng-1.2.38> copy scripts\makefile.mingw Makefile
C:\Temp\libpng-1.2.38> make prefix=/strawberry/c/local ZLIBLIB=/strawberry/c/local/lib ZLIBINC=/strawberry/c/local/include
This built the PNG library. Again, you may have to manually copy the .dll,
.a and .h files to the appropriate directories. I did because of my less
than perfect Strawberry environment.
Finally, get the JPEG library.
C:\Temp\jpeg-7> copy Makefile.ansi Makefile
Make sure to edit this file and set CC=gcc. Customize jconfig.h according
to the instructions in jconfig.txt. I used jconfig.dj as a basis.
You might also want to set
CFLAGS= -O2
SYSDEPMEM= jmemansi.o
in Makefile, and
#define DEFAULT_MAX_MEM 4*1024*1024
in jconfig.h. After running make, again copy the files as needed (and as explained by install.txt).
Once the libraries are installed, you can
C:\Temp> SET IM_INCPATH=C:\strawberry\c\local\include
C:\Temp> SET IM_LIBPATH=C:\strawberry\c\local\lib
C:\Temp> cpan
cpan> force install Imager
which yields:
gif: includes not found - libraries not found
ungif: includes not found - libraries not found
jpeg: includes found - libraries found
png: includes found - libraries found
tiff: includes not found - libraries not found
freetype2: includes not found - libraries not found
freetype2: not available
T1-fonts: includes not found - libraries not found
TT-fonts: includes not found - libraries not found
w32: includes found - libraries found
If all of this is too much work, it is ... sigh I just realized the
binaries are available at GnuWin32.