extracting and creating ipk files - package

ipk packages are the intallation packages used by opkg.
I'm trying to extract the contents of one of them and also create my own ipk.
I've read that I should be able to untar them but that is not true.
I've tried:
tar -zxvf mypack.ipk
and I get:
zip: stdin: not in gzip format
I've also tried:
tar -xvf mypack.ipk
and I get:
tar: This does not look like a tar archive
I've found that most of the information on the internet regarding ipk's are inaccurate.
My ipk was generated by bitbake. I'm having a hard time with bitbake and want to avoid using it.
Any ideas on how to extract and how to create ipk files? A simple template with a single package would be useful to have.

I figured it out.
You can extract the main package with the ar x command, then extract the control.tar.gz with the tar -zxf command.

I have tested "ar x package-name.ipk" command but it didn't help
I found bellow command which worked perfectly
tar zxpvf package-name.ipk
This extracts three files:
debian-binary
data.tar.gz
control.tar.gz
use the same command to open data.tar.gz and control.tar.gz files
for more information refer to
https://cognito.me.uk/computers/manual-extractioninstallation-of-ipk-packages-on-gargoyleopenwrt/

You need to create a control file, and then do some archiving using tar and ar. In my case, I was distributing just python scripts, so there was no architecture dependency. You should check the control and Makefile into version control, and delete all the other intermediate files.
Here are the contents of control
Package: my-thing-python
Version: 1.0
Description: python scripts for MyCompany
Section: extras
Priority: optional
Maintainer: John
License: CLOSED
Architecture: all
OE: my-thing-python
Homepage: unknown
Depends: python python-distutils python-pyserial python-curses python-mmap python-ctypes
Source: N/A
Here is my Makefile which sits in the same directory as all my python scripts.
all: my-thing-python.ipk
my-thing-python.ipk:
rm -rf ipk
mkdir -p ipk/opt/my-thing-python
cp *.py ipk/opt/my-thing-python
tar czvf control.tar.gz control
cd ipk; tar czvf ../data.tar.gz .; cd ..
echo 2.0 > debian-binary
ar r my-thing-python.ipk control.tar.gz data.tar.gz debian-binary
clean: FORCE
rm -rf ipk
rm -f control.tar.gz
rm -f data.tar.gz
rm -f my-thing-python.ipk
FORCE:

Extracting with these commands:
Extract the file by running the command:
ar -xv <.ipk file>
Extract the control.tar.gz file by running the command:
tar -zxvf control.tar.gz
data.tar.gz : untar by running the command:
tar –zxvf data.tar.gz

If you want a list of files in an ipk, you can do something like:
#!/bin/sh
for f
do
tar -x -z -f $f ./data.tar.gz -O | tar tvzf -
done
-O is extract to standard output.
ipk files used to be AR (like DPKG), but are now tgz.
I feel that some dpkg utility ought to cope with ipkg files, but I haven't found the right one.

Related

Does the "%defattr" directive affect the "%post" section also in rpm spec file?

I have a .spec file with code somewhat like this:
%files
%defattr(-,xyz, xyz)
%verify(md5 size mtime mode) %attr(755, xyz, xyz) /usr/bin/app1
%verify(md5 size mtime mode) %attr(755, xyz, xyz) /usr/bin/app2
%post
mkdir -p /apps/1/logs
mkdir -p /apps/2/logs
mkdir -p /apps/3/logs
mkdir -p /apps/4/logs
mkdir -p /apps/5/logs
ln -sf /usr/bin/app1 /usr/bin/app3
touch /home/xyz/abc.log
will the %defattr also affect the default attributes of files and directories getting created in the post section??
No. You'll need to explicitly chown/chmod anything you do in %post. It's preferred to not have them in %post because things can break that way (like rpm -V). Why wouldn't you want that to be done in %build?

COPY GZIP HDFS data into vertica

I want to COPY HDFS (gzipped)data into Vetica.
I am using following command. but its not working
COPY pix001 SOURCE Hdfs(url='http://hadoopnemenode.com:50070/webhdfs/v1/bq-upload/pix/m=03/d=01/03-01.txt.gz', username='xyz') GZIP DELIMITER E'\t';
Anyone know better way to do this
Thanks
Yes there is GZIP support just need to compile GZIP libs [Vertica Guys helped me finally :)]
here are the steps :
# cd /opt/vertica/sdk/examples/
# make
# vsql -f FilterFunctions.sql
dbadmin=> CREATE LIBRARY GZipLib AS '/opt/vertica/sdk/examples/build/GZipLib.so';
dbadmin=> CREATE FILTER GZip AS LANGUAGE 'C++' NAME 'GZipUnpackerFactory' LIBRARY GZipLib;
COPY abc002 SOURCE Hdfs(url='http://hadoop-namenode.com:50070/webhdfs/v1/03-01.txt.gz', username='xyz') filter GZip() DELIMITER E'\t';
Adding to roy answer,
Steps to make(build) are given below, (#2nd step on roy answer)
sudo apt-get install g++
sudo apt-get install zlib1g-dev # for gzip
g++ -lz -D HAVE_LONG_INT_64 -I /opt/vertica/sdk/include -Wall -shared -Wno-unused-value -fPIC -o /opt/vertica/sdk/examples/build/GZipLib.so /opt/vertica/sdk/examples/FilterFunctions/GZip.cpp /opt/vertica/sdk/include/Vertica.cpp
Hint: -lz flag to link the zlib library statically with GZip.so
Vertica Documentation for compiling UDF
It doesn't look like copying from HDFS supports GZIP?:
https://my.vertica.com/docs/7.0.x/HTML/Content/Authoring/HadoopIntegrationGuide/HDFSConnector/LoadingDataFromHDFS.htm
I don't see it in that doc, in any case.

How to replace MacPort's libiconv with Mac's default 64-bit version?

MacPorts installed "libiconv #1.14_0+universal" as a dependency on my system. This happens to be a 32-bit flavor and it started causing issue when I tried to compile a voice recognition software called Simon Listens.
While googling I found out that that Mac actually ships with a 64-bit flavor of libiconv by default and I was able to locate the said files on my system:
$ find /usr/lib -name libiconv*
/usr/lib/libiconv.2.4.0.dylib
/usr/lib/libiconv.2.dylib
/usr/lib/libiconv.dylib
In order to use the system library, the quickest way I could think of was to uninstall MacPort's version of libiconv so that the system's library would end up getting selected as a fallback as it has to present (my guess) somewhere downstairs on the PATH already.
But that failed due to dependecies:
$ sudo port uninstall libiconv #1.14_0+universal
Unable to uninstall libiconv #1.14_0+universal, the following ports depend on it:
...
So now my question is how can I tell MacPort to replace its dependency graph to point to and use the library already on my system?
Another approach to avoid MacPorts libiconv issues would be to build simon against a fresh MacPorts system plus the necessary packages such as cyrus-sasl2, zlib, portaudio and kdesdk4 in a custom location, e. g. /opt/macports-simon.
The following code worked for me on my machine running Mac OS X 10.6.8:
# compile simon on Mac OS X 10.6.8 using MacPorts for the installation of zlib, portaudio and kdesdk4
# http://www.simon-listens.org
# http://sourceforge.net/projects/speech2text/
# get a root shell
sudo -H -i
# prevent idle sleep
pmset -a force sleep 0 displaysleep 0 disksleep 0
mv -i /opt/local /opt/local-off
mv -i /usr/local /usr/local-off
cd /tmp
mkdir buildsimon || exit 1
cd buildsimon || exit 1
# create custom /opt/macports-simon to install zlib, portaudio and kdesdk4
# cf. http://guide.macports.org/#installing.macports.source.multiple
MP_PREFIX='/opt/macports-simon'
unset PATH
export PATH='/bin:/sbin:/usr/bin:/usr/sbin'
curl -L -O https://distfiles.macports.org/MacPorts/MacPorts-2.0.4.tar.bz2
tar -xjf MacPorts-2.0.4.tar.bz2
cd MacPorts-2.0.4 || exit 1
./configure --prefix="${MP_PREFIX}" --with-applications-dir="${MP_PREFIX}/Applications"
make
make install
cd /tmp/buildsimon
unset PATH
export PATH="${MP_PREFIX}/bin:/bin:/sbin:/usr/bin:/usr/sbin"
# get the Portfiles and update the system
port -v selfupdate
# install cyrus-sasl2
port -f uninstall cyrus-sasl2
port clean --all cyrus-sasl2
port extract cyrus-sasl2
cd "$(port dir cyrus-sasl2)"/work/cyrus-sasl-2.1.23
printf '%s\n' H '/\(darwin\[15\]\)/s//\1./g' wq | sudo ed -s config/ltconfig
printf '%s\n' H '/\(darwin\[15\]\)/s//\1./g' wq | sudo ed -s saslauthd/config/ltconfig
cd /tmp/buildsimon
port -f -s install cyrus-sasl2
otool -L /opt/macports-simon/lib/libsasl2.dylib
port -f install zlib
port -f install portaudio
port -f install kdesdk4
port installed zlib portaudio kdesdk4 cyrus-sasl2
# enable dbus with launchd
# http://www.freedesktop.org/wiki/Software/dbus
# open -e dbus-1.5.8/README.launchd
launchctl load -w /Library/LaunchDaemons/org.freedesktop.dbus-system.plist
launchctl load -w /Library/LaunchAgents/org.freedesktop.dbus-session.plist
sudo -u _mysql mysql_install_db5
sudo port load mysql5-server
# todo: how to configure simon to use /opt/macports-simon directly?
ln -isv "${MP_PREFIX}" /usr/local
cd /tmp/buildsimon
# http://sourceforge.net/projects/speech2text/
curl -L -O http://netcologne.dl.sourceforge.net/project/speech2text/simon/0.3.0/simon-0.3.0.tar.bz2
tar -xjf simon-0.3.0.tar.bz2
cd simon-0.3.0 || exit 1
# Note that /usr/local got symlinked to "${MP_PREFIX}" above!
unset PATH
export PATH='/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin'
# the following commands are taken from simon-0.3.0/build.sh
mkdir build 2> /dev/null
cd build || exit 1
cmake -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ..
# append ${MP_PREFIX}/lib/libiconv.dylib to gcc command in link.txt file
printf '%s\n' H '/\/usr\/bin\/gcc/s|\(.*\)|\1 '"${MP_PREFIX}"'/lib/libiconv.dylib|' wq |
ed -s julius/julius/CMakeFiles/juliusexe.dir/link.txt
# replace gcc option ' -bundle ' with ' -dynamiclib '
egrep -Ilsr -Z -e ' -bundle ' . |
xargs -0 -n 1 /bin/sh -c 'printf "%s\n" H "g/ -bundle /s// -dynamiclib /g" wq | /bin/ed -s "${1}"' argv0
make
touch ./julius/gramtools/mkdfa/mkfa-1.44-flex/*
make
make install
# ldconfig # not needed on Mac OS X
kbuildsycoca4
echo -e "**** Build completed ****\n\nThe executable file \"simon\" is now ready and has been installed.\n\nIssue \"simon\" to start it."
unset PATH
export PATH="${MP_PREFIX}/bin:/bin:/sbin:/usr/bin:/usr/sbin"
otool -L "${MP_PREFIX}/bin/simon"
simon
mv -i /opt/local-off /opt/local
mv -i /usr/local-off /usr/local

tar exclude file pattern

I am trying to create a tar with follwing command:
tar -cvf myFile.tar -X exclude-files.txt myContentDirectory
and my exclude-file has follwing patterns to exclude:
**/*.bak
**/*.db
**/*.html
But i dont see these file types being excluded out in my tar.
What am I doing wrong here?
I found that when i have just one pattern in my exclude-files.txt, lets say only
**/*.bak
it does work. But not with multiple file patterns (EACH ON NEW LINE)
I think this:
*.bak
*.db
*.html
is the correct format for the exclude file if you want to exclude a particular directory you could do:
some-dir/*.db
Also your command should look like this:
tar -cvf myFile.tar -X exclude-files.txt myContentDirectory
Sorry if this answer is a little late.
tar -cO --exclude=*.bak myContentDirectory | tar -O --delete '*.db' | tar -O --delete '*.html' > myFile.tar
See, what you're doing here is creating the tar, but sending it to stdout instead of to a file then piping that into tar to delete the stuff you don't want, one or more times and finally writing the output to a file.
You can even test it first like this:
tar -cO --exclude=*.bak myContentDirectory | tar -O --delete '*.db' | tar -O --delete '*.html' | tar -tv
Which will spit out a list of all the files remaining in the archive.
Most likely the order of the command is incorrect.
tar -cvf myFile.tar -X exclude-files.txt myContentDirectory
should be something like
tar cv -X exclude-files.txt -f myFile.tar myContentDirectory
PS. I haven't looked into the filters itself. Most likely order of the parameters is the issue.
If issues is in the filters/patterns - it's easier to test one by one with --exclude option.

Git ignore file for C projects

I've just started to learn C (using Thinking In C) and I'm wondering about what files I should be ignoring in a C project's git repository.
No suggestion can be too obvious -- I'm a total noob. Thanks!
I guess there will be a few generated files that you don't wan't to be sticking in your repo (assuming your build output dir is in your git heirachy):
object files (.o, o.obj)
libraries (.lib)
DLLs, shared objects (.so, .dll)
Executables (.exe, a.out ?)
GIT ignore files are something I tend to do iteratively. "Hey, I don't need those things in my repo" ...
Edit: re dmckee's comment
Yep, you definately want to be ignoring swap files, temp files etc. I have the following as a baseline for my .gitignore:
*.swp
.~
thumbs.db
You can also setup your build to happen in a subdirectory say build and then you can ignore the whole thing inside .gitignore
build/
And you're done.
I use this in my .gitignore
But I am building for micro-controllers, so I don't know if it helps you much.
The easiest way to know, is just do a make clean, then add all your files, then do a make all and see what extra stuff appears.
#Some of these are related to eclipse. So i keep them out of my repo
.cproject
.dep/
.project
.settings/
#files being edited
*~
# make and build files
*.lst
*.o
*.eep
*.lss
*.map
*.sym
# I keep these, since I prefer having the reference of the final build
# *.elf
# *.hex
Github's .gitignore file templates cover most of the common files for projects in a variety of languages.
The C .gitignore template looks like this:
# Prerequisites
*.d
# Object files
*.o
*.ko
*.obj
*.elf
# Linker output
*.ilk
*.map
*.exp
# Precompiled Headers
*.gch
*.pch
# Libraries
*.lib
*.a
*.la
*.lo
# Shared objects (inc. Windows DLLs)
*.dll
*.so
*.so.*
*.dylib
# Executables
*.exe
*.out
*.app
*.i*86
*.x86_64
*.hex
# Debug files
*.dSYM/
*.su
*.idb
*.pdb
# Kernel Module Compile Results
*.mod*
*.cmd
.tmp_versions/
modules.order
Module.symvers
Mkfile.old
dkms.conf
Using a *nix system and a Makefile, you could add each generated file to .gitignore.
As an example I use the following when creating an executable from a single source (Example for a C executable generation):
%: %.c
gcc -o $# $<
grep '^$#$$' .gitignore > /dev/null || echo '$#' >> .gitignore
The following line can be added to other recipes to add target $# to .gitignore file:
grep '^$#$$' .gitignore > /dev/null || echo '$#' >> .gitignore
Explanation:
grep '^$#$$' .gitignore : searches for the target in .gitignore
^ indicates start of line
$$ is a single $ (but Makefile needs $$ to work) and indicates the end of the line
'^$#$$' represents the target name
|| : executes the next command only if the left hand one failed
so only if grep ... does not find the target name in .gitignore, echo ... is executed
echo '$#' >> .gitignore: adds the target name to .gitignore
Eventually, you will add to clean and rebuild everything to make sure all files are correctly ignored

Resources