How to construct rule names according to an external factor? - shake-build-system

Till now i had the following code:
shakeArgsWith opts additionalFlags $ \flags targets -> return $ Just $ do
...
-- Get target arch from command line flags
let givenArch = listToMaybe [v | AFlag v <- flags]
let arch = fromMaybe defArch givenArch
...
-- Set the main target
let outName = masterOutName projType toolchain projName
let mainTgt = (case projType of
Binary _ -> "bin"
Archive -> "lib")
</> prettyShowArch arch
</> show variant
</> outName
-- Set the build directory for the current run
let buildDir = bldDir </> show toolchain </> prettyShowArch arch </> show variant
...
mainTgt %> \out -> do ...
...
buildDir <//> "*.o" %> \out -> do ...
...
Meaning that the names of the rules where constructed according to a command line flag that i was parsing (they contained the arch variable).
So if i gave shake --arch=x64 i was building the main target in bin/x64/Release directory and my intermidiate build files in the tmp/x64/Release folder accordingly.
But now instead of using the command line flag, i want the shared arch variable that is used to construct the rule names to be populated according to the output of some command, for example if i could define some top level action it would be this:
Stdout sout <- quietly $ cmd (EchoStdout False) (EchoStderr False) "gcc -dumpmachine" :: Action (Stdout String)
let foundArch = show (gccTripletToArch sout)
and use the variable foundArch instead of the arch when constructing the mainTgt and buildDir names. Obviously this cannot be done, as even the only top level rule that can be created with the action function returns Rule (). What can i do instead?

I think you should be able to do:
shakeArgsWith opts additionalFlags $ \flags targets -> do
Stdout sout <- cmd (EchoStdout False) (EchoStderr False) "gcc -dumpmachine"
let arch = show (gccTripletToArch sout)
return $ Just $ do
let buildDir = bldDir </> show toolchain </> prettyShowArch arch </> show variant
...
buildDir <//> "*.o" %> \out -> do ...
The target patterns in Shake do have to be statically known, as that ensures some important properties with respect to quick rebuilding (you can guarantee a change in one place has predictable effects). However, you can run commands to determine things like arch, compiler version etc. before you create the build script, and bake them in.
Another viewpoint is that you are dynamically generated a build system based on the arch. Using Shake, as a Haskell EDSL, that is no particular problem, and you arguably were doing that before with the command line.

Related

Unable to compile PnetCDF: 'tst_rec_vars' test fails

I am trying to install PnetCDF from source. I used the 1.12.2 version for installation. MPICH, Zlib, Szip, HDF5 and NetCDF-4 have been installed from source successfully (HDF5 and NetCDF-4 have been built with parallel I/O support).
MPICH is installed as follows (v3.4.2):
./configure --prefix=${DIR}/mpich --with-device=ch4:ofi CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" --enable-fortran
make
make check
make install
export PATH=${DIR}/mpich/bin:${PATH}
Where DIR is the root directory all the libraries are installed to (with their own directory).
Zlib (v1.2.11):
./configure --prefix=${DIR}/grib2
make
make check
make install
export CPPFLAGS="-I${DIR}/grib2/include"
export LDFLAGS="-L${DIR}/grib2/lib"
export LD_LIBRARY_PATH="${DIR}/grib2/lib:$LD_LIBRARY_PATH"
Szip (v2.1.1):
./configure --prefix=${DIR}/szip
CC=mpicc
make
make check
make install
export LD_LIBRARY_PATH="${DIR}/szip/lib:$LD_LIBRARY_PATH"
export CPPFLAGS="-I${DIR}/szip/include ${CPPFLAGS}"
export LDFLAGS="-L${DIR}/szip/lib ${LDFLAGS}"
HDF5 (v1.8.22):
export HDF5=${DIR}/hdf5
./configure --prefix=${HDF5} --with-szlib=${DIR}/szip --with-zlib=${DIR}/grib2 --enable-parallel CC=mpicc --enable-fortran FC=mpifort
make
make check
make install
export LD_LIBRARY_PATH="${HDF5}/lib:$LD_LIBRARY_PATH"
export CPPFLAGS="-I${HDF5}/include ${CPPFLAGS}"
export LDFLAGS="-L${HDF5}/lib ${LDFLAGS}"
export PATH="${HDF5}/bin:${PATH}"
NetCDF4-C (v4.8.0):
./configure --prefix=${DIR}/netcdf --disable-dap --enable-parallel-tests --enable-large-file-tests --disable-extreme-numbers CC=mpicc
make
make check
make install
export PATH="${NETCDF}/bin:${PATH}"
export LD_LIBRARY_PATH="${NETCDF}/lib:${LD_LIBRARY_PATH}"
export CPPFLAGS="-I${NETCDF}/include ${CPPFLAGS}"
export LDFLAGS="-L${NETCDF}/lib ${LDFLAGS}"
export LIBS="$(nc-config --libs) ${LIBS}"
NetCDF4-Fortran (v4.5.3):
./configure --prefix=${NETCDF} --enable-large-file-tests --enable-parallel-tests CC=${DIR}/mpich/bin/mpicc FC=mpifort F77=mpif77
make
make check
make install
The HDF5 configuration shown by h5pcc -showconfig is given below (only the part I think relevant is included):
Features:
---------
Parallel HDF5: yes
High-level library: yes
Build HDF5 Tests: yes
Build HDF5 Tools: yes
Threadsafety: no
Default API mapping: v18
With deprecated public symbols: yes
I/O filters (external): deflate(zlib),szip(encoder)
MPE:
Direct VFD: no
(Read-Only) S3 VFD: no
(Read-Only) HDFS VFD: no
dmalloc: no
Clear file buffers before write: yes
Using memory checker: no
Function stack tracing: no
Strict file format checks: no
Optimization instrumentation: no
And the output of nc-config --all:
--cc -> /media/case/work/hwrf/libraries/mpich/bin/mpicc
--cflags -> -I/media/case/work/hwrf/libraries/netcdf/include -I/media/case/work/hwrf/libraries/hdf5/include -I/media/case/work/hwrf/libraries/szip/include -I/media/case/work/hwrf/libraries/grib2/include
--libs -> -L/media/case/work/hwrf/libraries/netcdf/lib -lnetcdf
--static -> -lhdf5_hl -lhdf5 -lm -ldl -lz -lcurl
--has-c++ -> no
--cxx ->
--has-c++4 -> no
--cxx4 ->
--has-fortran -> yes
--fc -> mpifort
--fflags -> -I/media/case/work/hwrf/libraries/netcdf/include -I/media/case/work/hwrf/libraries/netcdf/include
--flibs -> -L/media/case/work/hwrf/libraries/netcdf/lib -lnetcdff
--has-f90 ->
--has-f03 -> yes
--has-dap -> no
--has-dap2 -> no
--has-dap4 -> no
--has-nc2 -> yes
--has-nc4 -> yes
--has-hdf5 -> yes
--has-hdf4 -> no
--has-logging -> no
--has-pnetcdf -> no
--has-szlib -> yes
--has-cdf5 -> yes
--has-parallel4 -> yes
--has-parallel -> yes
--has-nczarr -> no
--prefix -> /media/case/work/hwrf/libraries/netcdf
--includedir -> /media/case/work/hwrf/libraries/netcdf/include
--libdir -> /media/case/work/hwrf/libraries/netcdf/lib
--version -> netCDF 4.8.0
So far, so good. Everything was installed without any failure. All make checks have been passed.
The trouble arises when I try to compile PnetCDF from source. I tried PnetCDF version 1.11.2 for installation (in fact, I triedd several older versions as well, more on that later). The installation is attempted as following:
./configure --prefix=${DIR}/pnetcdf --enable-large-single-req --enable-netcdf4 --enable-shared=yes --with-netcdf4="${DIR}/netcdf" MPICC=mpicc MPIF77=mpif77 MPIF90=mpif90 MPICC=mpicc MPIF77=mpif77 MPIF90=mpif90
make
make tests
make check
make ptest
make ptests
But, it fails at make ptest. The failed test is tst_rec_vars.c and the error message is:
Error at line 78 of tst_rec_vars.c: expect dim X len 4 but got 3
Error at line 78 of tst_rec_vars.c: expect dim X len 4 but got 3 fail with 2 mismatches
make[2]: *** [Makefile:1354: ptest4] Error 1
make[2]: Leaving directory '/media/case/work/hwrf/libraries/source/pnetcdf-1.12.2/test/nc4'
make[1]: *** [Makefile:780: ptests] Error 1
make[1]: Leaving directory '/media/case/work/hwrf/libraries/source/pnetcdf-1.12.2/test'
make: *** [Makefile:965: ptests] Error 1
Heres the test that is failing (GitHub link): tst_rec_vars.c
It seems that the test is checking a NetCDF file for the lenth of dimension in it. I assume the NetCDF file it is checking is tst_rec_vars.nc. The output of ncdump -h on this file is given below:
netcdf tst_rec_vars {
dimensions:
X = UNLIMITED ; // (4 currently)
variables:
int U(X) ;
int V(X) ;
}
That is, the NetCDF file's X dimension indeed has a length 4, although the test program beleives it is only 3. But, if I were to run the test manually as ./tst_rec_vars, it runs correctly.
*** TESTING C tst_rec_vars for record variables to NetCDF4 file ------ pass
Could it be due to parallel processing/threading? The test in question is run under a section titled test/nc4: Parallel testing on 4 MPI processes. I assume the NetCDF file was tested before it was formed completely, which could explain why I got a length 1 X dimension in the NetCDF file from ncdump at one point.
>>>ncdump -h tst_rec_vars.nc
netcdf tst_rec_vars {
dimensions:
X = UNLIMITED ; // (1 currently)
variables:
int U(X) ;
int V(X) ;
}
What exactly is going on here? How do I fix it?
Edit:
I'm trying to install it on CentOS 7, since I was encountering the above mentioned issue, I tried to install it on a Debian 11 system. But I am getting the very exact error in both systems.

shake - rule finished running but did not produce file:

I try to use shake to convert some markdonw files to html ("bake"). The markdown files are in a directory "dough" and the html should go to "baked". The goal is to produce the index.html file, which links the other files.
This is my first use of shake!
The conversion works, but at the end the first rule produces the error
`rule finished running but did not produce file:`
The cause is perhaps that the index.html file is produced before (with the second rule). How can I tell the first rule not to expect a result (or force the production again)?
secondary question: how to change the first rule to collect files with extension "md" and "markdown"?
Thank you for the help! Suggestions for improvements are most welcome!
bakedD = "site/baked" -- toFilePath bakedPath
doughD = "site/dough"
shakeWrapped :: IO ()
shakeWrapped = shakeArgs shakeOptions {shakeFiles=bakedD
, shakeVerbosity=Loud
, shakeLint=Just LintBasic
} $
do
want ["index"<.>"html"]
"index"<.>"html" %> \out ->
do
mds <- getDirectoryFiles doughD ["//*.md"]
let htmlFiles = [bakedD </> md -<.> "html" | md <- mds]
need htmlFiles
liftIO $ bakeOneFileIO "baked/index.html"
(bakedD <> "//*.html") %> \out ->
do
let c = dropDirectory1 $ out -<.> "md"
liftIO $ bakeOneFileIO c
The error message notes that you declare the file to produce index.html, but it doesn't produce that file. From a read of your build system, it appears it produces based/index.html? If so, change the want line area to read:
do
want ["baked/index.html"]
"baked/index.html" %> \out ->
Now you are saying at the end of the execution you want to produce a file baked/index.html, and that here is a rule that produces baked/index.html. (If it's really producing site/baked/index.html then adjust appropriately.)
Addressing your second question, mds <- getDirectoryFiles doughD ["//*.md","//*.markdown"] will detect both extensions.
As for style tips, using "index" <.> "html" is not really helping - "index.html" is identical but clearer to read. Other than that, it seems pretty idiomatic.
The issue was that the first rule wants a file, but this file is included (and produced) by the second rule. There is an indication for that problematic case that the \out variable is not used and the production of the index.htm is not required in this rule (as it is included in the second rule). One can take this as an indication that a phony rule would be appropriate and simplify the code:
bakedD = "site/baked" -- toFilePath bakedPath
doughD = "site/dough"
shakeWrapped :: IO ()
shakeWrapped = shakeArgs shakeOptions {shakeFiles=bakedD
, shakeVerbosity=Loud
, shakeLint=Just LintBasic
} $
do
want ["allMarkdownConversion"]
phony "allMarkdownConversion" $
do
mds <- getDirectoryFiles doughD ["//*.md"] -- markdown ext ??
let htmlFiles = [bakedD </> md -<.> "html" | md <- mds]
-- liftIO $ putIOwords ["shakeWrapped - htmlFile", showT htmlFiles]
need htmlFiles
(bakedD <> "//*.html") %> \out ->
do
let c = dropDirectory1 $ out -<.> "md"
liftIO $ bakeOneFileIO c
I think that shake is a very convenient method to add a cache to a static site generator; it rebuilds only what is required!

How to compile a basic c file in yocto

I am working on yocto, I want to compile some C files in yocto and install the resulting binary to external filesystem.
Before doing that I tried creating a separate reciepe and compile c code from it.
I am unable to compile it.
I am not sure to understand the question since it is not precise enough.
Including C files in recipe tree
If you want to have the C files in your recipe, having a file tree like this:
recipe-example/example/example_0.1.bb
recipe-example/example/example-0.1/helloworld.c
You can generate this example when you create a new layer using
yocto-layer <your-layer-name>
Your bb file will look like this:
#
# This file was derived from the 'Hello World!' example recipe in the
# Yocto Project Development Manual.
#
SUMMARY = "Simple helloworld application"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
SRC_URI = "file://helloworld.c"
S = "${WORKDIR}"
do_compile() {
${CC} helloworld.c -o helloworld
}
do_install() {
install -d ${D}${bindir}
install -m 0755 helloworld ${D}${bindir}
}
It will compile the hello world file and install it into /usr/bin on your image.
From a Git repo
You also can compile from a git repository, I advise you to read the manual and examples in your yocto folder. Here is an example here of wiringPi:
DESCRIPTION = "A library to control Raspberry Pi GPIO channels"
HOMEPAGE = "https://projects.drogon.net/raspberry-pi/wiringpi/"
SECTION = "devel/libs"
LICENSE = "LGPLv3+"
LIC_FILES_CHKSUM = "file://COPYING.LESSER;md5=e6a600fd5e1d9cbde2d983680233ad02"
# tag 2.29
SRCREV = "d79506694d7ba1c3da865d095238289d6175057d"
S = "${WORKDIR}/git"
SRC_URI = "git://git.drogon.net/wiringPi \
file://0001-Add-initial-cross-compile-support.patch \
file://0001-include-asm-ioctl.h-directly-for-_IOC_SIZEBITS.patch \
"
COMPATIBLE_MACHINE = "raspberrypi"
CFLAGS_prepend = "-I${S}/wiringPi -I${S}/devLib"
EXTRA_OEMAKE += "'INCLUDE_DIR=${D}${includedir}' 'LIB_DIR=${D}${libdir}'"
EXTRA_OEMAKE += "'DESTDIR=${D}/usr' 'PREFIX=""'"
do_compile() {
oe_runmake -C devLib
oe_runmake -C wiringPi
oe_runmake -C gpio 'LDFLAGS=${LDFLAGS} -L${S}/wiringPi -L${S}/devLib'
}
do_install() {
oe_runmake -C devLib install
oe_runmake -C wiringPi install
oe_runmake -C gpio install
}
It is fetching from a git repository, applying patches generated by git, using oe_runmake to compile with the makefiles.
With devtool
It has been asked in a comment on how to add a recipe with devtool.
We will still use wiringPi as an example again. Download it doing
https://github.com/WiringPi/WiringPi
The Makefile is is the folder wiringPi.
You can then do
devtool add <name_of_recipe> <path_to_Makefile_folder>
Take care of the warning from devtool
NOTE: Creating workspace layer in /home/dbensoussan/new_poky/poky/build/workspace
NOTE: Enabling workspace layer in bblayers.conf
NOTE: Using source tree as build directory since that would be the default for this recipe
NOTE: Recipe /home/dbensoussan/new_poky/poky/build/workspace/recipes/project/project.bb has been automatically created; further editing may be required to make it fully functional
This is generating the recipe as follow:
# Recipe created by recipetool
# This is the basis of a recipe and may need further editing in order to be fully functional.
# (Feel free to remove these comments when editing.)
#
# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
# your responsibility to verify that the values are complete and correct.
LICENSE = "Unknown"
LIC_FILES_CHKSUM = "file://COPYING.LESSER;md5=e6a600fd5e1d9cbde2d983680233ad02"
# No information for SRC_URI yet (only an external source tree was specified)
SRC_URI = ""
# NOTE: this is a Makefile-only piece of software, so we cannot generate much of the
# recipe automatically - you will need to examine the Makefile yourself and ensure
# that the appropriate arguments are passed in.
do_configure () {
# Specify any needed configure commands here
:
}
do_compile () {
# You will almost certainly need to add additional arguments here
oe_runmake
}
do_install () {
# This is a guess; additional arguments may be required
oe_runmake install 'DESTDIR=${D}'
}
You can then edit your recipe to suit your configuration
With externalsrc
It is possible to use a directory present on the filesystem by using externalsrc.
I did not try it myself, nor have I the workspace ready to do, but #71GA in the comment tested the tutorial from the Koan software company https://wiki.koansoftware.com/index.php/Building_Software_from_an_External_Source and it worked. I will copy the content here:
in this case use the externalsrc class - you can inherit this in the original bb recipe or a bbappend:
inherit externalsrc
EXTERNALSRC = "/path/to/sources"
Depending on the type of build (eg, 'inherit module' for out of tree Linux kernel modules) you may or may not need to set EXTERNALSRC_BUILD.
inherit externalsrc
EXTERNALSRC = "/some/path"
EXTERNALSRC_BUILD = "/some/path"
If you're going to use it across a number of recipes you can inherit it globally at the configuration level (perhaps via an inc file that you include/require there):
INHERIT += "externalsrc"
EXTERNALSRC_pn-<recipename> = "/path/to/sources"
Recipe example using an external source for nInvaders package
#
# Recipe example with externalsrc
#
# (C)2019 Marco Cavallini - KOAN - <https://koansoftware.com>
#
LICENSE = "CLOSED"
LIC_FILES_CHKSUM = ""
inherit externalsrc
EXTERNALSRC = "/home/koan/yocto-qemuarm-sumo/ninvaders-0.1.1"
EXTERNALSRC_BUILD = "${EXTERNALSRC}"
DEPENDS = "ncurses"
EXTRA_OEMAKE = "-e"
do_install() {
install -d ${D}${bindir}
install -m 0755 nInvaders ${D}${bindir}
}
FILES_${PN} = "${bindir}/*"
You can just go to the official documentation to find your answer.
In the chapter 7.3 Writting a New Recipe of the yocto mega-manual, the very first example is exactly is what you need (Single .c File Package (Hello World!)).

SBT - Invoke GCC call failing in SBT, but not when I manually execute it

I'm working on a library that is going to talk to the I2C bus on my Raspberry PI from Scala. For this I need a little bit of JNI code that is going to interrupt with the OS on the device.
I tried to make a build file for this, which right now, looks like this:
name := "core"
organization := "nl.fizzylogic.reactivepi"
scalaVersion := "2.11.6"
val nativeClasses = List(
"nl.fizzylogic.reactivepi.i2c.I2CDevice"
)
val nativeDeviceSources = List(
"src/jni/nl_fizzylogic_reativepi_i2c_I2CDevice.c"
)
val nativeGenerateHeaders = taskKey[Int]("Generates JNI headers for the library")
val nativeCompile = taskKey[Int]("Compiles the native library")
nativeGenerateHeaders := {
("javah -classpath target/scala-2.11/classes -d src/jni " + nativeClasses.mkString(" ")) !
}
nativeCompile := {
("gcc -Wl -add-stdcall-alias -I$JAVA_HOME/include -I$JAVA_HOME/include/darwin -I$JAVA_HOME/include/linux -shared -o target/reactivepi.so " + nativeDeviceSources.mkString(" ")) !
}
So far the javah call is successful. But when I invoke the nativeCompile task, GCC tells me it can't find the jni.h file. However when I copy and paste the command from the build file to my terminal and execute it, it succeeds.
It looks like it is not picking up the include paths when I am executing gcc from my custom build task. But I have no idea what I'm doing wrong here.

Could not find configuration file /etc/startup.mk

I want to develop a simple C application for learning purposes using Eclipse (Juno) CDT.
For that, I created the sample project provided in the IDE that creates a simple Hello World executable.
But when I want to build the project, the following errors comes up:
make: all
make: Error -- Could not find configuration file /etc/startup.mk
What does this mean? I can't find a file named like this anywhere on my system. Curiously enough, it works on my OS X with the same setup perfectly. Just not on my Windows machine.
What is going wrong here?
Edit: make -V show the following output:
D:\>make -V
make - Version 7.0 build 1182
Built-in Rules (cannot be changed):
OS:=NT
.IMPORT .IGNORE : ROOTDIR
.MAKEFILES:makefile
.SOURCE : .NULL
#B = $(#:b)
#D = $(#:d)
#F = $(#:f)
%B = $(%:b)
%D = $(%:d)
%F = $(%:f)
*B = $(*:b)
*D = $(*:d)
*F = $(*:f)
<B = $(<:b)
<D = $(<:d)
<F = $(<:f)
?B = $(?:b)
?F = $(?:f)
?D = $(?:d)
System Configuration:
SWITCHAR = /
OSVERSION = 01
OSRELEASE = 5
DIRSEPSTR = /\:
SHELL = /mksnt/sh.exe
SHELLFLAGS = /c
SHELLMETAS =
GROUPSHELL = /mksnt/sh.exe
GROUPFLAGS =
GROUPSUFFIX =
MAKEDIR = D:/
PWD = D:/
The make executable that is in your path is not the GNU make that eclipse expects and needs. It is probably a tool that comes with some other product -- make is a pretty common name. If your windows is not too old, you could try where make to locate the culprit and remove it from your path. Once you have done that, eclipse will be able to the GNU make by itself if you used the standard installation of your toolchain, or you could add it to your path. Alternatively, you can customize the eclipse build settings and explicitly point it to the make you want to use.
The make you want is probably in C:\MinGW\msys\1.0\bin, but could be somewhere else depending on how you installed the toolchain. There is also a mingw32-make in C:\MinGW\bin -- all assuming that you are using MinGW

Resources