Bash parameter expansion and array index operations - arrays

The ${parameter,,pattern} parameter expansion converts alphabetic characters in parameter to lower-case.
On cygwin 1.7.11-1 Bash 4.1.10(4) and also on my debian squeeze Bash 4.1.5(1);
if i run the following, i get a curious result:
$ declare -a a=(Zero One Two Three); n=0; echo "${a[n],,}->${n}"; echo "${a[++n]}->${n}"; echo "${a[++n],,}->${n}"
zero->0
One->1
three->3
$
NB: similar results happen:
for ,,* or ^^ case conversion;
for some other expansions such as ${parameter##word};
for using either prefix/postfix ++ or -- operator;
for using $((++n)) instead of just ++n.
However, the length expansion ${#parameter} works as i might expect:
in the above snippet, echo "${#a[++n]}->${n}" instead of echo "${a[++n],,}->${n} would yield 3->2 instead of three->3 ~& the length of a[2]="two" is indeed 3 characters.
I imagine that the parameter expansion is happening twice. But why is this happening?

I grabbed the latest bash source & it appears that this issue is resolved for this version:
$ ./bash --version
GNU bash, version 4.2.24(1)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
$
Note however that on my Debian Squeeze stable, I have:
$ apt-cache policy bash
bash:
Installed: 4.1-3
Candidate: 4.1-3
Version table:
*** 4.1-3 0
500 http://ftp.uk.debian.org/debian/ squeeze/main amd64 Packages
100 /var/lib/dpkg/status
$
$ bash --version
GNU bash, version 4.1.5(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
I think this means that currently on Debian Squeeze, the latest stable version of bash is 4.1.5(1) & contains this bug. Also on Cygwin 1.7.11-1 currently the latest available bash package appears to be version 4.1.10(4) & contains this bug. Of course the issue is easy to work around, so no real need to build the 4.2 sources.

I got
Zero->0
One->1
Two->2
However I think "${a[++n]}->${n}" may not be well defined.
try substitute with
let n=$n+1 ; echo "${a[n],,}->${n}";
To inject the notion of sequence.

Related

How to use Intel Advisor to profile my parallel MPI application?

I am working on a remote Linux server where I have my application running in parallel with MPI. I want to profile it and test how good is the load balance in each MPI process and which are the heaviest parts of the code.
To run my application in parallel I usually run it like this:
mpirun -n # ${location}/myApp arg1 arg2 etc.
In the machine there is a module about Intel Advisor which I am going to use. The GUI command
advixe-gui does not work so I have to do it with advixe-cl
In case is helpful, when I type:
advixe-cl
it returns me this:
Intel(R) Advisor Command Line Tool
Copyright (C) 2009-2019 Intel Corporation. All rights reserved.
Usage: advixe-cl <--action> [--action-option] [--global-option] [[--] <target> [target options]]
Use --help for details.
Any idea about how to proceed further with profiling?
You have to use Advisor's command line (advixe-cl) and you have to "wrap" your advixe-cl command line by mpirun. And you can copy and view obtained profiles with GUI afterwards - with individual "result view" for each rank profiled.
You can "wrap" the command line in few ways, for example (Intel MPI specific):
$ mpirun -n 1 -gtool "advixe-cl -collect survey -no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples
or (generic MPI with SLURM):
$ srun –n 1 –c 32 advixe-cl --collect=survey --project-dir=./adv -- ./miniFE.x
This topic is described in many details (including selective rank analysis or e.g Cray or Intel MPI specifics) in following Intel "Cookbooks" and articles:
Intel MPI-specific : Analyzing Intel MPI applications with Intel Advisor
Generic MPI, SLURM, for famous WRF workfload: Analyze Vectorization and Memory Aspects of an MPI Application "cookbook"
Advisor for MPI apps on Cray system: Analyze Performance on Cray Systems "cookbook"
Advisor Documentation chapter
Yet another article
You need to provide an action in the command line - it is not optional according to the syntax:
$ advixe-cl <--action> [--action-options] [--global-options] [[--] target [target options]]
Where action would be to either collect or report. And each command has exactly one action. For example, you cannot use both the collect and report actions in the same command.
You can review the User Guide for Advisor here.

Discrepancy in behavior of Linux loaders (ld-linux-x86-64) between Glibc 2.12 and Glibc 2.17

I'm trying to compile the same lib on two x86 separate machines.
Both use the same toolchain (exactly same set of files) but have different Glibc versions.
When I run command LD_DEBUG=libs /lib64/ld-linux-x86-64.so.2 --list ./libl2ps.so I notice the following discrepancy between the 2 Linux loaders:
Machine 1 (with Glibc 2.12):
19943: find library=libm.so.6 [0]; searching
19943: search path=/ebs/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64:...:/ebs/frperies/repo/gnb/uplane/build_bbp/l2_ps/build/. (RPATH from file ./libl2ps.so)
19943: trying file=/ebs/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64/libm.so.6
19943:
19943: find library=libgcc_s.so.1 [0]; searching
...
In this case the Linux loader selects lib libm.so.6 from the toolchain path based on RPATH of lib libl2ps.so.
Machine 2 (with Glibc 2.17):
10699: find library=libm.so.6 [0]; searching
10699: search path=/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64:/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/lib64:/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib:/home/frperies/repo/gnb/uplane/build_bbp/l2_ps/build/. (RPATH from file ./libl2ps.so)
10699: trying file=/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64/libm.so.6
10699: trying file=/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/lib64/libm.so.6
10699: trying file=/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib/libm.so.6
10699: trying file=/home/frperies/repo/gnb/uplane/build_bbp/l2_ps/build/./libm.so.6
10699: search cache=/etc/ld.so.cache
10699: trying file=/lib64/libm.so.6
As for Machine 1, the loader attempts from RPATH of libl2ps.so to select lib libm.so.6 from toolchain path but skip it for some reason and try further other paths. Finally it selects libm.so.6from the system path /lib64/.
The RPATH of the 2 libs lib2ps.so are exactly the same. The two files libm.so.6 are also exactly the same on both machines (checked with md5sum).
I don't understand this differences of behavior between the 2 Linux loaders.
Do you see any reason what would explain this discrepancy ?
Thank you very much for your answers.
Update:
Thank you yugr for your answer.
Output of readelf -h gives only differences on fields "Entry point address" and "Start of section headers" and there is no other differences so I think it will not help.
Regarding using dlopen()/dlerror(), I've done a little executable with the following statement:
dlopen("/home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64/libm-2.28.so", RTLD_LAZY);
On machine 1 it works as expected:
C++ dlopen demo
Opening libm-2.28.so...
Closing library...
On machine 2 it fails and dlerror() gives the following output:
Cannot open library: /home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64/libm-2.28.so: cannot open shared object file: No such file or directory
but the file libm-2-28.so really exists on my file system:
$ ls -l /home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic-linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64/libm-2.28.so
-rwxr-xr-x 1 frperies linseeusers_lte_espoo 1682944 Oct 5 13:50 /home/frperies/repo/gnb/uplane/build/prefix-root/asik-x86_64-ps_lfs-dynamic- linker-on/toolchain/sysroots/core2-64-pc-linux-gnu/usr/lib64/libm-2.28.so
This is very weird, what could lead to this situation ???
Thanks
Update 2:
That is true that I haven't pointed out that machine 1 is a RHEL6.8 distro while machine 2 is RHEL7.4 distro. I (naively?) didn't think this really matters...
On machine 1:
$ cat /proc/sys/kernel/osrelease
4.4.115-1.NSN.el6.x86_64
$ uname -a
Linux sq24-3 4.4.115-1.NSN.el6.x86_64 #1 SMP Mon Feb 12 12:35:46 CET 2018 x86_64 x86_64 x86_64 GNU/Linux
$ readelf -n libl2ps.so
Notes at offset 0x00000270 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: b598468830fdf2f61eda25553b9a367c4d28cdc9
On machine 2:
$ cat /proc/sys/kernel/osrelease
3.10.0-693.el7.x86_64
$ uname -a
Linux localhost.localdomain 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
$ readelf -n libl2ps.so
Displaying notes found at file offset 0x00000270 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 5829181bc0502233748149369108915ea7b10e8f
Does it help ?
Thanks
Update 3:
$ readelf -n libm.so.6
Notes at offset 0x00000238 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 0d84c7247dd76008c096719043e5592735a1c4bd
Notes at offset 0x0000025c with length 0x00000020:
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 4.4.0
So, how to interpret this ABI version number set to 4.4.0 ??
Thanks
Thank you yugr and Employed Russian for your answers!!
I will give it a try by upgrading my Kernel version on Machine 2.
Thanks
Regards
The error message that you see is the infamously confusing ENOENT errno. I see two instances of it in dl-load.c:
checking OS compatibility
loading non-setuid to setuid process
I suspect the first one fails in your case which would mean that OS kernel is incompatible between two machines. ld.so manpage indeed says that
Each shared object can inform the dynamic linker of the
minimum kernel ABI version that it requires. (This
requirement is encoded in an ELF note section that is viewable
via readelf -n as a section labeled NT_GNU_ABI_TAG.) At run
time, the dynamic linker determines the ABI version of the
running kernel and will reject loading shared objects that
specify minimum ABI versions that exceed that ABI version.
NT_GNU_ABI_TAG is 4.4.0 which means that you run a program expecting a minimum 4.4 kernel on a 3.10 kernel. Theoretically newer Glibc should run on older kernels as well but in your case Glibc was probly built with explicit --enable-kernel flag which prevents it's usage on kernels before 4.4 (see e.g. this explanation of --enable-kernel).
As a workaround, you may try to fool Glibc by overriding kernel version on machine 2 via
export LD_ASSUME_KERNEL=4.4.0
but it may not work if libm makes 4.4-specific syscalls that are not really present on 3.10.

Verification of poly1305-donna-16.h code

The RFC from May 2015 Y. Nir et al, ChaCha20 and Poly1305 for IETF Protocols
(https://www.rfc-editor.org/rfc/rfc7539) contains a reference to the MIT/Public domain C library https://github.com/floodyberry/poly1305-donna.
I am just porting the C code to Pascal. The 8 bit code works OK (self-test, example, and RFC test vectors).
The port from poly1305-donna-16.h using 16->32 bit multiples and 32 bit additions failed. After some testing I compiled the original source with DJGPP GCC 4.7.3, MS VC 6.0 and BC 3.1 and all three failed too (poly1305 self-test).
Questions: Does this C version (build with -DPOLY1305_16BIT) fail for other compilers too? Is there a known fix available? (The blog of the author Andrew Moon at https://floodyberry.wordpress.com/ is inactive since 6 years)
I can confirm a build failure on a pretty vanilla Fedora 22 system:
% gcc poly1305-donna.c -c -DPOLY1305_16BIT
% gcc example-poly1305.c -o ex poly1305-donna.o -DPOLY1305_16BIT
% ./ex
poly1305 self test: failed
Notice that the test succeeds when I omit -DPOLY1305_16BIT.
Also notice:
% uname -rmp
4.0.8-300.fc22.x86_64 x86_64 x86_64
% gcc --version
gcc (GCC) 5.1.1 20150618 (Red Hat 5.1.1-4)
I suggest you submit a bug report. Andrew has been responsive in the past.
EDIT:
Compiling with clang version 3.5.0 yields the same results as the above gcc test.

C runtime and date command on cygwin

Background
$ uname -a
CYGWIN_NT-6.1 Owner-PC 1.7.34(0.285/5/3) 2015-02-04 12:14 x86_64 Cygwin
PS: just updated Cygwin 15 days ago. Should be razor close to current.
$ date --version
date (GNU coreutils) 8.23
Packaged by Cygwin (8.23-4)
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by David MacKenzie.
Extracted from man date page on cygwin:
%c locale's date and time (e.g., Thu Mar 3 23:05:25 2005)
output from C runtime (dtime is tiny piece of code):
$ ./dtime -f "%c"
Tue Feb 5 17:04:45 2030
Looks good to me...
To avert questions like: what does the code look like:
strcpy(p->fmt, "%c");
....
strftime(tmp, 80, p->fmt, localtime(&lt));
printf("%s\n", tmp);
dtime and other time routines I wrote in the C library behave as expected per standards. No problem.
On the other hand the date command is not correct AFAICT:
Owner#Owner-PC ~
$ date
Sun, Feb 22, 2015 11:41:44 AM
Owner#Owner-PC ~
$ date +%c
Sun, Feb 22, 2015 11:41:55 AM
The second line of output does not match the man page - or standards, I believe.
Question:
A bug? Something else I am missing?
Edit: per suggestion quote the format string. No. That is only need if there are white spaces in the format
Owner#Owner-PC ~
$ date '+%c'
Sun, Feb 22, 2015 3:13:51 PM
This seems related to the locale settings. Your C program uses the C locale by default, date will behave according to the environment variables LANG, LC_TIME and LC_ALL. What are the values of these variables ? Can you test this:
LC_ALL="C" date '+%c'

Function name is not displaying in GDB

I have the core dump file. I kept the c binary application(Built with Debug ON option) in the GDB path which caused the core dump along with the core file.
But when I do GDB, I am getting the stack information as below:
sura#hyde:/abc/def/ghi/appln [css_absmin] gdb application_name core_1206
HP gdb 3.0 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 3.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
warning: exec file is newer than core file.
Core was generated by 'application_name'.
Program terminated with signal 11, Segmentation fault.
#0 0xc0199640 in ?? ()
(gdb) where
#0 0xc0199640 in ?? ()
(gdb) bt
#0 0xc0199640 in ?? ()
(gdb)
The C Binary is compiled in PA-RISC (unix).
How can I get the offending function name?
Sundar,
You posted a question about gdb but HP-UX gdb is not a GNU gdb, but rather gdb customized by HP. So it is called Hewlett-Packard Wildebeest (based on GDB) - wdb. Your problem is that you have extremely old version of wdb. Right now HP offers gdb 6.5 (http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=29080f1bace021100f1bace02110275d6e10RCRD) for HP 9000 systems running HP-UX 11i v1, 11i v2, or 11i v3 and your version is 3.0!
You seem to have HP-UX 11.00, so you can ask on HP-UX forum what is the latest wdb for HP 11.00. Ask here: http://h30499.www3.hp.com/t5/Languages-and-Scripting/bd-p/itrc-150#. Also after installing the latest wdb for your PA-RISC try to get backtrace again with it. I hope you will get a correct backtrace with the latest wdb.

Resources