I need to track compiler flags used as part of a rule, e.g when supplied as arguments to a function. Does Shake have a way to track such flags as inputs in the same vein as need? As a trivial example, I would like for Shake to rebuild all .o files when the rule changes to pass -O2 to the C compiler instead of -O0.
There are two approaches to tracking things like whether to use optimisation or not.
1) Use Oracles
Oracles very closely match what you are asking for. To track something like
-O0 vs -O2 you'd need an oracle that tracks optimisation level:
newtype OptLevel = OptLevel ()
deriving (Show,Typeable,Eq,Hashable,Binary,NFData)
type instance RuleResult OptLevel = String
rules = do
addOracle $ \(OptLevel _) -> return $
if <whatever you use to decide> then "-O0" else "-O2"
"foo.o" %> \_ -> do
level <- askOracle $ OptLevel ()
cmd "gcc" level ...
Now the optimisation level is a tracked dependency which will update if anything changes. This example is based on the docs for addOracle.
2) Use different output files
For compiler flags another approach is to use different build directories, i.e. so build/opt (and build/opt/obj etc) has binaries and .o files built with -O, build/debug without, build/profile with profiling flags, and build/test with test flags. Some others, like build/doc and build/hsc, have generated files that aren't dependent on compiler flags.
The advantage of this approach is that you can keep all the files cached at once, and refreshing the debug or test build doesn't destroy the opt one.
The disadvantage is that it's just for a set of hardcoded
flags. But it's also not hard to add a new mode, you just need a new
(directory, flags) pair for it.
Related
Is there a built-in way to pass command-line arguments to a "shakefile"? I'd like to pass --env production|development|staging and then use it within my rules to (slightly) alter the build-steps for each environment.
There are two halves to this problem - first getting the flags into Shake, and secondly, using them to influence the behaviour.
You can get arguments into Shake using any Haskell command line parser, but Shake ships with support for one built it, which can often be easier:
data Flags = Production | Dev | Staging deriving Eq
flags = [Option "" ["production"] (NoArg $ Right Production) "Build production."
,Option "" ["dev"] (NoArg $ Right Dev) "Build dev."
,Option "" ["staging"] (NoArg $ Right Staging) "Build staging."
]
main = shakeArgsWith shakeOptionsn flags $ \flags targets -> do
want targets
... do whatever you want with the flags ...
return rules
For using the flags to influence, you might want to:
Completely isolate build outputs from each flag, in which case changing the directory and setting shakeFiles differently in each case makes each one fully distinct.
Use the flags to change the output paths, so you always have rules dev/main.js and prod/main.js, and then you consult the flags when doing want to pick up the right rules.
Put the flags into an Oracle and have them as tracked settings, so when you flip from prod to dev some things rebuild.
If the builds are 80%+ distinct I'd go for 1. If you change flags very rarely 3 can work. Otherwise, I tend to opt for 2. But they all work, so picking the simplest to start with is also not unreasonable.
I am trying to add warning as error flag in my makefiles. But I am getting the following problem.
When I am compiling without adding the flag it is successful. But when I am adding Werror flag in some ".mk" files, compilation is failing with some error. But in the successful build log warning was not there for that source file(".c") which is throwing error now(Werror).
I am adding he following flags.
UN_CDEFS := -Wno-error=%
CDEFS := -Wall -Werror -Wextra
SUB_CDEFS := -Wall -Werror -Wextra
So please suggest what might be the problem.
Caveat: This isn't a complete answer because we need more information, but it would become [too] lengthy for more top comments like the ones I've already posted.
As you refine the problem and/or post more data, I can edit this answer accordingly. At a minimum, posting your actual makefiles might help, as well as, the actual final cc commands and the compiler warning/error output for the failing .c file [There may be multiple ones, but the single/first one should be sufficient].
Below are some detailed instructions on how to debug this, based on my own experience with such issues.
But, before I get to that, I'll hazard a guess. I notice that you're doing:
CDEFS := -Wall -Werror
[leaving off the -Wextra as you mentioned in a comment].
If this is done as [nearly] the first thing in the makefile, it's fine. However, if it occurs in the middle, you are replacing CDEFS with your own value. If a prior line in the makefile did (e.g.):
CDEFS = -Dwont_build_cleanly_without_this_option
then, when you add your line, that could be the issue, because this gets [effectively] removed. You might try this instead:
CDEFS += -Wall -Werror
This just appends to the existing symbol, so any prior value will be retained.
Also, the base makefile might have something like:
ifndef CDEFS
CDEFS := -Dwont_build_cleanly_without_this_option
endif
Normally, make will output the full text of commands it executes to create targets. For compilation, this is (e.g.) cc -c foo.c.
Some fancier builds wrap the command in (e.g.) #doit cc -c foo.c where doit prints a message like compiling foo.c ... and only outputs the full command if there is an error. (e.g. the linux kernel build does this, IIRC). I'm assuming you don't have this, but if you do, there is usually a command line override such as make VERBOSE=1
So, there is some .c file somewhere that builds cleanly with the normal options but generates an error when extra compile options are added. Let's call this file badnews.c
What we want to see is the compilation command that make printed for badnews.c and the warning/error output for two cases:
without the extra options
with the extra options in various combinations
In particular, examining the case (1) command against the case (2) commands might show that options other than the -W are different. This indicates a makefile issue, similar to my "guess" above. You've said that [your equivalent of] case (1) is clean with no warnings, but, given the trouble you're having, it wouldn't hurt to double check.
You can cut and paste the case (1) cc command into a shell script and manually add the -W options. Watch out for things with spaces, such as -DSTRING="foo bar" in the makefile that may need extra quotes in a shell script.
To alleviate conflicts similar to yours, in my own makefiles I separate the symbols.
DFLAGS for all -DFOO=1
COPTS for -g, -O2, -Wall, -fno-inline-functions, etc.
Then, I either do:
CFLAGS := $(COPTS) $(DFLAGS)
Or:
%.o: %.c:
cc -c $(COPTS) $(DFLAGS) $<
There are other ways to do this as well.
UPDATE:
I am using following command to build: emq PRODUCT=ASG >build_log_0508.log
I'm unfamiliar with emq. I can't find a reference to it, except as "enterprise mail queue for JIRA", which [AFAICT] may be part of cPanel?
Getting the following error on compilation: prod/libs/app/app.c:720:5: error: incompatible implicit declaration of built-in function 'free' [-Werror] free(tmp_dn);
This is the smoking gun ...
I don't know what compiler you're using, or what OS/environment, but it appears to not flag this as a warning/error by default.
However, it is a bug in the source app.c that needs to be fixed. It was correctly flagged as a warning/error by the addition of -Wall and -Werror
Note: As I mentioned in my original answer, it would be helpful to have the final cc command line that produced this error [as well as the cc command when this file is not flagged].
I created a simple test case:
void
myfree(void *ptr)
{
free(ptr);
}
Here, under gcc, I did gcc -c test.c and I get:
test.c: In function 'myfree':
test.c:5:2: warning: implicit declaration of function 'free' [-Wimplicit-function-declaration]
free(ptr);
^
test.c:5:2: warning: incompatible implicit declaration of built-in function 'free'
test.c:5:2: note: include '<stdlib.h>' or provide a declaration of 'free'
So, gcc flags this by default [even without -Wall or -Werror]. But, your compiler does not unless it is given -Wall. This could occur if your compiler were clang and you also specified -std=c89
As I implied earlier, if you just specify -Wall but not -Werror, you should get the same warnings but they just won't stop the build. In a large build, they can be easily overlooked in the log [by a human (e.g.) me :-)].
Referring to the suggestions in my original answer, assuming that the cc commands between case (1) ["good"] and case (2) ["bad"] only differed by the addition of -Wall, the correct way to fix this is to edit app.c and add #include <stdlib.h> as part of the includes.
Is there any problem with "SUB_CDEFS := -Wall -Werror"?
It will have similar problems/benefits as with CDEFS.
I am adding at the end of the makefiles
This is all the more reason to use += instead of :=. You might be "killing off" the -std=c89 if that were specified somewhere.
UPDATE #2:
It worked after doing += instead of :=.
As I mentioned, using := removed some critical compile options, that were specified elsewhere in the makefile(s).
But, once again, the source code has a bug and is broken. It was broken before you ever touched it. By adding -Wall -Werror using :=, you uncovered this bug, that previously was masked incorrectly. This is a good thing.
Using += just sweeps the bug under the rug [again], by restoring some build options that were lost with :=. But, these "lost" build options were wrong. They allowed a genuine flaw in the C code to escape detection.
This is not about getting the build to work [with a workaround], but to fix the root cause of the build problems, which is to modify the C source code. There are probably other such C source code bugs and some may be more severe.
With the workaround to "fix" the build, you've now got a piece of built software that can not be trusted to run correctly. It could fail in intermittent ways on your system(s). Or, produce incorrect results. Or, allow your system to be hacked [and potentially expose you to legal liability] if you're putting this on a publicly facing site.
If you're not comfortable doing the source modification yourself, file a bug report with the original author of the software. The source code should have a README file, or BUGS file, or whatever that should outline a procedure for doing so.
Just need one more clarification for what is the difference between SUB_CDEFS, UN_CDEFS, and CDEFS
It's completely arbitrary.
Software projects built with make, can often build multiple programs or libraries. These often are placed in subdirectories. Each such subdirectory often has its own Makefile.
To avoid needless duplication [and potential error], the parts that would be common to these makefiles are placed in a single makefile, often called a rules file [but it's just a makefile]. The individual makefiles then have a line like: include ../common/rules.mk
The rules file expects that certain symbols are defined that help guide it to build the targets for the given subdirectory.
CDEFS et. al. are an example of such symbols. Names that are descriptive of function are [should be] chosen. That is, CDEFS [probably] means "C definitions". The actual symbol names and their function depends upon the rules file. We could use the symbol SHRONK instead of CDEFS. That doesn't help much with understanding things, but if all makefiles were edited to change CDEFS to SHRONK, it would work.
For example, in other software, instead of CDEFS, a similar symbol might be named CFLAGS or COPTS. This is fairly common.
Side note: It's a bit moot at this point, but things would have gone much more smoothly and quickly if you had edited your question and posted the output cc commands and [some of] your makefiles as I had requested. You would have gotten specific answers in a matter of hours instead of general guidelines [that took several days].
So, without the rules file, it's not possible to tell. Only make a guess, based upon the names:
CDEFS -- global cc options for a subdirectories
SUB_CDEF -- cc options for this particular subdirectory
UN_CDEFS -- specify -Ufoo options
The particular software you are building may have documentation for this in a documentation file or in comments in one or more of the makefiles.
To understand this generally, there are many online guides to make. Under Linux, there are "info" files. So, try info make. Other systems have detailed manpages, so do man make
Does the gcc output of the object file (C language) vary between compilations? There is no time-specific information, no change in compilation options or the source code. No change in linked libraries, environmental variables either. This is a VxWorks MIPS64 cross compiler, if that helps. I personally think it shouldn't change. But I observe that sometimes randomly, the instructions generated changes. I don't know what's the reason. Can anyone throw some light on this?
How is this built? For example, if I built the very same Linux kernel, it includes a counter that is incremented each build. GCC has options to use profiler information to guide code generation, if the profiling information changes, so will the code.
What did you analyze? The generated assembly, an objdump of object files or the executable? How did you compare the different versions? Are you sure you looked at executable code, not compiler/assembler/linker timestamps?
Did anything change in the environment? New libraries (and header files/declarations/macro definitions!)? New compiler, linker? New kernel (yes, some header files originate with the kernel source and are shipped with it)?
Any changes in environment variables (another user doing the compiling, different machine, different hookup to the net gives a different IP address that makes it's way into the build)?
I'd try tracing the build process in detail (run a build and capture the output in a file, and do so again; compare those).
Completely mystified...
I had a similar problem with g++. Pre 4.3 versions produced exactly the same object files each time. With 4.3 (and later?) some of the mangled symbol names are different for each run - even without -g or other recordings. Perhaps the use a time stamp or random number (I hope not). Obviously some of those symbols make it into the .o symbol table and you get a difference.
Stripping the object file(s) makes them equal again (wrt. binary comparison).
g++ -c file.C ; strip file.o; cmp file.o origfile.o
Why should it vary? It is the same result always. Try this:
for i in `seq 1000`; do gcc 1.c; md5sum a.out; done | sort | uniq | wc -l
The answer is always 1. Replace 1.c and a.out to suit your needs.
The above counts how many different executables are generated by gcc when compiling the same source for 1000 times.
I've found that in at least some environments, the same source may yield a different executable if the source tree for the subsequent build is located in a different directory. Example:
Checkout a pristine copy of your project to dir1. Do a full rebuild from scratch.
Then, with the same user on the same machine, checkout the same exact copy of your source code to dir2 (dir1 != dir2). Do another full rebuild from scratch.
These builds are minutes apart, with no change in the toolchain or any 3rd party libs or code. Binary comparison of source code is the same. However, the executable in dir1 has different md5sum than the executable in dir2.
If I compare the different executables in BeyondCompare's hex editor, the difference is not just some tiny section that could plausibly be a timestamp.
I do get the same executable if I build in dir1, then rebuild again in dir1. Same if I keep building the same source over and over from dir2.
My only guess is that some sort of absolute paths of the include hierarchy are embedded in the executable.
My gcc sometimes produces different code for exactly the same Input. The output object files differ in exactly one byte.
Sometimes this causes linker Errors, because one possible object file is invalid. Recompiling another version usually fixes the linker error.
The gcc Version is 4.3.4 on Suse Linux Enterprise.
The gcc Parameters are:
cc -std=c++0x -Wall -fno-builtin -march=native -g -I<path1> -I<path2> -I<path3> -o obj/file.o -c file.cpp
If someone experiences the same effect, then please let me know.
List of available LLVM passes are desribed here. I am interested in knowing which passes are included for different optimization levels. For example, for -O2 which passes are included. Is there any site or document which provides that information.
It's visible in PassManagerBuilder.cpp file. Take a look at populateModulePassManager method where OptLevel is integer that follows -O option.
from the command line you can use:
echo "" | opt -O3 -disable-output -debug-pass=Arguments
I know that I can discover the header file dependencies required when building an object file using a few tools (such as gcc -MD ...)
Is there a similar way to determine the static libraries that will be used when a component is linked?
In particular I am looking at some multi-level make files with lots of indirection and I would like to just be able to get a list of the depedencies for that build so I can streamline my build system's rebuild requests.
ex:
make foo.mak
foo.mak
OBJS = bar.o \
bar2.o
DEPS = core\
msg\
utils\
EXTRA_FLAGS += -Wall -Werror
include ../common/common.mak
within common.mak
the members of DEPS will be expanded in various ways depending opn what type of build this is. they may be static, shared or even kernel libraires and they may get pre- or post- fixes.
I would want to get
ABC_core_DEF.a
GEH_msg_IJK.a
(assuming that core and msg were the only dependencies to have expanded to actual static includes and that the pre and post fixes were as shown.)
If your build system supports a mode where the compilation commands are shown (e.g., some setting like VERBOSE=1), you could try and grep this output for items looking like -l (or whatever other kind of linker options your target toolchain uses).