Makefile - How to organize and build multiple library versions - c

Question

I am trying to create a build system, but the structure I am supposed to use in unclear.
I've got various library files with their associated includes.
These library files (or modules) may have various versions. So, module1 may have only 1 version, but ModuleN may have 50 versions.
Now my problem is that I am not sure how to organize my tree so that I can build my Library.a package for a given version.
My first thought was to organize my files in the following way:
Libraries Includes
¦ ¦
---------------------- ----------------------
¦ ¦ ¦ ¦ ¦ ¦
V1.0 V1.1 V1.2 V1.0 V1.1 V1.2
¦ ¦ ¦ ¦ ¦ ¦
Lib1.c Lib3.c Lib2.c Lib1.h Lib3.h Lib2.h
Lib2.c Lib3.c Lib2.h Lib3.h
Lib3.c Lib3.h
Lib4.c Lib4.h
Now, how would I build my package V1.2, considering it also needs to incorporate the Union of the latest libraries across the entire tree (ie. V1.2 package = V1.0 U V1.1 U V1.2). In this case, it would not be too difficult to define the rules manually, but it will quickly become unmanageable with 100`s of files.
Maybe duplicating unchanged files from a version to the next one in common practice, but it then becomes difficult to know where is what.
Is there a typical approach to this problem?

Answer 1

The proper way to handle this is with a version control system like Git. It was designed to handle exactly what you're describing.
There were several suggestions in the comments regarding the use of Git to accomplish what you want. However, you seem to have a misunderstanding of how exactly the files in your library would be stored.
You said in the comments:
I would like to build any version of the library out of the latest repository (pulled freshly from Git if you like), in one go, without having to do potentially 100's of pulls from Git
And:
But if you understood branches, you would know that by switching you will loose your previous snapshot. So if you start by building a branch (get V1.0), then switch to second branch to build V1.1, you will have lost V1.0.
Looking at these comments and your sample file structure, this suggests that you think the V1.0 branch would contain Lib1.c, Lib2.c, Lib3.c, and Lib4.c while the V1.1 branch would only contain Lib3.c and that the V1.2 branch would only contain Lib2.c and Lib3.c. That is not how Git works.
Here is an example of how you would store these files in Git:
Start by putting the V1.0 files into a directory structure:
[dbush#db-centos7 mylib]$ ls -lR
.:
total 0
drwxrwxr-x. 2 dbush dbush 62 Jan 20 10:19 inc
drwxrwxr-x. 2 dbush dbush 62 Jan 20 10:17 src
./inc:
total 16
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:18 Lib1.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:19 Lib2.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:18 Lib3.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:19 Lib4.h
./src:
total 16
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:16 Lib1.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib2.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib3.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib4.c
Then create a new git repo, add the files, and commit them:
[dbush#db-centos7 mylib]$ git init
Initialized empty Git repository in /home/dbush/mylib/.git/
[dbush#db-centos7 mylib]$ git add *
[dbush#db-centos7 mylib]$ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
# (use "git rm --cached <file>..." to unstage)
#
# new file: inc/Lib1.h
# new file: inc/Lib2.h
# new file: inc/Lib3.h
# new file: inc/Lib4.h
# new file: src/Lib1.c
# new file: src/Lib2.c
# new file: src/Lib3.c
# new file: src/Lib4.c
#
[dbush#db-centos7 mylib]$ git commit
Now you have the version 1.0 files committed, so next we create a tag based on the current version:
[dbush#db-centos7 mylib]$ git tag -a -m "version 1.0" V1.0
[dbush#db-centos7 mylib]$ git tag -l
V1.0
Right now, Lib3.c looks like this:
#include <stdio.h>
void lib3()
{
printf("lib3 1.0\n");
}
Now let's make a change for the 1.1 version:
#include <stdio.h>
void lib3()
{
printf("lib3 1.1\n");
}
We can see there's a difference between the working copy and the checked in version:
[dbush#db-centos7 mylib]$ git status
# On branch master
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: src/Lib3.c
#
no changes added to commit (use "git add" and/or "git commit -a")
[dbush#db-centos7 mylib]$ git diff
diff --git a/src/Lib3.c b/src/Lib3.c
index 3593018..12542d9 100644
--- a/src/Lib3.c
+++ b/src/Lib3.c
## -2,5 +2,5 ##
void lib3()
{
- printf("lib3 1.0\n");
+ printf("lib3 1.1\n");
}
Then we commit the change:
[dbush#db-centos7 mylib]$ git add src/Lib3.c
[dbush#db-centos7 mylib]$ git commit -m "updated Lib3.c for version 1.1"
And tag it as version 1.1:
[dbush#db-centos7 mylib]$ git tag -a -m "version 1.1" V1.1
Now let's suppose we want to get the 1.0 code to build. We checkout the tag:
[dbush#db-centos7 mylib]$ git checkout V1.0
Note: checking out 'V1.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at a1ee8c1... initial commit
[dbush#db-centos7 mylib]$ ls -lR
.:
total 0
drwxrwxr-x. 2 dbush dbush 62 Jan 20 10:19 inc
drwxrwxr-x. 2 dbush dbush 62 Jan 20 10:33 src
./inc:
total 16
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:18 Lib1.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:19 Lib2.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:18 Lib3.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:19 Lib4.h
./src:
total 16
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:16 Lib1.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib2.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:33 Lib3.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib4.c
[dbush#db-centos7 mylib]$ cat src/Lib3.c
#include <stdio.h>
void lib3()
{
printf("lib3 1.0\n");
}
[dbush#db-centos7 mylib]$ cat src/Lib2.c
#include <stdio.h>
void lib2()
{
printf("lib2 1.0\n");
}
You can see that the full source tree is available and that Lib3.c is the 1.0 version, as is Lib2.c. Now we want to build 1.1, so let's get the 1.1 tag:
[dbush#db-centos7 mylib]$ git checkout V1.1
Previous HEAD position was a1ee8c1... initial commit
HEAD is now at 95a429c... updated Lib3.c for version 1.1
[dbush#db-centos7 mylib]$ ls -lR
.:
total 0
drwxrwxr-x. 2 dbush dbush 62 Jan 20 10:19 inc
drwxrwxr-x. 2 dbush dbush 62 Jan 20 10:35 src
./inc:
total 16
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:18 Lib1.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:19 Lib2.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:18 Lib3.h
-rw-rw-r--. 1 dbush dbush 13 Jan 20 10:19 Lib4.h
./src:
total 16
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:16 Lib1.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib2.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:35 Lib3.c
-rw-rw-r--. 1 dbush dbush 62 Jan 20 10:17 Lib4.c
[dbush#db-centos7 mylib]$ cat src/Lib3.c
#include <stdio.h>
void lib3()
{
printf("lib3 1.1\n");
}
[dbush#db-centos7 mylib]$ cat src/Lib2.c
#include <stdio.h>
void lib2()
{
printf("lib2 1.0\n");
}
You can see from here that not only is the 1.1 copy of Lib3.c present but also all other files with their V1.0 content. You don't need to do multiple checkouts and you don't lose any files that weren't updated in the latest commit.
You can follow the above example to set up your repo. So you would:
Get the 1.0 files together in the desired directory structure
Create a new git repo and commit the files
Tag version 1.0
Copy in the files changes for version 1.1
Commit the changed files then tag version 1.1
Copy in the files changes for version 1.2
Commit the changed files then tag version 1.2
Etc...
This also works if you need to delete a file when going to a new version. So for example if in version 1.3 you remove Lib4.c, then you'll see the file when you check out tags 1.0, 1.1, and 1.2, but not when you check out tag 1.3.
So once you have your Git repo set up, all you need to do is checkout a single tag once and you have everything you need.
EDIT:
When it comes time use these files for multiple (possibly concurrent) builds, Git is still a good fit for this. If you were to keep your proposed folder structure, you would need to know the full history of the files to get the proper versions of each, and you would still want to make a copy of each relevant source file into you "active" build directory so that object files created by one build aren't overwritten by another. Git will fully manage all of this for you.
Git allows for cloning a local repository, and for performing a minimal clone of a particular branch or tag, that way you only copy what you need and don't grab the entire git history in the processes, so having multiple copies isn't an issue. You can do this as follows:
[dbush#db-centos7 mylib]$ git status
# On branch master
nothing to commit, working directory clean
[dbush#db-centos7 mylib]$ cd ..
[dbush#db-centos7 ~]$ git clone --single-branch --branch V1.0 ./mylib mylib-v1.0
Cloning into 'mylib-v1.0'...
done.
Note: checking out 'a1ee8c1bfdbb20f3e0716212582338371b60d9bc'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
[dbush#db-centos7 ~]$ cd mylib-v1.0
[dbush#db-centos7 mylib-v1.0]$ git status
# Not currently on any branch.
nothing to commit, working directory clean
[dbush#db-centos7 mylib-v1.0]$ git branch -l
* (no branch)
[dbush#db-centos7 mylib-v1.0]$
The --branch option says to clone a specific branch or tag, while the --single-branch option creates a shallow clone containing just the given tag so that the .git directory of the clone isn't huge and can be created quickly.

Answer 2

I know that OP said he did not want to use Git, so this answer is more for other people who reads this question.
A very common practice to build an arbitrary version from a repository is to use tags. Tagging is a way to highlight a specific commit. Git supports two types of tags: lightweight and annotated.
Lightweight tags is like a branch. It is just a pointer to a commit. The difference from a branch is that this pointer does not change with new commits. These tags are not recommended for your purpose. They are more for developers during developing. To create a lightweight tag, use this command
git tag <name>
The other tag type is annotated tags. They contain much more information and are checksummed. These are the ones you want to use for releases. As with commits, you specify a tag message for annotated tags. To create an annotated tag, use this command:
git tag -a <name>
This will open up an editor where you write a tag message. You can specify a message on command line too with -m "<message>"
To view all tags, just use:
git tag
You can checkout tags the same way you checkout branches:
git checkout <tag>
Read more about tagging here: https://git-scm.com/book/en/v2/Git-Basics-Tagging
So what you do is that you create a (preferably annotated) tag for each commit that corresponds to a release. Then, assuming you can build the project by just invoking make you can build any version with these two commands:
git checkout <tag>
make
Note: In the comments above, I was talking about branches. That was temporary confusion on my behalf. It is possible to use branches for this purpose, but tagging is the proper method.