Chapter 5. The Dependency Maintenance Tool

Table of Contents

1. Required Features
1.1. View Paths
1.2. Dynamic Include File Dependencies
2. Development Directory Style
2.1. View Path
2.2. Link the Baseline
2.3. Copy All Sources
2.4. Obsolete Features
3. Using Cook
3.1. Invoking Cook
3.2. The Recipe File
3.3. The Recipe for C
3.4. The Recipe for Yacc
3.5. The Recipe for Lex
3.6. Recipes for Documents
3.7. Templates
4. Using Cake
4.1. Invoking Cake
4.2. The Rules File
4.3. The Rule for C
4.4. The Rule for Yacc
4.5. The Rule for Lex
4.6. Rules for Documents
5. Using Make
5.1. Invoking Make
5.2. The Rule File
5.3. The Rule for C
5.4. The Rule for Yacc
5.5. The Rule for Lex
5.6. Rules for Documents
5.7. Other Makes
5.8. Templates
5.9. GNU Make VPATH Patch
5.10. GNU Make's VPATH+
6. Building Executable Scripts
7. GNU Autoconf
7.1. The Sources
7.2. Building
7.3. Testing
7.4. An Optimization
7.5. Signed-off-by
7.6. Importing the Next Upstream Tarball
7.7. Importing the Next Upstream Patch
8. No Build Required
8.1. Why This May Not Be Such A Good Idea

Aegis can place heavy demands on the dependency maintenance tool, so it is important that you select an appropriate one. This chapter talks about what features a dependency maintenance tool requires, and gives examples of how to use the various alternatives.

1. Required Features

The heart of any DMT is an inference engine. This inference engine accepts a goal of what you want it to construct and a set of rules for how to construct things, and attempts to construct what you asked for given the rules you specified. This is exactly a description of an expert system, and the DMT needs to be an expert system for constructing files. Something like PROLOG is probably ideal.

Aegis is capable of supporting a wide variety of development directory styles. The different development directory styles place different demands on the dependency maintenance tool. Development directory styles will be described in the next section, but here is a quick summary:

copy of all sources:

This is what CVS does, and what many other VC tool do. Because you have a complete copy of all source files, the dependency maintenance tool only needs to be aware of one directory tree.

copy of everything:

This is a small optimization of the previous case to cut down the time required for that first build,because the derived files from the integration build can be reused.

link all sources

The is an optimization of the "copy all sources" case, because linking a file is significantly faster than making a copy of a file. The dependency maintenance tool only needs to be aware of one directory tree.

link everything

This is an optimization of the previous case, again reusing derived files from the integration build, except that you need to ensure that your dependency maintenance tool is configured to remove the derived file outputs of each rule before creating them, to avoid corrupting the baseline or getting "permission denied" error.

view path

This is the most efficient development directory style, and it scales much better than any of the above, but the dependency maintenance tool must be able to cope with a hierarchy of parallel source directory trees. These trees for a "view path", a list of directories that programs search below to find the files of interest. The vpath\gP statements of GNU Make are almost, but not quite, capable of being used in this way.

1.1. View Paths

For the union of all files in a project and all files in a change (remembering that a change only copies those files it is modifying, plus it may add or remove files) for all files you must be able to say to the dependency maintenance tool,

"If and only if the file is up-to-date in the baseline, use the baseline copy of the file, otherwise construct the file in the development directory".

The presence of a source file in the change makes the copy in the baseline out-of-date.

Most DMTs with this capability implement it by using some sort of search path, allowing a hierarchy of directories to be scanned with little or no modification to the rules.

If your DMT of choice does not provide this functionality, the development_­directory_­style.­source_­file_­symlink field of the project configuration file may be set to true, which tells Aegis to maintain symbolic links in the development directory for all source files in the baseline which are not present in the development directory. (See aepconf(5) and aeb(1) for more information.) This incurs a certain amount of overhead when Aegis maintains these links, but a similar amount of work is done within DMTs which have search path functionality.

1.2. Dynamic Include File Dependencies

Include file dependencies are very important, because a change may alter an include file, and all of the sources in the baseline which use that include file must be recompiled.

Consider the example given earlier: the include file describing the interface definition of a function is copied into a change and edited, and so is the source file defining the function. It is essential that all source files in the baseline which include that file are recompiled, which will usually result in suitable diagnostic errors if any of the clients of the altered function have yet to be included in the change.

There are two ways of handling include file dependencies:

  • They can be kept in a file, and the file can be maintained by suitable programs (maintaining it manually never works, that's just human nature).

  • They can be determined by the DMT when it is scanning the rules to determine what needs updating.

1.2.1. Static File

Keeping include dependencies in a file has a number of advantages:

  • Most existing DMTs have the ability to include other rules files, so that when performing a development build from a baseline rules file, it could include a dependencies file in the development directory.

  • Reading a file is much faster than scanning all of the source files.

Keeping include dependencies in a file has a number of disadvantages:

  • The file is independent of the DMT, it is either generated before the DMT is invoked, in which case it may do more work than is necessary, or it may be invoked after the DMT (or after the DMT has scanned its rules), in which case it may well be out-of-date when the DMT needs it.

    For example, the use of gcc -M produces "dot d" files, which may be merged to construct such an includable dependency file. This happens after the DMT has read and applied the rules, but possibly before the DMT has finished executing.[18]

  • Many tools which can generate this information, such as the gcc -M option, are triggered by source files, and are unable to manage a case where it is an include file which is changing, to include a different set of other include files. In this case, the inaccurate dependencies file may contain references to the old set of nested include files, some of which may no longer exist, This causes the DMT to incorrectly generate an error stating that the old include file is missing, when it is actually no longer required.

If a DMT can only support this kind of include file dependencies, it is not suitable for use with Aegis.

1.2.2. Dynamic

In order for a DMT to be suitable for use with Aegis, it is essential that rules for the DMT may be specified in such a way that include file dependencies are determined "on the fly" when the DMT is determining if a given rule is applicable, and before the rule is applied.

This method suffers from the problem being rather slow; but this is amenable to some caching and the losses of performance are not as bad as could be imagined.

This method has the advantage of correctness in all cases, where a static file may at times be out-of-date.

2. Development Directory Style

The project configuration file, usually called aegis.conf, contains a field called development_­directory_­style which controls how the project sources are presented to the DMT.

See aepconf(5) for a complete description of this field.

There is a corresponding integration_­directory_­style field, which defaults to the same value as the development_­directory_­style. It is usually a very bad idea if these two are different.

2.1. View Path

By not setting development_­directory_­style at all the only source files present in the development directory are source files being create and/or modified.

By using information provided by the $search_path substitution, the build can access the unchanged source files in the branch baseline and deeper branch baselines. The great thing about this approach is that there are also "precompiled" object files on the viewpath, so if an object file does not need to be compiled (there are no soure files in the development directory that have anything to do with it) then the build can simply link the unchanged object files in the baseline without recompiling.

This build method scales the best, and is the Aegis default.

The difficulties of finding a DMT which is capable of coping with a view path means that this is not the only work area style. All other methods scale less well than a view path; some scale much less well.

2.2. Link the Baseline

The first two sub-fields of interest in the development_­directory_­style are source_­file_­link and source_­file_­symlink.

source_file_link = true; This field is true if hard links are to be used for project source files (which are not part of the change) so that the work area has a complete set of source files.

source_file_symlink = true; This field is true if symbolic links are to be used for project source files (which are not part of the change) so that the work area has a complete set of source files.

By using these settings, all source files are present in the development directory. They will be read-only. As you decide to modify files in the change set, the aecp command will remove the link and replace it with a read-write copy of the file.

You need both these sub-fields set, because hard links are not allowed to cross file system boundaries. Aegis will use hard links in preference to soft links when it can.

Maintaining the hard links can be time consuming for large projects, and add quite a noticeable delay before builds start doing anything. But see the -assume-symbolic-links option of the aeb(1) command; use it sparingly.

The biggest penalty with this method is that the initial build for a change set for a large project can be very time consuming. Recall that the baseline has a complete "prebuild" already available. To take advantage of these pre-built derived files, there are a few more sub-fields:

derived_file_copy = true; This field is true if copies are to be used for non-source files which are present in the project baseline but which are not present in the work area, so that the work area has a complete set of derived files.

derived_at_start_only = true; This settign causes the above fields controling the appearance of derived files to be acted upon only when the development directory is created (at aedb(1) time).

Copying files can be very time consuming and also eats a lot of disk space. If you are prepared to change your build slightly, it is possible to use the following fields:

derived_file_link = true; This field is true if hard links are to be used for non-source files which are present in the project baseline but which are not present in the work area, so that the work area has a complete set of derived files.

derived_file_symlink = true; This field is true if symbolic links are to be used for non-source files which are present in the project baseline but which are not present in the work area, so that the work area has a complete set of derived files.

Just as for source files, hard links will be used in preference to symbolic links if possible.

Note that every rule in your Makefile (or whatever your DMT uses) must remove its outputs before doing enything else, to break the links to the files in the baseline, otherwise you will corrupt the baseline. Aegis tries very hard to ensure that the baseline files (and thus the links) are read-only, so that you get an error from the build if you forget to break a link.

This development directory style is called "arch style" after Tom Lord's arch (tla) which does something very similar.

If you are placing an existing project under Aegis, do the above three things one step at a time. First get the source files available and integrate that. In a second change set get derived file copies working. In a third change set (if you do it at all) change the build and use derived file links.

2.3. Copy All Sources

The sub-fields of interest in the development_­directory_­style is source_­file_­copy.

source_file_copy = true; This field says copies are to be used for project source files (which are not part of the change) so that the work area has a complete set of source files. File modification time attributes will be preserved.

By using this setting, all source files are present in the development directory. They will be read-only. As you decide to modify files in the change set, the aecp command will remove the file and replace it with a read-write copy of the file.

Maintaining the copies can be time consuming for large projects, and add quite a noticeable delay before builds start doing anything. But see the -assume-symbolic-links option of the aeb(1) command; use it sparingly (yes, it applies to copies as well).

The biggest penalty with this method is that the initial build for a change set for a large project can be very time consuming. Recall that the baseline has a complete "prebuild" already available. To take advantage of these pre-built derived files, there are a few more sub-fields:

derived_file_copy = true; This says copies are to be used for non-source files which are present in the project baseline but which are not present in the work area, so that the work area has a complete set of derived files.

derived_at_start_only = true; This setting causes the above fields controlling the appearance of derived files to be acted upon only when the development directory is created (at aedb(1) time).

This development directory style is called "CVS style" after GNU CVS which does something very similar.

2.4. Obsolete Features

There are several fields in the aegis.conf file which are obsolete. Aegis will automatically transfer these to create a development_­directory_­style if you haven't specified one.

create_­symlinks_­before_­build: This is like setting both development_­directory_­style.source_­file_­symlink and development_­directory_­style.derived_file_symlink at the same time.

remove_­symlinks_­after_­build: This is like setting the development_­directory_­style.during_­build_­only field.

create_­symlinks_­before_­integration_­build: This is like setting both integration_­directory_­style.source_­file_­symlink and integration_­directory_­style.derived_file_symlink at the same time.

remove_­symlinks_­after_­integration_­build: This is like setting the integration_­directory_­style.during_­build_­only field.

Aegis will print a warning if you use any of these fields.

3. Using Cook

The Cook program is the only dependency maintenance tool, known to the author, which is sufficiently capable to supply Aegis' needs.[19] Tools such as cake and GNU Make are described later. They need a special tweak to make them work.

This section describes appropriate contents for the Howto.cook file, input to the cook program. It also discusses the build_command and integrate_­build_­command and link_­baseline and change_­file_­command and project_­file_­command and link_­integration_­directory fields of the configuration file. See aepconf for more information about this file.

3.1. Invoking Cook

The build_command field of the configuration file is used to invoke the relevant build command. In this case, it is set as follows

build_command =
  "cook -b ${s Howto.cook} -nl\
   project=$p change=$c version=$v";

This command tells Cook where to find the recipes. The ${s Howto.cook} expands to a path into the baseline during development if the file is not in the change. Look in aesub for more information about command substitutions.

The recipes which follow will all remove their targets before constructing them, which qualifies them for the next entry in the configuration file:

link_integration_directory = true;

The links must be removed first, otherwise the baseline would cease to be self-consistent.

3.2. The Recipe File

The file containing the recipes is called Howto.cook and is given to Cook on the command line.

The following items are preamble to the rest of the file; they ask Aegis for the source files of the project and change so that Cook can determine what needs to be compiled and linked.

project_files =
  [collect_lines aelpf
    -p [project] -c [change]];
change_files =
  [collect_lines aelcf
    -p [project] -c [change]];
source_files =
  [stringset [project_files]
    [change_files]];

This example continues the one from chapter 3, and thus has a single executable to be linked from all the object files

object_files =
  [fromto %.y %.o [match_mask %.y
    [source_files]]]
  [fromto %.l %.o [match_mask %.l
    [source_files]]]
  [fromto %.c %.o [match_mask %.c
    [source_files]]]
  ;

It is necessary to determine if this is a development build, and thus has the baseline for additional ingredients searches, or an integration build, which does not. The version supplied by Aegis will tell us this information, because it will be major.minor.Cchange for development builds and major.minor.Ddelta for integration builds.

if [match_mask %1C%2 [version]] then
{
  baseline = [collect aegis -cd -bl
    -p [project]];
  search_list = . [baseline];
}

The search_list variable in Cook is the list of directories to search for dependencies; it defaults to only the current directory. The resolve builtin function of Cook may be used to ask Cook for the name of the file actually used to resolve dependencies, so that recipe bodies may reference the appropriate file:

example: [object_files]
{
  [cc] -o example
    [resolve [object_files]]
    -ly -ll;
}

This recipe says that to Cook the example program, you need the object files determined earlier, and them link them together. Object files which were up to date in the baseline are used wherever possible, but files which were out of date are constructed in the current directory and those will be linked.

3.3. The Recipe for C

Next we need to tell Cook how to manage C sources. On the surface, this is a simple recipe:

%.o: %.c
{
  rm %.o;
  [cc] [cc_flags] -c %.c;
}

Unfortunately it has forgotten about finding the include file dependencies. The Cook package includes a program called c_incl which is used to find them. The recipe now becomes

%.o: %.c: [collect c_incl -eia %.c]
{
  rm %.o;
  [cc] [cc_flags] -c %.c;
}

The file may not always be present to be removed (causing a fatal error), and it is irritating to execute a redundant command, so the remove is mangled to look like this:

%.o: %.c: [collect c_incl -eia %.c]
{
  if [exists %.o] then
    rm %.o
      set clearstat;
  [cc] [cc_flags] -c %.c;
}

The "set clearstat" clause tells Cook that the command will invalidate parts of its stat cache, and to look at the command for what to invalidate.

Another thing this recipe needs is to use the baseline for include files not in a change, and so the recipe is altered again:

%.o: %.c: [collect c_incl -eia
  [prepost "-I" "" [search_list]]
    %.c]
{
 if [exists %.o] then
    rm %.o
      set clearstat;
  [cc] [cc_flags] [prepost "-I" ""
    [search_list]] -c %.c;
}

See the Cook Reference Manual for a description of the prepost builtin function, and other Cook details.

There is one last change that must be made to this recipe, it must use the resolve function to reference the appropriate file once Cook has found it on the search list:

%.o: %.c: [collect c_incl -eia
  [prepost "-I" "" [search_list]]
    [resolve %.c]]
{
  if [exists %.o] then
    rm %.o
      set clearstat;
  [cc] [cc_flags] [prepost "-I" ""
    [search_list]] -c [resolve %.c];
}

Only use this last recipe for C sources, the others are only shown so that the derivation of the recipe is clear; while it is very similar to the original, it looks daunting at first.

3.3.1. C Include Semantics

The semantics of C include directives make the

#include "filename"

directive dangerous in a project developed with the Aegis program and Cook.

Depending on the age of your compiler, whether it is AT&T traditional C or newer ANSI C, this form of directive will search first in the current directory and then along the search path, or in the directory of the including file and then along the search path.

The first case is fairly benign, except that compilers are rapidly becoming ANSI C compliant, and an operating system upgrade could result in a nasty surprise.

The second case is bad news. If the source file is in the baseline and the include file is in the change, you don't want the source file to use the include file in the baseline.

Always use the

#include <filename>

form of the include directive, and set the include search path explicitly on the command line used by Cook.

Cook is able to dynamically adapt to include file dependencies, because they are not static. The presence of an include file in a change means that any file which includes this include file, whether that source file is in the baseline or in the change, must have a dependency on the change's include file. Potentially, files in the baseline will need to be recompiled, and the object file stored in the change, not the baseline. Subsequent linking needs to pick up the object file in the change, not from the baseline.

3.4. The Recipe for Yacc

Having explained the complexities of the recipes in the above section about C, the recipe for yacc will be given without delay:

%.c %.h: %.y
{
  if [exists %.c] then
    rm %.c
      set clearstat;
  if [exists %.h] then
    rm %.h
      set clearstat;
  [yacc] [yacc_flags] -d
    [resolve %.y];
  mv y.tab.c %.c;
  mv y.tab.h %.h;
}

This recipe could be jazzed up to cope with the listing file, too, if that was desired, but this is sufficient to work with the example.

Cook's ability to cope with transitive dependencies will pick up the generated .c file and construct the necessary .o file.

3.5. The Recipe for Lex

The recipe for lex is vary similar to the recipe for yacc.

%.c: %.l
{
  if [exists %.c] then
    rm %.c
      set clearstat;
  [lex] [lex_flags] -d [resolve %.l];
  mv lex.yy.c %.c;
}

Cook's ability to cope with transitive dependencies will pick up the generated .c file and construct the necessary .o file.

3.6. Recipes for Documents

You can format documents, such as user guides and manual entries with Aegis and Cook, and the recipes are similar to the ones above.

%.ps: %.ms: [collect c_incl -r -eia
  [prepost "-I" "" [search_list]]
  [resolve %.ms]]
{
  if [exists %.ps] then
    rm %.ps
      set clearstat;
  roffpp [prepost "-I" ""
    [search_list]] [resolve %.ms]
    | groff -p -t -ms
    > [target];
}

This recipe says to run the document through groff, with the pic and tbl filters, use the ms macro package, to produce PostScript output. The roffpp program comes with Cook, and is like soelim but it accepts include search path options on the command line.

Manual entries may be handled in a similar way

%.cat: %.man: [collect c_incl -r -eia
  [prepost "-I" "" [search_list]]
  [resolve %.man]]
{
  if [exists %.cat] then
    rm %.cat
      set clearstat;
  roffpp [prepost "-I" ""
    [search_list]] [resolve %.man]
    | groff -Tascii -t -man
    > [target];
}

3.7. Templates

The lib/config.example/cook file in the Aegis distribution contains all of the above commands, so that you may readily insert them into your project configuration file.

4. Using Cake

This section describes how to use cake as the dependency maintenance tool. The cake package was published in the comp.sources.unix USENET newsgroup volume 12, around February 1988, and is thus easily accessible from the many archives around the internet.

It does not have a search path of any form, not even something like VPATH It does, however, have facilities for dynamic include file dependencies.

4.1. Invoking Cake

The build_command field of the configuration file is used to invoke the relevant build command. In this case, it is set as follows

build_command =
  "cake -f ${s Cakefile} \
   -DPROJECT=$p -DCHANGE=$c \
   -DVERSION=$v";

This command tells cake where to find the rules. The ${s Cakefile} expands to a path into the baseline during development if the file is not in the change. Look in aesub for more information about command substitutions.

The rules which follow will all remove their targets before constructing them, which qualifies them for the next entry in the configuration file:

link_integration_directory = true;

The links must be removed first, otherwise the baseline would be corrupted by integration builds.

Another field to be set in this file is

development_directory_style =
{
    source_file_symlink = true;
};

which tells Aegis to maintain symbolic links between the development directory and the baseline. This also requires that rules remove their targets before constructing them, to ensure that rules do not attempt to write their results onto the read-only versions in the baseline.

4.2. The Rules File

The file containing the rules is called Cakefile and is given to cake on the command line.

The following items are preamble to the rest of the file; they ask Aegis for the source files of the project and change so that cake can determine what needs to be compiled and linked.

#define project_files \
  [[aelpf -p PROJECT \
    -c CHANGE]];
#define change_files \
  [[aelcf -p PROJECT \
   -c CHANGE]];
#define source_files \
  project_files change_files

#define CC     gcc
#define CFLAGS -O

This example parallels the one from chapter 3, and thus has a single executable to be linked from all the object files

#define object_files \
  [[sub -i X.c %.o source_files]] \
  [[sub -i X.y %.o source_files]] \
  [[sub -i X.l %.o source_files]]

Constructing the program is straightforward

example: object_files
  rm -f example
  CC -o example object_files

This rule says that to construct the example program, you need the object files determined earlier, and them link them together. Object files which were up to date in the baseline are used wherever possible, but files which were out of date are constructed in the current directory and those will be linked.

4.3. The Rule for C

Next we need to tell cake how to manage C sources. On the surface, this is a simple rule:

%.o: %.c
  CC CFLAGS -c %.c

paralleling that found in most makes, however it needs to delete the target first, and to avoid deleting the .o file whenever cake thinks it is transitive.

%.o!: %.c
  rm -f %.o
  CC CFLAGS -c %.c

The -f option to the rm command is because the file does not always exist.

Unfortunately this rule omits finding the include file dependencies. The cake package includes a program called ccincl which is used to find them. The rule now becomes

%.o!: %.c* [[ccincl %.c]]
  rm -f %.o
  CC CFLAGS -c %.c

This rule is a little quirky about include files which do not yet exist, but must be constructed by some other rule. You may want to use gcc -MM instead, which is almost as quirky when used with cake. Another alternative, used by the author with far more success, is to use the c_incl program from the cook package, mentioned in an earlier section. The gcc -MM understands C include semantics perfectly, the c_incl command caches its results and thus goes faster, so you will need to figure which you most want.

4.3.1. Include Directives

Unlike cook described in an earlier section, using cake as described here allows you to continue using the

#include "filename"

form of the include directive. This is because the development directory appears, to the compiler, to be a complete copy of the baseline.

4.4. The Rule for Yacc

Having explained the complexities of the rules in the above section about C, the rule for yacc will be given without delay:

#define YACC yacc
#define YFLAGS

%.c! %.h!: %.y  if exist %.y
  rm -f %.c %.h y.tab.c y.tab.h
  YACC YFLAGS -d %.y
  mv y.tab.c %.c
  mv y.tab.h %.h

This rule could be jazzed up to cope with the listing file, too, if that was desired, but this is sufficient to work with the example.

Cake's ability to cope with transitive dependencies will pick up the generated .c file and construct the necessary .o file.

4.5. The Rule for Lex

The rule for lex is vary similar to the rule for yacc.

#define LEX lex
#define LFLAGS

%.c!: %.l  if exist %.l
  rm -f %.c
  LEX LFLAGS %.l
  mv lex.yy.c %.c

Cake's ability to cope with transitive dependencies will pick up the generated .c file and construct the necessary .o file.

4.6. Rules for Documents

You can format documents, such as user guides and manual entries with Aegis and cake, and the rules are similar to the ones above.

%.ps!: %.ms* [[soincl %.ms]]
  rm -f %.ps
  groff -s -p -t -ms %.ms > %.ps

This rule says to run the document through groff, with the soelim and pic and tbl filters, use the ms macro package, to produce PostScript output.

This suffers from many of the problems with include files which need to be generated, as does the C rule, above. You may want to use c_incl -r from the cook package, rather than the soincl supplied by the cake package.

Manual entries may be handled in a similar way

%.cat!: %.man* [[soincl %.man]]
  rm -f %.cat
  groff -Tascii -s -t -man %.man \
    > %.cat

5. Using Make

The make(1) program exists in many forms, usually one is available with each Unix™ version. The one used in the writing of this section is GNU Make 3.70 available by anonymous FTP from your nearest GNU archive site. GNU Make was chosen because it was the most powerful, it is widely available (usually for little or no cost) and discussion of the alternatives (SunOS make, BSD 4.3 make, etc), would not be universally applicable. "Plain vanilla" make (with no transitive closure, no pattern rules, no functions) is not sufficiently capable to satisfy the demands placed on it by Aegis.

With the introduction of the development_­directory_­style field of the project configuration file, any project which is currently using a "plain vanilla" make may continue to use it, and still manage the project using Aegis.

As mentioned earlier in this chapter, make is not really sufficient, because it lacks dynamic include dependencies. However, GNU Make has a form of dynamic include dependencies, and it has a few quirks, but mostly works well.

The other feature lacking in make is a search path. While GNU Make has functionality called VPATH the implementation leaves something to be desired, and can't be used for the search path functionality required by Aegis. Because of this, the development_­directory_­style.­source_­file_­symlink field of the project configuration file is set to true so that Aegis will arrange for the development directory to be full of symbolic links, making it appear that the entire project source is in each change's development directory.

5.1. Invoking Make

The build_command field of the project configuration file is used to invoke the relevant build command. In this case, it is set as follows

build_command =
  "gmake -f ${s Makefile} project=$p \
   change=$c version=$v";

This command tells make where to find the rules. The ${s Makefile} expands to a path into the baseline during development if the file is not in the change. Look in aesub for more information about command substitutions.

The rules which follow will all remove their targets before constructing them, which qualifies them for the next entry in the configuration file:

link_integration_directory = true;

The files must be removed first, otherwise the baseline would be corrupted by integration builds (or even by developer builds, if your aren't using a separate user for the project owner).

[Note]Note:

if you are migrating an existing project do not set this field; only set it after you have changed all of the Make rules. If in doubt, don't set this field.

Another field to be set in this file is

development_directory_style =
{
    source_file_symlink = true;
};

which tells Aegis to maintain symbolic links between the development directory and the baseline for source files (but not derived files). See aepconf(5) for more information.

5.2. The Rule File

The file containing the rules is called Makefile and is given to make on the command line.

The following items are preamble to the rest of the file; they ask Aegis for the source files of the project and change so that make can determine what needs to be compiled and linked.

project_files := \
  $(shell aelpf -p $(project) \
    -c $(change))
change_files := \
  $(shell aelcf -p $(project) \
    -c $(change))
source_files := \
  $(sort $(project_files) \
    $(change_files))
CC := gcc
CFLAGS := -O

This example parallels the one from chapter 3, and thus has a single executable to be linked from all the object files

object_files := \
  $(patsubst %.y,%.o,$(filter \
    %.y,$(source_files)))     \
  $(patsubst %.l,%.o,$(filter \
    %.l,$(source_files)))     \
  $(patsubst %.c,%.o,$(filter \
    %.c,$(source_files)))

Constructing the program is straightforward, remembering to remove the target first.

example: $(object_files)
  rm -f example
  $(CC) -o example $(object_files) \
    -ly -ll

This rule says that to make the example program, you need the object files determined earlier, and them link them together. Object files which were up to date in the baseline are used wherever possible, but files which were out of date are constructed in the current directory and those will be linked.

5.3. The Rule for C

Next we need to tell make how to manage C sources. On the surface, this is a simple rule:

%.o: %.c
  $(CC) $(CFLAGS) -c $*.c

This example matches the built-in rule for most makes. But it forgets to remove the target before constructing it.

%.o: %.c
  rm -f $*.o
  $(CC) $(CFLAGS) -c $*.c

The target may not yet exist, hence the -f option.

Something missing from this rule is finding the include file dependencies. The GNU Make User Guide describes a method for obtaining include file dependencies. A set of dependency files are constructed, one per .c file.

%.d: %.c
  rm -f %.d
  $(CC) $(CFLAGS) -MM $*.c \
  | sed 's/^\(.*\).o :/\1.o \1.d :/' \
  > $*.d

These dependency files are then included into the Makefile to inform GNU Make of the dependencies.

include $(patsubst \
  %.o,%.d,$(object_files))

GNU Make has the property of making sure all its include files are up-to-date. If any are not, they are made, and then GNU Make starts over, and re-reads the Makefile and the include files from scratch, before proceeding with the operation requested. In this case, it means that our dependency construction rule will be applied before any of the sources are constructed.

This method is occasionally quirky about absent include files which you have yet to write, or which are generated and don't yet exist, but this is usually easily corrected, though you do need to watch out for things which will stall an integration.

The -MM option to the $(CC) command means that this rule requires the gcc program in order to work correctly. It may be possible to use c_incl from cook, or ccincl from cake to build the dependency lists instead; but they don't understand the conditional preprocessing as well as gcc does.

This method also suffers when heterogeneous development is performed. If you include different files, depending on the environment being compiled within, the .d files may be incorrect, and GNU Make has no way of knowing this.

5.3.1. Include Directives

Unlike cook described in an earlier section, using GNU Make as described here allows you to continue using the

#include "filename"

form of the include directive. This is because the development directory appears, to the compiler, to be a complete copy of the baseline.

5.4. The Rule for Yacc

Having explained the complexities of the rules in the above section about C, the rule for yacc will be given without delay:

%.c %.h: %.y
  rm -f $*.c $*.h y.tab.c y.tab.h
  $(YACC) $(YFLAGS) -d $*.y
  mv y.tab.c $*.c
  mv y.tab.h $*.h

This rule could be jazzed up to cope with the listing file, too, if that was desired, but this is sufficient to work with the example.

GNU Make's ability to cope with transitive closure will pick up the generated .c file and construct the necessary .o file.

To prevent GNU Make throwing away the transitive files, and thus slowing things down in some cases, make them precious:

.PRECIOUS: \
  $(patsubst %.y,%.c,$(filter \
    %.y,$(source_files)))     \
  $(patsubst %.y,%.h,$(filter \
    %.y,$(source_files)))

5.5. The Rule for Lex

The rule for lex is vary similar to the rule for yacc.

%.c: %.l
  rm -f $*.c lex.yy.c
  $(LEX) $(LFLAGS) $*.l
  mv lex.yy.c $*.c

GNU Make's ability to cope with transitive closure will pick up the generated .c file and construct the necessary .o file.

To prevent GNU Make throwing away the transitive files, and thus slowing things down in some cases, make them precious:

.PRECIOUS: \
  $(patsubst %.l,%.c,$(filter \
    %.l,$(source_files)))

5.6. Rules for Documents

You can format documents, such as user guides and manual entries with Aegis and GNU Make, and the rules are similar to the ones above.

%.ps: %.ms
  rm -f $*.ps
  groff -p -t -ms $*.ms > $*.ps

This rule says to run the document through groff, with the pic and tbl filters, use the ms macro package, to produce PostScript output.

This omits include file dependencies. If this is important to you, the c_incl program from cook can be used to find them. Filtering its output can then produce the necessary dependency files to be included, rather like the C rules, above.

Manual entries may be handled in a similar way

%.cat: %.man
  rm $*.cat
  groff -Tascii -s -t -man $*.man \
    > $*.cat

5.7. Other Makes

All of the above discussion assumes that GNU Make and GCC are used. If you do not want to do this, or may not do this because of internal company politics, it is possible to perform all of the automated features manually.

This may, however, rapidly become spectacularly tedious. For example: if a user needs to copy the Makefile into their change for any reason, they will need to constantly use aed to "catch up" with integrations into the baseline.

Reviewers are also affected: they must check that each change to the Makefile accurately reflects the object list and the dependencies of each source file.

5.8. Templates

The lib/config.example/make file in the Aegis distribution contains all of the above commands, so that you may readily insert them into your project configuration file.

5.9. GNU Make VPATH Patch

Version 3.76 and later of GNU Make include this patch, so you don't need to read this section unless you have GNU Make 3.75 or earlier.

There is a patch available for GNU Make 3.75 and earlier which gives it improved VPATH semantics. At the time it was not maintained by the same person who maintained GNU Make. Since then, the maintaier changed, and the patch has been incorporated.

The patch is the work of Paul D. Smith <psmith AT BayNetworks.com> and may be fetched By Anonymous FTP from

Host:ftp.wellfleet.com
Dir:/netman/psmith/gmake
File:vpath+.README
File:vpath+.patch.version

The version numbers track the GNU Make version numbers.

For a description of the VPATH problem, and how this patch addresses it, see the README file referenced.

5.10. GNU Make's VPATH+

In theory, using GNU Make 3.76 or later (or a suitable patched earlier version) is similar to using Cook. The project configuration file now requires

link_integration_directory = false;

which is the default. The Makefile now requires

VPATH . bl

Assuming that bl is a symbolic link to the baseline. The .d files continue to be used.

6. Building Executable Scripts

Aegis treats source files as, well, source files. This means that it forgets any executable bits (and any other mode bits) you may set on the file. Usually, this isn't a problem - except for scripts.

So, just how do you get Aegis to give you an executable script? Well, you add a build rule. However, since it can't depend on itself, it needs to depend on something else.

Using a Cook example, we could write

bin/%: script/%.sh
{
    /* copy the script */
    cp script/%.sh bin/%;
    /* make it executable */
    chmod a+rx bin/%;
    /* syntax check */
    bash -n bin/%;
}

There is a small amout of value-added here: we did a syntax check along the way, which catches all sorts of problems.

The same technique also works for Perl

bin/%: script/%.pl
{
    cp script/%.pl bin/%;
    chmod a+rx bin/%;
    perl -cw bin/%;
}

The same technique also works for TCL

bin/%: script/%.tcl
{
    cp script/%.rcl bin/%;
    chmod a+rx bin/%;
    procheck -nologo bin/%;
}

The procheck(1) command is part of the TclPro package.

Many tools have a similar options.

You can also combine this with GNU Autoconf to produce architecture specific shell scripts from architecture neutral sources.

7. GNU Autoconf

If your projects uses GNU Make, GNU Autoconf and GNU Automake, here is a quick and simple method to import your project into Aegis and have it running fairly quickly.

7.1. The Sources

Once you have create and Aegis project to, your first change set should simply contain all of the source files, without removing or adding anything. The only additonal file is the Aegis project configuration file, usually called aegis.conf and usually located in the top-level directory.

Follow the directions in the section, above, on using Make for how to fill out this file.

Note that if you are working from a tarball, they usually contain several derived file. That is, files which are not primary sourec files, but are instead derived from other files. This is a convenience for the end-user but a nuisance at this point. Exanple of derived files in source tarballs include configure, Makefile.in, config.h.in, etc. You will need to exclude them form the first change set.

In this first change set, you don't even try to build anything.

build_command = "exit 0";

Which will allow the Aegis process to complete.

7.2. Building

You actually get your project to buld in the second change set. Once you have started development, you will see all of the source files in the development directory (well, symlinks to them).

In order to get you build to work, you have to bootstrap the Makefile. Using the usual GNU tool chain, this file is generated from Makefile.in which is in turn generated from Makefile.am, and this is not presently in the development directory.

This is done by creating a new primary source file called makefile at the top level

$ aenf makefile
$

and setting its contents to be

include Makefile

ifndef srcdir

bogus-default-target: Makefile
	$(MAKE) $(MAKEFLAGS) $(MAKECMDGOALS)

Makefile: configure Makefile.in \
  config.h.in
	rm -f config.cache
	./configure

configure: configure.ac
	autoconf

config.h.in: configure.ac
	autoheader

Makefile.in: Makefile.am
	automake

endif

This works because make(1) looks for makefile before Makefile, but also because our bootstrapping makefile includes the real Makefile if it exists, and the real file's rules will take precedence. At this point, GNU Make has a very useful feature: it will rebuild include files which are out-of-date before it does anything else. In oue new development directory, this will result in the necessary files being automagically generated and then acted upon.

Things that can go wrong: many projects include files such as install-sh and missing and mkinstalldirs in the directibution. You will need to include rules for these files in the conditonal part of your bootstrapping makefile rules.

AUTOMAKE_DIR=/usr/share/automake-1.7

install-sh: $(AUTOMAKE_DIR)/install-sh
	cp $^ $@

missing: $(AUTOMAKE_DIR)/missing
	cp $^ $@

mkinstalldirs: \
  $(AUTOMAKE_DIR)/mkinstalldirs
	cp $^ $@

You will have to tell the configure rule that it depends on these files as well.

Other things that can go wrong: some projects use different rules for constructing the config.h file. You should read the generated Makefile.in file for how, and duplicate into the bootstrapping makefile file. You may also need a rule for the aclocal.m4 file, and tell the configure rule it depends on it.

There is a template makefile installed in the /opt/aegis/share/config.example directory.

Now you can set the build command field of the project configuration file:

build_command =
	"make "
	"project=$project "
	"change=$change "
	"version=$version";

Aegis watches the eist status of the build command. Be aware that many build systems which use recursive make report false positives, because the exist status of the sub-make is often ignored by the top-level Makefile. This means that Aegis may think your project compiles when, in fact, it does not.

If, while trying to get it to build, you discover more derived files which should not be primary source files, simply use the aerm(1) command. The aeclean(1) command may come in handly, too.

Once this second change set builds, integrate it via the usual Aegis process.

7.3. Testing

If the project you are importing has tests, they are probably executed by saying

$ make check
lots of output
$

or something similar. Aegis expects each test to be in a separate shell script. Usually this is simple enough to arrange. See the chapter on Testing for some hints.

7.4. An Optimization

The first build in a new development directory can be quite time consuming. It is possible to short-ciruit this by using the pre-built object files in the baseline. To do this, use the following setting in the project configuration file:

development_directory_style =
{
    source_file_symlink = true;
    derived_file_copy = true;
    derived_at_start_only = true;
};

This causes Aegis to copy all of the derived file into your development directory at aedb time. This is usually much faster than compiling and linking all over again.

7.5. Signed-off-by

It is possible to get the Aegis process to automatically append Signed-off-by lines to the change description. Set the following field in the project configuration file:

signed_off_by = true;

Only open source projects should use this field. The OSDL definition of the Developer's Certificate of Origin 1.0 can be found at http://­www.osdl.org/­newsroom/­press_releases/­2004/­2004_05_24_dco.html and is defined to mean:

"By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it."

7.6. Importing the Next Upstream Tarball

If you are using Aegis to track your local changes, but the master sourecs are elsewhere, you will need to track upstream changes when they are released.

It is tempting to use the aetar(1) command, but it will not be able to detect derived files which have been added to the tarball. You will need to uppack the tarball and remove them manually.

Create a change set in the usual way, and aecd(1) into it. Copy the entire project into your change set, because you don't yet know what the tarball will want to change (and it will include unchanged files).

$ aecd
$ aecp .
$

(Yes, that dot is part of the command.) Now you can unpack the tarball. You need to strip off the leading directory somehow (most polite projects use a prefix). The author uses the tardy(1) command, like this:

$ zcat project-x.y.tar.gz | \
  tardy -rp project-x.y -now | \
  tar xf -
$

It pays to change that the tarball is the shape you expect before running this command.

At this point you have to once again remove all of the files which are in the tarball, but which are not primary source files, such as configure and Makefile.in and the like.

$ rm -f configure Makefile.in \
  config.h.in etc
$ rm -f aegis.log
$

It is useful if you place the rm(1) command in a shell script, and tell Aegis it is a source file, because you will have to do this every time.

Now you can have Aegis add any new files by using the follwoing command:

$ aenf .
$

(Yes, that dot is part of the command.) Note that if there are no new files, this command will give you an error, this is expected.

You will have to work out moved and removed files for yourself, and use the aemv(1) and aerm(1) commands.

At this point you should remove all the files which were present in the tarball but which dod not actually change from the change set. The follwoing command does this quickly and simply:

$ aecpu -unchanged
$

You change set now contains the minimum set of differences. Go ahead and complete it using the usual Aegis process.

7.7. Importing the Next Upstream Patch

In contrast to tarballs, patches tend to be far easier to cope with. In general, all that is necessary is to use the aepatch(1) command, something like this:

$ aepatch -receive -file \
  project-x.y.diff
$

which will create a change set, check-out only those file the patch alters, and copes with creates and removes automagically.

There are two problems with this method. The largest problem is patches whicg contains diff for derived fiels as well. This is unfortunately very common.

The simplest way of coping with this is to add the aepatch -trojan option, which will leave the change in the being developed state, where you can examine it and use the aenfu(1) command for any derived files it insisted on creating as primary source files.

The second problem is much simpler: if a patch only contains new files, Aegis can't work out how much of the leading path it should ignore on the filenames in the patch. You will need to use the aepatch -remove-prefix option in this case.

8. No Build Required

For some projects, particularly web sites and those written exclusively in interpreted languages, it may not be necessary to ever actually build your project.

For this kind of project you add the following line to the project configuration file:

build_command = "exit 0";

For a project configured in this way, the aede(1) and aeipass(1) commands will not check that a build has been performed.

8.1. Why This May Not Be Such A Good Idea

It isn't always desirable to configure a project this way, even when it may initialy appear to be a good idea.

Web sites:

You can use the build stage to check the HTML files against the relevant standards and DTDs. You can also check that all of you (internal) links are valid, and don't point to non-existant pages or anchors. Sometimes, if you have the space, you can resolve server side includes, to make it faster for Apache, by serving static pages.

Interpreted Languages:

A whole lot of simple errors, such as syntax errors, can be caught by a static check of the source files. For example, the perl -c option can syntax check your Perl files without executing them. See also the GNU awk -lint option, the Python built-in compile() function, and the php -l (lower case L) option. You can also check that all include files referenced actually exist.

Documentation:

Many systems allow documentation to be extracted from the source files, and turned into HTML or PDF files (e.g. Doxygen). This is a sensable thing to do at build time.



[18] See the Using Make section for how GNU Make may be used. It effectively combines both methods: keeping .d files and dynamically updating them. Because it combines both methods, it has some of the advantages and disadvantages of both.

[19] The version in use when writing this section was 1.5. All versions from 1.3 onwards are known to work with the recipes described here.