Tutorial
Introduction
KJam is a software build tool like make. KJam is designed to read in a user created jamfile, similar to the way make reads a makefile. It uses this to compute dependencies between targets. Most targets are either source files, such as c source files, or the generated files made by running system commands with the source files as input. KJam determines which targets exist in the file system, and what date they were last updated, and computes which targets need to be updated. These targets are then updated using defined update actions, consisting of shell scripts with operating system commands to update the targets.
The KJam Language
The KJam language consists of a series of statements each terminated with a semicolon. For example:
SOURCES = main.c file.c ;
There are four kinds of statements: (1) statements which set variables, (2) define a rule or action, (3) invoke a rule or action, and (4) flow of control statements. A rule is basically a KJam function. An action is an operating system command script which is called to update one or more targets. We will get to those later. First lets examine variables.
Setting Variables
Variables are set or modified using an assignment operator like this:
SOURCES = main.c file.c ;
The type of all variables in KJam is a list of strings. Variables which hold a single string are considered to be a list of strings with only one element. In the above statement, a variable called SOURCES has been created which holds a list with two elements, the string main.c and the string file.c.
In KJam, variable names and their values may include almost any character, including many characters which are also used as operators. For this reason it is necessary to put whitespace around most names. For example, in the above statement spaces are required around the operator = and before the final ;.
Had the statement above been written like this:
SOURCES=main.c file.c;
KJam would have interpreted this line as a the invocation of a rule called SOURCES=main.c being invoked with an argument file.c; and would have included the contents of the following line as further arguments to this rule invocation. The KJam parser works this way because very often it is necessary to create variables whose values include many special characters which are often found in file paths ( :;/\.* ) and in arguments to tools ( -=+ ). Not allowing these characters in values and variable names would require that many of these strings be expressed with quote marks. Maintainers of build systems are usually very engaged in managing the names of file targets, paths and command line arguments. The language is designed to make these things easy to manipulate at a minor cost to the flexibility of the language for writing statements.
KJam supports three variable assignment operators. This will overwrite the existing value of a variable:
SOURCE = main.c file.c ;
This will append the given value at the end of the existing value of the variable:
SOURCE += main.c file.c ;
This will create a variable with the given value if the variable does not already exist:
VALUE ?= default ;
You can use more than one variable name on the left hand side to set multiple variables at the same time. In the following case three variables are created, all of which are set to a hold a list of two strings.
VAR1 VAR2 VAR3 = value1 value2 ;
Literals
The value of a variable can be set to a literal, to the value of another variable or to a combination.
In KJam, for convenience literals don't have to be quoted, though they can be. If you have organized your jamfiles well, normally there will be one or more jamfiles full of rules and actions which you don't have to deal with very often, and the local jamfile which is full of literals containing just the names of your targets, important paths in your projects and sometimes additional command line options. Its nice when you have a file full of these literals not to have to put quotes around everything.
Every literal is interpreted as a string, even numbers. Unquoted literals may be composed of almost any character except whitespace or the " character. To create literals with these characters you must put quotes around it. For example here are several literals:
LIST = file.c 5 x=y "a b" "\"" "\n" ;
This list holds five elements. All of them are strings. The third value is the string x=y. The fourth value is the string a b. The fifth element is the value ". And the last is a string holding the newline character. Notice that the value of a quoted literal does not include the quote marks. Notice also that for literals expressed in quote marks, certain special characters can be expressed using c-like string syntax with escape codes for special characters.
KJam variable names are also strings. So it is possible to name a variable whatever you want by using quote marks. For example, this is legal:
" " = value ;
Variable Expansion
You can refer to the value of a variable by surrounding its name with $( and ). For reasons which will become clear, in KJam getting the value of a variable is called variable expansion. For example:
MSG = "This is a message" ;
print $(MSG) ;
The first line creates a variable holding a single string value. The second line invokes a built-in rule which prints that value. If a variable holds a list with more than one element, its expansion will be a list containing all of its elements. To refer to just one element in a variable with multiple elements use the subscript operator []:
SOURCES = file1.c file2.c file3.c ;
print $(SOURCES[1]) ;
This code would print file2.c. You can also use the subscript operator to get a range of elements from a list:
SOURCES = file1.c file2.c file3.c file4.c file5.c ;
print $(SOURCES[1-3]) ;
print $(SOURCES[2-]) ;
The second line would print file2.c file3.c file4.c. The third would print file3.c file4.c file5.c.
It is possible to use variable expansion on both the left hand or right hand side of a variable definition:
$(SOURCES) = file1.c file2.c file3.c ;
SRCS = $(SOURCES) ;
In this case the value of SRCS is set to hold a list of three strings: file1.c file2.c file3.c. Using variable expansion on the left hand side works also:
VARS = var1 var2 var3 ;
$(VARS) = value1 value2 ;
This sets three variables, var1 var2 and var3 to each hold a two string list, value1 value2.
The name of the variable given in an expansion may itself be an expansion. For example:
FILENAME = file ;
TYPE = FILE ;
SRCS = $($(TYPE)NAME).c ;
The value of SRCS is now set to file.c.
Variable Expansion Products
When variable expansions are combined with literals, with other expansions, the resulting expansion is the product of the elements. For example:
DIRS = dir1/ dir2/ ;
FILES = file1 file2 file3 ;
SRCS = $(DIRS)$(FILES) ;
The value of SRCS is a list with 6 elements: dir1/file1 dir1/file2 dir1/file3 dir2/file1 dir2/file2 dir2/file3. The product may have literals in it:
DIRS = dir1 dir2 ;
FILES = file1 file2 file3 ;
SRCS = ./$(DIRS)/$(FILES).c ;
Now the value of SRCS is a list with 6 elements: ./dir1/file1.c ./dir1/file2.c ./dir1/file3.c ./dir2/file1.c ./dir2/file2.c ./dir2/file3.c. If one of the expansions is an empty list then the resulting product is also an empty list:
// assume that DIRS is not defined
FILES = file1 file2 file3 ;
SRCS = ./$(DIRS)/$(FILES).c ;
Now the value of SRCS is a list with zero elements. It is also possible to explicitly set a variable to an empty list:
SRCS = ;
This is the same thing as undefining a variable. Variables may have some elements in the variable list set to a null string:
SRCS = file1.c "" file2.c ;
In this case the value of this variable is set to three strings. The second string is an empty string, but still holds its position. When expanding the product of two variables both of which contain empty strings, some of the combinations will be empty. New empty strings are automatically removed when generating products from lists with empty strings:
X = A "" B ;
Y = a "" b ;
PROD = $(X)$(Y) ;
The value of PROD is a list with 9 elements: Aa A Ab a b Ba B Bb.
Variable expansion also happens inside quoted strings the same way as when the variables are not quoted:
FILES = file1 file2 file3 ;
SRCS = "./dir/$(FILES).c";
The value of SRCS is now a list with the following three elements: ./dir1/file1.c ./dir1/file2.c ./dir1/file3.c.
Variable Expansion Modifiers
A variable expansion can also be written in the form $(NAME:MODIFIER). When applying a modifier to an expansion, the value returned may be changed in a number of ways. Each modifier consists of at least a single letter which determines its function. Here are some examples:
SRC = Dir1/File1.c ;
TEMP1 = $(SRCS:U) ; // return SRC in all upper case: e.g. DIR1/FILE1.C
TEMP2 = $(SRCS:L) ; // return SRC in all lower case: e.g. dir1/file1.c
TEMP3 = $(SRCS:D) ; // return just the directory part of the file name e.g. Dir1
TEMP4 = $(SRCS:B) ; // return just the base part of the file name e.g. File1
TEMP5 = $(SRCS:S) ; // return just the suffix part of the file name e.g. .c
TEMP6 = $(SRCS:H) ; // return the whole file name without the directory
// e.g. File1.c
SRC = /project/* ;
TEMP7 = $(SRCS:A) ; // returns all the directories in the file system which
// match this path expression.
// e.g. /project/src /project/include /project/bin etc.
SRC = Dir1/*.c ;
TEMP8 = $(SRCS:M) ; // returns all the files in the file system which match this
// path expression.
// e.g. Dir1/File1.c Dir1/File2.c Dir1/File3.c etc.
Some modifiers can take an argument, which is the letter followed by = and a string. For example:
TEMP6 = $(SRCS:D=newdir) ; // replace the directory name with the given string.
// e.g. newdir/File1.c
TEMP7 = $(SRCS:B=newfile) ; // replace the base part of the file name with the given
// string. e.g. Dir1/newfile.c
TEMP8 = $(SRCS:S=.o) ; // replace the suffix part of the file name with the
// given string. e.g. Dir1/File1.o
Multiple modifiers may be applied to a single expansion. In this case it may be required to quote any arguments so that the arguments don't run together with the following modifier:
TEMP9 = $(SRCS:D="newdir"S=".o"U); // return NEWDIR/FILE1.O
The modifiers are applied in the order that they are written in the modifier list. Putting the above modifiers in a different order produces different results:
TEMP10 = $(SRCS:UD="newdir"S=".o"); // return newdir/FILE1.o
It is possible to select some elements from a list using modifiers with regular expressions. For example:
FILE = file1.c file2.c file3.c file1.o file2.o file3.o file1.s file2.s file3.s ;
C_FILES = $(FILES:I=".c$"); // select just the c files.
SO_FILES = $(FILES:X=".c$"); // exclude only the c files.
CO_FILES = $(FILES:I=".c$"I=".o"); // include both c and o files.
CO_FILES = $(FILES:I=".c$"I=".o"X="^file1"); // include both c and o files, but
// not files that start with file1.
It is also possible with regular expression modifiers to replace parts of a matched expression. In regular expression syntax, parts of an expression delimited by () are called numbered subexpressions. Subexpression 0 is the entire expression. Expressions 1-9 are the first nine encountered () subexpressions. The modifier N supplies a match subexpression, and subsequent numbered modifiers either return or replace the part of the expression matched by that subexpression. For example:
TEXT = retired fragrance mortar ;
RTEXT1 = $(TEXT:N="r([^r]*)r"1); // return all the text between r's.
// e.g eti ag ta
RTEXT2 = $(TEXT:N="r([^r]*)r"0); // return all the text between r's,
// including the 'rs themselves.
// e.g retir ragr rtar
RTEXT1 = $(TEXT:N="r([^r]*)r"1="__"); // replace the text between r's with __.
// e.g r__red fr__rance mor__r
All variable expansion happens during the parsing phase so it is not possible to run system commands and assign their output to a KJam variable. In KJam external programs are only ever executed by command scripts during the updating phase.
Literal Expansion
Modifiers may also be applied to literals using the literal expansion syntax. As we saw above literals may be expressed directly as unadorned strings and as quoted strings. They can also be expressed by surrounding them with @( and ). For example the following three statements are equivalent:
TARGET = target.obj ;
TARGET = "target.obj" ;
TARGET = @(target.obj) ; // literal expansion syntax
When expressed using the third method, you can also apply expansion modifiers just like you can do with variable expansion. This is useful in cases where you find yourself assigning a literal string to a variable just so you can apply expansion modifiers to it. Here are several simple examples:
TARGET_DIR = @(target.obj:FD) ; // sets TARGET_DIR to the directory where
// target.obj is bound to
UPPERCASE = @(mIxEdCaSe:U) ; // sets UPPERCASE to MIXEDCASE
LIBS = @(libs/*:M) ; // sets LIBS to all the files found in the
// libs directory
Literal expansions and variable expansions may be nested in each other. So for example to find all the files with a given root file name but different three different file extensions:
EXTS = .txt .log .out ;
LOGFILES = @(mylogfile$(EXTS):F) ;
Suppose you have a list of source file name bases with the extensions already removed, and you want to get a list of any corresponding object files if they exist. You could do something like this:
SOURCES = source1 source2 source3 ;
EXISTING_OBJECTS = @($(SOURCES).obj:M) ;
As you can see combining literal and variable expansions can be very powerful.
Rules
KJam rules are functions which are interpreted and invoked during the parsing phase. They are used to set variables, to create dependencies between targets and to associate build actions with targets. Rules are defined like this:
rule RuleName ARG1 : ARG2 : ARG3
{
// rule statements go here, such as:
VAR1 = $(ARG1) ;
// flow of control statements
if ( $(VAR1) )
{
// and rule invocations
OtherRule $(ARG1) : $(ARG2) : $(ARG3) ;
}
}
A rule definition always starts with the keyword rule, and is followed by the name of the rule. This name is used later to invoke the rule. Rules definitions are then followed by a list of argument names. Rules may be defined with zero or more arguments. The arguments become variables defined during the invocation of that rule. This is then followed by the body of the rule which can include any KJam statement except the definition of another rule or action.
The first argument to a rule is interpreted by KJam as the target of that rule. This is normally the name of the files which this rule will build. The second argument is normally the name of the files which are used as the sources for the targets. Invoking a rule will cause KJam to create a dependency between the sources and the targets. This will be used later to compute which files need updating. Arguments after the second do not have a special meaning in KJam.
To invoke a rule, a statement begins with the rule name, and is followed by the arguments. Argument lists in KJam are always separated by :, and regular lists are simply series of token separated by whitespace. So a rule invocation with lists would look like this:
RuleName a b : c d e : f g h ;
Here the arguments to RuleName are three lists, "a b", "c d e", and "f g h".
Flow of Control Statements
Rule bodies may contain flow of control statements. Without flow of control statements, all statements in a rule would be executed in the order they appear. With flow of control statements, you have greater control over what gets executed and in what order. The most basic flow of control statement is if:
if ( $(ARG))
{
echo $(ARG);
}
This statement will print out the value of $(ARG) only if it is defined.
ARG = a b c ;
while( $(ARG) )
{
echo $(ARG);
ARG = $(ARG[2-]);
}
This statement will print out: "a b c", "b c", and "c".
Defining Actions
KJam actions are operating system command scripts. They are run during the updating phase to build targets. Actions are defined like this:
action ActionName ARG1 : ARG2
{%
cc $(ARG2) -o$(ARG1)
%}
An action definition always starts with the keyword action, and is followed by the name of the action. This name is used inside rule bodies to invoke the action.
Following this comes an argument list. In actions the first argument is the a list of targets which will be build built by this action. The second argument is a list of precursor files required to build it. More arguments are allowed, but have no special meaning to the KJam language.
Last comes the body of the action inside {% %} braces. The body of the action is command script which is passed to an operating system shell which then will run external applications to create the target files. Normally this shell is an sh-like shell which is built-in to KJam. By setting JAMSHELL however this shell may be a native operating system shell like bash, or Windows cmd.exe. Any expansions in the body of the script will be expanded before the script is passed to the shell.
Built-in Shell
The syntax of the built-in shell should be familiar to anyone who has written shell scripts. It supports piping output between processes with | and redirecting it to files using >, >> and <. For example:
ls | sort > sorted.txt
This would get a directory listing, send it to the sort command, and save the sorted output into a file called sorted.txt. The redirection also allows the error output to be redirected:
cc $(SRC) 1> output.txt 2> error.log
In this case the program 'cc' is run, and the regular output is redirected to output.txt, and the errors are redirected to error.log. The error stream can be redirected to the output stream and visa versa:
cc $(SRC) 2>&1 > output.txt
Here both stream are piped to output.txt. The > operator will create the output file, replacing any existing file by that name. The >> operator will concatenate the output to any existing file:
echo foo > tmp.txt
echo bar >> tmp.txt
The contents of tmp.txt would be foobar.
The input to a command chain can also be redirects by use of the < operator. In this case the input to the command grep is taken from the given file main.c:
echo foo > tmp.txt
echo bar >> tmp.txt
The contents of tmp.txt would be foobar.
The KJam built-in shell does not support any flow control statements found in shells, such as if, case, or while. It also does not support setting environment variables, job control or any common interactive features. The supported features should be enough to write most build actions.
If a certain build action requires more sophisticated shell features then an external shell must be used. To call out to an external shell, the JAMSHELL variable should be defined to point to the external shell to use. This JAMSHELL variable could be set in global scope to replace the built-in shell everywhere, or it can be set right before the action invocation to use a different shell just for the given action.
Variable Scope
By default all variables are defined in global scope. That is once a variable is defined, its value is available everywhere, in any rule or action until the value is overwritten by another variable definition. Variables may also be declared in local scope. To declare a variable in local scope you precede the definition with the keyword local:
local SOURCES = src1.c src2.c ;
This will define the variable for all statements inside the same block of statements. That is the variable remains defined until the next closing brace } is found. Variables in local scope will mask variables in global scope. So while a local variable is in scope, a global variable with the same name is not accessible. When the local variables goes out of scope, the global variable comes back into effect.
Variables may also be defined to be matched to a particular target such that they are in effect only while that target is part of the first argument to a rule or action. This is done with the on keyword:
HDRSCAN on src.c = $(HDRPATTERN) ;
Here, the variable HDRSCAN is set to $(HDRPATTERN) only while the file src.c is a target. While building a certain target, variables defined on that target will mask local or global variables with the same name.
Explicit Dependencies
In KJam most dependencies are expressed implicity by how rules are invoked. KJam defines two built-in rules which may be used to create explicit dependencies between targets. The rule depends creates a normal dependency between sources and targets:
depends target.o : source.c ;
This creates a dependency between target.o and source.c. When source.c is updated, KJam will determine that target.o needs to be updated as well.
include source.c : header.h ;
This rule creates an include dependency between source.c and header.h. This will cause KJam to determine that if header.h was updated that is should behave as if source.c was updated as well. Usually you will want to set up a build system such that the include dependencies are created automatically. This can be done with the built-in variable HDRSCAN and HDRRULE.
When $(HDRSCAN) is set on a target, if that target is a file, KJam will scan that file for inclusion statements. The value of $(HDRSCAN) should be set to a regular expression which matches inclusion lines in the given source file. The pattern should match the entire inclusion statement. The pattern should include a single () grouping which indicates the filename being included.
Whenever a match is found by KJam using $(HDRSCAN), KJam will invoke a the rule named by the $(HDRRULE) variable. $(HDRSCAN) and $(HDRRULE) must have the same number of elements. If these variables hold more than one element, then each file will be scanned by multiple patterns, and the approriate rule will be called.
Comments
KJam interprets every line as a KJam statement, unless the line is a comment. KJam support c-style /**/ multi-line comments, c++ like // comments, and makefile like # comments:
/* this is a comment and is not interpreted by KJam */
// so is this
# and so is this.
Making a basic Jamfile
Putting together the things we have learned we can make a basic jamfile:
CC ?= cl.exe /nologo ;
LIBEXE ?= lib.exe /nologo ;
HDRPATTERN = "^[\t ]*#[\t ]*include[\t ]*[\"]([^\"]*)[\"]" ;
HDRRULE = HeaderRule ;
rule Library TARGET : SOURCES : LIBS
{
HDRSCAN on $(SOURCES) = $(HDRPATTERN) ;
LOCATE on $(TARGET) = $(TARGET_DIR) ;
local C_SOURCES = $(SOURCES:I="\.(c|cpp)$") ;
local OBJECTS = $(C_SOURCES:S=.obj) ;
depends $(TARGET) : $(OBJECTS) $(LIBS) ;
for SRC_FILE in $(C_SOURCES)
Object $(SRC_FILE:S=.obj) : $(SRC_FILE) ;
LinkLibrary $(TARGET) : $(OBJECTS) $(LIBS) ;
}
rule Object TARGET : SOURCE
{
CompileObject $(TARGET) : $(SOURCE) ;
}
rule HeaderRule INCLUDER : INCLUDEDS
{
includes $(INCLUDER) : $(INCLUDEDS) ;
HDRSCAN on $(INCLUDEDS) = $(HDRPATTERN) ;
}
action CompileObject TARGET : SOURCE
{%
$(CC) $(CCOPTS) /c $(SOURCE) /o $(TARGET)
%}
action LinkLibrary TARGET : SOURCES
{%
$(LIBEXE) $(SOURCES) /out:$(TARGET:S=.lib)
%}
LIB1_SOURCES = src1.c src2.c src3.c ;
LIB2_SOURCES = src4.c src5.c src6.c ;
Library lib1.lib : $(LIB1_SOURCES) : ;
Library lib2.lib : $(LIB2_SOURCES) : lib1.lib ;
In the above example all of the code is in a single jamfile. Normally you want to separate the rules from the main rule invocation files where the top users source files and desired target files are named. To do this you use the include directive.
The main jamfile would look like this:
include "jambase" ;
LIB1_SOURCES = src1.c src2.c src3.c ;
LIB2_SOURCES = src4.c src5.c src6.c ;
Library lib1.lib : $(LIB1_SOURCES) : ;
Library lib2.lib : $(LIB2_SOURCES) : lib1.lib ;
The jambase file would include all the rules and actions. For most users they would never have to modify the jambase file. They would just add targets and sources to the jamfile. If you have written your rules well the user jamfiles will behave as if there are typed build targets. For example:
LIB1_SOURCES = src1.c src2.c src3.c ;
LIB2_SOURCES = src4.c src5.c src6.c ;
PROG_SOURCES = prog1.c ;
DATA_SOURCES = data.dat data2.dat ;
Library lib1.lib : $(LIB1_SOURCES) : ;
Library lib2.lib : $(LIB2_SOURCES) : lib1.lib ;
Dll lib1.dll : $(LIB1_SOURCES) : ;
Dll lib2.dll : $(LIB2_SOURCES) : lib1.dll ;
Executable program.exe : $(PROG_SOURCES) : lib1.lib lib2.lib ;
DataFile data.zip : $(DATA_SOURCES) ;
Running KJam in Network Mode
Normally KJam is simply invoked by running the KJam binary. In this mode KJam runs on the local machine and spawns build commands on the local machine. KJam can be run in network mode. In this mode a KJam server is started on build capable machines all over a local network. The servers listen for build instructions on a common port. Then a KJam is run in network build client mode. Just as in normal non network mode KJam will parse a jamfile, bind targets to the local file system, build a dependency graph and determine which targets need updating. This is all done locally just as in non-network mode. It will then issue the build commands to the KJam servers on the network.
To do this, first share the build directory on the network. Then on each machine on the network run KJam in server mode. The simplest way to do this is to just run it on the command line with the -z option. KJam can also be set up to run as a linux daemon or as a window service. All of the machines should be capable of running whatever programs are in the build actions. So for example, if a c-compiler is required for some of the actions, it must be installed on each server machine. All the server machines should also be able to see the shared drive.
So for example assume a network with three machines, M1, M2 and M3. Suppose on M1 the project directory is on c:/project. On M1 we share c:/project ss //M1/project. Then from M2 and M3 asking for a directory listing of //M1/project should work. Now on M1 M2 and M3 launch KJam -z. This will create a peer network of KJam servers. That is all the servers will automatically find each other on the network and organize themselves into a load balanced peer network.
On the client machine make //M1/project the working directory. This will not work if the current directory is set as c:/project, because this path is not visible on the network, and so the filename bindings will not work. From this directory start KJam with the -y option. This should build your project as normal, except that the build commands will be distributed onto all three machines.
Be aware that some programs when run on a shared directory my launch a little more slowly than they would if launched on a local directory. This means that for some programs which have very short execution times there may not be any time savings from running the build distributed on the network. For build steps which take a significant amount of time, network building can be sped up by a tremendous amount.
It is possible to control on which machines certain commands run. So for example suppose on a network you have 5 machines. Four of them, M1 M2 M3 and M4 have a compiler installed can run compiler build steps. And two of them, M4 and M5 have are connected to a common database and so can build data. In this case you would launch the servers on M1 M2 M3 with the flags "kjam -z -x compiler_system", M5 would have a server launched as "kjam -z -x data_system" and M4 would be started with "kjam -z -x compiler_system -x data_system". In the jamfile, the build actions for compiling would have the modifier "on compiler_system", and build actions for data would have the modifier "on data_system":
action on compiler_system CompileObject TARGET : SOURCE
{%
$(CC) $(SOURCE) -o $(TARGET)
%}
action on data_system MakeData TARGET : SOURCE
{%
makeData $(SOURCE) -o $(TARGET)
%}
Once this is set up, when build jobs are sent to the KJam build network, build commands are only sent to the appropriate systems.