KJam Reference
Usage
KJam [ -a ] [ -b ] [ -C ] [ -e ] [ -g ] [ -n ] [ -q ] [ -v ] [ -y ] [ -X ] [ -S ]
[ -c path ]
[ -d debug ]
[ -f inputFile ]
[ -h indent ]
[ -j jobs ]
[ -o outputFile ]
[ -s var=value ]
[ -t target ]
[ -z [ port ]]
[ -l power ]
[ -x type ]
[ -p subnet ]
[ -i [ port ]]
[ -P [ file ]]
[ -L [ file ]]
[ target ... ]
Description
KJam is a program construction tool, like make. KJam's design is based on Jam.
KJam recursively builds target files from source files, using dependency information and updating actions expressed in a user-provided jamfile, which is written in KJam's own interpreted language.
Command Line Options
If target is provided on the command line, KJam builds target; otherwise KJam builds the target 'all'.
Build Options: These options are available when KJam is running as a build client.
| -a | Build all targets, even if they are up-to-date. |
| -b | Build targets separately, even when they can be built together. |
| -f file | Read file instead of jamfile. |
| -g | Build from newest sources first. |
| -j n | Run up to n shell commands concurrently. This option is ignored in network client mode. |
| -n | Don't actually execute the updating actions. |
| -o file | Write the updating actions to file file. |
| -q | Stop as soon as a target fails to build. |
| -t target | Rebuild target, even if it is up-to-date. |
| -s x=y | Set variable x=y, overriding environment. |
| -S | Build command line targets in the order given on the command line. |
Network Options: These options control distributed nework building for servers and clients
| -z port | Run as network server. Listen for job requests on the network and run them. If optional port is given, the server will listen on the given port instead of on the default KJam server port (2297). |
| -y | Run as network client. Spawn jobs on known KJam servers. |
| -l power | In server mode, specify power of the server system. By default all servers are given a power rating of 1. Systems with greater (or smaller) computational resources can be given a larger ( or smaller ) rating to help the load balancer assign build jobs appropriately |
| -x type | In server mode, specify a server type, so that KJam build clients can assign jobs to appropriate servers. |
| -p subnet | Ip address and port number of subnet to scan for build servers. e.g. 127.0.0.* would scan the local subnet for servers. |
| -i subnet | Get information about the peer network from the server at this address. |
Common Options: These options apply when running as both client and server.
| -c path | Run KJam in the given directory. | ||
| -C | Do file system scanning and target binding in a case insensitive manner. | ||
| -d n | Display debugging information. If a number is given, then all flags with that number or lower are turned on. If a token is given, then only the given flag is turned on. | ||
| 0 | show no diagnostics | ||
| make | or 1 | show actions when executed (default) | |
| phases | or 2 | show start of phases | |
| fate | or 3 | show progress of bindAndScan | |
| build | or 4 | show progress of build | |
| compile | or 5 | show rule invocations | |
| header | or 6 | show result of header scan | |
| search | or 7 | show attempts at binding | |
| varSet | or 8 | show variable settings | |
| varGet | or 9 | show variable fetches | |
| varExp | or 10 | show variable expansions | |
| if | or 11 | show 'if' calculations | |
| scan | or 12 | show scanner tokens | |
| parse | or 13 | show parser rule matching | |
| makeQ | or 14 | show even quiet actions | |
| exec | or 15 | show text of actions | |
| thread | or 16 | show operation of threads | |
| targets | or 17 | show targets in detail | |
| actions | or 17 | show actions in detail | |
| depends | or 18 | show dependency graph | |
| network | or 19 | show network processing | |
| path | or 19 | show file path components | |
| regexp | or 19 | show regular expression parsing | |
| NETA | print out diagnostics for NetAgent library | ||
| NETAERROR | print out error diagnostics for NetAgent library | ||
| NETACONN | print out detail about connection and disconnection | ||
| PING | print out ping information | ||
| DL_LEX | print out the operation of the dl lexer | ||
| DL_YACC | print out the operation of the dl parser | ||
| -e | Display elapsed time and the average number of concurrent threads. | ||
| -h num | Indents the output by the given number of 3 character tabs. Useful for when kjam calls itself or is called by another process. | ||
| -v | Print the version of jam and exit. | ||
| -P filenName | Replace any currently built-in jamfiles with the given file. Multiple -P options may be given to support embedding multi-file build systems. | ||
| -X | Extract all the built-in jamfiles. | ||
| -D | Do not automatically include jambase in every jamfile | ||
| -L fileName | If given an argument, reads that license file. With no argument displays license information | ||
Operation
KJam has four phases of operation: start-up, parsing, binding, and updating.
Start-up
Upon start-up, KJam imports environment variable settings into KJam variables. Environment variables are split at blanks with each word becoming an element in the variable's list of values. Environment variables whose names end in PATH are split at $(SPLITPATH) characters (e.g., ":" for Unix).
To set a variable's value on the command line, overriding the variable's environment value, use the -s option. To see variable assignments made during KJam's execution, use the -d varSet option.
Parsing
In the parsing phase, KJam reads and executes the user provided jamfile. It is written in the KJam language. The purpose of the jamfile is to name built target and source files, construct the dependency graph among them, and associate build actions with targets.
Binding
After parsing, KJam recursively descends the dependency graph and binds every file target with a location in the file system.
Any string value in KJam can represent a target, and it does so if the depends or includes rules make it part of the dependency graph. Build targets are files to be updated. Source targets are the files used in updating build targets. Build targets and source targets are collectively referred to as file targets, and frequently build targets are source targets for other build targets. Pseudo-targets are symbols which represent dependencies on other targets, but which are not themselves associated with any real file.
A file target's identifier is generally the file's name, which can have a full path, a path relative to the directory of jam's invocation, or simply local (no directory). Most often it is the last case, and the actual file path is bound using the $(SEARCH) and $(LOCATE) variables.
The use of $(SEARCH) and $(LOCATE) allows KJam to separate the location of files from their names, so that jamfiles can refer to files locally (i.e. relative to the jamfile's directory), yet still be usable when jam is invoked from a distant directory.
After binding each target, KJam determines whether the target needs updating, and if so marks the target for the updating phase. A target is normally so marked if it is missing, it is older than any of its sources. This behavior can be modified by the application of special built-in rules.
During the binding phase, KJam also performs header file scanning, where it looks inside source files for the implicit dependencies on other files caused by C's #include syntax. This is controlled by the special variables $(HDRSCAN) and $(HDRRULE). The result of the scan is formed into a rule invocation, with the scanned file as the target and the found included file names as the sources. Note that this is the only case where rules are invoked outside the parsing phase.
Updating
After binding, KJam again recursively descends the dependency graph, this time executing the update actions for each target marked for update during the binding phase. If a target's updating actions fail, then all other targets which depend on that target are skipped.
The -j flag instructs jam to build more than one target at a time. On multiprocessor systems, or in when running in network mode this can result in a significant performance improvement. If there are multiple actions on a single target, they are run sequentially. The -g flag reorders builds so that targets with newest sources are built first. Normally, they are built in the order of appearance in the jamfile.
Language
Overview
KJam has an interpreted, procedural language with a few select features to effect program construction. Statements in KJam are rule (procedure) definitions, rule invocations, updating action definitions, flow-of-control structures, or variable assignments.
Variables
All KJam variables are one-dimensional lists of arbitrary strings. They arise as literal (whitespace-separated) tokens in the jamfiles, as the result of variable expansion, or as the return value from a rule invocation.
Rules
The basic KJam language entity is called a rule. A rule is simply a procedure definition, with a body of KJam statements to be run when the rule is invoked.
Rules take up to any number of named arguments and can have a return value (a single list). A rule's return value can be expanded in a list by enclosing the rule invocation with [ ].
The syntax of rule invocation makes it possible to write jamfiles that look a bit like makefiles.
Actions
A rule may have updating actions associated with it. Actions also take any number of named arguments. For purposes of determining dependencies, the first argument is interpreted by KJam as a list of targets, and the second argument as a list of sources. Actions also contain a script consisting of operating system shell commands to execute when updating the built targets of the rule.
When a rule with updating actions is invoked, those actions are added to those associated with its built targets before the rule's procedure is run. Later, to build the targets in the updating phase, the actions are passed to the OS command shell, with the input arguments expanded into the text of the script.
Statements
KJam's language has the following statements:
rulename arg1 : arg2 : ... : argN ;
Invoke a rule or action. The named rule or action is invoked with values in arg1 through argN. These arguments will get expanded into the body of the rule when it is evaluated, or into the script body of the action when it is executed.
rulename undergoes variable expansion. If the resulting list is more than one value, each rule or action is invoked with the same arguments, and the result of the invocation is the concatenation of all the results.
action [ modifiers ] rulename [ : args ] [ on typelist ] {% commands %}
Define a rule's updating actions, replacing any previous definition. args defines the names of the actions arguments. When an action is executed the arguments passed to the action in the action invocation are substituted into the command script. commands specifies the command script. This script is passed into the operating system shell and executed.
The following action modifiers are understood:
| existing | Includes only sources whose bound files are currently existing. Useful for the implementation of clean rules. |
| ignore | The return status of the commands is ignored. Normally if any of the commands return an error, the target will be assumed to have failed to build, even if a file is generated. Actions with the ignore modifier will behave as if the commands succeeded even if an error code is returned by the command script. |
| fail_expected | The return status of the commands is inverted. Actions with the this modifier will behave as if an action succeeded if it failed, and visa versa. |
| piecemeal [num] | Commands are repeatedly invoked with a subset of the sources small enough to fit in the command buffer on the current OS. If the optional number argument is given, the subset will never be larger than this number. |
| quietly | The action is not echoed to the standard output. |
| together | The sources from multiple invocations of the same action on the same built target are concatenated and if possible run as a single action. |
| updated | Only sources whose associated targets are marked for updating are included. Actions with this keyword must have the same number of sources and targets, and the sources and targets must match. |
| retry [num] | Normally kjam only tries to run each action once. If the action fails, kjam reports the error and terminates. The retry modifier tells kjam to try the action more than once, until the action succeeds. retry by itself will make kjam retry an action an unlimited number of times. retry plus a number will tell kjam to retry the action that number of times. retry 1 is requivalent to not setting retry at all. |
| timeout [num] | Normally all actions are given an unlimited amount of time to execute. This can be a problem when an action hangs. The timeout action modifier tells KJam to terminate an action if it takes longer than a certain amount of time. Timeout followed by a numerical argument tells kjam to allow the action to run for up to that number of seconds. Timeout without a number is equivalent to timeout 60. Timeouts shorter than about 5 seconds are rarely useful because KJam runs in parallel. Even a process which is expected to return instantly my take a few seconds to run if several other processes have started at the same time. |
| thread [num] | Limits the number of simultaneous threads which will be spawned to process actions of this type. For example thread 1 will only will process this action with a single thread, such that multiple calls to it will never occur in parallel. When this keyword is not used, KJam will spawn as many threads as neccesary until it reaches KJam's overall thread limit ( set with the -j command line option. This option is useful in cases where a particular action cannot be run in parallel ( because it modifies a global resource, or uses a tool which is not thread safe ), or where there are limits to the performance benefits for running it in more than a certain number of threads ( such as actions which interact with limited hardware resources ). |
In addition actions may have the "on" modifier which is followed by a list of server types. The "on" modifier is used when KJam is running in client-server network mode. In network mode, KJam servers may be given one or more types corresponding to the available system resources. For example if a server is running on a machine configured as a build server that server may advertize itself as type "build". A server configured as a rendering server or a database server may advertize as "render" or "database". Actions with "on" modifiers will only be spawned by the load balancer on on KJam servers of the given type. This allows distributed build systems which will only run certain commands on the appropriate system types.
break
Breaks out of the closest enclosing for or while loop.
continue
Jumps to the end of the closest enclosing for or while loop.
export varnames
Causes the named jam variables to be exported to any spawned shells. When a new command shell is spawned all exported jam variables are copied to the shell's environment variables.
for ( var in list ) { statements }
Executes statements for each element in list, setting the variable var to the current element in the list.
if ( cond ) { statements } [ else { statements } ]
Executes the given statements if the conditional expression cond is true. If the else clause is present the those statements will be run if the conditional expression is false.
The expression can be one of:
| a | true if any a element is a non-zero-length string |
| a = b | list a matches list b string-for-string |
| a != b | list a does not match list b |
| a < b | a[i] string is less than b[i] string, where i is first mismatched element in lists a and b |
| a <= b | every a string is less than or equal to its b counterpart |
| a > b | a[i] string is greater than b[i] string, where i is first mismatched element |
| a >= b | every a string is greater than or equal to its b counterpart |
| a in b | true if a is a subset of b. That is if all elements of a can be found in b, or if a has no elements |
| a intersects b | true if a intersects b. That is if a and b have at least one element in common. |
| ! cond | condition not true |
| cond && cond | true if both conditions are true, false otherwise. |
| cond || cond | true if either condition, or both are true. |
| ( cond ) | precedence grouping |
include file ;
Causes jam to read the named file. The file is bound like a regular target (see Binding above) but unlike a regular target the include file cannot be built. If the include file cannot be found an error is generated. Marking an include file target with the nocare rule makes it optional.
The include file is inserted into the input stream during the parsing phase. The primary input file and all the included file(s) are treated as a single file. That is, KJam infers no scope boundaries from included files.
[local] varnames [ on targets ][ ( = | += | -= | ?= ) values ] ;
Defines a variable. Variables associate a name with a list of zero or more string elements. An undefined variable is indistinguishable from a variable with an empty list. A defined variable may have one more elements which are null strings. The value of a variable may be referenced in KJam statements and in command scripts as $(variable).
varname is the name of the new variable. The value of the new variable is the list of strings specified by values. Variables may be global, local or target specific. A global variable it will expand to the given value everywhere until it is redefined with another global variable definition, or unless it is masked by a local or target specific variable. If a variable is defined locally, it will have that value until execution exits the nearest containing { }, or unless it is redefined locally within that block, or masked by a target specific variables. For target specific variables, the variable takes on the given value only during the target's binding, header file scanning, updating and during the evaluation of "on target" statement blocks.
The simplest variable definition, without additional keywords is global:
variable = values ;
Local variable definitions are specified by using the keyword local:
local variable = values ;
Target specific variables are defined by using the keyword on:
variable on target = values ;
The keyword on and local may not be combined in a single variable definition.
The variable definition uses one of three assignment operators:
| = | Creates a new variable, or overwrites and existing variable. |
| += | Creates a new variable, or adds new elements onto the end of the list of an existing variable. |
| -= | Removes any elements of the variable on the right hand side from the variable on the left, if it exists. |
| ?= | Creates a new variable if one does not already exist, but will not overwrite an existing variable. |
Multiple variable names may be set simultaneously by putting multiple variable names on the left hand side of the assignment operator. In the following case three variables are set to the same two string value:
var1 var2 var3 = value1 value2 ;
Variable names on the left hand side are subject to variable expansion before assignment. So it is possible to use the value of a variable as the name of a variable for assignment. The following case does the same thing as the one above:
VARNAME = var1 var2 var3 ;
$(VARNAME) = value1 value2 ;
on target { statements } ;
Run statement under the influence of target's target-specific variables. These variables become local copies during statement's run, but they may be updated as target-specific variables using the usual "variable on targets =" syntax.
return values ;
Within a rule body, the return statement sets the return value for an invocation of the rule and terminates the rule's execution.
rule rulename [ : args ] { statements }
Define a rule's procedure, replacing any previous definition. args is a series of argument names. When the rule is invoked, the arguments passed during invocation in are matched with the argument names as defined in the rule definition, and are set as local variables. These local variables may be referenced in variable expansion during rule invocation.
switch ( value )
{
case pattern1 : statements ;
case pattern2 : statements ;
...
default : statements ;
}
The switch statement executes zero or one of the enclosed statements, depending on which, if any, is the first case whose pattern matches value. The patterns can be any standard regular expression. The special default label will match any input value.
while ( cond ) { statements }
Repeatedly execute statements while cond remains true upon entry. (See the description of cond expression syntax under if ).
Variable Expansion
During parsing, KJam performs variable expansion on each token that is not a keyword or rule name. Such tokens with embedded variable references are replaced with zero or more tokens. Variable references are of the form $(v) or $(v:m), where v is the variable name, and m are optional modifiers.
Variable expansion in an action's command script is similar to variable expansion in statements, except that the action string is tokenized at whitespace regardless of quoting.
The result of a token after variable expansion is the product of the components of the token, where each component is a literal substring or a list substituting a variable reference. For example:
$(X) -> a b c
$(X) -> ta tb tc
$(X)z -> az bz cz
$(X)-$(X) -> a-a a-b a-c b-a b-b b-c c-a c-b c-c
The variable name and modifiers can themselves contain a variable reference, and this will be part of the product as well:
$(X) -> a b c
$(Y) -> 1 2
$(Z) -> X Y
$($(Z)) -> a b c 1 2
Because of this product expansion, if any variable reference in a token is undefined, the result of the expansion is an empty list. If any variable element is a null string, the result propagates the non-null elements:
$(X) -> a ""
$(Y) -> "" 1
$(Z) ->
_$(X)$(Y)_ -> _a_ _a1_ __ _1_
_$(X)$(Z)_ ->
Individual elements of a variable, or a subsection of the elements of a variable can be referred to using []:
| [n] | Select element number n (starting at 0). If the variable contains fewer than n elements, the result is a zero-element list. |
| [n-m] | Select elements number n through m. |
| [n-] | Select elements number n through the last. |
| [x,x] | Multiple elements and ranges can be selected by separating selection elements with , |
$(X) -> a b c d e f g
$(X[0]) -> a
$(X[2]) -> c
$(X[2-4]) -> c d e
$(X[3-]) -> d e f g
$(X[1,3,5-]) -> b d f g
When getting the value of a variable, the returned value can be modified by referring to the variable through the use of modifiers. Many modifiers consist of a single letter which normally selects a subset of each value. Some modifiers are followed by =, and an argument. Many modifiers only make sense if the variable string can be interpreted as a file name. A variable reference may include any number of modifiers.
The modifiers are:
| L | Replace uppercase characters with lowercase. |
| U | Replace lowercase characters with uppercase. |
| K | Remove duplicate elements. |
| V | Replace any backslashed character code with their proper replacement. e.g. \n will be replaced with the newline character etc. |
| C | Replace non-printable characters with their backslashed equivalents. e.g. newline will be replaced with \n etc. |
| P | Return parent directory. |
| Q | Given a relative path, returns the reverse path, such that if a script was to cd to the given path, a cd to the reverse path would return the script back to the original directory. For example the path one/two/../three/./four would have the reverse ../../... Some paths, such as global rooted paths, or relative paths which start with .. have no reverse. Paths with no reverse return an empty variable. All reverse paths consists just .. elements. |
| R | Normally zero length elements are removed. If R is given, these elements are preserved. |
| A | If the current variable is a path with wildcard charaters, expands the variable to the full set of directories in the file system which match the expression |
| M | If the current variable is a path with wildcard charaters, expands the variable to the full set of files in the file system which match the expression |
| D[=dir] | Select directory path. If an argument is given, replace the directory with the given string. |
| B[=base] | Return the filename base. If an argument is given, replace the base with the given string. |
| H[=file] | Return the filename without the directory. If an argument is given, filename with the given string. |
| D[=dir] | Select directory path. If an argument is given, replace the directory with the given string. |
| G[=grist] | Return the target grist. If an argument is given, replace the grist with the given string. |
| F[=file] | If the string is a target, return the bound name of the element, otherwise return the element unmodified. If an argument is given then the target will be bound explicitly to the given file name. If the file does not exist, then the binding is set to 'missing'. This will override any other binding that would normally be done automatically using the $(SEARCH) or $(LOCATE) variables. WARNING: the :F modifier causes a target to get bound to a file or directory as soon as it is evaluated. This may cause the SEARCH and LOCATE variables which may be evaluated later to be ignored. |
| X=regexp | Exclude elements which match the given regular expression. |
| I=regexp | Include only elements which match the given regular expression. |
| T | Return the given list in alphabetical order. |
| J | Join all the elements into a single string. |
| O[=split] | Split a string along boundaries defined by split. If none is given space is assumed. If split is a string with more than one character, multiple characters will be used as the split point. |
| E=value | If the returned list would be empty, return value. |
The following modifiers make it possible to select or replace parts of any element which match any part of a given regular expression, by using a combination of the N modifier a numbered modifier:
| N=regexp | Specifies the regular expression to use as the match to select elements from. |
| 0[=value] | With no argument returns the part or the element which matched the expression. With an argument this returns the element with the part matched by the expression replaced by value. |
| 1[=value] 3[=value] 4[=value] 5[=value] 6[=value] 7[=value] 8[=value] 9[=value] | If the regular expression has () groups, these modifiers refer to these groups. With no argument this will select only the part which matched the given group. With an argument this modifier will replace the given group. |
Some modifiers accept or require arguments. To give a modifier an argument follow it with = and the text of the argument like so:
SOURCES = foo.c bar.c ;
OBJECTS = $(SOURCES:S=.o) ;
# $(OBJECTS expands to foo.o bar.o
All text following the = is considered to be part of the argument until the expansions closing ). To express arguments with special characters, or to have multiple modifiers with arguments they must be quoted with " or ' like so:
TEXT = foo bar goo ;
TEXT2 = $(TEXT:N="oo"0="xx") ;
# $(TEXT2) expands to fxx bar gxx
TEXT = "foo=" "bar=" "goo=" l
TEXT2 = $(TEXT:N='='0='+') ;
# $(TEXT2) expands to foo+ bar+ goo+
Modifier arguments may themselves include variable expansions with their own modifiers. When a modifier argument expands to more than one element, only the first element will be used as the modifier argument. For example:
EXT = bmp png jpg ;
SOURCE_IMAGES = pony.mdl horse.mdl monkey.mdl ;
IMAGE_FILES = $(SOURCE_IMAGES:BS="$(EXT)") ;
# $(IMAGE_FILES) expands to pony.bmp horse.bmp monkey.bmp
Normal variable expansion happens before the command shell is spawned. In network mode this happens on the client. It is possible to cause variable
expansion to happen after the shell has spawned and during shell script execution, using server variable expansion syntax: $!(VAR)
In network mode, when variables are expanded this way, the variable settings used will be those on the server. This is also the way to expand using the environment variables as they are set in the spawned command shell. For example, to set an environment variable and then echo out its value:
{%
shellvar = foo
echo $!(shellvar)
%}
Literal Expansion
In the same way that modifiers can be applied to variables they can also be applied to literals. You can specify a literal either by using its name directly, in quotation marks, or surrounded by @( and ). For example the following three lines are equivalent:
LITERAL = string ;
LITERAL = "string" ;
LITERAL = @(string) ;
Using the third form allows for the use of modifiers. For example to uppercase a string literal:
LITERAL = @(string:U) ;
To get the directory where a certain literal file can be found:
MYFILE_DIR = @(myfile.txt:FD) ;
Variable expansions and literal expansions can be nested in each other. By combining literal and variable expansions with modifiers extremely powerful expressions can be specified. For example to find all the directories where log files with a certain name and different file extensions can be found:
LOGFILE_EXT = .log .txt .out ;
MYFILE_DIRS = @(myfile$(LOGFILE_EXT):FD) ;
Like variable expansion, literal expansion may be done on the server using @!(. In most cases this functions exactly like normal literal expansion and is only useful in combination with variable server expansion, that is with $!(.
Grist
Grist is used to make targets which would otherwise have the same file name unique. It is sometimes the case that there will be two different files which have the same name but exist in different directories. Sometimes this can be solved by using part of the directory name as part of the target name. Sometimes this is not practical. KJam allows target names to be prefixed with a "grist" string which is not used by Jam for binding, but is used in order to distinguish targets for the purposes of updating or computing dependencies. The grist string is a prefix on a target surrounded by < > :TESTS = <project1>test.data <project2>test.data ;
Built-in Rules
KJam has 11 built-in rules, all of which are pure procedure rules without updating actions. They are in three groups. The first builds the dependency graph. The second modifies it, and the third are utility rules.
Dependency Building: The following rules are used to explicitly add links to the dependency graph.
depends targets1 : targets2 ;
include targets1 : targets2 ;
depends foo.o : foo.c ;
includes foo.c : foo.h ;
"foo.o" depends on "foo.c" and "foo.h" in this example.
cleardep targets ;
clearinc targets ;
Modifying Binding: The six rules always, leaves, notfile, noupdate, and temporary modify the dependency graph so that KJam treats the targets differently during its target binding phase. Normally, KJam updates a target if it is missing, if its file system modification time is older than any of its dependencies (recursively), or if any of its dependencies are being updated. This basic behavior can be changed by invoking the following rules:
always targets ;
leaves targets ;
nocare targets ;
notfile targets ;
noupdate targets ;
temporary targets ;
Utility Rules: The remaining rules are utility rules.
echo args ;
print args ;
exit args ;
Built-in Variables
KJam has several special variables which change the behavior of the program.
SEARCH and LOCATE Variables
These two variables control the binding of file target names to locations in the file system. $(SEARCH) is used to find existing sources while $(LOCATE) is used to fix the location for built targets.
Rooted (absolute path) file targets are bound as is. Unrooted file target names are also normally bound as is, and thus relative to the current directory, but the settings of $(LOCATE) and $(SEARCH) alter this.
The $(SEARCH) path can be defined using wildcard characters to match multiple directories:
$(SEARCH) = obj/*/debug ;
HDRSCAN and HDRRULE Variables
These two variables control header file scanning. By using $(HDRSCAN) and $(HDRRULE) it is possible to eliminate the need for an explicit dependency making step, or the use of external dependency building tools such as makedepend.
When $(HDRSCAN) is set "on" a target, if that target is a file, KJam will scan that file for inclusion statements. The value of $(HDRSCAN) should be set to a regular expression which matches inclusion lines in the given source file. The pattern should match the entire inclusion statement. The pattern should include a single () grouping which indicates the filename being included.
Whenever a match is found by KJam using $(HDRSCAN), KJam will invoke a the rule named by the $(HDRRULE) variable. Typically this allows the user to create a rule to recursively scan files for inclusion and to build the dependency graph.
Both $(HDRSCAN) and $(HDRRULE) must be set for header file scanning to take place, and they should be set target-specific and not globally. If they were set globally, all files, including executables and libraries, would be scanned for header file include statements. Careful use of these variables allows for different $(HDRSCAN) patterns to be used for different source file types.
The $(HDRSCAN) and $(HDRRULE) variables must have the same number of elements. Normally, if there is only one pattern to scan for a given file type, then both variables will be set to lists with a single element. But this it not requires. You can set these variables to lists with multiple elements. This will cause the header scanning to use multiple regular expressions, and to call a different rule for each pattern. This can be useful for languages with more than one type of inclusion directive, and where the different types of inclusion directives must be treated differently.
The scanning for header file inclusions is not exact. The scanning mechanism errs on the side of inclusion. That is, it is more likely to return filenames that are not actually used by the compiler than to miss include files because it can't tell if #include lines are inside #ifdefs or other conditional logic. It is normally a good idea to apply nocare to each header file found during scanning so that if the file isn't found it doesn't cause the compilation to fail.
Scanning for regular expressions only works where the included file name is literally in the source file. It can't handle languages that allow including files using variable names (as the KJam language itself does).
JAM_SHELL Variable
By default KJam uses its own internal sh-like shell. This allows KJam command scripts to run in a common execution environment no matter what shells are installed on the target platform. The built-in shell may not be appropriate for all applications. To call out to a specific operating system shell, a jamfile may set the $(JAM_SHELL) variable:
JAM_SHELL = /bin/sh -c % ;
The % is replaced with the text of the command script. If % is not provided, the text of the command script is added at the end.
Jam Version Variables
| JAM_DATE | Time and date at jam start-up. |
| JAM_VERSION | KJam version, as reported by jam -v. |
| JAM_CWD | The directory Jam was invoked in. |
| JAM_USER | The name of the current user, as reported by the operating system. |
| JAM_PLATFORM | Identifies the major operating system family that this version of KJam is built to run on. For Windows this value is "Win32", on Linux the value is "Linux" |
| JAM_UNAME | For unix versions of KJam this variable indicates the specific operating system version as reported by a call to the uname() system call. The variable is a vector with five elements, machine, os version, os release number, node name, and system name. |
| JAM_NAME | The name of the jam executable. Usually set to kjam, but may be different if you rename KJam. Useful for actions which need to recursively call KJam. |
| JAM_CMD_OPTIONS | The command line options passed to KJam. |
| JAM_CMD_TARGETS | The top level build targets passed to KJam on the command line. |
| JAM_CMD_FILES | The names of the jamfiles passed on the command line with -f. |
| JAM_WRAP | This controls KJam's output autowrapping feature. Set this variable to a set of three numbers, start, end and margin. start< sets the column output gets wrapped to when a line is too long. end sets the maximum length of a line. Lines longer than this will be cut short and the remainer of the output will be printed indented to start. margin will cause the wrap feature to cut lines short on the first whitespace character on the column indicated by end - margin. For example JAM_WRAP = 10 $(COLUMNS:E=80) 20 ; will set reasonable wrap settings for most shells. Leaving JAM_WRAP unset, or setting it to 0 0 0 will turn off the autowrapping feature. |
| JAM_TARGETS | All the targets defined so far. |
| JAM_BUILDABLES | All the targets buildable defined so far, that is targets with at least one action. |
Built-in Shell
KJam uses its own built-in sh-like shell to execute command scripts during the updating phase. The KJam shell separates the script into command chains on newline boundaries. Each command chain consists of a series of operating system commands separated by | pipe operators. Each command is passed to the operating system and run. Its output stream is redirected to the input stream of the next command in the chain. The error stream of each command is redirected to a common error stream for the command chain. If any command returns a non zero return code, all of the commands in the chain are terminated and the shell will return this error code.
The output and error streams of the command chain may be redirected to one another or to a file by appending one of the following to the end of the command chain:
| > file | |
| 1> file | Redirects the output stream to the given file. If the file exists, its ttvious contents are replaced. |
| >> file | |
| 2>> file | Redirects the output stream to the given file. If the file exists, the output is appended to the file's existing contents. |
| 2> file | Redirects the error stream to the given file. If the file exists, its ttvious contents are replaced. |
| 2>> file | Redirects the error stream to the given file. If the file exists, the output is appended to the file's existing contents. |
| 1>&2 | Redirects the output stream to the error stream. |
| 2>&1 | Redirects the error stream to the output stream. |
Any output redirected to a file named nul will cause that output to be deleted.
A command chain may take its input from a file by use of input redirection:
| < file | Redirects the input stream to this command chain to come from the given file. |
The built-in shell includes several built-in commands which the shell interprets itself, instead of calling out to the operating system.
| echo arg | Prints the given arguments to the output stream. |
| rm [-r] files | Removes the given files. If -r is given, it will remove the given files and recursively delete the contents of any given directories. |
| mkdir dirs | Creates all the given directories. The directories may be more than one layer deep. |
| cd dir | Change the current working directory for the shell. |
| touch files | Update the time for the given files. If any file does not exist, it creates a new empty file by that name. |
| chmod files | Update the time permissions for the given files. |
| mv src dst | Moves the file given by src to the directory or filename given by dst. |
| cp [-h]src dst | Copies the file or directory given by src to the directory or filename given by dst. If the source is a directory the contents of the directory and its subdirectories will be copied. In a recursive copy, when the -h option is given hidden files and directories in subdirectories will not be copied. On Linux the non-copied files are files starting with ".". On Windows hiddden files either have the hidden file attribute set, or start with "." for compatibility with Linux |
| cat files | Contatenates all the given files and copies their contents to the output stream. |
| rgxrpl [-x expr] [-[0-9] expr] [-g expr] [-d #] | Manipulates the input stream using regular expressions, and copies the modified data to the output stream. |
These built-in commands may mask existing commands on your system. To use the operating system versions, a complete path to that command, an alias or simply surround the command name in quotes, or run your script using the operating systems own shell.
Comments begin with # and continue to the next newline character.
The KJam shell is very limited, and supports only the shell features normally used in short command scripts found in make files. For example it does not have support for flow control, like sh's if, do, while and case statements.
To set shell varibles use =, like so:
foo = barTo get the value of a shell environment variable use server variable expansion syntax:
echo $!(foo)Using normal variable expansion syntax will not work, because those variables will be expanded before the command shell is spawned, when the action script is still being assembled.
chmod
This tool changes the file permissions. Command takes arguments of the form:
The -R option changes the permissions of all files and subdirectories under a given directory. The -f option will cause chmod to not issue errors concerning file permissions if any are encountered.
The mode can either be an octal number representing the actual bit pattern of the file persmissions, or a symbolic mode of the form:
Where who is one of: u, g, o, a, which represents setting permissions for user, group, other or all respectively.
Where op is on of: +, -, which represents adding that permission or removings it respectively.
and where the permission itself is one of the following:
rgxrpl
This is a tool to manipulate text using regular expressions. The program reads data from standard input, changes it using the regular expressions given on the command line, and writes the modified data to standard output. Each command line option will either match some part or the input, or will modify matched data in some way. The command line operations are applied to the data from left to right as they appear on the command line. The operations are:
| -x expr | Scan the data for matches with the following regular expression |
| -[0-9] expr | Replace data matched with a previous regular expression with the given string. -0 will replace the entire matched expression. -1 will replace just the first subexpresion. |
| -d number | Delete the matched expression from the output. If the number is 0 the entire expression is deleted. Numbers 1-9 will delete the numbered sub-expression. |
| -g expr | Only copy lines which contain a match to the output stream. Like grep. |
Regular Expressions
The KJam language uses regular expressions anywhere where flexible matching of text is desired. As noted above this includes using regular expressions as arguments to the X, I and N variable expansion modifiers, as the labels in a switch statement, and as the value of for the HDRSCAN built-in variable. KJam's implementation of regular expressions matches closely with the syntax of standard Posix regular expressions, and should be familiar to many users.
A regular expression consists of one or more non-empty branches separated by |. A regular expression matches any text which matches at least one of those branches. For example, the expression one|two will match the string one and the string two.
A branch consists of one or more concatenated atoms. There are several types of atoms, and each one can be used to match a given text. The types of atoms are:
By itself each atom matches exactly one copy of the matching input text. Each atom may be followed by one of the following, which changes how many times the atom matches:
A bracket expression is a series of characters enclosed in []. It normally matches any single character from the list.
If the list begins with ^, it matches any single character not in the list. If two characters in the list are separated by
-, this is shorthand for the full range of characters between those two in the collating sequence.
For example the atom [abc]* would match the first 5 letters of bcbacXacd. The atom [^X]* would also match the first 5 letters of bcbacXacd. The atom [A-Z0-9] would match the first 5 letters of DS14Tabc.
A bracket expression may also contain inside its [] one of the following special character classes:
For example the atom [[:alpha:]_]* would match the string this_is_a_test.
A regular expression may also contain back references. All regular expression atoms, that is any atom surrounded by () are numbered. They are numbered in the order that their ( appears in the expression. So the leftmost regular expression atom would be numbered 1, the next one 2 etc. The entire expression is numbered as zero. A back reference atom matches the exact same string as the numbered atom it refers to.
For example a(b*)c\1d matches any string with the same number of b's on either side of the c such as the following strings: acd, abcbd, abbbcbbbd.
Network Execution
KJam may be run in a distributed client-server mode over a network. In this mode, KJam is started on each machine as a server process, with the -z option. If all the servers are on the same subnet and using the default KJam network port (2297), then all the servers will find each other and organize themselves into a peer network. It is possible to create peer networks of servers on alternative ports, and across subnets by using additional command line options.
KJam is run as a network client with the -y option. In this mode by default KJam will attempt to connect to a KJam server on the local machine, on the KJam network port (2297). It will run through its normal operation just like in non network mode, except when running command scripts, instead of running them locally, the scripts will be sent to be run on one of the KJam servers on the peer network. The peer network's own load balancing algorithms will distribute command scripts across the peer network. Multiple KJam clients may be interacting with the peer network at any time. By using additional command line options is it possible for the KJam client to connect to any server or port to get its initial pointer into the KJam peer network.
If KJam is started as a network client and it cannot find a peer network it will scan for one on the local subnet. While this works, it is considerably slower than either making sure there is a KJam server running on the local machine, or providing appropriate information about where one can be found.
In most cases it is important that the file system used by KJam to bind files on the local system be visible on all of the remote KJam servers in the exact same way. The commands passed to the remote servers will be run on those servers with the bindings determined on the KJam client. The files themselves are not transferred, but rather the remote server will attempt to read them from its view of the file system. Similarly, the output files and the programs being run are also not transferred. All of these things must be available on the remote server system. There are several easy ways to accomplish this. The simplest is to make the part of the file system that the KJam client is running in be visible on the network from all the machines. ( For example on windows by 'sharing' ). Then run the client in the subdirectory using the shared version of the directory. ( e.g. //SYSTEMNAME/PROJECTNAME ). In unix file systems this problem should be easier to address if all of the machines have had their local file systems mounted into one large virtual file system.
In network mode, special attention should be paid to the configuration of compilers and the versions of libraries and headers on all the KJam servers. It is possible with subtle changes in these tools for the build to appear to complete correctly but for the binary to have problems. It should also be noted that when running in network mode most programs will be running on one system but interacting with a file system on another. This often runs more slowly than if the entire process was local. For some programs which execute in very little time, or have heavy file access patterns it may be faster to run those command scripts entirely locally, and only distribute onto the network command scripts with more computationally intensive scripts.
Server command line options:
| -z [port] | Invokes KJam as a network server. The -z option can take an additional argument giving an alternate port number for the server to listen for peers and client. Only one -z and port number may be given. |
| -p scan | By default the server will search for peers on the local subnet ( 127.0.0.* ). Use -p to make the server search in other places. The argument to -p can be a complete ip address ( e.g. 192.168.100.100 ), or certain subnet to scan ( e.g. 192.168.100.* ). The address may be expressed by itself, in which case KJam will scan the default KJam port number, or with an explicit port number to scan ( e.g. 192.168.100.100:5555 ). Multiple -p options may be given. |
| -l power | This option sets the computational power this server will advertize itself as. Power may be any floating point number. By default all servers will advertize themselves as power 1. More powerful servers may be given a higher number, and less powerful ones a lower number. This power number affects how the load balancing algorithms distributes command scripts. |
| -j jobs | This option sets the maximum number of simultaneous commands scripts the given server can run. A good rule of thumb for most servers is to set this to two times the number of physical processors. |
| -x type | Sets this server to advertize itself as a server of a given type. Some build actions may be specified in the jamfile as only allowed to run on systems of a certain type. Multiple -x options may be given for systems which are configured for multiple job types. |
Client command line options:
| -y | Runs KJam in network client mode. |
| -p scan | By default the client will look for a server on the local machine, and on the standard KJam network port ( 127.0.0.1:2297). Use -p to make the client search in other places for its initial KJam server. The argument to -p can be a complete ip address ( e.g. 192.168.100.100 ), or certain subnet to scan ( e.g. 192.168.100.* ). The address may be expressed by itself, in which case KJam will scan the default KJam port number, or with an explicit port number to scan ( e.g. 192.168.100.100:5555 ). Multiple -p options may be given. It is not necessary for the given -p directives to cover the entire set of possible KJam server addresses. It is only necessary that the client be able to locate one of the peers quickly. Once that happens, the peer network will send the KJam client a complete list of available KJam servers on the peer network. |
It is possible to make KJam spawn jobs on only certain servers. To do this the servers can advertize one or more server 'types' using the -x option. Each action can be optionally labeled with a list of server types which the action can run on. When an action script has a server type, the peer network will only schedule that job to run on a server which advertizes that type. In addition to the server types specified by the -x option, KJam servers advertize several default types. All servers will advertize the local machine name as one of their types. This is so that scripts can be easily written where certain actions run only on a specifically named system. Linux machines will advertize as "Linux". All Microsoft Windows machines will advertize "Win32". Depending on the version of Windows, those machines will also advertize one of the following types: "WinXP" "Win2000" "WinNT" "Vista" "Longhorn" "Win2003ServerRc2" "Win2003Server" "Win95" "Win98" "WinME" "Win32s". Servers will also advertize the processor architecture, "i386", "i686" or "AMD64".
Case Sensitivity
The KJam language itself is always case sensitive. Case is important when referring to variables, targets, rules, and actions.
By default KJam also interacts with the file system in a case sensitive manner, even on operating systems with case insensitive file systems like Windows. In most cases, even on Windows this should produce correct behavior if the Jamfiles are written to carefully match the case of the actual file names.
In some cases this can be a problem when the same file is referred to in multiple places in the sources ( e.g. include directives ) with different cases. In this case KJam may fail to properly track some dependencies, or fail to find some targets. The preferred, cleaner solution is to fix the sources such that files are referred to with the correct case. If this is not possible KJam can be run in case insensitive mode using the -C option.
The -C option is almost never useful when KJam is running on an operating system with a case sensitive file system ( like Linux or Os X ). It can sometimes be useful on these platforms when generating build scripts for Windows using the -o option.
The -C option will cause a small performance penalty to file system scanning speed.
Managing Built-in Jamfiles
KJam comes configured with a set of built-in Jamfiles. This allows KJam to support many simple c and c++ projects with extremely simple user-defined Jamfiles.
For more complex projects most users will want to write their own custom Jamfiles. These can be written as separate Jamfiles and stored in the filesystem like regular files, or they may be embedded into KJam, replacing the built-in rules.
To extract the built-in Jamfiles, run KJam with the -X option. KJam will write out all its embedded rule files in the local directory with their original names. The KJam binary will still hold copies of the embedded files.
To replace the built-in Jamfiles, run KJam with one or more -P filename options. KJam will remove any existing embedded Jamfiles, and will embed all the files passed in through the -P option. If all the files passed in are found, and KJam is able to embed all the passed in files successfully, KJam will write out a new binary named kjam_new which has the new files embedded. This new binary will run as if all the embedded jamfiles were in the same directory as the kjam binary. Once it has been confirmed that the new binary runs correctly, it can be safely copied over the original kjam binary.
To remove all the embedded files, and replace them with nothing, use a single -P with no arguments.
Performance Considerations
KJam is designed to be able to compute dependencies between thousands of targets, and to determine which ones need to be updated in just a few seconds. Even for large projects, KJam's response time should be very fast. When writing a Jamfile the following things may adversely affect performance:
File System Latency: Each time KJam runs it must find every target in the file system, and get its time stamp. To scan for headers it may also need to read the contents of many files. If it is interacting with a slow file system, such as one which is mounted over the network, this may cause KJam to run much more slowly.
Too many directories in $(SEARCH): During the binding phase, each file is searched for in each directory specified in $(SEARCH). The number of checks scales as the product of the number of targets by the number of directories to search. For large projects with many files and directories it is important to limit the number of directories that each file is searched for. Divide up the targets into groups ( for example, by library, or separate data files from source files etc. ), and search for targets in each group in the smaller subset of directories in which they occur.
$(HDRSCAN) pattern too complex: A regular expression is used to scan files for header references. If there are a large number of files, or if the files are large the time spent scanning can be quite significant. An attempt to match each pattern is made at every byte offset in the file. That is for a 5k file, 5k matches must be attempted. Patterns where the first character in the pattern always occurs will execute much more quickly than patterns where the first element is optional. For example, the first header matching expression is much more efficient than the second one:
HDRPATTERN = "#[\t ]*include[\t ]*[\"]([^\"]*)[\"]" ;
HDRPATTERN = "[\t ]*#[\t ]*include[\t ]*[\"]([^\"]*)[\"]" ;
Very long files to scan: Some projects may include automatically generated source files which can be very large ( many megabytes ). It can take a significant amount of time to read in and scan these files for headers. In addition it is usually unnecessary to scan such files any further than the first few lines. To limit the how far into large files to scan set $(HDRSIZE) to the length in bytes of the depth to scan each file. Only the first $(HDRSIZE) bytes of each file will be checked. Setting this variable to zero or a non-number will cause the entire file to be scanned.
Overuse of flow control: Many things which are possible to do with flow control constructs like for and while can be done faster by use of expansion modifiers. For example:
for ( I in $(C_FILES) ) { O_FILES += $(I:S=".o") ; }
will execute faster when expressed as:
O_FILES = $(C_FILES:S=".o") ;
Re-evaluating expansions unnecessarily: In KJam it is much more efficient to create a new temporary variable than to re-evaluate a variable expansion. For example, the following code:
LOCATE on $(TARGET) = $(BIN_DIR)$(OBJ_SUBDIR) ;
depends $(TARGET) : $(BIN_DIR)$(OBJ_SUBDIR) ;
can be expressed faster as:
OUTPUT_DIR = $(BIN_DIR)$(OBJ_SUBDIR) ;
LOCATE on $(TARGET) = $(OUTPUT_DIR) ;
depends $(TARGET) : $(OUTPUT_DIR) ;
Diagnostics
In addition to generic error messages, KJam may emit one of the following:
warning: unknown rule X
updating N target(s)
can't find N target(s)
can't make N target(s)
warning: X depends on itself
don't know how to make X
X skipped for lack of Y
Warning: Target X will be built as a side effect of building another target.
Bugs and Limitations
A poorly set $(JAM_SHELL) is likely to result in silent failure.
Actions set with the 'together' modifier may not have more than three arguments.
When building in parallel sometimes the tools which KJam calls may try to access a file while it is being accessed by another tool in another thread or process. This may cause some of these programs to fail due to a file being locked or busy. KJam itself always checks for these conditions and should never have problems with files which are temporarily locked or busy. Some tools however do not check for this condition and may in rare instances fail unpredictably. The only way to fix this problem would be to rewrite those tools to interact with the file system more carefully.
KJam's dependency scanner is regular expression based. It can be used to create dependency scanners which trigger off simple include expressions, but cannot be used to evaluate inclusion statements which are affected by macros. Therefore it is possible for KJam to generate extra dependencies for included files which in reality are not actually included because the include statement is controlled by a conditional statement. It is often the case that conditionally included files may not actually exist on some platforms resulting in missing file errors. If this is the case, these errors can be silenced by labelling those files as nocare. In a correctly written jamfile this can only result in extra dependencies, and never in missing dependencies. It is possible if a jamfile has incorrectly set search paths for an include file which is necessary to not be found and yet for an error not to be generated because the header file was labeled nocare. This results in missing dependencies which can affect the correctness of builds. When jamfiles are being developed which include the keyword nocare, this condition should be checked for by running the KJam with the -d search command line option, and the list of files not found should be carefully checked for correctness.