KJam continues under rapid development.

We released a new version of KJam Beta today with an cool new feature, a new action modifier keyword 'immediate'. There are also a number of bug fixes, including a fix for a potential crash.

Changelog:

This release marks a major milestone for KJam - we have finished porting KJam to Apple Os X. This download includes a version of KJam for all the platforms currently supported: Microsoft Windows XP, Windows Vista, Linux ( Red Hat, Suse, Debian, Ubuntu etc. ) and now Apple Os X. Version 0.36 also has two interesting new major features, both added a as a result of feedback from KJam users.

The first feature is the -C command line option, which causes KJam to use case insensitive file system scanning. The default behavior is to match file names in a case sensitive manner. On operating systems with case insensitive file systems like Windows is can be useful to use a case insensitive search is the case of file targets is not predictable.

Another new feature is the addition of a new action modifier keyword 'thread', which limits the number of threads that KJam will spawn for actions of a given type. This is useful in cases where you want to have a highly parallel build with many execution threads for most build actions, but want to limit the number of threads that can run simultaneously for a particular type of action - for example limiting the number of simultaneous link steps. This is especially useful is some steps are not thread safe because they lock the same resource - such as writing to the same file. As usual, a number of bugs were identified and fixed.

Changelog

We skipped one public release, 0.34, and went straight to 0.35.

The main difference in this release is the addition of a new flexible licensing model. In previous versions the licensing code relied on non-modifiable license information which is compiled into the binary. After 0.35 the license information can be modified using a license file. This is in preparation for commercial licensing. Users in the beta program, or those using the demo version should not be affected. For them the new code should work the same as before.

We continue to add new features to kjam as we get feedback from users. If you have problem doing something with KJam we encourage you to write in and let us know. User comments drive the addition of new features.

One very interesting new feature is the timeout and retry keywords. These are designed to help build robust build systems even in environments where your tools can crash or hang unexpectedly. If you have an action that calls an unstable tool you can instruct kjam to retry that action several times before reporting an error. If you have a tool which is prone to hangs, you can specify a timeout after which kjam will terminate the presumably hung process.

As usual, a number of bugs were identified and fixed. The bugs keep getting fewer, smaller and more obscure which gives us confidence.

Changelog

Version 0.33 adds the few remaining features that were planned for 1.0. Namely built-in shell scripts now have their own environment variables. In previous versions build actions which needed environment variable support were forced to use external shells.

In network mode, you can now specify if you want a variable to be evaluated on the client or server. This is useful when you need to access a built-in variable as it exists in the server environment. For example if you need to get the path to a binary on the server machine.

The networking code has also been gone over again to make it more robust to failure. For example if a client or server is taken down unexpectedly, KJam will terminate any build commands spawned by the terminated processes.

As usual there are also many other smaller improvements and bug fixes.

Changelog

We are excited to have a kjam featured in an article in gamedev.net. link

Built-in Shell Improvements: This version of KJam has a new built-in shell command, chmod. It is portable between all supported platforms, with the same interface on all. On Linux this is a POSIX compliant chmod. On windows it supports as many file permissions as windows supports, and also the special windows file attributes like HIDDEN, SYSTEM etc. It also has a number of minor improvements to the existing built-in shell commands rm and touch.

Changelog
FEATURE: Added chmod as a built-in command for the built-in shell
FEATURE: The touch built-in shell command now reports an error on failure.
FEATURE: The rm built-in command now has a -f option, which forces removal for read-only files.

We have run some benchmarks comparing the performance of KJam against Jam, Gnu Make and MSBuild. The results are dramatic. KJam is significantly faster then Jam, and both KJam and Jam are often 20x faster than MSBuild and Gnu Make.

A new version of KJam Beta is now ready for download. There are a number of major improvement over previous versions.

Efficiency: Previous versions of KJam were already extremely fast. This new version is the fastest build tool we know of. The new file scanner in KJam 0.31 is over 100% faster than in 0.30, able to scan over 100,000 files per second. Regular expression compilation and matching in some cases is 10x faster. KJam is now even smarter about when it accesses the file system. The result is that KJam is now optimized to the point where performance is limited only by the speed of the disk subsystem. During the scanning phase, KJam is mostly idle waiting for i/o requests to be serviced by the operating system - even on systems with extremely fast SCSI or SATA RAID0 disk arrays.

We have measured KJam's performance extensively, measuring it against many other build tools. We have optimized it to make sure that KJam is faster than every tool we know of in every case we have tested. For incremental builds KJam is often 10-20x faster than Gnu Make or MSBuild. We will be publishing benchmarks in the next few days.

More Responsive: KJam runs many build commands in parallel, and caches the output of these commands to make sure that the output of different commands is never interspersed. In previous versions of KJam this sometimes results in KJam appearing to respond slowly because output of commands that have already terminated is not presented to the user while another command is busy sending output to the user. This new version if KJam has a whole new more intelligent caching mechanism which results in more responsive output in non-network mode. Commands which return output to the user quickly, like compiles are given priority over commands which take a long time to run, such as links.

Support for Even Larger Projects: Previous versions of KJam could manage up to 30,000 targets in a single Jamfile. The data structures in KJam which track targets have been widened. KJam can now manage over 2 billion simultaneous targets.

Improved Error Reporting: Many bugs related to error reporting have been fixed. Many errors which previously did not report the file and line number that generated the error now do so. Many cases where line numbers were incorrect have been fixed. When errors occur in network mode on remote servers, the errors are now reported with the name of the server which generated the error.

Many Other New Features: New elements were added to the regular expression syntax. A new variable expansion modifier Q has been added which returns the reverse of a relative path. New built-in variables JAM_CMD_FILES, JAM_TARGETS and JAM_BUILDABLES have been added which make it easy to write dependency rules triggered on changes to the Jamfiles themselves. The built-in Jamfiles include support for DJGPP, and are a good example of how to support tools which use response files.

Changes to Built-in Variables: We have changed the names of all of some of the built-in variables. The previous naming scheme was designed to match the naming scheme the original Jam/MR, but this produces variables which are all run together and hard to read. As we have added built-in variables, the necessity of readable variable names has become more apparent, so we have changed them. Names like JAMDATE have become JAM_DATE. A complete list of the changed variables is given in the changelog below. This change may break some of your jamfiles, and for this we are really sorry. Fortunately the fix should be trivial.

As usual many bugs have been fixed.

Future Work: We are finished with optimizations to KJam for now. Further optimization will not lead to any real world performance improvement, because KJam is now limited only by disk speed. We will start work on improving network build performance and robustness. Currently network build performance on Windows is limited by the slow speed of network mounted Windows shares. It may be necessary to write our build oriented network file system.

Changelog
FEATURE: Various major profile guided speed optimizations resulted in significant performance improvement.
FEATURE: Compiled regular expressions are now cached instead of being repeatedly re-compiled, resulting in a significant performance improvement.
FEATURE: Regular expression scanning speed for all expressions improved substantially.
FEATURE: Mutexes guarding shared data structures reduced to pass through no-op functions when no threads have yet been spawned, resulting in minor performance improvement during scanning phase.
FEATURE: Directory paths ending in / or \, or without a final slash all refer to the same directory, yet were being read and cached multiple times. Caching improvement now has those directories scanned and cached only once.
FEATURE: Added beginning of input (/A) and end of input (/Z) elements to the regular expression syntax. These are similar to the beginning of line (^) and end of line ($) elements which existed already. They are extremely efficient, and should be used instead of ^ and $ when you really want the beginning and end of input.
FEATURE: Added regular expression debugging output option -d regexp
FEATURE: Added new built-in variable JAM_CMD_FILES, listing all the jamfiles passed in on the command line
FEATURE: Added new built-in variable JAM_TARGETS, listing all the targets defined so far.
FEATURE: Added new built-in variable JAM_BUILDABLES, listing all the targets with associated build actions defined so far.
FEATURE: Internal symbol tables expanded. KJam can now track 2^32 variables and targets.
FEATURE: Added Q variable expansion modifier. Given a variable with a relative path, returns the reverse path from that relative path. Such that if you did a cd with the given path, you could get back to the original directory with the reverse path. Not all relative paths have a reverse path. Full paths, and variables which cannot be interpreted as paths also have no reverse. These will return an empty variable. For example "one/two/../three/./four" has a reverse of "../../..". This is useful for writing Jam-like SubDir and SubInclude rules
FEATURE: Errors from executing shell commands now report the file and line number of the jamfile where the command can be found.
FEATURE: Under windows, KJam now detects if a command is a 16-bit DOS application, and will issue the proper error message if the command fails due to a command line which is too long. 16-bit dos applications are limited to 128 byte command lines.
FEATURE: Added support in the built-in jambase files for DJGPP.
FEATURE: Added -S option, which builds targets given on the command line in command line order, as if Kjam were being run once for each target. Normally KJam will build all the targets given on the command line simultaneously so that it can take advantage of any dependencies between them. But sometimes it is desirable to build the targets in the order they are given on the command line. This is useful for explicitly running a clean target before some other target for example.
FEATURE: The built-in rm command now has a -c option which confirms that the file was actually deleted, and returns an error if the file still exists. This can sometimes be an issue on network drives, where remote files are locked.
FEATURE: In network mode, when reporting errors, KJam will now report the name of the server where the error occurred.
FEATURE: In non-network builds, when kjam calls a copy of itself, the output is now more responsive. Kjam caches output from multiple threads so that the output from the threads will not be interspersed. There were cases where this caching prevented KJam from printing out any output for a long period of time. This aspect has been improved.
FEATURE: Threads are now prioritized for improved responsiveness.
CHANGE: Many built-in variables have been renamed. As we have added more and more built-in variables the need for a more readable naming scheme has increased. The old naming scheme which matched closely with the Jam/MR naming scheme produces names where all the words are run together in an unreadable mess. The new scheme separates words out with _. The following built-in variables: JAMDATE, JAMCWD, JAMUNAME, JAMPLATFORM, JAMUSER, JAMVERSION, JAMNAME, JAMCMDOPTIONS, JAMCMDTARGETS, JAMWRAP, JAMSHELL have changed to: JAM_DATE, JAM_CWD, JAM_UNAME, JAM_PLATFORM, JAM_USER, JAM_VERSION, JAM_NAME, JAM_CMD_OPTIONS, JAM_CMD_TARGETS, JAM_WRAP, JAM_SHELL
CHANGE: Embedded jambase files for windows versions are now terminated with windows line termination (CRLF) instead of unix line termination (CR only).
BUGFIX: Certain complex expansions involving nested @() and $(), where one of those elements was the argument to a modifier were interpreted incorrectly. Fixed.
BUGFIX: Several cases where it was possible for KJam to read beyond certain array bounds were fixed.
BUGFIX: When using the built-in Jambase, and setting $(BIN_DIR) to a custom directory, on windows, certain alternative paths would cause the rules to fail. Fixed.
BUGFIX: Using a non-existent variable expansion modifier causes KJam to fail without printing an error message. An appropriate message is now printed.
BUGFIX: Regular expressions using \w \s \d and similar codes in certain cases would have the expression matched exactly, instead matching their special meaning. Fixed.
BUGFIX: Certain valid complex regular expressions involving | would fail to compile. Fixed.
BUGFIX: In many cases the line number reported for commands in an actions script was off by one or more lines. Fixed.
BUGFIX: :H variable expansion modifier would select part of file's path if the path contained a '//'. Fixed.
BUGFIX: On Linux, reading of output from spawned threads and processes used a blocking io call resulting in output being less responsive in highly parallel builds. A non-blocking call is now used, as designed, resulting in more responsive output.
KNOWN ISSUE: On Linux, in network mode, recursive calls to Kjam will fail if the calls are made using the $(JAM_NAME) variable. Calling KJam recursively should be done using this variable so that KJam can make sure the same binary that called kjam is run to execute the recursive call to KJam. This will fail for recursive calls to Kjam in network mode. To work around this just call kjam directly by literally using "kjam" as the command, or define your own variable. If you have only copy of kjam on your system, and the paths are set correctly this would work fine.

This latest version of KJam has a number of optimizations. KJam is now much faster and uses less memory. A number of minor bugs have also been fixed.

Significant Performance Improvement

The initial file scanning phase is now 500% faster than version 0.29. In our benchmark case, version 0.29 could scan about 10k files in 4.5 seconds. This new version can do it in under 1 second. For many projects, even those with many thousands of files, you may notice that KJam now starts building instantly.

This performance improvement was achieved mostly though more aggressive caching of file system operations. One side effect of this change is that on Windows KJam's file name matching is now case sensitive. Whereas in version 0.29, under Windows you could refer to files in a case insensitive way, now most file operations are case sensitive.

Significant Memory Footprint Reduction

KJam now uses less memory than before. Most invocations of KJam will use about 2.5megs of memory.

Changelog
FEATURE: More aggressive caching results in Binding Phase speed improvement of by 500%. Scanning time for about 8000k files reduced from 5 sec to under 1 sec.
FEATURE: Autowrap feature applied to parsing and binding phase errors and diagnostics.
FEATURE: Significant reduction in memory use.

CHANGE: On Windows, KJam is now more case sensitive when dealing with file names and paths.
CHANGE: Actions labeled as piecemeal will have their targets and sources sorted. This avoids building certain actions multiple times unnecessarily when an action is called multiple times with the same arguments in different orders. In the case of piecemeal actions we know that such calls are always redundant and can safely be skipped.

BUGFIX: In network mode KJam would poll network jobs for output too often, resulting in Jam's cpu load being higher than necessary. Fixed.
BUGFIX: When using the default jamfiles and running under shells which don't provide a COLUMNS variable, like cmd.exe, arguments to SubProject targets were being ignored. fixed.
BUGFIX: Many spawned job threads were being shut down one more time than necessary. Fixed.
BUGFIX: Under Windows, file system read operations were gaining exclusive locked access to files, when shared access was possible, resulting in a minor performance penalty, and an unfrequent error where scanned files would fail to open with "permission denied" errors. Fixed.