#!/bin/sh ####################################################################### ## ## ## Recompiling CMUCL is not an exact science... ## ## [Peter Van Eynde, September 1998] ## ## ## ####################################################################### # # The is the README file of the CMUCL build instructions by Martin # Cracauer . # # To use these build instructions, you need all files from the # cmucl-build/ web directory # Single browseable files: http://www.cons.org/cmucl/cmucl-build/ # All in one tarfile: http://www.cons.org/cmucl/cmucl-build.tar.gz # # There are alternative ways to build CMUCL: # - the Debian source package has its own self-building infrastructure # - the tools/ subdirectory of the CMUCL sources contains shellscripts # to aid in building CMUCL. Douglas T. Crosher uses polished # versions of these for his own rebuilding. His versions along with # a README are on #http://www2.cons.org:8000/ftp-area/cmucl/experimental/cmucl-dtc.src.tgz #http://www2.cons.org:8000/ftp-area/cmucl/experimental/cmucl-dtc.src.thread.tgz cat << EOF > /dev/null Basic understanding of the task: ================================ You have a running Lisp with a compiler. You use this Lisp to get a new CMUCL system from your to-be-build sources. 1) You have to build that startup code (the "lisp" Unix binary). To do so, you use your running Lisp to generate a header file that encodes various information from your to-be-build sources into constants in this header file. They become literal values in both C and Lisp code. 2) You use your running Lisp to compile each of the *.lisp files from the to-be-build sources. You may have to work around bootstrapping issues when a new file depends on other new files that are of course not loaded in your old running Lisp. 3) You generate a basic corefile that is compatible with your newly generated startup code. This is a "kernel.core", which contains just enough of CMUCL to load compiled files and write corefiles. 4) You use that kernel.core with your new "lisp" Unix binary to load in everything you need for the final lisp and dump a core. This time, the core includes a compiler. 5) You use that new Lisp core with it's up-to-date compiler to recompile all *.lisp files again. This time, you may also compile optional parts like PCL/CLOS. Again, you write a kernel.core from this stage. 6) You use that kernel.core to load everything you just compiled and dump a final core, this time containing the Compiler and if desired optional parts like PCL/CLOS. 7) Paranoid folks like to recompile just again, to be sure that the new system has no artifact from the old running Lisp. These instructions also build the very-optional parts like CLX at this stage. Doing this thing: ================= This looks like a shellscript, but you should use it as such only when you know it successfully runs through. As long as you are unsure whether it runs through, cut and paste the non-comment likes from here and drop them into a running interactive shell (Bourne shell compatible required, no csh). Before you start: - These instructions assume that you start from src/lisp/ - These instructions assume that the files that accompany these instructions are reachable from src/lisp as ../../crabuild/ The directory structure should look like this: $BASE/crabuild/README $BASE/crabuild/rmcompiled $BASE/src/lisp/ $BASE/src/tools/ $BASE/src/[other cmucl dirs] $BASE/fasl-crabuild/lisp # will maybe created $BASE/fasl-crabuild/tools # will maybe created ($BASE is just to make the point clear, you don't need to set it.) If you want a different structure, adjust the paths in env.lisp env-simple-raw.lisp and in the command in this README. - In src/lisp/, symlink the right Config.* to Config - Adjust path settings in env.lisp and env-simple-raw.lisp if needed. - The structure of the env things is like this: - env-simple.lisp is automatically generated by the crabuild:buildenv script. You edit env-simple-raw.lisp, the script makes env-simple.lisp from it. - env-simple.lisp is meant to contain as much as it can, but it must be readable by a lisp with just kernel.core. That means, no reader macros, no (if ...) and nothing else that uses the compiler must be in env-simple.lisp - Things that must use features not available in kernel.core go into env.lisp, which reads env-simple.lisp, so there's no need to add things to both files. - Of special importance in env*.lisp are *features* settings. For each combination of existing Lisp and to-be-compiled sources, make sure that all steps in your first try use the same set of features. That means, for your existing lisp and for all compiled Lisps, make sure that *features* is the same. Start them interactivly and print *features* to make sure. Add or remove features as needed in env.lisp and env-simple-raw.lisp so that everything matches. - Also see the crabuild:buildenv script and the resulting env-simple.lisp. The crabuild script is meant to automatically generate *feature* settings that for sure must be there when compiling normal sources in non-crosscompiling situations. Typically, it looks at the platform OS kind, OS version, libc version and object format and adds *features* for them. You may have to edit it when you changed sources' reactions to *features* or when you compile older sources than the script is aware of. - Only when compiling with the same set of *features* works, experiment with adding or removing features for the to-be-created Lisp. One at a time is highly recommended. - One thing besides *features* that env-simple*.lisp controls is the *cra-lisp-implementation-version* string, which will be used for *lisp-implementation-version* in the new image. If you don't add a string of your own manually to env-simple-raw.lisp, the buildenv script will automatically generate a string from the modification date of the newest *.lisp file in your CMUCL sources. - If you never rebuilt CMUCL with itself, you should start with an existing Lisp and sources from the same date. Recompiling newer sources on older binaries often causes bootstrapping problems. Use 'cvs -q update -dP -D 20000524' (or such) for current sources (current binaries should indicate the date of their sources at startup), for releases use an appropriate CVS tag. Make sure you don't mix up the HEAD and RELEASE_* or RELENG_* tags and branches. - The file CompileCL.howto is Paul's original document, from which this one is derived. People frequenty get this wrong. Thanks, Paul. - In these build directories is a number of boot-*.lisp files. Such a file may be needed when compiling newer sources on an older existing Lisp. The start of the file should indicate which file is for what. People changing the compiler so that it requires a bootstrap file release them on the cmucl-imp mailing list or in the experimental diretory of the CMUCL distribution server and I add them to these instructions. But there might be newer bootfile I didn't add so far, so check the list! - The build process describben here always reads one file named crabuild:boot.lisp, which is no specific bootfile, but a placeholder. The file included in these instructions is usually an example of the "full thing" which cover most bootstrapping problems at once. If you start tangling with CMUCL rebuilding, you should start with a Lisp and to-be-compiled sources from the same date and in that case, just empty boot.lisp, you don't need it. If you need just one bootstrap file, you might create a symlink. You may try jumping over several bootstrapping points at once (like the boot.lisp included here supports), but usually you should prefer to do things one step at a time, recompiling with the second bootstrap file using a complete lisp compiled with the first bootstrap file (and sources of an appropriate date, of course). Having an unneeded bootstrap file loaded is usually harmless, but if you run into problems not to be explained otherwise, this is a place to look into. - The file to-be-compiled.lisp is a file that gets loaded by before writing the final image. Hooks for cleanups of the image go here. See the file's comments for more documentation. Usually, you don't need to set this, at least it doesn't influence your chances of getting Lisp built. ---------- WAKE UP!!! ---------- Sorry for the interruption, the explanation was quite long, so here are the important things you have to check again: - You have symlinked boot.lisp, created an empty one or one from concatenating several boot files? - You created or symlinked src/lisp/Config? - You reviewed env-simple-raw.lisp and env.lisp for your needs? - Unless you know what you do, you have a running Lisp and sources from the same branch and from the same date? - You know you have to start the steps below from src/lisp/? OK, here we go... # End of NOTES EOF set -x set -e # we don't want coredumps ulimit -c 0 || true # Utility functions. If you execute this file by cut/paste, drop # these into your interactive shell (must be a Bourne shell clone), so # that they are available for later shell command lines. li () { lisp -batch -eval '(load "../../crabuild/env")' "$@" } li2 () { ./lisp -batch -eval '(load "../../crabuild/env")' -core "$@" } # # Here we go: # # generate env-simple.lisp ../../crabuild/buildenv > ../../crabuild/env-simple.lisp # look for the age of these sources ../../crabuild/sourcedate > version.tmp # Remove fasl files ../../crabuild/rmcompiled src touch Depends gmake clean # Set version number echo 'Map file for lisp version 1' > lisp.nm # Build internals.h li -eval '(load "crabuild:build-int_h")' # Compile C startup code touch Depends gmake depend gmake all # Compiles everything except PCL and builds kernel.core. kernel.core # does not contain all the newly compiled files, just those that are # needed to build a new full core from them. This step is running in # the old Lisp, it can compile the files, but it cannot load them to # build a core. Instead, a small core with loading stuff is built # with just a subset of the just compiled files (kernel.core). li -eval '(load "crabuild:build-the-subsystems")' # Compile the file containing last fixups li -eval '(progn (compile-file "../../crabuild/to-be-compiled") (quit))' # Here we start the *new* C startup binary (lisp/lisp) that we # compiled with gmake. We start it with kernel.core, where the # interal structures are expected to match. This is the first # critical step, it often fails. See the troubleshooting section at # the end of this document. # # If the new small Lisp runs, it is able to load all the compiled # files (it does not and can not compile them itself, this lisp does # not have a compiler) and build a real lisp.core from them, with # compiler and everything, but no PCL. cat << EOF | ./lisp -core kernel.core (load "../../crabuild/env-simple") (pushnew :no-pcl *features*) (load "crabuild:build-core") `cat version.tmp` (quit) EOF # Move core files around cp lisp.core tmp.core mv lisp.core nopcl.core mv kernel.core kernel.core.old # Now rebuild the compiler with itself. # This time pcl is automatically compiled, since it is disabled in the # running image. # # If you compile a newer version of the sources than your binary is, # do it several times. for i in 1 ; do # FIXME pcl build when doing more than one ../../crabuild/rmcompiled src # Same procedure as before. Use an old image to compile but not # load all files. Build just a basic loading-capable kernel.core. li2 tmp.core -eval '(load "../../crabuild/build-the-subsystems")' # Now use loading-capable core to build a new core. cat << EOF | ./lisp -core kernel.core (load "../../crabuild/env-simple") (load "crabuild:build-core") `cat version.tmp` (quit) EOF mv lisp.core tmp.core done # now recompile lesser-know subsystems and load them # li2 tmp.core -eval '(load "crabuild:build-some-subsystems")' # li2 tmp.core -eval '(load "crabuild:build-full-core")' cat << EOF > /dev/null TROUBLESHOOTING =============== So things went wrong, what to do? - Review *features* as indicated at the beginning of this document. Make sure they match after loading crabuild/env.lisp for: - your existing lisp - ./lisp -core kernel.lisp - ./lisp -core tmp.core Features are dropped and added in rather obscrure ways in the build process. Use env.lisp and env-simple.lisp to adjust them. - Did you understand the boot.lisp stuff from above. Make sure you load the ones you need and in case of further trouble none that you don't need. - Make sure env-simple-raw.lisp stays simple. If is used by a Lisp that has no compiler and no reader macros. - no #+ or #- - no #|...|# comments - no (if ...) ; requires compiler - no function creation, including no #'(lambda ...) - Usually, you cannot not load crabuild:boot.lisp from a kernel.core lisp, this must be done in env.lisp, not env-simple.lisp - If you still fail, especially with things like "gc lossage", signal 11/segfault or "object is of the wrong type", nuke the whole tree you used to build CMUCL and restart. Nuke everything: the checked-out sources, the target dir, the C code compile dir, everything, everything, everything. Restart by checking out a fresh src/ tree. Don't ask, just do it. Yes, you removed everything that was build, you are sure, blah, blah, blah. Shut up, sit down, nuke the damn thing and restart. This is what I do and since you're reading these instructions and seem to follow them, you should do so too. With the same head-shaking when it works afterwards although you was so sure you had it the old thing clean, it is just the feeling I have when it happens to me. And it does happen, did I mention that rebuilding CMUCL is not an exact science? CROSSCOMPILING ============== You usually should get crosscompilers from the cmucl mailing list or from the dtc sources in the experimental/ directory of the CMUCL distribution server. However, these build instructions contain slightly edited versions of various crosscompiler by other people (mostly dtc). cross-simple.lisp, cross.lisp and cross-loadbackend.lisp. My editing is done so that they work for non-experimental sources and/or don't require special features. Basic usage instructions: - Create fasl-cross1/ and fasl-cross2/ at the same level as src/, with all subdirectories. crabuild:makedirs will do this for you. - Uncomment the right block in the middle of the file for the kind of crosscompiler you want. - Just load cross.lisp - It uses the p86/ subdir when present. p86 is the cannonical name for "highly experimental", if you use it, you should also you crosscompiler you find at the same place. My copies here are usually hacked so that they do not require p86 stuff that is not in the normal CVS tree. - It will compile the compiler in fasl-cross1/, load it and use it to compile the world in fasl-cross2/. It also creates fasl-cross2/lisp/kernel.core, which you are suppoed to be able to do a worldload with. - You may have to compile your lisp startup code again, the cross compiler script will tell you if so. Happy Lisping Martin EOF