The execution environment of Etherboot

The network booting process

Since this is the part that the user sees first, let us first demystify how network booting works.

From time immemorial, well actually since the IBM XT appeared on the market, the PC architecture has a mechanism for invoking "extension BIOSes". The original reason for this mechanism was to allow adaptor cards that the main BIOS didn't know how to deal with to carry ROMs with initialisation code or drivers. An early example was the XT hard disk controller. The main BIOS of XTs only knew how to boot from floppies. When an XT hard disk controller is added, the code in the ROM on the controller appears in the memory space of the PC and is called as part of the machine initialisation. Another example is the BIOSes on VGA video adaptor cards, although strictly speaking that is a special case in terms of ROM address. When network adaptors were made for the PC, it was a natural step to put ROMs on them that could contact a server for network booting.

How does the main BIOS know that the code in the ROM is to be executed and why does it not execute some random code by accident? The ROM code has several conditions placed on it.

If such a ROM is detected and validated by a scan, then the main BIOS does a far call to ROMSEG:3, where ROMSEG is the segment of the ROM and 3 is the offset to transfer control to the discovered extension BIOS. Typically a network boot ROM does not take full control at this point. Instead the normal procedure to do some initialisation or probing of the hardware and then plant a vector that will be called when the BIOS is ready to boot the OS. The vector used for this purpose is normally interrupt 0x19 although interrupt 0x18 is sometimes used.

For PCI plug and play ROMs things are more complicated. For the full story, you need to get the specifications from Phoenix and Intel. Here is a quick summary.

The network boot process then works like this:

  1. The main BIOS detects the Etherboot ROM as an extension BIOS and passes control to it with a far call.

  2. For legacy ROMs, the Etherboot code hooks itself to interrupt 0x19 and returns control to the main BIOS. For PnP ROMs the Etherboot code indicates that it is a bootable device.

  3. The main BIOS finishes initialising other devices and boots the operating system by calling interrupt 0x19.

  4. The Etherboot code gains control.

  5. It initialises the network hardware so that it is ready to send and receive packets.

  6. It sends a Boot Protocol (BOOTP) or Dynamic Host Configuration Protocol (DHCP) broadcast query packet. An alternative is Reverse Address Resolution Protocol (RARP)

  7. Assuming a reply is received, the Etherboot code decodes the fields of the reply, sets its IP address and other parameters, and sends a Trivial File Transfer Protocol (TFTP) request to download the file. Be aware that the 16-bit counter field of TFTP may limit transfers to 16 MB (signed interpretation), 32 MB (unsigned interpretation) or 90 MB (unsigned, large block size option active). This is a rollover bug in many TFTP servers, but quite prevalent. An alternative loading protocol is Network File System (NFS) protocol. In this instance a mount of the remote filesystem is done (with the bare minimum of features) and the boot file is read off the filesystem.

  8. The file to be loaded is in a special format, it contains a "directory" in the first block that specifies where in memory the various pieces of the file are to be loaded. Formats that are supported are tagged Appendix A or Execution and Loader Format (ELF). One small extension to ELF has been made, the top bit in the e_flags longword of the ELF header has been used as ELF_PROGRAM_RETURNS_BIT, meaning that the program transferred to intends to return to Etherboot, e.g. a menu program, and that Etherboot should not disable the network interface yet. See the file core/osloader.c.

    Eric Biederman adds: The ELF format is documented in the SysV Generic ABI doc. Also check out the linux standard base, it has good links to all of these documents, in its related documents section.

  9. Etherboot transfers control to the loaded image.

Notice no assumption was made that the image is a Linux kernel. Even though loading Linux kernels is the most common use of Etherboot, there is nothing in the procedure above that is Linux specific. By creating the loaded file appropriately, different operating systems, e.g. FreeBSD, DOS, can be loaded.

In the case of a Linux kernel, there is some additional work to be done before the kernel can be called, so the segment of the file that Etherboot transfers to is not the startup segment of the kernel, but an initial stub, whose code is in first32.c in the mknbi package. This stub has several tasks, which either cannot be done by Etherboot, or should not be done by Etherboot because they are Linux specific. These are: to append kernel arguments from option 129 of the BOOTP or DHCP reply; to copy and expand special kernel parameters, in particular the vga= and the ip= parameters and then to point the kernel to the location of the parameter area; to move the RAMdisk, if there is one, to the top of memory (this last cannot be done at image creation time since the size of the RAM of the machine is not known then).

The kernel parameters are passed to the kernel as a pointer to the string written in a certain location in the original bootblock (boot.S from the Linux kernel sources). This is a 16-bit pointer and is the offset of the parameter area from the base of the bootblock. This is one reason why the parameter area must be in the same 64kB as the bootblock. If the components of Etherboot are to be relocated elsewhere, e.g. 0x80000 upwards, then they should be relocated together. In version 0x0202 and above of the Linux setup segment, this can be passed instead as a absolute 32-bit pointer in a certain location in the setup segment. This eases the relocation requirements. The address of the RAM disk, if it exists, is passed to the kernel as a 32-bit address in a particular location in the setup segment (setup.S from the Linux kernel sources). This is filled in with the final location of the RAM disk after it has been moved to the top of memory.

To compress or not to compress, that is the question

We simplified things a little when we talked about how the main BIOS detects the Etherboot ROM and passes control to it. At this point the code is executing from ROM. There are two problems with executing from ROM where x86 PCs are concerned.

  1. The x86 architecture does not easily support Position Independent Code (PIC). The main drawback when executing C code is referencing global entities (global and static variables). Since the ROM address is not known when building the image, addresses cannot be assigned to global entities. More advanced environments have a dynamic loader for adjusting references just before use. Etherboot has no such help. If the code were written in assembler, we could use a convention like always referencing global entities as offsets from a particular register. But we don't want to write in assembler and we don't have control over what the C compiler generates.

  2. C code assumes that data locations are writable. ROM locations are not writable, by definition. One could remedy this by locating the writable entities in a RAM segment, but this causes more complication.

For the reasons above, Etherboot copies itself to the top of memory or 4 GB, whichever is the lesser, and executes there. However that is the physical address. The logical address for execution is set in the Makefile and currently is 0x20000 for the x86 version. The translation is done using the address translation facility of the MMU. This allows Etherboot global variables and functions to be assigned known addresses at the linking step.

(Actually top of memory isn't completely true. On the x86 platform it executes in an even megabyte just below the top of memory. This is normally the second last megabyte. The reason has to do with the setting of the A20 gate. If Etherboot executes in an odd megabyte it will get gated out of existence when the A20 gate is disabled. By running in an even megabyte this is avoided. Note that we still have to disable the A20 gate because some operating systems react strangely if they find it enabled when they are expecting it not to be, but now we can do it from C instead of from real mode asm running in the lower 640kB.)

This gives us about 1 MB for code, data, and stack. There is no heap as is normally understood; Etherboot does not have dynamic storage allocation. This keeps things simple, makes it a bit less prone to programming errors, and also acts as a check against unthinking use of memory by programmers. There is however a small heap which can be used to provide small allocations of memory below 640 kB.

The ROM loader copies the payload to a temporary location starting at 0x80000 and continues execution from there. The reason for this first relocation is so that execution goes faster. Typically ROM has longer access times than RAM and is often accessed 8 bits at a time. The glue chipsets used in motherboards add wait states so that the CPU can interface with relatively slow ROM. This is to reduce ROM costs, as execution speed is not important at boot time. By copying itself to RAM first, the decompression goes faster. After relocation and resumption of execution, for the x86 platform, the loader then jumps to start16, which switches into protected mode, then this jumps to start32. On other platforms, there is no need to switch modes so it jumps to the beginning of the code directly. Other loaders are used when the code is not run from ROM but they also have the ultimate action of jumping to start16.

We usually want to minimise the size of the ROM used to hold Etherboot. Even if the network adapter accommodates large ROMs, there are many claims on the area between 0xC8000 to 0xEE000, by other extension BIOSes, by peripherals that use shared memory, and so forth. Compressed images are suffixed with .z*. The payload is a .zimg file. The first part of a .zimg file is the decompressor which decompresses its own payload to the intended execution location. The decompressor is assembled from arch/i386/prefix/unnrv2b.S. The second part is the .img result of a compilation. In non-compressed images, the .img is the one and only part in the payload. Compressed .zrom images should work just the same as .rom images, but are smaller (but this saving may or may not bring it below a particular ROM chip capacity which are powers of two). The compressor is a C program called util/nrv2b.c. This is run on the host platform when building images.

Here is a diagram of a full prefix stack for the x86 architecture. Other architectures will not have the real mode to protected mode transition and will have different runtime routines and decompressor:

|                       |
| Payload               | *.img
|                       |
| Decompressor          | unnrv2b.S
|                       |
| PM runtime routines   | start32.S
|                       |
| RM to PM transition   | start16.S
|                       |
| Media specific loader | loader.S, *prefix.S

Real and protected mode

One of the complications of the x86 architecture is the existence of the real and protected modes of execution. To simplify a long story, if we want to execute 32-bit code as generated by the C compiler, we need to be in protected mode. However the processor boots up in real mode, which is 16-bit. So the loaders must execute in real mode. In addition, the main BIOS calls the extension BIOS in real mode. So at least the prologue of the main code must be in real mode. BIOS calls must also be in real mode. In Etherboot this is handled by a pair of functions (written in assembler of course) called prot_to_real and real_to_prot that do the switching. The C code doesn't call this directly. They are implicitly called from routines that use BIOS services, such as printing characters to the screen or reading characters from the keyboard. In Etherboot 5.2 and above, the 16-bit code is executed inside special segments of memory created on the fly for running real-mode code fragments, as opposed to previous versions, where they ran inline with the 32-bit code. This allows the 32-bit stack to be separate from the 16-bit one.

The file first32 establishes a 32-bit stack which is separate from the 16-bit one and uses that thereafter.