PDP-11 stacks

Almost all PDP-11 CPU models have some form of stack overflow protection, which causes a CPU trap when the stack overflows. (LSI-11 CPUs have no overflow detection.) The details of the mechanism vary from model to model; unfortunately, the details are often documented poorly, or not at all.

Fixed Limit

The first PDP-11 CPU, the KA11 CPU of the PDP-11/20, had a fixed address limit of 0400; if the stack went below this, a trap occurred after the offending operation (instruction, trap, and probably also interrupt) was over.

The KD11-B CPU of the PDP-11/05 did the exact same thing; as do the KD11-D CPU of the PDP-11/04 and the KD11-E CPU of the PDP-11/34 (see EK-KD1EA-MM-001, pg. 2-33). KDF11 CPUs have the same mechanism (definitely on interrupts, as well; two PS/PC pairs, one from the interrupt, and one from the trap, are pushed when this happens).

KDJ11 CPUs have a similar mechanism, but although their CPU Error Register has separate Red and Yellow bits (below), interrupts when the SP is set below 0400 only set the Yellow bit, and push both PS/PC pairs; only a fault on the interrupt push causes the Red bit to be set, and then only the interrupt PS/PC are saved (in 0/2; see here).

Red/Yellow Zones and Stack Limit Registers

The most complex stack overflow protection mechanism first appeared in the KB11-A CPU of the PDP-11/45. In Kernel mode, it had a 'two-zone' scheme: the 'Yellow Zone' is a 16-word grace area, in which operations were allowed, but result in a trap after the operation is completed; in the 'Red Zone', all operations are prohibited, and result in aborting of the operation, and an immediate trap.

When Red Zone violations occur, the Stack Pointer is set to 4; the previous PC and PS are then saved in locations 0 and 2 by the resulting trap. (Odd stack addresses, and use of non-existent memory, result in identical handling.)

The address of the stack limit can be set with the Stack Limit Register register in the CPU. It is a word register (at 0777774), but the bottom byte is unused; it is cleared by a reset (e.g. when starting the CPU). The Red Zone runs up through the address given there plus 0337; the Yellow Zone starts at 0340, and runs up through 0377.

In User mode, there is a fixed Stack Overflow Boundary at 0400; attempts to write below that address cause an immediate trap.

The KB11-B CPU of the PDP-11/70 has exact same mechanism, as does the KD11-K CPU of the PDP-11/60.

The KD11-A CPU of the PDP-11/40 has the two-zone stack limitation as above, but at a fixed address of 0400. A similar Stack Limit Register is available as an option, the KJ11-A Stack Limit Register; when present, it functions identically to the others.