PDP-11 stacks

Almost all PDP-11 models have some form of stack overflow protection, which causes a CPU trap when the stack overflows. The details of the mechanism vary from model to model; unfortunately, the details are often documented poorly, or not at all.

Fixed Limit

The first PDP-11 CPU, the KA11 CPU of the PDP-11/20, had a fixed address limit of 0400; if the stack went below this, a trap occurred after the offending operation (instruction, trap, and probably also interrupt) was over. The KD11-B CPU of the PDP-11/05 did the exact same thing.

Red/Yellow Zones

The most complex stack overflow protection mechanism first appeared in the KB11-A CPU of the PDP-11/45. In kernel mode, it had a 'two-zone' scheme: the 'Yellow Zone' is a 16-word grace area, in which operations were allowed, but result in a trap after the operation is completed; in the 'Red Zone', all operations are prohibited, and result in an immediate trap.

When Red Zone violations occur, the Stack Pointer is set to 4; the previous PC and PS are then saved in locations 0 and 2 by the resulting trap. (Odd stack addresses, and use of non-existent memory, result in identical handling.)

The address of the stack limit can be set with the Stack Limit Register register in the CPU. It is a word register (at 0777774), but the bottom byte is unused; it is cleared by a reset (e.g. when starting the CPU). The Red Zone runs up through the address given there plus 0337; the Yellow Zone starts at 0340, and runs up through 0377.

In user mode, there is a fixed Stack Overflow Boundary at 0400; attempts to write below that address cause an immediate trap.

The KB11-B CPU of the PDP-11/70 has exact same mechanism. The KD11-A CPU of the PDP-11/40 has two-zone stack limitation as above, but at a fixed address of 0400; a similar Stack Limit Register is an option, the KJ11-A Stack Limit Register. When present, it functions identically to the other two.