A DOS Extender allows a program that is written for a 'protected mode' environment to run under MS-DOS. MS-DOS is a 'real mode' operating system, but with some 'tricks' it's possible to run programs meant for OS/2 1.x on MS-DOS.
Now you may be wondering, what is 'real mode', 'protected mode', and a 'DOS extender', not to mention how they all interact with each other. I'll try to explain this.
The old 8086/8088 CPUs could address 1MB of RAM on their 20-bit address bus (2^20 is 1,048,576) and this memory is chopped up into 64KB segments. The IBM PC reserves the top 384KB for adapters, hardware shared memory, and the ROM BIOS, leaving 640KB for MS-DOS and your user programs. This is where that mis-attributed quote "640KB ought to be enough for anyone" comes from. This 8086/8088 'mode' is called the 'real' mode.
Well remember leaving the old CP/M computers that typically had 48KB or 64KB of RAM, 640KB seemed plenty at the time. Naturally it was a matter of time until people were making programs where they ran out of memory.
The solution from Intel was the 80286 processor which was fully backwards compatible with the 8088/8086, however it included a new mode of operation, protected mode, which when activated would supply a virtual address space of 1GB (1,073,741,824), and a 24bit memory bus (2^24 is 16,777,216), allowing users to access 16MB of RAM.
Another big feature of protected mode is that memory is then protected so if a program accesses memory that it shouldn't, it'll generate a protection fault. The 80286 also introduced the Task State Segment (TSS) to support multi-tasking in hardware. The idea being with these two features a rouge program wouldn't crash the entire OS.
Intel's thinking was that the 8088/8086 software was a fad, and that once people saw the power of protected mode, they'd never want to leave it, so there is *NO* method for switching the 80286 back from protected mode to real mode once it's been entered.
Naturally this would require a whole new operating system to function in protected mode, and it would be incompatible with old MS-DOS programs as they simply will not function in protected mode. IBM & Microsoft started to collaborate on this new OS, called OS/2. The biggest obstacle that faced the team was that a new operating system that didn't support the growing 'legacy' base of MS-DOS applications would be doomed to fail. The IBM AT had this 'cheat' that while doing its POST in BIOS, the CPU would be switched to protected mode, to check & detect the memory. Then the instruction pointer would be saved in the start of RAM, and the keyboard controller would be used to reset the CPU, allowing the BIOS to continue back in real mode. While this did work, it was far too slow for an operating system to use where you may need to switch several thousand times a second.
Gordon Letwin spent a lot of time on trying to find a way to switch the 80286 from protected mode to real mode, when he hit upon what is known as the triple fault. The idea is that you place your instruction pointer much like the keyboard controller reset method, but you then remove all 'safety nets' by clearing the interrupt table for the CPU, and cause it to crash by throwing an interrupt. The CPU can't process the interrupt because the table is invalid, so it will end up doing a quick reset as the microcode assumes the processor has encountered a critical software problem.
This was a big save for OS/2 as now it could boot up into protected mode, run protected mode programs, and would allow the user to select a special 'DOS box' that would switch the processor into real mode, and run MS-DOS programs while switching back & forth between real & protected mode to service interrupts, and allow OS/2 programs to continue to execute in the background.
However when the first version of OS/2 was released in 1988, it had a confusing topview-like interface, which was full screen text, and didn't appear to multi task as you could only work on one thing at a time.
So with that the idea of a dos extender was born.
The people behind Tenberry worked with Microsoft, and decided that they could bring the number one benefit of protected mode, a larger address space, to MS-DOS by writing a special program, the DOS Extender. Since MS-DOS & the PC BIOS are real mode programs, and can't function in protected mode, what Tenberry did was write a special loader that could load OS/2 programs into protected mode, but every time the program would make a MS-DOS call the DOS Extender would intercept the call, switch the processor into real mode, make the call, then return to protected mode, and return the result to the program.
The result being that you didn't have to install a new operating system, just rebuild your applications, link with the DOS Extender and now you could have access to 16MB of RAM!
After seeing that the 8086 was not a passing fad, Intel then built the 80386 CPU which not only included a software method of switching from protected mode to real mode, but also included the ability to run 8086 programs in a hardware virtual machine. And since the 80386 is upwards compatible with the 80286, these DOS Extenders could continue to function, even if they relied on the triple fault.