Mfcttrf

I was reading Tanenbaum's "Operating Systems, Design and Implementation" 3rd edition (The MINIX Book), where I encountered the following quote that surprised me:

For instance, the 8086 and 8088 processors do not support detection of illegal instruction operation codes, but this capability is available on the 286 and above, which trap on an attempt to execute an illegal opcode.

What did the 8086 and 8088 do when it an encountered an illegal instruction?

Illegal opcodes were just instructions that hadn't been fully defined by the chip designers – a little like Undefined Behaviour in C, but much more predictable. Many people called these "undocumented instructions", because they functioned just like ordinary instructions, on the particular versions of the particular chips on which they are found. There was no special handling to prevent these instructions from executing.

Most of them were just NOPs (either because they weren't wired up to anything or because they did stuff like writing a register to itself) or duplicates of other instructions (because the instruction decoder ignored some bits when it didn't need to pay attention to them), but some of them were more interesting.

For instance, SALC, which does:

if (carry flag set)
 AL = 0xFF;
else
 AL = 0x00;

This can be used as a translation layer between C code and certain assembly returning conventions. It is equivalent to SBB AL, AL, except that it does not clobber flags.

For more information, see:

• 
  Stephen Kitt's answer to Use of undocumented opcodes.


• 
  supercat's answer to Use of undocumented opcodes, which has a good explanation of where they come from.


• A related question on Stack Overflow

It might be better to think about it this way:

On the 286 and above, a new thing was defined called an "illegal instruction", and this new thing came with a new behavior -- a trap that was generated when one was encountered. Before the 286, there was no such thing as an illegal instruction, just undocumented ones.

Your question then becomes, "what did the 8086 and 8088 do when encountering undocumented instructions" and the answer is simply that the behavior was undocumented. Some of them appeared to do nothing (i.e. they produced the same apparent result as a NOP instruction), while others did odd things, or simply the same thing as another opcode.

Here's a WayBack link that may shed more light on the situation:
http://web.archive.org/web/20190321200321/http://www.os2museum.com/wp/undocumented-8086-opcodes-part-i/