Donald E. Knuth has included a complete list of the syntax and semantics of DVI commands (opcode, symbolic name, number of parameters, meaning) in Computers & Typesetting Vol. E, §585, pp. 236–239. Where I can find a similar list for the additional commands in the 250–255 range, used by XeTeX?

Available information

Here is what I got (the information comes from https://github.com/simoncozens/dvipdfm-x/blob/libtexpdf/dvicodes.h):

• 
  begin_reflect 250. (No parameters???) TeX-XeT begin reflect


• 
  end_reflect 251. (No parameters???) TeX-XeT end reflect


• 
  xdv_native_font_def 252. Parameters??? Syntax???


• 
  xdv_glyphs 253. Parameters??? Possible values???


• ??? 254. ???


• 
  ptexdir 255. Parameters??? Possible values???

Needless to say, the information in the dvicodes.h file is lacunary. Can anyone complete this? (meaning of each command, number of parameters—if any, list of possible values)

Trying to decypher

I opened a very simple XDV file, obtained from the following TeX code:

documentclassarticle
usepackagepolyglossia
setdefaultlanguage[calendar=gregorian,hijricorrection=1,locale=mashriq]arabic
setmainfont[Script=Arabic,Extension=.ttf]Amiri-Regular
begindocument
Bonjourspecialligne x
beginArabicيانيسendArabic
Bonjour
enddocument

In the XDV file, some info is in specials (command xxx), that's fine. Then there is a xdv_native_font_def command followed (a) by the following bytes:

00 00 00 1E 00 0A 00 00 00 00

(b) by letter L and the absolute path to the font, (c) another four 00 bytes. Then the font 30 is called (although it has not been defined in the standard fnt_def way).

Then comes a xdv_glyphs command, followed by 2 bytes: 00 and 1F (???) and then another fnt_def command for font 50 (???) and then what looks like a lot of set_char commands for glyph 00. Since I don't know the xdv_native_font_def syntax, I probably got it all wrong.

What is really strange is that the Bonjour string of the file does never appear in the XDV file (contrarily to DVI files, where ASCII glyphs are assigned to opcodes having the numeric value of the ASCII position). Where the Bonjour string should have been, there is this byte string:

00 25 00 52 00 51 00 4d 00 52 00 58 00 55

where I can only conclude that 0025 is letter B (whose ASCII code is 0x42 and not 0x25), 0052 is letter o (whose ASCII code is 0x6F and not 0x52). Where on Earth comes the assignment B → 0x25, o → 0x52 from?

The xdv_native_font_def command appears again in the postamble. Although part of the document is in right-to-left mode, I don't see anywhere the begin_reflect and end_reflect commands, but not knowing the xdv_native_font_def syntax probably scrambles my analysis of the whole XDV file.

So any help will be greatly welcome!

For those who have no Unix tools handy, here is the hexdumped test file:

00000000 f7 07 01 83 92 c0 1c 3b 00 00 00 00 03 e8 1d 20 |.......;....... |
00000010 58 65 54 65 58 20 6f 75 74 70 75 74 20 32 30 31 |XeTeX output 201|
00000020 39 2e 30 36 2e 31 36 3a 32 31 31 30 8b 00 00 00 |9.06.16:2110....|
00000030 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 ff ff ff ff ef 14 70 64 66 3a 70 |...........pdf:p|
00000060 61 67 65 73 69 7a 65 20 64 65 66 61 75 6c 74 a0 |agesize default.|
00000070 02 79 00 00 8d a0 fd a3 00 00 a0 02 3f 00 00 8d |.y..........?...|
00000080 a0 fd e4 00 00 8d 92 01 2c 53 40 ef 61 70 64 66 |........,S@.apdf|
00000090 3a 64 6f 63 69 6e 66 6f 3c 3c 2f 42 49 44 49 2e |:docinfo<</BIDI.|
000000a0 46 75 6c 6c 62 61 6e 6e 65 72 28 54 68 69 73 20 |Fullbanner(This |
000000b0 69 73 20 74 68 65 20 62 69 64 69 20 70 61 63 6b |is the bidi pack|
000000c0 61 67 65 2c 20 56 65 72 73 69 6f 6e 20 33 35 2e |age, Version 35.|
000000d0 37 2c 20 52 65 6c 65 61 73 65 64 20 4d 61 72 63 |7, Released Marc|
000000e0 68 20 33 2c 20 32 30 31 39 2e 20 29 3e 3e 96 02 |h 3, 2019. )>>..|
000000f0 ee 00 fc 00 00 00 1e 00 0a 00 00 00 00 13 2e 2f |.............../|
00000100 41 6d 69 72 69 2d 52 65 67 75 6c 61 72 2e 74 74 |Amiri-Regular.tt|
00000110 66 00 00 00 00 c9 fd 00 1f dd 80 00 07 00 00 00 |f...............|
00000120 00 00 00 00 00 00 05 d2 00 00 00 00 00 00 0a 98 |................|
00000130 80 00 00 00 00 00 0f ca 80 00 00 00 00 00 12 20 |............... |
00000140 00 00 00 00 00 00 17 18 80 00 00 00 00 00 1c 20 |............... |
00000150 00 00 00 00 00 00 25 00 52 00 51 00 4d 00 52 00 |......%.R.Q.M.R.|
00000160 58 00 55 93 fd 00 13 27 c0 00 05 00 00 00 00 00 |X.U....'........|
00000170 00 00 00 00 0a ca 80 00 00 00 00 00 0d 3b 80 00 |.............;..|
00000180 00 00 00 00 0e f4 c0 00 00 00 00 00 11 40 40 00 |.............@@.|
00000190 00 00 00 06 5e 0e 9b 0e 7c 04 ef 05 2d 93 ef 07 |....^...|...-...|
000001a0 6c 69 67 6e 65 20 78 fd 00 1f dd 80 00 07 00 00 |ligne x.........|
000001b0 00 00 00 00 00 00 00 05 d2 00 00 00 00 00 00 0a |................|
000001c0 98 80 00 00 00 00 00 0f ca 80 00 00 00 00 00 12 |................|
000001d0 20 00 00 00 00 00 00 17 18 80 00 00 00 00 00 1c | ...............|
000001e0 20 00 00 00 00 00 00 25 00 52 00 51 00 4d 00 52 | ......%.R.Q.M.R|
000001f0 00 58 00 55 8e 8e 9f 1e 00 00 8d 92 00 e7 92 00 |.X.U............|
00000200 fd 00 05 dc 00 00 01 00 00 00 00 00 00 00 00 01 |................|
00000210 be 8e 8e 8c f8 00 00 00 2c 01 83 92 c0 1c 3b 00 |........,.....;.|
00000220 00 00 00 03 e8 02 79 f2 7f 01 97 00 00 00 03 00 |......y.........|
00000230 01 fc 00 00 00 1e 00 0a 00 00 00 00 13 2e 2f 41 |............../A|
00000240 6d 69 72 69 2d 52 65 67 75 6c 61 72 2e 74 74 66 |miri-Regular.ttf|
00000250 00 00 00 00 f9 00 00 02 14 07 df df df df df df |................|
00000260

Not a lot better than the source you gave but the xetex output is read by xdvipdfmx and the source for that in texlive svn has dvicodes.h which has

 /* XeTeX ".xdv" codes */
#define XDV_NATIVE_FONT_DEF 252 /* fontdef for native platform font */
#define XDV_GLYPHS 253 /* string of glyph IDs with X and Y positions */
#define XDV_TEXT_AND_GLYPHS 254 /* like XDV_GLYPHS plus original Unicode text */

#define PTEXDIR 255 /* Ascii pTeX DIR command */

These are handled by dvi.c in the same directory, but I guess your C is better than mine:-)

There are some comments as to the expected byte layout in the C eg

case XDV_GLYPHS:
 need_XeTeX(opcode);
 get_and_buffer_bytes(fp, 4); /* width */
 len = get_and_buffer_unsigned_pair(fp); /* glyph count */
 get_and_buffer_bytes(fp, len * 10); /* 2 bytes ID + 8 bytes x,y-location per glyph */
 break;
case XDV_TEXT_AND_GLYPHS:
 need_XeTeX(opcode);
 len = get_and_buffer_unsigned_pair(fp); /* utf16 code unit count */
 get_and_buffer_bytes(fp, len * 2); /* 2 bytes per code unit */
 get_and_buffer_bytes(fp, 4); /* width */
 len = get_and_buffer_unsigned_pair(fp); /* glyph count */
 get_and_buffer_bytes(fp, len * 10); /* 2 bytes ID + 8 bytes x,y-location per glyph */
 break;
case XDV_NATIVE_FONT_DEF:
 need_XeTeX(opcode);
 do_native_font_def(get_signed_quad(dvi_file));
 break;

dviasm (despite its name) can show xdv files, I had to change the font loading to find the font, but on your test file it reports

[preamble]
id: 7
numerator: 25400000
denominator: 473628672
magnification: 1000
comment: ' XeTeX output 2019.06.16:2127'

[postamble]
maxv: 633.947250pt
maxh: 407pt
maxs: 3
pages: 1

[font definitions]
fntdef: "/usr/local/texlive/2018/texmf-dist/fonts/truetype/public/amiri/amiri-regular.ttf" at 10pt

[page 1 0 0 0 0 0 0 0 0 0]
xxx: 'pdf:pagesize default'
down: 633pt
push:
 down: -605pt
 down: 575pt
 push:
 down: -540pt
 push:
 right: 300.325195pt
 xxx: 'pdf:docinfo<</BIDI.Fullbanner(This is the bidi package, Version 35.8, Released May 1, 2019. )>>'
 w: 2.929688pt
 fnt: "/usr/local/texlive/2018/texmf-dist/fonts/truetype/public/amiri/amiri-regular.ttf" at 10pt
 setglyphs: 31.865234pt gid37(0pt) gid82(5.820312pt) gid81(10.595703pt) gid77(15.791016pt) gid82(18.125000pt) gid88(23.095703pt) gid85(28.125000pt)
 w0:
 setglyphs: 19.155273pt gid2388(0pt) gid4497(10.791016pt) gid4466(13.232422pt) gid2021(14.956055pt) gid2083(17.250977pt)
 w0:
 xxx: 'ligne x'
 setglyphs: 31.865234pt gid37(0pt) gid82(5.820312pt) gid81(10.595703pt) gid77(15.791016pt) gid82(18.125000pt) gid88(23.095703pt) gid85(28.125000pt)
 pop:
 pop:
 down: 30pt
 push:
 right: 231.570312pt
 setglyphs: 5.859375pt gid447(0pt)
 pop:
pop:

With the tex file you gave apart from changing the font line to

setmainfont[Script=Arabic]Amiri

Based on the dvisvgm sources and dvipdfm-x sources: The truly XDV-specific opcodes are (as of the current Version 7) only three:

• 
  

  252 (fc): This is to define a font (code refers to it as XDV_NATIVE_FONT_DEF or XFontDef), and is the most complicated of the three. Parameters are:

  
  
  

  fontnum[4] ptsize[4] flags[2] psname_len[1] fontname[psname_len] fontIndex[4]

  
  
  

  followed by up to 16 more bytes depending on the flags.

  
  


• 
  

  253 (fd). This is a “string of glyph IDs with X and Y positions”, referred to in code by XDV_GLYPHS or XGlyphArray. Parameters are:

  
  
  

  w[4] n[2] xy[(4+4)n] g[2n]

  
  
  

  where w is the total width of the glyph array, n is the number of glyphs, xy is a sequence of (dx, dy) pairs (the relative horizontal and vertical positions of each glyph), and g contains the “FreeType indices of the glyphs to typeset”.

  
  


• 
  

  254 (fe): This is similar except it includes “a leading array of UTF-16 characters that specify the "actual text" represented by the glyphs to be printed. It usually contains the text with special characters (like ligatures) expanded so that it can be used for text search, plain text copy & paste etc. This XDV command was introduced with XeTeX 0.99995 and can be triggered by XeTeXgenerateactualtext1”. So its parameters are:

  
  
  

  parameters: l[2] t[2l] w[4] n[2] xy[8n] g[2n]

  
  

I don't think the TeX-XeT commands 250–251 nor the pTeX command 255 are used by XeTeX, which is consistent with you not seeing them in the file.

The hexdump in the question starts with f7 = 247, the DVI “pre” command, and the next byte is the DVI version, which here is 07. So we're looking at (XDV) version 7, as expected.

So in your file when you see (at either byte offset 278 or 423) bytes like fd 00 1f dd 80 00 07 00 00 00 and so on, it's actually not just two bytes that are the parameters, but rather 00 1f dd 80 are w, then 00 07 are n (the number of glyphs), then the next 56 bytes are xy or (dx, dy) (the offsets for each of these 7 glyphs), then the next 14 bytes are g or glyphs (the 7 glyphs). Needless to say, these 7 in your example are Bonjour:

 00 25 00 52 00 51 00 4d 00 52 00 58 00 55

As you observe, these are not ASCII codes, so where does this mapping of 00 25 to B, etc come from? Well it's the same as with the regular DVI format: these are the positions of the glyphs in the font, and the font can choose to put any glyph at any position. This is confirmed by opening the font and counting positions: maybe FontForge can show it but I couldn't find it in the UI, but I could find it with fonttools:

$ ttx amiri-regular.ttf 
Dumping "amiri-regular.ttf" to "amiri-regular.ttx"...

and the file contains:

 <GlyphID id="37" name="B"/>

where 37 is 0x25, etc.

The definitive source is the XeTeX source tree, and specifically the xetex.web file. Quoting from it:

yskipnoindent Commands 250--255 are undefined in normal .DVI
files, but the following commands are used in .XDV files.

yskiphangvboxhalign

yskiphang|set_glyphs| 253 |w[4]| |k[2]| |xy[8k]| |g[2k]|.

yskiphang|set_text_and_glyphs| 254 |l[2]| |t[2l]| |w[4]| |k[2]|
|xy[8k]| |g[2k]|.

yskipnoindent Commands 250 and 255 are undefined in normal .XDV
files.