Mfcttrf

allHexChars.txt

x01x02x03x04x05x06x07x08x09x0ax0bx0cx0dx0ex0fx10x11x12x13x14x15x16x17x18x19x1ax1bx1cx1dx1ex1fx20x21x22x23x24x25x26x27x28x29x2ax2bx2cx2dx2ex2fx30x31x32x33x34x35x36x37x38x39x3ax3bx3cx3dx3ex3fx40x41x42x43x44x45x46x47x48x49x4ax4bx4cx4dx4ex4fx50x51x52x53x54x55x56x57x58x59x5ax5bx5cx5dx5ex5fx60x61x62x63x64x65x66x67x68x69x6ax6bx6cx6dx6ex6fx70x71x72x73x74x75x76x77x78x79x7ax7bx7cx7dx7ex7fx80x81x82x83x84x85x86x87x88x89x8ax8bx8cx8dx8ex8fx90x91x92x93x94x95x96x97x98x99x9ax9bx9cx9dx9ex9fxa0xa1xa2xa3xa4xa5xa6xa7xa8xa9xaaxabxacxadxaexafxb0xb1xb2xb3xb4xb5xb6xb7xb8xb9xbaxbbxbcxbdxbexbfxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd7xd8xd9xdaxdbxdcxddxdexdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2xf3xf4xf5xf6xf7xf8xf9xfaxfbxfcxfdxfexff

allowedChars.txt

x01x02x03x04x05x06x07x08x09x31x32x33x34x35x36x37x38x39x3bx3cx3dx3ex41x42x43x44x45x46x47x48x49x4ax4bx4cx4dx4ex4fx50x51x52x53x54x55x56x57x58x59x5ax5bx5cx5dx5ex5fx60x61x62x63x64x65x66x67x68x69x6ax6bx6cx6dx6ex6fx70x71x72x73x74x75x76x77x78x79x7ax7bx7cx7dx7ex7f

How can I get this output?

x0ax0bx0cx0dx0ex0fx10x11x12x13x14x15x16x17x18x19x1ax1bx1cx1dx1ex1fx20x21x22x23x24x25x26x27x28x29x2ax2bx2cx2dx2ex2fx30x80x81x82x83x84x85x86x87x88x89x8ax8bx8cx8dx8ex8fx90x91x92x93x94x95x96x97x98x99x9ax9bx9cx9dx9ex9fxa0xa1xa2xa3xa4xa5xa6xa7xa8xa9xaaxabxacxadxaexafxb0xb1xb2xb3xb4xb5xb6xb7xb8xb9xbaxbbxbcxbdxbexbfxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd7xd8xd9xdaxdbxdcxddxdexdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2xf3xf4xf5xf6xf7xf8xf9xfaxfbxfcxfdxfexff

I've tried diff, vimdiff, sdiff, perl, awk, sed. I've tried echoing the contents of both files into one, and running the below:

perl -ne 'print unless $seen$_++' everything.txt
awk '!seen[$0]++' everything.txt 

But nothing seems to give me the output I need. Not sure if I'm just minterpreting, or if I need to specify the x as a delimiter, or replace it with something else. All I want is the delta between the two files: the hex characters that are in allHexChars.txt that don't exist in allowedChars.txt.

I don't mind how

sed -r 'H;$!d;x;s:n::g;:l;s:(\x..)(.*)1:2:;tl' allHexChars.txt allowedChars.txt > missingChars.txt

The above GNU sed script assumes two things as I understood them from the question:

• inside the files no hex character is listed more than one time


• the first file contains all the hex characters from the second file

To visualize the differences, use:

diff -y <(fold -4 allHexChars.txt) <(fold -4 allowedChars.txt)

With GNU awk for multi-char RS and RT:

$ awk -v RS='.4' 'NR==FNRa[RT];next !(RT in a)d=d RT ENDprint d' allowed all
x0ax0bx0cx0dx0ex0fx10x11x12x13x14x15x16x17x18x19x1ax1bx1cx1dx1ex1fx20x21x22x23x24x25x26x27x28x29x2ax2bx2cx2dx2ex2fx30x3ax3fx40x80x81x82x83x84x85x86x87x88x89x8ax8bx8cx8dx8ex8fx90x91x92x93x94x95x96x97x98x99x9ax9bx9cx9dx9ex9fxa0xa1xa2xa3xa4xa5xa6xa7xa8xa9xaaxabxacxadxaexafxb0xb1xb2xb3xb4xb5xb6xb7xb8xb9xbaxbbxbcxbdxbexbfxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd7xd8xd9xdaxdbxdcxddxdexdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2xf3xf4xf5xf6xf7xf8xf9xfaxfbxfcxfdxfexff

The programs diff, vimmdiff, awk and sed process the input line by line. The only way to be able to use them is to separate the values in separate lines.

Using the files from your question:

echo 'x01x02x03x04x05x06x07x08x09x0ax0bx0cx0dx0ex0fx10x11x12x13x14x15x16x17x18x19x1ax1bx1cx1dx1ex1fx20x21x22x23x24x25x26x27x28x29x2ax2bx2cx2dx2ex2fx30x31x32x33x34x35x36x37x38x39x3ax3bx3cx3dx3ex3fx40x41x42x43x44x45x46x47x48x49x4ax4bx4cx4dx4ex4fx50x51x52x53x54x55x56x57x58x59x5ax5bx5cx5dx5ex5fx60x61x62x63x64x65x66x67x68x69x6ax6bx6cx6dx6ex6fx70x71x72x73x74x75x76x77x78x79x7ax7bx7cx7dx7ex7fx80x81x82x83x84x85x86x87x88x89x8ax8bx8cx8dx8ex8fx90x91x92x93x94x95x96x97x98x99x9ax9bx9cx9dx9ex9fxa0xa1xa2xa3xa4xa5xa6xa7xa8xa9xaaxabxacxadxaexafxb0xb1xb2xb3xb4xb5xb6xb7xb8xb9xbaxbbxbcxbdxbexbfxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd7xd8xd9xdaxdbxdcxddxdexdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2xf3xf4xf5xf6xf7xf8xf9xfaxfbxfcxfdxfexff' > allHexChars.txt
echo 'x01x02x03x04x05x06x07x08x09x31x32x33x34x35x36x37x38x39x3bx3cx3dx3ex41x42x43x44x45x46x47x48x49x4ax4bx4cx4dx4ex4fx50x51x52x53x54x55x56x57x58x59x5ax5bx5cx5dx5ex5fx60x61x62x63x64x65x66x67x68x69x6ax6bx6cx6dx6ex6fx70x71x72x73x74x75x76x77x78x79x7ax7bx7cx7dx7ex7f' > allowedChars.txt

You can separate each hex value to one line with sed:

sed 's/..../&n/g' allHexChars.txt >one 
sed 's/..../&n/g' allowedChars.txt >two

You can generate the same files (each value in one line) with:

printf '\x%02xn' 1..255 > one
printf '\x%02xn' 1..9 49..57 59..62 65..127 >two

Then, get the difference with diff:

diff -y one two

Select only lines that have a < on the second column with awk:

diff -y one two | awk '$2=="<"print $1'

And remove all newlines with tr:

$ diff -y one two | awk '$2=="<"print $1' | tr -d 'n'
$ x0ax0bx0cx0dx0ex0fx10x11x12x13x14x15x16x17x18x19x1ax1bx1cx1dx1ex1fx20x21x22x23x24x25x26x27x28x29x2ax2bx2cx2dx2ex2fx30x3ax3fx40x80x81x82x83x84x85x86x87x88x89x8ax8bx8cx8dx8ex8fx90x91x92x93x94x95x96x97x98x99x9ax9bx9cx9dx9ex9fxa0xa1xa2xa3xa4xa5xa6xa7xa8xa9xaaxabxacxadxaexafxb0xb1xb2xb3xb4xb5xb6xb7xb8xb9xbaxbbxbcxbdxbexbfxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd7xd8xd9xdaxdbxdcxddxdexdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2xf3xf4xf5xf6xf7xf8xf9xfaxfbxfcxfdxfexff

A similar answer is posible with comm, but files need to be sorted first:

$ cat one | sort > one-s
$ cat two | sort > two-s

$ comm -23 one-s two-s | tr -d 'n'
x0ax0bx0cx0dx0ex0fx10x11x12x13x14x15x16x17x18x19x1ax1bx1cx1dx1ex1fx20x21x22x23x24x25x26x27x28x29x2ax2bx2cx2dx2ex2fx30x3ax3fx40x80x81x82x83x84x85x86x87x88x89x8ax8bx8cx8dx8ex8fx90x91x92x93x94x95x96x97x98x99x9ax9bx9cx9dx9ex9fxa0xa1xa2xa3xa4xa5xa6xa7xa8xa9xaaxabxacxadxaexafxb0xb1xb2xb3xb4xb5xb6xb7xb8xb9xbaxbbxbcxbdxbexbfxc0xc1xc2xc3xc4xc5xc6xc7xc8xc9xcaxcbxccxcdxcexcfxd0xd1xd2xd3xd4xd5xd6xd7xd8xd9xdaxdbxdcxddxdexdfxe0xe1xe2xe3xe4xe5xe6xe7xe8xe9xeaxebxecxedxeexefxf0xf1xf2xf3xf4xf5xf6xf7xf8xf9xfaxfbxfcxfdxfexff

But the result may not be sorted as the numeric value of the hex values (may be corrected, but becomes cumbersome).