I'm guessing that there's a pretty simple explanation for this, but I can't seem to see it.

[White "Donald Byrne"]
[Black "Robert James Fischer"]

[fen ""]

1. Nf3 Nf6 2. c4 g6 3. Nc3 Bg7 4. d4 O-O 5. Bf4 d5 6. Qb3 dxc4 7. Qxc4 c6 8. e4 Nbd7 9. Rd1 Nb6 10. Qc5 Bg4 11. Bg5 11. Be2 followed by 12 O-O would have been more prudent. The bishop move played allows a sudden crescendo of tactical points to be uncovered by Fischer. — Wade Na4! ! 12. Qa3 On 12. Nxa4 Nxe4 and White faces considerable difficulties. Nxc3 At first glance, one might think that this move only helps White create a stronger pawn center; however, Fischer’s plan is quite the opposite. By eliminating the Knight on c3, it becomes possible to sacrifice the exchange via Nxe4 and smash White’s center, while the King remains trapped in the center. 13. bxc3 Nxe4 The natural continuation of Black’s plan. 14. Bxe7 Forking Black's Queen and Rook - Brandon_J Qb6 15. Bc4 (15. Bxf8 Rxf8 16. Bd3 (16.Be2)(16.Bc4)) Nxc3 16. Bc5 Rfe8+ 17. Kf1 Be6!! If this is the game of the century, then 17…Be6!! must be the counter of the century. Fischer offers his queen in exchange for a fierce attack with his minor pieces. Declining this offer is not so easy: 18. Bxe6 leads to a ‘Philidor Mate’ (smothered mate) with …Qb5+ 19. Kg1 Ne2+ 20. Kf1 Ng3+ 21. Kg1 Qf1+ 22. Rxf1 Ne2#. Other ways to decline the queen also run into trouble: e.g., 18. Qxc3 Qxc5 18. Bxb6 Bxc4+ 19. Kg1 Ne2+ 20. Kf1 Nxd4+ This tactical scenario, where a king is repeatedly revealed to checks, is sometimes called a “windmill.” 21. Kg1 Ne2+ 22. Kf1 Nc3+ 23. Kg1 axb6 24. Qb4 Ra4 25. Qxb6 Nxd1 26. h3 Rxa2 27. Kh2 Nxf2 28. Re1 Rxe1 29. Qd8+ Bf8 30. Nxe1 Bd5 31. Nf3 Ne4 32. Qb8 b5 Every piece and pawn of the black camp is defended. The white queen has nothing to do. 33. h4 h5 34. Ne5 Kg7 35. Kg1 Bc5+ 36. Kf1 Ng3+ Now Byrne is hopelessly entangled in Fischer’s mating net. 37. Ke1 Bb4+ 38. Kd1 Bb3+ 39. Kc1 Ne2+ 40. Kb1 Nc3+ 41. Kc1 Rc2# 0-1

Why is 15. Bxf8 not played?

Here is the chessgames link to the game.

Instead of 15... Rxf8 like in your line, black could play 15... Bxf8. Bxf8 attacks the queen, so the queen has to move. 16. Qa4 let c3 uncovered and makes 16... Nxc3 with an upcoming Re8+ possible, 16. Qc1 (the only other square for the queen) should pretty much be lost after 16... Nxc3 (the queen can't take because of Bb4). On 17. Rd2 Re8+ looks strong, and against 17. Rd3 Bb4 looks pretty strong. 16... Re8 could also be strong instead of Nxc3.
All in all the white king is very vulnerable in the middle of the board and black can play pretty strong gaining some tempo by attacking the white queen. So black would have a strong initiative for the exchange, which gives compensation at least.

Short answer: Since after the bishop recapture on f8 (and not the rook recapture!) white is tactically and positionally completely busted, with 5 active black pieces against a completely exposed king in the centre and no foreseeable chance of consolidation in order to eventually benefit from the material advantage.

First observations:

• 
  15...Bxf8 is with tempo as it attacks the queen on a3
  


• White king is still stuck in the centre and being at least two tempi away from castling, not even sidestepping from the open file with Kf1 is possible.


• With the bishop recapturing on f8, Bb4 becomes a constant threat, which means Nxc3 is unstoppable for white as Qxc3 falls to Bb4 winning the queen in all ensuing lines.


• Although white is an exchange up temporarily, one of their rooks and the f1 bishop are completely out of play still and therefore it is in fact black who effectively appears to be material up (with last piece being the a8 rook that comes into play with check in fact Re8+), not in the raw count of pieces but most definitely in terms of the positional compensation.


• Most important consequence of all these observations: black is well in time to prevent white from ever consolidating, which means even without concrete assessment of the different lines we can safely bet that everything's about to fall apart for white quite shortly after 15...Bxf8 as black's initiative on the white king seems to lead to tactically and forcibly winning variations in every scenario.

Brief concrete assessment:

Now let's look at a few candidate lines concretely (by no means exhaustive, please make sure to check these and the connected sidelines with an engine on your own)

 [title "after 15. Bxf8 Bxf8"]
 [fen "r4bk1/pp3p1p/1qp3p1/8/3Pn1b1/Q1P2N2/P4PPP/3RKB1R w K - 0 2"]

 1. Qb3 Nxc3 2. Bc4 (2. Ra1 Re8+ 3. Kd2 (3. Ne5 Rxe5+ 4. dxe5 Bb4 and the mating net is set, white's completely defenseless.) Ne4+ and black's at least winning the f3 knight after for instance Kc2 and Qa5) (2. Qxb6 axb6 3. Ra1 Bb4 4. Bc4 Ne4+ 5. Ke2 Re8 only way to avoid the discovery is to play Kd3 but after Nxf2 Nxh1 we're back to a quite similarly lost endgame compared to the 2...Nxd1 Qxb6 line) Nxd1 and black is up a clear piece as white cannot in any way recapture the knight 3. Qxb6 (3. Qxd1 Qb4+) (3. Kxd1 Qxd4+ 4. Qd3 Qa1+ and the h1 rook is lost, notice the Bxf7 trick doesn't lead to anything for white.) axb6 4. Kxd1 Ra4 5. Bb3 Rxd4+ 6. Ke2 Re4+ 7. Kd1 Bb4 and black's clearly winning considering the bishop pair, the 2 pawns up and a passed one already, the exposed white king under constant threats and the out of play white rook.

these lines clearly demonstrate why the f8 rook is in fact not hanging and why white's best chances lie in Bc4 creating Luft on f1 for the king (or even short castles if black allows on a good day) side-stepping (at first glance) most of the tactical shenanigans we just witnessed after 13...Bxf8.

 [title "15. Bc4 only attempt at consolidating"]
 [fen "r4rk1/pp2Bpbp/1qp3p1/8/3Pn1b1/Q1P2N2/P4PPP/3RKB1R w K - 0 1"]

 1. Bc4 Nxc3 2. Bc5 Rfe8+ 3. Kf1 and the rest is as you see in the real game, with Fischer's devastating attack despite the tucked away king on f1, which is just too little too late.