Red Square has been updated and now works on real hardware. This has been tested by Jorgeche. For those of you who want to play it on your real Virtual Boy, now you can do that. The updated ROM is available from the official page. If you want to know what went into it, continue reading.
A lot of work went into debugging this issue from a lot of people, the following things about the game have changed on a technical level:
- Instruction Cache was enabled
- Wait Controller was configured
- VBlank period is now done via VIP interrupt
- VBlank interrupt is waited for using HALT
- Keypad polling was changed to initiate a hardware request at the start of a frame and poll the keys at the end of a frame
- FRMCYC was set to 0 on boot (as is done by commercial games)
All of these changes didn't fix the problem, but will provide a much greater battery life when playing the game. Specifically, changing the keypad polling method, and using HALT and the VIP interrupt. The real fix came from a very specific flag in the DPCTRL register.
The DPCTRL register has a flag called DPRST, which automatically resets the display (changing bits in INTPND and INTENB). I set this flag on hardware reset and never unset it. It turns out, this flag never gets unset. What happens when it remains set (it seems) is that every other frame is skipped because DPCTRL resets TIMEERR, FRAMESTART, GAMESTART, RFBEND, LFBEND and SCANERR in INTPND and INTENB. This was discovered when I was debugging retail games and found that their value for DPCTRL ($0302) did not match my value ($0303). My advice? Don't set DPRST, manually write to INTCLR and INTENB instead of relying on DPRST, and if you do, unset it very shortly after.
A lot of community support went into narrowing down the problem and offering solutions for this bug. The people on the PlanetVB Discord server have been extremely helpful, and I'd like to thank Jorgeche in particular for his time, knowledge, and help in testing and offering solutions to the problem.