Wow, Xot, what a cool overview you made
! You are always welcome
First of all, like Xot explained it, you must know very well the hardware and OS (ANTIC, POKEY, GTIA) and their interactions with the user (interrupts, I/O, dynamic registers...). I've never really coded any serious assembler program, but i've learned it very quickly because i already know others assemblers (68000, x86) and the hardware is like any other platform (cpu and specialized processors for sound, gfx and IO).
For reference books, I personnaly love these online links:
- Mapping the Atari: http://www.atariarchives.org/mapping/
- De'Re book: http://www.atariarchives.org/dere/
The debugger of Atari 800 win PLus is your friend. It gives you conditoinal breakpoints, history of instructions and jumps, execution to the next RTS/RTI, ...
After this, and the main work, is the disassembling. And for this, there's only one IMHO: IDA Pro ("Interactive DisAssembler"). It's not a freeware, but there's an old free version to begin with. I've bought the full version because it can be scripted and is multi processor (for atari ST coool). It can analyse the entry point you give him (through a memory dump for example or from a binarie) and it decompiles everything for you, going into procedures, following the jumps (but not vectors table, too bad)... It gives you cross references and dynamic follow up too. I prefer to work with a memory dump, because you can at any moment give him another one without redoing all the work and with the datas you know from the emulator.
But to really understand all the disassembled code, you must map the addresses to their meaning in renaming them (thanks to http://www.atariarchives.org/mapping/memorymap.php
and the use of the scripting).
After that, you go to the emulator and play your next disassembled game. You must perfectly know this game, because all the difficulties came from assembling the parts of the puzzle. And the better you know the game, the easier you'll be able to understand the meaning of memory addresses and code.
With try and retry, you'll have renamed lot of addresses and procedure, and things get clearer.
For example, i've just completely decompiled "Gateway to Apshai" this two last days (wow some interesting things discovered
Concerning alternate reality, i think it will be reaaaaally difficult because of the multiple loading phase and complexity of code, but it's my opinion. I think you should begin with a simpler game, loading in 1 step.
I hope it helps you, but if you have some questions, me (and surely Xot) will be happy to hear them