I know your skill Xot, but before recommending tools, i must say to other people: learn how the computer work!
For the Atari 8bit serie:
- MOS Technology 6502 assembly,
- Hardware periphals accesses by the OS (rom).
For the Atari ST serie:
- Motorola 68000 assembly,
- TOS (GEMDOS, BIOS and XBIOS),
- The GEM programming (structures, function calls...).
Without this knowledge, it's like trying to understand a mathematical formula without knowing the meaning of the variables... good luck...
You don't need to know exactly how it works (sounds, player missiles, rasters), but you need to know what the code is about: "wow, here it's a sound call, ok, let's comment it, but i don't need to know what blip it does".
I have to say, i only use ressources from the net. If you need some informations about the OS or assembly language, i'll add the references.
Now about the method. I've tried several ways to decompile/understand the code. But i've found only one way which was really faster and easier: working with the smallest memory dump we can have. Thus, you avoid (according the program only loads at the beginning):
- the decrunching/decrypting routines
- the floppy disk accesses routines
This solution is entirely linked with the tool i use: an old version of IDA Pro i've bought. This tool is really... wonderful. No need to decompile everything. You search the entry point (with the debugger) and you hit the touch "c" to decompile. The software instantaneously looks for the function calls and procedures. Nice! It does 75% of the work and you help him for the remaining code (indirect function calls, self-modifying codes...).
But the hard work comes when you must understand the decompiled code! Here the Steem Engine Debugger (or Atari 800 Win) comes in handy to follow the executed code with your decompiled code in IDA. And the good news with this method: you have the same addresses for your decompiled code and your current executing code in the debugger (use heavily the "save state" of the emulator, especially if the program doesn't load at the same memory address when you boot...). Next you look at your unknown code in IDA and you put break points on execution line or read/write access in the debugger. So easy
If your aim is to reverse engineer the code, you can start again the decompilation from the binary. It will be really easy because you already know the variables/functions meaning. You only have to understand the loading routines of the program and the data (raw copy from sectors, copy from files...).
Maybe the most painful part of the decompilation is the Gem part where big structures are used to call functions. Hopefully, IDA can use structure too.
That's the basics, now it's your turn! Start from an easy game and look how easy it is