• Welcome to Final Fantasy Hacktics. Please login or sign up.
 
March 29, 2024, 05:25:00 am

News:

Don't be hasty to start your own mod; all our FFT modding projects are greatly understaffed! Find out how you can help in the Recruitment section or our Discord!


Comprehensive Disassembly Text Files - Personal Project

Started by nitwit, March 30, 2020, 05:42:14 pm

nitwit

March 30, 2020, 05:42:14 pm Last Edit: February 15, 2022, 09:47:23 am by nitwit Reason: I suck
EDIT

I gave up on this project a few months ago on seeing how many errors there are on the wiki, and because it would be less effort just to design my own game using an existing MIT licensed 3D game engine.

If you're waiting on me to finish this to start your own hack, you can stop waiting now (if it wasn't already obvious that I had gave up.)

/EDIT

I'm posting this here on the chance that someone else is working on getting all the data tables and disassembled routines on the wiki into a form which can be assembled. Right now it's just copies of most of the disassembled routines for the files straight from the wiki.  Attached is a zipped folder with text files containing the disassembled routines.

It's more tedious than difficult, but with current circumstances being what they are I have more free time than usual. If you want to have the disassembled routines in a convenient off-line form, this would be it. If you have additional information not found in these files or one the wiki, please share or update the wiki.

Goals
  • Rip the disassembled routines from the files I haven't done yet from the wiki into the appropriate text files.
  • Add the missing routines into these text files.
  • Add data tables to the text files.
  • Make a duplicate set of files named with the *.asm extension instead of the *.txt extension, for assembling.
  • Replace all the routine calls in *.asm with labels.
  • Replace operands with labels where appropriate in *.asm; will require a bit of thought or maybe pseudo-opcodes.
  • Put data tables in the *.asm files in a form which my assembler will recognize, ideally using labels so they can be resized and altered at will.
  • Put compressed text tables, uncompressed text tables where they exist, and metadata/pointers for media assets into the files or devise some way to automate their creation given a set of inputs.
  • Do whatever else I currently don't know to get the *.asm files to assemble.
  • Apply all the essential and definitive bug-fixes, optimizations, and extensions to the *.asm files.
  • Improve comments in both *.txt and *asm files.
  • Discuss, plan, and implement new optimizations and extensions.
  • Port the codebase and assets to a platform that allows more resources for improving and expanding the codebase, data, and assets, such as the PS2 or PSP. Please note that the PSP version is hardly a real port.

It wouldn't be a bad idea to upload these files to a source host that allows incremental revision such as Github. I make no claims of copyright to any of these files.

I use the assembler ARMIPS.

This thread may be useful if it isn't already incorporated in the wiki.

As of the last edit of this post I have grabbed all the wiki entries on equip.out and require.out; they are in the zip file below. I'm going with the smallest files first. They are missing quite a few disassembled routines which weren't in or disappeared from the wiki, and the comments are sparse compared to SCUS and the battle.out file.

Let me know if you have any issues with these files.

Xifanie

Quote from: nitwit on March 30, 2020, 05:42:14 pmIt's more tedious than difficult
My ASM dyslexia says otherwise lol.

That's a really cool project though, I approve! I've always been super thankful for the wiki documentation and the more there is, the better. 🙂
I wish I could help (hell I wish I could've when I was healthy), but I'm going to leave it up to you and others!

Good luck! Great project.
  • Modding version: PSX
Love what you're seeing? https://supportus.ffhacktics.com/ 💜 it's really appreciated

Anything is possible as long as it is within the hardware's limits. (ie. disc space, RAM, Video RAM, processor, etc.)
<R999> My target market is not FFT mod players
<Raijinili> remember that? it was awful

nitwit

If anyone's wondering, I'm still working on this but it's going very slow. I'm halfway through world.bin right now. Should be done in another 12 hours of work, which for reasons of sanity I can't do all at once.

You can find out if any routines are missing by searching for the text "missing" in any given file. There are quite a few that were never disassembled, or were disassembled and lost. A handful of wiki pages seem like they existed at one point but now don't. I'll hold off on disassembling those routines and adding them until after I'm finished with all the others and after I add the data tables.

edit

It's pointless to do this manually, because it's just too much work and the wiki changes too often for such an effort to remain relevant. I need to automate it. Henceforth I will discuss how to automate this task.

I did some testing and it turns out wget works just fine with URLs containing /, *, ?, and so on, it just names the file incorrectly. I didn't think it worked before, but I was mistaken.

Solution is to write a script that calls wget and feeds it the correct name to save the web page as. Python is the ideal language for this, called via Cygwin or MinGW if you're on WIndows, as it has at least one built in URL grabber and you can just call wget from python.

Now that that is taken care of, how would you get all the pages, and in the correct order?

Q. Why does the correct order matter?
A. So we can put all the commented assembly code in the correct order.


I think the best solution is to take the source of a wiki page for a specific file, then use common sense and find/replace in notepad2 to cut as much out as possible. For example:
https://ffhacktics.com/wiki/REQUIRE.OUT
</p><p><br />
<a href="/wiki/001bf01c_-_001bf1f0" title="001bf01c - 001bf1f0">001bf01c - 001bf1f0</a>:
</p><p><a href="/wiki/001bf1f4_-_001bf224" title="001bf1f4 - 001bf224">001bf1f4 - 001bf224</a>:
</p><p><a href="/wiki/001bf228_-_001bf250" title="001bf228 - 001bf250">001bf228 - 001bf250</a>:
</p><p><a href="/wiki/001bf254_-_001bf2a8" title="001bf254 - 001bf2a8">001bf254 - 001bf2a8</a>:
</p><p><a href="/wiki/001bf2ac_-_001bf30c" title="001bf2ac - 001bf30c">001bf2ac - 001bf30c</a>:
</p><p><a href="/wiki/001bf310_-_001bf370" title="001bf310 - 001bf370">001bf310 - 001bf370</a>:
</p><p><a href="/wiki/001bf374_-_001bf798" title="001bf374 - 001bf798">001bf374 - 001bf798</a>:

By replacing this string with nothing, I get just parts of the URLs to these pages (the exception being the first URl, which I had to do manually):
</p><p><a href="
Now I have this:
/wiki/001bf01c_-_001bf1f0" title="001bf01c - 001bf1f0">001bf01c - 001bf1f0</a>:
/wiki/001bf1f4_-_001bf224" title="001bf1f4 - 001bf224">001bf1f4 - 001bf224</a>:
/wiki/001bf228_-_001bf250" title="001bf228 - 001bf250">001bf228 - 001bf250</a>:
/wiki/001bf254_-_001bf2a8" title="001bf254 - 001bf2a8">001bf254 - 001bf2a8</a>:
/wiki/001bf2ac_-_001bf30c" title="001bf2ac - 001bf30c">001bf2ac - 001bf30c</a>:
/wiki/001bf310_-_001bf370" title="001bf310 - 001bf370">001bf310 - 001bf370</a>:
/wiki/001bf374_-_001bf798" title="001bf374 - 001bf798">001bf374 - 001bf798</a>:

This is a little harder There's no way to wildcard string replacement in notepad2, but I can replace part of this with newline characters.

Part replaced:
" title="
Result:
/wiki/001bf01c_-_001bf1f0
001bf01c - 001bf1f0">001bf01c - 001bf1f0</a>:
/wiki/001bf1f4_-_001bf224
001bf1f4 - 001bf224">001bf1f4 - 001bf224</a>:
/wiki/001bf228_-_001bf250
001bf228 - 001bf250">001bf228 - 001bf250</a>:
/wiki/001bf254_-_001bf2a8
001bf254 - 001bf2a8">001bf254 - 001bf2a8</a>:
/wiki/001bf2ac_-_001bf30c
001bf2ac - 001bf30c">001bf2ac - 001bf30c</a>:
/wiki/001bf310_-_001bf370
001bf310 - 001bf370">001bf310 - 001bf370</a>:
/wiki/001bf374_-_001bf798
001bf374 - 001bf798">001bf374 - 001bf798</a>:

Now every other line starting with the first is a partially correct URL, and the rest are trash. Very easy to cut out the lines, just write a little Python script to iterate through it and only print the odd numbered lines. Actually now that I think about it, you don't even need a script for smaller files, just paste it into a spreadsheet, highlight the offending rows, delete them, then paste them back into notepad2. This won't scale well with even a medium sized file, but it's good to know and faster than writing code.

Result:
/wiki/001bf01c_-_001bf1f0
/wiki/001bf1f4_-_001bf224
/wiki/001bf228_-_001bf250
/wiki/001bf254_-_001bf2a8
/wiki/001bf2ac_-_001bf30c
/wiki/001bf310_-_001bf370
/wiki/001bf374_-_001bf798

Now I just use the replace dialog again to prepend the rest of the wiki URL to the beginning of each line, and I'm done (I could also do this in the Python script in the previous step, but learning what you can do with find/replace in a text editor is more important).

Find (make sure you have the "transform backslashes" checkbox checked in notepad2):
\r\n
Replace:
\r\nhttps://ffhacktics.com/
Result:
https://ffhacktics.com//wiki/001bf01c_-_001bf1f0
https://ffhacktics.com//wiki/001bf1f4_-_001bf224
https://ffhacktics.com//wiki/001bf228_-_001bf250
https://ffhacktics.com//wiki/001bf254_-_001bf2a8
https://ffhacktics.com//wiki/001bf2ac_-_001bf30c
https://ffhacktics.com//wiki/001bf310_-_001bf370
https://ffhacktics.com//wiki/001bf374_-_001bf798
https://ffhacktics.com/

Whoops, looks like I left a stray newline at the end of the file, but no big deal, just delete it.

Now I could write a python script that accepts a text file with this list of URLs as input, and have it wget every URL in sequence... with an appropriate wait of let's say 15 to 45 seconds so the server doesn't experience issues.

BTW, you should only run such a script with the staff's permission, and I probably won't post the python/wget script publicly unless I also donate some money to server costs.

I looked at the source pages for all the files on the wiki, and this part (massaging the web pages for each file's list of commented disassembly routines) should work for all of them. It can't be completely automated because each page is slightly different, but as long as your computer is powerful enough to hold a 5 MB text files in memory it shouldn't be an issue.

A few files aren't worth the effort to automate, as they have less than a dozen pages that you can just manually copy and paste, presumably interspersed with data.

Once I get everything in one place, I'll upload them here and ask Glain to upload them to the FFH github, and after that improvements to the comprehensive disassemblies can be made by anyone.

TODO: add the little Python script to only print odd lines, then do the next step.