• Welcome to Final Fantasy Hacktics. Please login or sign up.
 
March 28, 2024, 08:51:30 am

News:

Use of ePSXe before 2.0 is highly discouraged. Mednafen, RetroArch, and Duckstation are recommended for playing/testing, pSX is recommended for debugging.


Formula mapping

Started by Orkney, December 19, 2020, 11:45:11 am

Orkney

Hi everybody. (and sorry for the poor english)

I'm submitting an excel sheet at an alpha stage, but quite functionnaly. Mostly because i'm losing faith in this project.

At the beginning i want to learn ASM hacking, and i was very pleased by Xifanie videos tutorial. But at a moment in the video, she's looking for a formula that will fit her project (physical damage). At this stage i ask myself how will i pick the good formula for my own project. I start digging in the wiki and a went quite lost.

So i decided to make a sheet with all the formulas in order to quickly know when a formula his affected by zodiac compatibility, elemental modification, physical damage reduction etc...

I've a hard time doing it and i decided to start a map. I went quite mad about it (jal jal jal, then vba vba vba)

Today all the formulas and the routine are on the sheet, they are connected to each other. Even if the very right of the sheet is not clean (because those are the unknown routines, so i was less motivated)
-There is an arrow on each of them that link to the wiki of each of them.
-When you click on a formula/routine, all the non connected routines vanish (if not, the sheet is not usable)
-If you click again on the same routine the sheet return to it base stage (the "what the f...?" stage)
-If you click on another routine (visible so connected to the first) it will either reboot the visible routine (as if you click it at the first time) or either add to the visible routines all those which are connected to the one you just click (i'm quite sure i'm mad at this time). If the word "ADD" is in the cell B1 you'll be in the second case.

I was planning to add another function to color the formulas and routines.
I tried to keep the routine in the order they are called. (that why some link are colored or not). It's probably not totaly acurate at this time.
There are probably some bug too (with connectors to nothing, because missnamed some of the connector (there are a lot of them !!!)
The VBA is probably not ultra efficient (mostly the remaping of the shape) and it could take a few seconds to work - mostly with the first formulas)
And the présence of arrow or not on the links, looks totaly random.

At this stage, i need to know if i'm trying to do something that will help to ASM coding formulas or if i'm (definitively) loosing my time here... Maybe some suggestions will help me to move on this project (or drop it)

Here is an image of what happen when you click on the formula 0C



I was drive to put the routines on different column to avoid multiple link direction, the the jal always go to the right (not true at the very right, but it's understandable i guess)

FORM ALPHA.xlsm
  • Modding version: PSX

nitwit

What is your goal?

I'm working on other games right now, but you should skip the spreadsheets (unless Raven is willing to help, then you should continue with this) and instead ask for a copy of the Wiki's SQL database (minus the user login credentials) and try to make a commented disassembly.

I started working on it here:
https://ffhacktics.com/smf/index.php?topic=12443.msg227843#msg227843

... but I won't be back to it for probably a few more years.

If you have experience with Python or a similar scripting language, it shouldn't be hard to do.  Here's the order of operations:
1. Get all the stuff in the SQL database into text files corresponding to each of the main game files (SCUS, BATTLE.BIN, etc).
2. Everything that can be replaced with a pseudo-opcode should be, as this allows you to use labels to identify subroutines and data structures rather than addresses... which means you can rearrange data and code as you see fit, without needing to manually recalculate things.
3. Replace all hard-coded addresses operands with labels.
4. Some other stuff I can't remember off the top of my head.

All of this can be easily automated with a scripting language built to handle text and judicious use of revision control software to unscrew mistakes as you find them. It makes absolutely no sense to try to do it by hand.

Good luck.

BTW, here's a useful tool for translating things:
https://www.deepl.com/translator

Orkney

Hi and thank you for your answer

I'm trying to make a tool that gives an overview of how the routines interact.

For example, I'm looking at changing the way bravery and faith impact routines, so I need to know what routines are impacted and how they are impacted. In the file I made, if I click on the faith-routine, I will see all the formulas related to this routine (directly or via another routine)

I have very little experience with python, a little more with Excel (experience, no formation). I've downloaded your file, and I'll take a look at it to see if I can find a way to use it.


Thanks for the translation link!
  • Modding version: PSX

nitwit

I looked at it, and while it doesn't work for me (Libre Office - I don't pay for software) it seems that you click on a formula and all subroutines (and their subroutines, and so on) would appear or be highlighted, to make it obvious what is used where.

Seems pretty useful, though a nightmare to code by hand. I assume you wrote that in VBA, or in a scripting language for Excel? I don't have much to contribute or any ways I can think to help you, since all the ways I could automate it require me to interface with the wiki*.

What exact changes do you want to make? Do your wished-for changes depend on what is feasible? Or are you trying to map out how to do the things you want?

My project to rip the commented disassemblies from the wiki dead-ended when it became apparent that the wiki allows non-standard URL characters. Links with '/' in them would be interpreted as another directory instead of a link name, making wget throw an error. It's especially difficult to debug because you must go veeeeeerrry slow to avoid DDOSing the wiki and running up their bandwidth bill, which no one wants. I haven't tried curl or other Linux apps, and I'm afraid I'll need to spin my own python web crawler. It will be doable, but frustrating, so until then I'll work on building a labeled and pseudo-opcoded disassembly straight from a game binary and release the tools I used to do so.

My files aren't terribly useful and are out of date, as I made them by manually copying and pasting text from the wiki. It's not feasible even for smaller files.

* Which isn't feasible due to the non-standard characters in the wiki URLs and the inadvisability of publicly releasing what amounts to a DDOS tool specifically for the wiki.

Orkney

Thanks for looking at it.

Yes I did it in VBA with Excel and a lot of work by hand as well. Since I saw that the VBA language for excel and the office language were different, I gave up the idea of relearning how to write macros.

Since it takes me a long time, I wanted to get some feedback. To find out if I was doing something that could be useful to someone else or not.

I may be considering doing it for something other than formulas (although to this day I have no idea what routines are involved in a fight, or on the roster).

So I'm not particularly looking for help, unless this beginning of work gives someone here bigger ideas (but that would be me helping him at this point).

My skills are relatively limited, and I don't understand much about what you are trying to do on your side... (sorry).

But you have partly answered my question by telling me that you think it is quite useful. The file has been downloaded 6 times (!!) so the subject seems to interest a few people (the current tool maybe less). I will take a little more time to make it more presentable.

  • Modding version: PSX

EnderC

I have a feeling Orkney, that you might be after the same goal that I am: better understanding of the codebase. The wiki is useful when you want to reference an individual routine, but it's very difficult to see the bigger picture.

I'm going through the same pain as you and I'm trying to find a better way to explore the code.

Nitwit's suggestion is a good one, to scrape the wiki and (most importantly) attach labels to it. It took looking at some of Pride and Clain's patches that I realized labels were a thing. See this code and how much easier it is to reason over:

<Location file="BATTLE_BIN" offset="15C9F0" mode="ASM" offsetMode="RAM">
    @text_delay_hook:
            .eqv    %max_normal_delay, 2
            .eqv    %min_hard_stop, 45

            li      t2, %max_normal_delay
            sltiu  t1, t0, %min_hard_stop
            beq    t1, zero, text_delay_hook_end
            sltu    t1, t0, t2
            bne    t1, zero, text_delay_hook_end
            nop

            move    t0, t2

        text_delay_hook_end:
            jr      ra
            nop
</Location>
<Location file="BATTLE_BIN" offset="132718" mode="ASM" offsetMode="RAM">
    jal    @text_delay_hook
</Location>

As an aside, it's really handy that you can reference the label even across <Location> elements! Had no idea!

There is a lot going on here that would help me read and write assembly that I wasn't even aware of, and it would certainly help understanding larger sections of code if I had labelled it all up like this!

If I could hijack the thread just for a sec, I think this would be useful, but I have a few questions:

What's the difference here between @text_delay_hook: syntax and the later text_delay_hook_end:?

Why are the registries written as t1, t0 instead of r1, r0? (does it matter?)

Is .eqv an op or is it defining variables that get compiled away?

Sorry for interjecting with questions of my own, but I think there's value here. If not I can start a new thread, nbd.
  • Modding version: PSX
  • Discord username: EnderC

Glain

For some of these questions, it would make sense to look over the MassHexASM thread, which explains some of this.  In short:

The "@" in front of a label means it persists across Location tags.
t1 and t0 are names for the first two "temp" registers (and actually refer to $9 and $8, respectively). Check out the calling conventions.
.eqv is a preprocessor instruction that defines a simple macro which replaces the first string with the second before actually encoding the block.
  • Modding version: Other/Unknown

nitwit

Adding labels* is only part of the solution, you also need pseudo-opcodes.

https://ffhacktics.com/wiki/Dark/Confuse_Caclulation
00185328: 3c028019 lui r2,0x8019
0018532c: 8c422d94 lw r2,0x2d94(r2) Load Attacker's Data Pointer
It isn't possible to add a label here, because the address is split across two lines. However, one could write a python script to iterate through a text file line by line, and when it encounters a lui operation followed by a lw operation, it could replace them with something that the assembler will later split into a lui and lw. This allows you to call this new thing with a complete address... and you can substitute a label for that address.

There are a few edge cases (when the second line is a negative number, and therefore subtracts from the first address after the first address is bit-shifted right I think  32 bits) but these are literally like less than 20 lines of python code. Not hard at all. The only hard part is figuring out how to get it from the wiki into text files, and periodically adding changes from the wiki to it; or somehow syncing the wiki with a git repository, though tbh the wiki will probably die of neglect once you have a commented, labeled, and pseudo instructed disassembly as the latter is much more valuable.

Here's a list of pseudo instructions (I used the wrong term earlier):
https://github.com/MIPT-ILab/mipt-mips/wiki/MIPS-pseudo-instructions

Text manipulation is very, very easy to do even for novice programmers.

*which can be largely automated with a set of simple python scripts if you have all the routines in text files.

Orkney

Well, I'm completely out of it...

I don't understand much of what you're saying.

You're talking about formatting the raw code of the wiki in a form that's more convenient to program?

I was just trying to format it to understand it better (at my humble level), as EnderC suggested.
At this stage I haven't written any ASM code yet, I've only read most of the tutorials and links on the site.

I understand that you want to give names to groups of lines (like the confusion example), so that you can then call them up and have them substituted again by the original code . But how can we create these labels (in what form? in what programme?), and above all how can we use them again?

  • Modding version: PSX

EnderC

Quote from: Orkney on December 22, 2020, 02:31:14 pmYou're talking about formatting the raw code of the wiki in a form that's more convenient to program?

Correct. Nitwit seems to be suggesting you write a script to fetch all the code on the wiki and format it for you. That might be putting the cart before the horse IMO.

Quote from: Orkney on December 22, 2020, 02:31:14 pmI understand that you want to give names to groups of lines (like the confusion example), so that you can then call them up and have them substituted again by the original code . But how can we create these labels (in what form? in what programme?), and above all how can we use them again?

ASM code is still code. You can use whatever text editor you'd like, though I'd suggest something that supports color formatting. I'm doing something similar with the Damage Formula code in VSCode myself:


I found this very useful in helping me understand what was going on, and you'll get insights from doing this bit by bit that you wouldn't if you were handed the output of the entire wiki.
  • Modding version: PSX
  • Discord username: EnderC

nitwit

December 23, 2020, 05:15:36 am #10 Last Edit: December 23, 2020, 05:34:59 am by nitwit
Quote from: Orkney on December 22, 2020, 02:31:14 pmWell, I'm completely out of it...

I don't understand much of what you're saying.
What is your native language? I may be able to find some resources for you in your language so we can leap the language barrier. Besides, there are a few speakers of other languages here who can help you.

Quote from: Orkney on December 22, 2020, 02:31:14 pmYou're talking about formatting the raw code of the wiki in a form that's more convenient to program?
Yes. Besides that:
1. There are a lot of tools for different operating systems to deal with text files.
2. It's the easiest format to write your own programs for.
3. Programmer's tools like Integrated Development Environments (IDEs) or programmer's text editors allow you to highlight syntax, do simultaneous editing, and fold code, as well as quickly navigate through a file with the Find and/or Replace dialogs.

Quote from: Orkney on December 22, 2020, 02:31:14 pmI was just trying to format it to understand it better (at my humble level), as EnderC suggested.
At this stage I haven't written any ASM code yet, I've only read most of the tutorials and links on the site.
I apologize for coming on so strong. I'm just sharing my own findings in case anyone else is interested.

It's your life, and you should do what works best for yourself. If that's Excel spreadsheets, by all means keep doing as you are; your personal goals should matter more to you than my goals and failures.

You can do quite a lot with spreadsheets, so it's not a dead end by any means.

Quote from: Orkney on December 22, 2020, 02:31:14 pmI understand that you want to give names to groups of lines (like the confusion example), so that you can then call them up and have them substituted again by the original code . But how can we create these labels (in what form? in what programme?), and above all how can we use them again?
You could use anything that edits a text (*.txt) file. Notepad would work, though it would be painful to use with so few features.

I use the program Notepad2 to do most of my work - it is a very fast but has everything I need for most tasks - though Notepad++ (which is more like an IDE) is good as well.

I have like 20 different text editors installed on my primary computer, from back when I was testing out different ones to see which I liked. I couldn't tell you what most of them do without spending a day testing them all again (and actually taking notes this time), but there are plenty of options for Windows, Mac, and Linux.

Labels are useful because without them you need to manually tell the assembler (the program or tool which turns the code on the wiki into bytes that run on a PS1 emulator) what address to jump to for jal opcodes and various other opcodes. Needing to manually calculate all that stuff is a nightmare, though I assume you could do it in a spreadsheet (for all of BATTLE.BIN it would be an enormous spreadsheet, you'd probably need tens of gigabytes of RAM to open it) and have the spreadsheet automatically generate these addresses based on a starting address you input somewhere.

Pseudo-instructions are to my limited knowledge useful primarily because they let you use labels.

If you don't use labels or automatically generate addresses, you need to manually edit them whenever you add or delete a line of assembly code, or you are stuck keeping everything where it currently is. This is very limiting. Right now it's dealt with by jumping to free space somewhere, doing stuff there, and then jumping back; but that wastes I think 2 lines of code at least, and there's a limited amount of free space.

With labels and pseudo-instructions, we would never need to jump to free space, and we would have fewer limits on what we could do, and editing assembly would be considerably easier.

Orkney

Thank you both for your explanations.

I was just trying to get an overview of the execution of the different routines and the implication of the different parameters of the game (dodge, parry, affinity etc).

But what you're talking about is also very interesting (I understand a bit better now). I don't know much about it (for the moment?).

I'm French (I thought it must be obvious from the way I write in English :) ). And I'm looking for a possible link to see if I can consider diving into it.

Having said that, Niwit was talking about starting from the wiki, I found 2-3 errors while making my file. I don't know if it will impact the extraction to the text file or not, I plan to make a post to bring up what I found (time to settle down and find or do it).
  • Modding version: PSX

nitwit

I noticed a few issues with the wiki too. Some routines overlap with others, some are missing, and some are duplicates.

It's not a serious issue for my purposes until I get around to getting them all in one place, and even then all I would need to do is disassemble the file in which it's found and add that routine.

It could be a serious issue if someone is trying to do something with that routine. It will all eventually get disassembled and dumped somewhere, probably when someone is working on a mod and needs a hack that modifies that routine and an assembly language hacker is interested in it.

Reasons a section of RAM may be missing from the routine locations for a file:
  • The missing area may not be a routine, but a table. These are the things that FFTPatcher edits.
  • The missing area may be empty space, junk that is never called anywhere, or malfunctioning code that makes no sense and can be safely removed.

Orkney

Quote from: Orkney on December 26, 2020, 02:39:48 pmAnd I'm looking for a possible link to see if I can consider diving into it.

I've been fooled by the translator.(and i don't double check it)
If you could point some link to me, i'll go ahead and check it !
  • Modding version: PSX