• Welcome to Final Fantasy Hacktics. Please login or sign up.
 
February 19, 2020, 09:20:56 pm

News:

Use of ePSXe before 2.0 is highly discouraged. Mednafen/RetroArch is recommended for playing/testing, pSX is recommended for debugging.


Data Structures in SCUS_942.21

Started by Goladus, March 31, 2017, 08:24:39 pm

Goladus

Hi, just want to compliment you all on what a great forum you have here.  FFT was my favorite game for a long time.  I hope you'll indulge my question.

Is there a comprehensive map of all the data structures in SCUS_942.21 and the ENTD files?  What I'm looking for would be similar to what's on the wiki for World Stats: http://ffhacktics.com/wiki/World_Stats.

I am writing a command-line tool to query game data from the ISO and am assembling the data.  I was able to extract the basic structure from the FFTPatcher source code (https://github.com/Glain/FFTPatcher/blob/master/PatcherLib/ISOPatching/PsxIso.cs), most of which is also on the wiki (http://ffhacktics.com/wiki/CD_Full_Index) but I'm trying to figure out how data is laid out within the individual files.

The FFTPatcher source should have all of the individual data structures, but it looks like it's fairly heavily baked into the C# source code and not easy to extract with some simple text munging.  FFTPatcher credits ZodiacFFTM's excel spreadsheets, but I am unable to find these.

This:
static PsxIso()
        {
            Propositions = new KnownPosition( Sectors.WORLD_WLDCORE_BIN, 0x36380, 0xA7C );
            Abilities = new KnownPosition( Sectors.SCUS_942_21, 0x4F3F0, 9414 );
            AbilityEffects = new KnownPosition( Sectors.BATTLE_BIN, 0x14F3F0, 0x2E0 );
            ItemAbilityEffects = new KnownPosition( Sectors.BATTLE_BIN, 0x14F6D0, 0x1C );
            ReactionAbilityEffects = new KnownPosition( Sectors.BATTLE_BIN, 0x014F73C, 0x40 );
            ActionEvents = new KnownPosition( Sectors.SCUS_942_21, 0x564B4, 224 );
            InflictStatuses = new KnownPosition( Sectors.SCUS_942_21, 0x547C4, 0x300 );
            Jobs = new KnownPosition( Sectors.SCUS_942_21, 0x518B8, 0x1E00 );
            JobLevels = new KnownPosition( Sectors.SCUS_942_21, 0x568C4, 0xD0 );
            MonsterSkills = new KnownPosition( Sectors.SCUS_942_21, 0x563C4, 0xF0 );
            OldItemAttributes = new KnownPosition( Sectors.SCUS_942_21, 0x54AC4, 0x7D0 );
            OldItems = new KnownPosition( Sectors.SCUS_942_21, 0x536B8, 0x110A );
            PoachProbabilities = new KnownPosition( Sectors.SCUS_942_21, 0x56864, 0x60 );
            StatusAttributes = new KnownPosition( Sectors.SCUS_942_21, 0x565E4, 0x280 );
            SkillSets = new KnownPosition( Sectors.SCUS_942_21, 0x55294, 0x1130 );
            ENTD1 = new KnownPosition( Sectors.BATTLE_ENTD1_ENT, 0, 81920 );
            ENTD2 = new KnownPosition( Sectors.BATTLE_ENTD2_ENT, 0, 81920 );
            ENTD3 = new KnownPosition( Sectors.BATTLE_ENTD3_ENT, 0, 81920 );
            ENTD4 = new KnownPosition( Sectors.BATTLE_ENTD4_ENT, 0, 81920 );
            MoveFindItems = new KnownPosition( Sectors.BATTLE_BIN, 0x8EE74, 0x800 );
            StoreInventories = new KnownPosition( Sectors.WORLD_WORLD_BIN, 0xAD844, 0x200 );
            NumberOfSectorsBigEndian = new KnownPosition( (Sectors)16, 0x54, 4 );
            NumberOfSectorsLittleEndian = new KnownPosition( (Sectors)16, 0x50, 4 );

            // Ability animations go down until the Support abilities, so down up to and including 0x1C5 = 0x1C6 * 3 = 0x552 bytes
            AbilityAnimations = new KnownPosition( Sectors.BATTLE_BIN, 0x2CE10, 0x552 );
        }


Is easy enough to convert to this:


KnownPositions:
  Propositions: [WORLD_WLDCORE_BIN, 0x36380, 0xA7C]
  Abilities: [SCUS_942_21, 0x4F3F0, 9414]
  AbilityEffects: [BATTLE_BIN, 0x14F3F0, 0x2E0]
  ItemAbilityEffects: [BATTLE_BIN, 0x14F6D0, 0x1C]
  ReactionAbilityEffects: [BATTLE_BIN, 0x014F73C, 0x40]
  ActionEvents: [SCUS_942_21, 0x564B4, 224]
  InflictStatuses: [SCUS_942_21, 0x547C4, 0x300]
  Jobs: [SCUS_942_21, 0x518B8, 0x1E00]
  JobLevels: [SCUS_942_21, 0x568C4, 0xD0]
  MonsterSkills: [SCUS_942_21, 0x563C4, 0xF0]
  OldItemAttributes: [SCUS_942_21, 0x54AC4, 0x7D0]
  OldItems: [SCUS_942_21, 0x536B8, 0x110A]
  PoachProbabilities: [SCUS_942_21, 0x56864, 0x60]
  StatusAttributes: [SCUS_942_21, 0x565E4, 0x280]
  SkillSets: [SCUS_942_21, 0x55294, 0x1130]
  ENTD1: [BATTLE_ENTD1_ENT, 0, 81920]
  ENTD2: [BATTLE_ENTD2_ENT, 0, 81920]
  ENTD3: [BATTLE_ENTD3_ENT, 0, 81920]
  ENTD4: [BATTLE_ENTD4_ENT, 0, 81920]
  MoveFindItems: [BATTLE_BIN, 0x8EE74, 0x800]
  StoreInventories: [WORLD_WORLD_BIN, 0xAD844, 0x200]


But turning this into a simple data structure is going to be more of a hassle:


        public Job( Context context, byte value, string name, IList<byte> bytes, Job defaults )
        {
            Value = value;
            Name = name;
            int equipEnd = context == Context.US_PSP ? 13 : 12;

            SkillSet = context == Context.US_PSP ? SkillSet.PSPSkills[bytes[0]] : SkillSet.PSXSkills[bytes[0]];
            InnateA = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[1], bytes[2] )];
            InnateB = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[3], bytes[4] )];
            InnateC = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[5], bytes[6] )];
            InnateD = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[7], bytes[8] )];
            Equipment = new Equipment( bytes.Sub( 9, equipEnd ), defaults == null ? null : defaults.Equipment );
            HPConstant = bytes[equipEnd + 1];
            HPMultiplier = bytes[equipEnd + 2];
            MPConstant = bytes[equipEnd + 3];
            MPMultiplier = bytes[equipEnd + 4];
            SpeedConstant = bytes[equipEnd + 5];
            SpeedMultiplier = bytes[equipEnd + 6];
            PAConstant = bytes[equipEnd + 7];
            PAMultiplier = bytes[equipEnd + 8];
            MAConstant = bytes[equipEnd + 9];
            MAMultiplier = bytes[equipEnd + 10];
            Move = bytes[equipEnd + 11];
            Jump = bytes[equipEnd + 12];
            CEvade = bytes[equipEnd + 13];
            PermanentStatus = new Statuses( bytes.Sub( equipEnd + 14, equipEnd + 18 ), defaults == null ? null : defaults.PermanentStatus );
            StatusImmunity = new Statuses( bytes.Sub( equipEnd + 19, equipEnd + 23 ), defaults == null ? null : defaults.StatusImmunity );
            StartingStatus = new Statuses( bytes.Sub( equipEnd + 24, equipEnd + 28 ), defaults == null ? null : defaults.StartingStatus );
            AbsorbElement = new Elements( bytes[equipEnd + 29] );
            CancelElement = new Elements( bytes[equipEnd + 30] );
            HalfElement = new Elements( bytes[equipEnd + 31] );
            WeakElement = new Elements( bytes[equipEnd + 32] );

            MPortrait = bytes[equipEnd + 33];
            MPalette = bytes[equipEnd + 34];
            MGraphic = bytes[equipEnd + 35];

            if( defaults != null )
            {
                Default = defaults;
                AbsorbElement.Default = defaults.AbsorbElement;
                CancelElement.Default = defaults.CancelElement;
                HalfElement.Default = defaults.HalfElement;
                WeakElement.Default = defaults.WeakElement;
            }
        }


The ability code would be an even more complex example.  https://github.com/Glain/FFTPatcher/blob/master/Datatypes/Abilities/Ability.cs.

I'm just looking for a layout with offsets binary data types.  If this exists on the wiki and I'm just not seeing it, or if someone has the ZodiacFFTM spreadsheets that FFTPatcher was based on, I would be grateful if someone could point me in the right direction!

Thanks.
  • Modding version: PSX

Xifanie

I don't exactly have any good news. I'm the one who created those spreadsheets, and they are extremely outdated, as they were last updated 10 years ago... you don't want them.

I am the one who created the World Stats and Battle Stats

I can only suggest to edit things in FFTPatcher and see the resulting Gameshark Code... remove the 3/8 header, and you have yourself a memory address for the SCUS (subtract 0xF800 if you want the file offset). However, that will not work for the ENTD tab, as ENTDs slots are loaded dynamically and thus not part of the SCUS.

You could still make some changes, save/export, and see which bytes changed in a hex editor... no, it's not practical, but I have no other solution to offer than to just look at the source code of FFTPatcher.


Either way, I can probably help you out, because I know about all the data structures in the SCUS covered by FFTPatcher (except the propositions).
I also know that the last 4 bytes of the item table overlap with the beginning of another table (Inflict Statuses IIRC?).

Also, you might be thinking of a simple data structure, but that would require more coding. For example, FFTPatcher should really auto-calculate Inflict Statuses and Item Stats, fuse the duplicates together, and automatically assign an ID that the user should not even be able to see... i.e. those tabs should not even exist and they should be part of the Abilities and Items tabs. Of course, those tables have a limited number of rows, but if this was all automatically calculated, it would not be that easy to reach the max. I had to make a spreadsheet to compensate for that missing feature.

You might want to drop in chat to discuss with more ease.
  • Modding version: PSX
<R999> My target market is not FFT mod players
<Raijinili> remember that? it was awful

Raijinili

Quote from: Goladus on March 31, 2017, 08:24:39 pm
I am writing a command-line tool to query game data from the ISO and am assembling the data.  I was able to extract the basic structure from the FFTPatcher source code (https://github.com/Glain/FFTPatcher/blob/master/PatcherLib/ISOPatching/PsxIso.cs), most of which is also on the wiki (http://ffhacktics.com/wiki/CD_Full_Index) but I'm trying to figure out how data is laid out within the individual files.


I was considering IronPython for this purpose: load up the FFTPatcher libs, write some wrappers, and query from Python. IronPython is a Python that runs on the same virtual machine that C# uses, allowing reuse of the existing C# objects.

Make me do it.
  • Modding version: Other/Unknown

Goladus

Thanks Xifanie,

The ideas you crossed out are probably what I'm going to wind up doing.  For job data, simply knowing that FFTPatcher displays the HPC/HPM/etc. stats in the right order makes it easy to search memory for those byte sequences (eg [11, 125, 11] gets me 3 addresses corresponding to Ramza's 3 Squire jobs).  I've already used the gameshark export tab to make some manual save state modifications (like adding abilities to a skillset), just for the sake of experimentation.

I was just checking to see if my work had already been done for me!  I'll keep at it and report back.

QuoteAlso, you might be thinking of a simple data structure, but that would require more coding.


Right, I'm interested in the raw, low-level data structure with the assumption that I might need to do some additional coding to properly interpret that data, and the recognition that some things might be unusually complicated (such as the overlapping tables you mention).

QuoteFor example, FFTPatcher should really auto-calculate Inflict Statuses and Item Stats, fuse the duplicates together, and automatically assign an ID that the user should not even be able to see... i.e. those tabs should not even exist and they should be part of the Abilities and Items tabs. Of course, those tables have a limited number of rows, but if this was all automatically calculated, it would not be that easy to reach the max.


This is true if your goal is to design an intuitive UI for modders (which is the purpose of FFTPatcher), but for would-be hackers that want to understand the game it could be confusing.  If the UI suggested that each ability included its own bit array for inflicting statuses, when what's actually there is a reference to a status payload from a different list, that would confuse anyone who tried to write their own tool to do something that FFTPatcher wasn't designed for.  And I'm not really sure what I'm going to to want to do until later.  For now, the first thing I want to do to query job stats and sort them, like this.  (Data for this came from Aerostar's Battle Mechanics Guide, which is great but doesn't work for mods)

$ python jobdata.py HPM MPM SPM PAM MAM | egrep "(Generic|Holy.Knight|Holy.Swordsman)"
63  [65, 80, 50, 50, 70]           Calculator (0x5a)    Generic
70  [55, 50, 100, 30, 115]         Bard (0x5b)          Generic
80  [80, 70, 100, 75, 75]          Mediator (0x54)      Generic
82  [80, 75, 100, 75, 80]          Chemist (0x4b)       Generic
82  [90, 50, 110, 100, 60]         Thief (0x53)         Generic
83  [60, 50, 100, 110, 95]         Dancer (0x5c)        Generic
87  [70, 50, 120, 120, 75]         Ninja (0x59)         Generic
88  [120, 50, 100, 120, 50]        Lancer (0x57)        Generic
89  [100, 75, 100, 90, 80]         Squire (0x4a)        Generic
91  [75, 110, 100, 50, 120]        Oracle (0x55)        Generic
91  [100, 65, 100, 110, 80]        Archer (0x4d)        Generic
92  [70, 125, 90, 50, 125]         Summoner (0x52)      Generic
93  [75, 75, 100, 128, 90]         Samurai (0x58)       Generic
95  [75, 120, 100, 50, 130]        Time Mage (0x51)     Generic
100 [120, 80, 100, 120, 80]        Knight (0x4c)        Generic
101 [75, 120, 100, 60, 150]        Wizard (0x50)        Generic
102 [80, 120, 110, 90, 110]        Priest (0x4f)        Generic
104 [110, 95, 100, 110, 105]       Geomancer (0x56)     Generic
106 [135, 80, 110, 129, 80]        Monk (0x4e)          Generic
108 [140, 100, 100, 100, 100]      Holy Knight (0x1e)   Special
108 [140, 100, 100, 100, 100]      Holy Knight (0x34)   Special
109 [140, 50, 120, 120, 115]       Mime (0x5d)          Generic
114 [135, 100, 110, 120, 105]      Holy Knight (0x5)    Special
122 [160, 120, 110, 122, 100]      Holy Swordsman (0xd) Special


Quote from: RaijiniliI was considering IronPython for this purpose: load up the FFTPatcher libs, write some wrappers, and query from Python. IronPython is a Python that runs on the same virtual machine that C# uses, allowing reuse of the existing C# objects.

Make me do it.


That's a great idea, though I confess setting up a new dev environment wasn't originally in my immediate plans.
  • Modding version: PSX

Xifanie

Not everything is in order, unfortunately, but overall it tends to be. You have to be careful because some checkboxes were reversed. It was easier for the creator of FFTPatcher (melonhead) to come up with a flag name after reversing the value... so, true is false and false is true. A horrible practice you'll have to watch out for. I can't remember offhand which is which, and I don't know if Glain fixed those, but I wouldn't think so.
  • Modding version: PSX
<R999> My target market is not FFT mod players
<Raijinili> remember that? it was awful

Raijinili

April 03, 2017, 01:17:47 am #5 Last Edit: April 04, 2017, 06:10:37 pm by Raijinili
After so much time wishing I'd do it, I got IronPython working on FFTPatcher. Best part is, you can use it with the compiled/built/released Patcher, instead of the source code.

You need to change the "folder" variable below. Remember the r if you write backslashes. You also need to toggle the comments at the end if you want to load WotL. Only one of the versions can be loaded at a time, but you can extract the values from FFTPatch before you load the other.

import clr
from os.path import abspath, join

folder = r'path\to\FFTPatcher_491'  #Path to the folder with FFTPatcher.exe and PatcherLib.dll.
folder = abspath(folder)  #clr requires absolute path.
def load_dll(filename):
    clr.AddReferenceToFileAndPath(join(folder, filename))

load_dll('FFTPatcher.exe')
load_dll('PatcherLib.dll')

from FFTPatcher.Datatypes import FFTPatch
from PatcherLib.Datatypes import Context

try:
    FFTPatch.New(Context.US_PSX) #for USPSX
    # FFTPatch.New(Context.US_PSP) #for WotL
except SystemError:
    # If it errors, run it again. Static constructor silliness.
    print("Mou Ikkai!")
    FFTPatch.New(Context.US_PSX) #for USPSX
    # FFTPatch.New(Context.US_PSP) #for WotL


After the above code runs, the FFTPatch class will hold all the data. Use its source code as a reference for what's in there.
- FFTPatch.Abilities
- FFTPatch.AbilityAnimations
- FFTPatch.ActionMenus
- FFTPatch.Context
- FFTPatch.ENTDs
- FFTPatch.InflictStatuses
- FFTPatch.ItemAttributes
- FFTPatch.Items
- FFTPatch.JobLevels
- FFTPatch.Jobs
- FFTPatch.MonsterSkills
- FFTPatch.MoveFind
- FFTPatch.PoachProbabilities
- FFTPatch.SkillSets
- FFTPatch.StatusAttributes
- FFTPatch.StoreInventories
- FFTPatch.Propositions

You may need to deeper to get to the good stuff. For example, the (iterable!) Array of abilities is at FFTPatch.Abilities.Abilities, and the Array of jobs is at FFTPatch.Jobs.Jobs.

To see the properties of an object, look at .DigestableProperties (e.g. "FFTPatch.Jobs.Jobs
  • .DigestableProperties". Or use the dir(obj) and vars(obj) Python functions to inspect an object and see what properties it has.

    Python being Python. you can rename FFTPatch to be less annoying to type.
  • Modding version: Other/Unknown