Please login or register.

Login with username, password and session length
Advanced search  

News:

Please use .png instead of .bmp when uploading unfinished sprites to the forum!

Data Structures in SCUS_942.21  (Read 719 times)
Pages: [1]
Goladus [Posts: 2]
  • View Profile
  • share
  • [April 01, 2017, 12:24:39 AM]
Data Structures in SCUS_942.21
« on: April 01, 2017, 12:24:39 AM »
Hi, just want to compliment you all on what a great forum you have here.  FFT was my favorite game for a long time.  I hope you'll indulge my question.

Is there a comprehensive map of all the data structures in SCUS_942.21 and the ENTD files?  What I'm looking for would be similar to what's on the wiki for World Stats: http://ffhacktics.com/wiki/World_Stats.

I am writing a command-line tool to query game data from the ISO and am assembling the data.  I was able to extract the basic structure from the FFTPatcher source code (https://github.com/Glain/FFTPatcher/blob/master/PatcherLib/ISOPatching/PsxIso.cs), most of which is also on the wiki (http://ffhacktics.com/wiki/CD_Full_Index) but I'm trying to figure out how data is laid out within the individual files.

The FFTPatcher source should have all of the individual data structures, but it looks like it's fairly heavily baked into the C# source code and not easy to extract with some simple text munging.  FFTPatcher credits ZodiacFFTM's excel spreadsheets, but I am unable to find these.

This:
Code: [Select]
static PsxIso()
        {
            Propositions = new KnownPosition( Sectors.WORLD_WLDCORE_BIN, 0x36380, 0xA7C );
            Abilities = new KnownPosition( Sectors.SCUS_942_21, 0x4F3F0, 9414 );
            AbilityEffects = new KnownPosition( Sectors.BATTLE_BIN, 0x14F3F0, 0x2E0 );
            ItemAbilityEffects = new KnownPosition( Sectors.BATTLE_BIN, 0x14F6D0, 0x1C );
            ReactionAbilityEffects = new KnownPosition( Sectors.BATTLE_BIN, 0x014F73C, 0x40 );
            ActionEvents = new KnownPosition( Sectors.SCUS_942_21, 0x564B4, 224 );
            InflictStatuses = new KnownPosition( Sectors.SCUS_942_21, 0x547C4, 0x300 );
            Jobs = new KnownPosition( Sectors.SCUS_942_21, 0x518B8, 0x1E00 );
            JobLevels = new KnownPosition( Sectors.SCUS_942_21, 0x568C4, 0xD0 );
            MonsterSkills = new KnownPosition( Sectors.SCUS_942_21, 0x563C4, 0xF0 );
            OldItemAttributes = new KnownPosition( Sectors.SCUS_942_21, 0x54AC4, 0x7D0 );
            OldItems = new KnownPosition( Sectors.SCUS_942_21, 0x536B8, 0x110A );
            PoachProbabilities = new KnownPosition( Sectors.SCUS_942_21, 0x56864, 0x60 );
            StatusAttributes = new KnownPosition( Sectors.SCUS_942_21, 0x565E4, 0x280 );
            SkillSets = new KnownPosition( Sectors.SCUS_942_21, 0x55294, 0x1130 );
            ENTD1 = new KnownPosition( Sectors.BATTLE_ENTD1_ENT, 0, 81920 );
            ENTD2 = new KnownPosition( Sectors.BATTLE_ENTD2_ENT, 0, 81920 );
            ENTD3 = new KnownPosition( Sectors.BATTLE_ENTD3_ENT, 0, 81920 );
            ENTD4 = new KnownPosition( Sectors.BATTLE_ENTD4_ENT, 0, 81920 );
            MoveFindItems = new KnownPosition( Sectors.BATTLE_BIN, 0x8EE74, 0x800 );
            StoreInventories = new KnownPosition( Sectors.WORLD_WORLD_BIN, 0xAD844, 0x200 );
            NumberOfSectorsBigEndian = new KnownPosition( (Sectors)16, 0x54, 4 );
            NumberOfSectorsLittleEndian = new KnownPosition( (Sectors)16, 0x50, 4 );

            // Ability animations go down until the Support abilities, so down up to and including 0x1C5 = 0x1C6 * 3 = 0x552 bytes
            AbilityAnimations = new KnownPosition( Sectors.BATTLE_BIN, 0x2CE10, 0x552 );
        }

Is easy enough to convert to this:

Code: [Select]
KnownPositions:
  Propositions: [WORLD_WLDCORE_BIN, 0x36380, 0xA7C]
  Abilities: [SCUS_942_21, 0x4F3F0, 9414]
  AbilityEffects: [BATTLE_BIN, 0x14F3F0, 0x2E0]
  ItemAbilityEffects: [BATTLE_BIN, 0x14F6D0, 0x1C]
  ReactionAbilityEffects: [BATTLE_BIN, 0x014F73C, 0x40]
  ActionEvents: [SCUS_942_21, 0x564B4, 224]
  InflictStatuses: [SCUS_942_21, 0x547C4, 0x300]
  Jobs: [SCUS_942_21, 0x518B8, 0x1E00]
  JobLevels: [SCUS_942_21, 0x568C4, 0xD0]
  MonsterSkills: [SCUS_942_21, 0x563C4, 0xF0]
  OldItemAttributes: [SCUS_942_21, 0x54AC4, 0x7D0]
  OldItems: [SCUS_942_21, 0x536B8, 0x110A]
  PoachProbabilities: [SCUS_942_21, 0x56864, 0x60]
  StatusAttributes: [SCUS_942_21, 0x565E4, 0x280]
  SkillSets: [SCUS_942_21, 0x55294, 0x1130]
  ENTD1: [BATTLE_ENTD1_ENT, 0, 81920]
  ENTD2: [BATTLE_ENTD2_ENT, 0, 81920]
  ENTD3: [BATTLE_ENTD3_ENT, 0, 81920]
  ENTD4: [BATTLE_ENTD4_ENT, 0, 81920]
  MoveFindItems: [BATTLE_BIN, 0x8EE74, 0x800]
  StoreInventories: [WORLD_WORLD_BIN, 0xAD844, 0x200]

But turning this into a simple data structure is going to be more of a hassle:

Code: [Select]
        public Job( Context context, byte value, string name, IList<byte> bytes, Job defaults )
        {
            Value = value;
            Name = name;
            int equipEnd = context == Context.US_PSP ? 13 : 12;

            SkillSet = context == Context.US_PSP ? SkillSet.PSPSkills[bytes[0]] : SkillSet.PSXSkills[bytes[0]];
            InnateA = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[1], bytes[2] )];
            InnateB = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[3], bytes[4] )];
            InnateC = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[5], bytes[6] )];
            InnateD = AllAbilities.DummyAbilities[PatcherLib.Utilities.Utilities.BytesToUShort( bytes[7], bytes[8] )];
            Equipment = new Equipment( bytes.Sub( 9, equipEnd ), defaults == null ? null : defaults.Equipment );
            HPConstant = bytes[equipEnd + 1];
            HPMultiplier = bytes[equipEnd + 2];
            MPConstant = bytes[equipEnd + 3];
            MPMultiplier = bytes[equipEnd + 4];
            SpeedConstant = bytes[equipEnd + 5];
            SpeedMultiplier = bytes[equipEnd + 6];
            PAConstant = bytes[equipEnd + 7];
            PAMultiplier = bytes[equipEnd + 8];
            MAConstant = bytes[equipEnd + 9];
            MAMultiplier = bytes[equipEnd + 10];
            Move = bytes[equipEnd + 11];
            Jump = bytes[equipEnd + 12];
            CEvade = bytes[equipEnd + 13];
            PermanentStatus = new Statuses( bytes.Sub( equipEnd + 14, equipEnd + 18 ), defaults == null ? null : defaults.PermanentStatus );
            StatusImmunity = new Statuses( bytes.Sub( equipEnd + 19, equipEnd + 23 ), defaults == null ? null : defaults.StatusImmunity );
            StartingStatus = new Statuses( bytes.Sub( equipEnd + 24, equipEnd + 28 ), defaults == null ? null : defaults.StartingStatus );
            AbsorbElement = new Elements( bytes[equipEnd + 29] );
            CancelElement = new Elements( bytes[equipEnd + 30] );
            HalfElement = new Elements( bytes[equipEnd + 31] );
            WeakElement = new Elements( bytes[equipEnd + 32] );

            MPortrait = bytes[equipEnd + 33];
            MPalette = bytes[equipEnd + 34];
            MGraphic = bytes[equipEnd + 35];

            if( defaults != null )
            {
                Default = defaults;
                AbsorbElement.Default = defaults.AbsorbElement;
                CancelElement.Default = defaults.CancelElement;
                HalfElement.Default = defaults.HalfElement;
                WeakElement.Default = defaults.WeakElement;
            }
        }

The ability code would be an even more complex example.  https://github.com/Glain/FFTPatcher/blob/master/Datatypes/Abilities/Ability.cs.

I'm just looking for a layout with offsets binary data types.  If this exists on the wiki and I'm just not seeing it, or if someone has the ZodiacFFTM spreadsheets that FFTPatcher was based on, I would be grateful if someone could point me in the right direction!

Thanks.
French Maid
Xifanie (Webmistress) [Posts: 4398]
  • View Profile
  • Final Fantasy Hacktics
  • http://steamcommunity.com/id/Xifanie
  • share
  • [April 01, 2017, 12:57:14 AM]
Re: Data Structures in SCUS_942.21
« Reply #1 on: April 01, 2017, 12:57:14 AM »
I don't exactly have any good news. I'm the one who created those spreadsheets, and they are extremely outdated, as they were last updated 10 years ago... you don't want them.

I am the one who created the World Stats and Battle Stats

I can only suggest to edit things in FFTPatcher and see the resulting Gameshark Code... remove the 3/8 header, and you have yourself a memory address for the SCUS (subtract 0xF800 if you want the file offset). However, that will not work for the ENTD tab, as ENTDs slots are loaded dynamically and thus not part of the SCUS.

You could still make some changes, save/export, and see which bytes changed in a hex editor... no, it's not practical, but I have no other solution to offer than to just look at the source code of FFTPatcher.


Either way, I can probably help you out, because I know about all the data structures in the SCUS covered by FFTPatcher (except the propositions).
I also know that the last 4 bytes of the item table overlap with the beginning of another table (Inflict Statuses IIRC?).

Also, you might be thinking of a simple data structure, but that would require more coding. For example, FFTPatcher should really auto-calculate Inflict Statuses and Item Stats, fuse the duplicates together, and automatically assign an ID that the user should not even be able to see... i.e. those tabs should not even exist and they should be part of the Abilities and Items tabs. Of course, those tables have a limited number of rows, but if this was all automatically calculated, it would not be that easy to reach the max. I had to make a spreadsheet to compensate for that missing feature.

You might want to drop in chat to discuss with more ease.

    • Modding version: PSX
  • <R999> My target market is not FFT mod players
    <Raijinili> remember that? it was awful
    Raijinili [Posts: 78]
    • View Profile
    • share
    • [April 01, 2017, 01:27:23 AM]
    Re: Data Structures in SCUS_942.21
    « Reply #2 on: April 01, 2017, 01:27:23 AM »
    I am writing a command-line tool to query game data from the ISO and am assembling the data.  I was able to extract the basic structure from the FFTPatcher source code (https://github.com/Glain/FFTPatcher/blob/master/PatcherLib/ISOPatching/PsxIso.cs), most of which is also on the wiki (http://ffhacktics.com/wiki/CD_Full_Index) but I'm trying to figure out how data is laid out within the individual files.

    I was considering IronPython for this purpose: load up the FFTPatcher libs, write some wrappers, and query from Python. IronPython is a Python that runs on the same virtual machine that C# uses, allowing reuse of the existing C# objects.

    Make me do it.
    Goladus [Posts: 2]
    • View Profile
    • share
    • [April 01, 2017, 01:59:34 AM]
    Re: Data Structures in SCUS_942.21
    « Reply #3 on: April 01, 2017, 01:59:34 AM »
    Thanks Xifanie,

    The ideas you crossed out are probably what I'm going to wind up doing.  For job data, simply knowing that FFTPatcher displays the HPC/HPM/etc. stats in the right order makes it easy to search memory for those byte sequences (eg [11, 125, 11] gets me 3 addresses corresponding to Ramza's 3 Squire jobs).  I've already used the gameshark export tab to make some manual save state modifications (like adding abilities to a skillset), just for the sake of experimentation.

    I was just checking to see if my work had already been done for me!  I'll keep at it and report back.

    Quote
    Also, you might be thinking of a simple data structure, but that would require more coding.

    Right, I'm interested in the raw, low-level data structure with the assumption that I might need to do some additional coding to properly interpret that data, and the recognition that some things might be unusually complicated (such as the overlapping tables you mention).

    Quote
    For example, FFTPatcher should really auto-calculate Inflict Statuses and Item Stats, fuse the duplicates together, and automatically assign an ID that the user should not even be able to see... i.e. those tabs should not even exist and they should be part of the Abilities and Items tabs. Of course, those tables have a limited number of rows, but if this was all automatically calculated, it would not be that easy to reach the max.

    This is true if your goal is to design an intuitive UI for modders (which is the purpose of FFTPatcher), but for would-be hackers that want to understand the game it could be confusing.  If the UI suggested that each ability included its own bit array for inflicting statuses, when what's actually there is a reference to a status payload from a different list, that would confuse anyone who tried to write their own tool to do something that FFTPatcher wasn't designed for.  And I'm not really sure what I'm going to to want to do until later.  For now, the first thing I want to do to query job stats and sort them, like this.  (Data for this came from Aerostar's Battle Mechanics Guide, which is great but doesn't work for mods)

    Code: [Select]
    $ python jobdata.py HPM MPM SPM PAM MAM | egrep "(Generic|Holy.Knight|Holy.Swordsman)"
    63  [65, 80, 50, 50, 70]           Calculator (0x5a)    Generic
    70  [55, 50, 100, 30, 115]         Bard (0x5b)          Generic
    80  [80, 70, 100, 75, 75]          Mediator (0x54)      Generic
    82  [80, 75, 100, 75, 80]          Chemist (0x4b)       Generic
    82  [90, 50, 110, 100, 60]         Thief (0x53)         Generic
    83  [60, 50, 100, 110, 95]         Dancer (0x5c)        Generic
    87  [70, 50, 120, 120, 75]         Ninja (0x59)         Generic
    88  [120, 50, 100, 120, 50]        Lancer (0x57)        Generic
    89  [100, 75, 100, 90, 80]         Squire (0x4a)        Generic
    91  [75, 110, 100, 50, 120]        Oracle (0x55)        Generic
    91  [100, 65, 100, 110, 80]        Archer (0x4d)        Generic
    92  [70, 125, 90, 50, 125]         Summoner (0x52)      Generic
    93  [75, 75, 100, 128, 90]         Samurai (0x58)       Generic
    95  [75, 120, 100, 50, 130]        Time Mage (0x51)     Generic
    100 [120, 80, 100, 120, 80]        Knight (0x4c)        Generic
    101 [75, 120, 100, 60, 150]        Wizard (0x50)        Generic
    102 [80, 120, 110, 90, 110]        Priest (0x4f)        Generic
    104 [110, 95, 100, 110, 105]       Geomancer (0x56)     Generic
    106 [135, 80, 110, 129, 80]        Monk (0x4e)          Generic
    108 [140, 100, 100, 100, 100]      Holy Knight (0x1e)   Special
    108 [140, 100, 100, 100, 100]      Holy Knight (0x34)   Special
    109 [140, 50, 120, 120, 115]       Mime (0x5d)          Generic
    114 [135, 100, 110, 120, 105]      Holy Knight (0x5)    Special
    122 [160, 120, 110, 122, 100]      Holy Swordsman (0xd) Special

    Quote from: Raijinili
    I was considering IronPython for this purpose: load up the FFTPatcher libs, write some wrappers, and query from Python. IronPython is a Python that runs on the same virtual machine that C# uses, allowing reuse of the existing C# objects.

    Make me do it.

    That's a great idea, though I confess setting up a new dev environment wasn't originally in my immediate plans.
    French Maid
    Xifanie (Webmistress) [Posts: 4398]
    • View Profile
    • Final Fantasy Hacktics
    • http://steamcommunity.com/id/Xifanie
    • share
    • [April 01, 2017, 02:20:05 AM]
    Re: Data Structures in SCUS_942.21
    « Reply #4 on: April 01, 2017, 02:20:05 AM »
    Not everything is in order, unfortunately, but overall it tends to be. You have to be careful because some checkboxes were reversed. It was easier for the creator of FFTPatcher (melonhead) to come up with a flag name after reversing the value... so, true is false and false is true. A horrible practice you'll have to watch out for. I can't remember offhand which is which, and I don't know if Glain fixed those, but I wouldn't think so.

    • Modding version: PSX
  • <R999> My target market is not FFT mod players
    <Raijinili> remember that? it was awful
    Raijinili [Posts: 78]
    • View Profile
    • share
    • [April 03, 2017, 05:17:47 AM]
    Re: Data Structures in SCUS_942.21
    « Reply #5 on: April 03, 2017, 05:17:47 AM »
    After so much time wishing I'd do it, I got IronPython working on FFTPatcher. Best part is, you can use it with the compiled/built/released Patcher, instead of the source code.

    You need to change the "folder" variable below. Remember the r if you write backslashes. You also need to toggle the comments at the end if you want to load WotL. Only one of the versions can be loaded at a time, but you can extract the values from FFTPatch before you load the other.

    Code: [Select]
    import clr
    from os.path import abspath, join

    folder = r'path\to\FFTPatcher_491'  #Path to the folder with FFTPatcher.exe and PatcherLib.dll.
    folder = abspath(folder)  #clr requires absolute path.
    def load_dll(filename):
        clr.AddReferenceToFileAndPath(join(folder, filename))

    load_dll('FFTPatcher.exe')
    load_dll('PatcherLib.dll')

    from FFTPatcher.Datatypes import FFTPatch
    from PatcherLib.Datatypes import Context

    try:
        FFTPatch.New(Context.US_PSX) #for USPSX
        # FFTPatch.New(Context.US_PSP) #for WotL
    except SystemError:
        # If it errors, run it again. Static constructor silliness.
        print("Mou Ikkai!")
        FFTPatch.New(Context.US_PSX) #for USPSX
        # FFTPatch.New(Context.US_PSP) #for WotL

    After the above code runs, the FFTPatch class will hold all the data. Use its source code as a reference for what's in there.
    - FFTPatch.Abilities
    - FFTPatch.AbilityAnimations
    - FFTPatch.ActionMenus
    - FFTPatch.Context
    - FFTPatch.ENTDs
    - FFTPatch.InflictStatuses
    - FFTPatch.ItemAttributes
    - FFTPatch.Items
    - FFTPatch.JobLevels
    - FFTPatch.Jobs
    - FFTPatch.MonsterSkills
    - FFTPatch.MoveFind
    - FFTPatch.PoachProbabilities
    - FFTPatch.SkillSets
    - FFTPatch.StatusAttributes
    - FFTPatch.StoreInventories
    - FFTPatch.Propositions

    You may need to deeper to get to the good stuff. For example, the (iterable!) Array of abilities is at FFTPatch.Abilities.Abilities, and the Array of jobs is at FFTPatch.Jobs.Jobs.

    To see the properties of an object, look at .DigestableProperties (e.g. "FFTPatch.Jobs.Jobs[0].DigestableProperties". Or use the dir(obj) and vars(obj) Python functions to inspect an object and see what properties it has.

    Python being Python. you can rename FFTPatch to be less annoying to type.
    « Last Edit: April 04, 2017, 10:10:37 PM by Raijinili »
    • Modding version: Other/Unknown
  • Pages: [1]