Final Fantasy Hacktics

Where would I locate the formulas spells use and how would I edit those? Like, what's the recommended programs?

You would have to modify the game's compiled assembly code directly (ASM hack). There are no programs that could do it for you, though there are some that make it easier. If you're really still interested, this thread would be a good place to start.