Crystal Dynamics Hash Generator

software by Ben Lincoln and Andrew Fradley, article by Ben Lincoln

The Crystal Dynamics Hash Generator utility converts human-readable filenames into the proprietary hashes used in Crystal Dynamics bigfiles.

Supports the following filename-hashing algorithms:

Blood Omen (all known versions)
Original Gex 2 algorithm, used by:
- Akuji: The Heartless
- Gex 2
- Gex 3
- Soul Reaver (Lighthouse / "proto1" demo)
- Walt Disney World Magical Racing Tour
Soul Reaver (PlayStation and PC release and beta versions)
Soul Reaver (known alpha builds)
Soul Reaver (all known Dreamcast versions)
Soul Reaver 2 (Air Forge demo builds)
Soul Reaver 2 (all known release versions)
Defiance (all known versions)

This is a command-line tool, since its main use is generating hashtable lists for use with Soul Spiral.

Options

--hashtype or -t: selects the algorithm to use. Valid choices are:

bloodomen or bo: all known versions of Blood Omen
proto1: Gex 2, Gex 3, Akuji: The Heartless, Soul Reaver (Lighthouse / "proto1" demo), Walt Disney World Magical Racing Tour
soulreaveralpha or sra: alpha builds of Soul Reaver
soulreaver or sr: PlayStation and PC release and beta builds of Soul Reaver
soulreaverdreamcast or srdc: Dreamcast builds of Soul Reaver
soulreaver2airforge or sr2af: Air Forge demo builds of Soul Reaver 2
soulreaver2 or sr2: PlayStation 2 and PC release builds of Soul Reaver 2
defiance: all known builds of Defiance

--input or -i: specify a file to read input strings from, one per line, e.g. --input filenames.txt.

--match or -m: specify a file to read hashes (in ASCII hexadecimal format) that should be tested for matches against the program output, one per line, e.g. --match known_hashes.txt. If this option is specified, only hashes that match an entry in the list will be output.

--output or -o: specify a path to write results in tab-delimited format, one per line, e.g. --output new_hash_lookup_list.txt.

--separator or -s: specify a separator to use instead of the default tab, e.g. --separator ,.

--hash-raw-text: disable the use of any game-specific logic for generating hashes. For example, the Soul Reaver algorithm handles six file extensions using special logic that will be ignored if this option is specified. It is very unlikely you want to use this option unless you are reverse-engineering an unknown build of a game.

--big-endian: generate hashes in big-endian byte order, e.g. for matching against data from Sega Saturn versions of games.

--base64: output hashes as base64-encoded binary instead of ASCII hexadecimal.

--append: append to an existing output file instead of overwriting it.

--overwrite: overwrite an existing output file.

Generating a single hash

Generate a hash using the Soul Reaver release build algorithm for the string kain2\area\add\bin\add1.pcm:

CDHashGen.exe -t sr kain2\area\add\bin\add1.pcm

Should generate the following output:

998E1D20 kain2\area\add\bin\add1.pcm

Generating hashes from a list

Generate a list of hashes from a list of complete filenames using the algorithm for Gex 3:

CDHashGen.exe --hashtype proto1 --input gex3_filenames.txt --output gex3_hashes-01.txt

Searching for hashes based on known information

When analyzing an unknown build of a game, if there are any files with hashes that aren't already in the lookup files included with Soul Spiral, it's usually necessary to test for them using available information.

For example, Soul Reaver dynamically constructs filenames like the following when it processes an instruction to load the pillars 1 area:

\kain2\area\pillars\bin\pillars1.crm
\kain2\area\pillars\bin\pillars1.drm

...or like the following when loading the character model for Raziel:

\kain2\object\raziel\raziel.crm
\kain2\object\raziel\raziel.drm

Similarly, every level in Gex 3 may have up to five associated files, with names in the following form, where # may be any number from 1 to 99, sometimes zero-padded, sometimes not:

\g3\level\LEVEL_NAME\LEVEL_NAME#.smp
\g3\level\LEVEL_NAME\LEVEL_NAME#.vrm
\g3\level\LEVEL_NAME\LEVEL_NAME#.snd
\g3\level\LEVEL_NAME\LEVEL_NAME#.bin
\g3\level\LEVEL_NAME\LEVEL_NAME#.drm

e.g.

\g3\level\mob\mob01.smp
\g3\level\mob\mob01.vrm
\g3\level\mob\mob01.snd
\g3\level\mob\mob01.bin
\g3\level\mob\mob01.drm

For this type of scenario, you can build up a dictionary of potential name components by analyzing the game binary and bigfile(s). For example:

kain
raziel
akuj
city

Then generate a list of permutations. I do this on Linux, because the Windows command-line is not very good for this purpose. E.g. for PlayStation versions of Soul Reaver:

while read name; do for number in `seq 1 99`; do for extension in drm crm tim smp snd smf snf; do echo "\\kain2\\area\\${name}\\bin\\${name}${number}.${extension}" >> soul_reaver_filenames.txt;done; done; done

while read name; do for extension in drm crm tim smp snd smf snf; do echo "\\kain2\\object\\${name}\\${name}.${extension}" >> soul_reaver_filenames.txt; done; done

Gex 3 is a little more complicated because of the potential for zero-padding:

while read name; do for number in `seq 1 99`; do for extension in drm vrm bin smp snd; do echo "\\g3\\level\\${name}\\${name}${number}\\${name}${number}.${extension}" >> gex_filenames.txt; echo "\\g3\\level\\${name}\\${name}0${number}\\${name}0${number}.${extension}" >> gex_filenames.txt; done; done; done

You'll probably want a list of hashes to match against so that the output file isn't enormous. I do this by exporting the game using Soul Spiral and building a list from the filenames in the Default directory:

ls Default | sed -E 's/\.[a-z]{3}$//g' | sort -u > unknown_hashes.txt

Finally, run the hash generator tool using the data you've collected:

CDHashGen.exe --hashtype sra --input soul_reaver_filenames.txt --match unknown_hashes.txt --output soul_reaver_hashes-01.txt

Updates in Version 0.3:

Fixed an old-but-undiscovered bug which caused filename hashes generated for the PlayStation release version of Soul Reaver to be incorrect if the extension was .drm. Thank you to Gh0stBlade and Andrew Fradley for reporting the issue.

Download
File	Size	Version	Release Date	Author
Crystal Dynamics Hash Generator	38 KiB	0.3	2023-07-19	Ben Lincoln and Andrew Fradley

Download
File	Size	Version	Release Date	Author
Crystal Dynamics Hash Generator (Source Code)	444 KiB	0.3	2023-07-19	Ben Lincoln and Andrew Fradley

Download
File	Size	Version	Release Date	Author
Crystal Dynamics Hash Generator	33 KiB	0.2	2020-05-12	Ben Lincoln and Andrew Fradley

Download
File	Size	Version	Release Date	Author
Crystal Dynamics Hash Generator (Source Code)	201 KiB	0.2	2020-05-12	Ben Lincoln and Andrew Fradley