Games Unravelling the Secrets of "Normality" (1996) (Read 5054 times) ANALYZE

  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630


I am working out how to rip resources from the game files for the 3D point and click adventure game Normality by Gremlin Entertainment. I strongly suspect there to be many inaccessible/unused/secret things here. I want to look at them.


I will be updating this thread with my progress. I have not got very far yet, many of the files are encoded with a proprietary kind of compression. I will include decompression code once I have reverse engineered it, then I can start working out what's in the raw data


I believe this to be the first project of its kind anywhere on the web... the secrets of Normality will be spilled here, for the first time, live


Also: Analysis of the underlying politics, ideology and intentions of the Normality authors very welcome in this thread
 
 
 
 

Last Edit: February 03, 2013, 05:28:13 pm by denzquix
  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
Wowwww
Normality is one of my favorite games ever. I must have played through it about a dozen times. And yeah, there's something MYSTERIOUS about certain things. Like the park at the beginning of the game that you show. I've never been able to get in the normal way, but a while back I recorded some videos that show what happen if you do get in(by modifying the player's coordinates with a debugger).

Normality inaccessible area: the TV station set
http://www.youtube.com/watch?v=hP_QNRw5UmY

Normality inaccessible area: Saul's cell
http://www.youtube.com/watch?v=KclwqUUg8RM

Normality inaccessible area: the Plush-Rest furniture machine
http://www.youtube.com/watch?v=eKe-3YjnWiY

Normality inaccessible area: the park
http://www.youtube.com/watch?v=82TCgjop_jY

If you could find some way to rip the graphics, that would be awesome. I've always sort of had it in the back of my mind to try this but given that it's COMPLEX STUFF that I have no experience with, I've never given it a try.
  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
Quote
Also: Analysis of the underlying politics, ideology and intentions of the Normality authors very welcome in this thread
Check out the comments by BL1TZEN on this vid

http://www.youtube.com/watch?feature=player_embedded&v=MB_bACKgt9Y
Last Edit: January 19, 2013, 07:04:35 pm by dada
  • Avatar of Vellfire
  • TV people want to leave
  • PipPipPipPipPipPipPipPipPipPip
  • Group: Premium Member
  • Joined: Feb 13, 2004
  • Posts: 9602
I was gonna freak out and alert dada about this topic but he already found it and now I feel foolish for thinking that a topic about Normality could show up on this forum and him not be in it IMMEDIATELY.
I love this hobby - stealing your mother's diary
BRRING! BRRING!
Hello!  It's me, Vellfire!  FOLLOW ME ON TWITTER! ... Bye!  CLICK!  @gidgetnomates
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630


h'okay, I think I have cracked the decompression algorithm!!

the rest of this post will only be of interest to hardcore bytehackers who want to follow in my filthy footsteps


so, this compression scheme applies to any files that have the extension .MGL and this is some (Lua-ish) pseudocode code to decompress it
repeat
  local b = read_byte()
  if (b == 0) or (b == nil) then
    -- zero byte marks the end of the stream
    break
  elseif b >= 0xE0 then
    local b2 = read_byte()
    local b3 = read_byte()
    local offset = ((b - 0xE0) * 256) + b2 + 3
    local length = b3 + 5
    reuse_bytes(offset, length)
  elseif b >= 0xC0 then
    local b2 = read_byte()
    local offset = ((b % 4) * 256) + b2 + 3
    local length = 4 + math.floor((b - 0xC0) / 4)
    reuse_bytes(offset, length)
  elseif b >= 0x80 then
    local offset = (b - 0x80) + 3
    reuse_bytes(offset, 3)
  elseif b >= 0x70 then
    local reps = (b - 0x70) + 2
    reuse_bytes(2, 2, reps)
  elseif b >= 0x60 then
    local reps = (b - 0x60) + 3
    reuse_bytes(1, 1, reps)
  elseif b >= 0x50 then
    local length = (b - 0x50) + 2
    uint16LE_pattern_sequence(length)
  elseif b >= 0x40 then
    local length = (b - 0x40) + 3
    byte_pattern_sequence(length)
  else
    local length = b
    copy_input_bytes(length)
  end
until (b == 0) or (b == nil)
this is the core loop, it requires the following functions be defined first:
  • read_byte(): Read 1 byte from the input stream, and return its value as a number. If the end of the stream is reached, return nil.
  • copy_input_bytes(length): Take a chunk of bytes from the input stream, and write it direct to the output stream.
  • reuse_bytes(offset, length[, repeats]): Take a chunk of bytes that have previously been written to the output stream, and re-append it to the end. The chunk begins offset bytes backwards from the end of the stream. If length is greater than offset, repeat the chunk like a pattern - for example if reuse_bytes(5, 9) is called and the last five bytes written spell "Hello" in ASCII, "HelloHell" should be written to the output stream. If the optional parameter repeats is given, repeat the chunk that number of times.
  • byte_pattern_sequence(length): Use the last two bytes written to the output stream to continue with a pattern. For example if the last two byte values were 02 03, the sequence would continue 04 05 06 07...
  • uint16LE_pattern_sequence(length): Same as byte_pattern_sequence() but with 16-bit short integers (Little-Endian encoded) instead of bytes. So note the total number of bytes written will be twice the value of length
Last Edit: January 24, 2013, 02:21:03 pm by denzquix
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630
Quote
Normality inaccessible area: the TV station set
Normality inaccessible area: Saul's cell
Normality inaccessible area: the Plush-Rest furniture machine
Normality inaccessible area: the park
 
Quote
Check out the comments by BL1TZEN on this vid
Wow this is great stuff!! Thank you, I've been doing searches to find anything Normality-related in preparation for this project but somehow missed your YouTubes completely...
Last Edit: January 21, 2013, 01:00:16 pm by denzquix
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630


all right this is starting to come together...

first. some more Nerd Shit. this is a Work-In-Progress description doc for the .DAS texture/sprite pack files in the MAPS folder:
 
/*
=======================================
Normality .DAS Texture/Sprite Pack File
=======================================

These files are found in the MAPS folder.
Any .MGL files found here are actually .DAS files that need to be decompressed first.

All multibyte integers are Little-Endian encoded.

(Pseudocode C struct syntax is descriptive only, not intended for actual code use...)
*/

// first in the file
struct File_Header {
  char[6] file_id;        // always "DASP\0\5"
  uint16_t unknown_1;     // 0x8000-0x8600 ?
  uint32_t always_0x28;
  uint32_t addr_palette;  // file address of Palette
  uint32_t unknown_3;
  uint32_t addr_names;    // file address of Name_Table
  uint32_t unknown_4;
  uint32_t unknown_5;
  uint32_t unknown_6;
};

struct Palette_Entry {
  uint8_t red, green, blue; // component values are 0-63 (VGA 6-bit)
};

struct Palette {
  Palette_Entry entries[256];
};

struct Image_Table {
  Image_Record records[?];
};

struct Image_Record {
  uint32_t addr_data; // file address of corresponding Image_Data
  uint32_t unknown;
};

struct Image_Data {
  uint16_t flags; /*
    ANIMATED: ((flags & 0x0100) != 0)
  */
  uint16_t width;
  uint16_t height;
  Animation anim[(flags & 0x0100) ? 1 : 0];
  uint8_t data[width * height];
};

struct Animation {
  uint32_t unknown_1;
  uint16_t length;
  uint16_t unknown_2;
  uint8_t always_0xFFFFFF[3];
  uint8_t speed; /*
    how speed values translate into frame duration (approximate!!!):
      0x02: 70ms
      0x03: 57ms
      0x04: 100ms
      0x06: 128ms
      0x07: 170ms
      0x08: 186ms
      0x0A: 214ms
      0x0E: 270ms
      0x10: 57ms
  */
  uint32_t addr_delta[(length / 4) - 5]; /*
    file address of delta for frame i is
 the address of the "length" field + addr_delta[i]
      unless addr_delta[i] is zero. then just keep previous frame
    final (non-zero) delta always just restores the original image
      any further zeroes after that frame seem to be meaningless...
  */
  uint32_t unknown_3;
  uint32_t unknown_4;
};

struct Name_Table {
  uint16_t texture_count;
  uint16_t sprite_count;
  Name_Record texture_records[texture_count];
  Name_Record sprite_records[sprite_count];
};

struct Name_Record {
  uint16_t unknown_1;
 
  uint16_t index; /*
    if top bit is set, index is a sprite
    file address is File_Header.addr_textures + (index * 4)
  */
   
  char short_name[?]; // null terminated
  char long_name[?];  // null terminated
};
edit: see this post for info about animation frame delta encoding

here are a few examples of what this can extract so far (edit: out of date!! palette and animation is now understood...)
*http://www.youtube.com/watch?v=UcCHRW8G9yY starts playing....*


BUILDIN3 "Fluffys" from EUREKA0.DAS



SKIPWAL5 "Skip wall5" from DEN.DAS



BUSTOP "Bus Stop" from FACTOUT.DAS



TUNENT1 "Shoe Shop" from MINTMALL.DAS



PILLOW "Pillow!" from EUREKA0.DAS



CINDOORS "Cinema Doors" from TRUCKTV.DAS



still early days.....
Last Edit: January 24, 2013, 03:07:14 pm by denzquix
  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
fantastic, glad you're making progress :)
  • Avatar of jamie
  • ruined former youth seeking atonement
  • PipPipPipPipPipPipPipPipPip
  • Group: Premium Member
  • Joined: Jun 4, 2003
  • Posts: 3581
keep going...
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630






good news friends!! i have worked out where the palettes are in the .DAS files


i still don't know how to get the animation frames, and some textures/sprites that i don't think are even animated are still coming out all messed up


but, time for some more http://www.youtube.com/watch?v=UcCHRW8G9yY ...





















  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630
bonus: piece on Normality's mocap tech by Violet Berlin for 90s gaming TV show "Bad Influence"


here's the bernard manning placeholder from the early game footage in the clip




don't think he ever appears fully but did get used for one of the CD covers in the music store


  • Avatar of Swordfish
  • Comrade!?! I AM NOT A FUCKING RUSSIAN
  • PipPipPipPipPipPipPip
  • Group: Premium Member
  • Joined: Sep 12, 2003
  • Posts: 1074
I think that what you are doing is super cool, but what's your ultimate end-game? As far as I know your goal is to find secret stuff but I want to know if you got any other ideas.
RIP DoktorMartini

My brute!
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630
Quote
I think that what you are doing is super cool, but what's your ultimate end-game? As far as I know your goal is to find secret stuff but I want to know if you got any other ideas.
no i don't got any further plans at the moment


but i'd love to see offshoot project(s) if someone wants to take em up, & i'd help where i can


ideas...
 
  • an exporter application
  • modding!! why not propose marriage to your loved one via craftily modified wall textures... or, do we dare to dream, an all-new game on the Normality engine...? don't hold your breath on that tbh... i still have no idea how much of the room objects, events etc. are hardwired into the exe
  • using ultimate papercraft 3D to make a sweet diorama of Kent's apartment, an ideal centerpiece for the modern home
  • faithfully recreating part of the game in Unity3D or w/e
Last Edit: January 21, 2013, 04:45:35 pm by denzquix
  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
bonus: piece on Normality's mocap tech by Violet Berlin for 90s gaming TV show "Bad Influence"
Awesome, I knew this existed but I never bothered to look it up. I can't for the life of me figure out what cutscene they're recording for though.
  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
I would love to make a faithful recreation of Kent's Apartment inside a browser window using maybe something like WebGL.
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630
i've been trying to decode the .GDV cutscene video files... this is actually documented for once http://wiki.multimedia.cx/index.php?title=Gremlin_Digital_Video


not lookin so good yet though






there's actually an existing tool out there that can play .GDV files and even export every frame as images, but the reason i'm doin this is I suspect the animated textures/sprites will use same/similar encoding to the video....
  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
imo that actually looks wonderful in its own right.

Nice to see you're still making progress :)
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630





  • Avatar of dada
  • VILLAIN
  • PipPipPipPipPipPipPipPip
  • Group: Administrator
  • Joined: Dec 27, 2002
  • Posts: 5533
whoa cool, is that something you found among the files or is that FANART?
  • Avatar of denzquix
  • PipPipPipPipPip
  • Group: Member
  • Joined: Aug 22, 2012
  • Posts: 630
it was in the manual PDF that came with the gog.com installation, i dunno if it was in the original game materials or what