Bogus BNK File Entries

2008-11-05, 12:54 by Keshire

Rules for posting on this site.
Follow them or have your posting rights removed or be banned.

CUT: Full List Of Rules

If you have a problem with these rules, don't come here.

Failure to not read the stickies on forums (like this one) is not an excuse. Failure to read this post is also not an excuse.

LittleCodingFox

2008-11-11, 09:40 #1

Member since Nov 2008 · 47 posts
Group memberships: Members

Show profile ·

Link to this post

Subject: Bogus BNK File Entries

Hello everyone.

I'm having a bit of an issue regarding BNK File Entries.

I read the BNK File Header, and then seek to the offset of the File Data (as described on the Formats thread), however when i read the File Table, i get a bogus number of entries (about 120 million), which is obviously wrong.

When exactly should i read the BNK File Entries? Should i read it just after reading the BNK Header and CompressedTable?

Here is my File Table reading function:

void IFileTable::DeSerialize(CStream Stream)
{
    unsigned long FileCount = 0;
    READV(FileCount, sizeof(unsigned long), 1);
    Entries.Allocate(FileCount); //I breakpoint here to check FileCount
    for(unsigned long i = 0; i < Entries.Size(); i++)
    {
      Entries[i].Swap(new IFileEntry);
      Entries[i]->DeSerialize(Stream);
    };
};

The BNK Header reading function:

void IBNKHeader::DeSerialize(CStream Stream)
{
    READV(Offset, sizeof(unsigned long), 1);
    READV(Unknown, sizeof(char), 4);
    READV(CompressFileData, sizeof(unsigned char), 1);
    READV(CompressedTableSize, sizeof(unsigned long), 1);
    READV(DecompressedTableSize, sizeof(unsigned long), 1);
    if(CompressedTableSize)
    {
      CompressedTable.Allocate(CompressedTableSize);
      READV(CompressedTable[0], sizeof(char), CompressedTableSize);
    };
};

And here's what READV is:

#define READV(object, size, count)\
Stream->Read(&object, size, count);\
EndianConvert(&object, size * count, true);

I hope someone can give me a hint as to what i am doing wrong.

Thank you for your time regarding this issue, and have a nice day!

EDIT: If it isn't obvious, the "CompressFileData" flag is 0.

This post was edited on 2008-11-11, 09:53 by LittleCodingFox.

JohnDoe (Moderator) 2008-11-11, 14:50 #2

User title: Community Pet
Member since Nov 2008 · 37 posts · Location: Texas
Group memberships: Global Moderators, Members

Show profile ·

Link to this post

Wrong endian perhaps? XBox360 uses big endian. The only reason I think of this is that someone else had such a misunderstanding, maybe it applies here but without knowing which file and the exact number of entries, I can't check. ;-)

An optimist is a pessimist in the making.

LittleCodingFox

2008-11-11, 23:39 #3

Member since Nov 2008 · 47 posts
Group memberships: Members

Show profile ·

Link to this post

I'm already converting from Big Endian to Little Endian. Maybe the issue is with how i convert it then.

I'm not at home right now, so i cant show you the code for my endian conversion code, but i'll post it later.

As a side note, when i open skeletalmorphs.bnk on notepad (i'm using it for my BNK Reader, too), i can actually read the text file in there as a normal text file, not an inverted text file (as you'd expect from big to little endian). Is that normal?

Keshire (Administrator) 2008-11-12, 01:16 #4

Member since Nov 2008 · 43 posts
Group memberships: Administrators, Members

Show profile ·

Link to this post

I didn't think ascii files followed endian?

Apathy Cannot Inspire.
Ambivalence cannot lead.
Loved me. Feared me.
Changed me. Killed me.
Anything would be something.
But nothing is worst of all.

LittleCodingFox

2008-11-12, 03:17 #5

Member since Nov 2008 · 47 posts
Group memberships: Members

Show profile ·

Link to this post

Oh, right, sorry. I dont have a very good understanding about endianess, so i guess that'd be what you call a "newbie error".

Anyhow, here's my EndianConvert code:

int IsBigEndian = -1;
void EndianInit()
{
    short int word = 0x0001;
    char *byte = (char *) &word;
    IsBigEndian = (byte[0]) ? (0) : (1);
}

void EndianConvert(void *Data, unsigned int Size, bool Reverse)
{
    unsigned char *RealData = (unsigned char*)Data;
    if(IsBigEndian == -1)
      EndianInit();
    if(IsBigEndian == !Reverse)
    {
      register int i = 0;
      register int j = Size-1;
      unsigned char Tmp;
      while (i<j)
      {
      Tmp = RealData[i];
      RealData[i] = RealData[j];
      RealData[j] = Tmp;
      i++, j--;
      }
    };
}

Please notice that i am not the original creator of the endianess code.

Using globals_model_headers.bnk:

Offset is 65536, not compressing data, unknown is {3, 0, 0, 0}, CompressedTableSize is 7967, DecompressedTableSize is 65536.

Should i read CompressedTableSize bytes? I'm doing that right now.

Afterwards, i seek to Offset, and deserialize the file Table. It tells me it has 1298494312 files.

If i dont seek to Offset, it tells me it has 6965 files. However, when i read the filenames and offset and such, no filenames are valid (first one starts with 0, 2nd one is binary trash, and so on).

Should i seek to Offset like fseek(File, Offset, SEEK_CUR), or should i seek relatively to the current position after reading the Header?

Any tips would be helpful.

TodX (Administrator) 2008-11-12, 09:54 #6

Member since Oct 2008 · 17 posts · Location: San Diego
Group memberships: Administrators, Members

Show profile ·

Link to this post

Offset just tells you where the file data section starts, it has nothing to do with the file table. You need to decompress the file CompressedFileData and use the output of that to read in the file table. The file table will then have offsets into the file data section for each file entry.

The number you're seeing is actually the first peice of data of the first entry:
1298494312 = 4D 65 73 68 = "Mesh"

I suggest getting a hex editor and trying some of these formats by hand. It helps a lot in understanding the data formats. Or use hex workshop and the structs we provide and it'll do most of the parsing for you.

Also the endian conversion code you have looks kind of odd. If I'm reading it right it is using global variables for a few things and using loops to do the endian conversion which is relatively slow, and can add up if you're going to be doing it on every little peice of file data. I'll be posting the code to the BNK extractor soon, you can see an alternate way of doing it in there.

P.S. This format is inherently not serial as it requires random access (aka using seek). It's just terminology, but I thought I'd mention it since you have Serialize and Deserialize names in there.

This post was edited on 2008-11-12, 10:00 by TodX.

LittleCodingFox

2008-11-13, 02:07 #7

Member since Nov 2008 · 47 posts
Group memberships: Members

Show profile ·

Link to this post

Oh, i call DeSerialize to some functions because if i have a DeSerialize function per "component", it becomes easier to understand the code, and it's "prettier". I consider DeSerializing the process of converting file data into usable data.

I apologise for my "ways of coding", i'm self-taught so some things i consider to work in a way should actually be called something else, or work a bit differently.

LittleCodingFox

2008-11-13, 11:03 #8

Member since Nov 2008 · 47 posts
Group memberships: Members

Show profile ·

Link to this post

Sorry for double posting.

I've fixed all my bugs, and now have a proper reader. Thanks for your help everyone, including those who contributed for the codebase (which i used to figure out that you dont get one entry per "compressedsize/decompressedsize block", and solved everything).

I'll be working on my codebase, i'll let you guys know if i have anything minimally interesting to share.

EDIT: there's sample output of my "BNK Archive File Lister" test attached.

Does anybody know of any compressed archives i can use to test it with compressed files? Not that i'm lazy, but i've got to leave ASAP so i cant search right now.

The author has attached one file to this post:

files.txt | Save 125.9 kBytes, downloaded 5 times

This post was edited on 2008-11-13, 11:55 by LittleCodingFox.

Announcement