Bogus BNK File Entries

2008-11-05, 13:54 by Keshire

Rules for posting on this site.
Follow them or have your posting rights removed or be banned.

[cut=Full List Of Rules]* If a topic makes no sense to you, do not ask broad questions about it. We have a lot of developer posts that will make no sense to someone who doesn't know how to program or has not taken the time to explore the game files.

* All modifications we release for the xbox will require a modified xbox. Ask for instruction and then tell us you don't have one will result in banning/deletion of all posts for failure to read instructions. THERE WILL BE NO SAVES ON THIS SITE SO DO NOT REQUEST THEM.

* We do not endorse pirating. Suggesting or asking for such will get you banned. If you like a game BUY IT. People worked hard to make something to entertain you, it's the least you can do. If you are caught without one of these you will be banned.

* Posting of original game files is strictly prohibited. If you have code to share post just the code you have made. Only attach files if you have created an application or something similar that requires the attachment. If you made many changes to a file but did not change ALL of it then post a PPF or something similar to avoid posting any original code at all. Do not link to original files either. Either way it's just the same as posting them here.

* Harassing admins/moderators through PM/Email because of your own inability to work things out with other people in the forums will result in loss of PM an removal of ability to post.

* Harassing admins/moderators through PM/Email because of your own inability to read posts and stickies in the forums with any compitence will result in loss of PM an removal of ability to post.

* Harassing or just flooding a topic/person because you are too lazy to use search will get your posting rights removed. Search exists for a reason and we won't deal with people that constantly ask for things that are easy to find with search.

* Public bashing of other members will get your posts removed. If it is done repeatedly you will get your account banned.

* Posting a reply to merely say you don't know or you also want to know something is a waste of space and all of our collective time (assuming you weren't asked directly, but if you were it probably should have been done in PM). Repeated offenders will get their posting rights removed.

* Do not cross post questions/requests in forums. Pick the appropriate forum and post it there. People read in all of the forums and don't need to see the extra spam.

* Everything on this site is provided AS IS. If you use a tool that someone wrote here and it crashes your computer, we don't care. The programmer should be informed with as much information as you can provide in order to prevent that from happening again though.

* Being an open forum, try to contribute to it. As in by helping crack file formats or developing new tools or game modifications. Leeches are never appreciated. Especially when they are ungrateful for the work that other people put into doing something for them. Being ungrateful is a very quick path to being banned.

* Lack of acting like a mature person or inability to type in proper english or something close to it will not be tolerated (i.e. l33t,ALL CAPS, or IM shorthand).

Basically, this is a technically based Fable modding site. This is NOT a playground. [/cut]

If you have a problem with these rules, don't come here.

Failure to not read the stickies on forums (like this one) is not an excuse. Failure to read this post is also not an excuse.

LittleCodingFox

2008-11-11, 10:40 #1

Member since Nov 2008 · 34 posts
Group memberships: Members

Show profile ·

Link to this post

Subject: Bogus BNK File Entries

Hello everyone.

I'm having a bit of an issue regarding BNK File Entries.

I read the BNK File Header, and then seek to the offset of the File Data (as described on the Formats thread), however when i read the File Table, i get a bogus number of entries (about 120 million), which is obviously wrong.

When exactly should i read the BNK File Entries? Should i read it just after reading the BNK Header and CompressedTable?

Here is my File Table reading function:

void IFileTable::DeSerialize(CStream Stream)
{
    unsigned long FileCount = 0;
    READV(FileCount, sizeof(unsigned long), 1);
    Entries.Allocate(FileCount); //I breakpoint here to check FileCount
    for(unsigned long i = 0; i < Entries.Size(); i++)
    {
      Entries[i].Swap(new IFileEntry);
      Entries[i]->DeSerialize(Stream);
    };
};

The BNK Header reading function:

void IBNKHeader::DeSerialize(CStream Stream)
{
    READV(Offset, sizeof(unsigned long), 1);
    READV(Unknown, sizeof(char), 4);
    READV(CompressFileData, sizeof(unsigned char), 1);
    READV(CompressedTableSize, sizeof(unsigned long), 1);
    READV(DecompressedTableSize, sizeof(unsigned long), 1);
    if(CompressedTableSize)
    {
      CompressedTable.Allocate(CompressedTableSize);
      READV(CompressedTable[0], sizeof(char), CompressedTableSize);
    };
};

And here's what READV is:

#define READV(object, size, count)\
Stream->Read(&object, size, count);\
EndianConvert(&object, size * count, true);

I hope someone can give me a hint as to what i am doing wrong.

Thank you for your time regarding this issue, and have a nice day!

EDIT: If it isn't obvious, the "CompressFileData" flag is 0.

This post was edited on 2008-11-11, 10:53 by LittleCodingFox.

JohnDoe Moderator 2008-11-11, 15:50 #2

User title: Community Pet
Member since Nov 2008 · 19 posts · Location: Texas
Group memberships: Global Moderators, Members

Show profile ·

Link to this post

Wrong endian perhaps? XBox360 uses big endian. The only reason I think of this is that someone else had such a misunderstanding, maybe it applies here but without knowing which file and the exact number of entries, I can't check. ;-)

An optimist is a pessimist in the making.

LittleCodingFox

2008-11-12, 00:39 #3

Member since Nov 2008 · 34 posts
Group memberships: Members

Show profile ·

Link to this post

I'm already converting from Big Endian to Little Endian. Maybe the issue is with how i convert it then.

I'm not at home right now, so i cant show you the code for my endian conversion code, but i'll post it later.

As a side note, when i open skeletalmorphs.bnk on notepad (i'm using it for my BNK Reader, too), i can actually read the text file in there as a normal text file, not an inverted text file (as you'd expect from big to little endian). Is that normal?

Keshire Administrator 2008-11-12, 02:16 #4

Member since Nov 2008 · 29 posts
Group memberships: Administrators, Members

Show profile ·

Link to this post

I didn't think ascii files followed endian?

Apathy Cannot Inspire.
Ambivalence cannot lead.
Loved me. Feared me.
Changed me. Killed me.
Anything would be something.
But nothing is worst of all.

LittleCodingFox

2008-11-12, 04:17 #5

Member since Nov 2008 · 34 posts
Group memberships: Members

Show profile ·

Link to this post

Oh, right, sorry. I dont have a very good understanding about endianess, so i guess that'd be what you call a "newbie error".

Anyhow, here's my EndianConvert code:

int IsBigEndian = -1;
void EndianInit()
{
    short int word = 0x0001;
    char *byte = (char *) &word;
    IsBigEndian = (byte[0]) ? (0) : (1);
}

void EndianConvert(void *Data, unsigned int Size, bool Reverse)
{
    unsigned char *RealData = (unsigned char*)Data;
    if(IsBigEndian == -1)
      EndianInit();
    if(IsBigEndian == !Reverse)
    {
      register int i = 0;
      register int j = Size-1;
      unsigned char Tmp;
      while (i<j)
      {
      Tmp = RealData[i];
      RealData[i] = RealData[j];
      RealData[j] = Tmp;
      i++, j--;
      }
    };
}

Please notice that i am not the original creator of the endianess code.

Using globals_model_headers.bnk:

Offset is 65536, not compressing data, unknown is {3, 0, 0, 0}, CompressedTableSize is 7967, DecompressedTableSize is 65536.

Should i read CompressedTableSize bytes? I'm doing that right now.

Afterwards, i seek to Offset, and deserialize the file Table. It tells me it has 1298494312 files.

If i dont seek to Offset, it tells me it has 6965 files. However, when i read the filenames and offset and such, no filenames are valid (first one starts with 0, 2nd one is binary trash, and so on).

Should i seek to Offset like fseek(File, Offset, SEEK_CUR), or should i seek relatively to the current position after reading the Header?

Any tips would be helpful.

TodX Administrator 2008-11-12, 10:54 #6

Member since Oct 2008 · 16 posts · Location: San Diego
Group memberships: Administrators, Members

Show profile ·

Link to this post

Offset just tells you where the file data section starts, it has nothing to do with the file table. You need to decompress the file CompressedFileData and use the output of that to read in the file table. The file table will then have offsets into the file data section for each file entry.

The number you're seeing is actually the first peice of data of the first entry:
1298494312 = 4D 65 73 68 = "Mesh"

I suggest getting a hex editor and trying some of these formats by hand. It helps a lot in understanding the data formats. Or use hex workshop and the structs we provide and it'll do most of the parsing for you.

Also the endian conversion code you have looks kind of odd. If I'm reading it right it is using global variables for a few things and using loops to do the endian conversion which is relatively slow, and can add up if you're going to be doing it on every little peice of file data. I'll be posting the code to the BNK extractor soon, you can see an alternate way of doing it in there.

P.S. This format is inherently not serial as it requires random access (aka using seek). It's just terminology, but I thought I'd mention it since you have Serialize and Deserialize names in there.

This post was edited on 2008-11-12, 11:00 by TodX.

LittleCodingFox

2008-11-13, 03:07 #7

Member since Nov 2008 · 34 posts
Group memberships: Members

Show profile ·

Link to this post

Oh, i call DeSerialize to some functions because if i have a DeSerialize function per "component", it becomes easier to understand the code, and it's "prettier". I consider DeSerializing the process of converting file data into usable data.

I apologise for my "ways of coding", i'm self-taught so some things i consider to work in a way should actually be called something else, or work a bit differently.

LittleCodingFox

2008-11-13, 12:03 #8

Member since Nov 2008 · 34 posts
Group memberships: Members

Show profile ·

Link to this post

Sorry for double posting.

I've fixed all my bugs, and now have a proper reader. Thanks for your help everyone, including those who contributed for the codebase (which i used to figure out that you dont get one entry per "compressedsize/decompressedsize block", and solved everything).

I'll be working on my codebase, i'll let you guys know if i have anything minimally interesting to share.

EDIT: there's sample output of my "BNK Archive File Lister" test attached.

Does anybody know of any compressed archives i can use to test it with compressed files? Not that i'm lazy, but i've got to leave ASAP so i cant search right now.

The author has attached one file to this post:

files.txt | Save 125.9 kBytes, downloaded 374 times

This post was edited on 2008-11-13, 12:55 by LittleCodingFox.

Announcement