Jump to Content
Register
Forgot your password?
Third-party login
Log in with Discord

Handmade Hero
Forums
Episode Guide
twitter
Handmade Hero»Forums»Code
← Back to index
Scott Hunt
41 posts
Father, Thinker, Mechanical Engineer @NASA, and C/C++ Hobby Enthusiast / @TexxStudio
#16800
stream and buffer usage
Edited by Scott Hunt on
Good evening everyone, couple questions as I'm attempting to learn the stream and buffer mechanisms Casey has added recently.

Making a quick OBJ 3d model parser, I've pulled in the readstream from the file reader and while reading through it using a slightly modified version of the token struct when debugging I've noticed that the stream pointer (At) gets corrupted at the end of it's data. It's not clear to me why this is happening except maybe it is reading off the end of the stream as the contents? It doesn't fail as the contents.count hits 0 and breaks the reading, but I can't figure out why it's doing this.

Also is there a better way to handle this text stream data reading? I wanted to learn how to tokenize strings so I went this direction in-lieu of the fscanf and FILE handle and while it works, I'm not convinced this is the data approach Casey's buffer and streams were intended as I'm having to cast to a (char*) to convert to a value.

Any thoughts are appreciated.

Sample code: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
static void ConsumeLine(FString* Source)
{
    for(;;)
    {
        FToken Token = PopToken(Source);
        if(StringsAreEqual(Token.Text, "\n"))
        {
            break;
        }
    }
}

static FOBJModelData ParseOBJ(FMemoryArena* Arena, FStream File)
{
FStream* At = &File;
while(At->Contents.Count > 0)
{
    FToken = PopToken(&At->Contents);
    if(StringsAreEqual(Token.Text, "#"))
    {
        ConsumeLine(&At->Contents);
    }
    else if(StringsAreEqual(Token.Text, "v"))
    {
        Vertices[VertexCount++] = vec3{strtof((char*)PopToken(&At->Contents).Text.Data, 0),
                strtof((char*)PopToken(&At->Contents).Text.Data, 0),
                strtof((char*)PopToken(&At->Contents).Text.Data, 0)};
        PopToken(&At->Contents);
    }

    ...

    }
}
}
Simon Anciaux
1340 posts
#16808
stream and buffer usage
Edited by Simon Anciaux on Reason: typo
It's hard to find the problem with you At pointer without more code (if possible a simple reproduction case that we can compile and debug). At least we would need your PopToken function and the FStream and FString struct definitions.

I haven't watched the last few episodes of HmH so I don't know what is Casey's approach, but if your problem is the casting, you can use a union to have several pointer type. 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
typedef struct token {
    umm length;
    union {
        void* data;
        char* text;
        umm offset;
    };
} token;

token t = PopToken( &stream );
r32 number = strtof( t.text, 0 );
Scott Hunt
41 posts
Father, Thinker, Mechanical Engineer @NASA, and C/C++ Hobby Enthusiast / @TexxStudio
#16898
stream and buffer usage
Edited by Scott Hunt on
Hey Simon, sorry for the delayed follow-up, been a hectic week. I added you to a git repository I'm using as a test bed for the things I'm learning from Handmade Hero. If you have a quick moment without inconvenience, you can clone and build it (using the HH style build.bat), any insight into the Visual Studio showing the junk at the end of the stream would be beneficial. I believe it's from memory being written over, but I can't tell for sure. The part I'm curious about is *At stream when loading in a OBJ file.
Simon Anciaux
1340 posts
#16903
stream and buffer usage
Is there something to do to load a obj ? The code I downloaded doesn't call ParseOBJ as it is, and the test code in UpdateAndRender is in a #if 0. Switching it to 1 causes a few compile error.
Scott Hunt
41 posts
Father, Thinker, Mechanical Engineer @NASA, and C/C++ Hobby Enthusiast / @TexxStudio
#16908
stream and buffer usage
Edited by Scott Hunt on
Hey Simon, I apologize for not providing better information and appreciate you taking time to review.

It uses Casey's asset file style. Use oeaedit to create a local.oea under the data folder. Once done any .obj files in the art folder will get automatically parsed in to the local.oea for future runs. If the local.oea is already there, you can delete it and recreate with oeaedit.
Simon Anciaux
1340 posts
#16919
stream and buffer usage
Edited by Simon Anciaux on Reason: typo
There doesn't seem to be a problem with the At pointer, as it's never modified in the ParseOBJ function.
At->Content->Data points to garbage data after parsing the file because when you Pop a token you advance that pointer. At the end of the loop it's expected that it would point 1 byte past the end of the valid data. The garbage data is whatever was in memory before you used it to read the file. If there was a 0 after the file content (you could for instance make your file reading function add a zero at the end) the debugger would stop showing the garbage as it would consider that a null terminated string.

It's a personal choice but instead of using a counter, I like to write those loops like this: 
1
2
3
4
5
6
uint8_t* current = At->Content->Data;
uint8_t* end = current + At->Content->Count;

while ( current < end ) { /* The loop ends when the current pointer is past the end pointer. */
    ...
}


To avoid casting to char* you could change you FBuffer struct to be 
1
2
3
4
5
6
7
8
struct FBuffer
{
    umm Count;
    union {
        u8* Data;
        char* Text;
    };
};


Your PopToken function could just return a FBuffer since you don't seem to use the "value" part of the Token.

The following code could potentially read invalid memory if the file was corrupted (but maybe you can trust file and don't care about that), since you don't test for the length of the token before using it: if the file was truncated it would return a token with the data pointer after the end of the valid data and a count of 0. 
1
2
3
4
5
Vertices[VertexCount++] = vec3{
    strtof((char*)PopToken(&At->Contents).Text.Data, 0),
    strtof((char*)PopToken(&At->Contents).Text.Data, 0),
    strtof((char*)PopToken(&At->Contents).Text.Data, 0)
};
Scott Hunt
41 posts
Father, Thinker, Mechanical Engineer @NASA, and C/C++ Hobby Enthusiast / @TexxStudio
#16923
stream and buffer usage
Thanks Simon, makes sense about it reading garbage due to no null termination. I went ahead and updated the FBuffer struct with the union, it's definitely cleaner. Appreciate your time in reviewing through.
← Back to index Log in to reply