← Back to index

Asset System - Loading files in general (unknown allocation size)

Matyas
#21411
Hi everyone,

I have been working on an asset system I have architected similarly to the one Casey did in Handmade Hero.

The requirements in my system turned out to be different though, since I don't store "raw" decoded data along with its size in files like Casey does for .hha files. (Maybe this has changed in newer episodes though?)
For instance, I store glyphs in ttf-s and rasterize them with freetype2 on the fly.

Here's an example: 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
internal_fn void
load_font(Assets *assets, Font_ID id)
{
    if (interlocked_exchange(assets->fonts[id.value].state, Asset_State_Queued, Asset_State_Unloaded)) {
        
       // NOTE(Matyas): Allocate font slot, from the Asset stack arena
       Loaded_Font *font = Push_Struct(&assets->arena, Loaded_Font);
 
       // NOTE(Matyas): Initialize font struct...

       Load_Font_Work *work = Push_Freelist_Node(&assets->work_freelist, Load_Font_Work);
       
       // NOTE(Matyas): Initialize work struct...

       // NOTE(Matyas): Start thread
       Platform->add_entry(assets->high_queue, decode_font_work, work);
       
    } else {
        assets->fonts[id.value].state = Asset_State_Unloaded;
    }
}

// NOTE(Matyas): This is called by load_font, runs on a separate thread!
thread_safe internal_fn
WORK_QUEUE_CALLBACK(decode_font_work)
{
    // NOTE(Matyas): Load the font from file, then decode it
}


The issue I have is that I am decoding data on-the-fly, so I don't _actually know_ what size I should allocate for in load_font for the decoded asset. This means of course that I can't actually make my allocation in load_font implying I have to do it in decode_font_work.
The issue with this is that since decode_font_work can allocate at any time from any thread, a simple stack or free list allocation strategy wouldn't work... (at least that's my assumption)

Can anyone suggest a good way to architect such a system memory/allocation-wise?
Any suggestion is much much appreciated!

PS.
Sorry if this issue came up before, I haven't found anything similar to this case.
Edited by Matyas on
ratchetfreak
#21412
Is there anything stopping you from storing the "raw" loaded size along side the asset description?
Matyas
#21413
Well I could store the allocated size in the Font_Info struct in decode_font_work, my issue however is that I don't know what allocation strategy to use!
I have a fixed size stack arena for Assets like Casey does, the problem is that since decode_font_work is on a different thread, I am not sure how to solve allocation conflicts: suppose two threads call Push_Size(...) at the same time for instance.
Another issue is that decode_font_work might allocate temporary memory just for the decoding,
and I would need to free that later somehow (probably by using a free-list, but the same issue of allocating on different threads at once arise).
ratchetfreak
#21414
Give each worker thread its own Arena for that temp storage.

And when creating the job allocate the target memory in the main thread and pass the pointer along through the work struct.

Simon Anciaux
#21415
You could make push_size thread safe. It boils down to use InterlockedCompareExchange on the "used" variable of the arena. Something like (not tested): 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
typedef struct arena_t {
    umm reserved, used;
    u8* bytes;
} arena_t;

void* push_size_safe( arena_t* arena, umm size ) {
    
    void* result = 0;
    
    while ( !result ) {
               
        umm used = arena->used;
        umm new_used = used + size;
        umm reserved = arena->reserved;
        
        if ( new_used <= reserved ) {
            
            long used_check = InterlockedCompareExchange( &arena->used, new_used, used );
            
            if ( used_check == used ) {
                result = arena->bytes + used;
            }
            
        } else {
            break;
        }
    }
    
    return result;
}


For the temporary memory, one solution is to have a "working memory block" per thread that you allocate when the thread is created and that is reset each time a new task is starting.
Matyas
#21416
Thank you guys for your help!

@ratchetfreak
Yep, giving each thread its own dedicated arena seems to be the way to go.

"And when creating the job allocate the target memory in the main thread and pass the pointer along through the work struct"

I can't allocate memory beforehand in the main thread, because the allocation size is determined while doing the decoding. In the case of loading fonts, packing glyphs is an NP problem by nature and I don't know what the size of the atlas will be before actually doing the decoding work on the thread. I also cannot approximate the size of the allocation, since the atlas size can range fron 256x256 to 2048x2048 for CJK fonts.

@mrmixer
This function is what I've been looking for I think! I'll try implementing it in my system, I think it should solve this problem.

Thanks again!

Edited by Matyas on
Reply to Thread
← Back to index