Hi, I have a problem with queuing jobs into fixed size circular queue when multiple threads can produce new work. I've read a lot about this problem on the web. One way to extend the code Casey wrote is to use two semaphores instead of one (one to wake up consumer threads when there's new job to do and other to wake up producer threads when there's free space in work queue) + mutex for accessing the queue. There are a lot of resources on this topic on the web, I'll put here the shortest video I was able to find:
But if I'm understanding this well, implicit in all of these examples is the fact that producers and consumers are separate threads. But in my example thread is both consumer and producer at the same time. To be more concrete, I'm trying to recursively explore a directory structure using multiple threads. I'm using win API FindFirstFileW / FindNextFileW. The code starts from the root directory, and for each directory inside, it puts a new job in the queue. So job == directory. Other threads kick in and they start to pull those jobs. But, if directory has nested directories, thread will want to put new job/s after it finished its job. When I try to run this code on lets say Windows directory, I quickly end up in situation where all threads want to put new job into the queue, but the queue is full so they go to sleep. I don't know if I'm missing something obvious, but I don't think this particular problem can be solved using general abstraction of fixed size circular work queue.
I have two ideas how I might solve this:
1) Use growing queue.
2) Instead of queuing the jobs, each thread dig multiple directories on its own. When it's finished it can go and help other unfinished threads.
Any opinion on this?
P.S. It would be awesome if Casey at some point implements multiple producers/consumers queue on the stream. If not for the game, then for educational purposes.