When run with either underflow or overflow checking enabled, mprotect will fail after around 32,570 "protections", with errno 12 (ENOMEM Internal kernel structures could not be allocated.)

I am assuming this this is because of a limit on the space allowed in the kernel to store memory protection structures, I guess per process. It seems to consistently fail around that 32,570 number on this machine, whether underflow or overflow checking is used.

Is anyone familiar with this, or am I mistaken? Windows seems fine with almost a million "protections".

I think this is governed by the limit in /proc/sys/vm/max_map_count. You could try increasing that in the file or temporarily with:
sysctl -w vm.max_map_count=nnn

Both of these require root authority.

I think I may have been mistaken, I was trying to use the "Size" member of the "platform_memory_block" as the size to deallocate using munmap (as it requires a size to deallocate, unlike VirtualFree), but I forgot that the "Size" value isn't the full size of the allocation, it doesn't include the headers or extra protection pages.

It turns out it was a combination of both using the wrong size, and the vm.max_map_count limit being set to it's default of 65536. I increased vm.max_map_count to 2097152, (not sure if it matters if it is a power of two) and split LoopingFlags from it's previous size of u64 into a u32 and another u32 to store the extra size of the allocation.

When I just fixed the size, mprotect would fail after ~78000 "protections", not sure why it was more than the 65536 limit it was set to.

Thanks FieryDrake for the sweet info :)

edit: As for the ~78000 total "protections" before it failed, I think this is from a lot of those "protections" being deallocated, and the total number of mmap's and mprotect's in flight at any one time only reaches >65536 after around 78000 mprotect's have been done.

The guard page trick is neat, but it has limitations. If you are already using malloc/free (or can swap your allocations to use it instead), then using something like Address Sanitizer will generally give you better results.

From that link:

The tool can detect the following types of bugs:

* Out-of-bounds accesses to heap, stack and globals
* Use-after-free
* Use-after-return (runtime flag ASAN_OPTIONS=detect_stack_use_after_return=1)
* Use-after-scope (clang flag -fsanitize-address-use-after-scope)
* Double-free, invalid free
* Memory leaks (experimental)

Typical slowdown introduced by AddressSanitizer is 2x.

It's built into Clang (and recent versions of GCC), you can just pass -fsanitize=address to the compiler and linker.