I think you are greatly confused about phrase "from scratch" here.
From scratch in Handmade Hero context means when you write thing, you understand how it works from A-Z. Not using OpenGL and writing bunch of pixels to memory buffer is as "from scratch" as it can be. This memory buffer can be blitted to screen with GDI, or with OpenGL or even with Vulkan - that all is fine. It's just a minor detail.
But even drawing triangles with OpenGL also is "from scratch" - if you prepare input data yourself and call necessary GL functions yourself. As long as you don't relay on things like Unity or UE4. Because for them you don't know what happens in their insides to get graphics out. Which means it is harder to debug things and harder to know why it works with worse performance than you expect, etc...
You can avoid talking to Xserver if you want to do raw OpenGL. But that will involve few things:
1) not running X11 at all, because that overrides who is accessing GPU - which means you need to reboot to console mode to run your application
2) using libdrm library (can be avoided if you directly talk to kernel KMS) or using EGLStreams (if you are on Nvidia binary driver as they don't work the same as rest of open-source gpu drivers). This is needed to initialize GL context, handle memory allocations.
3) using whole Mesa library (or nvidia binary driver) if you don't want to reimplement OpenGL. Because GPU drivers in kernel do not know OpenGL. It's responsibility of Mesa to translate GL API calls to something that GPU understands.
You can see how this can be done in my github repo here: https://github.com/mmozeiko/rpi/tree/master/gles2_drm
This code works on RaspberryPi or on Intel GPU. It will probably work also with AMD using open-source driver, but I have not tested that. For Nvidia it will require replacing libdrm & libgbm with EGLStreams.
In general that is a lot of pain, and unless your goal is to learn how program without X11 server, don't do this.
Using Mesa library will still mean you are using GL. At that point what is reason to avoid X11? That is such a tiny party of application. Rest of your code will be multiple of magnitudes larger and more complex.