Here's what I want to do:
1. Load a font
2. Generate all bitmaps of the font
3. Get the baseline value for each character (I don't care about kerning)
4. Unload font (I never use any font directly, I only need the generated bitmap)

What's the simplest way to do this in win32.

Simplest way would be using stb_truetype.h
Load font with stbtt_InitFont.
Get codepoint metrics with stbtt_GetCodepointBox / stbtt_GetCodepointBitmapBox.
Get codepoint bitmaps stbtt_MakeCodepointBitmap / stbtt_GetCodepointBitmap.

With win32 you would need more steps to initialize GDI.
To load font AddFontResourceEx + CreateFont
Create Bitmap & HDC - CreateDIBSection, CreateCompatibleDC, SelectObject.
Get font metrics - GetTextMetrics.
Get codepoint size - GetTextExtentPoint32
Draw text to memory bitmap - TextOut / ExTextOut / etc...

I have my custom functions to draw text, just need to get all the bitmaps first. What about unloading font from memory after initializing?

There's nothing to call in stb_truetype.h - that just uses whatever memory you pass in.
In GDI case you delete bitmap with DeleteObject, hdc with DeleteDC and remove font resource with RemoveFontResource. These all functions are mentioned in MSDN docs when you read about their Create* functions.

The GetTextExtentPoint32 method calculates the size of the final bitmap for a given string, does it take to account the kerning?

How can I get the font name to pass in CreateFont? I heard that the name from the font file name doesn't always equal the actual font name, is it true?

You call GetTextExtentPoint32 on each character individually, not on whole string. Because you need to know size of each char in the bitmap anyway, so you can copy it out for your needs.

File name of font is irrelevant. You can rename arial font to a.ttf and it will still work. Font name is embedded in font.
I don't think there is a good and official way to get font name from file in GDI.
Hacky way would be using undocumented GetFontResourceInfo function with QFR_DESCRIPTION query: https://superuser.com/a/1452012
Documented way would be to enumerate currently installed fonts, call AddFontResourceEx, then enumerate again and find the name that is new.

I need to know the size of all the bitmaps upfront to allocate it first. Can I call GetTextMetrics and GetTextExtentPoint32 right after calling CreateFont, or do I must call it after CreateDIBSection?

Also, do I need to create a new device context with CreateCompatibleDC every time? Windows has multiple functions that use or create a hdc (CreateCompatibleDC, GetDC, BeginPaint, etc). Can I just create and store a device context at the beginning of the program from one of these functions and just keep using it? Do all the device contexts the same throughout the program?

Calling font functions will require passing HDC as argument. So that means font must be loaded & selected into this HDC for it to work.
DIB section matters only when you actually render glyphs - as it needs to write to memory.

As long as you want to use one font at a time, you can keep one HDC around forever. No need to recreate it. Switching to different font will require either calling SelectObject, or just keeping different HDC around.

How can I know how many characters a font support? And what are those characters? TextMetrics only has tmFirstChar and tmLastChar, but some custom fonts may only support parts of the alphabet (like some fonts don't support ~!@#$ or in entirely different languages like Chinese).

One ttf font can have max 65535 glyphs.
If you want to know exact glyphs that font provides, then it seems you can use GetFontUnicodeRanges function. I've never tried it, not sure if it'll work - if it doesn't then you'll need to parse ttf tables yourself.

Are all the glyphs in a font have the same height (equal to textMetric.tmHeight)? Are all the bitmaps of each glyph start at the left-most pixel and end at the right-most pixel? All the default Windows fonts that I was testing always have these two characteristics, so I wonder if this is the same for other fonts.
Also, can I pass all the characters (i.e "ABCDEFG...789") to the TextOut function?

Each glyph has different height when rendered to bitmap. Look at . and M characters - they very different height.
I don't know what you mean by "glyphs start at left-most pixel"? What is start? Bitmap starts on left and right side of bitmap. Glyph origin can be anywhere in bitmap - that you need to calculate.
Yes, you can render whole strings with TextOut. That will include kerning and I believe that can include things like ligatures and other combining character modifications to string.

If I use the GetTextMetrics and GetTextExtentPoint32W then all the characters will have the same size.cy (and equal to textMetric.tmHeight). But when I use the TextOut method after that, all the generated bitmaps have different heights, for some reason? Also, CreateFont makes you pass in a height argument (which I pass in zero, the same for the width argument), is it related to that? When I said "glyphs start at the...", I mean does the generated bitmap clamp to the actual size that a glyph needs, are there any empty pixel rows or columns? Does the first column of the bitmap have a valid pixel in it or does it empty?