Why would you want somebody to prevent running two instances of game simultaneously? Sometimes it actually could be useful, for example, to test some networking functionality.
Anyways, if you really want to disable second instance, you need to know that there is one already running. You need to have something in system that you can check. On Windows it can be one of several things:
1) Named mutex - create it with CreateMutex
2) Named event - use CreateEvent
3) Named pipe - use CreateNamedPipe
4) Memory mapped file - use CreateFileMapping (with handle = -1)
5) ... probably something else that has name
All these methods accept some string as name and can return status if object with such name existed before or not.
If you open pipe you can even communicate between instances you open later and first instance (by passing strings around).
Alternative way would be to enumerate process list with EnumProcesses function and check executable location if it is same as ours. But with this method you must be a bit careful so both processes don't think they are second instance and both think that they need to exit - you could for example make process with lowest process identifier (PID) as "first" process, and don't exit if your process id is less that other process id.