This is second (and last) post about my quest to make a real-time game playable in stock Notepad.exe. In the previous article, I talked through using a quick and dirty memory scanner to get access to Notepad’s on screen text buffer (and build a ray tracer that rendered into it). In this post I’m going to talk about how I handled getting user input, and finally ended up at a fully playable Snake game in stock Notepad.
The title of this post gives away the fact that I ended using hooks to capture user input, but I originally thought I could do it with just DLL injection instead. I barely knew what DLL injection was but I knew it could cause things to happen in an already running process. This seemed like a decent place to start. As it turns out, you need to understand dll injection to work with hooks anyway, so it’s not a bad spot to start this blog post too.
I started by googling the hell out of “DLL injection,” and found this excellent article that breaks down what DLL Injection is and has a great github repo with examples of different ways to go about it. I didn’t have a clue about how I was going to use any of this capture keyboard input, but I figured I’d try to inject something simple into a running Notepad process anyway.
Based on the injection article I just linked, the easiest way to inject a dll seems to be:
Writing a DLL that does something in dllmain() is really easy if you aren’t doing a whole lot with it. I found later on that there’s a whole lot of stuff that you can’t do in dllmain (more info here), but for my first test project I just popped open a message box. The entire code for the DLL payload was just a few lines.
The tricky part, as you might imagine, was getting Notepad to load this in the first place. Just like the above payload, my injection code was almost entirely copied from the InjectAllTheThings repo I linked above. Unlike the payload, it’s a lot longer. I’m including it here because if you’ve never seen how to do this before, I assume this will be more convenient than having to click a link to github, but I’m not going to dive into how it works because the article/repo I linked above can teach you about it a whole lot better than I can.
This was enough to get a message box popping up in a running instance of Notepad, which was super cool. Unfortunately I realized pretty much immediately after I got this working that I had no idea how to go from popping a message box to using this to actually change Notepad’s behaviour.
My message box app could make something new happen in another process, but I actually needed to be able to change the behaviour of the target process. I had heard vaguely about api hooking before, and my limited understanding of it was that it allowed you to either replace existing code paths, or add additional functionality to them. This seemed roughly in line with what I wanted, so I dove down this rabbit hole next.
Googling for how hooking works was less straightforward than dll injection, mostly because hooking is much more complicated. I eventually realized that as long as I wanted to change a program’s reponse to a Windows system message, I could bypass a lot of this complexity and use a Win32 hook. Given that keyboard input is sent to Windows processes via WH_KEYBOARD messages, I was in luck.
The MDSN page for hooks provides some basic information about how these types of hooks work, but the general idea is like this (note: I’m a super beginner at all of this so take everything I say with a grain of salt):
Given this information, it seemed reasonable to try to intercept the keyboard events sent to Notepad by creating a hook function which intentionally didn’t call the next function in the hook chain. After persuing the msdn docs page about using hooks, I figured out that I was going to need to install a WH_KEYBOARD hook into Notepad’s Edit control.
The docs also point out that if you want to install a hook in a process other than your own, what you’re really doing is a form of dll injection. You need to place the hook function in a dll, and use SetWindowsHookEx() to load that dll’s code into the target application.
So with all that in mind, I put on my robe and wizard hat and got to work.
I started off by just trying to prevent Notepad from receiving keyboard input at all. All I needed to do for this was to hook the WH_KEYBOARD and then not call the next hook in the hook chain, which seemed like an easy place to start. To write a hook function for WH_KEYBOARD, all you need to do is make sure to match the function signature of KeyboardProc(). Given that I needed this function to do basically nothing, this was pretty easy:
The code for installing a windows hook is very straightforward (and shown below).
The threadId function argument is used to install the hook only for Notepad’s Edit control (otherwise it becomes a global hook). Getting the thread id is juat a matter of calling GetWindowThreadProcessId() on the HWND for the Edit control. You can get the HWND with the GetWindowForProcessAndClassName() function from my last post. Here’s that function again:
One thing to note about the installRemoteHook() function is that because it gets the function pointer for the callback with GetProcAddress(), the compiled name of the hook callback is important. This meant that I needed to make sure that to export that function using “extern C” to prevent the compiler from mangling the function name.
If you want to see what all of this looks like in pactice, the github repo for this blog post has a proof of concept hooking app uses the hook payload described above to disable key input to an instance of Notepad.
Simply preventing Notepad from getting keyboard input was cool and all, but it was a far cry from being able to redirect that output to a game. What I wanted to be able to do was both prevent Notepad from getting keyboard input (so that the user couldn’t type characters and mess up what I was rendering), and redirect that key input to the process I was using to control my game logic.
Redirecting the key input to a different process wasn’t much more difficult than preventing key input. I just copy/pasted the code for disabling key input and made the following changes:
I’m not going to walk through how to set up windows sockets (but all the code for doing so is on the github page for this project). Instead, I just want to share the hook payload that I used to make this all happen.
Extracting the key state from the lparam was a little weird, but it seemed like the best way to get at that information. If you wanted to write a more robust input handling hook, you’d probably care about more of the data in that parameter than I did, but this was enough for getting WASD.
Once this was working, it was a very small jump from there to a working real time game.
So yeah, the fruit of all this labor isn’t super exciting. I made Snake. It lends itself super well to ascii graphics (even if the fact that characters are taller than they are wide is a bit annoying), and I already had the gameplay logic written from a couple posts ago.
There’s not really much interesting to say about implementing Snake, and I’ve already talked through everything else, so I’m going to end things off with another gif of me playing snake in a hijacked Notepad.exe window. I hope you enjoyed the process of getting here as much as I did, because the end product is (as promised) super dumb.