I actually have wondered how JIT (Just-In-Time compiling) works for quite a long time. It is basic knowledge that the code segment in the computer’s memory is write-protected, and you cannot move the CPU instruction pointer to the stack frame or the heap for security reasons. But to JIT, you need to generate new machine code in memory and instantly execute it. How?
I stumbled upon this article titled “The Joy of Simple JITs”, which get me started. Apparently, you can do mmap (POSIX) or VirtualAlloc (Win32) to allocate virtual files in memory. And you can write to this section. And you can execute it. Looking great?
If you have read that article, you would have noticed that the hardest part of doing JIT is the instruction encoding. I am not going to go into the full detail here – you can easily find how to encode instructions on various platforms. I have used some JIT engines in other projects before (Avisynth was using SoftWire, but it no longer using JIT). And the JIT library alone is half of the binary size!
DynASM, mentioned in that article, is very interesting. It encodes all instructions compile-time! At runtime, it’s just concatenation and substituting parameters. This makes the runtime very small.
So to play with it, I wrote some little brainfuck JIT program. (Technically not JIT, it’s a brainfuck dynamic code generator) with some optimization. It was quite fun. This is the very first time I have actually written some assembly myself (normally I use C intrinsics function). The hardest part is actually to get the function prologue and epilogue right.
The code is available on GitHub.