Today, I emailed L spiro, asking how Restricted mode works, and this is what he said:
Opening in Restricted Mode causes MHS to access the target process without being attached to it. It does not call OpenProcess() so hooks on that function are avoided, not does it maintain a HANDLE to the target, so HANDLE enumerations searching for the protected process also fail.
If you want to make your own, you need a kernel-mode driver and you must code everything from scratch.