Hooking D3D (all newbies pls read so u won't need to C+P others code)

[falzarex]

Ok I have found a very good reference on hooking techniques for direct3d
credits to a guy named Wang Jijun
EDIT: sorry for the long page but my iPod cant upload and I didn't wanna link

Hooking Direct3d
By Jijun Wang
Introduction:
In the field of hooking technologies, hooking COM based applications is a big challenging because you can’t know their member functions’ real addresses previously. There are many papers on the Internet that talk about how to hook (somebody call it hijack) an application. However, seldom of them mention how to hook COM. Since I have seen some people asked questions about this topic and nobody answered their questions clearly, I decide to write this paper to share my experience with the guys who are interested in hooking COM.
*
In this paper, I use direct3d, which is widely used in current gaes, as an example of COM. At first I will briefly introduce the hooking technologies and explain why I select Detours to hook direct3d. Then more details and examples of Detours will be presented and I will discus virtual functions and find the way to hook direct3d.
Hooking Technologies
The basic idea of hooking is injecting your code into a piece of code. When the target code executes your code will be invoked. To do this, you need to at first attach your code into the target process. And then inject your code into the target process’ code.
The ways to attach your code into another thread or process include
1) Register your DLL to the registry table.
This method registers your DLL to the key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs and the dll will be loaded when user32.dll initializes itself. It is a safe method. But it only works for the applications that use user32.dll. And it’s not convenient to activate/deactivate your DLL.
2) System-wide Windows Hooks
This method use SetWindowsHookEx(),CallNextHookEx() and UnhookWindowsHookEx() winapis to monitor the system’s message traffic and attach your code into the processes that fit the hook filter. Since it is system-wide message monitor, the system’s performance will be significantly affected.
3) Create Remote Thread
This method use CreateRemoteThread()to inject your DLL into another thread (the to-be-hooked thread). It’s a very effective method. But there is one thing we need to consider. You must make sure you have enough privileges to access the to-be-hooked thread.
*

To inject your code into another process’ code, we could use:
1) Proxy DLL
This method writes a new DLL to replace the to-be-hooked DLL. The new DLL must have the same name, exported functions and variables as the old one. For example, the GLtrace is a replacement of the opengl32.dll. It is used to trace OpenGL. Although you could use function forward to reduce the time spent in rewriting functions, it’s a tedious work in some cases. *
2) Altering of the IAT (Import Address Table)
This is a wildly used hooking method. Windows use IAT to find the functions’ addresses. Through changing the data of IAT, you could use your own function to replace the to-be-hooked function. The method is robust and easy to implement. However, it only works for statically linked API calls. And sometimes you may miss API calls since you must patch every module in the target application.
3) Code Overwriting
The basic idea is to overwrite the API’s binary code in memory. We could save and replace the first several codes of a function call to a JUMP code to jump to your codes and then go back to the original position and restore the saved codes. It’s very hard to implement. However, once you have mastered it, it’s really a good hooking method.
*

For more details about hooking technologies you can read the wonderful paper “API hooking revealed” written by Ivo Ivanov and madshi’s discussion about hooking methods. There are some libraries or tools that wrap the hooking methods. "Detours" is a library developed by a Microsoft’s research term. It uses code overwriting technology. “madCodeHook” is a library built by madshi. It’s also based on code overwriting. “hookapi” is another tool can help you develop your hooking applications. Since Detours is free for research purpose and it’s stable and effective, I will use it to hook direct3d.
Detours
“Detours” is originally built for change a standalone system to distributed system. It can inject your code into another win32 function. Some utilities that attach your codes to another process are also provided in this library. The typical case you may need Detours is that you need to modify an application’s behavior without knowing its source code. For more details please visit Detours’ web site: Detours - Microsoft Research
*

As an example of how to use Detours, we will try to catch the function “SetTimer()”. Then change the elapse time to a big number. So for the applications use SetTimer(), you can steal time by hooking this function. The first step is to create TRAMPOLINE. It changes SetTimer()’s binary code and lets it jump to the Real_SetTimer() function.
*

DETOUR_TRAMPOLINE(
UINT WINAPI Real_SetTimer(HWND hWnd, ​// handle of window for timer messages
​ * * * *UINT nIDEvent, * * * * // timer identifier
​ * * * *UINT uElapse, * * * * * // time-out value
​ * * * *TIMERPROC lpTimerFunc), // address of timer procedure
SetTimer);

*

Then we write our own SetTimer function.

UINT WINAPI Mine_SetTimer(
​HWND hWnd, * * * * * // handle of window for timer messages
​UINT nIDEvent, * * * *// timer identifier
​UINT uElapse, * * * * * // time-out value
​TIMERPROC lpTimerFunc * // address of timer procedure
​)
{
​uElapse=10* uElapse;
​return Real_SetTimer(hWnd,nIDEvent,uElapse,lpTimerFunc);
}

*

Then write the functions to intercept the Real_SetTimer() function to your own Mine_SetTimer() function and the function that removes the interception.
*

BOOL TrampolineWith(VOID)
{
* *DetourFunctionWithTrampoline((PBYTE)Real_SetTimer,
* * * * * * * * * * * * * * * *​(PBYTE)Mine_SetTimer);
* *return TURE;
}
*
BOOL TrampolineRemove(VOID)
{
* *DetourRemove((PBYTE)Real_SetTimer,
* * *​ (PBYTE)Mine_SetTimer);
* *return TRUE;
}

*

At last, we write the DLLmain function.
*

BOOL APIENTRY DllMain(HINSTANCE hModule, DWORD dwReason, PVOID lpReserved)
{
​switch (dwReason) {
​ *case DLL_PROCESS_ATTACH:
​return TrampolineWith();
​ *case DLL_PROCESS_DETACH:
​return TrampolineRemove();
​ *case DLL_THREAD_ATTACH:
​return TrampolineWith();
​ *case DLL_THREAD_DETACH:
​return TrampolineRemove();
​}
​return TRUE;
}

*

By including the “detours.h” file and linking the “detours.lib”, you can build this DLL (let’s say it is myTest.dll). Then you can write your own application to attach the DLL into another application, or use the “withdll.exe”, which is included in Detours 1.5’s samples, to attach it. If you place the DLL file, withdll.exe and winmine.exe in the same directory, then play the Mine game use command “withdll.exe –d:myTest.dll winmine.exe”, you will get a very high score.
*

This is the simplest case of using Detours. If you want to hook a member function of a class, you need to write your own class and your member function. You can find an example of hooking member function in Detours 1.5’s samples.
*
To hook direct3d, you may create a class, write your own member functions, for example CreateDevice(), Present(), and then detour direct3d’s member function to your own functions. Unfortunately this doesn’t work. The reason is that Directx is COM based, it uses virtual functions. When you detour the member functions, you will use the virtual function’s address not the real implement function’s address. So the key of hooking direct3d is finding out the real function address. To do this we need to know how virtual function works.
Understanding Virtual Functions and Hooking Direct3d
Virtual function is used for reusable and polymorphic purposes. It is runtime bound to the real implement function. For more details about how virtual functions work, please read Shivesh V.’s paper “Virtual Functions and their implementation in C”.
*
Basically, after you call the Direct3DCreate8() function (let’s assume we use directx8), you will get a pointer to the created IDirect3D8 object. In the pointer is the address of the vtable, a structure that contains all the member functions’ addresses. The address in the vtable is the real implement function’s address. So if we want to build a benchmark application to test the frame update rate, we could do it in the following steps:
1) Catch Direct3DCreate8() to get the IDirect3D8 object
2) Use the pointer of IDirect3D8 object, catch the CreateDevice() function to get the IDirect3DDevice8 object.
3) Use the pointer of IDirect3DDevice8 object, catch Present() function.
4) In the Present() function calculates the frame rate.
*

However, to catch the member function, there is a very important parameter we need to know. It’s the address data in the vtable that contain the to-be-hooked function’s address. There is no general way to know the offset in the vtable. For direct3d, you could use the d3d8.h file to figure out the offset. Or you can debug a direct3d application and get the offset by read the disassembly code. The following is a piece code of the directx 8.1 SDK’s Text3D sample code. The comments of the disassembly code are added by the author.
*

873: * * *// Create the device
874: * * *hr = m_pD3D->CreateDevice( m_dwAdapter, pDeviceInfo->DeviceType,
875: * * * * * * * * * * * * * * * * m_hWndFocus, pModeInfo->dwBehavior, &m_d3dpp,
876: * * * * * * * * * * * * * * * * &m_pd3dDevice );
00403114 * mov * * * * eax,dword ptr [this]
00403117 * add * * * * eax,2A4A8h
0040311C * push * * * *eax​// push the sixth parameter &m_pd3dDevice
0040311D * mov * * * * ecx,dword ptr [this]
00403120 * add * * * * ecx,2A464h
00403126 * push * * * *ecx​//push the fifth parameter &m_d3dp
00403127 * mov * * * * edx,dword ptr [pModeInfo]
0040312A * mov * * * * eax,dword ptr [edx+0Ch]
0040312D * push * * * *eax​//push the fourth parameter pModeInfo->dwBehavior
0040312E * mov * * * * ecx,dword ptr [this]
00403131 * mov * * * * edx,dword ptr [ecx+2A49Ch]
00403137 * push * * * *edx​//push the third parameter m_hWndFocus
00403138 * mov * * * * eax,dword ptr [pDeviceInfo]
0040313B * mov * * * * ecx,dword ptr [eax]
0040313D * push * * * *ecx​//push the second parameter pDeviceInfo->DeviceType
0040313E * mov * * * * edx,dword ptr [this]
00403141 * mov * * * * eax,dword ptr [edx+2A448h]
00403147 * push * * * *eax​//push the first parameter m_dwAdapter
00403148 * mov * * * * ecx,dword ptr [this]
0040314B * mov * * * * edx,dword ptr [ecx+2A4A4h]​//calculate the return address
00403151 * mov * * * * eax,dword ptr [this]
00403154 * mov * * * * ecx,dword ptr [eax+2A4A4h]
0040315A * mov * * * * eax,dword ptr [ecx]​//calculate the vtable address. It’s the value
​//stored in ecx register.
0040315C * push * * * *edx​//push the return address
0040315D * call * * * *dword ptr [eax+3Ch]​//call the CreateDevice() function
00403160 * mov * * * * dword ptr [hr],eax

*
From the assembler code, it’s obvious that the offset is 0x3ch. You can do the same thing for the Present() function. And the offset happen is also 0x3ch.
*
The sample code can be found at https://usl.sis.pitt.edu/wjj/U***ient/direct3d8.zip. To compile it you need Directx 8.1 SDK. Please note that the method of display frame count is not every effective. If you remove the displaying code, you will find Detours has almost no effect to the frame rate.
Reference:
[1] “API hooking revealed”, Ivo Ivanov
[2] “madCodeHook”, madshi
[3] "Detours", Galen Hunt and Doug Brubacher
[4] “Virtual Functions and their implementation in C”, Shivesh V.
[5] “Pointers to C++ Member Functions”, Michael D. Crawford

original source here:
https://usl.sis.pitt.edu/wjj/U***ient...20Direct3d.doc

[why06]

Looks nice, but can you clean it up a bit with some codetags? Lol. though that's probably pretty tricky on ur iPod. If you can't do it PM me a link to the main article and I'll fix it myself.

[Hell_Demon]

Ok I have found a very good reference on hooking techniques for direct3d
credits to a guy named Wang Jijun
EDIT: sorry for the long page but my iPod cant upload and I didn't wanna link

Hooking Direct3d
By Jijun Wang
Introduction:
In the field of hooking technologies, hooking COM based applications is a big challenging because you can’t know their member functions’ real addresses previously. There are many papers on the Internet that talk about how to hook (somebody call it hijack) an application. However, seldom of them mention how to hook COM. Since I have seen some people asked questions about this topic and nobody answered their questions clearly, I decide to write this paper to share my experience with the guys who are interested in hooking COM.
*
In this paper, I use direct3d, which is widely used in current games, as an example of COM. At first I will briefly introduce the hooking technologies and explain why I select Detours to hook direct3d. Then more details and examples of Detours will be presented and I will discus virtual functions and find the way to hook direct3d.
Hooking Technologies
The basic idea of hooking is injecting your code into a piece of code. When the target code executes your code will be invoked. To do this, you need to at first attach your code into the target process. And then inject your code into the target process’ code.
The ways to attach your code into another thread or process include
1) Register your DLL to the registry table.
This method registers your DLL to the key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows\AppInit_DLLs and the dll will be loaded when user32.dll initializes itself. It is a safe method. But it only works for the applications that use user32.dll. And it’s not convenient to activate/deactivate your DLL.
2) System-wide Windows Hooks
This method use SetWindowsHookEx(),CallNextHookEx() and UnhookWindowsHookEx() winapis to monitor the system’s message traffic and attach your code into the processes that fit the hook filter. Since it is system-wide message monitor, the system’s performance will be significantly affected.
3) Create Remote Thread
This method use CreateRemoteThread()to inject your DLL into another thread (the to-be-hooked thread). It’s a very effective method. But there is one thing we need to consider. You must make sure you have enough privileges to access the to-be-hooked thread.
*

To inject your code into another process’ code, we could use:
1) Proxy DLL
This method writes a new DLL to replace the to-be-hooked DLL. The new DLL must have the same name, exported functions and variables as the old one. For example, the GLtrace is a replacement of the opengl32.dll. It is used to trace OpenGL. Although you could use function forward to reduce the time spent in rewriting functions, it’s a tedious work in some cases. *
2) Altering of the IAT (Import Address Table)
This is a wildly used hooking method. Windows use IAT to find the functions’ addresses. Through changing the data of IAT, you could use your own function to replace the to-be-hooked function. The method is robust and easy to implement. However, it only works for statically linked API calls. And sometimes you may miss API calls since you must patch every module in the target application.
3) Code Overwriting
The basic idea is to overwrite the API’s binary code in memory. We could save and replace the first several codes of a function call to a JUMP code to jump to your codes and then go back to the original position and restore the saved codes. It’s very hard to implement. However, once you have mastered it, it’s really a good hooking method.
*

For more details about hooking technologies you can read the wonderful paper “API hooking revealed” written by Ivo Ivanov and madshi’s discussion about hooking methods. There are some libraries or tools that wrap the hooking methods. "Detours" is a library developed by a Microsoft’s research term. It uses code overwriting technology. “madCodeHook” is a library built by madshi. It’s also based on code overwriting. “hookapi” is another tool can help you develop your hooking applications. Since Detours is free for research purpose and it’s stable and effective, I will use it to hook direct3d.
Detours
“Detours” is originally built for change a standalone system to distributed system. It can inject your code into another win32 function. Some utilities that attach your codes to another process are also provided in this library. The typical case you may need Detours is that you need to modify an application’s behavior without knowing its source code. For more details please visit Detours’ web site: Detours - Microsoft Research
*

As an example of how to use Detours, we will try to catch the function “SetTimer()”. Then change the elapse time to a big number. So for the applications use SetTimer(), you can steal time by hooking this function. The first step is to create TRAMPOLINE. It changes SetTimer()’s binary code and lets it jump to the Real_SetTimer() function.
*

DETOUR_TRAMPOLINE(
UINT WINAPI Real_SetTimer(HWND hWnd, ​// handle of window for timer messages
​ * * * *UINT nIDEvent, * * * * // timer identifier
​ * * * *UINT uElapse, * * * * * // time-out value
​ * * * *TIMERPROC lpTimerFunc), // address of timer procedure
SetTimer);

*

Then we write our own SetTimer function.

UINT WINAPI Mine_SetTimer(
​HWND hWnd, * * * * * // handle of window for timer messages
​UINT nIDEvent, * * * *// timer identifier
​UINT uElapse, * * * * * // time-out value
​TIMERPROC lpTimerFunc * // address of timer procedure
​)
{
​uElapse=10* uElapse;
​return Real_SetTimer(hWnd,nIDEvent,uElapse,lpTimerFunc);
}

*

Then write the functions to intercept the Real_SetTimer() function to your own Mine_SetTimer() function and the function that removes the interception.
*

BOOL TrampolineWith(VOID)
{
* *DetourFunctionWithTrampoline((PBYTE)Real_SetTimer,
* * * * * * * * * * * * * * * *​(PBYTE)Mine_SetTimer);
* *return TURE;
}
*
BOOL TrampolineRemove(VOID)
{
* *DetourRemove((PBYTE)Real_SetTimer,
* * *​ (PBYTE)Mine_SetTimer);
* *return TRUE;
}

*

At last, we write the DLLmain function.
*

BOOL APIENTRY DllMain(HINSTANCE hModule, DWORD dwReason, PVOID lpReserved)
{
​switch (dwReason) {
​ *case DLL_PROCESS_ATTACH:
​return TrampolineWith();
​ *case DLL_PROCESS_DETACH:
​return TrampolineRemove();
​ *case DLL_THREAD_ATTACH:
​return TrampolineWith();
​ *case DLL_THREAD_DETACH:
​return TrampolineRemove();
​}
​return TRUE;
}

*

By including the “detours.h” file and linking the “detours.lib”, you can build this DLL (let’s say it is myTest.dll). Then you can write your own application to attach the DLL into another application, or use the “withdll.exe”, which is included in Detours 1.5’s samples, to attach it. If you place the DLL file, withdll.exe and winmine.exe in the same directory, then play the Mine game use command “withdll.exe –d:myTest.dll winmine.exe”, you will get a very high score.
*

This is the simplest case of using Detours. If you want to hook a member function of a class, you need to write your own class and your member function. You can find an example of hooking member function in Detours 1.5’s samples.
*
To hook direct3d, you may create a class, write your own member functions, for example CreateDevice(), Present(), and then detour direct3d’s member function to your own functions. Unfortunately this doesn’t work. The reason is that Directx is COM based, it uses virtual functions. When you detour the member functions, you will use the virtual function’s address not the real implement function’s address. So the key of hooking direct3d is finding out the real function address. To do this we need to know how virtual function works.
Understanding Virtual Functions and Hooking Direct3d
Virtual function is used for reusable and polymorphic purposes. It is runtime bound to the real implement function. For more details about how virtual functions work, please read Shivesh V.’s paper “Virtual Functions and their implementation in C”.
*
Basically, after you call the Direct3DCreate8() function (let’s assume we use directx8), you will get a pointer to the created IDirect3D8 object. In the pointer is the address of the vtable, a structure that contains all the member functions’ addresses. The address in the vtable is the real implement function’s address. So if we want to build a benchmark application to test the frame update rate, we could do it in the following steps:
1) Catch Direct3DCreate8() to get the IDirect3D8 object
2) Use the pointer of IDirect3D8 object, catch the CreateDevice() function to get the IDirect3DDevice8 object.
3) Use the pointer of IDirect3DDevice8 object, catch Present() function.
4) In the Present() function calculates the frame rate.
*

However, to catch the member function, there is a very important parameter we need to know. It’s the address data in the vtable that contain the to-be-hooked function’s address. There is no general way to know the offset in the vtable. For direct3d, you could use the d3d8.h file to figure out the offset. Or you can debug a direct3d application and get the offset by read the disassembly code. The following is a piece code of the directx 8.1 SDK’s Text3D sample code. The comments of the disassembly code are added by the author.
*

873: * * *// Create the device
874: * * *hr = m_pD3D->CreateDevice( m_dwAdapter, pDeviceInfo->DeviceType,
875: * * * * * * * * * * * * * * * * m_hWndFocus, pModeInfo->dwBehavior, &m_d3dpp,
876: * * * * * * * * * * * * * * * * &m_pd3dDevice );
00403114 * mov * * * * eax,dword ptr [this]
00403117 * add * * * * eax,2A4A8h
0040311C * push * * * *eax​// push the sixth parameter &m_pd3dDevice
0040311D * mov * * * * ecx,dword ptr [this]
00403120 * add * * * * ecx,2A464h
00403126 * push * * * *ecx​//push the fifth parameter &m_d3dp
00403127 * mov * * * * edx,dword ptr [pModeInfo]
0040312A * mov * * * * eax,dword ptr [edx+0Ch]
0040312D * push * * * *eax​//push the fourth parameter pModeInfo->dwBehavior
0040312E * mov * * * * ecx,dword ptr [this]
00403131 * mov * * * * edx,dword ptr [ecx+2A49Ch]
00403137 * push * * * *edx​//push the third parameter m_hWndFocus
00403138 * mov * * * * eax,dword ptr [pDeviceInfo]
0040313B * mov * * * * ecx,dword ptr [eax]
0040313D * push * * * *ecx​//push the second parameter pDeviceInfo->DeviceType
0040313E * mov * * * * edx,dword ptr [this]
00403141 * mov * * * * eax,dword ptr [edx+2A448h]
00403147 * push * * * *eax​//push the first parameter m_dwAdapter
00403148 * mov * * * * ecx,dword ptr [this]
0040314B * mov * * * * edx,dword ptr [ecx+2A4A4h]​//calculate the return address
00403151 * mov * * * * eax,dword ptr [this]
00403154 * mov * * * * ecx,dword ptr [eax+2A4A4h]
0040315A * mov * * * * eax,dword ptr [ecx]​//calculate the vtable address. It’s the value
​//stored in ecx register.
0040315C * push * * * *edx​//push the return address
0040315D * call * * * *dword ptr [eax+3Ch]​//call the CreateDevice() function
00403160 * mov * * * * dword ptr [hr],eax

*
From the assembler code, it’s obvious that the offset is 0x3ch. You can do the same thing for the Present() function. And the offset happen is also 0x3ch.
*
The sample code can be found at https://usl.sis.pitt.edu/wjj/U***ient/direct3d8.zip. To compile it you need Directx 8.1 SDK. Please note that the method of display frame count is not every effective. If you remove the displaying code, you will find Detours has almost no effect to the frame rate.
Reference:
[1] “API hooking revealed”, Ivo Ivanov
[2] “madCodeHook”, madshi
[3] "Detours", Galen Hunt and Doug Brubacher
[4] “Virtual Functions and their implementation in C”, Shivesh V.
[5] “Pointers to C++ Member Functions”, Michael D. Crawford

original source here:
https://usl.sis.pitt.edu/wjj/U***ient...20Direct3d.doc

[Mr.Magicman]

)nice good tut i might use this for my latest project

[falzarex]

lol thanks HD
was too lazy to fix it

[why06]

lol thanks HD
was too lazy to fix it

Copy, pasted, done.

[crushed]

Copy, pasted, done.

You mean just like the rest of the so called C++ community on MPGH? lol.
We have our talented ones, as well as our tainted ones. As in tainted by idiocy.

GJ to Fal for finding this wonderful knowledge. LUL, although I won't understand this, I know a people will find this useful.

[Void]

You mean just like the rest of the so called C++ community on MPGH? lol.
We have our talented ones, as well as our tainted ones. As in tainted by idiocy.

GJ to Fal for finding this wonderful knowledge. LUL, although I won't understand this, I know a people will find this useful.

I'm tainted.

[falzarex]

lol we're all tainted

[crushed]

Oh dear, the community that is talented is turning especially emo. I just hope away your parents hide the razors from you.

[falzarex]

My Swiss knife is sitting on my table as we speak
*reaches out for knife*