Am currently trying to inline a VMProtect target and from what I could see, VMProtect uses two tricks. One is a hash (or DWORD value) of the main code sections of the protected application (along with its own code), and two, the ASProtect trick - CreateFileMapping + MapViewOfFile. Detailing what I did with observations, maybe it will help get this to its end :-)
TEST 1:
a. get application, make a copy, hex edit copy in such a way that I changed last byte from 00 to 01; saved, launched, error, corrupted
Wanted to see if changing the byte in real-time does the trick. So:
b. opened application in Olly, found last byte, changed from 01 to 00; ran app, error, corrupted
Therefore, it must be making an image of the file on the disk. Known methods - ReadFile, CreateFileMapping + MapViewOfFile. Since VMProtect is tricky, decided to break on end of each API. Ran app, result:
->> 0012F6BC 007FF83A RETURN to pclaw.007FF83A from pclaw.007FD391 (yeah, guess you know the target now, hehe)
7C80B9C1 C2 1400 RETN 14 // MapViewOfFile
->> 0012F6C0 007FF83A RETURN to pclaw.007FF83A from pclaw.007FD391
As you can see, returns to same "wrapper" ;-)
So, pushed this further a little bit. Time to find out the parameters for those two APIs, since the custard doesn't let me change 01 to 00 in the mapped copy T_T..
That's why :-) Same trick as with ASProtect -> 4th parameter is FILE_MAP_READ. Let's trick the custard with 01 (FILE_MAP_COPY). Mapping done, EAX = 1010000. Scrolled to end of map, found my byte, changed to 00.