Driver Debugging with WinDbg and VMWare
One of the first useful things you will want to do when in the bowels of ring 0 is attack the thing from a debugger point of view. In my case I like using Windows Debugger [windbg] (hey its free, fully functional and does remote debugging really well), and found it neccessary to find a way to have it work with VMWare.
The trick to get it to work in VMWare is to get the host OS to believe it is able to connect to a serial port. Through VMWare, to accomplish this, you need to create a named pipe. The following steps will guide you to configuring VMWare (in my case v3.2):
Open up the Configuration Editor (Settings->Configuration Editor)
Click Add to run the Hardware Wizard
Select Serial Port, and then click the Next button
Select the Use named pipe radio button
Use the default pipe name. It SHOULD be \\.\pipe\com_1. If it is not, change it to that.
Select This end is the Server.
Select The other end is an application.
Click the Advanced Button.
Select the Yield CPU on poll checkbox
This is an important step as the Kernel in the target virtual machine uses the port in polled mode, not interrupt mode.
Click the Finish button, and then click Ok to close the Configuration Editor.
Once you have configured your VMWare session, you need to power it on. You will be able to confirm that the new virtual serial port is added by clicking on the Devices->serial0 menu item. You should note it is saying "Connecting". This means its now ready for a remote connection.
Once the actual virtual machine is configured, you need to configure the target operating system installed in the virtual machine to support remote debugging. This is accomplished by editing the boot.ini found in the root of the C:\ drive. To do this, you need to add a /debugport=some_com_port /baudrate=some_baud_rate to the end of an [operating systems] line. I would not recommend doing it to the one that is there. It would be better to copy the line and paste it with the ammendments, and then use the OS's menu selection to determine which mode you would like to boot into. This is what my boot.ini looks like for XP Pro as the operating system in the virtual machine:
[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - Debug" /fastdetect /debugport=com1 /baudrate=115200
You will notice that the second option sets the debugport to com1, and sets the baudrate to 115200. I am told you can tweak this out to get even more speed out. But it seems fine for me at this speed, so I haven't mucked with it at all. If you do try this, drop me an email and let me know how it works out.
If you set up your boot.ini fine, save it and reboot. You should be prompted with something that looks like this:
At this point you have completed setting up the VMWare side of things. Now you need to set up the host to connect to it. This is actually rather easy. You just need to tell windbg at the command like to connect to the pipe, like this:
windbg -k com:port=\\.\pipe\com_1,pipe
If you are using WinDbg 6.x or newer, a better alternative is to use:
windbg -b -k com:pipe,port=\\.\pipe\com_1,resets=0
Thanks needs to go to Randhir Dugal for pointing out the new format for the latest Windbg version.
I am a rather anal type guy when it comes to security, so I actually made a shortcut on my desktop to WinDbg and added these command line arguments to the Target line. Originally this was so I could run WinDbg with differnet credentials as I do not run with administrator privileges on a day to day basis. I found that with XP's normal security settings for com ports, you can still work in a least privileged environment while doing the development WITHOUT having to use 'runas'. (You are logged into W2K/XP as a least priviledged environment aren't you? If not, you really should read my article on how, and why this is important to do so.)
At this point fire up the debugger. With any luck you should see something that looks like this:
If you are using a WinDbg version earlier than 6.x, you will find one issue with this approach. If you cannot seem to connect right away, close Windbg and restart it... it will then work. Seems flaky to me. But it works. And thats a Good Thing™. With the latest versions of WIndbg (6.x and newer) a new resets flag prevents this sort of hanging.
That is all there is to it. At this point, you can now go nuts with remote debugging. To make sure it works for ya just add some DbgPrint() messages to your ring 0 driver and watch them echo to the screen. Past that, I will leave it to your imagination how to use the debugger :)
Good luck!
Kernel debugging Windows XP inside of a virtual machine
Something that is incredibly easy to do with virtual machines is kernel debugging. You can do this by either connecting two virtual machines to the same named pipe (creating a virtual null-modem cable as it were), or by debugging a virtual machine directly over a named pipe from the host operating system.
If you are using two virtual machines you will need to configure both to have COM1 connected to a shared named pipe (e.g. \\.\pipe\debugPipe). If you are going to debug a virtual machine from your host computer you will just need to configure the one virtual machine to have its COM1 connected to the named pipe.
You will then need to start the virtual machine you want to debug and do the following:
Go to My Computer --> Properties --> Advanced --> Settings under Startup and Recovery --> Edit
You should see a line that is something like 'multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect /NoExecute=OptIn'
You will need to add '/DEBUG /DEBUGPORT=COM1' to the end of this line
Once you have done all of this you should shut down the virtual machine
Now you will need to download the debugging tools from: http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx and either install them on your host computer, or in a second virtual machine (depending on where you want to debug from). Once you have done this you will need to do the following:
For host based debugging run "kd -y SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols -k com:pipe,port=\\.\pipe\debugPipe,resets=0,reconnect" and then launch your target virtual machine.
For virtual machine based debugging run "kd -y SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols -k com:port=com1" and then launch your target virtual machine.
Once the target virtual machine starts to boot - the debugger should connect and you will be good to go.