x64 bit debugging

 

x64 debugging, as daily task in today’s dev. life

 

There are some subtle differences between using the Debugging Tools for Windows (DTW) toolset on x86 and x64 that are worth mentioning, especially if you are new to doing x64 debugging. Most of this post applies to all of the debuggers shipped in the DTW package, which is why I avoid talking about WinDbg or ntsd or cdb specifically, and often just refer to the “DTW debuggers”. This is the first post in a multipart series, and it provides a general overview of the options you have for doing 32-bit and 64-bit debugger on an x64 machine, and how to setup the debugger properly to support both, using either the 32-bit or 64-bit packages.

There are many ways to do x64 debugging, which can get confusing, simply because there are so many different choices. You can use both the 32-bit and 64-bit DTW packages, with some restrictions. Here’s a summary of the most common cases (including “cross-debugging” scenarios, where you are using the 32-bit debugger to debug 64-bit processes). For now, I’ll just limit this to user mode, although you can use many of these options for kernel debugging too.

  • Natively debugging 64-bit processes on the same computer using the 64-bit DTW package
  • Natively debugging 32-bit (Wow64) processes on the same computer using the 64-bit DTW package
  • Debugging 32-bit (Wow64) processes on the same computer using the 32-bit DTW package (running the debugger itself under Wow64)
  • Debugging 64-bit processes or 32-bit (Wow64) processes on the same or a different computer using either the 64-bit or 32-bit DTW package, with the remote debugging support (e.g. dbgsrv.exe, or -remote/-server). This requires a 64-bit remote debugger server.
  • Debugging 32-bit (Wow64) processes on the same or a different computer using either the 64-bit or 32-bit DTW package, with the remote debugger support (e.g. dbgsrv.exe, or -remote/-server). This works with a 32-bit remote debugging server.
  • Debugging a 64-bit or 32-bit dump file using the 32-bit or 64-bit DTW package. Both DTW packages are capable of doing this task natively.

There are actually even more combinations, but to keep it simple, I just listed the major ones. Now, as for which setup you want to use, there are a couple of considerations to keep in mind. Most of the important differences for the actual debugging experience stem from whether the process that is making the actual Win32 debugger API calls is a 64-bit or 32-bit process. For the purposes of this discussion, I’ll call the process that makes the actual debugger API calls (e.g. DebugActiveProcess) the actual debugger process.

If the actual debugger process is a 32-bit process under Wow64, then it will be unable to interact meaningfully with 64-bit processes (if you are using WinDbg, 64-bit processes will all show as “System” in the process list). For 32-bit processes, it will see them exactly as you would under an x86 Win32 system; there is no direct indication that they are running under Wow64, and the extra Wow64 functionality is completely isolated from the debugger (and the person driving the debugger). This can be handy, as the extra Wow64 infrastructure can in many cases just get in the way if you are debugging a pure 32-bit program running under Wow64 (unless you suspect a bug in Wow64 itself, which is fairly unlikely to be the case).

If the actual debugger process is a native 64-bit process, then the whole debugging environment changes. The native 64-bit debugging environment allows you to debug both 32-bit (Wow64) and 64-bit targets. However, when you are debugging 32-bit targets, the experience is not the same as if you were just debugging a 32-bit program on a 32-bit Windows installation. The 64-bit debugger will see all of the complexities of Wow64, which often gets confusing and can get in your way. I’ll go into specifics of what exactly is different and how the 64-bit debugger can sometimes be annoying when working with Wow64 processes in a moment; for now, stick with me.

So, if you need to do development on 64-bit computers, which debugging package is the best for you to use? Well, that really depends on what you are doing, but I would recommend installing both the 32-bit and 64-bit DTW packages. The main reason to do this is that it will allow you to debug 32-bit processes without having to deal with the Wow64 layer all the time, but it at the same time it will allow you to debug native 64-bit processes.

After you have installed the DTW packages, one of the familiar first steps with setting up the debugger tools on a new system is to register WinDbg as your default post-portem debugger. This turns out to be a bit more complicated on 64-bit systems than on 32-bit systems, however, in large part due to a new concept added to Windows to support Wow64: registry reflection. Registry reflection allows for 32-bit and 64-bit applications to have their own virtualized view of several key sections of the registry, such as HKEY_LOCAL_MACHINE\Software. What this means in practice is that if you write to the registry from a 32-bit process, you might not see the changes from 64-bit processes (and vice versa), depending on which registry keys are changed. Alternatively, you might see different changes than you made, such as if you are registering a COM interface in HKEY_CLASSES_ROOT.

So, what does all of this mean to you, as it relates to doing debugging on 64-bit systems? Well, the main difference that impacts you is that there are different JIT handlers for 32-bit and 64-bit processes. This means that if you register a 32-bit DTW debugger as a default postmortem debugger, it won’t be activated for 64-bit processes. Conversely, if you register a 64-bit DTW debugger as a default postmortem debugger, it won’t be activated for 32-bit processes.

This leaves you with a couple of options: Register both the 32-bit and 64-bit DTW packages as default postmortem debuggers (if you only want to use the 64-bit DTW package on 64-bit processes and not 32-bit (Wow64) processes as a JIT debugger), or register the 64-bit DTW debugger as a default postmortem debugger for both 32-bit and 64-bit processes. If you want to do the former, then what you need to do is as simple as logging in as an administrator and running both the 32-bit and 64-bit DTW debuggers with the -I command line option (install as default postmortem debugger), and then you’re set. However, if you want to use the 64-bit debugger for both 64-bit and 32-bit processes as a JIT debugger, then things are a bit more complicated. The best way to set this up is to install the 64-bit DTW debugger as a default postmortem debugger (run it with -I), and then open the 64-bit version of regedit.exe, navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug, and copy the value of the “Debugger” entry into the clipboard. Then, navigate to the 32-bit view of this key, located at HKEY_LOCAL_MACHINE\Software\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug, create (or modify, if it already exists) the “Auto” string value and set it to “1″, then create (or modify, if it already exists) the “Debugger” string value and set it to the value you copied from the 64-bit view of the AeDebug key. For my system, the “Debugger” value is set to something like “C:\Program Files\Debugging Tools for Windows 64-bit\WinDbg.exe” -p %ld -e %ld -g. If you don’t see a Wow6432Node registry key under HKEY_LOCAL_MACHINE\Software, then you are probably accidentally running the 32-bit version of regedit.exe and not the 64-bit version of regedit.exe.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s