WinDbg for .NET developers: "1. The Beginning"

WinDbg is indeed a powerful debugger which is widely known among low-level system developers and is unfairly neglected by more high-level application developers. Although, a well-known Visual Studio Debugger provides both easy and obvious ways to debug your software in the development stage, WinDbg gives much more powerful tools to debug your software in the user environment, when all what you have is just a memory dump of a died program... or some strange things happen in your test lab including your PC. There is a bunch of good books and some blog posts which contain an excessive information on WinDbg usage over the Internet. Unfortunately, a lot of them have a very steep curve (like vi) and due to I could not find any docs which would have made it possible to quickly start to use WinDbg, it was decided to gather my gained experience by means of short practical stories. First part "1. The beginning" is dedicated to some very basic things which you really should do and uncover the most frequent scenario of WinDbg usage: post-mortem debugging.

Disclaimer

Like everything in this blog, this post can be absolutely cruel and contain terrible things. Some of them can be made in absolutely incorrect and irrational way and even destroy your brain. If you have another opinion or know more logical way to do something relevant to the current topic, I will be very glad if you express your thoughts in comments or pull-requests.

Installation

Despite the fact that some websites offer small-sized repacks of WinDbg, I’d recommend not to safe your traffic and download WinDbg as a part of WinSDK from the official Microsoft Dev Center. For minimal WEB installation only ~150Mb is required and a few minutes of your time but this action would prevent a lot of troubles connected with possibly incorrect unofficial repacks.

/posts/2018/windbg_course/winsdk.png

Also I’d advise to install the following WinDbg extensions which would make your .NET debugging much easier:

  • sosex (SOS extended) is an awesome extension which speeds up a lot of frequent scenarios and provides much better output for some commands.
  • psscor4 provides a super-set of SOS functionality and is more tolerant to some issues in collected memory dumps.
  • soswow64 gives ability to open x86 dumps memory collected by x64 versions of Task Manager or ProcDump. I do hope that it will not be necessary as your dumps will be always collected by a right tool but in cases where it has already happened this extension is the only way to investigate such memory dumps.

You can download these extensions and copy in any directory and load them by commands such as .load C:\windbg\exts\x86\sosex.dll. But I’d recommend to copy them to:

C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\winext // for x86
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext // for x64

These actions allow you to use much shorter commands as .load sosex.

Omg... where are my glasses?!

/posts/2018/windbg_course/windbg-init.png

WinDbg is a tool with quite a long story with no changes in its UI. I have found a screenshot dating as far back as 2001 where WinDbg looks absolutely the same as nowadays after the first run. Default font is absolutely non-readable in the most modern high DPI screens (such as, for example, my FullHD 13 inch laptop) and a high contrast white-black theme can cause some eye pain which get used to modern dark themes such as Darcula. You can change font and font size in the menu View - Font.. and colors in the menu View - Options.... Unfortunately colors settings are quite challenging and contain more than 60 colors, so I recommend to limit your configuration tuning only by using font settings or take some predefined configuration, e.g. my Solarized Themes for WinDbg (it requires installed Ubuntu Mono Font). It changes UI of WinDbg much close to the modern text editors and IDEs.

/posts/2018/windbg_course/windbg-theme.png

The First Investigation

For the following actions you can use any crash dump for .NET application. If you do not have it or you want to make your life a bit easier, you can take this demo application prepared for this course. For memory dump collection I prefer to use ProcDump from Window Sysinternals. There is an important moment here, as a lot of things in CLR are based on the usage of heap memory, a “Mini” memory dump which is collected by almost all programs by default will not be enough in most cases. A lot of commands will simply throw exceptions and do nothing. So, you should be careful while collecting memory dumps and specify that you need a "Full" memory dump. If you decided to use procdump and demo program, then you should run demo application, open command prompt from procdump directory, enter procdump.exe -ma -e WinDbgCourse.exe and click the button Basic Null Reference Crash in demo application. After a few seconds you take a collected memory dump with a filename such as WinDbgCourse.exe_180527_142004. Then click File - Open Crash Dump... (or simply press Ctrl+D) and you load your memory dump in WinDbg (see image above). Because it is crash-dump WinDbg shows message such as:

(2450.b88): Access violation - code c0000005 (first/second chance not available)

Unfortunately, it is an unmanaged exception which does not show anything for .NET developer. So, it is the right time to use SOS extension (sometimes some commands do not work from the first attempt, so you should repeat them):

0:000> .loadby sos clr

0:000> !PrintException
c0000005 Exception in C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos.PrintException debugger extension.
   PC: 0f99b8f3  VA: 00000000  R/W: 0  Parameter: 00000000

0:000> !PrintException -lines

Exception object: 027c934c
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
*** WARNING: Unable to verify checksum for WinDbgCourse.exe
   SP       IP       Function
   001CEA98 02523133 WinDbgCourse!WinDbgCourse.BasicNullRefCrashCommand.Execute(System.Object)+0x73 [C:\Users\wwwzl\Dropbox\l-projects\windbg-course\BasicNullRefCrashCommand.cs @ 15]
   001CEAD0 56C6D2F1 PresentationFramework_ni!MS.Internal.Commands.CommandHelpers.CriticalExecuteCommandSource(System.Windows.Input.ICommandSource, Boolean)+0xb1
   001CEAF4 56D290C8 PresentationFramework_ni!System.Windows.Controls.Primitives.ButtonBase.OnClick()+0x54
   001CEB04 56BED59D PresentationFramework_ni!System.Windows.Controls.Button.OnClick()+0x55
...

An output already contains a link to your source code, we can open and read a source file and understand that problem in null-valued argument parameter. To confirm this, we can look at CLR stack:

0:000> !clrstack -a
OS Thread Id: 0xb88 (0)
Child SP       IP Call Site
001cea98 02523133 WinDbgCourse.BasicNullRefCrashCommand.Execute(System.Object) [C:\Users\wwwzl\Dropbox\l-projects\windbg-course\BasicNullRefCrashCommand.cs @ 15]
   PARAMETERS:
      this (0x001ceabc) = 0x0275cc48
      parameter (0x001ceab4) = 0x00000000
...

Bingo! We have found a root cause so we can patch our code and send a patched version to a client.

Deep Into

Existence of this part shows that something has gone wrong. A patched version is still crashing. And it is not so good news for us because we know that clients do not like updates which do not patch anything. So, try to be more careful in the future. Firstly, let’s find IP (instruction pointer) from the upper line in stack trace (02523133) and try to understand some assembler code:

0:000> !U 02523133
Normal JIT generated code
WinDbgCourse.BasicNullRefCrashCommand.Execute(System.Object)
Begin 025230c0, size a4
...

C:\Users\wwwzl\Dropbox\l-projects\windbg-course\BasicNullRefCrashCommand.cs @ 15:
025230ed e846e9d953      call    PresentationFramework_ni+0x231a38 (562c1a38) (System.Windows.Application.get_Current(), mdToken: 0600028a)
025230f2 8945e8          mov     dword ptr [ebp-18h],eax
025230f5 8b4de8          mov     ecx,dword ptr [ebp-18h]
025230f8 3909            cmp     dword ptr [ecx],ecx
025230fa e879e9d953      call    PresentationFramework_ni+0x231a78 (562c1a78) (System.Windows.Application.get_MainWindow(), mdToken: 0600028c)
025230ff 8945e4          mov     dword ptr [ebp-1Ch],eax
02523102 e831e9d953      call    PresentationFramework_ni+0x231a38 (562c1a38) (System.Windows.Application.get_Current(), mdToken: 0600028a)
02523107 8945e0          mov     dword ptr [ebp-20h],eax
0252310a 8b4de0          mov     ecx,dword ptr [ebp-20h]
0252310d 3909            cmp     dword ptr [ecx],ecx
0252310f e864e9d953      call    PresentationFramework_ni+0x231a78 (562c1a78) (System.Windows.Application.get_MainWindow(), mdToken: 0600028c)
02523114 8945dc          mov     dword ptr [ebp-24h],eax
02523117 8b4ddc          mov     ecx,dword ptr [ebp-24h]
0252311a 3909            cmp     dword ptr [ecx],ecx
0252311c e82feedb53      call    PresentationFramework_ni+0x251f50 (562e1f50) (System.Windows.FrameworkElement.get_DataContext(), mdToken: 06000551)
02523121 8945d8          mov     dword ptr [ebp-28h],eax
02523124 8b55d8          mov     edx,dword ptr [ebp-28h]
02523127 b9a00bbf04      mov     ecx,4BF0BA0h (MT: WinDbgCourse.MainWindow)
0252312c e89f88bf0c      call    clr!JIT_IsInstanceOfClass (0f11b9d0)
02523131 8bc8            mov     ecx,eax
>>> 02523133 3909            cmp     dword ptr [ecx],ecx
02523135 e8e6e6db53      call    PresentationFramework_ni+0x251820 (562e1820) (System.Windows.Window.get_Title(), mdToken: 06000d20)
0252313a 8945d4          mov     dword ptr [ebp-2Ch],eax
0252313d 8b4dec          mov     ecx,dword ptr [ebp-14h]
02523140 8b01            mov     eax,dword ptr [ecx]
02523142 8b4028          mov     eax,dword ptr [eax+28h]
02523145 ff10            call    dword ptr [eax]
02523147 8945d0          mov     dword ptr [ebp-30h],eax
0252314a ff75d0          push    dword ptr [ebp-30h]
0252314d 8b4de4          mov     ecx,dword ptr [ebp-1Ch]
02523150 8b55d4          mov     edx,dword ptr [ebp-2Ch]
02523153 e884496054      call    PresentationFramework_ni+0xa97adc (56b27adc) (System.Windows.MessageBox.Show(System.Windows.Window, System.String, System.String), mdToken: 0600077b)
02523158 8945f0          mov     dword ptr [ebp-10h],eax
0252315b 90              nop
...

By function call we can realize that our code even does not reach instructions which use parameter. If we carefully match our source code to an assembly listing, we can understand our mistake and get to the root of the problem. We can make it a little bit easier if also show intermediate IL code:

0:000> !ip2md 02523133

MethodDesc:   04bf1e2c
Method Name:  WinDbgCourse.BasicNullRefCrashCommand.Execute(System.Object)
Class:        04c3043c
MethodTable:  04bf1e48
mdToken:      06000006
Module:       00ab4014
IsJitted:     yes
CodeAddr:     025230c0
Transparency: Safe critical
Source file:  C:\Users\wwwzl\Dropbox\l-projects\windbg-course\BasicNullRefCrashCommand.cs @ 15

0:000> !dumpil 04bf1e2c
ilAddr = 000320dd
IL_0000: nop
IL_0001: call System.Windows.Application::get_Current
IL_0006: callvirt System.Windows.Application::get_MainWindow
IL_000b: call System.Windows.Application::get_Current
IL_0010: callvirt System.Windows.Application::get_MainWindow
IL_0015: callvirt System.Windows.FrameworkElement::get_DataContext
IL_001a: isinst WinDbgCourse.MainWindow
IL_001f: callvirt System.Windows.Window::get_Title
IL_0024: ldarg.1
IL_0025: callvirt System.Object::ToString
IL_002a: call System.Windows.MessageBox::Show
IL_002f: pop
IL_0030: ret

Argument for !dumpil command taken from MethodDesc: 04bf1e2c line. We have had enough of that and can understand that original line contains two issues: unnecessary getting property DataContext and null-valued parameter. Bad news: it is so easy only for very trivial part of code such as, for example, in the demo. Good news: it is time to introduce SOSEX extension which can make our life much easier.

SOSEX

To load SOSEX extension you should enter command .load sosex. Unfortunately, for all extension commands WinDbg use the same namespace started with ! so SOS help will be overwritten by SOSEX help:

0:000> .loadby sos clr
0:000> !help
-------------------------------------------------------------------------------
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "!help <functionname>" for detailed info on that function.
...

0:000> .load sosex
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000> !help
SOSEX - Copyright 2007-2015 by Steve Johnson - http://www.stevestechspot.com/
To report bugs or offer feedback about SOSEX, please email sjjohnson@pobox.com
...

But it is correct only for a shorthand form of an extension command and you can access to any command by a full form:

0:000> !sos.help
-------------------------------------------------------------------------------
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "!help <functionname>" for detailed info on that function.
...

0:000> !sosex.help
SOSEX - Copyright 2007-2015 by Steve Johnson - http://www.stevestechspot.com/
To report bugs or offer feedback about SOSEX, please email sjjohnson@pobox.com
...

These commands are especially important because you can use only them to understand what each command does and do not read dump blog posts like that one:

0:000> !sos.help clrstack
-------------------------------------------------------------------------------
!CLRStack [-a] [-l] [-p] [-n]
!CLRStack [-a] [-l] [-p] [-i] [variable name] [frame]

CLRStack attempts to provide a true stack trace for managed code only. It is
handy for clean, simple traces when debugging straightforward managed
programs. The -p parameter will show arguments to the managed function. The
-l parameter can be used to show information on local variables in a frame.
SOS can't retrieve local names at this time, so the output for locals is in
the format <local address> = <value>. The -a (all) parameter is a short-cut
for -l and -p combined.
...

If have decided to read this post further, let’s move to the next very useful command (I specially do not say anything about arguments and hope that you will go to and try to execute !sosex.help muf -_-):

0:000> !muf 02523133
...
{
   IL_0000: nop
        025230ec 90              nop
MessageBox.Show(Application.Current.MainWindow, (Application.Current.MainWindow.DataContext as MainWindow).Title, parameter.ToString());
   IL_0001: call System.Windows.Application::get_Current
        025230ed e846e9d953      call    PresentationFramework_ni+0x231a38 (562c1a38)  [System.Windows.Application.get_Current()]
        025230f2 8945e8          mov     dword ptr [ebp-18h],eax
   IL_0006: callvirt System.Windows.Application::get_MainWindow
        025230f5 8b4de8          mov     ecx,dword ptr [ebp-18h]
        025230f8 3909            cmp     dword ptr [ecx],ecx
   IL_0006: callvirt System.Windows.Application::get_MainWindow
        025230fa e879e9d953      call    PresentationFramework_ni+0x231a78 (562c1a78)  [System.Windows.Application.get_MainWindow()]
        025230ff 8945e4          mov     dword ptr [ebp-1Ch],eax
   IL_000b: call System.Windows.Application::get_Current
   IL_000b: call System.Windows.Application::get_Current
        02523102 e831e9d953      call    PresentationFramework_ni+0x231a38 (562c1a38)  [System.Windows.Application.get_Current()]
        02523107 8945e0          mov     dword ptr [ebp-20h],eax
   IL_0010: callvirt System.Windows.Application::get_MainWindow
        0252310a 8b4de0          mov     ecx,dword ptr [ebp-20h]
        0252310d 3909            cmp     dword ptr [ecx],ecx
   IL_0010: callvirt System.Windows.Application::get_MainWindow
        0252310f e864e9d953      call    PresentationFramework_ni+0x231a78 (562c1a78)  [System.Windows.Application.get_MainWindow()]
        02523114 8945dc          mov     dword ptr [ebp-24h],eax
   IL_0015: callvirt System.Windows.FrameworkElement::get_DataContext
        02523117 8b4ddc          mov     ecx,dword ptr [ebp-24h]
        0252311a 3909            cmp     dword ptr [ecx],ecx
   IL_0015: callvirt System.Windows.FrameworkElement::get_DataContext
        0252311c e82feedb53      call    PresentationFramework_ni+0x251f50 (562e1f50)  [System.Windows.FrameworkElement.get_DataContext()]
        02523121 8945d8          mov     dword ptr [ebp-28h],eax
   IL_001a: isinst WinDbgCourse.MainWindow
        02523124 8b55d8          mov     edx,dword ptr [ebp-28h]
        02523127 b9a00bbf04      mov     ecx,4BF0BA0h
        0252312c e89f88bf0c      call    clr!JIT_IsInstanceOfClass (0f11b9d0)
        02523131 8bc8            mov     ecx,eax
>>>>>>>>02523133 3909            cmp     dword ptr [ecx],ecx
   IL_001f: callvirt System.Windows.Window::get_Title
        02523135 e8e6e6db53      call    PresentationFramework_ni+0x251820 (562e1820)  [System.Windows.Window.get_Title()]
        0252313a 8945d4          mov     dword ptr [ebp-2Ch],eax
...

Brilliant! SOSEX gives us much better result in one command against the result which we can achieve with three command of SOS.

Conclusion

It is just the first part of this course which is a basic scenario of crash dump analysis. And despite of easiness of this scenario, a well-known Visual Studio Debugger will not be helpful for such investigation. List of scenarios which can be investigated by WinDbg is really terrifying: crash dumps with incorrectly handled exception, application hangs, deadlock, memory and resource leaks etc. I hope that this part shows that WinDbg is not much harder than Visual Studio Debugger and you include it in your standard set of tools.

Comments

Comments powered by Disqus