Sometimes we encounter scenarios which are hard to debug using our favorite IDE debugger. A typical scenario is a hard-to-reproduce production issue. These scenarios usually occur more frequently as load and users increase on production systems – causing performance-issues, timing-issues and user behavior which can be hard, or even impossible, to reproduce in development environments.
It further complicates the debugging process that production systems are often locked-down and isolated. Thus, it may not be possible to attach a remote debugger or install 3rd party software on these environments.
In the following, we will outline how to approach debugging such cases.
Creating Process Dumps
To start debugging a production issue, we’ll need a way to look inside our running process to figure out what is going on. Luckily, Windows has built-in support for dumping processes which takes a snapshot of the process at the given time incl. thread stacks and full memory and writes it to a file. This allows us to examine the problem without further affecting the running process or to copy the dump to a remote system to perform the analysis there. The simplest way to obtain such a dump is using Task Manager and performing Right-click Process -> Create dump file.
Writing the dump file will hang the process for some seconds (depending on process size), but as soon as it is done your process will continue executing and serving user requests. In case your process is in a broken state, you could now restart it to resume normal operation and you can start debugging your dump file without worrying about affecting production stability.
There are several ways to obtain a dump file and if you are running a GUI-less Windows Server installation this PowerShell script might come in handy.
In case of company policy preventing developer access to production systems, the above steps can also be performed by a System Administrator, who can forward the resulting dump file for remote analysis by a Developer.
Debugging dump files originating from another system can be a bit tricky as Windows and .NET versions might mismatch. Ensure you have Microsoft Symbol Server correctly configured.
Analyzing Dumps
While Visual Studio’s capability to analyze dumps is improved with every major version, the console-based and light-weight WinDbg remains the de facto debugging tool for Windows dump analysis. It can be installed on the troubling server for quick analysis or used remotely and comes with a wide range of commands allowing both high-level analysis and deep diving into every little detail of your dumped process.
WinDbg can be obtained from Microsoft where it is bundled with the Windows Driver Kit or the Windows SDK.
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/
For debugging directly on a server where installers can be problematic to execute, WinDbg is also xcopy deployable (use above installer on your local PC and copy “C:\Program Files (x86)\Windows Kits\10\Debuggers” directory to the target server).
Getting started with WinDbg
To start analyzing a process dump, go to File -> Open Crash Dump and open the .dmp file.
WinDbg only supports native process debugging out-of-the-box. To debug managed (.NET) applications we need to make use of the SOS (Son of Strike) extension, which is installed as part of the .NET Framework. To load this extension type .loadby sos clr – if you receive no output back, the command was successful.
Now we have access to the managed commands offered by SOS.
Here is a few commands to get us started:
!threadpool – Thread pool information (Note that if CPU utilization shows 81% this is often an indication that the process is currently performing Garbage Collection)
!runaway – View CPU time per thread (Use this to identify problematic threads in a high CPU scenario)
!threads – Lists all application threads
~17s – Select thread number 17
!clrstack – View stack information for currently selected thread
~*e!clrstack – View stack information for all threads
!dumpheap -stat – View memory information by type (good starting point for tracking down memory issues)
Most debugging scenarios start by running one of the above commands and digging into the details from there using further commands to explore the memory and stacks in more detail.
For a comprehensive list of common WinDbg commands see: http://windbg.info/doc/1-common-cmds.html
The right dump at the right time
Using the Task Manager to dump processes can be convenient when timing is not critical – such as high CPU usage or high memory consumption scenarios.
For other scenarios timing is essential. Often you need to create a dump at the right time in the right method to see the value of a variable or you may need to create a dump when a certain exception is raised to further investigate the problem.
Several tools exist to help automating process dumps.
Here is a list of the most commonly used ones:
ADPlus
Comes with WinDbg and can be found in the WinDbg installation folder. As with WinDbg, ADPlus can also be xcopied to the target server.
ADPlus is a command line utility that supports creating automated dump files when it detects certain scenarios such as crashes or hangs.
For usage see: https://support.microsoft.com/en-us/help/286350/how-to-use-adplus-vbs-to-troubleshoot-hangs-and-crashes
Debug Diagnostic Tool (DebugDiag)
Download: https://www.microsoft.com/en-us/download/details.aspx?id=49924
Debug Diagnostic Tool is a rule based GUI tool useful for creating automated dump files.
Supports breakpoints, exceptions, hangs, crashes and memory/handle leak scenarios.
ProcDump
Download: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump
One of the most powerful tools for automated dump creation is a small console-based tool initially developed by Mark Russinovich and today part of the Sysinternals suite. The tool supports a wide variety of configuration options.
Besides covering the most common scenarios, it also offers built in support for taking dumps based on performance counters.
WinDbg as post-mortem debugger
For easy debugging of crashing applications on your developer machine you can setup WinDbg as your default post-mortem debugger.
Using an elevated command prompt, run:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64> windbg.exe -I
When an application crashes, it will now automatically launch an instance of WinDbg attached to the process. From Task Manager you can also automatically attach to the desired process by performing Right-click Process -> Debug.
If you also wish to configure WinDbg as the default application for opening .dmp files, run:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64> windbg.exe -IA
Further reading
Production debugging and dump analysis is a big topic and there are a wealth of tools, blogs and documentation available. This blog post is meant as an introduction to the topic and primary tools involved for .NET applications. To continue this journey this StackOverflow post is a good reference and contains links to many relevant blog posts, videos and tools.
