ACorns.Debugging – The .Net Deadlock Detector
July 30, 2008 11:38 pm .Net, Debugging, Improve Your Code, Tools, Visual StudioAs nothing exciting has happed in my yard since the last release of Hawkeye I’ve decided to spice up my life and write a new .Net tool: The .Net Deadlock Detector.
The punch line: The .Net Deadlock Detector is the only* tool that is able to detect and report a deadlock inside a running .Net process in a production environment or out of a memory dump.
(* Disclaimer: I couldn’t find any other tool to do this. If you know of one please let me know)
Production environment is an environment in which you don’t (want to) have installed a debugging tool like Visual Studio.
The .Net Deadlock Detector
- The tool does not require to have the code re-compiled in any way or form, with any external dependencies, nor reference any external library or have you modify your code to use any special type of locks inside your code
- It works on release builds with no PDB files
- It works on running processes or previously captured memory dumps
- It detects deadlocks across multiple threads and returns detailed call-stack and lock usage information
- It only detect deadlocks in which threads are actively waiting for locks acquired by other threads
- It does not detect the dining philosophers problem or deadlocks created in combination of time waits + wake/check + lock
- It has an external dependency on the cdb.exe (part of the the free Debugging Tools for Windows package from Microsoft)
- It requires absolutely no installation. It an xcopy deployment
- And best of all it’s free (source code to be published soon)
What’s a deadlock
A deadlock is a situation wherein two or more competing actions are waiting for the other to finish, and thus neither ever does. It is often seen in a paradox like ‘the chicken or the egg‘. (wikipedia)
For example Thread 1 locks resource A, Thread 2 locks resource B, Thread 1 wants resource B and starts a wait on resource B, Thread 2 wants resource A and starts a wait on resource A.
In this moment the two threads are considered deadlocked as each of them owns a resource while trying to acquire another resource owned by a different thread.
- Two threads, two resources
- Thread 1 acquires resource A
- Thread 2 acquires resource B
- Thread 1 wants resource B and starts waiting for it
- Thread 1 wants resource A and starts waiting for it
Now both threads wait for the other one to release their resource.
How does the .Net Deadlock Detector work
The .Net Deadlock detector works by loading the cdb.exe (one of the native Windows Debuggers) on the target process and hooking the input and output streams of it to allow it to send commands and receive output from the debugger.
Then the tool is loading the sos (Son-on-Strike) debugging extensions into the cdb and starts sending commands to the cdb and sos and parse the output.
Then the tool follows a standard procedure in trying to find a managed deadlock by analysing the locks, the threads and the callbacks for each of the threads. As always one of the best examples to understand deadlocks is is Tess’s Deadlock case study.
So more or less the tool is a glorified macro system and command automation that is using standard cdb and sos commands to understand what is happening with the process and does some intensive analysis (including circular references search) to detect the deadlock.
How to “install” the .Net Deadlock Detector
- First of all before you can use the .Net Deadlock Detector you need to install (on your development machine not on the production machine) the Debugging Tools for Windows from Microsoft
- Then Download ACorns.Debugging.FindDeadlock.1.0.1.zip and unzip
- Now you need to copy the cdb.exe from the installation folder (defaults in C:\Program Files\Debugging Tools for Windows\cdb.exe) into the folder where the ACorns.Debugging.FindDeadlock.exe was unzipped
- You are now ready to use the tool
- You can now copy this new folder containing (ACorns.Debugging.Cdb.dll, ACorns.Debugging.FindDeadlock.exe and cdb.exe) to your production machine or target machine and start finding your deadlock
(Note: I’d love to deliver the a complete tool but the cdb.exe is not redistributable.)
How to use the .Net Deadlock Detector
If you finally got the right files prepared you are ready to try to find your deadlock.
The tool can be used with a set of command line parameters (exclusive):
ACorns.Debugging.FindDeadlock.exe [/pn=<processname>|/pid=<processid>|/DumpFile=<path to memory dump file>]
- /pn=name of a process
- /pid=id of a proces
- /DumpFile=path to the file
For example to try the tool on the provided demo application start the application then start the deadlock detector with:
ACorns.Debugging.FindDeadlock.exe /pn=ACorns.Debugging.DeadlockTests.exe
Interpreting the results
The tool will output a bunch of less relevant details as it tries to understand the deadlock and at the end it presents a “graphical” representation of the deadlock with some general details about the threads and the locks:
Thread 4 owns a lock named B and waits for lock A. Thread 3 owns the lock A and waits for B.
Then the most relevant part of the analysis comes: the callstack involved in the deadlock. (the callstacks should be read bottom to top with the Using of locks referring to the next (up) method)
Thread 4 has the following callstack:
We can see that method StartThraed2 is using lock B and the Thread2Worker is trying to use lock A.
Thread 3 has the following callstack:
Method StartThread1 is using lock A and Thread1Worker is trying to use lock B.
With all this information at hand you should be able to find and fix your deadlock. Good luck!
If you have a memory dump of a process that has a deadlock that the tool can’t detect please let me know as I’d like to debug it and improve the tool.
Now, start the download of the ACorns.Debugging.FindDeadlock.1.0.1.zip tool and then head to Microsoft to download the Debugging Tools for Windows.
The tool is based on an idea by Tatham Oddie and Paul Stovell. Thanks guys!
July 31st, 2008 at 3:32 am
Your tool is excellent.
I have 2 things to mention:
1) There’s a great windbg extension, written by Steve Johnson, called sosex which has a deadlock finding command like yours. You can find it at http://www.stevestechspot.com/downloads/sosex_32.zip and you can read more about it
here: http://www.stevestechspot.com/SOSEXANewDebuggingExtensionForManagedCode.aspx
and here: http://www.stevestechspot.com/SOSEXUpdatedV11Available.aspx .
2) Your application works great for monitors, but not for other types of synchronization mechanism, like ReaderWriterLockSlim to name one.
Here’s a simple application that deadlocks, but so far you fail to detect it:
using System;
using System.Threading;
namespace DeadLockTest
{
class Program
{
private static void Main()
{
Thread.CurrentThread.Name = "Thread_A";
Console.WriteLine("{0}: Starting thread.", Thread.CurrentThread.Name);
// locks
var lock_A = new object();
var lock_B = new ReaderWriterLockSlim();
// thread b code
new Thread(() =>
{
Console.WriteLine("{0}: Starting thread.", Thread.CurrentThread.Name);
Console.WriteLine("{0}: Locking A", Thread.CurrentThread.Name);
lock (lock_A)
{
Console.WriteLine("{0}: Locked A", Thread.CurrentThread.Name);
Thread.Sleep(100);
Console.WriteLine("{0}: Locking B", Thread.CurrentThread.Name);
lock_B.EnterWriteLock();
{
Console.WriteLine("{0}: Locked B", Thread.CurrentThread.Name);
}
lock_B.ExitWriteLock();
Console.WriteLine("{0}: Released B", Thread.CurrentThread.Name);
}
Console.WriteLine("{0}: Released A", Thread.CurrentThread.Name);
}) {Name = "Thread_B"}.Start();
// thread a (main) code
{
Console.WriteLine("{0}: Locking B", Thread.CurrentThread.Name);
lock_B.EnterWriteLock();
{
Console.WriteLine("{0}: Locked B", Thread.CurrentThread.Name);
Thread.Sleep(100);
Console.WriteLine("{0}: Locking A", Thread.CurrentThread.Name);
lock (lock_A)
{
Console.WriteLine("{0}: Locked A", Thread.CurrentThread.Name);
}
Console.WriteLine("{0}: Released A", Thread.CurrentThread.Name);
}
lock_B.ExitWriteLock();
Console.WriteLine("{0}: Released B", Thread.CurrentThread.Name);
}
Console.ReadLine();
}
}
}
— Omer Mor.
July 31st, 2008 at 2:01 pm
Hi Omer,
Thanks for the nice words. I’ve used SOSEX before and I actually wanted to include and use it now but I had no time to integrate it.
I’ve noticed that thinlocks are not correctly reported so I’m working on that and hope to release a new version soon. Your sample is perfect for me to start testing and improve my code.
Thanks,
Corneliu.
July 31st, 2008 at 5:29 pm
[...] ACorns.Debugging – The .Net Deadlock Detector – Corneliu announces a new deadlock detection tool, currently a binary release only, but sourcecode will follow at a later date. [...]
August 3rd, 2008 at 4:31 am
Pretty cool idea! I’ve been looking into deadlocks quite a bit in the past, and it seems that one of the main problem is that it seems to be impossible (without running something like a device driver in kernel space … then it would be easy) to find locks/blocks/deadlocks on kernel-level synchronization elements (Mutexes, …).
August 3rd, 2008 at 7:03 am
Would it be possible to use NSTD instead of CBD? That would have the advantage of NTSD generally being pre-installed on the target machine.
August 3rd, 2008 at 8:05 pm
Ingo,
The tool is only looking at managed locks that can be found as reported by the SOS. I’m now looking to add support for thinlocks as well but I’m not sure if I’ll try to go any further.
John,
That’s an interesting idea. I’ll look into adding support for NSTD as well.
August 5th, 2008 at 9:22 am
Excellent tool Corneliu!!!
Have you checked out the Windows WCT API (Wait Chain Traversal). It allows you to traverse the wait chain in any process (managed/unmanaged). May be you’ll find it usuful.
Also, IIRC, there is a knob in CLR hosting API that you can hookup your own logic to detect deadlocks and possibly recover from them (like SQL server for example). I think Joe Duffy’s got some sample stuff over here.
http://msdn.microsoft.com/en-us/magazine/cc163618.aspx
Finally, it would be really nice if you could do a simple graph travesal to automatically alert the user if there is a lock
.
Just some ideas… Anyway… cool stuff dude!
Cheers,
Buddhike
August 8th, 2008 at 12:58 pm
Hi Corneliu,
I just had a nasty deadlock and was scratching my head. I decided to break out the big guns and give ACorns.Debugging.FindDeadlock.exe a go.
It instantly found the lockup, and made it obvious where the lock order had been violated. It didn’t display the thread names, which might have helped – VS.NET 2008 wasn’t even responding to me, so I couldn’t use thread view from there – but the call stack display of the offending threads was a total lifesaver.
I think you just saved me at least a day of pain and misery.
THANKS!
Andrew
August 20th, 2008 at 4:25 am
[...] out all about it at Corneliu’s website.
August 20th, 2008 at 5:06 am
[...] info By Roman Hnatiuk Categories: .Net, Debugging and Tools ACorns.Debugging – The .Net Deadlock Detector is able to detect and report a deadlock inside a running .Net process in a production environment [...]
September 9th, 2008 at 12:15 am
[...] have to finish the Deadlock detector as I think I have a fix for the tool not finding deadlocks between a lock() and a slim lock or [...]
September 11th, 2008 at 1:05 pm
the description for figure 5 is wrong:
Thread 1 wants resource A and starts waiting for it
should be:
Thread 2 wants resource A and starts waiting for it
September 12th, 2008 at 12:00 pm
Hi, I downloaded your tools and tried to use it with your demo in that zip, but it always shows this output below:
Using ACorns.Debugging Tools: 1.0.1.0
Locks with waiting threads: SyncBlockInfo
ThreadInfo
Found: 0 locks ...
The command is: “ACorns.Debugging.FindDeadlock.exe /pn=ACorns.Debugging.DeadlockTests.exe” (you provided in the article)
I don’t know why…In fact if I use my own program with deadlocks the result is still the same…
Thanks
September 19th, 2008 at 5:30 pm
Hello, I’m from Japan.
And I encountered the same message as the previous H’s comment. It says,
Using ACorns.Debugging Tools: 1.0.1.0
Locks with waiting threads: SyncBlockInfo
ThreadInfo
Found: 0 locks …
My system is Windows Server 2003 R2 Standard Edition Service Pack 2. And I tested your tool on Windows XP with SP3 and I made sure it works fine on the Win XP.
Thank you,
September 19th, 2008 at 8:16 pm
Can I correct my previous comment?
I made sure that your tool is working fine even on Windows Server 2003 R2 w/ SP2.
But, your test scenario, ACorns.Debugging.FindDeadlock.exe detects a deadlock created by ACorns.Debugging.DeadlockTests.exe, does not work on a system incorporated dual-core CPU.
Thank you,
September 19th, 2008 at 8:49 pm
Hi Takeshi,
I think you just provided me with a critical piece of information: The fact that you tried it on W2K3 and it failed. I’ll have it a try on W2K3 to see what’s happening and I’ll try to fix it and post an update.
Thanks,
Corneliu.
September 19th, 2008 at 8:50 pm
Ok, Thanks for this as well. I’ll try to debug it on a dual core and see what’s happening.
Thanks for the details.
Corneliu.
October 1st, 2008 at 5:47 am
I also cannot get it to work with the included Test program. I’m running XP SP3, and have a single core CPU. Any ideas?