TechEd done and dusted

.Net, Debugging, Readify, Tools, Visual Studio 5 Comments

Well, it seems like everyone is doing a post TechEd review of their own.

Me … I have to start with the comment that this was the best TechEd I’ve been too mostly because of the comments I’ve received for my sessions.

Some comments were good, some were great, some were funny and some were really eye-openers. (oh, yes, some were dull but they are worthless).

On Sunday I flew to TechEd NZ and much to my surprise when I’ve tried to book into my reserved hotel I was informed that I was upgraded and moved to the hotel on the other side of the road. Well, thank you. This was much appreciated. I was basically in the hotel just above the conference. I was taking the lift to level 4-5 and I was existing directly in the middle of the conference. You godda love that.

On Tuesday I had to do both my talks one after the other and then run straight to the airport. I didn’t get any feedback back as TechEd NZ seems to use paper to collect feedback and evaluations and heard no news from them. I hope they were good as I want to go back next year … one week earlier to do a bit of skiing as well …

On Wednesday I had my talk on debugging which went very well. The room was packed and the demos worked a threat. The only issue I had were the lights which were pointing straight into my eyes giving me the feeling of an FBI interview. It was hard to see the people in the room and the raised hands for questions.

The reviews&evals were really great. So great, I jumped straight in the top 10 best sessions and top 5 best speakers. Thanks everyone for filling in the reviews.

I have to write here some of the comments:

  • Best session I’ve been to for several Techeds, congratulations. this is what it is all about….
  • Simply amazed by this session, by far the best one I have seen during Tech-ed so far.
  • Utterly brilliant – this session paid for the entire entrance fee Excellent

Well, there were over 50 such comments so thanks everyone for taking the time and writing them.

Now, some of the funny ones included no comments but a rating of “Too Technical“. Well, for what we considered (and it was marked as) a 400 level presentation and some considered in their comments as a 300 level presentation, a “Too Technical” comment can only be considered a compliment. If you left that comment, then mate, the session was supposed to be technical ;)

Some of the eye-opener comments:

  • I understand there was alot to cover, but at times the information was a bit rushed, and I found myself unable to keep up with some of the more arcane sections
  • The only thing is that sometimes he’d speak a little too quickly, which combined with the accent made it hard to hear some things
  • Well delivered. He did have a “balky” moment when he mentioned he might shoot himself in the “leg”.(Woops. It was supposed to be “foot”. Sorry for that.)
  • Speaker’s accent was hard to understand at times

Comments from the security talk:

  • Ease up on the hand waving Corneliu :-) When flicking through code please linger a little longer to give people at least 2-3 seconds to process the context and the detail.
  • Speaks very fast
  • The guy was like a ferret on speed. Great knowledge, but flicked between code wayyy too quick. There were very few web developers in the room, and most of it was about web stuff. Can’t remember what was in the description, but this was disappointing.
  • Very web oriented, which was not obvious from the title.

Thanks a lot for these comments.

I know exactly what I need to focus on for next year.

  • Speak slower (and keep that rhythm of talking): I know I was quite conscious at times of this during my debugging talk but I completely lost myself in the security one as I had the feeling I’m running out of time
  • Present less: Have a smaller amount of information to talk about in one hour. Plan to have the session for only 50 minutes so I have time to speak slower. It’s easy to have an ace up the sleeves prepared that can fill in the last 5 minutes if you finish early.
  • Fix my accent: Maybe if I can just speak a bit slower it’s easier with my accent as well. I know my diction in English is quite bad and I’m hard to understand at time (let me tell you I speak as bad and hard to understand in my native Romanian as well). I’ve already engaged an English teacher (my lovely wife) to prepare a course for me to improve my diction and accent.
  • Work on some of the “quotes”/”jokers” that I use in my talks and make sure they are right. I’ll have to make sure you shot yourself in the “foot” not in the “leg”. (The original joke I was trying to refer to is How to Shoot Yourself In the Foot)

Well, now, back to my small corner of development:

  1. I have to finish the Deadlock detector as I think I have a fix for the tool not finding deadlocks between a lock() and a slim lock or read/write lock.
  2. I want to publish a Secure Web library for automatic encoding for ASP.Net controls that I was talking about in my security talk.
  3. I need to publish a new site for my games shop
  4. I need to start planning my upcoming trip to Europe

And most importantly I want to start planning for some new RDN talks and for some cool talks for next year’s TechEd.

PS>> If you were in my sessions and are looking for the demos/slides, please logon to TechEd portal and navigate to the session scheduler and you can find the pptx attached to the description of the sessions.

ACorns.Debugging – The .Net Deadlock Detector

.Net, Debugging, Improve Your Code, Tools, Visual Studio 18 Comments

As nothing exciting has happed in my yard since the last release of Hawkeye I’ve decided to spice up my life and write a new .Net tool: The .Net Deadlock Detector.

The punch line: The .Net Deadlock Detector is the only* tool that is able to detect and report a deadlock inside a running .Net process in a production environment or out of a memory dump.

(* Disclaimer: I couldn’t find any other tool to do this. If you know of one please let me know)

Production environment is an environment in which you don’t (want to) have installed a debugging tool like Visual Studio.

The .Net Deadlock Detector

  1. The tool does not require to have the code re-compiled in any way or form, with any external dependencies, nor reference any external library or have you modify your code to use any special type of locks inside your code
  2. It works on release builds with no PDB files
  3. It works on running processes or previously captured memory dumps
  4. It detects deadlocks across multiple threads and returns detailed call-stack and lock usage information
  5. It only detect deadlocks in which threads are actively waiting for locks acquired by other threads
  6. It does not detect the dining philosophers problem or deadlocks created in combination of time waits + wake/check + lock
  7. It has an external dependency on the cdb.exe (part of the the free Debugging Tools for Windows package from Microsoft)
  8. It requires absolutely no installation. It an xcopy deployment
  9. And best of all it’s free (source code to be published soon)

What’s a deadlock

A deadlock is a situation wherein two or more competing actions are waiting for the other to finish, and thus neither ever does. It is often seen in a paradox like ‘the chicken or the egg‘. (wikipedia)

For example Thread 1 locks resource A, Thread 2 locks resource B, Thread 1 wants resource B and starts a wait on resource B, Thread 2 wants resource A and starts a wait on resource A.

In this moment the two threads are considered deadlocked as each of them owns a resource while trying to acquire another resource owned by a different thread.

  1. Two threads, two resources
    image
  2. Thread 1 acquires resource A
    image
  3. Thread 2 acquires resource B
    image
  4. Thread 1 wants resource B and starts waiting for it
    image
  5. Thread 1 wants resource A and starts waiting for it
    image

Now both threads wait for the other one to release their resource.

How does the .Net Deadlock Detector work

The .Net Deadlock detector works by loading the cdb.exe (one of the native Windows Debuggers) on the target process and hooking the input and output streams of it to allow it to send commands and receive output from the debugger.

Then the tool is loading the sos (Son-on-Strike) debugging extensions into the cdb and starts sending commands to the cdb and sos and parse the output.

Then the tool follows a standard procedure in trying to find a managed deadlock by analysing the locks, the threads and the callbacks for each of the threads. As always one of the best examples to understand deadlocks is is Tess’s Deadlock case study.

So more or less the tool is a glorified macro system and command automation that is using standard cdb and sos commands to understand what is happening with the process and does some intensive analysis (including circular references search) to detect the deadlock.

How to “install” the .Net Deadlock Detector

  1. First of all before you can use the .Net Deadlock Detector you need to install (on your development machine not on the production machine) the Debugging Tools for Windows from Microsoft
  2. Then Download ACorns.Debugging.FindDeadlock.1.0.1.zip and unzip
  3. Now you need to copy the cdb.exe from the installation folder (defaults in C:\Program Files\Debugging Tools for Windows\cdb.exe) into the folder where the ACorns.Debugging.FindDeadlock.exe was unzipped
  4. You are now ready to use the tool
  5. You can now copy this new folder containing (ACorns.Debugging.Cdb.dll, ACorns.Debugging.FindDeadlock.exe and cdb.exe) to your production machine or target machine and start finding your deadlock

(Note: I’d love to deliver the a complete tool but the cdb.exe is not redistributable.)

How to use the .Net Deadlock Detector

If you finally got the right files prepared you are ready to try to find your deadlock.

The tool can be used with a set of command line parameters (exclusive):

ACorns.Debugging.FindDeadlock.exe [/pn=<processname>|/pid=<processid>|/DumpFile=<path to memory dump file>]

  • /pn=name of a process
  • /pid=id of a proces
  • /DumpFile=path to the file

For example to try the tool on the provided demo application start the application then start the deadlock detector with:

ACorns.Debugging.FindDeadlock.exe /pn=ACorns.Debugging.DeadlockTests.exe

Interpreting the results

The tool will output a bunch of less relevant details as it tries to understand the deadlock and at the end it presents a “graphical” representation of the deadlock with some general details about the threads and the locks:

image

Thread 4 owns a lock named B and waits for lock A. Thread 3 owns the lock A and waits for B.

Then the most relevant part of the analysis comes: the callstack involved in the deadlock. (the callstacks should be read bottom to top with the Using of locks referring to the next (up) method)

Thread 4 has the following callstack:

image

We can see that method StartThraed2 is using lock B and the Thread2Worker is trying to use lock A.

Thread 3 has the following callstack:

image

Method StartThread1 is using lock A and Thread1Worker is trying to use lock B.

With all this information at hand you should be able to find and fix your deadlock. Good luck!

If you have a memory dump of a process that has a deadlock that the tool can’t detect please let me know as I’d like to debug it and improve the tool.

Now, start the download of the ACorns.Debugging.FindDeadlock.1.0.1.zip tool and then head to Microsoft to download the Debugging Tools for Windows.

The tool is based on an idea by Tatham Oddie and Paul Stovell. Thanks guys!