(Unit-)Testing State of Mind

My mind has been steeped deep in the quagmire of automated testing for the past few days. If ever there were an area of software development where I wished for a silver bullet that I could just load and fire…

I’ve gone from a job where testing was predominantly a lightly automated activity, to a job where large swaths of the testing effort are automated to a high degree. I definitely prefer the latter in principle, if only there were a definitive book of recipes and best-practices to follow.

I enjoyed watching the Pluralsight video on Code Testability; it has the best explanation of what makes testing code hard that I’ve ever seen. The side-by-side examples in which he shows how using dependency injection increased what he calls the “sphere of influence” (or the directly testable surface) of the code was an A-Ha moment for its sheer clarity.

And another video that has gotten lost in the blur in my memory made a great point regarding keeping tests descriptive and linear. Unit Tests are easy to write because they do not branch or loop. They test one thing well at a time, and as a result feel more reliable; being able to read and trivially understand a test is a major strength.

And yet… it bothers me when a lot of the hands-on examples in videos still show how to test an “ArgumentException” type gets thrown. I blogged about exceptions what feels like an eternity ago (Exceptions – 3 specifically), and ArgumentExceptions should not be caught; they aren’t for code to deal with, they are for code to try and avoid at all cost. So why test them? Like… ever?

There are plenty of proto-guides on (unit-)testing best practices, but few if any get beyond the obvious: isolate your test cases, pick structured names, be wary of mocking, test one thing only. But that still doesn’t really give any guidance on what is worth testing. Writing tests is easy enough, writing the right tests, and no more than necessary, is the hard part.

Are there resources out there I haven’t found yet?

Exceptions – 5

Before diving into the final category of exceptions, I want to make a little detour into fundamentals. I promise this will be relevant shortly.

Preconditions / Post-conditions / Invariants
Formal Methods and Design by Contract are often a dim memory after a few years in a full-time software development job. It would be easy to conclude that therefore they do not get applied in real business. But in actual fact, contracts are everywhere.

Whenever you write a method, it’s name, result type, parameter names and parameter types are part of an ad-hoc specification covering preconditions and post-conditions.

int Square(int n);
void ValidateOrder(Order order);
Order MergeOrders(params Order[] validOrders);

I should be able to reasonably assume without looking at the code that:

  • Square” will return me the square of its argument
  • ValidateOrder” will probably throw some exception when the contents of “order” do not meet validation standards
  • MergeOrders” will create a new single order object out of a collection of other orders, provided they can be combined (and if not, likely throws an exception). Also, the name of the argument strongly implies that validation may need to be done prior to calling it.

It is of course possible that the names and types are misleading and these methods do something completely different, but in that case I’d argue that they are not meeting their implicit contracts.

Compare this with the following signatures were they to have exactly the same implementations:

int Calculate(int n);
void Process(Order order);
object HandleTogether(IEnumeration toCombine);

By simply changing some names and types I have destroyed a lot of the implicit documentation this method provided:

  • There is no indication what relationship there is between inputs and outputs for “Calculate“. Even worse, I can no longer reasonably assume this method succeeds for all integers “n” without looking at the documentation or implementation.
  • The name “Process” although technically accurate (but then, isn’t everything processing in some sense?) gives the misleading impression that it might in some sense execute the order. Exceptions could still be expected if processing fails, but it might prompt a defensive implementation predicated on the false assumption that there may have been side-effects.
  • And “HandleTogether” pretty much completely obscures both the nature of the operation and the preconditions that must be satisfied by its arguments. Let’s hope the documentation comments are actually helpful!

As these examples already alluded to, exceptions logically form a part of the specification of a method.

/// <summary> ... </summary>
/// <param name="validOrders"> ... </param>
/// <exception cref="ArgumentException">
/// validOrders == null || validOrders.Length == 0
/// </exception>
/// <exception cref="ValidationException">
/// Any of "validOrders" fails validation.
/// </exception>
/// <exception cref="MergeException">
/// Not all "validOrders" have the same customer details.
/// </exception>
Order MergeOrders(params Order[] validOrders);

There could potentially be a lot more involved, but now the exception documentation confirms and enhances the specification implicit in the method signature itself.

Note that there is one further improvement that could be made above; currently the first exception is an “ArgumentException“, which therefore corresponds to a precondition (see Usage Exceptions post). The “ValidationException” is presumably the exception thrown by the “ValidateOrder” method that we’d be using internally to make sure all the orders are valid before attempting the merge. And the “MergeException” is a new exception specific to this method that indicates incompatible orders.

In reality, all these should probably be preconditions to the method, and therefore be implemented as “ArgumentException” instances. It is in most cases much better to fail early before any calculations have been done.

Vexing Exceptions
In practice, “Vexing Exceptions” pretty much need to be dealt with in the same way as any other “Logical Errors”, but they indicate a badly designed API (see original overview post). In the remainder of this post I will not treat them separately, but I want to dedicate a few moments here to recognising and avoiding them when writing new code.

In the previous section I had a few example methods to illustrate extending method signatures with exception specifications.

/// <exception cref="ValidationException" />
void ValidateOrder(Order order);

/// <exception cref="ArgumentException" />
/// <exception cref="ValidationException" />
/// <exception cref="MergeException" />
Order MergeOrders(params Order[] validOrders);

And I commented that the “ValidationException” in “MergeOrder” corresponded to a possible result of using the “ValidateOrder” method, and that all conditions on “MergeOrders” would be better served being “ArgumentException” across the board.

To do so for the validation exception would mean that “MergeOrders” needs to implement a catch handler during its precondition checks and wrap a “ValidationException” into a descriptive “ArgumentException“. This is precisely a “Vexing Exception”, because we would be much better served by a second API variant of “ValidateOrder” that returns errors, or even just a boolean:

IEnumeration<ValidationError> ValidateOrder(Order order);
bool TryValidateOrder(Order order);

Then we can do the validation in our merging routine without having to catch exceptions and wrap them.

Whether or not transforming a “MergeException” into a “Usage Error” makes sense depends on a number of factors, including whether an up-front check would have to re-implement substantial portions of the logic from the body of the method. Sometimes it may be better to leave the exception unchanged.

Note however that either way we really need a further method:

  • If we make it a precondition, then the caller needs to be able to avoid passing in incompatible orders
  • If we leave it unchanged, we have another potentially vexing exception in case compatible orders cannot be guaranteed

The caller of the merge method needs to either structure the code so that it is implicitly guaranteed that orders passed into the method will be compatible, or there needs to be an “bool AreOrdersCompatible(...);” method so that a failing call can be potentially avoided where it might otherwise routinely occur.

Logical Errors” / “Exogenous Exceptions
And now that we have eliminated everything else, what exactly are we left with? It turns out that I lied in the last section… I am not quite done with “Vexing Exceptions” yet.

Since “Vexing Exceptions” are thrown under expected circumstances, where providing an API alternative that directly returns a result indicating those circumstances is the preferable approach, I think “Exogenous Exceptions” can best be summarised as follows:

“Exogenous Exceptions” correspond to unexpected circumstances that cannot be avoided but can potentially be resolved by the caller.

So, in a perfect world:

  • “Usage Errors” are only ever thrown and never caught, because they indicate a caller that does not respect preconditions
  • “System Failures” are only ever thrown by the environment and never by code and they are never trapped, because they indicate the environment has become unreliable and the application should be allowed to terminate
  • “Vexing Exceptions” never occur because all our APIs provide method alternatives to avoid them
  • “Exogenous Exceptions” are only ever thrown if a method cannot satisfy its post-conditions due to unexpected, unavoidable circumstances outside its control, and each type of exception corresponds uniquely to one type of remedial action

Let’s start with an illustrative example of a “Logical Error” from the .NET Framework itself.

    using (FileStream fs = new FileStream("...", FileMode.Open))
        ... load resource ...
catch (FileNotFoundException)
    ... load from elsewhere ...

As I was trying to come up with a good example of an “Exogenous Exception”, it became more and more clear to me that in-principle there are none. Every time an exception is thrown by a method, the question remains: is this an expected or unexpected exception? And the only code that can answer this question is the caller.

In the fragment above, it is tempting to say something along the lines of “you cannot avoid this exception; it is thrown when a file does not exist when you try to open it“, but somewhere in the implementation of the “FileStream” constructor, there is a line that determines whether the low-level Windows API succeeded or failed, and turns that into an exception. If I write code using the “FileStream” API, where I can routinely expect files I am trying to open will no longer exist, then this is suddenly a “Vexing Exception”.

The only reason I have no choice but to use an exception handler is that using “File.Exists(...)” does not help, because the file may go missing between calling this and the “FileStream” constructor. And there is no constructor alternative “FileStream.TryCreate(...)” that can allow me to normally handle this condition. Vexing indeed.

Note however that this does not mean that all is lost, and the naysayers about exceptions were right after all. Far from it. I think “FileStreamshould throw an exception if the file does not exist. But it should also have an alternative that doesn’t.

And this goes for all methods, because ultimately the only arbiter of what is expected to go wrong (“Vexing”) and what is not (“Exogenous”) is the calling code; it’s the use that determines the nature of the exception.

(Sidenote: this possibly explains the ongoing religious war over whether exceptions or error codes are the best way to handle errors. Those against exceptions tend to look at “Vexing Exceptions” as their rationale, whereas those in favour can only see “Exogenous Exceptions”. It turns out we really need both.)

To Be Continued…
I was going to finish up here with a description of how to implement methods, how to consume methods, and what can be done to formalise and automate some of the required discipline in all this…

…but this post is getting a bit long already, and I think having just the guidelines in a single post will provide a better reference.

(Really just a stalling technique so I can let my most recent lightbulb-moments filter into this before I come to a final decision.)

Exceptions – 4

In my last post on exceptions I covered “Boneheaded Exceptions” and why they should not be caught (and what to do about them instead). Next-up is another category that should hardly ever be caught… except in a very specific fashion.

System Failures” / “Fatal Exceptions” (also: the system is down)
These are exceptions that originate in the implementation of the execution environment. Some can get thrown by specific (types of) IL instructions, such as “TypeLoadException” or “OutOfMemoryException“. Others can get thrown at literally any instruction, such as “ExecutionEngineException“.

The two key observations about these exceptions is that they cannot be prevented (because they originate from the low-level execution of your code itself), and there are virtually no circumstances where your application code can do anything to resolve the indicated problem (something went wrong that is by definition out of the control of your code). They can happen at any time and there is no way to fix them; it should be obvious why they should not normally be caught.

If like me, you find yourself trying to construct a scenario where you might want to catch one of these, ask the following questions. If a type fails to load cleanly indicating a broken deployment, can you trust any further remedial action to even work? If you run out of memory, what kind of logic could you write that does not itself need to allocate memory? Worst of all; if the execution engine failed in some unspecified way, can you even rely upon correct execution of any further instructions?

Even if there are specific corner-cases where anything can be done at all, how much value would it add over just letting the application terminate from its illegal state and construct some external mechanism to restart it into a valid state instead?

So, what to do?
If the foregone conclusion is that these cannot be handled in any way, then all that is left is ensuring the application dies as gracefully as possible.

First and foremost, use the “try {} finally {}” pattern wherever possible. There may be cases where the “finally” will fail in part or whole due to the nature of the system failure, but it maximizes the chances that files flush the last useful fragments, transactions get cleanly aborted, and shared system resources are restored to a safer state.

Very few “System Errors” / “Fatal Exceptions” get caught explicitly in a handler. This is precisely because there is nothing specific that can be done to remedy them. There is however a very commonly used handler that deserves scrutiny; the much-reviled “catch (Exception ex) {}“.

Since there are precious few fatal exceptions that can be meaningfully handled in any fashion, it should be obvious that writing a handler purporting to deal with all of them is even more preposterous. That is why the following is the only valid pattern for a general exception handler:

    // Some code
catch (Exception ex)
    // ???

Only by re-throwing the exception at the end of the handler can we guarantee that all the various fatal exceptions keep bubbling up to the top of the application, where termination of the application is the final signal of an unrecoverable problem.

The following two questions need to be answered then:

  • What kind of “some code” could be protected in this structure?
  • What kind of logic can sensibly be placed at “???”

To start with the latter; when something non-specified goes wrong, the only sensible options are to either record details not generally available in a stack trace in some fashion, or to make general fixes to the state-space that “some code” may have trashed.

Recording additional detail can be done by either logging something somewhere about values of relevant variables at the time the execution failed, or alternately to wrap the exception in a custom exception that records the values in its properties (in which case it should hold the original exception as an inner exception!)

Writing a general fix for corrupted state-space can be difficult. As one extreme, the fatal exception may have occurred in the middle of an allocation inside the “Dictionary.Add()” method, and now you’re stuck with a dictionary in an inconsistent and unrecoverable state. It may however be possible to just replace the dictionary with a new empty dictionary in the catch handler, providing that does not break any invariants that need to hold. In many cases, the “some code” will have made state-space changes that cannot be credibly put back in some correct default state, at which point you should resist the temptation to write any catch handler. If you cannot do anything,… then don’t.

Now, it should be obvious what “some code” could be; anything that either can benefit from additional information about the local state-space being recorded when a problem occurs, or anything for which affected state-space can be restored to some kind of safe default that does not break any invariants. (An example of the latter might be a manipulation of a cache of some sort that fails; restoring the cache to an empty state does not invalidate it’s invariants. It may hurt ongoing performance, but it does neatly restore the local state into a valid default.)

How to fix Fatal Handlers?
Many libraries or applications will have fallen prey to catching and swallowing “Exception” somewhere (including code I have written myself). The logical-sounding rationale usually is something like “If anything goes wrong while doing this, then let me put some default behaviour in that is good enough in its stead”. Default behaviour can range from returning a default value, all the way up to just logging the exception and moving on, hoping for the best.

while (...some high-level loop...)
        ...some piece of processing logic...
    catch (Exception ex)
        LogException("Could not process, retry next iteration", ex)

On the face of it, it is easy to make yourself believe this improves the robustness of the above processing loop. Now, if anything goes wrong, it will try again some number of times depending on the high-level loop.

But as we’ve seen above, this really just makes a whole range of potential problems worse rather than better. There is no guarantee that the next iteration of the loop will even do the same thing that the failed iteration did. Instead of producing a file, the next loop could be deleting them. Rather than having a simple understandable error fall out of the application at the point of the original problem, we may end up doing all kinds of unpredictable things that are going to be impossible to diagnose or recover after-the-fact.

When you find code that contains general exception handlers, warning bells should be ringing. There is a reason there is an FxCop rule that triggers on this coding pattern. It is an evil pattern that must be exorcised.

The only valid fixes for “Exception” handlers are as follows:

  • Re-throw the original exception at the end of the handler (see “what to do?” above)
  • Throw a new exception that includes further details about the problem, and which must include the original exception as an inner-exception (see “what to do?” above)
  • Make the exception type more specific so that a problem that can be credibly recovered from is caught instead (and make sure the handling logic actually addresses that problem!)
  • Remove the handler altogether, and just let the exception mechanism do it’s thing

Some of these remedies edge into the territory of “Logical Errors” / “Exogenous and Vexing Exceptions” and my next post will dig much deeper into how to deal with those. That’ll be where the rubber meets the road on what many would consider actual exception handling, and what kind of exceptions you can declare and throw yourself (and how to do so).

Exceptions – 3

In my last post I presented a classification of exceptions by two Microsoft employees that should know what they are talking about. Here, I want to pick off the low hanging fruit and discuss just one of the categories of exceptions. A category that should not be caught.

Usage Errors” / “Boneheaded Exceptions” (also: Preconditions)
It may sound strange at first to say that any category of exceptions should never be caught. All documentation on exceptions keeps drilling home the message that exceptions exist to be caught and handled; unfortunately that isn’t true.

This category of exceptions serves the sole purpose of notifying the programmer that a method cannot be validly called with the given arguments and/or in the current state of the object it is called on. It signals that the programmer did not honour the preconditions of the method.

Handling such an exception is putting the cart before the horse. Let’s say we have the following method (note that the non-null condition is purely by way of a simple example; a better implementation would gracefully handle null strings):

/// <summary>
/// Manipulate a string in some fashion
/// </summary>
/// <param name="value">
/// The string to manipulate, must be non-null
/// </param>
/// <returns>The manipulated result</returns>
public string SomeMethod(string value);

And we use this method in some code, only to discover later that due to logic elsewhere nulls can end up making their way to this method. After careful analysis, it turns out for null values we want the result also to be null, so the following code fragment is written:

string result;
    result = SomeMethod(someValue);
catch (ArgumentNullException)
    result = null;

See? All fixed. Isn’t it wonderful how we are handling the exception, and everything is perfect, right?

Of course this is wrong. The correct way to deal with this situation is to not violate the precondition in the first place and just do it right:

string result =
    someValue == null
    ? null
    : SomeMethod(someValue);

Whenever you feel tempted to handle a “Usage Error” / “Boneheaded Exception” you should immediately wonder why the code isn’t checking the precondition, or alternately designed to honour the precondition by definition.

Letting these exceptions fall out of your application is not a sign that you forgot to add an exception handler. It’s a sign that you didn’t write the code correctly. Rather than spending time writing exception handlers, put that time into guard conditions on the call. Not only will it make your assumptions much more explicit, it also performs much better. An exception will typically take 1000s of times longer to process than an equivalent guard statement on your call.

Catching these exceptions doesn’t just indicate you’ve lost control of your code. It also signals that you don’t really care about performance at all.

So, what to do?
Because these exceptions are never intended to be caught but rather to tell the programmer that a precondition was not honoured, the exception class matters very little. What does matter is to make sure that the exception message is very explicit about the precondition that was not satisfied, and in what way it was not satisfied. Make the message as long and wordy as you need to, because a programmer will need to be able to read it, understand what the problem was, and then fix the code.

Although the type of the exception does not matter much, because a programmer is going to mainly go off the message, there are a few exception types in the .NET Framework that are specifically suitable to be used for these.

class ArgumentException
    class ArgumentNullException
    class ArgumentOutOfRangeException
    /// various others ...
class InvalidOperationException
    class ObjectDisposedException
    /// various others ...

Use “ArgumentException” or one of its sub-classes to signal when an argument to a method does not satisfy a precondition. Some recommend using the most derived class that is appropriate to the error that occurred, but as long as you make sure the exception message will make complete sense to the programmer it is fine to just use “ArgumentException” itself and no others.

Use “InvalidOperationException” or one of its sub-classes to signal when an object property does not satisfy a precondition to the method call being made. The same advice goes for sub-classes here as for “ArgumentException“.

By just using these two exception (trees) it also is very easy to make sure you never end up catching a “Usage Error” / “Boneheaded Exception”. Creating an FxCop rule that forbids these types occurring in exception handlers should be a breeze.

Also note that for reasons not related to exception semantics, “InvalidOperationException” should probably never occur. An API that could credibly throw this exception for the reason outlined above is very badly designed and should probably be refactored. (An example scenario is a class that has a flag to indicate a processing direction (input / output) and methods that only are allowed to be called for one of these modes. A better implementation would have a general base-class for shared functionality and then subclass into an input and an output class that each only have suitable methods on them.)

How to fix Boneheaded Handlers?
Whenever you encounter code that handles a Boneheaded Exception “X”, corresponding to precondition “P” that looks as follows:

catch (X)

Replace the code with the following instead:

if (P)

(Note that this assumes that exception ‘X’ actually corresponds to a precondition ‘P’, and gets thrown before any functional logic runs. Otherwise side-effects may make the transformation more complicated.)

Next post I’ll pick off the next-lowest-hanging fruit; “System Failures” / “Fatal Exceptions”.

Exceptions – 2

At this point, it seems appropriate to put some terminology in place for my ongoing discussion of throwing and handling exceptions. As a matter of fact, I will be providing two sets of terminology for the price of one!

These sets of terminology are by Krzysztof Cwalina (leader of the effort to develop API guidelines) and Eric Lippert (senior member of the team designing the C# language). I think it’s fair to say that between the two of them there is a lot of experience with how to do things and how not to do them using C# and .NET.

Krzysztof Cwalina classifies exceptions into “usage errors” and “system errors“, the latter of which can be split into “logical errors” and “system failures“. Eric Lippert classifies them into “fatal“, “boneheaded“, “vexing” and “exogenous“.

All that may just sound like a jumble of (colourful) word-soup, so the following sections will make the terms and what they mean a bit more concrete.

Usage Errors” / “Boneheaded Exceptions
These exceptions signal that there is a problem the calling code could have avoided itself. As a result this is a bug in the caller, not a fault in the called code. Typically these are the result of broken preconditions and invariants.

public void DoSomething(string nonNullValue)
    if (value == null)
        throw new ArgumentNullException("value");

You must make sure your code passes this method a non-null value; if you do not, then the resulting exception is your own fault. You broke the contract.

System Failures” / “Fatal Exceptions
These exceptions cannot be handled under any circumstance. They signal a fundamental problem with the state of the virtual machine, such as “Out of Memory” or “Thread Aborted” or “Type Load” exceptions that can occur at almost any instruction in your program.

The only correct thing to do with these exceptions is to let them climb up the stack until they eventually terminate the program. There is nothing an application should try to do to recover from these, because there is no sensible way to recover. This is also why catching “Exception” is so heavily frowned upon.

There are ways to write general recovery handlers, but they have to follow a very specific pattern to make sense. More on that in a later post.

Logical Errors” / “Exogenous and Vexing Exceptions
These are the “real” exceptions. They indicate that the method could not make good on its promises in some fashion. You asked the method to open a file for reading, but the file doesn’t exist; too bad! You asked the method to parse a string into an integer, but there were letters in the string; oops!

The reason that Eric Lippert presents two options here is specifically for my second example. Some exceptions thrown by methods that cannot satisfy their contract indicate that the API was just badly designed; sometimes you have to expect certain failures, and code accordingly.

using (var reader = new StreamReader("DataFile.txt"))
    var line = reader.ReadLine();
        var value = int.Parse(line);
    catch (FormatException ex)
        // Is this Exogenous or Vexing?

As the exception handler asks… is this exception exogenous, or vexing? I really can’t say, because it depends on the context. If the “DataFile.txt” was created by an end user, and is supposed to contain a single line with an integer value on it, then this is almost certainly a vexing exception, and use of the “int.TryParse(...)” method would have been more appropriate. Betting on a human-generated file to contain correctly formatted input is wishful thinking; you have to assume there may be problems.

If however that file were produced by another application, then this may very well be an appropriate way to deal with the situation. We can safely assume that if the other program produces an integer one time, it will likely do so every time, and if it doesn’t that is genuinely worthy of an exception and associated logic (albeit in reality probably a few levels further up the stack than my simplistic example).

Up Next…
In the next post I hope to distill down some initial advice on these three categories of exceptions. I will not treat vexing exceptions as a category from here on in, since a vexing exception really indicates an incomplete API that needs to be redesigned. Usually it is just a matter of adding alternatives that allow the caller to avoid the exception in favour of a more complex method contract, similar to the way the “TryParse” calls were added in the .NET Framework to many classes that didn’t have them before).

Exceptions – 1

As much as exception-handling is pretty much part-and-parcel of any modern programming language, resources and guides on how to most effectively and correctly use them is fairly thin on the ground.

I noticed recently that when I throw exceptions, or catch them, there is a sort-of-almost-there structure and pattern to the way I use them. But what bothered me is that I didn’t have a strong underlying philosophy regarding why I was using them in that way (and I bet my usage hasn’t been 100% consistent either).

So on this holiday I’ve kinda set out on some self-study to try and formulate what I think is the best way to use exceptions in C#/.NET … there’s some resources that provide good basic information in this matter, but none of them go to the point of taking it to the logical conclusion of fully thought-through recipes that will result in correct and consistent use; so that’s my goal at the end of this series of posts: an article that conclusively formalises my opinion on throwing and handling exceptions.

The initial materials that form the basis for these posts are as follows: