Why xUnit Should Be Called xTest

JUnit is one of the simplest, yet also perhaps the most effective of all the software frameworks I've ever seen: it solves a very specific class of problems (writing and executing automated tests in Java) very well, with very little code and very little ceremony, which is especially true of version 4, which takes great advantage of Java annotations. Consequently, I use it whenever I'm writing Java code using Test Driven Development (TDD) method.
There is one big problem with JUnit, however, that keeps popping up over and over again: its name.
Every now and then, I will see Joe the developer broadcasting an e-mail to explain that Mary from the other group broke several unit tests of Joe's component by changing behavior of her component that Joe's component depends on. This is typically followed by a comment by Bob saying that he's very well aware of Mary's unfortunate change, because it also broke five unit tests of Bob's component.
What's going on here is that a typical developer these days seems to think that the very fact that a test is written using JUnit, or any other xUnit framework (where x is typically substituted by the first letter, or an acronym of programming language that the framework supports) automatically makes that test a "unit test".
Majority of automated tests written using JUnit or any other xUnit testing framework that I've had the opportunity to see are in reality integration tests, which test not only the unit that the author is interested in testing (typically a single class, or a cluster of tightly coupled classes), but also every bit of code (including 3rd party libraries, services and systems) that this unit happens to depend on. That's probably because xUnit tests become integration tests by mere inertia, while making them unit tests requires focused effort on mocking out dependencies.
Why does incorrectly categorizing such tests matter? Because the properties of unit and integration tests are extremely different! For example, by calling the JUnit-based test that you've written a "unit test", you are implicitly signalling to other developers that it can be run quickly (in a matter of milliseconds) and reliably, without impacting any database, or other system. But if that test turns out to be integration test, these implicit assumptions are going to be violated, and cause a lot of grief, potentially not just to the person who ran the test (e.g., by having to wait 30 minutes for it to finish), but also to the people that depend on this test (e.g., by corrupting their database, and thus conflicting with the automated tests they've gotten accustomed to depend on).
Over years, I've adopted a simple, but effective naming scheme that minimizes confusion about what kind of test a particular xUnit class implements: if the class is a collection of true unit tests, then it has suffix "UnitTest", and if it's a collection of integration tests, then its suffix is "IntegrationTest". This does make the test class names a bit longer than with the usual xUnit suffix "Test", but it takes the guess work out of the question. It also allows automated running of only unit tests, or all automated tests, depending on their type by simple pattern matching.
Perhaphs the biggest advantage of such approach when applied with discipline is that it forces me to think about the type of the test I'm about to write before I write a single line.
If I have the choice, I will almost always write a unit test, because it's always going to be faster, more reliable and easier to maintain than the integration test of the same functionality. But when the code I want to test depends on a library or framework that's hard to replace with a testing dummy or a mock, integration test may just be the right kind of test to write.
Recently, I ran into such a case when using Apache Velocity templating engine, which allows adding few lines of its (quite awkward) proprietary scripting language to the HTML page in order to populate it with data. This is accompanied with a few lines of Java code that provides that data to Velocity framework. So, while it might be even possible to mock Velocity's Java APIs using an advanced mocking technology, that kind of unit test would not be particularly valuable, as it would not test the script code embedded in the HTML. So, in this case, I decided to create a class that hides the presence of Velocity library from the rest the system, and then write an integration test for this class, which tested the implementation of that class together with its interactions with Velocity and the script embedded in HTML. It did not have to use any database, but it did end up reading and writing files on the local file system. I used HTMLUnit testing framework to verify that the output HTML was as expected for a simplified, but still non-trivial set of inputs.
This testing strategy was useful because it addressed the highest risk of the component being tested: ensuring that all pieces of the puzzle, each simple enough on its own right, fit together and produce expected output. This integration test was definitely not as fast as a unit test: one full run would take up to a minute. It was also harder to write and maintain than a unit test: having a non-trivial set of input data meant there was a non-trivial set of outputs, which thus required a lot of assertions in the test. But, a minute long feedback loop was still useful enough to allow its repeated incremental execution, and this integration test was still an order of magnitude faster than an equivalent end-to-end test would be. It was also much easier to write and maintain, since the input data set was still an order of magnitude simpler than it would have been in an end-to-end test (where input data set for this component would have full complexity of a real world system).
This integration test paid off very quickly, when it allowed me to detect several issues, always caused by a single missing, or misplaced character (did I mention that this syntax of Velocity scripting language is really awkward?). Being able to quickly detect such issue, see the error message produced by Velocity framework, and iterate to find the fix was priceless. If I was depending on a unit test, I would never have noticed the problem at the first place.
And if I had relied solely on an end-to-end test, it could have easily taken several days, instead of minutes to find the right fix and verify it.
In conclusion, integration tests lie between unit tests on one end of the test automation spectrum, and end-to-end tests, on another end. Most of the time, they provide the least bang for the buck, and developers are better off covering their code with unit tests, and addressing highest project risks with few well-chosen end-to-end tests. But there are some special circumstances where integration tests may be the most cost effective choice. The key is to be clear what kind of automated test one is writing, and make that decision consciously by assessing the project risks and associated constraints before actually starting to write the test. Following a naming scheme like the one I've described above helps to enforce that, and avoid the trap of automatically considering every test written using an xUnit framework a unit test.