Skip to end of metadata
Go to start of metadata

When you are coding, how often do you run your unit test suite? Once a day? Once an hour? Once a minute? Could you even run the suite once a minute if you wanted to? Project Background

Also see Extreme XP Tools.

General Information

Introduction

Regardless of your political beliefs, as an agile and test-infected developer, we'd like you to consider the benefits of adhering to strict, some might say "dictatorial", limits on the way that you write your unit tests as well as the code being tested (SUT).

We should clarify that these restrictions apply only to unit tests (and not system tests or integration tests, which are also of great importance to the success of any project). However, your unit tests, at least one for each class in your codebase, should be clean and decoupled from one another. Your unit test suite should run ultra-fast, so that you're encouraged to run it often. We think you should be able to run thousands of unit tests and get your green bar in a few seconds. Achieving that kind of performance takes major discipline on the part of the developer, and Ashcroft helps you learn that discipline by failing tests which stray from acknowledge best practices.

Remember, the most severe restrictions often lead to the most creative solutions.

Ten Commandments of Unit Tests

  1. I am the class being tested. Thou shalt not test any other class but me.
  2. Thou shalt write isolated tests
  3. Thou shalt not access files during unit tests
  4. Thou shalt not write two tests which depend upon each other

There are more of these to come... we are currently soliciting feedback.

How Does This Work

Ashcroft has an implementation of the Java seurity manager. Once registered (See FAQ), it will start monitoring the process. Ashcroft can be configured through additional VM parameter (See Ashcroft Configuration).

More Details

Technical FAQ

Design FAQ

Java Security Manager

Download

You can download the attached jar at this page.

Or you can build it from the Subversion repository: svn://beaver.codehaus.org/ashcroft/scm

Open issues and bugs

com.atlassian.confluence.macro.MacroExecutionException: JIRA project does not exist or you do not have permission to view it.

License

Copyright 2004-2005 (C) Obie Fernandez and Aslak Hellesoy. All Rights Reserved.

Redistribution and use of this software and associated documentation
("Software"), with or without modification, are permitted provided
that the following conditions are met:

1. Redistributions of source code must retain copyright
statements and notices. Redistributions must also contain a
copy of this document.

2. Redistributions in binary form must reproduce the
above copyright notice, this list of conditions and the
following disclaimer in the documentation and/or other
materials provided with the distribution.

3. The name "ashcroft" must not be used to endorse or promote
products derived from this Software without prior written
permission of Aslak Hellesoy and Obie Fernandez. For written permission,
please contact aslak.hellesoy@gmail and obiefernandez@gmail.

THIS SOFTWARE IS PROVIDED BY OBIE FERNANDEZ AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
OBIE FERNANDEZ OR HIS CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.

  • No labels

20 Comments

  1. Anonymous

    I don't know about "Thou shald not access files during unit tests"...

    What about - my program produces a report and saves it as a pdf. To test that, by definition, I have to access the PDF, correct? The same applies across the system - mock objects don't do it - they just allow you to check that we wanted to produce a file - I want to actually test that my system is actually capable of producing a PDF!

    I would assert: "it is impossible to test functionality that produces or changes external files, without accessing files"...

  2. Anonymous

    You don't need to write the PDF to disk to check that it got produced correctly. Use a ByteArrayOutputStream, for example.

  3. Anonymous

    I wonder how many of the commandments appear in the book "JUnit Recipes" as guidelines. I like the idea – in a perverse way – of making them into "commandments". I would have written about them that way, if it weren't for the risk of offending some would-be readers.

  4. Why not allow setting system properties? Is there some sort of performance penalty I'm not aware of?

  5. Anonymous

    "You don't need to write the PDF to disk to check that it got produced correctly. Use a ByteArrayOutputStream, for example."

    What if (as is the case) - I am using a third party printer driver to produce the pdf's? I want to check that my communication to the printer driver is correct, that the printer driver is correctly installed, and that the whole thing works together to produce a pdf.

    I could mock up the printer driver, but that actually tells me nothing at all - it tells me that I'm sending out the stuff I thought I was sending out, under my understanding of the driver. Does not imply in the slightest that the aim of the piece of code - ie "to produce a valid pdf" - can be satisfied.

  6. Anonymous

    Mmmm... sounds to me like you are writing an integration test. Not a unit test.

  7. Why not allow setting of system properties? Here are some reasons:
    1) It has a sticky side effect that may impact other tests.
    2) If your class requires access to system properties it is not a unit, but part of a larger system. It is wiser to instantiate your class by passing in a Properties object. That allows you to test it in isolation. At deploy/integration test time you'd typically instantiate it with the Properties object returned by System.getProperties(), but during unit tests you shouldn't.

  8. Re: JUnit Recipes - I didn't know that book. I will definitely take a look at it. I am sure it could help us improve both the Ashcroft tool and its upcoming documentation.
    Our plan is that the message of each thrown CantDoThat will contain a http link to a page that contains:
    o An explanation of why you shouldn't do what you (or someone else) did.
    o One or more suggestions/tips for how you should refactor your code.

    It is funny, Obie and I started Ashcroft because we wanted to improve the execution speed of an existing codebase. We discovered that as we started to fix the tests that Ashcroft failed, we ended up refactoring the code into being much more decoupled. So Ashcroft gives you two things:
    1) Faster running unit tests
    2) More decoupled and elegant code

  9. Re: 3rd party driver to generate PDFs. In that case you're talking about an integration test - integration between your code and that driver. You can't unit test that - such tests should go into an integration testsuite that does not use Ashcroft.

    In other words, Ashcroft is encouraging you to group your tests in different test suites (unit, system and integration), where the majority of them should be in the unit testsuite.

    Of course, using JUnit as the framework for unit, system and integration tests is fine. -But just because a test extends from junit.framework.TestCase, that doesn't make it a unit test per se. At least not according to me (and many others). We'll have to work on these definitions, I know.

  10. Anonymous

    Here's one more commandment:

    Thou shalt only test public and protected methods.

    In practice, I suspect this commandment may in fact run into conflicts with some of the other principles, particularly the ones about separating unit tests from fucntionality tests. However this principle is critical for using unit tests and code coverage tools for identifying unreachable code and branches with code coverage tools. If you can't write a test that reaches code solely by exercising the public/protected API, then the code should be deleted.

  11. I think protected methods fall in the same category as private methods. Why would you want to test a protected method? They are always called as a direct consequence of calling a public method right? (smile)

  12. Anonymous

    Well, yes... and no.

    I insert an interface in front of the File API, so that I can test my objects without writing to a real File. I still have to implement my new interface with something that delegates to a File.

    The good news is this: I can test my new file-delegating implementation /once/, with tests that actually writes to files, then tuck it away and trust it. For every other application I ever write, I test my objects against the file interface and never need to write to a real file.

  13. Anonymous

    Only wimpy languages like Java and C# need "protected". Real men write Smalltalk – all data is private and all methods are public, period, end of sentence. Fortunately, I can also do that in Java and C#.

  14. Anonymous

    If protected methods were the same as private methods, there wouldn't be two keywords. Protected methods are part of the published interface of a class. They can be invoked by other, user-written classes, just not by every object. To unit test a protected method, you may need to write a subclass, but that's legal and far less skanky than some of the hacks I've seen people use to unit test private and default methods.

  15. I like and agree with your reasoning, thanks for the project and tip.

  16. How about some of these, probably some of which I believe help define a unit test:

    • Thou shalt only test one bit of functionality at a time.
    • Thou shalt not override in order to test anything.
    • If we're going for the individual 'file' thing, then there should be: Thou shalt not access a database during a unit test.
    • Thou shalt not write to System.out and System.err
    • Thou shalt not have tests without assertions
  17. Anonymous

    Nonsense. Unless you never use inheritance (final/sealed classes).

    Public: the interface the class provides towards the system as a whole. Protected+public: the interface the class provides towards descendants. A well designed inheritance interface is as important as the public interface in any framework, and needs to be tested at least as much as the public interface – otherwise, you should smack on the final/sealed modifier on your class.

    Ah, for the religious wars of old... (smile)

  18. Do you consider private Inner Classes as bad for testability of code ? The only way to test them is to test the surrounding Parent class.

  19. Anonymous

    In the item "How Does This Work"

    ....Java seurity manager....
    it's missing a c there...