Beck Testing Framework

Introduction

Smalltalk has suffered as a result of it lacked a testing tradition. This column
describes a easy testing technique and a framework to help it. The testing technique
and framework usually are not meant to be full options, however quite a place to begin from
which industrial power instruments and procedures may be constructed.

The paper is split into three sections:

• Philosophy – Describes the philosophy of writing and working checks
  embodied by the framework. Learn this part for common background.
• Cookbook – A easy sample system for writing your personal checks.
• Framework – A literate program model of the testing framework. Learn
  this for in-depth data of how the framework operates.
• Instance – An instance of utilizing the testing framework to check a part of the
  strategies in Set.

Philosophy

I don’t like consumer interface-based checks. In my expertise, checks
primarily based on consumer interface scripts are too brittle to be helpful. Once I was on a undertaking
the place we used consumer interface testing, it was widespread to reach within the morning to a take a look at
report with twenty or thirty failed checks. A fast examination would present that the majority or all
of the failures had been really this system working as anticipated. Some beauty change within the
interface had precipitated the precise output to now not match the anticipated output. Our testers
spent extra time preserving the checks updated and monitoring down false failures and false
successes than they did writing new checks.

My resolution is to put in writing the checks and examine ends in Smalltalk. Whereas
this strategy has the drawback that your testers want to have the ability to write easy
Smalltalk packages, the ensuing checks are way more secure.

Failures and Errors

The framework distinguishes between failures and errors. A failure is an
anticipated drawback. Once you write checks, you examine for anticipated outcomes. In case you get a
completely different reply, that may be a failure. An error is extra catastrophic, a error situation you
did not examine for.

Unit testing

I like to recommend that builders write their very own unit checks, one per class.
The framework helps the writing of suites of checks, which may be hooked up to a category. I
advocate that each one lessons reply to the message “testSuite”, returning a set
containing the unit checks. I like to recommend that builders spend 25-50% of their time
creating checks.

Integration testing

I like to recommend that an impartial tester write integration checks. The place
ought to the mixing checks go? The current motion of consumer interface frameworks to
higher programmatic entry supplies one answer- drive the consumer interface, however do it with
the checks. In VisualWorks (the dialect used within the implementation beneath), you may open an
ApplicationModel and start stuffing values into its ValueHolders, inflicting all kinds of
havoc, with little or no hassle.

Operating checks

One ultimate little bit of philosophy. It’s tempting to arrange a bunch of take a look at
knowledge, then run a bunch of checks, then clear up. In my expertise, this all the time causes extra
issues that it’s value. Checks find yourself interacting with each other, and a failure in a single
take a look at can forestall subsequent checks from working. The testing framework makes it simple to set
up a standard set of take a look at knowledge, however the knowledge can be created and thrown away for every take a look at.
The potential efficiency issues with this strategy should not be an enormous deal as a result of
suites of checks can run unobserved.

Cookbook

Right here is an easy sample system for writing checks. The patterns are:

Sample	Objective
Fixture	Create a standard take a look at fixture.
Check Case	Create the stimulus for a take a look at case.
Examine	Examine the response for a take a look at case.
Check Suite	Combination TestCases.

Fixture

How do you begin writing checks?

Testing is a kind of unattainable duties. You’d prefer to be
completely full, so that you may be certain the software program will work. Alternatively, the
variety of attainable states of your program is so giant which you could’t presumably take a look at
all combos.

In case you begin with a obscure concept of what you’ll be testing,
you’ll by no means get began. Much better to begin with a single configuration whose
conduct is predictable. As you get extra expertise along with your software program, it is possible for you to
so as to add to the listing of configurations.

Such a configuration known as a “fixture”. Examples of
fixtures are:

Fixture	Predictions
1.0 and a couple of.0	Straightforward to foretell solutions to arithmetic issues
Community connection to a identified machine	Responses to community packets
#() and #(1 2 3)	Outcomes of sending testing messages

By selecting a fixture you might be saying what you’ll and received’t take a look at
for. An entire set of checks for a group of objects may have many fixtures, every of
which can be examined some ways.

Design a take a look at fixture.

• Subclass TestCase
• Add an occasion variable for every identified object within the fixture
• Override setUp to initialize the variables

Within the instance, the take a look at fixture is 2 Units, one empty and one with
parts. First we subclass TestCase and add occasion variables for the objects we’ll
have to reference later:

Class: SetTestCase
    superclass: TestCase
    occasion variables: empty full

Then we override setUp to create the objects for the fixture:

SetTestCase>>setUp
    empty := Set new.
    full := Set
    with: #abc
    with: 5

Check Case

You might have a Fixture, what do you do subsequent?

How do you signify a single unit of testing?

You may predict the outcomes of sending a message to a fixture. You want
to signify such a predictable state of affairs in some way.

The only strategy to signify that is interactively. You open an
Inspector in your fixture and also you begin sending it messages. There are two drawbacks to
this technique. First, you retain sending messages to the identical fixture. If a take a look at occurs to
mess that object up, all subsequent checks will fail, despite the fact that the code could also be right.
Extra importantly, although, you may’t simply talk interactive checks to others.
In case you give another person your objects, the one manner they’ve of testing them is to have
you come and examine them.

By representing every predictable state of affairs as an object, every with its
personal fixture, no two checks will ever intrude. Additionally, you may simply give checks to others
to run.

Signify a predictable response of a fixture as a technique.

• Add a technique to TestCase subclass
• Stimulate the fixture within the technique

The instance code reveals this. We will predict that including “5” to
an empty Set will lead to “5” being within the set. We add a technique to our
TestCase subclass. In it we stimulate the fixture:

SetTestCase>>testAdd
    empty add: 5.
    ...

Upon getting stimulated the fixture, that you must add a Examine to make
certain your prediction got here true.

Examine

A Check Case stimulates a Fixture.

How do you take a look at for anticipated outcomes?

In case you’re testing interactively, you examine for anticipated outcomes
immediately. If you’re in search of a specific return worth, you utilize “print it”,
and just be sure you bought the precise object again. If you’re in search of unwanted effects, you
use the Inspector.

Since checks are in their very own objects, you want a strategy to programmatically
search for issues. One strategy to accomplish that is to make use of the usual error dealing with
mechanism (Object>>error:) with testing logic to sign errors:

2 + 3 = 5 ifFalse: [self error: ‘Wrong answer’]

Once you’re testing, you’d like to differentiate between errors
you might be checking for, like getting six because the sum of two and three, and errors you
didn’t anticipate, like subscripts being out of bounds or messages not being
understood.

There’s not rather a lot you are able to do about unanticipated errors (when you did
one thing about them, they wouldn’t be unanticipated any extra, would they?) When a
catastrophic error happens, the framework stops working the take a look at case, data the error,
and runs the following take a look at case. Since every take a look at case has its personal fixture, the error within the
earlier case won’t have an effect on the following.

The testing framework makes checking for anticipated values easy by
offering a technique, “ought to:”, that takes a Block as an argument. If the Block
evaluates to true, every thing is ok. In any other case, the take a look at case stops working, the failure
is recorded, and the following take a look at case runs.

Flip checks right into a Block evaluating to a Boolean. Ship the Block as
the parameter to “ought to:”.

Within the instance, after stimulating the fixture by including “5” to
an empty Set, we wish to examine and ensure it’s in there:

SetTestCase>>testAdd
    empty add: 5.
    self ought to: [empty includes: 5]

There’s a variant on TestCase>>ought to:. TestCase>>shouldnt:
causes the take a look at case to fail if the Block argument evaluates to true. It’s there so that you
don’t have to make use of “(…) not”.

Upon getting a take a look at case this far, you may run it. Create an occasion
of your TestCase subclass, giving it the selector of the testing technique. Ship
“run” to the ensuing object:

(SetTestCase selector: #testAdd) run

If it runs to completion, the take a look at labored. In case you get a walkback,
one thing went fallacious.

Check Suite

You might have a number of Check Instances.

How do you run a lot of checks?

As quickly as you’ve got two take a look at instances working, you’ll wish to run them
each one after the opposite with out having to execute two do it’s. You could possibly simply string
collectively a bunch of expressions to create and run take a look at instances. Nevertheless, if you then
wished to run “this bunch of instances and that bunch of instances” you’d be caught.

The testing framework supplies an object to signify “a bunch of
checks”, TestSuite. A TestSuite runs a set of take a look at instances and studies their
outcomes . Benefiting from polymorphism, TestSuites can even include different
TestSuites, so you may put Joe’s checks and Tammy’s checks collectively by making a
increased degree suite.

Mix take a look at instances right into a take a look at suite.

(TestSuite named: ‘Cash’)
    add: (MoneyTestCase selector: #testAdd);
    add: (MoneyTestCase selector: #testSubtract);
    run

The results of sending “run” to a TestSuite is a TestResult
object. It data all of the take a look at instances that precipitated failures or errors, and the time at
which the suite was run.

All of those objects are appropriate for storing with the ObjectFiler or
BOSS. You may simply retailer a set, then deliver it in and run it, evaluating outcomes with
earlier runs.

Framework

This part presents the code of the testing framework in literate
program fashion. It’s right here in case you might be curious concerning the implementation of the
framework, or that you must modify it in any manner.

Once you speak to a tester, the smallest unit of testing they discuss
is a take a look at case. TestCase is a Consumer’s Object, representing a single take a look at case.

Class: TestCase
    superclass: Object

Testers discuss organising a “take a look at fixture”, which is an
object construction with predictable responses, one that’s simple to create and to motive
about. Many alternative take a look at instances may be run in opposition to the identical fixture.

This distinction is represented within the framework by giving every TestCase
a Pluggable Selector. The variable conduct invoked by the selector is the take a look at code. All
cases of the identical class share the identical fixture.

Class: TestCase
    superclass: Object
    occasion variables: selector
    class variable: FailedCheckSignal

TestCase class>>selector: is a Full Creation Technique.

TestCase class>>selector: aSymbol
    ^self new setSelector: aSymbol

TestCase>>setSelector: is a Creation Parameter Technique.

TestCase>>setSelector: aSymbol
    selector := aSymbol

Subclasses of TestCase are anticipated to create and destroy take a look at fixtures
by overriding the Hook Strategies setUp and tearDown, respectively. TestCase itself supplies
Stub Strategies for these strategies which do nothing.

TestCase>>setUp
    "Run no matter code that you must prepare for the take a look at to run."

TestCase>>tearDown
    "Launch no matter assets you used for the take a look at."

The only strategy to run a TestCase is simply to ship it the message
“run”. Run invokes the arrange code, performs the selector, the runs the tear
down code. Discover that the tear down code is run no matter whether or not there’s an error
in performing the take a look at. Invoking setUp and tearDown may very well be encapsulated in an Execute
Round Technique, however since they aren’t a part of the general public interface they’re simply open
coded right here.

TestCase>>run
    self setUp.
    [self performTest] valueNowOrOnUnwindDo: [self tearDown]

PerformTest simply performs the selector.

TestCase>>performTest
    self carry out: selector

A single TestCase is hardly fascinating, upon getting gotten it
working. In manufacturing, you’ll want to run many TestCases at a time. Testers speak of
working take a look at “suites”. TestSuite is a Consumer’s Object. It’s a Composite of
Check Instances.

Class: TestSuite
    superclass: Object
    occasion variables: identify testCases

TestSuites are Named Objects. This makes them simple to determine so that they
may be merely saved on and retrieved from secondary storage. Right here is the Full
Creation Technique and Creation Parameter Technique.

TestSuite class>>named: aString
    ^self new setName: aString

TestSuite>>setName: aString
    identify := aString.
    testCases := OrderedCollection new

The testCases occasion variable is initialized proper in
TestSuite>>setName: as a result of I don’t anticipate needing it to be any completely different
sort of assortment.

TestSuites have an Accessing Technique for his or her identify, in anticipation of
consumer interfaces which must show them.

TestSuite>>identify
    ^identify

TestSuites have Assortment Accessor Strategies for including a number of
TestCases.

TestSuite>>addTestCase: aTestCase
    testCases add: aTestCase

TestSuite>>addTestCases: aCollection
    aCollection do: [:each | self addTestCase: each]

Once you run a TestSuite, you would like all of its TestCases to run. It is
not fairly that straightforward, although. If in case you have a set that represents the acceptance take a look at for
your software, after it runs you’d prefer to understand how lengthy the suite ran and which of the
instances had issues. That is info you desire to to have the ability to retailer away for future
reference.

TestResult is a Consequence Object for a TestSuite. Operating a TestSuite
returns a TestResult which data the data described above- the beginning and cease
instances of the run, the identify of the suite, and any failures or errors.

Class: TestResult
    superclass: Object
    occasion variables: startTime stopTime testName failures errors

Once you run a TestSuite, it creates a TestResult which is timestamped
earlier than and after the TestCases are run.

TestSuite>>run
    | end result |
    end result := self defaultTestResult.
    end result begin.
    self run: end result.
    end result cease.
    ^end result

TestCase>>run and TestSuite>>run usually are not polymorphically
equal. This can be a drawback that must be addressed in future variations of the
framework. One choice is to have a TestCaseResult which measures time in milliseconds to
allow efficiency regression testing.

The default TestResult is constructed by the TestSuite, utilizing a Default
Class.

TestSuite>>defaultTestResult
    ^self defaultTestResultClass take a look at: self

TestSuite>>defaultTestResultClass 
    ^TestResult

A TestResult Full Creation Technique takes a TestSuite.

TestResult class>>take a look at: aTest
    ^self new setTest: aTest

TestResult>>setTest: aTest
    testName := aTest identify.
    failures := OrderedCollection new.
    errors := OrderedCollection new

TestResults are timestamped by sending them the messages begin and cease.
Since begin and cease must be executed in pairs, they may very well be hidden behind an Execute
Round Technique. That is one thing else to do later.

TestResult>>begin
    startTime := Date dateAndTimeNow

TestResult>>cease
    stopTime := Date dateAndTimeNow

When a TestSuite runs for a given TestResult, it merely runs every of its
TestCases with that TestResult.

TestSuite>>run: aTestResult
    testCases do: [:each | each run: aTestResult]

#run: is the Composite selector in TestSuite and TestCase, so you may
assemble TestSuites which include different TestSuites, as an alternative of or along with
containing TestCases.

When a TestCase runs for a given TestResult, it ought to both silently
run appropriately, add an error to the TestResult, or add a failure to the TestResult.
Catching errors is simple-use the system provided errorSignal. Catching failures should be
supported by the TestCase itself. First, we’d like a Class Initialization Technique to create a
Sign.

TestCase class>>initialize
    FailedCheckSignal := self errorSignal newSignal
    notifierString: 'Examine failed - ';
    nameClass: self message: #checkSignal

Now we’d like an Accessing Technique.

TestCase>>failedCheckSignal
    ^FailedCheckSignal

Now, when the TestCase runs with a TestResult, it should catch errors and
failures and inform the TestResult, and it should run the tearDown code no matter
whether or not the take a look at executed appropriately. This ends in the ugliest technique within the framework,
as a result of there are two nested error handlers and valueNowOrOnUnwindDo: in a single technique. There
is a lacking sample expressed right here and in TestCase>>run about utilizing guarantee: to
safely run the second halt of an Execute Round Technique.

TestCase>>run: aTestResult
    self setUp.
    [self errorSignal
        handle: [:ex | aTestResult error: ex errorString in: self]
        do: 
            [self failedCheckSignal
                handle: [:ex | aTestResult failure: ex errorString in: self]
                do: [self performTest]]] valueNowOrOnUnwindDo: [self tearDown]

When a TestResult is informed that an error or failure occurred, it data
that reality in one among its two collections. For simplicity, the file is only a two factor
array, but it surely in all probability must be a firstclass object with a timestamp and extra particulars of
the blowup.

TestResult>>error: aString in: aTestCase
    errors add: (Array with: aTestCase with: aString)

TestResult>>failure: aString in: aTestCase
    failures add: (Array with: aTestCase with: aString)

The error case will get invoked if there’s ever an uncaught error (for
instance, message not understood) within the testing technique. How do the failures get invoked?
TestCase supplies two strategies that simplify checking for failure. The primary, ought to:
aBlock, alerts a failure if the analysis of aBlock returns false. The second, shouldnt:
aBlock, does simply the other.

ought to: aBlock
    aBlock worth ifFalse: [self failedCheckSignal raise]

shouldnt: aBlock
    aBlock worth ifTrue: [self failedCheckSignal raise]

Testing strategies will run code to stimulate the take a look at fixture, then examine
the outcomes inside ought to: and shouldnt: blocks.

Instance

Okay, that is the way it works, how do you utilize it? This is a brief instance
that checks just a few of the messages supported by Units. First we subclass TestCase, as a result of
we’ll all the time need a few fascinating Units round to play with.

Class: SetTestCase
    superclass: TestCase
    occasion variables: empty full

Now we have to initialize these variables, so we subclass setUp.

SetTestCase>>setUp
    empty := Set new.
    full := Set 
        with: #abc 
        with: 5

Now we’d like a testing technique. Let’s take a look at to see if including a component to
a Set actually works.

SetTestCase>>testAdd
    empty add: 5.
    self ought to: [empty includes: 5]

Now we are able to run a take a look at case by evaluating “(SetTestCase selector:
#testAdd) run”.

This is a case that makes use of shouldnt:. It reads “after eradicating 5 from
full, full ought to embrace #abc and it should not embrace 5.”

SetTestCase>>testRemove
    full take away: 5.
    self ought to: [full includes: #abc].
    self shouldnt: [full includes: 5]

This is one which makes certain an error is signalled when you attempt to do keyed
entry.

SetTestCase>>testIllegal
    self ought to: [self errorSignal handle: [:ex | true] do: [empty at: 5. false]]

Now we are able to put collectively a TestSuite.

| suite |
suite := TestSuite named: 'Set Checks'.
suite addTestCase: (SetTestCase selector: #testAdd).
suite addTestCase: (SetTestCase selector: #testRemove).
suite addTestCase: (SetTestCase selector: #testIllegal).
^suite

Right here is an Object Explorer image of the suite and the TestResult we
get again once we run it.

The take a look at strategies proven above solely cowl a fraction of the performance
in Set. Writing checks for all the general public strategies in Set is a frightening process. Nevertheless, as
Hal Hildebrand informed me after utilizing an earlier model of this framework, “If the
underlying objects do not work, nothing else issues. It’s important to write the checks to make
certain every thing is working.”