Software Test Plan
The test plan is based on the outline provided by ANSI/IEEE
Std. 829-1991 but has been streamlined for CPE 206..
2. Introduction
Lucky Strike Software is committed to an effective and thorough testing
effort for our Sokoban game release. Each release will use a more
thorough and involved testing procedure. It is our hope to release
a product which is not only believed to be high in quality, but proven.
6. Approach
For our Stage 2 delivery, Lucky Strike Software will be creating and automating
a full suite of regression system testing for all features promised in
Stages 1 and 2 using the GUIRobot tool. Unit testing will be conducted
by individual developers as they see fit. Daily Smoke Testing will
provide a strong tool to monitor the stability of all newly developed modules
while assuring the quality of each day's build.
During the development of our Stage 3 delivery, automated system tests
for the new features will be added to the existing test suite while continually
updating and improving the Smoke Testing Procedure. In addition,
unit testing will be conducted by each developer using the JUnit test tool
to verify the execution of every line of code while testing boundary values
to ensure correctness.
6.1 Unit Testing
Unit testing is testing of the source code for an individual unit (class
or method) by the developer who wrote it. A unit is to be tested after
initial development, and again after any change or modification. The developer
is responsible for testing that his new code does not break his or her
private build before committing the new source code to change control.
The concept of unit testing is that a unit is tested in isolation, independently
of the rest of the system, and as such requires the creation of a test
driver. All test driver code must be turned over to the test team when
the unit is submitted to the project build. The test team will store all
the units tests in a test library for creation of a regression test suite
(see section 6.5).
During Stage 2, the test driver may be built from scratch, but during
Stage 3, all unit tests will be created with the JUnit unit testing tool.
During Stage 2, unit testing will be carried out informally.
During Stage 3, unit testing will be conducted more formally, as described
next.
Unit Testing for Methods
The following test types are explained in Braude Chapter 8. Items 1 - 6
are required in Stage 3, the remainder are recommended for all stages.
-
Verify operation at representative parameter values
(black box - test derived via Equivalence Partitioning)
-
Verify operation at limit parameter values
(black box - Boundary Value Analysis)
-
Verify operation outside parameter values
(black box - Boundary Value Analysis)
-
Verify preconditions and postconditions via assertion testing
-
Ensure that all instructions execute
(white box - statement coverage)
-
Check all paths, including both sides of all branches (including loops)
(white box - decision coverage)
-
Source Code Tracing
Step through every line of code with an interactive debugger and verify
all values.
-
Check the use of all called objects
-
Verify the handling of all data structures
-
Verify the handling of all files
-
Verify control flow invariants via assertion testing
-
Check normal termination of all loops
(part of a correctness proof)
-
Check abnormal termination of all loops
-
Check normal termination of all recursions
-
Check abnormal termination of all recursions
-
Verify the handling of all error conditions
-
Check timing and synchronization
-
Verify all hardware dependencies
Unit Testing for Classes
Item 1 is required in Stage 3, the remainder are recommended for all stages.
-
Exercise methods in combination
-
2-5, usually
-
choose most common sequences first
-
include sequences likely to cause defects
-
requires hand-computing the resulting attribute values
-
Focus unit tests on each attribute
-
initialize, then execute method sequences that affect it
-
Verify that each class invariant is unchanged
-
verify that the invariant is true with initial values
-
execute a sequence (e.g., the same as in 1.)
-
verify that the invariant still true
-
Verify that objects transition among expected states
-
plan the state / transition event sequence
-
set up the object in the initial state by setting variables
-
provide first event & check that transition occurred . etc.
6.2 Daily Smoke Tests
With every build a smoke test will be run after the build successfully
compiles. The smoke test is designed to ensure that the newly developed
code has not damaged the system and that the system, with the newly integrated
code, is stable enough to be subjected to further integration and system
testing.
A daily smoke
test report will be filled out and posted on the project team web site
for each build. The requirements for the daily smoke test will change throughout
the project with newly implemented features and requirements, so the actual
features tested will be described in the report.. Any defects or missing
functionality will be logged as a defect and the defect tracking number
entered in the daily smoke test report. After the smoke test is complete
and the results posted on the web, the integration team may notify the
test team that the build is available for further testing.
Smoke tests may be manual or automated as determined by the test team.
It is recommended that there be some convention or mechanism to "freeze"
the source code repository during daily build and smoke test so that new
code doesn't get committed in the middle of a build. (See Braude fig 9.21).
6.3 Integration Testing
Integration testing is exercising newly developed code that has been committed
to the build in combination with other code that has already been integrated.
Integration tests are performed while builds are in progress. Braude discusses
integration testing at length in Chapter 9. For our project, integration
tests will emphasize what Braude calls "interface testing."
It is highly recommended to automate integration test cases via the
JUnit test framework.
6.4 System Testing
System testing is testing the functionality of the system as a whole in
order to verify the requirements in the SRS.
Follow these guidelines for writing
system test cases.
We will automate all our functional system tests using the java.awt.Robot
tool. Follow these directions for writing
system test scripts.
Non-functional requirements may be verified using manual test methods.
6.5 Regression Testing
At least once a week, a regression test will be run (see Braude Chapter
9) to verify that recent additions to the system have not caused it to
regress to a less functional state than in a previous build. Regression
testing is not a different type of testing, it doesn't require writing
any new test cases. Instead, it refers more to a schedule for running a
suite of tests that have already passed. The tests that comprise the regression
test suite come from a "test library" of stored tests (including unit,
integration, and system tests).
From a practical perspective, regression testing must be automated as
there are hundreds of tests to be run. Dr. Dalbey has a handout "Automating
Regression Tests" which has many useful tips.
6.6 Validation/Acceptance Testing
Validation testing is ensuring that the system responds and behaves in
a reasonable manner when placed under normal or expected operating conditions
such as those that would be used by an average user. Validation testing
for our project will be in the form of an acceptance test implemented by
the customer. The customer may formalize this testing on his own, or it
may be nothing more than an informal smoke test in which he steps through
what he feels is a normal use case. The customer will report on the results
of his acceptance test with each release. The acceptance of our build by
the customer in this manner is his stamp of approval that all contracted
functionality for a specific release is present and behaving properly.
9. Test deliverables
The test deliverables are :
-
this Test Plan document (may be modified with approval of instructor)
-
Test Matrix
-
Unit Test Cases (Stage 3)
-
Unit Test Results
-
System Test Cases
-
System Test Scripts
-
System Test Results
-
Smoke Test reports
Each of these items has a link on the team home page template.
11. Environmental needs
The hardware necessary to conduct proper testing
are as follows:
-
2 PC-Based computers (>100 MHz). Each computer
must have the ability to establish a network connection and each must have
an attached monitor, keyboard, and mouse.
-
Both computers must be running Windows95 or later with the capabilities
to run a java program and establish a TCP/IP network connection.
-
Lucky Strike Software will use stand-alone testing.
11.1 Test tools
To run our full testing suite, the testing parties must have copies of
the JUnit and GUIRobot source code. JUnit and GUIRobot can be obtained
from the 206 web site under Professor Dalbey's resources link.
12. Responsibilities
No separate list of responsibilities is included here. The test responsibilities
should be included in the Project Plan under Job Descriptions or Responsibilities.
14. Schedule
No separate test schedule is included here. All test milestones should
be included in the project detailed schedule.
CPE 206
Home