2. Introduction

6. Approach

The implementation manager will perform the daily build and daily smoke test.

The test group will perform integration tests and system tests (both functional and non-functional). (Mynatt Ch 7).

The test group will perform automated regression testing.  (Mynatt Ch 7, Pressman Ch 17).

6.1 Unit Testing

The concept of unit testing is that a unit is tested in isolation, independently of the rest of the system, and as such requires the creation of a test driver. In CPE 309 these test drivers will be written as JUnit test cases.  (There is a training on JUnit in the  Training Plan.) The unit tests must demonstrate that every method in the class works correctly. All test driver code must be placed in the team source code repository when the unit is submitted to the project build.  Here is a tutorial on setting up the source code structure for testing units in isolation.  (This will be required for Stage 2).

JUnit is not for GUI testing.

JUnit test code does not have to follow the class coding standard to the letter.  However, the instructor will read the test code so it needs to be readable.  In particular, the purpose of each test case must be documented, either as a method comment or as a JUnit test message.  Be sure to cross reference any tests that were created in response to a defect report to the defect number.  Also, here are the recommended tips on JUnit style.

As a simple check that developers are writing thorough unit tests, the QA person may require that unit tests demonstrate that a certain level of statement coverage is achieved before they are allowed to be checked in. (See 6.7).  It would be logical to require coverage levels that are the same as those specified in the Release Criteria. One way to do this is to require that the developer post the Emma coverage report in a designated topic on the team Forum when they commit their code to the team repository.  Even more rigorous would be to require that the test manager inspects the unit tests before a new module can be checked in.

Also, don't forget that McConnell recommends Source Code Tracing - step through every line of code with an interactive debugger and verify all values.  The QA manager may choose to require tracing documentation from each developer.

Unit Testing for Methods

1. Verify operation at representative parameter values (black box - test derived via Equivalence Partitioning)
2. Verify operation at limit parameter values (black box - Boundary Value Analysis)
3. Verify operation outside parameter values (black box - Boundary Value Analysis)

1. Verify preconditions and postconditions via assertion testing (if your team is DBC).
   
2. Ensure that all instructions execute (white box - statement coverage)
3. Check both sides of all branches (including loops) (white box - branch testing)
4. Check all paths in the basis set (basis path testing).  Each developer is responsible for writing formal basis path test case documentation for at least one important unit. The documentation can be in the form of the Basis Path Testing Worksheet. Append the JUnit source code which implements the test and test output
   

1. Check the use of all called objects
2. Verify the handling of all data structures
3. Verify the handling of all files
4. Verify control flow invariants via assertion testing
5. Check normal termination of all loops
   
6. Check abnormal termination of all loops
7. Check normal termination of all recursions
8. Check abnormal termination of all recursions
9. Verify the handling of all error conditions
10. Check timing and synchronization
11. Verify all hardware dependencies

Unit Testing for Classes

1. Exercise methods in combination
• 2-5, usually
• choose most common sequences first
• include sequences likely to cause defects
• requires hand-computing the resulting attribute values
2. Focus unit tests on each attribute
• initialize, then execute method sequences that affect it
3. Verify that each class invariant is unchanged
• verify that the invariant is true with initial values
• execute a sequence (e.g., the same as in 1.)
• verify that the invariant is still true
4. Verify that objects transition among expected states
• plan the state / transition event sequence
• set up the object in the initial state by setting variables
• provide first event & check that transition occurred . etc.

Unit Test Results

6.2 Daily Smoke Tests

It is recommended that there be some convention or mechanism to "freeze" the source code repository during daily build and smoke test so that new code doesn't get committed in the middle of a build. For example, the team might set a deadline of 6pm for all new code to be commited to the repository.  Then the daily build is run every day at 7pm and after the build report and smoke test report is posted the repository is "unfrozen." (See Braude fig 9.21).

6.2.1 Failed Smoke Test Reporting

6.3 Integration Testing

• What components will be available for testing in each build increment?
• What drivers, stubs, and fakes must be available at what time in order to test each build increment?
• What new test cases (in addition to unit tests) are needed to test the interactions between the components?

Each team decides who is responsible for writing and running the integration tests. Obvious alternatives are a) the developer who developed the new code or b) the Test Manager or c) the developer writes the tests and the Test Manager runs them. The instructor recommends option B.

Describe your team's decision here.

6.4 System Testing

Follow these guidelines for writing system test cases.  The tests must be evenly assigned to all developers.  Each developer must submit their test cases as part of their individual project grade.

It is suggested but not required that the functional System Tests be automated as they will be run at least once a week.  An automated test case is called a test "script."  Follow these directions for writing system test scripts.  Automation of GUI elements of your functional system tests can be achieved with the instructor's GUI Robot tool or another tool such as Abbot.  It his highly recommended that you follow a thin GUI testing strategy.

Non-functional requirements may be verified using manual test methods.

Randomness
If your system has a non-deterministic component, such as dealing cards in random order, that part has to be isolated and tested manually.
For example, have the Deck class contain a shuffle() method and a nonShuffle() method. For automated testing, have a command line argument --noshuffle that runs the game calling the nonShuffle() method to deal the cards in a known order. The expected results can then be determined.
The random game is tested manually and observed to see that cards are presented in an unpredictable order.

6.4.1 System Test Results