CPE 309 Software Quality Assurance Plan
[Test Plan] [Standards]
[Metrics]
[Defects]
[Tools]
[Integration] [Release]
The table of contents of this SQAP follow that of IEEE standard 730-1989.
1. Purpose
This document describes the plan by which the CPE 309 project
will produce a quality product.
2. Referenced documents
Project
Plan
3. Management
3.1 Organization
The team will designate an individual "quality assurance leader," named
in section 3.3. The course may also have a QA "guru" who
coordinates QA activities between teams.
3.2 Tasks
QA tasks during CPE 309 shall include
- documentation,
- informal, internal QA reviews
- software inspections
- daily build
- defect reporting and tracking
- requirements traceability
- verification (including testing),
- metrics collection and analysis
These tasks are detailed in this document.
3.3 Responsibilities
Each team member is responsible for the quality of his or her
work. Each person should be sure their work conforms to the QA
criteria provided on the course web page.
It is the quality assurance leader's responsibility to see to it
that the tasks in 3.2 are done, and to ensure that the prescriptions in
this document are followed, including scheduling the inspections
specified and the collection of metrics data. The QA leader is
ultimately responsible for the quality of the deliverables.
The Quality Assurance leader for team (insert
name here) _ is (insert
name
here) .
4. Documentation
All the documentation required for this project is described in the
project plan (SPMP).
The GNOME
style guide is a very good reference for writing tips for user
documentation. You might require your analyst to read this
document. Of particular use to QA people is the section on quality
reviews.
5. Standards, practices, conventions and metrics
5.1 Purpose
This section describes the standards, practices, conventions and
metrics to be used for the CPE 309 project. These are intended not only
to ensure quality for CPE 309 project, but also to obtain quantitative
metric data on the SQA process itself. This data is to be used to help
establish the key practices of a CMM
level four organization.
5.2 Standards
- The IEEE standards, with appropriate modifications, are to be
used for documentation. In most cases teams should follow the
templates Dr. Dalbey has prepared for major documents.
- Quality criteria and checklist are to be used for specification
and design documents.
- UML notation is recommended for Analysis and Design documents.
- Detailed Designs must follow the class pseudocode
standard.
- Java source code must adhere to the class coding
standard.
- E-mail message "subject lines" must follow the guidelines in the
project plan.
- Directory names in the team account must begin with a
capital letter.
5.3 Practices
- The document which describes the quality attributes the finished
software must exhibit is the non-functional
requirements section of the SRS. The QA manager should work
with the Analyst to ensure this document is well written and meets the
customer's needs. Also assist the Test Manager in writing tests
for these requirements.
- From a technical perspective, the success of the software depends
on the detailed designs (pseudocode). It's worth putting extra
effort into making these correct and very detailed. Internal
review is required.
- Because delaying quality is expensive, engineers are strongly
encouraged to apply quality precepts while they work, rather than as an
afterthought. "If you don't have time to do it right in the first
place, when are you going to find time to fix it after it breaks?"
- QA is an important function in CPE 309 as part of the student
course grade depends on the quality of the project deliverables.
A
cooperative, team-oriented attitude is crucial.
- Even though the QA person is ultimately responsible for quality,
the QA person is NOT expected to edit and rework every artifact that
s/he receives. Each individual must produce quality work.
The QA job is to assure that the product has achieved the required
level
of quality before it is delivered.
5.4 Conventions
- HTML (or Wiki) is the preferred format for documents that are to
be shared
in computer readable form.
- Guidelines for all written work are given in the syllabus.
- Source code printouts must follow the guidelines given in the
syllabus.
- The web pages on the team web site must be viewable in Internet
Explorer and Firefox.
5.5 Metrics
5.5.1 Individual Metrics
Individuals will track the number of defects found by the inspectors of
their code during formal software
inspections.
(Recommended) Programmers will plan and track their own unit
development work using the Personal Software Process (PSP). Consult the
PSP
script for details on gathering size, cost, and quality data.
During
Stage 1, only size and cost measures will be required. During
Stage 2, quality measures will also be required.
5.5.2 Team Metrics
The QA person is responsible for compiling and reporting the following
metrics, but each person is responsible for gathering their own data.
- Productivity Metrics (Size and Cost)
Actual Lines of Code. Use the LOC procedure. Categorize by
individual and team total. Keep a separate total for the lines of
testing code.
Actual Hours of effort (Summary of Individual hours, categorized by
project phase, as gathered from timecards. Also team total by phase.)
For example:
| Name |
Stage 1 |
Stage 2 |
Total |
| Sue |
55.25 |
47.75 |
113.5 |
| Tom |
63.50 |
54.25 |
128.75 |
| Bill |
50.25 |
56.25 |
115.5 |
| Team Total |
179.00 |
158.25 |
357.75 |
- Defect Metrics
Number of issues (software inspections). Inspections will be
conducted using a formal process and any issues found
will be recorded.
Number of compile defects. That is, when a defect causes
integration failure ("broken build"). This can be during the
official daily build, or unofficially by any developer who compiles the
code from the team repository.
Any time after a module is checked in to the repository, if the system
won't compile it counts as a compile defect. Each of these
defects is entered into the team defect reporting system All
developers must gather
this data.
Number of test defects. Count of defects found during integration and
system testing (not unit testing). Just as above,
once a module is checked in to the team repository, if it causes a
defect during execution, it gets counted as a test defect.
Each of these
defects is entered into the team defect reporting system, so this
measure
can be obtained from the reporting system itself.
Defects per KLOC. Compute as: total test defects * 1000 / Total LOC.
Total Defect Repair time. Repair times are logged for each defect
in the defect reporting system. The total must be computed by
manually summing the all the individual times.
- Software Design Quality Metrics (recommended)
It is recommended that each team use some objective means
to determine the quality of their design.
The QA person should assist the Designer
in gathering objective measures of the
quality of design and code.
There are several free tools that can do this, such
as JDepend.
(Eclipse
example). CheckStyle also provides warnings
for some of the metrics below.
The Designer
is responsible for understanding the meaning and how to interpret each
of these metrics.
The QA Person should supervise the designer
in collecting these metrics every other week
and giving a briefing to the team.
Example metrics: Lines of Code, Comment Ratio, Number of Classes, Lack of Cohesion of
Methods, Cyclomatic Complexity, Response for Class, Weighted Methods
Per
Class, Coupling between Objects, Coupling Factor, Fan Out, Depth of
Inheritance Hierarchy.
- Change Metrics (computed by Change manager)
Number of Formal Change Proposals Submitted.
Number of Formal Change Proposals Accepted.
Number of Uncontrolled changes to SRS. Suggested technique:
Save a copy of the
original baselined SRS. Perform an automated "diff" against the final
SRS. Count each item appearing in the diff for which there isn't
a change request.
Number of Uncontrolled changes to High Level Design (javadocs).
Save a copy of the original baselined javadocs. Perform an automated
"diff"
against the final javadocs. Count each item appearing in the diff
for
which
there isn't a change request.
5.6 Goals and Reporting
[Note to the student: The quality targets used here are based on
historical data obtained from previous student teams on similar
projects. Based on previous team data, these targets are
aggressive but not unrealistic.]
CPE 309 quality goals for delivered products are as follows.
- Software Inspections: no more than one severe and three minor
issues per inspection.
- Compile: no more than one broken build for the entire quarter.
- Integration and System Test: no more than fifty defects per KLOC
(thousand lines of code).
- Acceptance Test: no defects during acceptance testing.
The actual metric data gathered from this project (all the metrics in
5.5.2) are
to be reported in the
final project submission.
at each release.
6. Demos and Reviews
6.1 Purpose
The purpose of demos and reviews is to provide a means of focusing the
attention of engineers on quality of the application as it develops.
- Internal Reviews are a peer review by the QA leader or
another team member of individual work products. Here is a recommended
QA submittal
form.
- Software
Inspections are peer reviews of source code before unit testing.
- Class Demos are presentations of completed work products
or deliverables in front of the entire class.
- Customer Demos are a formal presentation for approval of
the customer.
6.2 Minimum requirements
6.2.1 Internal Reviews
All new detailed designs in Stage 1 and 2 will
undergo internal review.
6.2.2 Software Inspections
All new source code modules for Stage 1 will
undergo internal
review.
All new source code modules for Stage 2 require a
formal code inspection.
6.2.3 Post mortem review
The team will conduct a post-mortem review at the end of CPE 309 to
gather the insights and lessons learned during the course.
7. Test Plan
A software project test plan describes the objectives, scope, approach,
activities, and procedures your team will use for software
verification.
It explains who is responsible for each kind of testing, how it is to
be
carried out, and how it will be documented and tracked.
Here is the Test
Plan to be followed for CPE 309, based on the IEEE standard.
8. Defect reporting, correction,
and tracking
Read McConnell page 129-130.
Defects in the software found during integration and system test (NOT
unit test) are tracked using an automated web-based tool, Trac.
According to
the Integration Procedure (10.1) only unit tested code can be submitted
to the repository. So usually any tests that are run against the
repository codebase are integration or system tests. Any defect
found in such a test must be reported in Trac as a defect ticket.
Defects are corrected using this Defect
Repair Procedure. Make sure all developers follow this
procedure to the letter!
9. Tools,
techniques and methodologies
SQA techniques include the application of standards,
requirements tracing, FTR's and software verification (testing).
The SQA tools consist of form templates and checklists.
- Form Templates are provided on the course web site.
- QA Criteria Checklists for each major deliverable are provided in
the deliverables
list.
9.1 LOC Counting Procedure
As a measure of product size we will use logical lines of code. Do not
include unit test code as part of the product size measure, only code
in
the product itself. Our procedure requires following the class
coding standard and a tool to count lines of code. We do not
include comment lines in our line count. Here are two suggested
tools for you to consider:
9.2 Requirements
Traceability
See McConnell page 151. Each requirement in the specification is traced
to each module(s) in design and code that implements it. Table
format works fine for this.
9.3 Unit Development
Procedure (Recommended)
This procedure describes every step that a developer undertakes in
producing a single module (unit). The Unit
Development Procedure (aka "PSP script") for CPE 309 is a more
elaborate version of Table 14-1 pg 203. (Note that McConnell calls this
an "integration" procedure, but we will call it a Unit Development
Procedure).
This script has been painstakingly prepared for you. Read the
directions completely, carefully, and be sure you
understand every word. There is also a FAQ.
If there is anything in these directions about which you have the
slightest question, ask for clarification! Completing
the PSP forms is recommended, but not required. You are
expected to complete the PSP forms accurately and precisely.
Be sure you understand this procedure and follow it religiously. If
the procedure doesn't work for you, seek the instructor's permission to
modify it. Note this procedure only applies to new code, not to
maintenance tasks.
9.3.1 PSP Submission
Procedure
The PSP forms for a unit you develop are to be submitted to the QA
manager during the first class session after the unit has been
submitted
to the build. The QA manager gathers all the PSP forms and may
publicize which have been completed (but do not publicize individual
data). The QA manager submits all the PSP forms to the instructor
with the Release. (Completing these forms is an individual choice
as described in the previous section).
9.4 Internal Review
Internal reviews are the best way for a team to ensure quality. The
QA person must write the procedure for internal review and place it
here. At a minimum it requires review of the work product by one
other team member and documentation using a form such as this review
form. The QA leader may create a scoring mechanism,
checklists, or other artifacts that seem helpful for the team.
The
reviews may be staff from a different team. The QA leader may
choose to do all the reviews himself (or herself). Be sure to
work
with the team manager to allocate time for review meetings.
9.5 Software Code Inspections
All new modules for Stage 2 require a code inspection.
Read these guidelines to Software
Inspections for Software Engineering Student Teams.
Here is the required CPE 309 Software
Inspection Process.
9.5.1 Software Inspection
Report
A summary report of results from the software inspections is to be
posted on the team web page under the link "Software Inspection Report"
9.6 Design By Contract
(Recommended)
Design by Contract is a module specification technique that intends
to improve design reliability and reduce development costs. After
completing the homework activities and participating in the class
discussion about DBC, each team must decide which specification style
they choose to adopt: Design by Contract or Error Checking. For
the second stage of development, all modules must be specified in
the style the team has chosen. It is the QA manager's job to
enforce conformance to this style.
On (decision date) our
team decided to adopt the following module specification style (check
one):
___ Design by Contract
___ Error Checking
10. Integration Plan
This section is now a separate document,
the Integration Plan.
11. Software Release
Release Criteria
These criteria dictate when the software can be released. Pay
attention to these; the software can't be released until these criteria
are met (regardless of the planned schedule). Discussed in Chapter
16. Here are the
CPE
309 release criteria. (negotiable until end of week 3).
Release Checklist
This is a list of all tasks that must be completed before the product
can be delivered to the customer. Discussed in Pfleeger Chapter
10
and McConnell Chapter 16. See McConnell's template.
It is the
responsibility of the QA mgr to create the release
checklist and to supervise related tasks.
13. Records collection, maintenance & retention
The SQA records collected and archived shall include ...
Individual Status Reports- Individual PSP forms (if collected)
- Internal Review Forms
- Software Inspection Reports
- The Current Defect List
- Release Checklist for each Release (and associated documentation)
14. Training
The only SQA training planned is class lectures. If additional training
seems necessary, describe it here.
Change History
3/31/09
|
JD
|
Moved section 10 to Integration Plan. Moved
deployment section to Integration Plan.
|
1/6/09
|
JD
|
Revised integration procedure for Winter 2009.
|
4/23/08
|
JD
|
Changed metrics report (Sec 5.6) delivery from
final report to each release. |
5/18/04
|
JD
|
Added testing code to LOC metrics required.
Added Total Defect Repair Time to metrics required.
|
4/13/04
|
JD
|
Added directions for setting up Elementool to
section 8.
|
3/27/04
|
JD
|
Minor editing and additions for Spring 2004
|
2/18/04
|
JD
|
Added section 9.5 on DBC
|
| 1/12/04 |
JD
|
Revised for Winter 2004. PSP forms made
recommended, not required. Revised quality goals. Added
more
explanation to metrics. |
| 4/8/03 |
JD
|
Changed "teamatic" to "elementool" |
Home