Chapter 7 - Software Quality and Software Quality Assurance (QA)

What is Quality?

• Quality is the characteristic that distinguishes the grade of excellence or superiority of a process, product, or service
• In general usage, quality means different things
• The meaning of quality varies considerably across specific disciplines and applications

Quality Trilogy

Three interconnected pillars of quality management:

Quality Planning

• Identify the customer
• Determine the customers' needs
• Develop product features
• Establish quality goals
• Develop a process
• Prove process capability

Quality Control

• Choose control subjects
• Choose units of measurement
• Establish measurement
• Establish standards for performance
• Measure actual performance
• Interpret the difference (actual vs standard)
• Take action on the difference

Quality Improvement

• Prove the need for improvement
• Identify specific projects
• Organize for diagnosis
• Provide remedies
• Prove that the remedies are effective under operating conditions
• Provide for control to hold gains

Analogy: Think of building an iOS app — Quality Planning = designing your feature spec, Quality Control = running unit tests against that spec, Quality Improvement = analyzing crash reports and fixing root causes. Same loop.

ISO 9126 Quality Model (1991)

Six software quality characteristics arranged in a hexagon:

• Functionality — Are the required functions available in the software?
• Reliability — How reliable is the software?
• Usability — Is the software easy to use?
• Efficiency — How efficient is the software?
• Maintainability — How easy is it to modify the software?
• Portability — How easy is it to transfer the software to another environment?

(Image — ISO/IEC 9126 hexagon diagram)

Analogy: Like the six sides of a die — all sides must be balanced for the product to "roll" well in production.

Software Quality Assurance (SQA)

Key Principles

• Conformance to requirements is the foundation from which software quality is measured
• Specified standards define the development criteria used to guide how software is engineered
• Software must conform to:
  • Explicit requirements (stated functional requirements)
  • Implicit requirements (ease of use, maintainability, reliability, etc.)

Quality Concepts

• Quality of Conformance — degree to which design specifications are followed in manufacturing the product
• Quality Control — series of inspections, reviews, and tests used to ensure conformance of a work product to its specifications
• Quality Assurance — auditing and reporting procedures used to provide management with data needed to make proactive decisions

Analogy: QC = catching a bug before it ships; QA = the process that makes sure bugs are systematically prevented in the first place.

Variation Control

• Variation control is the heart of quality control
• Software engineers strive to control:
  • The process applied
  • Resources expended
  • End product quality attributes

Quality Costs

Analogy: Prevention is writing tests before you code; External failure is a 1-star App Store review because something broke in production.

SQA Group Activities

1. Prepare SQA plan for the project
2. Participate in development of the project's software process description
3. Review software engineering activities to verify compliance with the defined software process
4. Audit designated software work products to verify compliance
5. Ensure deviations in software or work products are documented and handled according to a documented procedure
6. Record noncompliance evidence and report it to management

Software Reviews

• Purpose: find defects (errors) before they are passed on to another activity or released to the customer
• Software engineers conduct Formal Technical Reviews (FTR) — also called walkthroughs or inspections
• FTRs are an effective means for improving software quality

Review Roles

Formal Technical Reviews — Guidelines

• Involves 3 to 5 people (including reviewers)
• Advance preparation: no more than 2 hours per person
• Duration: less than 2 hours
• Focus is on a discrete work product
• The product is under review, not the producer
• Producer walks reviewers through the product
• Recorder writes down significant issues
• Reviewers decide to accept or reject the work product

Why Do Peer Reviews?

• Improve quality
• Catches ~80% of all errors if done properly
• Catches both coding errors and design errors
• Enforces organizational standards
• Provides training and insurance for the team

Formality and Timing

• Formal reviews resemble conference presentations
• Informal reviews are less detailed but equally correct
• Early reviews tend to be informal (may lack enough info)
• Late reviews tend to be more formal (but feedback may come too late to avoid rework)

When to Perform Reviews

• When analysis is complete
• When design is complete
• After first compilation
• After first test run
• After all test runs
• Any time you complete an activity that produces a complete work product

Review Guidelines

• Keep it short (< 30 minutes)
• Don't schedule two reviews in a row
• Don't review product fragments
• Use standards to avoid style disagreements
• Let the coordinator run the meeting and maintain order

Formal SQA Approaches

Two main formal approaches:

1. Proof of correctness
2. Statistical quality assurance

Statistical Quality Assurance

• Information about software defects is collected and categorized
• Each defect is traced back to its cause
• Apply the Pareto Principle: 80% of the defects can be traced to 20% of the causes — isolate the "vital few" defect causes
• Move to correct the problems that caused the defects

Analogy: Like App Store crash analytics — you fix the top 20% of crash types that account for 80% of all crashes first.

Software Reliability

• Definition: The probability of failure-free operation of a computer program in a specified environment for a specified time period
• Can be measured directly and estimated using historical and developmental data
• Software reliability problems can usually be traced back to errors in design or implementation

Reliability Metrics

Probability of Failure on Demand (POFOD)

• Example: POFOD = 0.001 → for one in every 1000 requests, the service fails per time unit

Rate of Fault Occurrence (ROCOF)

• Example: ROCOF = 0.02 → two failures for each 100 operational time units

Mean Time to Failure (MTTF)

• Average time between observed failures (also called MTBF — Mean Time Between Failure)

Availability Formula: $\boxed{Availability = \frac{MTBF}{MTBF + MTTR}}$

Where:

• $MTBF$ = Mean Time Between Failure
• $MTTR$ = Mean Time to Repair

Reliability Formula: $\boxed{Reliability = \frac{MTBF}{1 + MTBF}}$

Time Units for Reliability Measurement

Software Safety

• SQA activity that focuses on identifying potential hazards that may cause a software system to fail
• Early identification allows developers to specify design features to eliminate or control potential hazards
• Note: Reliability = likelihood of failure; Safety = impact/consequences of failure

Validation Perspectives

Validation Techniques

Static Techniques

• Design reviews and program inspections
• Mathematical arguments and proof

Dynamic Techniques

• Statistical testing
• Scenario-based testing
• Run-time checking

Process Validation

• SE processes should minimize chances of introducing system defects

Static Validation Techniques

• Concerned with analysis of documentation
• Focus on finding system errors and identifying potential problems
• May use structured arguments and mathematical proofs

Static Safety Validation

• Demonstrating safety by testing alone is difficult
• Testing all possible operational situations is impossible
• Normal correctness reviews may be supplemented with safety-specific techniques

Safety Reviews

• Are the intended system functions correct?
• Is structure maintainable and understandable?
• Verify algorithm and data structure design against specification
• Check code consistency with algorithm and data structure design
• Review adequacy of system testing

Hazard-Driven Analysis

• Effective safety assurance relies on hazard identification
• Safety can be assured by:
  • Hazard avoidance
  • Accident avoidance
  • Protection systems
• Safety reviews should demonstrate one or more of these techniques have been applied to all identified hazards

SQA Plan

Section 1 — Management

• Describes the place of SQA in the structure of the organization

Section 2 — Documentation

• Describes each work product produced as part of the software process

Section 3 — Standards, Practices, and Conventions

• Lists all applicable standards/practices applied during the software process
• Any metrics to be collected as part of the software engineering work

Section 4 — Reviews and Audits

• Provides an overview of the approach used in reviews and audits conducted during the project

Section 5 — Test

• References the test plan and procedure document
• Defines test record keeping requirements

Section 6 — Problem Reporting and Corrective Action

• Defines procedures for reporting, tracking, and resolving errors or defects
• Identifies organizational responsibilities for these activities

Section 7 — Other

• Tools, SQA methods, change control, record keeping, training, and risk management

Standards for Software Process Improvement

CMMI, TSP, PSP Overview

(Image — CMMI/TSP/PSP umbrella diagram)

Personal Software Process (PSP)

• PSP-trained engineers learn process discipline using:
  • Defined methods to do their work
  • Data to plan and manage their work

Essential Elements of PSP

• Performance measures
• Estimating and planning skills
• Quality management skills

PSP Process Flow

(Image — PSP Process Flow diagram)

Steps in order:

1. Planning
2. Design
3. Design Review
4. Code
5. Code Review
6. Compile
7. Test
8. Postmortem → Finished Product

Supporting artifacts: Scripts (guide), Logs (Time & Defects), Results → Plan Summary → Project and Process Data Summary Report

PSP Process Script — Phases

(Image — PSP Process Script table)

Inputs Required:

• Problem description
• PSP Project Plan Summary form
• Historical estimated and actual size and time data
• Time and Defect Recording Logs
• Defect Type Standard

Exit Criteria:

• Thoroughly tested program
• Completed Project Plan Summary
• Completed design templates
• Completed Design Review & Code Review Checklists
• Completed Test Report Template
• Complete PIP forms
• Completed Defect and Time Recording Logs

(Image — Project Plan Summary form)

TSP (Team Software Process)

• Developed to address the need for software engineering teams who can build quality products within cost and schedule constraints
• Key goals:
  • Building teams quickly and reliably
  • Optimizing team performance throughout a project
  • Accelerating software process improvement
  • Making use of mature processes normal and expected
• TSP ≈ CMM/CMMI maturity level 5 process for teams

TSP Strategy Rests on PSP

(Image — TSP Strategy diagram)

TSP improves performance from the bottom-up:

CMMI (Capability Maturity Model Integration)

What is CMMI?

• A collection of characteristics of effective processes that provides guidance for improving an organization's processes
• Covers management of development, acquisition, and maintenance of products or services
• Helps an organization:
  • Examine the effectiveness of its processes
  • Establish priorities for improvement
  • Implement these improvements

Goal: Improving processes for better products

Evolution of Process Capability (5 Maturity Levels)

(Image — Maturity levels diagram)

Analogy: Like iOS developer tiers — Level 1 is "works on my machine," Level 5 is Apple's own release pipeline with automated regression testing, statistical process monitoring, and continuous integration.

CMMI Models (Three Constellations)

(Image — CMMI three constellations Venn diagram)

All three share 16 Core Process Areas:

CMMI-DEV Process Areas by Maturity Level

CMMI-DEV Process Areas by Category

Process Management

• Organizational Innovation and Deployment (OID)
• Organizational Process Definition (OPD)
• Organizational Process Focus (OPF)
• Organizational Process Performance (OPP)
• Organizational Training (OT)

Support

• Causal Analysis and Resolution (CAR)
• Configuration Management (CM)
• Decision Analysis and Resolution (DAR)
• Measurement and Analysis (MA)
• Process and Product Quality Assurance (PPQA)

Project Management

• Integrated Project Management (IPM)
• Project Monitoring and Control (PMC)
• Project Planning (PP)
• Quantitative Project Management (QPM)
• Requirements Management (REQM)
• Risk Management (RSKM)
• Supplier Agreement Management (SAM)

Engineering

• Product Integration (PI)
• Requirements Development (RD)
• Technical Solution (TS)
• Validation (VAL)
• Verification (VER)

Process Area (PA) Components

(Image — Process Area Components diagram)

Each Process Area contains:

Required Components (must be present for satisfaction):

• Specific Goals (SG)
• Generic Goals (GG)

Expected Components (describe what is typically done):

• Specific Practices (SP)
• Generic Practices (GP)

Informative Components (provide guidance):

• Purpose Statement
• Introductory Notes
• Related Process Areas
• Example Work Products
• Subpractices
• Generic Practice Elaborations

CMMI Model Structure

(Image — CMMI pyramid structure diagram)

From bottom to top:

1. Institutionalization (foundation)
   • Policies, Plans, Resources, Responsibilities, Training, Managing Configurations, Stakeholder Involvement, Monitoring and Control, Objective Evaluation, Management Visibility, Defined Process, Improvement Information
2. CMMI Model Foundation — Core Process Areas (shared by all constellations)
3. Constellation-specific PAs (CMMI-DEV / CMMI-SVC / CMMI-ACQ)
4. Benchmark Ratings (Goals, Process Areas, Maturity Levels, Capability Levels)

Analogy: The pyramid is like a codebase architecture — the Institutionalization layer is your AppDelegate and project config (infrastructure), the Core PAs are your shared frameworks, the constellation-specific PAs are your feature modules, and Benchmark Ratings are your test coverage reports and App Store metrics at the top.

End of CSS323 Software Quality Assurance Notes