Purpose of software testing: A study of how errors, defects, and failures happen, and how we catch them to increase confidence in delivered software. Analogy (car delivery vs software delivery): In car manufacturing, expectations (no damage, correct parts, documentation, pre-sales checks) are treated as baseline requirements. Software delivery often fails to meet expectations—commonly because testing is inadequate. Testing challenges: Testing is not inherently “complex,” but it’s often applied with insufficient rigour, leading to an assurance vacuum. Common issues include: Rigour gap (not done thoroughly enough) Human cost (time/coordination effort) Tooling shortfall (underuse of technology that could speed up testing) Why software fails (five cases): Airbus A380: Wiring/cabling integration problems during assembly. UK tax returns: A privacy/data leakage issue where users could see prior users’ earnings. UK “Top 10 Wanted” website: Performance collapse due to load spikes after widespread publicity. Smartphone mapping app: Data accuracy/cartographic errors (e.g., incorrect placement of a museum). US military breach: Security breach exposing sensitive personnel financial and identity data. Root causes: Designers make mistakes because of pressures like deadlines, system complexity, organizational complexity, and changing technology. Error → defect → failure model: Error: incorrect thinking/assumption/forgotten detail (invisible) Defect: the error becomes a fault in documentation/specs/design/code Failure: the defect causes the system to fail (not always, but often) Detection pitfalls: Test outcomes can be misleading: False positive (looks broken, actually works) False negative (works “in tests,” but fails in real use because it wasn’t detected) How to handle testing: Start testing early (“begin testing the moment errors begin”) to prevent propagation. Continue until confident, right through to the end of development. What’s at stake: Incorrect software can cause money loss, time loss, reputation damage, injury, or even death—especially in safety-critical systems.

 

Software Engineering / Lecture 01 01 / 14

Introduction
to Software
Testing.

Why systems fail · How we catch them

A study of errors, defects & failures.

◉ EST. 2024 · DEPT. OF CS

The analogy · expectations 02 / 14

If you were buying a new car…

A level of expectation is set the moment you walk in.

You would not expect to take delivery from the showroom with a scratch down the side of the vehicle. The car should have the right number of wheels, a steering wheel, an engine — and a spare for emergencies.

It should come with appropriate documentation, with all pre-sales checks completed and passed satisfactorily. The car you receive should be the car described in the sales literature.

If your expectations are not met, you will feel justifiably aggrieved.

✓Bodywork integrityVISUAL

✓Wheels + spareCOMPONENT

✓Engine specificationPERFORMANCE

✓Colour scheme matchAESTHETIC

✓Ordered extrasCONFIG

✓Fuel consumptionMETRIC

✓DocumentationPAPERWORK

The contrast · delivery expectations 03 / 14

Car delivery test vs. Software delivery

Automotive

Pre-sales
checks
passed.

Brochures, test drives, experience — all set a baseline. Deviation is the exception, not the rule.

Expectation92%

versus

Software

Delivered
not working
— often.

Examples of software being delivered not working as expected, or not working at all, are common. No single cause, but inadequate testing is a major factor.

Confidence34%

The discipline · challenges 04 / 14

Software testing challenges

Software testing is neither complex nor difficult to implement, yet it is a discipline that is seldom applied with anything approaching the necessary rigour to provide confidence in delivered software.

It is costly in human effort, or in the technology that can multiply that effort — yet seldom implemented at a level that provides any assurance software will operate effectively, efficiently or even correctly.

01

Rigour gap. Seldom applied at the level needed to inspire confidence.

02

Human cost. Expensive in people, time, and coordination.

03

Tooling shortfall. Technology that multiplies effort is underused.

04

Assurance vacuum. Rarely does testing guarantee correct operation.

Why software fails · five cases 05 / 14

Case studies in software failure

Why does software fail?

Five real-world examples where inadequate testing or design flaws led to failures of varying scale — from financial loss to national security exposure.

CASE 01

Airbus
A380

Manufacturing

CASE 02

UK Tax
Returns

Privacy

CASE 03

Top 10
Wanted

Performance

CASE 04

Mapping
App

Accuracy

CASE 05

US Military
Breach

Security

Case 01 · Airbus A380 06 / 14

Case Study 01

01

Cabling could not be joined.

Assembly
incompatibility.

After successful test flights and airworthiness accreditation, problems arose in the manufacture of the Airbus A380 aircraft. Assembly of the large sub-parts into the completed aircraft revealed enormous cabling and wiring problems.

The wiring of large sub-parts could not be joined together. The issue was quickly fixed and the aircraft entered into service within 18 months of the cabling difficulties being identified.

Estimated rectification cost

$6.1BILLION

Case 02 · UK Government tax filing 07 / 14

Case Study 02

02

You could see what the previous user earned.

Privacy
breach.

When the UK Government introduced online filing of tax returns, a user could sometimes see the amount that a previous user earned.

This was regardless of the physical location of the two applicants — a session or data leakage defect with serious privacy implications for every citizen using the system.

Failure category

DATALEAKAGE

Case 03 · UK's top 10 wanted 08 / 14

Case Study 03

03

The site was taken offline.

Performance
collapse.

In November 2005, information on the UK's top 10 wanted criminals was displayed on a website. The publication was described in newspapers, on morning radio and television, and as a result many people attempted to access the site.

The performance of the website proved inadequate under this load and it had to be taken offline. The publicity created performance peaks beyond the capacity of the website.

Failure category

LOADSPIKE

Case 04 · Smartphone mapping app 09 / 14

Case Study 04

04

A museum sat in the middle of a river.

Cartographic
errors.

A new smartphone mapping application was introduced in September 2012. Among many other problems, a museum was incorrectly located in the middle of a river, and Sweden's second city, Gothenburg, seemed to have disappeared from at least one map.

A high-profile launch with low-quality data undermined user trust within hours.

Failure category

DATAACCURACY

Case 05 · US military security breach 10 / 14

Case Study 05

05

Perhaps "all" personnel compromised.

Personnel
exposure.

Security breaches at the US military resulted in the payment details of many personnel — perhaps even "all" — being compromised.

The exposed data included names, addresses, email addresses and bank details: the full financial identity of serving members of one of the world's most security-conscious organisations.

Failure category

SECURITYBREACH

Root cause · human fallibility 11 / 14

Do system designers make mistakes?

Of course they do. People are fallible.

To understand what's going on, we need to start at the beginning — with the people who design systems. Pressures bear down on designers and increase the likelihood of defects in specifications, designs and code.

P/01

Deadlines

P/02

System complexity

P/03

Organisational complexity

P/04

Changing technology

Anatomy · error → defect → failure 12 / 14

Error, Defect, Failure

From invisible thought to visible collapse.

STAGE / 01

Error

An incorrect thought, a wrong assumption, or a thing forgotten. Intangible — you cannot touch it. Where major system failures usually begin.

↓

STAGE / 02

Defect

When the error is written down, it becomes a fault or defect in a document. If used to specify a component, the component may be built faulty.

↓

STAGE / 03

Failure

A faulty component built into a system causes the system to fail. Not always guaranteed — but errors in thought tend to lead to faulty components, and faulty components cause system failure.

Detection · false signals & strategy 13 / 14

Not every apparent failure is a real failure

Test results can lie — in both directions.

False Positive

+

Looks broken.
Isn't.

The tester misunderstood what should happen in those precise circumstances. The application or system is actually performing correctly.

False Negative

−

Looks fine.
Isn't.

A real failure exists, but is not identified as such — the test is seen as correct, and the defect slips through into production.

AVOIDANCEBegin testing the moment errors begin — at the start of development. Find faults before they propagate.

RECTIFICATIONContinue until confident no serious failures remain — right through to the end of the development process.

Conclusion · what is at stake 14 / 14

Incorrect software can harm…

People. Companies.
The environment.

— LOSS / 01

Money

— LOSS / 02

Time

— LOSS / 03

Reputation

— LOSS / 04

Injury

— LOSS / 05

Death

A train carrying nuclear waste. An aircraft at altitude. A hospital life support system. Software failures can cause all of these at once.