HomeAcceptance testing guide

Guide · Counting acceptance testing

Measuring the accuracy of a passenger counting system.

Before accepting an automatic passenger counting system, the transport authority and the operator compare its results against a reference. Here is a six-step method, the indicators to calculate and a reusable acceptance grid.

Request a demo
© Shixart1985 · CC BY 2.0

Why acceptance testing

Counting data that drives decisions.

Counts are used to size the service, track the targets of a public service contract and sometimes to calculate revenue or penalties. Documented acceptance testing sets out, before deployment, what is measured, how it is measured and the expected level.

The method

Six steps, from reference to report.

  1. 1

    Choose the reference

    Manual counts by staff on board, or frame-by-frame video review; the reference must be independent of the system under test.

  2. 2

    Define the sample

    Representative vehicles, lines and periods: peak and off-peak hours, school services, weekends; aim for at least 1,000 passages in each direction.

  3. 3

    Cover the difficult cases

    Heavy crowding, double doors, pushchairs, bicycles, wheelchairs, children, night-time and low light.

  4. 4

    Align the data

    Same timestamps, same breakdown by door, stop and direction for the reference and the system.

  5. 5

    Calculate the differences

    Overall bias, absolute error, results by door, by direction and by load band (see the indicators below).

  6. 6

    Report

    Results, test conditions, missing data, uncounted trips and excluded cases, with their justification.

The indicators

Four calculations to assess a system.

IndicatorCalculationWhat it reveals
Overall bias(system total − reference total) ÷ reference totalA tendency to overcount (positive) or undercount (negative)
Absolute errorΣ |system − reference| ÷ Σ reference, by stop and by doorDifferences that cancel out in the total but distort load and origin-destination data
Reliability ratesystem total ÷ reference totalThe indicator often quoted in annual reports (example below)
Share of trips countedtrips with valid data ÷ trips operatedData completeness, essential for extrapolating ridership

A reliability rate close to 100% can hide errors that cancel each other out: always calculate the absolute error as well as the total.

The acceptance grid

A grid to include in your specifications.

CheckMethodAcceptance thresholdResult
Overall accuracyBias and absolute error across the whole sampleTo be set in the specifications…
By door and directionSame calculations for each door, for boardings and alightingsTo be set in the specifications…
By load bandLow, medium and heavy crowding, analysed separatelyTo be set in the specifications…
Difficult casesPushchairs, bicycles, wheelchairs, night-time, double doorsTo be set in the specifications…
CompletenessShare of trips counted over the periodTo be set in the specifications…
ConsistencyDifference between cumulative boardings and alightings at the end of the tripTo be set in the specifications…

Benchmark: the accuracy of the 2FLOW solution is certified above 98%, measured in a controlled environment on at least 1,000 passages in each direction.

Published example

A real acceptance test on a suburban network.

Niort Agglo · Tanlib network

368 passengers counted by WEBREATHE against 371 counted by hand

99.2%published reliability rate (368 ÷ 371)

The network's 2023 annual report publishes an example of a cross-check against manual counts carried out in week 38, on a vehicle serving suburban lines 21, 22 and 24 and a school service. For full acceptance testing, this kind of example should be complemented by the absolute error, several vehicles and several periods.

  • Reference: manual count on board
  • Sample: one vehicle, one week
  • Checks required by the contract: two campaigns per year cross-checking against manual counts

Go further

Prepare your project.

Let's prepare your system's acceptance testing. Our studies office is here to support you.

Request a demo