From risk to requirement in six questions
Qué es
Turning each identified risk into a control, and each control into AI system requirements that can be verified and validated.
- Who
- The provider of the AI system. Engineering does it, with quality and whoever manages the risks.
- When
- After the risks are identified and evaluated, during design. It is repeated for every version of the system.
- Where
- In the system requirements specification, traced to the risk documentation and to the tests.
- Why
- EN 18286 asks for requirements that are complete, unambiguous and verifiable. A risk without a requirement cannot be tested.
- How
- Risk → control, in the order of prEN 18228. Control → requirement, written in EARS. Requirement → test. Details below.
What EN 18286 asks for
In clause 8.3.2.1, EN 18286 asks the provider to determine the AI system requirements needed to comply with the AI Act. Its example is the requirements that come from risk control. They have to be complete, unambiguous, verifiable and validatable. It then asks for them to be reviewed (8.3.2.2), turned into specifications (8.3.2.3), and verified and validated (8.4).
On what makes a good requirement, the standard refers to two life cycle standards:
- ISO/IEC/IEEE 15288: the life cycle processes of a system, from concept to retirement.
- ISO/IEC/IEEE 29148: requirements engineering within that life cycle.
It takes the AI system life cycle model from ISO/IEC 22989, clause 6. The requirements engineering these standards take for granted is in the SWEBOK, the IEEE guide to the software engineering body of knowledge, in its requirements area.
From risk to control
The example follows the CV screening from the risk identification session. The system is fictitious. It ranks the CVs for each vacancy and proposes the top twenty to Recruitment. Chain 2 ended like this: anyone with a career gap of more than a year is ranked lower because of that gap. The only barrier is a sample of ten random CVs that Recruitment opens by hand, and there is no preventive barrier.
prEN 18228, the standard for Article 9, still a draft, ranks controls in three levels:
| Level | What it is | In CV screening |
|---|---|---|
| 1. Inherently safe design | Removes the hazard from the system. It is the preferred control. | The model does not receive the months out of work or the end date of the last job, which brings them back in. |
| 2. Protective measures | Detect or stop the harm when design is not enough. | Each version is measured separately with and without a career gap, and is not released if the difference exceeds the threshold. The system draws the random sample, and it includes CVs with gaps. |
| 3. Information | Instructions and training for whoever uses the system. Never on its own. | The list tells Recruitment that it is a proposal and rejects no one. |
Two points from the standard. An internal procedure does not count as a control: the sample Recruitment opens by hand becomes one when the system draws it and records it. And each control is reviewed in case it creates a new hazard, like the end date of the last job, which brings the gap back in.
From control to requirement, with EARS
EARS (Easy Approach to Requirements Syntax) is a way of writing requirements with five patterns. Alistair Mavin and others published it in 2009, at Rolls-Royce. The first author keeps an open guide. Froga writes its own requirements this way and asks the same of the requirements its users write.
| Pattern | Form |
|---|---|
| Ubiquitous | The <system> shall <response>. |
| State-driven | While <state>, the <system> shall <response>. |
| Event-driven | When <trigger>, the <system> shall <response>. |
| Unwanted behaviour | If <unwanted condition>, then the <system> shall <response>. |
| Optional feature | Where <feature is present>, the <system> shall <response>. |
First, a term: the gap disparity is the difference between CVs with and without a career gap of more than a year in the share of suitable applicants left out of the top twenty.
The screening controls, written this way:
- Ubiquitous. The screening system shall exclude the months out of work and the end date of the last job from its inputs.
- Event-driven. When a new version is trained, the screening system shall calculate the gap disparity.
- Unwanted behaviour. If the gap disparity exceeds 2 points, then the screening system shall block the release of the version.
- Event-driven. When there are CVs with a career gap of more than a year outside the list, the screening system shall include at least three in the sample.
- State-driven. While Recruitment has not opened the ten CVs in the sample, the screening system shall keep the process open.
- Optional feature. Where the screening system is integrated into the company's recruitment tool, the top-twenty list shall show the notice "Proposal. Rejects no one".
The 2-point threshold is made up for the example. The organisation sets it in its risk acceptance criteria.
Froga's writing rules, which apply to any requirement:
- A subject that cannot be read two ways. Here, always "the screening system".
- One demand per requirement. If two things can fail separately, they are two requirements.
- One sentence, and no reason: the reason goes elsewhere, traced to the risk.
- No "may". The requirement says what the system does, not what it is allowed to do.
- No judgement words. "Appropriate", "fair" or "in time" are replaced by a threshold, a condition or a data point.
- Prohibitions, as unwanted behaviour: "If…, then…".
- The threshold and the deadline, inside the response.
From requirement to test
prEN 18228 asks for two checks per control (9.2): that it is implemented and that it reduces the risk.
| Requirement | Implemented (verification) | Reduces the risk (validation) |
|---|---|---|
| 1 · Excludes the variables | Inspection: the two variables do not reach the model. | Pairs of CVs that are identical except for the gap land in the same position. |
| 2 · Measures the disparity | The disparity report exists for each version. | The broken-down validation data that the session was missing. |
| 3 · Blocks the version | A version with a 5-point disparity, seeded on purpose, is not released. | The disparity stays under the threshold in real processes. |
| 4 · Sample with gaps | In a test process with CVs with gaps outside the list, the sample includes at least three. | The record of the sample per process, also missing. |
| 5 · Sample opened | A process with the sample unopened does not close. | The same record: how many samples are opened in full. |
| 6 · Notice on the list | The notice appears in the tool. | Recruitment opens CVs from outside the list. Checked in the record. |
Each row leaves signed evidence in the technical documentation: the risk, its control, the requirement, the verification and the validation, linked together.
Froga runs that chain: risks, controls, requirements in EARS, verification and validation, signed on top of the tools the team already uses. Froga, from risk to proof.
