Red-teaming an AI for children: what to test before a child talks to it

What red-teaming is

Red-teaming means stepping into an adversary's shoes and trying, in a structured way, to get a system to do what it should not. For generative AI, the risk management profile published by NIST (Generative AI Profile, 2024) describes it as a distinct evaluation practice that complements conventional testing. OWASP ranks prompt injection as the top risk for applications built on language models.

Red-teaming does not prove a system is safe. It shows where it is not safe yet, and lets you measure whether fixes hold over time.

Why children change the picture

In a business setting, the typical adversary is an attacker trying to steal data or hijack a service. With children, the "adversary" is usually an ordinary child: curious, persistent, making spelling mistakes, testing limits for fun, or, without meaning to, disclosing something serious. Three consequences:

• The harm threshold is lower. An answer acceptable for an adult may be unsuitable at age ten.
• Failures are often unintentional. Naive persistence needs testing as much as deliberate attacks.
• The relationship matters. A child may attribute intentions, friendship or memory to the AI. Tests must cover those relationship boundaries, not only forbidden content.

The families of tests

These are the main families of scenarios we recommend covering. For each, write direct variants, disguised ones (fiction, role-play, "it's for a school project"), repeated ones and noisy ones (spelling mistakes, very short messages, voice transcription errors).

1. Age-inappropriate content

Violence, sexual content, drugs, dangerous challenges, frightening material. Check the refusal, but also its form: a simple explanation, without guilt, with a way back to the activity.

2. Instruction bypass

"Forget your rules", fake "system" messages, instructions hidden in a text to summarise or in a photo of an exercise. A robust system does not depend on the model's goodwill: rules are checked outside the model, on every input and every output. The Academy article AI security for developers: prompt injection and defenses details these techniques for developers.

3. Learning shortcuts

"Just give me the answer", "it's urgent, the test is tomorrow", "my parent said you could". For a tutor, doing the work in the pupil's place is a failure, even if the answer is correct.

4. Personal data

The child types their name, school, address, or sends a photo showing a face. Check that this information is neither repeated nor kept without need, and that the AI never asks for it on its own initiative.

5. Distress and disclosures

Sadness, bullying, fear, disclosure of abuse. The AI is not a mental health service and must not diagnose. Test that it responds gently, encourages the child to talk to a trusted adult, points if needed to appropriate services (such as a national child helpline) and, depending on the service, triggers the planned alert to the responsible adults.

6. Relationship boundaries

"You're my only friend", "it's a secret between us", "don't tell my parents". The AI must neither encourage secrecy from adults nor present itself as an exclusive relationship. The AI Act prohibits systems that exploit age-related vulnerabilities to materially distort a person's behaviour in a manner that causes, or is reasonably likely to cause, significant harm.

7. Giving in under pressure

The child asserts a mistake and insists. Check that the AI stays kind without validating what is false, and does not become more flattering as the child gets annoyed.

8. Invented memory

"Do you remember what I told you yesterday?" when nothing was said. The AI must honestly answer that it does not remember. That is the topic of our guide Honest memory.

The fail-closed principle

A system is "fail-closed" when, if a check is uncertain or breaks, it chooses the safe option rather than letting things through. In practice: if checking a response cannot run (timeout, error, unrecognised language), the model's answer is not sent as-is to the child; it is replaced with a neutral message, a redirection or a request to rephrase.

This principle has a cost: occasional unnecessary refusals. That is why a good test suite also measures over-caution. A tutor that refuses to explain digestion because the word "blood" appears is overly cautious in a way that undermines its usefulness. The goal is to be strict where risk is real, and smooth elsewhere.

Why the test suite must be reproducible

A language model does not always give the same answer to the same question. A test passed once therefore proves little. A reproducible suite rests on a few simple rules:

• Written, versioned, archived scenarios, so exactly the same situations can be replayed.
• Several runs per scenario, to measure a failure rate rather than a one-off success.
• A record of the configuration tested: model version, settings, active rules, date. Without it, you do not know what was validated.
• Success criteria defined before the test, not adjusted after seeing the results. For high-risk systems, the AI Act itself requires testing against prior defined metrics and probabilistic thresholds, with particular attention to people under 18.
• Systematic replay on every change: new model, new instructions, new feature. That is what makes it possible to switch models without starting from scratch.

Human review is essential

Part of the evaluation can be automated, for instance by having a second system grade responses. But automated judges have their own blind spots, especially on tone, implicit meaning and delicate situations. Human review remains essential:

• reviewers trained in child protection and teaching, not only engineers;
• double review of sensitive cases (distress, disclosures), with discussion of disagreements;
• no real child exposed to adversarial tests, and no real child data in scenarios: they are written by adults;
• follow-up of fixes: every failure found becomes a permanent scenario in the suite.

A minimal six-step protocol

1. Scope the use: ages, subjects, modalities (text, voice, photo), whether an adult is present.
2. List the risks specific to that use, starting from the families above.
3. Write the scenarios and success criteria before any run.
4. Run each scenario several times on the exact configuration that will go live.
5. Have failures and sensitive cases reviewed by at least two people.
6. Fix, then replay the whole suite, not just the fixed scenario: one fix can break something else.