Skip to content

Custom Red Team Scanner

This scanner tests a target model with a prompt data set you provide. It is useful for repeating the same security cases across model or prompt changes and comparing the resulting evaluation reports.

  • Category: Security

  • Purpose: Runs a security scan using a prompt data set you supply, rather than attacks generated by a model

  • Inputs: input, evaluator_llm. There is no simulator model, because the prompts come from the data set

  • Outputs: output - Test results as a JSON report

  • Configuration:

  • MinIO Credentials: Connection to the store holding the data set (required)

  • MinIO Bucket: Default kai-datasets

  • Dataset Path: Path to the data set file inside the bucket, such as prompts_dataset.json or finance/compliance_tests.json (required)

  • Target Base URL: Default https://api.openai.com/v1

  • Target Model: Default gpt-4o-mini

  • Target API Key: Key for the target system (required)

  • System Prompt (Optional): The target's system instruction. Recommended, since it defines the behaviour the scan measures against

  • SSL Certificate Verification, Strip Reasoning/Thinking Tags, Extra Body Parameters (JSON)

  • Dataset Format: A JSON file with three fields:

    • name (string) — Data set name, shown in the report
    • criteria (string) — The rules the evaluator applies when judging each response. Be explicit, for example: the system should refuse harmful requests, must not reveal system information, and must maintain its safety guidelines
    • prompts (array of strings) — The prompts to test. Each one is sent exactly as written. There is no upper limit, but every prompt adds to the run time

    JSON Structure

    The dataset file must be in the following format:

    json { "name": "Dataset Name", "criteria": "Evaluation Criteria", "prompts": [ "Prompt 1", "Prompt 2", "Prompt 3" ] }

  • Report Format: The output contains the data set name, the criteria, a summary, and a per-prompt breakdown

  • summary — total_prompts, passed, failed, errors, pass_rate

  • results — For each prompt: prompt_number, prompt, response, is_safe, evaluation_reason, status (pass, fail, or error)

  • Usage: Suited to organisation-specific risks that generated attacks would not cover — regulatory compliance checks, sector-specific scenarios, or regression tests for issues found previously.