LawZero is a non-profit building safe-by-design AI systems. We’re building the Scientist AI, an advanced AI system designed from the ground up to be both highly capable and safe. As we develop both general‑purpose Scientist AI models and safety guardrails for frontier LLMs, we need rigorous, independent evaluation of every capability and safety claim we make. We are looking for a Director of Evaluations to build, lead, and grow LawZero’s Evaluations Team.

This is a foundational hire. You will define what world‑class evaluation looks like at LawZero, build the team and infrastructure to deliver it, and ensure that evaluations remain independent of the main research stream so that capability and safety claims can be trusted both internally and externally by the wider AI and AI safety community.

Key responsibilities

Define LawZero’s evaluations strategy and roadmap, prioritising what needs to be measured and when, in close coordination with both research and product teams.
Build up the Evaluations Team during your first 3–6 months, scaling to roughly 8–10 people across research, engineering, dataset and benchmark design, and red‑teaming.
Operate the team independently of the main research and product streams in order to avoid conflicts of interest, including designing novel benchmarks that can be applied apples‑to‑apples to evaluate both the Scientist AI and frontier LLMs.
Oversee the design and construction of new datasets, tasks, and virtual or interactive environments to measure performance of the Scientist AI across capabilities, safety (including honesty and goal-directedness), explainability, causal mechanisms and detecting adversarial attacks.
Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models, including its ability to comply with harm specifications, detect and block harmful responses, explain its decisions, and resist adversarial attacks such as jailbreaks, prompt injection, and data poisoning.
Establish and lead our automated and manual red‑teaming programmes, both in‑house and in partnership with external providers, to stress test the Scientist AI as a general‑purpose model and as a guardrail.
Lead the construction of internal tooling and infrastructure needed to run evaluations at scale, automating and standardizing the pipeline wherever possible.
As needed and where possible, directly support research and product streams with their own internal requirements w.r.t. evaluations and benchmarking to unblock and speed up.
Own LawZero’s public communication of evaluation results, including model and system cards, technical reports, peer‑reviewed publications and blog posts, to build trust with the wider AI safety community.

Key responsibilities

Define LawZero’s evaluations strategy and roadmap, prioritising what needs to be measured and when, in close coordination with both research and product teams.
Build up the Evaluations Team during your first 3–6 months, scaling to roughly 8–10 people across research, engineering, dataset and benchmark design, and red‑teaming.
Operate the team independently of the main research and product streams in order to avoid conflicts of interest, including designing novel benchmarks that can be applied apples‑to‑apples to evaluate both the Scientist AI and frontier LLMs.
Oversee the design and construction of new datasets, tasks, and virtual or interactive environments to measure performance of the Scientist AI across capabilities, safety (including honesty and goal-directedness), explainability, causal mechanisms and detecting adversarial attacks.
Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models, including its ability to comply with harm specifications, detect and block harmful responses, explain its decisions, and resist adversarial attacks such as jailbreaks, prompt injection, and data poisoning.
Establish and lead our automated and manual red‑teaming programmes, both in‑house and in partnership with external providers, to stress test the Scientist AI as a general‑purpose model and as a guardrail.
Lead the construction of internal tooling and infrastructure needed to run evaluations at scale, automating and standardizing the pipeline wherever possible.
As needed and where possible, directly support research and product streams with their own internal requirements w.r.t. evaluations and benchmarking to unblock and speed up.
Own LawZero’s public communication of evaluation results, including model and system cards, technical reports, peer‑reviewed publications and blog posts, to build trust with the wider AI safety community.

Director, Evaluations

Director, Evaluations

LawZero

LawZero

Benefits

LawZero