Define what an AI security evaluation can actually establish

Establish the access assumptions before testing a model

Suppose a classifier prioritizes suspicious transactions for human review. The team wants to know whether unauthorized training-data changes could undermine that queue. Before modifying a dataset, find out who can change labels, which pipeline accepts them and how a training run records its inputs. Otherwise, the test may assume an attacker capability that no relevant actor actually has.

In an isolated evaluation, compare the unchanged training dataset with a controlled, documented modification. Keep the evaluation set separate from training and tuning. Report missed cases, false alerts and the resulting review workload, including uncertainty. Overall accuracy may stay similar even while one subgroup changes substantially, so it cannot answer the operational question on its own.

Record what the evaluation is intended to establish

Complete the evaluation card before choosing the benchmark. Another reader should be able to see why the test was relevant, what it measured and how it informed the decision. Retain the configuration and artifacts needed to reproduce it without publishing sensitive records or credentials.

Evaluate the control and the model separately

For the classifier, approved label changes, immutable dataset versions and a traceable promotion decision would address how training inputs enter the system. Check that unauthorized changes are rejected and authorized changes produce attributable new inputs. Then repeat the bounded evaluation on the candidate model. This checks both enforcement of the control and the behavior of the model trained under it.

Also examine the surrounding application. A model artifact might be replaceable without authority, sensitive examples might enter logs, or evaluation data might cross a tenant boundary. These are access and dependency questions in their own right. Keep their results separate from the model's behavior so a correction at one layer is not mistaken for a correction everywhere.

State where the result can be used

NIST's adversarial machine-learning taxonomy distinguishes attack stages and objectives. Its generative AI risk profile adds context for systems that generate content. Use OWASP's LLM guidance when the application actually uses a language model. The selected threat analysis still needs to match the type of system being evaluated.

Report the evaluated versions, access assumptions and data distribution. Reproducing a result shows that it can occur again under the recorded conditions. It does not establish that the result generalizes to another population or deployment. State which decision the evidence supports and which questions about safety, fairness and adaptive attacks remain open.