Arcezia / AI agent security / Healthcare / Audit procedure
Audit procedure: AI agent action records (healthcare)
For an internal auditor or an external audit team. It tests whether an organisation’s records of AI agent action checks are genuine, ordered and complete for a period, and, with the organisation’s own logs, whether executed actions were checked and held actions were approved. It does not test whether any decision was correct.
In six steps: request the complete export, the key fingerprint and the organisation’s own logs; run the offline checker over every record; read what each result proves; sample what the checker cannot reach; confirm retention; and state a conclusion of defined scope.
1. What to request
- The complete export for each account in scope: every page, from the account’s first record, not a date slice. Watch it being downloaded, or receive it directly, and note the SHA-256 of each file at hand-over.
- The fingerprint of the account’s record-signing key, from the organisation directly, not from the export: it is shown on the organisation’s dashboard (API Keys page, “Record-signing key fingerprint”) and by
GET /v1/account/record_keys. The organisation is the source of this fingerprint, not Arcezia. Compare two copies taken at different times if you can. - Checkpoints kept outside Arcezia, if the organisation keeps them (for example copies sent to its security log system).
- The inventory of agents and tools in scope, and which of them send every call for a check.
- Execution logs for the same period from the systems the agents act on, ideally carrying the record id for each executed call.
- Approval records for held actions: who approved, when, for which session, keyed by the approval’s one-time id (the id the decision record names), and the fingerprint of the organisation’s registered approval key.
- Erased accounts: a list of any accounts deleted in the period, since their records remain only as hashes, and for each the signed erasure statements fetched with its deletion receipt.
2. How to run the check
pip install "arcezia[verify]"
arcezia-verify-records page-1.ndjson page-2.ndjson ... \
--use-export-keys --fingerprint <fingerprint from the organisation> \
--checkpoint kept-checkpoint.json \
--json workpaper.jsonFor a deleted account, add --erasure erasure-statements.json, the signed erasure statements the organisation fetched with its deletion receipt.
Exit status 0: every record PASS or PASS-ATTESTED and the export is complete. 1: at least one FAIL. 2: nothing failed, but something is UNVERIFIED or completeness is unknown. 3: the input could not be read. Keep workpaper.json with your working papers. The checker covers every record, so integrity needs no sampling. Do not pass --partial: it is for excerpts, such as the sample export, and reports a missing start or end instead of failing it.
3. What each check proves, and what it does not
| Check | A PASS proves | It does not prove |
|---|---|---|
| Signature | The signed content is exactly what was signed with the key you supplied | That the key is the account’s, unless its fingerprint came to you from the organisation, not from the export |
| Header | Each page’s header is the one signed for this account: its record count, first and last record, next page and export time match the file | Anything, for a header with no signature (exports made before headers were signed): it is reported and relied on for nothing |
| Approvals | The approval fields in a record (id, role, approval key fingerprint, call binding, whether it counted) are the ones signed | Who the person was: the organisation’s approval system maps the id to the person |
| Attestation (PASS-ATTESTED) | A record from before the current public format is in its place in the sequence (its hash links to a signed checkpoint), and a current signed statement vouches for its content | The original signature over its content, which is not in the export: you rely on the statement for the content |
| Agreement | The plain columns beside the signed content say the same thing | Anything about the plain time column, which is not signed |
| Account | Each record was signed for the account the export is for | Which person or agent used the account |
| Link and order | No record is missing between the first and last record present, and none is repeated | That the first record is the account’s first, unless the export starts the sequence or a signed removal statement explains the start |
| Checkpoints | No record was removed from the end up to the strongest verified checkpoint, and the counts agree | Anything after the newest checkpoint; a checkpoint inside the export is weaker than one kept elsewhere |
| Removal statements | Records removed at the start were removed under a signed statement | That the retention period chosen was the right one |
| Erasure statements | Each erased record was erased under a signed statement naming it and the hash it carried | What the record said; why the account was deleted |
FAIL is an exception to report. Note how many records are PASS-ATTESTED and say so in the conclusion. UNVERIFIED must be resolved before concluding: obtain the missing key, pages or erasure statements, or record why it cannot be resolved.
4. Sampling for what the checker cannot reach
Use your firm’s sample-size method for a control that operates many times a day.
- Coverage. From the execution logs, select executed agent calls. Trace each to a record with ALLOW for the same session and time, or record id. An executed call with no record, or with a BLOCK or REVIEW record and no later ALLOW, is an exception.
- Approvals. From the records, select ALLOW records whose signed record names an approval that counted. Trace each approval id to the approval system’s record of the person who approved. Confirm the approval key fingerprint is the organisation’s registered approval key, and, where the approval names a call, that it is the record’s own call (the two bindings are equal).
- Refusals. Select BLOCK records. Confirm in the execution logs that the call did not run.
- Patient data. Open a sample of records and confirm they hold no patient data beyond what the organisation chose to put in names.
5. Retention
Confirm the period the export covers against the period the organisation must keep logs for. Common reference periods:
| Rule | Period |
|---|---|
| HIPAA 45 CFR 164.316(b)(2)(i), documentation | six years (whether records count as documentation is the entity’s determination) |
| EU AI Act Art. 26(6), deployers of high-risk systems | at least six months |
| India DPDP Rules 2025, Rule 6(1)(e) and Rule 8(3) | one year |
6. Scope of the conclusion
A clean result supports a statement that the records of AI agent action checks for the period are genuine, ordered and complete, and, to the extent of the samples, that executed calls were checked and held calls were approved. It is not a certification of the organisation, of Arcezia, or of compliance with any law or standard.
How the records relate to each rule, with sources: mapping. Record format, export and retention settings: developer docs, receipts and compliance.