Commission
Research
All research areas →

Agent Evaluation and Coordination

Generated-Code and GPU-Kernel Correctness

Federated Learning and Privacy

Decentralised Systems and Protocol

Methods
Papers
Collaborate
About Contact Bring a technical claim

Research topic · Privacy threat models

A privacy claim is only as good as the threat model behind it

Federated and on-device learning are often described as private. Before that claim can be tested, someone has to say who the adversary is, what they can see, what counts as leakage and what risk is left over.

The situation

What adversary and leakage assumptions must be specified before a distributed ML approach can make a privacy claim?

What you leave with

An explicit threat model, stated leakage hypotheses, a test design for each, and a written account of the residual uncertainty.

For: Privacy researcher; security evaluation team

Established background

The points below are general field knowledge, stated cautiously; they are not my results.

  • Data locality is not privacy. In federated learning, raw data stays on clients but model updates leave them. The research literature reports attacks that infer whether a record was in training (membership inference), infer properties of a client’s data, and in some settings partially reconstruct inputs from updates.
  • The adversary matters more than the architecture. An aggregating server that follows the protocol but inspects what it receives is a different threat from one that deviates, from colluding clients, or from an outsider who only sees the final model.
  • Defences have assumptions too. Secure aggregation hides individual updates from the server but not the aggregate; differential privacy bounds leakage only for its stated parameters and accounting; neither addresses metadata such as who participated, when, or with how much data.

What my work contributes, and what it does not

Fed-Focal Loss and CatFedAvg are about accuracy and communication under heterogeneous client data. They appear here because people sometimes read “federated” as “private”. It is worth saying directly: neither paper makes or tests a privacy guarantee.

When Privacy Moves ML-Mediated Decisions On Device (preprint, 2026) is the closest of my papers to this topic, from an unusual angle. It does not measure leakage. It shows that moving decisions onto privacy-preserving clients changes the information structure of the system: shared budgets come to depend on stale information, and an on-device simulation shows large overspend from staleness plus an incentive misalignment in payment units. The lesson for threat modelling is that a privacy architecture should be evaluated for what it protects and for what it breaks.

Illustrative example — not a client engagement. A provider says its cross-hospital model is “private because data never leaves the site”. The restated claim is that the aggregating server cannot determine whether a given patient record was used in training. The threat model fixes an honest-but-curious server that sees individual updates. Membership-inference tests run against the system as deployed and against a variant with the provider’s planned noise addition. The report states that the first configuration does not support the claim, the second weakens the attack under the tested conditions, and participation metadata remains visible in both.

Open questions

  • How should evaluations report leakage when the strongest known attack changes over time?
  • What is the right comparison point for an empirical attack: no defence, a published defence, or a pooled baseline?
  • How do privacy-motivated architectures trade against other properties — accuracy on minority clients, budget compliance, incentive alignment — and can those trade-offs be measured in one study?

Useful if

  • A vendor or partner describes its distributed learning as private and you need to know what that would have to mean to be checkable.
  • Your own team is about to publish or market a privacy property and wants the assumptions stated before reviewers ask.
  • A consortium needs to agree what each member is protected against before sharing model updates.

Not the right fit if

  • You need a legal or regulatory compliance opinion. This is technical evaluation, not legal advice.
  • You want a checklist that certifies a system as private. No checklist does that; a threat model makes the claim testable, not true.
  • The system has no defined data flow yet. Start with the design; the threat model follows it.

What this produces

  • Threat model. Who the adversary is (server, other clients, an outside observer of outputs), whether they are honest-but-curious or actively malicious, what they observe, and what they can change.
  • Leakage hypotheses. Concrete statements of what might leak — membership, attributes, reconstructable inputs, label distributions, participation patterns — each phrased so a test can fail.
  • Test design. For each hypothesis, the attack or measurement used, the data it runs on, the success criterion and the comparison point.
  • Residual-uncertainty statement. What the tests did not cover, which assumptions were taken on trust, and what would change the conclusion.

What you need before starting

  • The claim in writing. The exact wording of the privacy claim and where it is made.
  • Data-flow description. What leaves each client, in what form, how often, and who can see it at each hop.
  • Defences in place. Any secure aggregation, noise addition, clipping or access control, with parameters.
  • Access for testing. A test deployment or reference implementation and representative non-sensitive data; production data is not required.

Protocol

  1. 1
    Restate the claim. Turn the marketing or paper wording into a property: what an adversary with a given capability should not be able to learn.
  2. 2
    Fix the adversary. Write down capabilities and trust boundaries before choosing attacks, so the attacks are chosen for the model rather than the other way round.
  3. 3
    Map leakage channels. Trace every artefact that crosses a trust boundary — updates, gradients, metadata, outputs, timing — and attach a hypothesis to each.
  4. 4
    Test and compare. Run the agreed attacks against the system and against a reference with no defence, so the effect of the defence is measured rather than assumed.
  5. 5
    Report what remains. State what was shown, what was not tested and which assumptions the conclusion rests on.

Limits and unfavourable results

  • Empirical attacks give a lower bound on leakage: a failed attack shows that attack failed, not that the system is private.
  • Formal guarantees such as differential privacy hold only under their stated parameters and accounting; the study checks that the stated mechanism is the one implemented, within the access provided.
  • I have no published paper that measures leakage in a federated system. The evidence below is adjacent work, and this page treats the topic as a research direction.

A study may conclude the claim cannot be supported under any reasonable threat model, or that it holds only against a weaker adversary than the wording implies. Either is reported as found.

Evidence behind this page

  • on-device-auction-audit (GitHub) — Results, calibration data and analysis scripts for the on-device auction paper. Code MIT, data CC BY-NC 4.0; simulator source withheld with hashes recorded.

Questions

Does federated learning guarantee privacy?

No. It keeps raw data on clients, which removes one exposure, but model updates and outputs can still carry information about training data. Any stronger statement needs a named adversary, a defence and evidence.

Is Fed-Focal Loss a privacy technique?

No. Fed-Focal Loss addresses class imbalance across clients in federated learning. It inherits whatever privacy properties the surrounding federated setup has and adds none; the paper does not measure leakage.

Which threat assumptions should a vendor's privacy claim state?

Who is trusted and who is not (aggregating server, other participants, model users), whether adversaries are passive or can deviate from the protocol, whether they can collude, what each observes, and which defences are in place with their parameters.

Can a privacy evaluation prove a system is private?

An empirical evaluation can show that specific attacks succeed or fail under stated conditions. It cannot prove the absence of leakage. A formal guarantee can bound leakage, but only for the mechanism and parameters actually implemented.

How is this commissioned and published?

As a fixed-scope study whose fee is set at scoping and does not depend on the result. Data rights, review windows and publication terms are agreed before work starts; see publication, IP and independence.

See also

Commission a scoped study

Send the privacy claim as worded, a description of what leaves each client, and the decision the evaluation informs. Code and data access are agreed afterwards.

Last reviewed 2026-10-07.