Research area · Federated learning and privacy
Federated learning research that states its baselines and its privacy assumptions
My federated-learning work began with class imbalance across data holders. The questions I take on now are about evidence: which baselines and metrics a distributed-learning study needs, and what a privacy claim has to assume before it means anything.
The situation
Our data holders have very different class distributions. What baselines and metrics should a federated-learning study use?
What you leave with
A methodological research partnership: an agreed study design with baselines, partitions, metrics, privacy assumptions and a reproducible result.
For: Distributed ML researcher; data-sharing consortium
Two kinds of question in this area
Work here splits into two questions that are often confused:
- Does federated training help, and for whom? This is about accuracy, fairness across clients and communication cost under data that is not independent and identically distributed. My 2020 papers sit here.
- Is the approach private, and against whom? This is about adversaries, leakage and residual risk. Federated learning changes where data sits; it does not by itself answer this question.
If your question is the first, this page and a study design are the starting point. If a vendor, partner or paper is describing a distributed method as private, go to the narrower route on privacy claims and threat models.
Designing a study when clients hold very different class distributions
The decision a consortium usually faces is whether pooling learning across members beats each member alone by enough to justify the cost. A study that answers it fixes four things before running anything:
- Partitions. Natural client splits where they exist, plus controlled label skew at several strengths, so the result can be read as a curve rather than a single point. Quantity skew (some clients have far less data) is tested separately from label skew.
- Baselines. Local-only training, plain federated averaging, a heterogeneity-aware method, an imbalance-aware loss such as a focal-style reweighting, and a pooled reference where policy allows it.
- Metrics. Per-class recall and balanced accuracy or macro-F1 rather than accuracy alone; the worst-client and spread of per-client results; rounds and bytes communicated.
- Uncertainty. Repeats over seeds and over partition draws, with intervals reported, and the rule for calling a difference meaningful written down in advance (see uncertainty and repeated trials).
Where my published work fits
Fed-Focal Loss adapts focal loss to the federated setting so that training is not dominated by majority classes on clients whose data is skewed, without needing the global class distribution. CatFedAvg, a preprint from the same period, addresses the trade-off between communication efficiency and classification accuracy in federated averaging. Both are candidates to include as baselines; neither is a privacy result.
More recently, When Privacy Moves ML-Mediated Decisions On Device (preprint, 2026) looks at a different cost of privacy-motivated decentralisation: once decisions move onto clients, shared constraints depend on stale information, and the simulation shows large overspend and an incentive misalignment. It is relevant here because it treats “move it to the device” as a design choice with measurable side-effects rather than a free gain.
Useful if
- You are planning or reviewing a federated-learning study where clients hold very different label distributions.
- A consortium wants to know whether training across members beats each member training alone, and by how much, before committing infrastructure.
- A distributed-learning method is described as private and you need the assumptions behind that word written down.
Not the right fit if
- You need a production federated-learning system built and operated. That is delivery work (see the link below).
- You want a privacy certification or compliance sign-off. Research can test stated assumptions; it cannot certify a system as private.
- The aim is to confirm a result already announced. A study here can conclude the method does not help.
Limits and unfavourable results
- My published federated-learning papers date from 2020. They are useful baselines and context, not the current state of the art.
- Fed-Focal Loss and CatFedAvg address accuracy and communication under heterogeneous data. Neither paper measures privacy leakage or provides a formal privacy guarantee.
- Results on simulated partitions of public datasets do not transfer automatically to a real consortium's data; the study design says which partitions were tested.
Evidence behind this page
- Fed-Focal Loss for imbalanced data classification in Federated Learning
2020 · Peer-reviewed workshop paper · cited by 96 (Google Scholar, October 2026)
- Catfedavg: Optimising communication-efficiency and classification accuracy in federated learning
2020 · Preprint (not peer reviewed) · cited by 4 (Google Scholar, October 2026)
- When Privacy Moves ML-Mediated Decisions On Device: Information and Incentive Misalignment in Auctions
2026 · Preprint (not peer reviewed)
Narrower routes from here
Questions
What is Fed-Focal Loss?
A loss function for federated learning that adapts focal loss to class imbalance across clients, down-weighting well-classified examples so training concentrates on harder, typically minority-class, examples. It was a workshop paper at the IJCAI 2020 Workshop on Federated Learning for Data Privacy and Confidentiality (FL-IJCAI'20), co-authored with A Narang and S Rai, and had 96 citations on Google Scholar as of October 2026.
What baselines should a federated-learning study on imbalanced data include?
At minimum: each client training alone, plain federated averaging, a method designed for client heterogeneity, and an imbalance-aware loss or reweighting. Where policy allows, a pooled centralised model gives a reference point. The new method is compared against all of them on the same partitions and seeds.
Which metrics matter when class distributions differ across clients?
Overall accuracy hides minority-class failure. Report per-class recall, balanced accuracy or macro-F1, the distribution of per-client performance (including the worst client), communication cost, and the run-to-run spread over seeds and partition draws.
Is federated learning private by default?
No. Keeping raw data on the client reduces one kind of exposure, but model updates can still leak information. A privacy claim needs an explicit adversary, a leakage hypothesis and a test; that is the subject of the privacy threat models page.
Does CatFedAvg have published code or data?
No code artefact is listed for CatFedAvg or Fed-Focal Loss. A new study would reimplement the baselines and publish its own package if the terms allow.
See also
Method
Uncertainty, Repeated Trials and Evaluation Claims
Repeated-trial design, cluster bootstrap over items, rank stability and sensitivity comparisons for small performance gaps.
Method
Pre-Specified Evaluation Plans
Versioned research question, primary outcomes, exclusion and analysis rules, and an amendment log, fixed before results.
Collaboration
Industry-Academic AI Research Collaboration
Map a shared research question, split contributions and settle data and publication dependencies before the work starts.
Papers & Artefacts
Every paper with its publication status, co-authors, code and data.
Discuss a distributed-learning question
Describe the data holders, how their distributions differ, what decision the study informs and what privacy is being claimed. A non-confidential summary is enough to start.
Last reviewed 2026-10-07.