๐Ÿ“–
FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging
Fatema Siddika, Abhijit Chunduru, Zilinghan Li, Ali Jannesari, Ravi Madduri

FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging

Science & Mathematics Published 2026 Verification Pending

Overview & Synopsis

Federated continual learning trains a model on a sequence of tasks across several clients without sharing their data while retaining knowledge of earlier tasks. Sparse mixture-of-experts models support this setting by adapting selected parameters on a frozen backbone. However, clients learn different expert content as tasks evolve, making it difficult to share knowledge across clients without disrupting task-specific and client-specific specialization. Matching tile coordinates alone does not establish whether their learned content is compatible. We present FedSEAM, a federated continual learning framework that organizes task knowledge into sparse experts and selectively shares their content across tasks and clients. On each client, task-aware tile selection allocates capacity to a new task, while a subspace-based compatibility test determines which existing expert tiles to reuse and where to introduce fresh capacity. On the server, Subspace Expert Agreement Merging (SEAM) compares client updates as subspaces and moves each client towards consensus only along agreed-upon directions, leaving the rest of its update unchanged. We identify two remaining sources of forgetting, routing drift and merge erasure, and address them with three protections, one of which exactly preserves tile components in a selected prior-overlap subspace at every merge. We evaluate FedSEAM on TRACE using Qwen3-4B and Llama-3.2-3B, with component-wise ablations and studies of data size to assess its balance between learning new tasks and retaining prior knowledge.

Indexed from repository: arXiv (arxiv.org)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging
Author / Organization Fatema Siddika, Abhijit Chunduru, Zilinghan Li, Ali Jannesari, Ravi Madduri
Subject & Discipline cs.LG
Categories Science & Mathematics
Publication Year 2026
License / Rights arXiv Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website arXiv (arxiv.org)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-116436
Original Source Link Visit Source Page on arxiv.org »