FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction
Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains...
Kaydedildi:
| Asıl Yazarlar: | , , , |
|---|---|
| Materyal Türü: | Makale |
| Baskı/Yayın Bilgisi: |
2022
|
| Konular: | |
| Online Erişim: | https://arxiv.org/abs/2212.01548 http://arxiv.org/abs/2212.01548 |
| Etiketler: |
Etiket eklenmemiş, İlk siz ekleyin!
|
| _version_ | 1879174315461574656 |
|---|---|
| author | Alam, Samiul Liu, Luyang Yan, Ming Zhang, Mi |
| author_browse | Alam, Samiul Liu, Luyang Yan, Ming Zhang, Mi |
| author_facet | Alam, Samiul Liu, Luyang Yan, Ming Zhang, Mi |
| author_sort | Alam, Samiul |
| building | arXiv |
| collection | Acceso abierto |
| description | Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. We show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout and evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex |
| first_indexed | 2026-10-16T00:11:15Z |
| format | Article |
| fullrecord | <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ https://www.openarchives.org/OAI/2.0/oai_dc.xsd"><identifier>arxiv-2212.01548</identifier><datestamp>2023-01-24</datestamp>
<dc:title>FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction</dc:title>
<dc:creator>Alam, Samiul</dc:creator>
<dc:creator>Liu, Luyang</dc:creator>
<dc:creator>Yan, Ming</dc:creator>
<dc:creator>Zhang, Mi</dc:creator>
<dc:subject>Machine Learning</dc:subject>
<dc:subject>Cryptography and Security</dc:subject>
<dc:subject>Computer Vision and Pattern Recognition</dc:subject>
<dc:subject>Distributed, Parallel, and Cluster Computing</dc:subject>
<dc:description>Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. We show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout and evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex </dc:description>
<dc:description>20 pages, 7 Figures, Published in 36th Conference on Neural Information Processing And Systems</dc:description>
<dc:date>2022-12-03</dc:date>
<dc:date>2023-01-20</dc:date>
<dc:type>text</dc:type>
<dc:identifier>https://arxiv.org/abs/2212.01548</dc:identifier>
<dc:identifier>http://arxiv.org/abs/2212.01548</dc:identifier>
</oai_dc:dc> |
| id | arxiv-2212.01548 |
| institution | arXiv |
| last_indexed | 2026-10-16T00:11:15Z |
| publishDate | 2022 |
| publishDateRange | 2022 |
| publishDateSort | 2022 |
| record_format | oai_dc |
| spelling | arxiv-2212.015482023-01-24 FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction Alam, Samiul Liu, Luyang Yan, Ming Zhang, Mi Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. We show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout and evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex 20 pages, 7 Figures, Published in 36th Conference on Neural Information Processing And Systems 2022-12-03 2023-01-20 text https://arxiv.org/abs/2212.01548 http://arxiv.org/abs/2212.01548 |
| spellingShingle | Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing Alam, Samiul Liu, Luyang Yan, Ming Zhang, Mi FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction |
| title | FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction |
| title_full | FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction |
| title_fullStr | FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction |
| title_full_unstemmed | FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction |
| title_short | FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction |
| title_sort | fedrolex model heterogeneous federated learning with rolling sub model extraction |
| topic | Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2212.01548 http://arxiv.org/abs/2212.01548 |
| work_keys_str_mv | AT alamsamiul fedrolexmodelheterogeneousfederatedlearningwithrollingsubmodelextraction AT liuluyang fedrolexmodelheterogeneousfederatedlearningwithrollingsubmodelextraction AT yanming fedrolexmodelheterogeneousfederatedlearningwithrollingsubmodelextraction AT zhangmi fedrolexmodelheterogeneousfederatedlearningwithrollingsubmodelextraction |