Efficient Machine Learning on Decentralized Data at Scale

Supervisor: Dr Ahmed Sayed

Research group(s): Networks

AI/ML systems are becoming an integral part of user products and applications as well as the main revenue driver for most organizations. This resulted in shifting the focus to bringing the intelligence towards where the data are produced including training the models on these data. Existing approaches operate as follows: 1) the data is collected on multiple servers and processed in parallel (e.g., Distributed Data-Parallel); 2) the server coordinates the training rounds and collects model updates from the clients (e.g., Federated Learning); 3) the server splits the model training between the clients and the server (e.g., Split Learning); or 4) the clients coordinate among themselves via gossip protocols (i.e., Decentralized Training). The challenges that manifest themselves are the highly heterogeneous learners, configurations, and environment, communication and synchronization overheads, fairness and bias, and privacy and security. Therefore, existing approaches fail to scale with a large number of learners and produce models with low qualities and high bias at prolonged training times. It is imperative to build systems that provide high-quality models in a timely manner. This project addresses this gap by exploring novel ideas and proposing efficient and scalable ML systems for decentralized data.