Ongoing Student Projects

The following projects are currently pursued by students in our lab, and are therefore not available anymore. They are published for reference and inspiration.

Improving communication-efficiency in Decentralized and Federated Learning Systems

Improving communication-efficiency in Decentralized and Federated Learning Systems

Contact and Supervisor: Rishi Sharma (also CC Anne-Marie Kermarrec)

​Deep neural networks are huge and communicating models during training has a high communication overhead. We would like to reduce this overhead, without adversely affecting the accuracy and generalization of the models. Various techniques such as neighbour selection and message compression are used. In Federated Learning, gradient sparsification techniques are used to reduce communication costs.

This project requires building over a Decentralized Learning framework to implement algorithms to reduce the communication cost such as Neighbour selection and Parameter selection. We would also like to investigate and visualize the importance of individual model layers/parameters to the decentralized/federated learning process. This would also include running multi-machine decentralized and federated learning runtimes to evaluate the algorithms over different datasets and graph topologies.

Sub-goals

  1.  Implement random parameter sampling used in Gossip Learning [2].
  2. Implement non-iid versions for datasets such as CIFAR. Evaluate using the framework.
  3. Implement Random-Model Walk and compare with the existing results. Investigate if gradient sparsification techniques work well.
  4. The framework gives a partition of the model based on the importance of the parameters on every iteration of training, identify patterns (maybe by visualizing using tensorboard) over the entire training process.
  5. Optional, based on calculus and the gradients available during the training process, find a better parameter sampling.

Must-haves​

  1. Python programming experience
  2. Knowledge of Machine Learning

Good to know​

  1. Computer networks
  2. Experience with Pytorch
  3. Calculus
  4. Concurrency

You may like to read

[1] R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2015, pp. 909-910, doi: 10.1109/ALLERTON.2015.7447103.
https://ieeexplore.ieee.org/document/7447103
[2] István Hegedűs, Gábor Danner, Márk Jelasity. “Decentralized learning works: An empirical comparison of gossip learning and federated learning.” In Journal of Parallel and Distributed Computing, Volume 148, 2021, Pages 109-124, ISSN 0743-7315. https://www.sciencedirect.com/science/article/pii/S0743731520303890
[3] McMahan, Brendan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. “Communication-efficient learning of deep networks from decentralized data.” In Artificial Intelligence and Statistics, pp. 1273-1282. PMLR, 2017. https://arxiv.org/abs/1602.05629