Telč 2023
Federated AI communication protocol
Václav Oujezský
Telč 2023
Overview
● Motivation for the topic
● Federated learning
● Current research projects on the given topic
● Current challenges and conclusion
2
Telč 2023
Motivation
● Safety
○ Data privacy, transfer of sensitive data
● Efficiency
○ Federated learning, federated computing
● Development
○ New possibilities of use (ad hoc encryption, mobile
communication, others)
3
Telč 2023
Federated learning?
Data processing is increasingly done on end devices
● Improving responsiveness
● Benefits of data security
● Analytics
● Learning
4
Telč 2023
Federated learning?
5
Telč 2023
Federated learning?
Specifically defined in 2019 in the publication
Advances and Open Problems in Federated Learning (https://arxiv.org/abs/1912.049770)
Citation:
Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or
whole organizations) collaboratively train a model under the orchestration of a central server (e.g.
service provider), while keeping the training data decentralized. FL embodies the principles of
focused data collection and minimization, and can mitigate many of the systemic privacy risks and
costs resulting from traditional, centralized machine learning and data science approaches.
Motivated by the explosive growth in FL research, this paper discusses recent advances and
presents an extensive collection of open problems and challenges.
6
Telč 2023
Federated learning
7
1
Characteristics
2
Basic
types
3
Term
inology
Telč 2023
Federated learning ‒ characteristic
● Data is generated locally
● Data remains decentralized
● Data is not shared and distributed
● A central orchestrator coordinates training,
but never sees its own data
8
Telč 2023
Federated learning ‒ basic types
Cross-silo federated learning (between objects, institutions)
● smaller number of clients, high availability
● object identity
● each object participates in each round of learning
● computational complexity is the primary weak point
Cross-device federated learning (between end devices)
● hundreds of temporarily available clients
● no identification
● usually each client participates only once
● communication is the primary weak point
Decentralized learning
● peer-to-peer, without a centralized orchestrator
9
Telč 2023
Federated learning ‒ terminology
● Client
○ Computing entity holding local data (mobile device, IoT,
institution, …)
● Server
○ Federated learning coordinator, nowadays, more than
one device
10
Telč 2023
Federated learning
11
1
Algorithm
2
Variations
3
Fram
ew
orks
Telč 2023
Federated learning ‒ basic algorithm
● Client
○ The current state of the model is sent to the client from the central element
○ The client trains it with local data
○ The principle of training with the SGD (Stochastic Gradient Descent) method or its
derivation as a function of evaluation
○ The result is model parameter weights that are sent to the central element
● Server
○ It obtains parameter weights from clients and performs averaging (Federated
Average) and updates the original model
○ The next procedure already depends on the specific solution (cross-silo,
cross-device)
12
Telč 2023
Federated learning ‒ basic principles
● Optimization of the objective function for the purpose of
convergence of weights
● Hyperparameters
○ number of epochs, learning rate, batch size
13
Telč 2023
Federated learning ‒ list of variations
● Federated stochastic gradient descent (FedSGD)
○ transpose SGD, swap gradients, random number of clients, gradients are
averaged and gradient step performed
● Federated averaging (FedAvg)
○ generalization of FedSGD, exchange of updated weights, not gradients
● Federated Learning with Dynamic Regularization (FedDyn)
○ 2021(Acar, et al.), the issue of heterogeneous data distribution. It addresses the
dilemma of the difference between device error minimization and global error
minimization. Using dynamic control to converge local and global error.
● Hybrid Federated Dual Coordinate Ascent (HyFDCA)
○ 2022 (Overman et al.), solves the problems of hybrid federated learning, where
each client solves only a certain subset of data samples (features, samples).
● KafkaFed
○ An example of a design of a federated learning algorithm using Apache Kafka as
a communication medium
14
Telč 2023
Federated learning ‒ frameworks
● TensorFlow Federated ‒ opensource Google Brain 2019, use TFLite flatbuffer files, mostly
focused on simulations, still in development, FedAvg integration
● FederatedAITechnologyEnabler(FATE) ‒ Webank, the first open source at an industrial level ‒
in the area of credit risk control, object detection and anti-money laundering
● IBM Federated Learning ‒ possible to use for free, tools for implementation in real situations
● PySyft ‒ the open source solution OpenMined, originally using PyTorch, then also using
TensorFlow
● DeepLearning4J ‒ opensource Konduit for JVM, Python, C++
● Visioner - based on TensorFlow, uses inference graph* files,
org.tensorflow:tensorflow-android)
● Flower ‒ uses TensorFlow, possibility of use on AWS, Azure, Android, iOS, Raspberry, Nvidia
Jetson
● FedML ‒ cross platform solution
15
Telč 2023
Federated learning
16
1
Com
m
unication
2
Protocols
3
Research
project
Telč 2023
Federated learning ‒ communication
17
Telč 2023
Federated learning ‒ communication
Transmission of updated data
● most often gRPC (HTTP/2 and protocol buffer, Google 2015)
● also REST HTTP API (Ktor, Retrofit2, others)
● or Apache Kafka, dynamic databases like Google Firebase
https://www.tensorflow.org/federated/tutorials/building_your_own_federated_lear
ning_algorithm
18
Telč 2023
Federated learning ‒ communication
19
Telč 2023
Federated learning ‒ communication
protocol, ideas
20
Telč 2023
Federated learning ‒ proposed
communication protocol
21
Compatible with both, Apache Kafka and Realtime database
Using a text format for the transmission of messages
Automatic selection of central nodes
Seemless communication
Telč 2023
Federated learning ‒ proposed
communication protocol
22
Telč 2023
Federated learning ‒ proposed
communication protocol
23
J. Michalek, V. Skorpil and V. Oujezsky, "Federated Learning on Android - Highlights from Recent Developments," 2022
14th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Valencia,
Spain, 2022, pp. 27-30, doi: 10.1109/ICUMT57764.2022.9943382.
J. Michalek, V. Oujezsky, and V. Skorpil An Android Federated Learning Framework for Emergency Management
Applications, 2023, ICUMT.
Telč 2023
Federated learning ‒ research project
Android federated learning framework for crisis
management applications
24
Telč 2023
Current challenges
Communication problems
● Security risks
○ Transfer of model, weights
● Principles of data transfer, protocols, synchronization
synchronization
● Decentralized learning
25
Telč 2023
Thank you
oujezsky@vut.cz
This presentation is based upon the grant of the Ministry of the Interior of the Czech Republic, Open challenges in security research,
VK01030152, ``Android federated learning framework for emergency management applications ''.
Telč 2023
Federated learning ‒ appendix
27
Telč 2023
● Federated stochastic gradient descent (FedSGD)
○ transpose SGD, swap gradients, random number of clients, gradients are
averaged and gradient step performed
Federated learning ‒ FedSGD
28
Loss on the data
sample with the parameters of the
weightsA set of client dataClient Data Count
The server
aggregates the
gradients and
applies the update
Model weights in
the communication
round
training rate
Telč 2023
● Federated averaging (FedAvg)
○ generalization of FedSGD, exchange of updated weights, not gradients
In practice, calculations are performed on clients in batches of data
Federated learning ‒ FedAvg
29
One step GD locally on the client
The server updates the model with a weighted average of the weights