Federated Logistic Regression

Logistic Regression(LR) is a widely used statistic model for classification problems. FATE provided two modes of federated LR: Homogeneous LR (HomoLR) and Heterogeneous LR (HeteroLR and Hetero_SSHE_LR).

Below lists features of each LR models:

Linear Model	Multiclass(OVR)	Arbiter-less Training	Weighted Training	Multi-Host	Cross Validation	Warm-Start/CheckPoint
Hetero LR	✓	✗	✓	✓	✓	✓
Hetero SSHELR	✓	✓	✓	✗	✓	✓
Homo LR	✓	✗	✓	✓	✓	✓

We simplified the federation process into three parties. Party A represents Guest， party B represents Host while party C, which also known as "Arbiter", is a third party that holds a private key for each party and work as a coordinator. (Hetero_SSHE_LR have not "Arbiter" role)

Homogeneous LR

As the name suggested, in HomoLR, the feature spaces of guest and hosts are identical. An optional encryption mode for computing gradients is provided for host parties. By doing this, the plain model is not available for this host any more.

Models of Party A and Party B have the same structure. In each iteration, each party trains its model on its own data. After that, all parties upload their encrypted (or plain, depends on your configuration) gradients to arbiter. The arbiter aggregates these gradients to form a federated gradient that will then be distributed to all parties for updating their local models. Similar to traditional LR, the training process will stop when the federated model converges or the whole training process reaches a predefined max-iteration threshold. More details is available in this Practical Secure Aggregation for Privacy-Preserving Machine Learning.

Heterogeneous LR

The HeteroLR carries out the federated learning in a different way. As shown in Figure 2, A sample alignment process is conducted before training. This sample alignment process is to identify overlapping samples stored in databases of the two involved parties. The federated model is built based on those overlapping samples. The whole sample alignment process will not leak confidential information (e.g., sample ids) on the two parties since it is conducted in an encrypted way.

In the training process, party A and party B compute out the elements needed for final gradients. Arbiter aggregate them and compute out the gradient and then transfer back to each party. More details is available in this: Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption.

Multi-host hetero-lr

For multi-host scenario, the gradient computation still keep the same as single-host case. However, we use the second-norm of the difference of model weights between two consecutive iterations as the convergence criterion. Since the arbiter can obtain the completed model weight, the convergence decision is happening in Arbiter.

Heterogeneous SSHE Logistic Regression

FATE implements a heterogeneous logistic regression without arbiter role called for hetero_sshe_lr. More details is available in this following paper: When Homomorphic Encryption Marries Secret Sharing: Secure Large-Scale Sparse Logistic Regression and Applications in Risk Control. We have also made some optimization so that the code may not exactly same with this paper. The training process could be described as the following: forward and backward process.

The training process is based secure matrix multiplication protocol(SMM), which HE and Secret-Sharing hybrid protocol is included.

Param

logistic_regression_param

Classes

LogisticParam(penalty='L2', tol=0.0001, alpha=1.0, optimizer='rmsprop', batch_size=-1, shuffle=True, batch_strategy='full', masked_rate=5, learning_rate=0.01, init_param=InitParam(), max_iter=100, early_stop='diff', encrypt_param=EncryptParam(), predict_param=PredictParam(), cv_param=CrossValidationParam(), decay=1, decay_sqrt=True, multi_class='ovr', validation_freqs=None, early_stopping_rounds=None, stepwise_param=StepwiseParam(), floating_point_precision=23, metrics=None, use_first_metric_only=False, callback_param=CallbackParam())

Bases: LinearModelParam

Parameters used for Logistic Regression both for Homo mode or Hetero mode.

Parameters:

Name	Type	Description	Default
penalty		Penalty method used in LR. Please note that, when using encrypted version in HomoLR, 'L1' is not supported.	'L2'
tol	float, default	The tolerance of convergence	0.0001
alpha	float, default	Regularization strength coefficient.	1.0
optimizer		Optimize method.	'rmsprop'
batch_strategy	str	Strategy to generate batch data. a) full: use full data to generate batch_data, batch_nums every iteration is ceil(data_size / batch_size) b) random: select data randomly from full data, batch_num will be 1 every iteration.	'full'
batch_size	int, default	Batch size when updating model. -1 means use all data in a batch. i.e. Not to use mini-batch strategy.	-1
shuffle	bool, default	Work only in hetero logistic regression, batch data will be shuffle in every iteration.	True
masked_rate		Use masked data to enhance security of hetero logistic regression	5
learning_rate	float, default	Learning rate	0.01
max_iter	int, default	The maximum iteration for training.	100
early_stop		Method used to judge converge or not. a) diff： Use difference of loss between two iterations to judge whether converge. b) weight_diff: Use difference between weights of two consecutive iterations c) abs: Use the absolute value of loss to judge whether converge. i.e. if loss < eps, it is converged. Please note that for hetero-lr multi-host situation, this parameter support "weight_diff" only.	'diff'
decay		Decay rate for learning rate. learning rate will follow the following decay schedule. lr = lr0/(1+decay*t) if decay_sqrt is False. If decay_sqrt is True, lr = lr0 / sqrt(1+decay*t) where t is the iter number.	1
decay_sqrt		lr = lr0/(1+decay*t) if decay_sqrt is False, otherwise, lr = lr0 / sqrt(1+decay*t)	True
encrypt_param		encrypt param	EncryptParam()
predict_param		predict param	PredictParam()
callback_param		callback param	CallbackParam()
cv_param		cv param	CrossValidationParam()
multi_class		If it is a multi_class task, indicate what strategy to use. Currently, support 'ovr' short for one_vs_rest only.	'ovr'
validation_freqs		validation frequency during training.	None
early_stopping_rounds		Will stop training if one metric doesn’t improve in last early_stopping_round rounds	None
metrics		Indicate when executing evaluation during train process, which metrics will be used. If set as empty, default metrics for specific task type will be used. As for binary classification, default metrics are ['auc', 'ks']	None
use_first_metric_only		Indicate whether use the first metric only for early stopping judgement.	False
floating_point_precision		if not None, use floating_point_precision-bit to speed up calculation, e.g.: convert an x to round(x * 2**floating_point_precision) during Paillier operation, divide the result by 2**floating_point_precision in the end.	23

Source code in federatedml/param/logistic_regression_param.py

def __init__(self, penalty='L2',
             tol=1e-4, alpha=1.0, optimizer='rmsprop',
             batch_size=-1, shuffle=True, batch_strategy="full", masked_rate=5,
             learning_rate=0.01, init_param=InitParam(),
             max_iter=100, early_stop='diff', encrypt_param=EncryptParam(),
             predict_param=PredictParam(), cv_param=CrossValidationParam(),
             decay=1, decay_sqrt=True,
             multi_class='ovr', validation_freqs=None, early_stopping_rounds=None,
             stepwise_param=StepwiseParam(), floating_point_precision=23,
             metrics=None,
             use_first_metric_only=False,
             callback_param=CallbackParam()
             ):
    super(LogisticParam, self).__init__()
    self.penalty = penalty
    self.tol = tol
    self.alpha = alpha
    self.optimizer = optimizer
    self.batch_size = batch_size
    self.learning_rate = learning_rate
    self.init_param = copy.deepcopy(init_param)
    self.max_iter = max_iter
    self.early_stop = early_stop
    self.encrypt_param = encrypt_param
    self.shuffle = shuffle
    self.batch_strategy = batch_strategy
    self.masked_rate = masked_rate
    self.predict_param = copy.deepcopy(predict_param)
    self.cv_param = copy.deepcopy(cv_param)
    self.decay = decay
    self.decay_sqrt = decay_sqrt
    self.multi_class = multi_class
    self.validation_freqs = validation_freqs
    self.stepwise_param = copy.deepcopy(stepwise_param)
    self.early_stopping_rounds = early_stopping_rounds
    self.metrics = metrics or []
    self.use_first_metric_only = use_first_metric_only
    self.floating_point_precision = floating_point_precision
    self.callback_param = copy.deepcopy(callback_param)

Attributes

penalty = penalty instance-attribute

tol = tol instance-attribute

alpha = alpha instance-attribute

optimizer = optimizer instance-attribute

batch_size = batch_size instance-attribute

learning_rate = learning_rate instance-attribute

init_param = copy.deepcopy(init_param) instance-attribute

max_iter = max_iter instance-attribute

early_stop = early_stop instance-attribute

encrypt_param = encrypt_param instance-attribute

shuffle = shuffle instance-attribute

batch_strategy = batch_strategy instance-attribute

masked_rate = masked_rate instance-attribute

predict_param = copy.deepcopy(predict_param) instance-attribute

cv_param = copy.deepcopy(cv_param) instance-attribute

decay = decay instance-attribute

decay_sqrt = decay_sqrt instance-attribute

multi_class = multi_class instance-attribute

validation_freqs = validation_freqs instance-attribute

stepwise_param = copy.deepcopy(stepwise_param) instance-attribute

early_stopping_rounds = early_stopping_rounds instance-attribute

metrics = metrics or [] instance-attribute

use_first_metric_only = use_first_metric_only instance-attribute

floating_point_precision = floating_point_precision instance-attribute

callback_param = copy.deepcopy(callback_param) instance-attribute

Functions

check()

Source code in federatedml/param/logistic_regression_param.py

def check(self):
    descr = "logistic_param's"
    super(LogisticParam, self).check()
    self.predict_param.check()
    if self.encrypt_param.method not in [consts.PAILLIER, None]:
        raise ValueError(
            "logistic_param's encrypted method support 'Paillier' or None only")
    self.multi_class = self.check_and_change_lower(self.multi_class, ["ovr"], f"{descr}")
    if not isinstance(self.masked_rate, (float, int)) or self.masked_rate < 0:
        raise ValueError("masked rate should be non-negative numeric number")
    if not isinstance(self.batch_strategy, str) or self.batch_strategy.lower() not in ["full", "random"]:
        raise ValueError("batch strategy should be full or random")
    self.batch_strategy = self.batch_strategy.lower()
    if not isinstance(self.shuffle, bool):
        raise ValueError("shuffle should be boolean type")
    return True

HomoLogisticParam(penalty='L2', tol=0.0001, alpha=1.0, optimizer='rmsprop', batch_size=-1, learning_rate=0.01, init_param=InitParam(), max_iter=100, early_stop='diff', encrypt_param=EncryptParam(method=None), re_encrypt_batches=2, predict_param=PredictParam(), cv_param=CrossValidationParam(), decay=1, decay_sqrt=True, aggregate_iters=1, multi_class='ovr', validation_freqs=None, early_stopping_rounds=None, metrics=['auc', 'ks'], use_first_metric_only=False, use_proximal=False, mu=0.1, callback_param=CallbackParam())

Bases: LogisticParam

Parameters:

Name	Type	Description	Default
re_encrypt_batches	int, default	Required when using encrypted version HomoLR. Since multiple batch updating coefficient may cause overflow error. The model need to be re-encrypt for every several batches. Please be careful when setting this parameter. Too large batches may cause training failure.	2
aggregate_iters	int, default	Indicate how many iterations are aggregated once.	1
use_proximal		Whether to turn on additional proximial term. For more details of FedProx, Please refer to https://arxiv.org/abs/1812.06127	False
mu		To scale the proximal term	0.1

Source code in federatedml/param/logistic_regression_param.py

def __init__(self, penalty='L2',
             tol=1e-4, alpha=1.0, optimizer='rmsprop',
             batch_size=-1, learning_rate=0.01, init_param=InitParam(),
             max_iter=100, early_stop='diff',
             encrypt_param=EncryptParam(method=None), re_encrypt_batches=2,
             predict_param=PredictParam(), cv_param=CrossValidationParam(),
             decay=1, decay_sqrt=True,
             aggregate_iters=1, multi_class='ovr', validation_freqs=None,
             early_stopping_rounds=None,
             metrics=['auc', 'ks'],
             use_first_metric_only=False,
             use_proximal=False,
             mu=0.1, callback_param=CallbackParam()
             ):
    super(HomoLogisticParam, self).__init__(penalty=penalty, tol=tol, alpha=alpha, optimizer=optimizer,
                                            batch_size=batch_size,
                                            learning_rate=learning_rate,
                                            init_param=init_param, max_iter=max_iter, early_stop=early_stop,
                                            encrypt_param=encrypt_param, predict_param=predict_param,
                                            cv_param=cv_param, multi_class=multi_class,
                                            validation_freqs=validation_freqs,
                                            decay=decay, decay_sqrt=decay_sqrt,
                                            early_stopping_rounds=early_stopping_rounds,
                                            metrics=metrics, use_first_metric_only=use_first_metric_only,
                                            callback_param=callback_param)
    self.re_encrypt_batches = re_encrypt_batches
    self.aggregate_iters = aggregate_iters
    self.use_proximal = use_proximal
    self.mu = mu

Attributes

re_encrypt_batches = re_encrypt_batches instance-attribute

aggregate_iters = aggregate_iters instance-attribute

use_proximal = use_proximal instance-attribute

mu = mu instance-attribute

Functions

check()

Source code in federatedml/param/logistic_regression_param.py

def check(self):
    super().check()
    if type(self.re_encrypt_batches).__name__ != "int":
        raise ValueError(
            "logistic_param's re_encrypt_batches {} not supported, should be int type".format(
                self.re_encrypt_batches))
    elif self.re_encrypt_batches < 0:
        raise ValueError(
            "logistic_param's re_encrypt_batches must be greater or equal to 0")

    if not isinstance(self.aggregate_iters, int):
        raise ValueError(
            "logistic_param's aggregate_iters {} not supported, should be int type".format(
                self.aggregate_iters))

    if self.encrypt_param.method == consts.PAILLIER:
        if self.optimizer != 'sgd':
            raise ValueError("Paillier encryption mode supports 'sgd' optimizer method only.")

        if self.penalty == consts.L1_PENALTY:
            raise ValueError("Paillier encryption mode supports 'L2' penalty or None only.")

    return True

HeteroLogisticParam(penalty='L2', tol=0.0001, alpha=1.0, optimizer='rmsprop', batch_size=-1, shuffle=True, batch_strategy='full', masked_rate=5, learning_rate=0.01, init_param=InitParam(), max_iter=100, early_stop='diff', encrypted_mode_calculator_param=EncryptedModeCalculatorParam(), predict_param=PredictParam(), cv_param=CrossValidationParam(), decay=1, decay_sqrt=True, sqn_param=StochasticQuasiNewtonParam(), multi_class='ovr', validation_freqs=None, early_stopping_rounds=None, metrics=['auc', 'ks'], floating_point_precision=23, encrypt_param=EncryptParam(), use_first_metric_only=False, stepwise_param=StepwiseParam(), callback_param=CallbackParam())

Bases: LogisticParam

Source code in federatedml/param/logistic_regression_param.py

def __init__(self, penalty='L2',
             tol=1e-4, alpha=1.0, optimizer='rmsprop',
             batch_size=-1, shuffle=True, batch_strategy="full", masked_rate=5,
             learning_rate=0.01, init_param=InitParam(),
             max_iter=100, early_stop='diff',
             encrypted_mode_calculator_param=EncryptedModeCalculatorParam(),
             predict_param=PredictParam(), cv_param=CrossValidationParam(),
             decay=1, decay_sqrt=True, sqn_param=StochasticQuasiNewtonParam(),
             multi_class='ovr', validation_freqs=None, early_stopping_rounds=None,
             metrics=['auc', 'ks'], floating_point_precision=23,
             encrypt_param=EncryptParam(),
             use_first_metric_only=False, stepwise_param=StepwiseParam(),
             callback_param=CallbackParam()
             ):
    super(HeteroLogisticParam, self).__init__(penalty=penalty, tol=tol, alpha=alpha, optimizer=optimizer,
                                              batch_size=batch_size, shuffle=shuffle, batch_strategy=batch_strategy,
                                              masked_rate=masked_rate,
                                              learning_rate=learning_rate,
                                              init_param=init_param, max_iter=max_iter, early_stop=early_stop,
                                              predict_param=predict_param, cv_param=cv_param,
                                              decay=decay,
                                              decay_sqrt=decay_sqrt, multi_class=multi_class,
                                              validation_freqs=validation_freqs,
                                              early_stopping_rounds=early_stopping_rounds,
                                              metrics=metrics, floating_point_precision=floating_point_precision,
                                              encrypt_param=encrypt_param,
                                              use_first_metric_only=use_first_metric_only,
                                              stepwise_param=stepwise_param,
                                              callback_param=callback_param)
    self.encrypted_mode_calculator_param = copy.deepcopy(encrypted_mode_calculator_param)
    self.sqn_param = copy.deepcopy(sqn_param)

Attributes

encrypted_mode_calculator_param = copy.deepcopy(encrypted_mode_calculator_param) instance-attribute

sqn_param = copy.deepcopy(sqn_param) instance-attribute

Functions

check()

Source code in federatedml/param/logistic_regression_param.py

def check(self):
    super().check()
    self.encrypted_mode_calculator_param.check()
    self.sqn_param.check()
    return True

Features

• Both Homo-LR and Hetero-LR

1. L1 & L2 regularization

2. Mini-batch mechanism

3. Weighted training

4. Six optimization method:

   • sgd
     gradient descent with arbitrary batch size

   • rmsprop
     RMSProp

   • adam
     Adam

   • adagrad
     AdaGrad

   • nesterov_momentum_sgd
     Nesterov Momentum

5. Three converge criteria:

   • diff
     Use difference of loss between two iterations, not available for multi-host training;

   • abs
     use the absolute value of loss;

   • weight_diff
     use difference of model weights

6. Support multi-host modeling task.

7. Support validation for every arbitrary iterations

8. Learning rate decay mechanism

• Homo-LR extra features

1. Two Encryption mode

   • "Paillier" mode
     Host will not get clear text model. When using encryption mode, "sgd" optimizer is supported only.

   • Non-encryption mode
     Everything is in clear text.

2. Secure aggregation mechanism used when more aggregating models

3. Support aggregate for every arbitrary iterations.

4. Support FedProx mechanism. More details is available in this Federated Optimization in Heterogeneous Networks.

• Hetero-LR extra features

• Support different encrypt-mode to balance speed and security

• Support OneVeRest
• When modeling a multi-host task, "weight_diff" converge criteria

  is supported only.
  

• Support sparse format data
• Support early-stopping mechanism
• Support setting arbitrary metrics for validation during training
• Support stepwise. For details on stepwise mode, please refer stepwise.

• Support batch shuffle and batch masked strategy.

• Hetero-SSHE-LR extra features

  1. Support different encrypt-mode to balance speed and security
  2. Support OneVeRest
  3. Support early-stopping mechanism
  4. Support setting arbitrary metrics for validation during training
  5. Support model encryption with host model

---

Last update: 2022-07-12