跳转至

Federated Logistic Regression

Logistic Regression(LR) is a widely used statistic model for classification problems. FATE provided two modes of federated LR: Homogeneous LR (HomoLR) and Heterogeneous LR (HeteroLR and Hetero_SSHE_LR).

Below lists features of each LR models:

Linear Model Multiclass(OVR) Arbiter-less Training Weighted Training Multi-Host Cross Validation Warm-Start/CheckPoint
Hetero LR
Hetero SSHELR
Homo LR

We simplified the federation process into three parties. Party A represents Guest, party B represents Host while party C, which also known as "Arbiter", is a third party that holds a private key for each party and work as a coordinator. (Hetero_SSHE_LR have not "Arbiter" role)

Homogeneous LR

As the name suggested, in HomoLR, the feature spaces of guest and hosts are identical. An optional encryption mode for computing gradients is provided for host parties. By doing this, the plain model is not available for this host any more.

Figure 1 (Federated HomoLR Principle)

Models of Party A and Party B have the same structure. In each iteration, each party trains its model on its own data. After that, all parties upload their encrypted (or plain, depends on your configuration) gradients to arbiter. The arbiter aggregates these gradients to form a federated gradient that will then be distributed to all parties for updating their local models. Similar to traditional LR, the training process will stop when the federated model converges or the whole training process reaches a predefined max-iteration threshold. More details is available in this Practical Secure Aggregation for Privacy-Preserving Machine Learning.

Heterogeneous LR

The HeteroLR carries out the federated learning in a different way. As shown in Figure 2, A sample alignment process is conducted before training. This sample alignment process is to identify overlapping samples stored in databases of the two involved parties. The federated model is built based on those overlapping samples. The whole sample alignment process will not leak confidential information (e.g., sample ids) on the two parties since it is conducted in an encrypted way.

Figure 2 (Federated HeteroLR Principle)

In the training process, party A and party B compute out the elements needed for final gradients. Arbiter aggregate them and compute out the gradient and then transfer back to each party. More details is available in this: Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption.

Multi-host hetero-lr

For multi-host scenario, the gradient computation still keep the same as single-host case. However, we use the second-norm of the difference of model weights between two consecutive iterations as the convergence criterion. Since the arbiter can obtain the completed model weight, the convergence decision is happening in Arbiter.

Figure 3 (Federated Multi-host HeteroLR
Principle)

Heterogeneous SSHE Logistic Regression

FATE implements a heterogeneous logistic regression without arbiter role called for hetero_sshe_lr. More details is available in this following paper: When Homomorphic Encryption Marries Secret Sharing: Secure Large-Scale Sparse Logistic Regression and Applications in Risk Control. We have also made some optimization so that the code may not exactly same with this paper. The training process could be described as the following: forward and backward process. Figure 3 (forward) Figure 4 (backward)

The training process is based secure matrix multiplication protocol(SMM), which HE and Secret-Sharing hybrid protocol is included. Figure 5 (SMM)

Param

logistic_regression_param

Classes

LogisticParam(penalty='L2', tol=0.0001, alpha=1.0, optimizer='rmsprop', batch_size=-1, shuffle=True, batch_strategy='full', masked_rate=5, learning_rate=0.01, init_param=InitParam(), max_iter=100, early_stop='diff', encrypt_param=EncryptParam(), predict_param=PredictParam(), cv_param=CrossValidationParam(), decay=1, decay_sqrt=True, multi_class='ovr', validation_freqs=None, early_stopping_rounds=None, stepwise_param=StepwiseParam(), floating_point_precision=23, metrics=None, use_first_metric_only=False, callback_param=CallbackParam())

Bases: LinearModelParam

Parameters used for Logistic Regression both for Homo mode or Hetero mode.

Parameters:

Name Type Description Default
penalty

Penalty method used in LR. Please note that, when using encrypted version in HomoLR, 'L1' is not supported.

'L2'
tol float, default

The tolerance of convergence

0.0001
alpha float, default

Regularization strength coefficient.

1.0
optimizer

Optimize method.

'rmsprop'
batch_strategy str

Strategy to generate batch data. a) full: use full data to generate batch_data, batch_nums every iteration is ceil(data_size / batch_size) b) random: select data randomly from full data, batch_num will be 1 every iteration.

'full'
batch_size int, default

Batch size when updating model. -1 means use all data in a batch. i.e. Not to use mini-batch strategy.

-1
shuffle bool, default

Work only in hetero logistic regression, batch data will be shuffle in every iteration.

True
masked_rate

Use masked data to enhance security of hetero logistic regression

5
learning_rate float, default

Learning rate

0.01
max_iter int, default

The maximum iteration for training.

100
early_stop

Method used to judge converge or not. a) diff: Use difference of loss between two iterations to judge whether converge. b) weight_diff: Use difference between weights of two consecutive iterations c) abs: Use the absolute value of loss to judge whether converge. i.e. if loss < eps, it is converged.

Please note that for hetero-lr multi-host situation, this parameter support "weight_diff" only. In homo-lr, weight_diff is not supported

'diff'
decay

Decay rate for learning rate. learning rate will follow the following decay schedule. lr = lr0/(1+decay*t) if decay_sqrt is False. If decay_sqrt is True, lr = lr0 / sqrt(1+decay*t) where t is the iter number.

1
decay_sqrt

lr = lr0/(1+decay*t) if decay_sqrt is False, otherwise, lr = lr0 / sqrt(1+decay*t)

True
encrypt_param

encrypt param

EncryptParam()
predict_param

predict param

PredictParam()
callback_param

callback param

CallbackParam()
cv_param

cv param

CrossValidationParam()
multi_class

If it is a multi_class task, indicate what strategy to use. Currently, support 'ovr' short for one_vs_rest only.

'ovr'
validation_freqs

validation frequency during training.

None
early_stopping_rounds

Will stop training if one metric doesn’t improve in last early_stopping_round rounds

None
metrics

Indicate when executing evaluation during train process, which metrics will be used. If set as empty, default metrics for specific task type will be used. As for binary classification, default metrics are ['auc', 'ks']

None
use_first_metric_only

Indicate whether use the first metric only for early stopping judgement.

False
floating_point_precision

if not None, use floating_point_precision-bit to speed up calculation, e.g.: convert an x to round(x * 2**floating_point_precision) during Paillier operation, divide the result by 2**floating_point_precision in the end.

23
Source code in federatedml/param/logistic_regression_param.py
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
def __init__(self, penalty='L2',
             tol=1e-4, alpha=1.0, optimizer='rmsprop',
             batch_size=-1, shuffle=True, batch_strategy="full", masked_rate=5,
             learning_rate=0.01, init_param=InitParam(),
             max_iter=100, early_stop='diff', encrypt_param=EncryptParam(),
             predict_param=PredictParam(), cv_param=CrossValidationParam(),
             decay=1, decay_sqrt=True,
             multi_class='ovr', validation_freqs=None, early_stopping_rounds=None,
             stepwise_param=StepwiseParam(), floating_point_precision=23,
             metrics=None,
             use_first_metric_only=False,
             callback_param=CallbackParam()
             ):
    super(LogisticParam, self).__init__()
    self.penalty = penalty
    self.tol = tol
    self.alpha = alpha
    self.optimizer = optimizer
    self.batch_size = batch_size
    self.learning_rate = learning_rate
    self.init_param = copy.deepcopy(init_param)
    self.max_iter = max_iter
    self.early_stop = early_stop
    self.encrypt_param = encrypt_param
    self.shuffle = shuffle
    self.batch_strategy = batch_strategy
    self.masked_rate = masked_rate
    self.predict_param = copy.deepcopy(predict_param)
    self.cv_param = copy.deepcopy(cv_param)
    self.decay = decay
    self.decay_sqrt = decay_sqrt
    self.multi_class = multi_class
    self.validation_freqs = validation_freqs
    self.stepwise_param = copy.deepcopy(stepwise_param)
    self.early_stopping_rounds = early_stopping_rounds
    self.metrics = metrics or []
    self.use_first_metric_only = use_first_metric_only
    self.floating_point_precision = floating_point_precision
    self.callback_param = copy.deepcopy(callback_param)
Attributes
penalty = penalty instance-attribute
tol = tol instance-attribute
alpha = alpha instance-attribute
optimizer = optimizer instance-attribute
batch_size = batch_size instance-attribute
learning_rate = learning_rate instance-attribute
init_param = copy.deepcopy(init_param) instance-attribute
max_iter = max_iter instance-attribute
early_stop = early_stop instance-attribute
encrypt_param = encrypt_param instance-attribute
shuffle = shuffle instance-attribute
batch_strategy = batch_strategy instance-attribute
masked_rate = masked_rate instance-attribute
predict_param = copy.deepcopy(predict_param) instance-attribute
cv_param = copy.deepcopy(cv_param) instance-attribute
decay = decay instance-attribute
decay_sqrt = decay_sqrt instance-attribute
multi_class = multi_class instance-attribute
validation_freqs = validation_freqs instance-attribute
stepwise_param = copy.deepcopy(stepwise_param) instance-attribute
early_stopping_rounds = early_stopping_rounds instance-attribute
metrics = metrics or [] instance-attribute
use_first_metric_only = use_first_metric_only instance-attribute
floating_point_precision = floating_point_precision instance-attribute
callback_param = copy.deepcopy(callback_param) instance-attribute
Functions
check()
Source code in federatedml/param/logistic_regression_param.py
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
def check(self):
    descr = "logistic_param's"
    super(LogisticParam, self).check()
    self.predict_param.check()
    if self.encrypt_param.method not in [consts.PAILLIER, consts.PAILLIER_IPCL, None]:
        raise ValueError(
            "logistic_param's encrypted method support 'Paillier' or None only")
    self.multi_class = self.check_and_change_lower(
        self.multi_class, ["ovr"], f"{descr}")
    if not isinstance(self.masked_rate, (float, int)) or self.masked_rate < 0:
        raise ValueError(
            "masked rate should be non-negative numeric number")
    if not isinstance(self.batch_strategy, str) or self.batch_strategy.lower() not in ["full", "random"]:
        raise ValueError("batch strategy should be full or random")
    self.batch_strategy = self.batch_strategy.lower()
    if not isinstance(self.shuffle, bool):
        raise ValueError("shuffle should be boolean type")
    return True
HomoLogisticParam(penalty='L2', tol=0.0001, alpha=1.0, optimizer='rmsprop', batch_size=-1, learning_rate=0.01, init_param=InitParam(), max_iter=100, early_stop='diff', predict_param=PredictParam(), cv_param=CrossValidationParam(), decay=1, decay_sqrt=True, aggregate_iters=1, multi_class='ovr', validation_freqs=None, metrics=['auc', 'ks'], callback_param=CallbackParam())

Bases: LogisticParam

Parameters:

Name Type Description Default
aggregate_iters int, default

Indicate how many iterations are aggregated once.

1
Source code in federatedml/param/logistic_regression_param.py
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
def __init__(self, penalty='L2',
             tol=1e-4, alpha=1.0, optimizer='rmsprop',
             batch_size=-1, learning_rate=0.01, init_param=InitParam(),
             max_iter=100, early_stop='diff',
             predict_param=PredictParam(), cv_param=CrossValidationParam(),
             decay=1, decay_sqrt=True,
             aggregate_iters=1, multi_class='ovr', validation_freqs=None,
             metrics=['auc', 'ks'],
             callback_param=CallbackParam()
             ):

    super(HomoLogisticParam, self).__init__(penalty=penalty, tol=tol, alpha=alpha, optimizer=optimizer,
                                            batch_size=batch_size,
                                            learning_rate=learning_rate,
                                            init_param=init_param, max_iter=max_iter, early_stop=early_stop,
                                            predict_param=predict_param,
                                            cv_param=cv_param, multi_class=multi_class,
                                            validation_freqs=validation_freqs,
                                            decay=decay, decay_sqrt=decay_sqrt,
                                            metrics=metrics,
                                            callback_param=callback_param)
    self.aggregate_iters = aggregate_iters
Attributes
aggregate_iters = aggregate_iters instance-attribute
Functions
check()
Source code in federatedml/param/logistic_regression_param.py
194
195
196
197
198
199
200
201
202
def check(self):

    super().check()
    if not isinstance(self.aggregate_iters, int):
        raise ValueError(
            "logistic_param's aggregate_iters {} not supported, should be int type".format(
                self.aggregate_iters))

    return True
HeteroLogisticParam(penalty='L2', tol=0.0001, alpha=1.0, optimizer='rmsprop', batch_size=-1, shuffle=True, batch_strategy='full', masked_rate=5, learning_rate=0.01, init_param=InitParam(), max_iter=100, early_stop='diff', encrypted_mode_calculator_param=EncryptedModeCalculatorParam(), predict_param=PredictParam(), cv_param=CrossValidationParam(), decay=1, decay_sqrt=True, sqn_param=StochasticQuasiNewtonParam(), multi_class='ovr', validation_freqs=None, early_stopping_rounds=None, metrics=['auc', 'ks'], floating_point_precision=23, encrypt_param=EncryptParam(), use_first_metric_only=False, stepwise_param=StepwiseParam(), callback_param=CallbackParam())

Bases: LogisticParam

Source code in federatedml/param/logistic_regression_param.py
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
def __init__(self, penalty='L2',
             tol=1e-4, alpha=1.0, optimizer='rmsprop',
             batch_size=-1, shuffle=True, batch_strategy="full", masked_rate=5,
             learning_rate=0.01, init_param=InitParam(),
             max_iter=100, early_stop='diff',
             encrypted_mode_calculator_param=EncryptedModeCalculatorParam(),
             predict_param=PredictParam(), cv_param=CrossValidationParam(),
             decay=1, decay_sqrt=True, sqn_param=StochasticQuasiNewtonParam(),
             multi_class='ovr', validation_freqs=None, early_stopping_rounds=None,
             metrics=['auc', 'ks'], floating_point_precision=23,
             encrypt_param=EncryptParam(),
             use_first_metric_only=False, stepwise_param=StepwiseParam(),
             callback_param=CallbackParam()
             ):
    super(HeteroLogisticParam, self).__init__(penalty=penalty, tol=tol, alpha=alpha, optimizer=optimizer,
                                              batch_size=batch_size, shuffle=shuffle, batch_strategy=batch_strategy,
                                              masked_rate=masked_rate,
                                              learning_rate=learning_rate,
                                              init_param=init_param, max_iter=max_iter, early_stop=early_stop,
                                              predict_param=predict_param, cv_param=cv_param,
                                              decay=decay,
                                              decay_sqrt=decay_sqrt, multi_class=multi_class,
                                              validation_freqs=validation_freqs,
                                              early_stopping_rounds=early_stopping_rounds,
                                              metrics=metrics, floating_point_precision=floating_point_precision,
                                              encrypt_param=encrypt_param,
                                              use_first_metric_only=use_first_metric_only,
                                              stepwise_param=stepwise_param,
                                              callback_param=callback_param)
    self.encrypted_mode_calculator_param = copy.deepcopy(
        encrypted_mode_calculator_param)
    self.sqn_param = copy.deepcopy(sqn_param)
Attributes
encrypted_mode_calculator_param = copy.deepcopy(encrypted_mode_calculator_param) instance-attribute
sqn_param = copy.deepcopy(sqn_param) instance-attribute
Functions
check()
Source code in federatedml/param/logistic_regression_param.py
239
240
241
242
243
def check(self):
    super().check()
    self.encrypted_mode_calculator_param.check()
    self.sqn_param.check()
    return True

Functions

Features

  • Both Homo-LR and Hetero-LR
  1. L1 & L2 regularization

  2. Mini-batch mechanism

  3. Weighted training

  4. Six optimization method:

    • sgd
      gradient descent with arbitrary batch size

    • rmsprop
      RMSProp

    • adam
      Adam

    • adagrad
      AdaGrad

    • nesterov_momentum_sgd
      Nesterov Momentum

  5. Three converge criteria:

    • diff
      Use difference of loss between two iterations, not available for multi-host training;

    • abs
      use the absolute value of loss;

    • weight_diff
      use difference of model weights

  6. Support multi-host modeling task.

  7. Support validation for every arbitrary iterations

  8. Learning rate decay mechanism

  • Homo-LR extra features
  1. Two Encryption mode

    • "Paillier" mode
      Host will not get clear text model. When using encryption mode, "sgd" optimizer is supported only.

    • Non-encryption mode
      Everything is in clear text.

  2. Secure aggregation mechanism used when more aggregating models

  3. Support aggregate for every arbitrary iterations.

  4. Support FedProx mechanism. More details is available in this Federated Optimization in Heterogeneous Networks.

  • Hetero-LR extra features

  • Support different encrypt-mode to balance speed and security

  • Support OneVeRest
  • When modeling a multi-host task, "weight_diff" converge criteria
    is supported only.
    
  • Support sparse format data
  • Support early-stopping mechanism
  • Support setting arbitrary metrics for validation during training
  • Support stepwise. For details on stepwise mode, please refer stepwise.
  • Support batch shuffle and batch masked strategy.

  • Hetero-SSHE-LR extra features

    1. Support different encrypt-mode to balance speed and security
    2. Support OneVeRest
    3. Support early-stopping mechanism
    4. Support setting arbitrary metrics for validation during training
    5. Support model encryption with host model

最后更新: 2022-07-12