Homogeneous Neural Networks¶

Neural networks are probably the most popular machine learning algorithms in recent years. FATE provides a federated homogeneous neural network implementation. We simplified the federation process into three parties. Party A represents Guest，which acts as a task trigger. Party B represents Host, which is almost the same with guest except that Host does not initiate task. Party C serves as a coordinator to aggregate models from guest/hosts and broadcast aggregated model.

Basic Process¶

As its name suggested, in Homogeneous Neural Networks, the feature spaces of guest and hosts are identical. An optional encryption mode for model is provided. By doing this, no party can get the private model of other parties.

_build_temp/python/doc/images/homo_nn.png

Figure 1 (Federated Homo NN Principle)¶

The Homo NN process is shown in Figure 1. Models of Party A and Party B have the same neural networks structure. In each iteration, each party trains its model on its own data. After that, all parties upload their encrypted (with random mask) model parameters to arbiter. The arbiter aggregates these parameters to form a federated model parameter, which will then be distributed to all parties for updating their local models. Similar to traditional neural network, the training process will stop when the federated model converges or the whole training process reaches a predefined max-iteration threshold.

Please note that random numbers are carefully generated so that the random numbers of all parties add up an zero matrix and thus disappear automatically. For more detailed explanations, please refer to [Secure Analytics: Federated Learning and Secure Aggregation]. Since there is no model transferred in plaintext, except for the owner of the model, no other party can obtain the real information of the model.

Param¶

class HomoNNParam(api_version: int = 0, secure_aggregate: bool = True, aggregate_every_n_epoch: int = 1, config_type: str = 'nn', nn_define: Optional[dict] = None, optimizer: Union[str, dict, types.SimpleNamespace] = 'SGD', loss: Optional[str] = None, metrics: Optional[Union[str, list]] = None, max_iter: int = 100, batch_size: int = -1, early_stop: Union[str, dict, types.SimpleNamespace] = 'diff', encode_label: bool = False, predict_param=<federatedml.param.predict_param.PredictParam object>, cv_param=<federatedml.param.cross_validation_param.CrossValidationParam object>)¶

Parameters used for Homo Neural Network.

Parameters

Args –

secure_aggregate: enable secure aggregation or not, defaults to True. aggregate_every_n_epoch: aggregate model every n epoch, defaults to 1. config_type: one of “nn”, “keras”, “tf” nn_define: a dict represents the structure of neural network. optimizer: optimizer method, accept following types:

a string, one of “Adadelta”, “Adagrad”, “Adam”, “Adamax”, “Nadam”, “RMSprop”, “SGD”

a dict, with a required key-value pair keyed by “optimizer”,
with optional key-value pairs such as learning rate.

defaults to “SGD”

loss: a string metrics: max_iter: the maximum iteration for aggregation in training. batch_size : batch size when updating model.

-1 means use all data in a batch. i.e. Not to use mini-batch strategy. defaults to -1.

early_stopstr, ‘diff’, ‘weight_diff’ or ‘abs’, default: ‘diff’

Method used to judge converge or not.

diff： Use difference of loss between two iterations to judge whether converge.
weight_diff: Use difference between weights of two consecutive iterations
abs: Use the absolute value of loss to judge whether converge. i.e. if loss < eps, it is converged.

encode_label : encode label to one_hot.

Features¶

tensorflow backend

supported layers

Dense

{
    "layer": "Dense",
    "units": ,
    "activation": null,
    "use_bias": true,
    "kernel_initializer": "glorot_uniform",
    "bias_initializer": "zeros",
    "kernel_regularizer": null,
    "bias_regularizer": null,
    "activity_regularizer": null,
    "kernel_constraint": null,
    "bias_constraint": null
  }

Droupout

{
    "rate": ,
    "noise_shape": null,
    "seed": null
}

other layers listed in tf.keras.layers will be supported in near feature.

supported optimizer

all optimizer listed in tf.keras.optimizers

Adadelta

adadelta link

{
      "optimizer": "Adadelta",
      "learning_rate": 0.001,
      "rho": 0.95,
      "epsilon": 1e-07
}

Adagrad

adagrad link

{
      "optimizer": "Adagrad",
      "learning_rate": 0.001,
      "initial_accumulator_value": 0.1,
      "epsilon": 1e-07
}

Adam

adam link

{
      "optimizer": "Adam",
      "learning_rate": 0.001,
      "beta_1": 0.9,
      "beta_2": 0.999,
      "amsgrad": false,
      "epsilon": 1e-07
}

Ftrl

ftrl link

{
      "optimizer": "Ftrl",
      "learning_rate": 0.001,
      "learning_rate_power": -0.5,
      "initial_accumulator_value": 0.1,
      "l1_regularization_strength": 0.0,
      "l2_regularization_strength": 0.0,
      "l2_shrinkage_regularization_strength": 0.0
}

Nadam

nadam link

{
      "optimizer": "Nadam",
      "learning_rate": 0.001,
      "beta_1": 0.9,
      "beta_2": 0.999,
      "epsilon": 1e-07
}

RMSprop

rmsprop link

{
      "optimizer": "RMSprop",
      "learning_rate": 0.001,
      "pho": 0.9,
      "momentum": 0.0,
      "epsilon": 1e-07,
      "centered": false
}

SGD

sgd link

{
      "optimizer": "SGD",
      "learning_rate": 0.001,
      "momentum": 0.0,
      "nesterov": false
}

supported losses

all losses listed in tf.keras.losses

binary_crossentropy
categorical_crossentropy
categorical_hinge
cosine_similarity
hinge
kullback_leibler_divergence
logcosh
mean_absolute_error
mean_absolute_percentage_error
mean_squared_error
mean_squared_logarithmic_error
poisson
sparse_categorical_crossentropy
squared_hinge

support multi-host

In fact, for model security reasons, at least two host parties are required.

pytorch backend

There are some difference in nn configuration build by pytorch compared to tf or keras.

config_type

pytorch, if use pytorch to build your model

nn_define

Each layer is represented as an object in json.

supported layers

Linear

{
"layer": "Linear",
"name": #string,
"type": "normal",
"config": [input_num,output_num]
}

other normal layers

BatchNorm2d
dropout

activate

{
  "layer": "Relu",
  "type": "activate",
  "name": #string
}

other activate layers

Selu
LeakyReLU
Tanh
Sigmoid
Relu
Tanh

optimizer

A json object is needed

"optimizer": {
  "optimizer": "Adam",
  "learning_rate": 0.05
}

optimizer include “Adam”,”SGD”,”RMSprop”,”Adagrad”

loss

A string is needed, supported losses include:

“CrossEntropyLoss”
“MSELoss”
“BCELoss”
“BCEWithLogitsLoss”
“NLLLoss”
“L1Loss”
“SmoothL1Loss”
“HingeEmbeddingLoss”

metrics

A string is needed, supported metrics include:

auccuray
precision
recall
auc
f1
fbeta

Use¶

Note

For more information on task configuration, please refer to the [doc] under example first. In this part we only talk about the parameter configuration.

Since all parties training Homogeneous Neural Networks have the same network structure, a common practice is to configure parameters under algorithm_parameters, which is shared across all parties. The basic structure is:

{
      "config_type": "nn",
      "nn_define": [layer1, layer2, ...]
      "batch_size": -1,
      "optimizer": optimizer,
      "early_stop": {
        "early_stop": early_stop_type,
        "eps": 1e-4
      },
      "loss": loss,
      "metrics": [metrics1, metrics2, ...],
      "max_iter": 10
}

Note

Some detailed examples can be found in the example directory

nn_define

Each layer is represented as an object in json. Please refer to supported layers in Features part.

optimizer

A json object is needed, please refer to supported optimizers in Features part.

loss

A string is needed, please refer to supported losses in Features part.

others

batch_size: a positive integer or -1 for full batch
max_iter: max aggregation number, a positive integer
early_stop: diff or abs
metrics: a string name, refer to [metricsdoc], such as Accuracy, AUC …