Hetero-NN Customize your Dataset¶
The FATE system primarily supports tabular data as its standard data format. However, it is possible to utilize non-tabular data, such as images, text, mixed data, or relational data, in neural networks through the use of the Dataset feature of the NN module. The Dataset module within the NN module allows for the customization of datasets so that user can use them in more complex data scenarios. This tutorial will cover the use of the Dataset feature in the Hetero-NN. For the ease of demonstration, We will use the MNIST handwriting recognition dataset as an example to simulate a Hetero Federation task to illustrate these concepts.
Prepare MNIST Data¶
Please download the guest/host MNIST dataset from the link below and place it in the project examples/data folder:
guest data: https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/fate/examples/data/mnist_guest.zip
host data: https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/fate/examples/data/mnist_host.zip
The mnist_guest is a simplified version of the MNIST dataset, with a total of ten categories, which are classified into 0-9 10 folders according to labels. The mnist_host has the same images as the mnist_guest, but it is not labeled.
! ls ../../../../examples/data/mnist_guest
0 1 2 3 4 5 6 7 8 9
! ls ../../../../examples/data/mnist_host
not_labeled
Dataset¶
In version FATE-1.10, FATE introduces a new base class for datasets called Dataset, which is based on PyTorch's Dataset class. This class allows users to create custom datasets according to their specific needs. The usage is similar to that of PyTorch's Dataset class, with the added requirement of implementing two additional interfaces when using FATE-NN for data reading and training: load() and get_sample_ids().
To create a custom dataset in Hetero-NN, users need to:
- Develop a new dataset class that inherits from the Dataset class
- Implement the __len__() and __getitem__() methods, which are consistent with PyTorch's Dataset usage. The __len__() method should return the length of the dataset, while the __getitem__() method should return the corresponding data at the specified index. However, please notice that different __getitem__() methods may have different behaviors between different parties. In the guest party(party with labels), __getitem__() method return features and labels, while in the host parties(parties without label), __getitem__() method return features only.
- Implement the load(), get_sample_ids(), get_classes() methods
For those unfamiliar with PyTorch's Dataset class, more information can be found in the PyTorch documentation: Pytorch Dataset Documentation
load()¶
The first additional interface required is load(). This interface receives a file path and allows users to read data directly from the local file system. When submitting a task, the data path can be specified through the reader component. Hetero-NN will use the user-specified Dataset class, utilizing the load() interface to read data from the specified path and complete the loading of the dataset for training. For more information, please refer to the source code in /federatedml/nn/dataset/base.py.
get_sample_ids()¶
The second additional interface is get_sample_ids(). This interface should return a list of sample IDs, which can be either integers or strings and should have the same length as the dataset. This function is important in Hetero-NN, there are some points you need to know:
- When using a customized dataset in Hetero-NN, it is important to ensure that your sample IDs are aligned with those of other parties. You can do this by using the intersection component and extracting the results, or by agreeing on the sample IDs to be used with the other parties.
- You don't have to put your ids in order, Hetero-NN component will sort them.
get_classes¶
The third function return a list of all unique labels. It will be called in the guest party. If it is not a classification task, just return an empty list.
Example: Implement a simple image dataset¶
In order to better understand the customization of Dataset, here we implement a simple image dataset to read MNIST images, and then complete a federated image classification task in a vertical scene. For convenience here, we use the jupyter interface of save_to_fate to update the code to federatedml.nn.dataset, named mnist_dataset.py, of course you can manually copy the code file to the directory.
- This dataset has a parameter 'return_label', when guest party(party with label) is using it, we set return_label=True, else return_label=False
- It is developed based on ImageFolder, and we take image name as the sample id.
from pipeline.component.nn import save_to_fate
%%save_to_fate dataset mnist_dataset.py
import numpy as np
from federatedml.nn.dataset.base import Dataset
from torchvision.datasets import ImageFolder
from torchvision import transforms
class MNISTDataset(Dataset):
def __init__(self, return_label=True):
super(MNISTDataset, self).__init__()
self.return_label = return_label
self.image_folder = None
self.ids = None
def load(self, path):
self.image_folder = ImageFolder(root=path, transform=transforms.Compose([transforms.ToTensor()]))
ids = []
for image_name in self.image_folder.imgs:
ids.append(image_name[0].split('/')[-1].replace('.jpg', ''))
self.ids = ids
return self
def get_sample_ids(self, ):
return self.ids
def get_classes(self, ):
return np.unique(self.image_folder.targets).tolist()
def __len__(self,):
return len(self.image_folder)
def __getitem__(self, idx): # get item
ret = self.image_folder[idx]
img = ret[0][0].flatten() # flatten tensor 784 dims
if self.return_label:
return img, ret[1] # img & label
else:
return img # no label, for host
Now we test our dataset class:
# guest party
! ls ../../../../examples/data/mnist_guest
ds = MNISTDataset().load('../../../../examples/data/mnist_guest/')
print(len(ds))
print(ds[0][0])
print(ds.get_classes())
print(ds.get_sample_ids()[0: 10])
0 1 2 3 4 5 6 7 8 9 1309 tensor([0.0118, 0.0000, 0.0000, 0.0118, 0.0275, 0.0118, 0.0000, 0.0118, 0.0000, 0.0431, 0.0000, 0.0000, 0.0118, 0.0000, 0.0000, 0.0118, 0.0314, 0.0000, 0.0000, 0.0118, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0196, 0.0000, 0.0471, 0.0000, 0.0627, 0.0000, 0.0000, 0.0157, 0.0000, 0.0078, 0.0314, 0.0118, 0.0000, 0.0157, 0.0314, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0000, 0.0039, 0.0078, 0.0039, 0.0471, 0.0000, 0.0314, 0.0000, 0.0000, 0.0235, 0.0000, 0.0431, 0.0000, 0.0000, 0.0235, 0.0275, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0118, 0.0000, 0.0000, 0.0078, 0.0118, 0.0000, 0.0000, 0.0000, 0.0471, 0.0000, 0.0000, 0.0902, 0.0000, 0.0000, 0.0000, 0.0000, 0.0431, 0.0118, 0.0000, 0.0000, 0.0157, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0039, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0235, 0.0000, 0.0980, 0.1059, 0.5333, 0.5294, 0.7373, 0.3490, 0.3294, 0.0980, 0.0000, 0.0000, 0.0118, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0157, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.3451, 0.9686, 0.9255, 1.0000, 0.9765, 0.9804, 0.8902, 0.9412, 0.5333, 0.1451, 0.0039, 0.0000, 0.0078, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0118, 0.0000, 0.0000, 0.0157, 0.1059, 0.7569, 0.9843, 0.9922, 1.0000, 1.0000, 1.0000, 1.0000, 0.9412, 0.9961, 1.0000, 0.8353, 0.3490, 0.0000, 0.0000, 0.0549, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.0000, 0.0706, 0.2196, 0.9647, 1.0000, 0.9922, 0.9529, 0.9843, 1.0000, 0.9608, 1.0000, 1.0000, 0.9961, 1.0000, 0.9059, 0.4667, 0.0275, 0.0000, 0.0196, 0.0000, 0.0000, 0.0000, 0.0000, 0.0157, 0.0000, 0.0000, 0.0471, 0.0510, 0.0000, 0.2549, 0.7451, 0.9647, 1.0000, 1.0000, 0.9843, 1.0000, 0.4275, 0.3451, 0.7804, 1.0000, 0.9686, 0.9804, 1.0000, 0.9176, 0.3608, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0392, 0.0039, 0.0000, 0.0000, 0.0706, 0.6392, 0.9725, 1.0000, 0.9216, 0.8471, 0.5882, 0.5020, 0.1765, 0.0235, 0.0314, 0.0863, 0.8314, 1.0000, 1.0000, 0.9882, 0.6745, 0.0000, 0.0588, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0157, 0.0196, 0.0000, 0.0000, 0.7333, 1.0000, 0.9961, 0.3686, 0.2235, 0.0275, 0.0039, 0.0000, 0.0235, 0.0000, 0.0000, 0.5451, 0.9490, 1.0000, 1.0000, 0.8549, 0.2431, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0196, 0.0078, 0.0000, 0.0000, 0.0431, 0.2196, 0.9882, 0.9216, 0.9922, 0.0784, 0.0196, 0.0078, 0.0196, 0.0039, 0.0000, 0.0039, 0.0078, 0.0000, 0.3804, 0.9765, 0.9725, 0.9765, 0.6510, 0.0314, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.2745, 1.0000, 1.0000, 0.9608, 0.0980, 0.0392, 0.0000, 0.0000, 0.0039, 0.0000, 0.0157, 0.0392, 0.0000, 0.0392, 1.0000, 0.9647, 0.9804, 0.6078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0275, 0.0471, 0.0000, 0.3412, 0.8863, 1.0000, 0.7216, 0.0000, 0.0118, 0.0000, 0.0392, 0.0196, 0.0000, 0.0000, 0.0000, 0.0353, 0.0000, 0.7176, 0.9843, 1.0000, 0.8706, 0.0588, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0196, 0.0039, 0.0000, 0.0745, 0.9020, 1.0000, 0.9529, 1.0000, 0.1373, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0353, 0.0314, 0.0000, 0.0000, 0.2745, 0.9608, 0.9490, 1.0000, 0.0549, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0157, 0.0118, 0.0000, 0.0745, 0.9843, 0.9373, 1.0000, 0.9686, 0.1176, 0.0039, 0.0000, 0.0157, 0.0157, 0.0549, 0.0000, 0.0000, 0.0078, 0.0000, 0.1843, 1.0000, 1.0000, 0.9686, 0.0824, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.0078, 0.0078, 0.0000, 0.6784, 0.9686, 0.9882, 0.9804, 0.1098, 0.0392, 0.0000, 0.0000, 0.0314, 0.0000, 0.0000, 0.0000, 0.0314, 0.0000, 0.2627, 0.9765, 1.0000, 1.0000, 0.0471, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0118, 0.0000, 0.3451, 1.0000, 0.9843, 1.0000, 0.7373, 0.0824, 0.0000, 0.0588, 0.0000, 0.0314, 0.0078, 0.0627, 0.0000, 0.1373, 0.7843, 0.9686, 0.9843, 0.5255, 0.0157, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0118, 0.0039, 0.0000, 0.0431, 0.8275, 0.9686, 0.9765, 1.0000, 0.7412, 0.2980, 0.0000, 0.0000, 0.0157, 0.0000, 0.0078, 0.0000, 0.6627, 1.0000, 1.0000, 0.9686, 0.1843, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.0000, 0.0078, 0.0000, 0.2314, 0.8039, 1.0000, 0.9412, 1.0000, 0.7137, 0.1608, 0.2196, 0.1098, 0.1294, 0.1647, 0.9373, 0.9647, 0.9843, 0.9333, 0.6157, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0039, 0.0000, 0.0000, 0.0078, 0.0392, 0.0000, 0.4078, 0.9373, 1.0000, 0.9412, 1.0000, 0.9922, 0.9686, 0.9294, 1.0000, 1.0000, 0.9804, 1.0000, 0.9373, 1.0000, 0.3922, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0000, 0.0118, 0.0000, 0.0000, 0.0275, 0.0000, 0.0157, 0.4471, 1.0000, 1.0000, 1.0000, 1.0000, 0.9686, 0.9765, 0.9922, 0.9843, 0.9961, 0.9294, 0.9843, 0.3490, 0.0000, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0353, 0.0000, 0.0000, 0.0039, 0.0510, 0.0000, 0.0549, 0.6549, 1.0000, 0.9647, 0.9922, 1.0000, 1.0000, 0.9961, 0.9490, 1.0000, 0.9569, 0.2392, 0.0000, 0.0745, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0039, 0.0275, 0.0000, 0.0000, 0.0157, 0.0000, 0.0549, 0.0000, 0.1059, 0.2392, 0.5608, 1.0000, 1.0000, 0.9882, 1.0000, 0.5843, 0.0824, 0.0235, 0.0627, 0.0000, 0.0000, 0.0275, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] ['img_1', 'img_1029', 'img_1046', 'img_1047', 'img_1076', 'img_108', 'img_1091', 'img_1093', 'img_1096', 'img_110']
# host party
! ls ../../../../examples/data/mnist_host # no label
ds = MNISTDataset(return_label=False).load('../../../../examples/data/mnist_host')
print(len(ds))
print(ds[0]) # no label
not_labeled 1309 tensor([0.0118, 0.0000, 0.0000, 0.0118, 0.0275, 0.0118, 0.0000, 0.0118, 0.0000, 0.0431, 0.0000, 0.0000, 0.0118, 0.0000, 0.0000, 0.0118, 0.0314, 0.0000, 0.0000, 0.0118, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0196, 0.0000, 0.0471, 0.0000, 0.0627, 0.0000, 0.0000, 0.0157, 0.0000, 0.0078, 0.0314, 0.0118, 0.0000, 0.0157, 0.0314, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0000, 0.0039, 0.0078, 0.0039, 0.0471, 0.0000, 0.0314, 0.0000, 0.0000, 0.0235, 0.0000, 0.0431, 0.0000, 0.0000, 0.0235, 0.0275, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0118, 0.0000, 0.0000, 0.0078, 0.0118, 0.0000, 0.0000, 0.0000, 0.0471, 0.0000, 0.0000, 0.0902, 0.0000, 0.0000, 0.0000, 0.0000, 0.0431, 0.0118, 0.0000, 0.0000, 0.0157, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0039, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0235, 0.0000, 0.0980, 0.1059, 0.5333, 0.5294, 0.7373, 0.3490, 0.3294, 0.0980, 0.0000, 0.0000, 0.0118, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0157, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.3451, 0.9686, 0.9255, 1.0000, 0.9765, 0.9804, 0.8902, 0.9412, 0.5333, 0.1451, 0.0039, 0.0000, 0.0078, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.0118, 0.0000, 0.0000, 0.0157, 0.1059, 0.7569, 0.9843, 0.9922, 1.0000, 1.0000, 1.0000, 1.0000, 0.9412, 0.9961, 1.0000, 0.8353, 0.3490, 0.0000, 0.0000, 0.0549, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.0000, 0.0706, 0.2196, 0.9647, 1.0000, 0.9922, 0.9529, 0.9843, 1.0000, 0.9608, 1.0000, 1.0000, 0.9961, 1.0000, 0.9059, 0.4667, 0.0275, 0.0000, 0.0196, 0.0000, 0.0000, 0.0000, 0.0000, 0.0157, 0.0000, 0.0000, 0.0471, 0.0510, 0.0000, 0.2549, 0.7451, 0.9647, 1.0000, 1.0000, 0.9843, 1.0000, 0.4275, 0.3451, 0.7804, 1.0000, 0.9686, 0.9804, 1.0000, 0.9176, 0.3608, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0392, 0.0039, 0.0000, 0.0000, 0.0706, 0.6392, 0.9725, 1.0000, 0.9216, 0.8471, 0.5882, 0.5020, 0.1765, 0.0235, 0.0314, 0.0863, 0.8314, 1.0000, 1.0000, 0.9882, 0.6745, 0.0000, 0.0588, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0157, 0.0196, 0.0000, 0.0000, 0.7333, 1.0000, 0.9961, 0.3686, 0.2235, 0.0275, 0.0039, 0.0000, 0.0235, 0.0000, 0.0000, 0.5451, 0.9490, 1.0000, 1.0000, 0.8549, 0.2431, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0196, 0.0078, 0.0000, 0.0000, 0.0431, 0.2196, 0.9882, 0.9216, 0.9922, 0.0784, 0.0196, 0.0078, 0.0196, 0.0039, 0.0000, 0.0039, 0.0078, 0.0000, 0.3804, 0.9765, 0.9725, 0.9765, 0.6510, 0.0314, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0000, 0.2745, 1.0000, 1.0000, 0.9608, 0.0980, 0.0392, 0.0000, 0.0000, 0.0039, 0.0000, 0.0157, 0.0392, 0.0000, 0.0392, 1.0000, 0.9647, 0.9804, 0.6078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0000, 0.0275, 0.0471, 0.0000, 0.3412, 0.8863, 1.0000, 0.7216, 0.0000, 0.0118, 0.0000, 0.0392, 0.0196, 0.0000, 0.0000, 0.0000, 0.0353, 0.0000, 0.7176, 0.9843, 1.0000, 0.8706, 0.0588, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0196, 0.0039, 0.0000, 0.0745, 0.9020, 1.0000, 0.9529, 1.0000, 0.1373, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0353, 0.0314, 0.0000, 0.0000, 0.2745, 0.9608, 0.9490, 1.0000, 0.0549, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0157, 0.0118, 0.0000, 0.0745, 0.9843, 0.9373, 1.0000, 0.9686, 0.1176, 0.0039, 0.0000, 0.0157, 0.0157, 0.0549, 0.0000, 0.0000, 0.0078, 0.0000, 0.1843, 1.0000, 1.0000, 0.9686, 0.0824, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.0078, 0.0078, 0.0000, 0.6784, 0.9686, 0.9882, 0.9804, 0.1098, 0.0392, 0.0000, 0.0000, 0.0314, 0.0000, 0.0000, 0.0000, 0.0314, 0.0000, 0.2627, 0.9765, 1.0000, 1.0000, 0.0471, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0118, 0.0000, 0.3451, 1.0000, 0.9843, 1.0000, 0.7373, 0.0824, 0.0000, 0.0588, 0.0000, 0.0314, 0.0078, 0.0627, 0.0000, 0.1373, 0.7843, 0.9686, 0.9843, 0.5255, 0.0157, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0118, 0.0039, 0.0000, 0.0431, 0.8275, 0.9686, 0.9765, 1.0000, 0.7412, 0.2980, 0.0000, 0.0000, 0.0157, 0.0000, 0.0078, 0.0000, 0.6627, 1.0000, 1.0000, 0.9686, 0.1843, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0235, 0.0000, 0.0000, 0.0078, 0.0000, 0.2314, 0.8039, 1.0000, 0.9412, 1.0000, 0.7137, 0.1608, 0.2196, 0.1098, 0.1294, 0.1647, 0.9373, 0.9647, 0.9843, 0.9333, 0.6157, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0039, 0.0000, 0.0000, 0.0078, 0.0392, 0.0000, 0.4078, 0.9373, 1.0000, 0.9412, 1.0000, 0.9922, 0.9686, 0.9294, 1.0000, 1.0000, 0.9804, 1.0000, 0.9373, 1.0000, 0.3922, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0039, 0.0000, 0.0118, 0.0000, 0.0000, 0.0275, 0.0000, 0.0157, 0.4471, 1.0000, 1.0000, 1.0000, 1.0000, 0.9686, 0.9765, 0.9922, 0.9843, 0.9961, 0.9294, 0.9843, 0.3490, 0.0000, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0353, 0.0000, 0.0000, 0.0039, 0.0510, 0.0000, 0.0549, 0.6549, 1.0000, 0.9647, 0.9922, 1.0000, 1.0000, 0.9961, 0.9490, 1.0000, 0.9569, 0.2392, 0.0000, 0.0745, 0.0000, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0078, 0.0039, 0.0275, 0.0000, 0.0000, 0.0157, 0.0000, 0.0549, 0.0000, 0.1059, 0.2392, 0.5608, 1.0000, 1.0000, 0.9882, 1.0000, 0.5843, 0.0824, 0.0235, 0.0627, 0.0000, 0.0000, 0.0275, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000])
Good! It’s ready to use, so let’s use this developed dataset to run a Hetero-NN model, and both parties use the two datasets mnist_guest & mnist_host aligned by id to conduct a hetero federated training
The same as the Homo-NN(see Customize your Dataset), here we will not follow the usage of conventional FATE components, but directly bind the data path to a FATE name&namespace and pass it to the Hetero-NN component through the reader, and Hetero-NN imports your custom dataset class through the DatasetParam you set, then read data from the path.
pipeline initialization¶
Here we define the pipeline to run a hetero task
import os
import torch as t
from torch import nn
from pipeline import fate_torch_hook
from pipeline.component import HeteroNN
from pipeline.component.hetero_nn import DatasetParam
from pipeline.backend.pipeline import PipeLine
from pipeline.component import Reader, Evaluation, DataTransform
from pipeline.interface import Data, Model
from pipeline.component.nn import save_to_fate
fate_torch_hook(t)
# bind path to fate name&namespace
fate_project_path = os.path.abspath('../../../../')
guest = 10000
host = 9999
pipeline_img = PipeLine().set_initiator(role='guest', party_id=guest).set_roles(guest=guest, host=host)
guest_data = {"name": "mnist_guest", "namespace": "experiment"}
host_data = {"name": "mnist_host", "namespace": "experiment"}
guest_data_path = fate_project_path + '/examples/data/mnist_guest/'
host_data_path = fate_project_path + '/examples/data/mnist_host/'
pipeline_img.bind_table(name='mnist_guest', namespace='experiment', path=guest_data_path)
pipeline_img.bind_table(name='mnist_host', namespace='experiment', path=host_data_path)
{'namespace': 'experiment', 'table_name': 'mnist_host'}
guest_data = {"name": "mnist_guest", "namespace": "experiment"}
host_data = {"name": "mnist_host", "namespace": "experiment"}
reader_0 = Reader(name="reader_0")
reader_0.get_party_instance(role='guest', party_id=guest).component_param(table=guest_data)
reader_0.get_party_instance(role='host', party_id=host).component_param(table=host_data)
hetero_nn_0 = HeteroNN(name="hetero_nn_0", epochs=3,
interactive_layer_lr=0.01, batch_size=512, task_type='classification', seed=100
)
guest_nn_0 = hetero_nn_0.get_party_instance(role='guest', party_id=guest)
host_nn_0 = hetero_nn_0.get_party_instance(role='host', party_id=host)
# define model
# image features 784, guest bottom model
# our simple classification model:
guest_bottom = t.nn.Sequential(
t.nn.Linear(784, 8),
t.nn.ReLU()
)
# image features 784, host bottom model
host_bottom = t.nn.Sequential(
t.nn.Linear(784, 8),
t.nn.ReLU()
)
# Top Model, a classifier
guest_top = t.nn.Sequential(
nn.Linear(8, 10),
nn.Softmax(dim=1)
)
# interactive layer define
interactive_layer = t.nn.InteractiveLayer(out_dim=8, guest_dim=8, host_dim=8)
# add models
guest_nn_0.add_top_model(guest_top)
guest_nn_0.add_bottom_model(guest_bottom)
host_nn_0.add_bottom_model(host_bottom)
# opt, loss
optimizer = t.optim.Adam(lr=0.01)
loss = t.nn.CrossEntropyLoss()
# use DatasetParam to specify dataset and pass parameters
guest_nn_0.add_dataset(DatasetParam(dataset_name='mnist_dataset', return_label=True))
host_nn_0.add_dataset(DatasetParam(dataset_name='mnist_dataset', return_label=False))
hetero_nn_0.set_interactive_layer(interactive_layer)
hetero_nn_0.compile(optimizer=optimizer, loss=loss)
pipeline_img.add_component(reader_0)
pipeline_img.add_component(hetero_nn_0, data=Data(train_data=reader_0.output.data))
pipeline_img.add_component(Evaluation(name='eval_0', eval_type='multi'), data=Data(data=hetero_nn_0.output.data))
pipeline_img.compile()
<pipeline.backend.pipeline.PipeLine at 0x7fae4e81f460>
pipeline_img.fit()
2022-12-24 17:37:26.846 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:83 - Job id is 202212241737239009640 2022-12-24 17:37:26.858 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:98 - Job is still waiting, time elapse: 0:00:00 m2022-12-24 17:37:27.906 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - 2022-12-24 17:37:27.908 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component reader_0, time elapse: 0:00:01 2022-12-24 17:37:28.931 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component reader_0, time elapse: 0:00:02 2022-12-24 17:37:29.954 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component reader_0, time elapse: 0:00:03 2022-12-24 17:37:30.982 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component reader_0, time elapse: 0:00:04 2022-12-24 17:37:32.045 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component reader_0, time elapse: 0:00:05 2022-12-24 17:37:33.096 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component reader_0, time elapse: 0:00:06 m2022-12-24 17:37:35.464 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - 2022-12-24 17:37:35.465 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:08 2022-12-24 17:37:36.528 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:09 2022-12-24 17:37:37.565 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:10 2022-12-24 17:37:38.595 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:11 2022-12-24 17:37:39.631 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:12 2022-12-24 17:37:40.660 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:13 2022-12-24 17:37:41.689 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:14 2022-12-24 17:37:42.735 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:15 2022-12-24 17:37:43.763 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:16 2022-12-24 17:37:44.843 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:17 2022-12-24 17:37:45.868 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:19 2022-12-24 17:37:46.898 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:20 2022-12-24 17:37:47.923 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:21 2022-12-24 17:37:48.948 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:22 2022-12-24 17:37:49.977 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:23 2022-12-24 17:37:51.022 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:24 2022-12-24 17:37:52.046 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:25 2022-12-24 17:37:53.069 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:26 2022-12-24 17:37:54.110 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:27 2022-12-24 17:37:55.145 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:28 2022-12-24 17:37:56.177 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:29 2022-12-24 17:37:57.216 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:30 2022-12-24 17:37:58.258 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:31 2022-12-24 17:37:59.289 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:32 2022-12-24 17:38:00.315 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:33 2022-12-24 17:38:01.336 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:34 2022-12-24 17:38:02.384 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:35 2022-12-24 17:38:03.411 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:36 2022-12-24 17:38:04.433 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:37 2022-12-24 17:38:05.455 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:38 2022-12-24 17:38:06.475 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:39 2022-12-24 17:38:07.494 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:40 2022-12-24 17:38:08.513 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:41 2022-12-24 17:38:09.532 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:42 2022-12-24 17:38:10.557 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:43 2022-12-24 17:38:11.579 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:44 2022-12-24 17:38:12.615 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:45 2022-12-24 17:38:13.662 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:46 2022-12-24 17:38:14.698 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:47 2022-12-24 17:38:15.718 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:48 2022-12-24 17:38:16.809 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:49 2022-12-24 17:38:17.833 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:50 2022-12-24 17:38:18.857 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:52 2022-12-24 17:38:19.896 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:53 2022-12-24 17:38:20.934 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:54 2022-12-24 17:38:21.995 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:55 2022-12-24 17:38:23.025 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:56 2022-12-24 17:38:24.056 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:57 2022-12-24 17:38:25.080 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:58 2022-12-24 17:38:26.104 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:00:59 2022-12-24 17:38:27.151 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:00 2022-12-24 17:38:28.175 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:01 2022-12-24 17:38:29.194 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:02 2022-12-24 17:38:30.215 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:03 2022-12-24 17:38:31.234 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:04 2022-12-24 17:38:32.255 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:05 2022-12-24 17:38:33.278 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:06 2022-12-24 17:38:34.443 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:07 2022-12-24 17:38:35.470 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:08 2022-12-24 17:38:36.488 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:09 2022-12-24 17:38:37.510 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:10 2022-12-24 17:38:38.532 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:11 2022-12-24 17:38:39.558 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:12 2022-12-24 17:38:40.587 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:13 2022-12-24 17:38:41.612 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:14 2022-12-24 17:38:42.635 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:15 2022-12-24 17:38:43.666 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:16 2022-12-24 17:38:44.707 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:17 2022-12-24 17:38:45.733 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:18 2022-12-24 17:38:46.776 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:19 2022-12-24 17:38:47.806 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:20 2022-12-24 17:38:48.835 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:21 2022-12-24 17:38:49.858 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:23 2022-12-24 17:38:50.878 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:24 2022-12-24 17:38:52.364 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:25 2022-12-24 17:38:53.600 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:26 2022-12-24 17:38:54.635 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:27 2022-12-24 17:38:55.684 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:28 2022-12-24 17:38:56.725 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:29 2022-12-24 17:38:57.747 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:30 2022-12-24 17:38:58.771 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:31 2022-12-24 17:38:59.795 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:32 2022-12-24 17:39:00.815 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:33 2022-12-24 17:39:01.839 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:34 2022-12-24 17:39:02.877 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:36 2022-12-24 17:39:03.901 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:37 2022-12-24 17:39:04.923 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:38 2022-12-24 17:39:05.965 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:39 2022-12-24 17:39:07.009 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:40 2022-12-24 17:39:08.047 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:41 2022-12-24 17:39:09.082 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:42 2022-12-24 17:39:10.122 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:43 2022-12-24 17:39:11.151 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:44 2022-12-24 17:39:12.176 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:45 2022-12-24 17:39:13.198 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:46 2022-12-24 17:39:14.218 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:47 2022-12-24 17:39:15.240 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:48 2022-12-24 17:39:16.262 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:49 2022-12-24 17:39:17.285 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:50 2022-12-24 17:39:18.315 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:51 2022-12-24 17:39:19.337 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component hetero_nn_0, time elapse: 0:01:52 m2022-12-24 17:39:21.406 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - 2022-12-24 17:39:21.407 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component eval_0, time elapse: 0:01:54 2022-12-24 17:39:22.427 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component eval_0, time elapse: 0:01:55 2022-12-24 17:39:23.448 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component eval_0, time elapse: 0:01:56 2022-12-24 17:39:24.485 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component eval_0, time elapse: 0:01:57 2022-12-24 17:39:25.504 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component eval_0, time elapse: 0:01:58 2022-12-24 17:39:26.539 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:127 - Running component eval_0, time elapse: 0:01:59 2022-12-24 17:39:27.576 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:89 - Job is success!!! Job id is 202212241737239009640 2022-12-24 17:39:27.577 | INFO | pipeline.utils.invoker.job_submitter:monitor_job_status:90 - Total time: 0:02:00
pipeline_img.get_component('hetero_nn_0').get_output_data() # get result
id | label | predict_result | predict_score | predict_detail | type | |
---|---|---|---|---|---|---|
0 | img_1 | 0 | 8 | 0.2643284201622009 | {'0': 0.1221427395939827, '1': 0.0131008885800... | train |
1 | img_3 | 4 | 7 | 0.28708773851394653 | {'0': 0.020947180688381195, '1': 0.10722759366... | train |
2 | img_4 | 0 | 8 | 0.2315242737531662 | {'0': 0.1916944831609726, '1': 0.0062320181168... | train |
3 | img_5 | 0 | 8 | 0.2495078295469284 | {'0': 0.04027773439884186, '1': 0.039246417582... | train |
4 | img_6 | 7 | 7 | 0.5058534145355225 | {'0': 0.011092708446085453, '1': 0.06033106520... | train |
... | ... | ... | ... | ... | ... | ... |
1304 | img_32537 | 1 | 7 | 0.228757843375206 | {'0': 0.02710532397031784, '1': 0.178657308220... | train |
1305 | img_32558 | 1 | 7 | 0.22664928436279297 | {'0': 0.03342469036579132, '1': 0.192812800407... | train |
1306 | img_32563 | 1 | 7 | 0.2404891550540924 | {'0': 0.02606056071817875, '1': 0.177977174520... | train |
1307 | img_32565 | 1 | 7 | 0.44030535221099854 | {'0': 0.014257671311497688, '1': 0.11480070650... | train |
1308 | img_32573 | 1 | 1 | 0.24856531620025635 | {'0': 0.028993327170610428, '1': 0.24856531620... | train |
1309 rows × 6 columns