Skip to content

Task data: structure

Agent is a task manages service. It knows only what data have to be downloaded, and how to upload task results. Agent prepare necessary data and upload results based on task type.

Import

For example, user started import and the system assigned task_id = 3039. Special task directory will be created at ~/.supervisely-agent/tasks/3039. Directory structure is the following:

3039
├── data
│   ├── chase
│   │   ├── ann
│   │   │   ├── Image_01R.json
│   │   │   ├── ...
│   │   │   └── Image_13R.json
│   │   └── img
│   │       ├── Image_01R.png
│   │       ├── ...
│   │       └── Image_13R.png
│   └── meta.json
├── logs
│   └── log_2018-06-10_18:18:20.txt
├── results
│   └── vessels6
│       ├── chase
│       │   ├── ann
│       │   │   ├── Image_01R.json
│       │   │   ├── ...
│       │   │   └── Image_13R.json
│       │   └── img
│       │       ├── Image_01R.png
│       │       ├── ...
│       │       └── Image_13R.png
│       └── meta.json
├── task_settings.json
└── tmp

data - this directory contains all content (user drag and drop it) that user want to upload to the system.

task_settings.json - contains resulting project name.

logs - agent puts all task logs in this directory.

results - import converts input data to Supervisely format and put resulting project here.

Agent chooses necessary docker image to start based on choosen element from drop-down list (import interface). When import finished, agent will upload resulting project to Supervisely.

NN: training

For example, user started NN training and the system assigned task_id = 777. Special task directory will be created at ~/.supervisely-agent/tasks/777. Directory structure is the following:

777
├── data
│   └── mapillary_cityscapes_filtered
│       ├── ...
│       ...
├── logs
│   └── log_2018-06-10_15:24:50.txt
├── model
│   ├── model.config
│   └── model_weights
│       ├── checkpoint
│       ├── model.ckpt.data-00000-of-00001
│       ├── model.ckpt.index
│       └── model.ckpt.meta
├── results
│   ├── 00000000
│   │   ├── model.config
│   │   └── model_weights
│   │       ├── checkpoint
│   │       ├── model.ckpt.data-00000-of-00001
│   │       ├── model.ckpt.index
│   │       └── model.ckpt.meta
│   ├── 00000001
│   │   ├── model.config
│   │   └── model_weights
│   │       ├── checkpoint
│   │       ├── model.ckpt.data-00000-of-00001
│   │       ├── model.ckpt.index
│   │       └── model.ckpt.meta
│   ├── 00000002
│   │   ├── model.config
│   │   └── model_weights
│   │       ├── checkpoint
│   │       ├── model.ckpt.data-00000-of-00001
│   │       ├── model.ckpt.index
│   │       └── model.ckpt.meta
│   ├── 00000003
│   │   ├── model.config
│   │   └── model_weights
│   │       ├── checkpoint
│   │       ├── model.ckpt.data-00000-of-00001
│   │       ├── model.ckpt.index
│   │       └── model.ckpt.meta
│   └── 00000004
│       ├── model.config
│       └── model_weights
│           ├── checkpoint
│           ├── model.ckpt.data-00000-of-00001
│           ├── model.ckpt.index
│           └── model.ckpt.meta
├── task_settings.json
└── tmp

data - this directory contains project in Supervisely format (images with annotations) that will be used in as training data.

task_settings.json - training config that user defined in training wizard.

logs - agent puts all task logs in this directory.

results - training produces checkpoints (model weights) while training. As you can see, this training example produced four five checkpoints

When the next checkpoint is created, Agent copies checkpoint to its local storage and notifies Supervisely Server about this event.

NN: inference

For example, user started NN inference and the system assigned task_id = 123. Special task directory will be created at ~/.supervisely-agent/tasks/123. Directory structure is the following:

123
├── data
│   └── lemon_test
│       ├── test_dataset
│       │   ├── ann
│       │   │   ├── IMG_0748_1A8L0ZMw1N.json
│       │   │   ├── IMG_0748_6C3orKKYgt.json
│       │   │   ├── ...
│       │   │   └── IMG_8144_zRgoFGi2GC.json
│       │   └── img
│       │       ├── IMG_0748_1A8L0ZMw1N.jpg
│       │       ├── IMG_0748_6C3orKKYgt.jpg
│       │       ├── ...
│       │       └── IMG_8144_zRgoFGi2GC.jpg
│       └── meta.json
├── model
│   ├── model.config
│   └── model_weights
│       ├── checkpoint
│       ├── model.ckpt.data-00000-of-00001
│       ├── model.ckpt.index
│       └── model.ckpt.meta
├── task_settings.json
├── logs
│   └── log_2018-06-10_16:51:04.txt
├── results
│   ├── inference_results
│   │   ├── ann
│   │   │   ├── IMG_0748_1A8L0ZMw1N.json
│   │   │   ├── IMG_0748_6C3orKKYgt.json
│   │   │   ├── ...
│   │   │   └── IMG_8144_zRgoFGi2GC.json
│   │   └── img
│   │       ├── IMG_0748_1A8L0ZMw1N.jpg
│   │       ├── IMG_0748_6C3orKKYgt.jpg
│   │       ├── ...
│   │       └── IMG_8144_zRgoFGi2GC.jpg
│   └── meta.json
└── tmp

data - this directory contains project that will be used in inference.

task_settings.json - inference config that user defined in inference wizard.

logs - agent puts all task logs in this directory.

results - inference produces resulting project with NN predictions, that will be stored here in Supervisely format.

When inference finished, agent will upload resulting project to Supervisely.

DTL

For example, user started DTL and the system assigned task_id = 3038. Special task directory will be created at ~/.supervisely-agent/tasks/3038. Directory structure is the following:

3038
├── data
│   └── lemon6_tr1
│       ├── lemon6__ds
│       │   ├── ann
│       │   │   ├── IMG_0748_1A8L0ZMw1N.json
│       │   │   ├── IMG_0748_6C3orKKYgt.json
│       │   │   ├── ...
│       │   │   └── IMG_8144_zRgoFGi2GC.json
│       │   └── img
│       │       ├── IMG_0748_1A8L0ZMw1N.jpg
│       │       ├── IMG_0748_6C3orKKYgt.jpg
│       │       ├── ...
│       │       └── IMG_8144_zRgoFGi2GC.jpg
│       └── meta.json
├── graph.json
├── logs
│   └── log_2018-06-10_16:51:04.txt
├── results
│   ├── lemon6_tr1__lemon6__ds
│   │   ├── ann
│   │   │   ├── IMG_0748_1A8L0ZMw1N.json
│   │   │   ├── IMG_0748_6C3orKKYgt.json
│   │   │   ├── ...
│   │   │   └── IMG_8144_zRgoFGi2GC.json
│   │   └── img
│   │       ├── IMG_0748_1A8L0ZMw1N.jpg
│   │       ├── IMG_0748_6C3orKKYgt.jpg
│   │       ├── ...
│   │       └── IMG_8144_zRgoFGi2GC.jpg
│   └── meta.json
└── tmp

data - this directory contains projects (that are used in DTL) in Supervisely format (images with annotations).

graph.json - DTL graph.

logs - agent puts all task logs in this directory.

results - DTL produces resulting project, that will be stored here in Supervisely format.

When DTL finished, agent will upload resulting project to Supervisely.

Task finalizing and Error handling

If the task is finished successfully, agent will clean all task directory content except logs. If task is failed, directory will not be cleaned. It is useful because developer can download entire task directory and debug error locally without any Supervisely infrastructure.

Note

It is applicable for all task types.