Merge datasets

Here is a tutorial that describes how to merge and split datasets in a project.
A simple case
For example we have project "lemons_annotated" with dataset "ds1" and project "lemons_test" with dataset "ds1". Also we want to rename and merge all datasets into another project.
Step 1. Go to DTL page.
Step 2. Write DTL query.
You can see a simple computational graph which contains a few layers:
2 input data layers
2 dataset naming layers
1 save data layer
First and second layer define the project we will use. Third and fourth is the Dataset layer that moves data flow to the selected dataset. Fifth layer just saves all input data to the new lemon_merged project.
Here is the DTL query from our example:
[
  {
    "dst": "$project_1_sample",
    "src": [
      "lemons_annotated/*"
    ],
    "action": "data",
    "settings": {
      "classes_mapping": "default"
    }
  },
  {
    "dst": "$project_2_sample",
    "src": [
      "lemons_test/*"
    ],
    "action": "data",
    "settings": {
      "classes_mapping": "default"
    }
  },
  {
    "dst": "$output_1",
    "src": [
      "$project_1_sample"
    ],
    "action": "dataset",
    "settings": {
      "name": "ds_annotated"
    }
  },
  {
    "dst": "$output_2",
    "src": [
      "$project_2_sample"
    ],
    "action": "dataset",
    "settings": {
      "name": "ds_test"
    }
  },
  {
    "dst": "lemon_merged",
    "src": [
      "$output_1",
      "$output_2"
    ],
    "action": "supervisely",
    "settings": {}
  }
]
Step 3. View results.
When the task is finished, the new project "lemon_merged" will appear on the "Projects" page. This project will contain 2 datasets ("ds_annotated" and "ds_test").
Here is the result:
Conflict case
For example we have project "Project_AB" (contains datasets dsA, dsB) and project "Project_A" (contains only the dataset dsA).
DTL query
We want to just merge all datasets into another a new project. We don't want to rename conflicting names:
[
  {
    "dst": "$output_1",
    "src": [
      "Project_AB/*"
    ],
    "action": "data",
    "settings": {
      "classes_mapping": "default"
    }
  },
  {
    "dst": "$output_2",
    "src": [
      "Project_A/*"
    ],
    "action": "data",
    "settings": {
      "classes_mapping": "default"
    }
  },
​
  {
    "dst": "Project_AB_and_A",
    "src": [
      "$output_1",
      "$output_2"
    ],
    "action": "supervisely",
    "settings": {}
  }
]
Names conflict resolving results
When the task is finished, the new project "Project_AB_and_A" will appear on the "Projects" page. That project will contain 3 datasets ("Project_AB__dsA", "Project_A__dsA", "dsB").
Here is the result:

Examples / Use cases

Split datasets

Last modified 2yr ago

Copy link

Outline

A simple case

Step 1. Go to DTL page.

Step 2. Write DTL query.

Step 3. View results.

Conflict case

DTL query

Names conflict resolving results