Skip to content

Labeling Jobs

Labeling Jobs allow assigning annotation tasks to labelers.

While you can call annotation interface directly from the Projects page, there are some difficulties you may face:

  • Job management - the need to describe a particular task: what kind of objects to annotate and how
  • Progress monitoring - tracking annotation status and reviewing submitted results
  • Access permissions - limiting access only to specific datasets and classes within a single job

Labeling Jobs address these problems in the following way:

Example

Consider this case: we want to annotate large project with two people, split the whole process into separate jobs and track overall progress. How to do that?

1. Set up a new team

This step is optional, but it's much more easier to manage access permissions when you have specialized teams for annotation jobs.

Click on the teams selector in the left menu to open teams list. Add a new team, for example, "Labeling team".

You can upload datasets to be annotated here or use "Clone to..." action to copy existing datasets from the other team.

2. Create accounts for labelers

You can invite an existing user from "Members" page or create new ones from "Users" section of admin user. In this example we will show both ways.

Click "Signup" button and enter login and password for a new labeler account. We will check "Restricted" option - in this case the new user will have no access to Explore section, won't be able to create or switch teams and no personal team will be created for them during signup.

#50%

Now go to the "Members" page and invite some existing user to your "Labeling team". Choose an "Annotator" role - that means that user won't have access to any page apart from "Labeling Jobs". You can choose a "Developer" role if you want that user to be able to upload new data, but we advice to upload datasets in a separate "Working" team and then clone them to "Labeling" team - that way it will be easier to separate research from annotation and manage labeling jobs.

3. Provide a description

Let's make some README for the project to be annotated. Open the "Projects" page, choose your project and go to the "Info" tab. Click the "edit" button to edit project readme information. We support markdown so you can attach images with annotation examples, describe valid and invalid cases and so on.

4. Create a new labeling Job

Open "Labeling Jobs" page in the left menu and click "Create Job".

Choose your project, dataset and user to be assigned to that job. Our example project has two classes. Lets assign user "labeler1" just one class. We will also change the images range to make them label only a half of the project. You can also use DTL to randomly split an existing project into several datasets and assign them as separate jobs.

Click "Create" to finish. Repeat the same for the user "labeler2", but choose the second half of the project.

5. Complete Jobs

Let's sign in as a labeler. We will use "User List" from the admin menu and select "Login as..." from the context menu.

As you can see, "labeler1" can only access labeling jobs and cannot even switch the team. But they see all jobs assigned to them. The "Info" button provides the necessary information about the labeling job and project readme we set up earlier.

If we click on the job title, the annotation tool will open. This will automatically change the job status from "Pending" to "In progress" so that manager can see that the job has been started.

As you can see, we can't create new classes and can only see the single class that were assigned to us. Let's annotate an image. When we are done, click "Confirm" to mark that image as completed. Manager will see our progress in the jobs list. He can also track labeling statistics like annotation average time per class.

When the labeler has completed the annotation, they need to go back to "Labeling Jobs" and click the "Submit" button. This will mark job as completed and remove it from their list.

#50%

Manager can review annotations and, if job needs to be redone, he can choose "Clone" in the context menu and start it again as a new job.