Labeling Jobs provide functionality to assign annotation tasks to labelers.
While you can call annotation interface directly from Projects page, where are some cons you may face:
- Job management - you need to describe a particular task: what kind of objects to annotate and how
- Progress monitoring - track annotation status and review submitted results
- Access permissions - give access only to a specific datasets and classes within a single job
Labeling Jobs address those problems in the following way.
Consider this case: we want to annotate large project with two people, split the whole process in separate jobs and track overall progress. How to do that?
1. Setup a new team¶
This step is optional, but it's much more easier to manage access permissions when you have special teams for annotation jobs.
Click on teams selector in left menu to open teams list. Add a new team, for example, "Labeling team".
You can upload datasets to be annotated here or use "Clone to..." action to copy existing datasets from other team.
2. Create accounts for labelers¶
You can invite an existing users from "Members" page or create new ones from "Users" section of admin user. In this example we will show both ways.
Click "Signup" button and enter login and password for a new labeler account. We will check "Restricted" option - in this case new user will have no access to Explore section, won't be able to create or switch team and no personal team will be created for him during signup.
Now go a "Members" page and invite some existing user to your "Labeling team". Choose "Annotator" role - that means, that user won't have access to any page apart from "Labeling Jobs". You can choose "Developer" role if you want that your to be able to upload new data, but we advice to upload datasets in a separate "Working" team and then clone them to "Labeling" team - that way it would be easier to separate research from annotation and manage labeling jobs.
3. Provide description¶
Let's make some README for project to be annotated. Open Projects page, choose project and go to "Info" tab. Click "edit" button to edit project readme information. We support markdown so you can attach images with annotation examples, describe valid and invalid cases and so on.
4. Create a new labeling Job¶
Open "Labeling Jobs" page in left menu and click "Create Job".
Choose your project, dataset and user to be assigned on that job. Our example project has two classes. Lets assign to user "labeler1" just one class. I will also change images range to make him label only half of the project. You can also use DTL to randomly split an existing project into several datasets and assign them as a separate jobs.
Click "Create" to finish. Repeat the same for user "labeler2", but choose the second half of the project.
5. Complete Jobs¶
Let's sign in as a labeler. I will use "User List" from admin menu and select "Login as..." from context menu.
As you can see, "labeler1" can only see labeling jobs and cannot even switch the team. But he sees all jobs assigned to him. "Info" button provides necessary information about labeling job and project readme we set up earlier.
If we click on job title, annotation tool will open. This would automatically change job status from "Pending" to "In progress" so that manager will see that the job has been started.
As you can see, we can't create new classes and can only see single class that were assigned to me. Let's annotate an image. When we are done, click "Confirm" button to mark that image as completed. Manager will see progress in jobs list. He can also track labeling statistics like annotation average time per class.
When labeler has completed annotation, go back to "Labeling Jobs" and click "Submit" button. This will mark job as completed and remove it from his list.
Manager can review annotations and, if job needs to be redone, he can choose "Clone" in context menu and start it again as a new job.