Visualize manual label jobs

In a manual label job, the label spec is processed manually by a human expert to refine existing labels or add new labels. Once the manual labeling job process is complete, you can view the results, review them, and accept or reject them depending on their accuracy.

Refer to the Manual Label Jobs topic for creating and submitting a manual label job.

This article describes the steps to visualize the manual label job results and categorize the labels.

Once the manual labeling is completed, you can do the following activities:

• Slice and filter the results based on various parameters.

• Review sample jobs and accept or reject the results.

• Save accepted results as a ResultSet for reference.

• Export ResultSets to a datalog table or download the labels

Visualize manual label jobs

1. For manual label jobs, the job results appear under Expert Review Stats based on the classes defined manually.
   
   
   By default, all the results appear under the Pending Review tab.

Visualize Classification-based manual label job

1. On the Labeling Jobs page, locate the label job card with the label M-LA-CL for which the status is shown as READY, and click Visualize.
   

   The manual label job page displays the top classes segregated by Class Changed conditions in a tabular format. The ‘Class Changed’ indicates whether the manual labeler changed the class compared to the pre-label supplied when job was created. This is an indicator of the degree of refinement that manual labeled needed to do on the pre-labels that could have been supplied from an auto label job.
   
   By default, all the results appear under the Pending Review tab. The Class Changed table displays the results for different classes specified for the manual label job.

2. View results based on the class changed conditions (for example, True or False).

   1. Select one of the condition values in the Class Changed table.

   2. Click the image icon on the top-right corner of the section.
      The right-side panel displays the corresponding image classified based on the expert review.
      

      
      Based on the class defined during expert review, you can review and accept or reject the results.

Visualize Object Detection-based manual label job

Similar to visualizing the manual label job results for Classification type, you can view the label job results for Object Detection type.

1. On the Labeling Jobs page, locate the label job card with the label M-LA-OD for which the status is shown as READY, and click Visualize.
   

   The manual label job page displays the results as per the Class Changed condition under the Expert Review Stats section.
   
   By default, all the results appear under the Pending Review tab.

   1. The Class Changed table displays the results for class changed conditions as specified for the images. The ‘Class Changed’ indicates whether the manual labeler changed the class compared to the pre-label supplied when job was created. This is an indicator of the degree of refinement that manual labeled needed to do on the pre-labels that could have been supplied from an auto label job.

   2. Unlike Auto Label Jobs, it does not have a multi-class result, as the images are manually labelled and classified under a single class.

2. View results based on the class changed manually for the images.

   1. Select one of the values under the condition applicable for the class changed (for example, True or False).

   2. Click the image icon on the top-right corner of the section.
      The right-side panel displays the corresponding image detected with the matching condition that you have selected.
      

Based on the class defined during expert review, you can review and accept or reject the results.

Filter manual label job results

1. On the label job page, click the Filter icon.

2. In the Filters window, specify the required details to filter the results.
   

   1. Select the Expert Review Class, as needed.

   2. Enable Show Advanced Filter option, if needed.

   3. Specify the Class Changed Condition as True or False, depending on the Expert review.

   4. Set the Min and Max values for Expert Review Area and Current to Auto/Pre Label Adjustment Factor.

   5. Select Group by, and click Apply.
      The manual label job page displays the results based on the filter parameters.
      

Review manual label job results

On the manual label job page, you can view the accuracy results of the images for different class changed conditions. For illustration purposes, let us consider review of the manual label job of Object Detection type. Once you are satisfied with the results based on the detected components indicated with annotations, you can accept to move the results to the Accepted category.

1. Navigate to Label > Jobs, locate the job card, and click Catalog.

2. In the label job page, review the images for values and accept or reject the results.

   1. Click a value under the Class Changed table to view the images.

   2. Review the images for accuracy based on the expert review parameters.
      

3. Accept or Reject the results.

   1. To accept the results of all images displayed for the selected value, click the Accept button.
      The set of images will be accepted and moved under the Accepted tab.
      

4. Move an accepted result to a Pending for review or rejected state.

   1. On the label page, under the Accepted tab, click the required value, and click the image icon on the top right of that section.

   2. Click the Reject or Move to pending button located at the bottom-right corner.
      
      

You can accept the results of the label job and move all the images under the Accepted tab, any time. For future reference, you can also create a result set and export it to the corresponding catalog.