Analyze Segmentation

This type of job analyzes the output of an instance segmentation model, which has class, segments in the form of polygons, and confidence score.

Catalog

This type of job requires the following fields to be present in the catalog.

1. Ground truth class(string) (gt_class)

2. Ground truth segment(string) (gt_segment) - The ground truth segment must be provided in one of the following forms.

   1. Three individual catalog columns for

      1. segment: A JSON string in the form of a list of polygons with each polygon being a list of (x, y) coordinates normalized with respect to the mask_width and mask_height columns. A sample is as below
         [[x11, y11, x12, y12,..,x15,y15], [x21, y21, x22, y22,…,x26, y26]] → This represents a 6 co-ordinate polygon with (x1n, y1n) representing individual vertices with values normalized with respect to mask_width and mask_height.

      2. mask_width: An integer for the maximum width against which the coordinates are normalized.

      3. mask_height: An integer for the maximum height against which the coordinates are normalized.

   2. A single column representing a segment as a JSON of three keys as shown below.

      1. {segment: [[x11, y11, x12, y12,..,x15,y15], [x21, y21, x22, y22,…,x26,y26]], mask_height: h, mask_width: w}

      2. The individual keys have the same requirements as described for individual columns in the previous format.

   3. segment in COCO format.

3. Predicted class(string)(pd_class)

4. Prediction segment(string) (pd_segment) - Same format as ground truth segment

5. Prediction confidence score(float) (pd_score)

Below is a sample catalog table with ground truth and predictions for a single image with the format where a single column holds the JSON of three keys(segment, mask_height, mask_width) described above. The file_path or file_name column is required to correlate(join) this table with one of the pipeline-generated tables. The below sample table captures a model output where car and bus classes are correct predictions, and train is a wrong prediction on an image 1.jpg

Note:

• If an object has multiple ground truth and prediction labels, each row in the catalog should specify one ground truth or one prediction label.

• The ground truth label and ground truth segment information should be in the same row.

• The prediction confidence and predicted segment information should be in the same row as the prediction label.

The above catalog must be loaded into a table using instructions in Quick Import catalog

Create a view

Before creating an analyze segmentation job, the catalog with ground truth and prediction information must be joined with one of the pipeline tables using a view. To create a view, follow the steps in the guide below.

Other types of virtual columns

The guide above defined segments using a single catalog column with a JSON string. The below table lists other virtual column types relevant to segmentation catalog information.

Create a job

Create an analyze segmentation job by following the steps in the guide below.

Visualize job

Once the job is in READY state, click on the VISUALIZE button as shown below.

Statistics panel

The numbered elements on the screenshot of the statistics panel are described below.

1. IOU threshold: A slider to adjust the Intersection-over-Union(IOU) threshold of the prediction to consider it to match the ground truth. Changing the IOU threshold updates the confusion matrix and the precision-recall curve.

2. Confidence threshold: A slider to adjust the prediction's confidence threshold so that it is considered a match against the ground truth. Changing the confidence threshold updates the confusion matrix and the precision-recall curve.

3. Action bar: The icons on the action bar from right to left are

   1. Show images: After selecting any column/row/cell in the confusion matrix, the ‘Show Images’ icon will populate the right panel with thumbnails corresponding to the selected cell(s) in the confusion matrix.

   2. Show plot: Shows the points of the selected column/row/cell in a plot view.

   3. Clear selection: Clear selected column/row/cell.

4. Confusion matrix: The confusion matrix shows the break-up of object counts for each ground truth Vs prediction combination. The row legend shows the precision value, and the column legend shows the recall value at the IOU and confidence thresholds selected by the sliders described earlier.

The precision-recall curve shows the precision and recall values for the chosen IOU threshold and each confidence threshold in the slider. The precision and recall values for the currently selected confidence threshold are highlighted by the larger dot in the chart. The left panel allows viewing precision-recall values for an individual class. The left panel also shows the F1 score at the current chosen confidence threshold.

The IOU Vs. confidence histogram shows the distribution of predictions for each range of IOU and confidence. The average IOU confidence score is presented on the left panel, and a higher score indicates a larger percentage of high IOU and high confidence predictions, indicating good model behavior. The left panel also allows drilling down on an individual class.

The video below demonstrates the statistics panel's capabilities