Job creation and visualization

The job creation flow consists of selecting objects from the catalog and submitting an 'Analyze Classification' or 'Analyze Object Detection' type of job, as shown in the below animation.

Catalog columns

Analyze Classification

This type of job requires the following catalog fields to be present in the catalog.

• Ground truth class (string)
• Predicted class (string)
• Prediction confidence (float)

A sample catalog table with ground truth and predictions for a single image is provided below. The file_path column is a mandatory column for all rows that stores the image file name.

Note:

• Ground truth and prediction must be imported to the catalog as 2 separate rows, with each prediction in its own row.
• Only one ground truth per image is supported.
• The prediction and corresponding confidence score must be present in the same row.

Analyze object detection

This type of job requires the following catalog fields to be present in the catalog.

1. Ground truth class(string) (gt_class)
2. Ground truth bounding box(string) (gt_box)
   1. The string should be formatted as a JSON with x_left, x_right, y_top, and y_bottom.  
   2. All key values must be normalized between 0 and 1 using the width and height of the image.
   3. For example {'x_left': 0.5, 'y_top': 0.75, 'x_right': 0.7, 'y_bottom': 0.95} represents a valid bounding box.
3. Predicted class(string)(pd_class)
4. Prediction bounding box(string) (pd_box) - Same format as ground truth bounding box
5. Prediction confidence score(float) (pd_score)

A sample catalog table with ground truth and predictions for a single image is provided below. The file_path column is a mandatory column for all rows that stores the image file name. The below sample table captures a model output where car and bus classes are true positives and train is a false positive predicted on the image 1.jpg

Note:

• If an object has multiple ground truth and prediction labels, each row in the catalog should specify one ground truth or one prediction label.
• The ground truth label and ground truth bounding box should be present in the same row.
• The prediction confidence and prediction bounding box should be present in the same row as the prediction label.

Visualization

Analyze-visualization The 'analyze' job is visualized in the same way as an explore job with model performance characteristics used as features to cluster and present the plot view. The clusters get formed based on aspects like 'certain class X detected as class Y', 'mispredictions in the top left grid of the image' etc. All controls in plot view to sample, view thumbnail on hover, select images for similarity search, etc are available similar to explore job type.

Statistics

Analyze Classification job

Prediction confidence histogram

The histogram provides a histogram of the number of samples for different ranges of prediction confidence. The left-side selectors can be used to filter the histogram on one or more classes.

Analyze-classification-confidence-histogramEach histogram bar can be selected for follow on operations

1. Show plot - The points belonging to the selected bar are highlighted in the plot. This gives insight into the feature space region where the points corresponding to the selected bar lie.
2. Show images - The thumbnails for the points corresponding to the selected bar are populated in the right selection panel for further follow-up actions like 'selecting images for similarity search' or 'adding images to resultset'
3. Reset - Reset the selection

Analyze-classification-confidence-histogram-actions

 PR curve and confusion matrix

Analyze-classification-PR-curve-and-confusion-matrix

1. The confidence threshold slider selects the confidence threshold to decide if a prediction should be treated as a positive classification. The confidence threshold can be set to one of the predefined values.
2. The precision-recall curve plots the precision against recall values for different confidence thresholds and highlights the value corresponding to the above chosen confidence threshold with a 'dot'.
3. The confusion matrix provides a 2-dimensional relationship between ground truth and predicted classes corresponding to the chosen confidence threshold. For further actions, a cell, row(s), column(s), the intersection of row and columns, or diagonal cells can be selected. On the selected cells, there are the following actions supported.
   • Highlight selected points - Shows the points corresponding to the selected cells in a plot view.
   • Show Images - Shows thumbnails corresponding to the selected cells.

Analyze object detection job

IOU Vs confidence heatmap

IOU Vs confidence heatmap

This heatmap colors each cell based on the number of bounding boxes with a specific value of IOU (Intersection over union) and prediction confidence. The bottom right cells represent good model behavior with high IOU and prediction confidence. The class labels on the left are sorted from the class that shows the best model performance to the class that shows the worst model performance. The visualization uses an algorithm where the class with darker shades on the heatmap towards the bottom right will be considered as the best-performing class.

IOU vs confidence heatmap for 1 classThe above heatmap shows the view when a single class is selected. Along with the heatmap representing the number of bounding boxes for the specific intersections of IOU Vs. confidence values, the following additional controls are available.

1. Each cell shows the number of objects (images) corresponding to the intersection.
2. On selecting a cell, the following actions are available.
   1. Highlight selected points - Shows the points corresponding to the selected cells in a plot view.
   2. Show Images - Shows thumbnails corresponding to the selected cells.

PR curve and confusion matrix

Analyze-object-detection-PR-curve-and-confusion-matrix

 The object detection statistics panel provides additional control to interact using the IOU (Intersection over Union) threshold values for object detection.

1. The PR curve plots the precision and recall values for the chosen class(aeroplane in the above image) for different confidence threshold values. The chosen class is considered to be a true positive if the prediction has an IOU value greater than or equal to the IOU threshold chosen in the slider against at least one ground truth bounding box.
2. The dot on the PR curve indicates the precision and recall value for the chosen confidence threshold.
3. The confusion matrix is presented for the chosen IOU threshold and the confidence threshold values.
4. The confusion matrix provides a 2-dimensional relationship between ground truth and predicted classes corresponding to the chosen IOU and confidence thresholds. For further actions, a cell, row(s), column(s), the intersection of row and columns, or diagonal cells can be selected. On the selected cells, there are the following actions supported.
   1. Highlight selected points - Shows the points corresponding to the selected cells in a plot view.
   2. Show Images - Shows thumbnails corresponding to the selected cells.