Catalog structure for analyze jobs

Data Explorer supports the 'Analyze' type of job used for model performance analysis of an image classification or object detection model. This analysis requires ground truth and prediction information to be available in the catalog.

Analyze job for classification model

Create a comma-separated file similar to the following sample.

file_path,frame_idx_in_file,score,prediction_class,ground_truth_class
sdnet_analyze/Walls/Cracked/7069-154.jpg,0,0.9209516048431396,non-cracked,
sdnet_analyze/Walls/Cracked/7069-154.jpg,0,0.07904839515686035,cracked,
sdnet_analyze/Walls/Cracked/7069-154.jpg,0,,,cracked
sdnet_analyze/Walls/Cracked/7069-183.jpg,0,0.9039033353328705,non-cracked,
sdnet_analyze/Walls/Cracked/7069-183.jpg,0,0.09609666466712952,cracked,
sdnet_analyze/Walls/Cracked/7069-183.jpg,0,,,cracked

The first line should be the header line with the names of the fields. As described in the general structure, common fields file_path and frame_idx_in_file fields must be present. Additionally, for an analyze classification job, the following fields must be present.

1. score(float): Probability of the image being classified correctly. The value must be between 0 and 1.
2. prediction_class(string): A predicted class label for input data.
3. ground_truth_class(string): Ground truth class label.

Additionally,

1. Ground truth and prediction must be imported into the catalog as separate rows.
2. Only one ground truth per image/video frame is supported.
3. The prediction and corresponding confidence score must be present in the same row.
4. For the ground truth row, the prediction and score are expected to be empty. Similarly, for the prediction row, the ground truth column is expected to be empty.

Analyze job for object detection model

Create a comma-separated file that must be in the following format.

file_path,frame_idx_in_file,gt_class,gt_box,pd_class,pd_box,pd_score
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2011_000753.jpg,0,car,"{'x_left': 0.812666902937676, 'x_right': 0.9925274478643895, 'y_top': 0.5254668930645845, 'y_bottom': 0.9725518810661693}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2011_000753.jpg,0,traffic-light,"{'x_left': 0.25302248112220327, 'x_right': 0.9489973045211927, 'y_top': 0.3737651964479025, 'y_bottom': 0.4424076757749313}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2011_000753.jpg,0,truck,"{'x_left': 0.821515851648852, 'x_right': 0.9616102232755864, 'y_top': 0.04334663063624866, 'y_bottom': 0.413286574667091}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2011_000753.jpg,0,"","",traffic-light,"{'x_left': 0.08149561800260563, 'x_right': 0.9668002399072667, 'y_top': 0.25440137329378776, 'y_bottom': 0.8837804257735178}",0.5483175214891738
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2011_000753.jpg,0,"","",truck,"{'x_left': 0.06777141251658725, 'x_right': 0.9858757232625857, 'y_top': 0.0036388237825385195, 'y_bottom': 0.8911654794471764}",0.18220726194488202
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2011_000753.jpg,0,"","",car,"{'x_left': 0.4113943543536298, 'x_right': 0.9943189514949846, 'y_top': 0.15488761456109107, 'y_bottom': 0.9785810352571493}",0.17037061975029022
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2010_005364.jpg,0,truck,"{'x_left': 0.7211499479874045, 'x_right': 0.9808525551062757, 'y_top': 0.9075398530491968, 'y_bottom': 0.965455589700479}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2010_005364.jpg,0,car,"{'x_left': 0.9463358841265729, 'x_right': 0.9740749436395301, 'y_top': 0.7288394870413163, 'y_bottom': 0.7581365702756696}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2010_005364.jpg,0,pedestrian,"{'x_left': 0.14286272565145897, 'x_right': 0.9203391402990826, 'y_top': 0.643751508026241, 'y_bottom': 0.8944058141852372}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2010_005364.jpg,0,traffic-light,"{'x_left': 0.35961338630076967, 'x_right': 0.40658112854256917, 'y_top': 0.8308644102469491, 'y_bottom': 0.8732949131563695}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2010_005364.jpg,0,bus,"{'x_left': 0.5833657273948736, 'x_right': 0.8374848805983268, 'y_top': 0.30027764239612054, 'y_bottom': 0.3814474567028219}","","",""
VOCtrainval_11-May-2012/VOCdevkit/VOC2012/JPEGImages/2010_005364.jpg,0,"","",truck,"{'x_left': 0.1696867615155303, 'x_right': 0.9984250232135757, 'y_top': 0.23597941793683774, 'y_bottom': 0.9982859463247853}",0.8464981925824753

The first line should be the header line with the names of the fields. As described in the general structure, common fields file_path and frame_idx_in_file fields must be present. Additionally, for an analyze object detection job, the following fields must be present.

1. gt_class(string): Ground truth class label.
2. Ground truth bounding box: The bounding box can be inputted in different forms
   1. JSON string: Ground truth bounding box coordinates as a JSON string. This form can be directly used to define an analyze object detection job. The other two forms below must be inputted into a virtual column definition during Catalog view creation.
      1. The string must be formatted as a JSON with x_left, x_right, y_top and y_bottom keys.
      2. All key values must be normalized between 0 and 1 using the width and height of the image.
      3. A sample string is {'x_left':0.5, 'y_top':0.75, 'x_right':0.7, 'y_bottom':0.95}.
   2. 4 columns with normalized values: 4 individual columns representing 0-1 normalized values of x_left, y_top, x_right, and y_bottom can also be used to specify the bounding box.
   3. Absolute coordinates with height and width:  6 individual columns with four columns for absolute x_left, y_top, x_right and y_bottom coordinates and two columns for image height and width can also be used to specify the bounding box.
3. pd_class(string);  Predicted class label.
4. Prediction bounding box:  Prediction bounding box with the same format as the ground truth bounding box defined above.
5. pd_score(float): Prediction confidence score.

Additionally,

1. Each row in the catalog should specify one ground truth or one prediction. So, if an image/frame has five ground truths and six predictions, then eleven rows must be present in the input CSV.
2. The ground truth label and corresponding ground truth bounding box must be in the same row.
3. The prediction confidence, prediction label and prediction bounding box must be in the same row.