Video sequence search

This feature is currently in the BETA stage.

A video dataset is ingested as a sequence of frames. For certain use cases like searching for a specific action, the features of a sequence of frames need to be viewed as an aggregate unit. In this direction, Video sequence search allows searching for matching sequences. A video sequence search is available in an 'Explore Sequence' type of job.

Creating an Explore-sequence job

1. On the catalog page of a video dataset, navigate to the 'Edit query' and the 'Advanced' tab. Review and edit the default values provided for the following fields.
   1. Sequence length: Duration of the sequence in seconds.
   2. Sequence stride: Gap between successive sequences in seconds. The video is divided into chunks of sequence stride length, and 1 frame matching the catalog query condition is chosen as the anchor(representative) frame from each stride. The sequence stride determines the overlap between successive sequences with a sequence stride = 0.5 * sequence length, implying a  50% overlap.
   3. Sequence offset: Time before the anchor frame(the frame chosen by the catalog query) that defines the start of the sequence. This is intended for use cases where the sequence must start a few frames before the frame with a specific attribute in the catalog. For example, the catalog may have a column  'traffic_light=yes/no', which will be used as the catalog filter condition. If there is a frame at the 5th second with traffic_light=yes, then a sequence offset=1 will start the sequence at the 4th second.
      
2. Review other fields of catalog query and click 'Submit for Visualize'.  Select the job type as 'Explore Sequence'. Fill in other parameters and submit the job for visualization.
   

Explore sequences

The job visualization view has the following video sequence related capabilities.

1. The icon indicates the presence of video sequence capabilities.
2. Every thumbnail has a 'Generate video' button to produce a thumbnail video for the sequence. The video has play/pause, expand and speed controls.
3. The job details page presented on clicking the job name has sequence configuration parameters.
4. The job details page has a 'Version' field. A version >= 2.0 supports video sequence search capabilities.
5. A click on the thumbnail opens the action bar. The 'Catalog tags' action shows catalog tags for the anchor frame(the representative frame for the sequence). The 'thumbs-up' and 'thumbs-down' actions can be used to add the sequences to similarity search.

Searching video sequences

1. Click the 'thumbs-up' or 'thumbs-down' buttons to add the sequence to the search set as mentioned in the previous section.
2. On the action bar on the thumbnail in the search set, click the highlighted button to expand the sequence and annotate keyframes that distinctly characterize the sequence to search.
   
3. Select keyframes and draw bounding boxes on the selected keyframes. The bounding boxes are available only if the 'Patch' toggle is on in the previous screen.
   
4. Click 'OK' to come back to the main search screen.
5. The number on the 'Select key frames' button shows the number of keyframes annotated.
   
6. Review other parameters and click the 'Quick Search' or 'Search' button.
7. Review the results. Use the 'Show matched key frame' button highlighted below to get additional details.
   
8. On clicking the 'Show matched key frame' button, the page shown has controls to view the matching keyframes and the matching regions of the keyframe that resulted in the sequence appearing in the search results.
   
   
9. Use the 'add to resultset' button to add the entire sequence to a resultset.