Video sequence search

A video dataset is ingested as a sequence of frames. For certain use cases like searching for a specific action, the features of a sequence of frames need to be viewed as an aggregate unit. In this direction, Video sequence search allows searching for matching sequences. A video sequence search is available in an 'Explore Sequence' type of job.

Creating an Explore-sequence job

1. On the catalog page of a video dataset, navigate to the 'Edit query' and the 'Advanced' tab. Review and edit the default values provided for the following fields.

   1. Sequence length: Duration of the sequence in seconds.

   2. Sequence stride: Gap between successive sequences in seconds. The video is divided into chunks of sequence stride length, and 1 frame matching the catalog query condition is chosen as the anchor(representative) frame from each stride. The sequence stride determines the overlap between successive sequences with a sequence stride = 0.5 * sequence length, implying a  50% overlap.

   3. Sequence offset: Time before the anchor frame(the frame chosen by the catalog query) that defines the start of the sequence. This is intended for use cases where the sequence must start a few frames before the frame with a specific attribute in the catalog. For example, the catalog may have a column  'traffic_light=yes/no', which will be used as the catalog filter condition. If there is a frame at the 5th second with traffic_light=yes, then a sequence offset=1 will start the sequence at the 4th second.
      

      

2. Review other fields of catalog query and click 'Submit for Visualize'.  Select the job type as 'Explore Sequence'. Fill in other parameters and submit the job for visualization.
   

   

Explore sequences

The job visualization view has the following video sequence related capabilities.

1. The icon indicates the presence of video sequence capabilities.

2. Every thumbnail has a 'Generate video' button to produce a thumbnail video for the sequence. The video has play/pause, expand and speed controls.

3. The job details page presented on clicking the job name has sequence configuration parameters.

4. The job details page has a 'Version' field. A version >= 2.0 supports video sequence search capabilities.

5. A click on the thumbnail opens the action bar. The 'Catalog tags' action shows catalog tags for the anchor frame(the representative frame for the sequence). The 'thumbs-up' and 'thumbs-down' actions can be used to add the sequences to similarity search.

Searching video sequences

Video sequence search has the following steps

1. Select sequences that has the event of interest.

2. For each sequence, mark ‘key frames’ that distinctly define the sequence. Only the key frames influence the search results.

3. On each ‘key frame’, mark the object of interest with bounding boxes.

4. Run search. Review results and provide positive and negative feedback to further refine the search.

The video sequence search has a ‘trajectory mode’ toggle that influences the search criteria. Use trajectory mode when you are looking to find results with the same object and trajectory as in the input(query) sequences. For example, if you want to look for ‘person moving from left to right’ Vs ‘sequence with persons’, then the ‘trajectory mode’ must be enabled.

If you have created a video sequence job with a “patch” pipeline run, you can search for the frames by defining the trajectory path.

1. Navigate to Repo > Dataset, and click the Visualize button on the job card for the video sequence dataset.
   

2. Click a data point to view all image frames with associated video sequence, select an image, and click the thumbs-up icon.

3. Click an image frame and enable the Patch option.
   
   By default, the Trajectory mode is enabled for the selected frame for which the patch pipeline is already run on the video sequence.
   

4. Select the Frames icon on the selected images to open the series of image frames.
   

5. Select the key frames to mark the objects and define the trajectory path.
   

   1. Draw the bounding box to mark the same object in a specific direction, say from Left to Right, in the frames.

   2. Click OK.

6. Click Search on the dataset page to search for the frames based on the defined trajectory mode.
   

7. From the search results, select an image frame. and click the Frames icon to view the results with the matching trajectory as per the key frames.
   

   The page displays the number of matched keyframes.
   

8. Enable the options Show Matching Frames Only and Show Matching Regions.
   As you can see that the image frames displays the matching regions as per the trajectory you had defined with the bounding box for search.
   You can review each such frame from the search results.

9. Once you are satisfied with the search results, click the + (plus) sign at the bottom-right corner of the page, and select all points to save the results as a result set.
   

You can save the result to an existing resultset or create a new result set as explained in the Resultset topic.