Text search

Vision Copilot provides various visual input-based search techniques, including full-image search, patch search, and full-object search. Text search augments these capabilities with support for free-form text inputs.

1. A dataset will get text search capability through a pre-registered pipeline named 'AkridataImageClipPipeline'(for the Image data type) and 'AkridataVideoClipPipeline'(for the Video data type). Read the article to learn more about pipelines.

2. The corresponding pipeline must be attached to the dataset. Read the article to learn more about pipeline attachments to datasets.

3. The pipeline must be run to produce features that allow text search. Read the article to learn more about executing a pipeline.

4. The text feature pipeline will populate a catalog on the dataset catalog page. Read the article to learn more about catalog operations.

Create a Visualization Job with Text Search

To create a job with text search capabilities, follow the steps below.

1. On the Dataset page, click the Catalog button to open the catalog page. Note that in the catalog page, the default pipeline is set to “all_pipeline_tables”.
   

2. Click Visualize to create a visualization job. Refer to the article Create an explore job for the detailed steps.
   Once the job is created, note that the job card indicates “Text search supported” by default.
   

Create a View for Text Search

The all_pipeline_tables view described above supports text search and is created by default for every dataset. The below section describes creating a custom view with text search support.

1. Create a view that joins the catalog table from a pipeline that ingested image features and the text-search capable pipeline. The file_id and frame_idx_in_file fields must be used to join the catalog tables. Read the article to learn more about catalog views.

   • The screenshot below shows two pipeline tables that can be used to create a view that supports text search jobs.
     

     

   • Create a view with file_id and frame_idx_in_file as the join columns. The 'frame_idx_in_file' column is not needed for the image type of the dataset.
     

2. Once the view is created, run the catalog query and create a visualize job.

   1. The icon highlighted below can identify a job with text search support.
      

      

   2. The job visualization shows a 'Text Search' tab shown below.

      

Perform Text Search on the Job

1. On the job page, type text prompt search as per the following steps:

   1. Enter text prompt.

   2. Enter the required number of results.

   3. Optionally select the filters and tunables to further reduce the search space. Refer to the article for details about tunables and filters.  

   4. Click the 'Search' button.
       

2. Use the following controls on the text search results:

   1. Group Images: For Video datasets, successive images that are expected to be similar can be grouped together using this control to reduce redundancy in the results.
      

   2. Sim score settings: You can view the sim search match score and filter results above a threshold score.
       

   3. Highlight points: View the results on a 2D plot that shows the relative position of the results in the content feature space.