Import catalog general structure

In SaaS mode, the following options to import/connect to the catalog are available.

1. Registering an External catalog and creating a view - this connects Data Explorer, and the entries are fetched only when queries are run.
2. Import catalog on the web portal UI and create a view - this uploads the catalog entries in a CSV(comma-separated value) file to a table within Data Explorer.

To create a view (virtual table), the imported (or connected external catalog) must have one or more columns that can be used to join the imported/connected table with the tables produced by the Data Explorer featurization process. The columns available for this purpose are:

1. file_path: Full path name of the file whose features are ingested to Data Explorer.
2. file_name: Relative path of the file with respect to the URI registered in Container.
3. frame_idx_in_file: For Video datasets, the imported/connected table must join on a per-frame basis, and hence a field that represents the frame in a video file is required.

Either among (file_path or file_name) and frame_idx_in_file (for Video) must be present in the imported/connected catalog. The column names for these fields can be anything, but they should represent the entity described above. Beyond these mandatory fields, there can be additional fields in the imported/connected catalog.

A sample CSV file for the imported/connected catalog table is below. It must be noted that the abs_file_path and frame_in_file represent file_path and frame_idx_in_file, respectively, with different column names.

abs_file_path, frame_in_file, class, confidence
s3a://my-bucket/my-file-1.mov, 0, car, 0.9
s3a://my-bucket/my-file-1.mov, 4, bus, 0.7
s3a://my-bucket/my-file-2.mov, 0, person, 0.68