adectl - Resultset Upload

The resultset upload operation enables users to materialize objects in a resultset onto a target S3 bucket, Azure blob store, or local file system for further analysis or connection to downstream pipelines (e.g., training pipelines).

Configure the Target Location

adectl resultset config

This command will select the type of destination location (S3/Azure/GCP. etc.) and necessary credentials and configuration information. As an example, the fields captured for the S3 destination are as follows:

Select store type [s3 | azure | GCP | file | hdfs] : s3
Enter S3 Bucket Name: bucket
Enter S3 Access Key: xxxxx
Enter S3 Secret Key: yyyyy
Enter S3 Endpoint [default: https://s3.amazonaws.com]: 
Configured S3 Store Successfully

For Azure blob store as a destination, storage account key fields are needed.

Upload resultset objects

You can upload resultsets using either resultset ID or resultset name.

adectl resultset upload -n <resultsetname> -t <target-location>

OR

adectl resultset upload -r <resultsetid> -t <target-location> 

The command starts the upload operation as an asynchronous operation, and the status of this operation is available by running the following command.

adectl show

Note

If the resultset name is not unique, the command will prompt you to invoke the adectl resultset upload command with resultsetid.

Dump Resultset Objects

The command copies all the object file names available in the selected resultset.

adectl resultset dump -n <resultsetname> -o <output-dir>

OR

adectl resultset dump -r <resultsetid> -o <output-dir>