Data Transfer

Uploading Data

Uploading data to the platform makes it available for consumption by analysis workflows and tools. There are multiple methods to upload data. Prior to performing an upload operation, a volume must be created.

Projects created via the platform console GUI automatically create a volume with the same name as the project. To make data available within a project, data should be uploaded to the volume associated with the project.

The examples below assume a project "myProject" and its associated volume have been created.

Method 1 - Command-line Interface

The example below demonstrates uploading a file test.txt to the project volume.

ica files upload ~/test.txt gds://myproject/test.txt

Method 2 - Cloud-native Tooling

The object models for volume, file, and folder resources include upload credentials for the underlying cloud-native object store. These credentials can be used for uploading data through the cloud-native tooling including command-line interfaces, SDKs, etc. To retrieve the upload credentials, create the data resource object using the GDS API or command-line interface. In the example below, a file resource is created and AWS S3 credentials are returned and used to upload the file content.

$ ica files create gds://myproject/test.txt --with-access
id: fil.578bd17b70f2447840fc08d82EXAMPLE
gdsPath: gds://myproject/test.txt
objectStoreAccess:
        sessionId:
        awsS3TemporaryUploadCredentials:
                bucketName: ica-gds-use1
                region: us-east-1
                accessKeyId: ASIA2...
                secretAccessKey: oYx2...
                sessionToken: Fwo...
                keyPrefix: 0203b81e-336d-4b2a-8f82-08d812430000/test.txt
                expirationDate: 2020-09-16 22:15:26 -0700 PDT

The temporary upload credentials can then be used with the AWS CLI to upload data directly to the underlying S3 storage.

export AWS_ACCESS_KEY_ID=ASIA2...
export AWS_SECRET_ACCESS_KEY=oYx2...
export AWS_SESSION_TOKEN=Fwo...
aws s3 cp ~/test.txt s3://ica-gds-use1/0203b81e-336d-4b2a-8f82-08d812430000/test.txt

After the upload finishes the S3, refer to GDS Upload Sessions for indicating that the upload has finished so the data can be immediately used.

For more information about using the AWS CLI, visit the AWS CLI documentation.

Method 3 - Drag-and-drop

Use the following instructions to upload data manually via the drag-and-drop interface in the platform UI. This method works best for small data transfers.

1. Select a project.

2. From the projects menu, select Data.

3. To add data, use one of the following methods. Make sure the Illumina Connected Analytics tab is open while data uploads.

   • Drag a file from your system into Choose a file or drag it here box.

   • Select the Choose a file or drag it here box, and then choose a file. Select Open to upload the file.

Your file or files are added to the Data page when upload completes.

Downloading Data

The data object models for content hosted in the platform contain direct URLs to the underlying cloud storage used to retrieve the data content. Use the command-line interface to download data through the GDS API.

Method 1 - Command-line interface download command

The command-line interface provides download commands for files and folders.

ica files download gds://myproject/test.txt .

Method 2 - Direct URL

The data object model shows the direct URL to be used for retrieving content through an HTTP client, such as a browser.

ica files get gds://myproject/test.txt
...
path          /test.txt
presignedUrl  https://ica-gds-use1.s3.us-west-2.amazonaws.com/0203b81e-EXAMPLE
...

The presignedUrl in the resource model output contains a link to the data content. Pasting the link into a web browser or using an HTTP client using a GET request on the URL will retrieve the data content.

Connectors

The platform UI provides a data transfer mechanism called Service Connectors used to synchronize data stored on the server to a machine with the client application running. Data connectors check your system for data to upload or download based on the rules set when creating the connector.

See Connectors for more information.