Object Storage Integration

Storage Provider Setup

To set up your object storage for Score:

1. Register with a provider of your choice and follow their instructions for setting up and configuring their service.

2. Create two data buckets for Score to use:

• A bucket to store object data
• A bucket to store and maintain state information

3. You may need to create a /data sub-folder in advance for each bucket. This requirement will depend on your storage provider and is summarized below:

4. Record the URL, access key and secret key used to access your storage service. These will be required later for configuring Score. Keep these values safe and secure.

Environment Variable Setup

Once your object storage is established, the next step involves configuring Score for connection. The specific configuration settings will vary based on your object storage provider. Below are the detailed setup instructions.

AWS, Ceph, or Minio

To connect Score with AWS, Ceph, or Minio storage, modify your .env.score file as follows:

bash

Copy

# ============================
# Object Storage Configruation Variables
# ============================

SPRING_PROFILES_ACTIVE=prod,aws

S3_ENDPOINT="http://localhost:9000"
S3_ACCESS_KEY="minio"
S3_SECRETKEY="minio123"
S3_SIGV4ENABLED="true"

BUCKET_NAME_OBJECT="object.bucket"
BUCKET_NAME_STATE="state.bucket"
BUCKET_SIZE_POOL=0
BUCKET_SIZE_KEY=2

UPLOAD_PARTSIZE=1048576
UPLOAD_RETRY_LIMIT=10
UPLOAD_CONNECTION_TIMEOUT=60000
UPLOAD_CLEAN_CRON="0 0 0 * * ?"
UPLOAD_CLEAN_ENABLED="true"

The following table summarizes the necessary variables:

Azure

For connecting Score with Azure storage, update .env.score as shown:

bash

Copy

SPRING_PROFILES_ACTIVE=prod,azure

AZURE_ENDPOINT_PROTOCOL=https
AZURE_ACCOUNT_NAME={{storage_account_name}}
AZURE_ACCOUNT_KEY={{storage_account_secret_key}}

BUCKET_NAME_OBJECT={{object_bucket}} # Object data storage bucket/container name
BUCKET_POLICY_UPLOAD={{write_policy}} # Access policy name for write operations
BUCKET_POLICY_DOWNLOAD={{read_policy}} # Access policy name for read operations

UPLOAD_PARTSIZE=104587
DOWNLOAD_PARTSIZE=250000000 # Default part size for downloads

OBJECT_SENTINEL=heliograph # Required sample object/file name for `ping` operations; default is `heliograph

The following table summarizes the Azure-specific variables:

Access Policy Configuration for Azure

For Azure storage, you must define a storage access policy for your container.

1. Access the Azure dashboard: Select containers from the left-hand menu.

2. Locate your container: Choose Access Policy from the dropdown menu.

   

3. Create write and readonly access policies: