Bucket Replication

• 1: Requirements to Set Up Bucket Replication
• 2: Enable One-Way Server-Side Bucket Replication
• 3: Enable Two-Way Server-Side Bucket Replication
• 4: Enable Multi-Site Server-Side Bucket Replication
• 5: Resynchronize Bucket from Remote Replica

1 - Requirements to Set Up Bucket Replication

Bucket replication uses rules to synchronize the contents of a bucket on one MinIO deployment to a bucket on a remote MinIO deployment.

Replication can be done in any of the following ways:

• Active-Passive Eligible objects replicate from the source bucket to the remote bucket. Any changes on the remote bucket do not replicate back.
• Active-Active Changes to eligible objects of either bucket replicate to the other bucket in a two-way direction.
• Multi-Site Active-Active Changes to eligible objects on any bucket set up for bucket replication replicate to all of the other buckets.

Ensure you meet the following prerequisites before you set up any of these replication configurations.

Permissions Required for Setting Up Bucket Replication

Bucket replication requires specific permissions on the source and destination deployments to configure and enable replication rules.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget",
                "admin:ListBatchJobs",
                "admin:DescribeBatchJob",
                "admin:StartBatchJob",
                "admin:CancelBatchJob"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}

wget -O - https://silo.pgsty.com/extra/examples/ReplicationAdminPolicy.json | \
mc admin policy create TARGET ReplicationAdminPolicy /dev/stdin
mc admin user add TARGET ReplicationAdmin LongRandomSecretKey
mc admin policy attach TARGET ReplicationAdminPolicy --user=ReplicationAdmin

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}

wget -O - https://silo.pgsty.com/extra/examples/ReplicationRemoteUserPolicy.json | \
mc admin policy create TARGET ReplicationRemoteUserPolicy /dev/stdin
mc admin user add TARGET ReplicationRemoteUser LongRandomSecretKey
mc admin policy attach TARGET ReplicationRemoteUserPolicy --user=ReplicationRemoteUser

See mc admin user, mc admin user svcacct, and mc admin policy for more complete documentation on adding users, access keys, and policies to a MinIO deployment.

Matching Object Encryption Settings for Bucket Replication

MinIO supports replication of objects encrypted using SSE-KMS and SSE-S3:

• For objects encrypted using SSE-KMS, MinIO requires that the target bucket support SSE-KMS encryption of objects using the same key names used to encrypt objects on the source bucket.
• For objects encrypted using SSE-S3, MinIO requires that the target bucket also support SSE-S3 encryption of objects regardless of key name.

As part of the replication process, MinIO decrypts the object on the source bucket and transmits the unencrypted object over the network. The destination MinIO deployment then re-encrypts the object using the encryption settings from the target. MinIO therefore strongly recommends enabling TLS on both source and destination deployments to ensure the safety of objects during transmission.

MinIO does not support replicating client-side encrypted objects (SSE-C).

Bucket Replication Requires MinIO Deployments

MinIO server-side replication only works between MinIO deployments. Both the source and destination deployments must run MinIO Server with matching versions.

To configure replication between arbitrary S3-compatible services, use mc mirror.

Replication Requires Versioning

MinIO relies on the immutability protections provided by versioning to support replication and resynchronization.

Use mc version info to validate the versioning status of both the source and remote buckets. Use the mc version enable command to enable versioning as necessary.

If you exclude a prefix or folder from versioning within the source bucket, MinIO cannot replicate objects within that folder or prefix.

Matching Object Locking State With Bucket Replication

MinIO supports replicating objects held under WORM Locking. Both replication buckets must have object locking enabled for MinIO to replicate the locked object. For active-active configuration, MinIO recommends using the same retention rules on both buckets to ensure consistent behavior across sites.

You must enable object locking during bucket creation as per S3 behavior. You can then configure object retention rules at any time. Configure the necessary rules on the unhealthy target bucket prior to beginning this procedure.

2 - Enable One-Way Server-Side Bucket Replication

The procedure on this page creates a new bucket replication rule for one-way synchronization of objects from one MinIO bucket to another MinIO bucket. The buckets can be on the same MinIO deployment or on separate MinIO deployments.

• To configure two-way “active-active” replication between MinIO buckets, see Enable Two-Way Server-Side Bucket Replication.
• To configure multi-site “active-active” replication between MinIO deployments, see Enable Multi-Site Server-Side Bucket Replication

Requirements

Replication requires all participating clusters meet the following requirements. This procedure assumes you have reviewed and validated those requirements.

For more details, see the Bucket Replication Requirements page.

Considerations

Click to expand any of the following:

Procedure

• Configure One-Way Bucket Replication Using the Command Line

  • Create a Replication Remote Target
  • Create a New Bucket Replication Rule
  • Validate the Replication Configuration

Configure One-Way Bucket Replication Using the Command Line mc

This procedure uses the aliases SOURCE and REMOTE to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments.

This procedure assumes each alias corresponds to a user with the necessary replication permissions.

1) Create a New Bucket Replication Rule

Use the mc replicate add command to add a new replication rule to each MinIO deployment.

mc replicate add ALIAS/BUCKET \
   --remote-bucket 'https://USER:PASSWORD@HOSTNAME:PORT/BUCKET' \
   --replicate "delete,delete-marker,existing-objects"

• Replace ALIAS with the alias of the origin MinIO deployment. The name must match the bucket specified when creating the remote target in the previous step.

• Replace BUCKET with the name of the bucket to replicate from on the origin deployment.

• Replace the --remote-bucket to specify the remote MinIO deployment and bucket to which the ALIAS/BUCKET replicates.

  The USER:PASSWORD must correspond to a user on the remote deployment with the necessary replication permissions.

  The HOSTNAME:PORT must resolve to a reachable MinIO instance on the remote deployment. The BUCKET must exist and otherwise meet all other replication requirements.

• The --replicate "delete,delete-marker,existing-objects" flag enables the following replication features:

  • Replication of Deletes
  • Replication of existing Objects

  See mc replicate add --replicate for more complete documentation. Omit any field to disable replication of that component.

Specify any other supported optional arguments for mc replicate add.

2) Validate the Replication Configuration

Use mc cp to copy a new object to the replicated bucket on one of the deployments.

mc cp ~/foo.txt ALIAS/BUCKET

Use mc ls to verify the object exists on the destination bucket:

mc ls ALIAS/BUCKET

3 - Enable Two-Way Server-Side Bucket Replication

The procedure on this page creates a new bucket replication rule for two-way “active-active” synchronization of objects between MinIO buckets.

• To configure replication between arbitrary S3-compatible services, use mc mirror.
• To configure one-way “active-passive” replication between MinIO clusters, see Enable One-Way Server-Side Bucket Replication.
• To configure multi-site “active-active” replication between MinIO clusters, see Enable Multi-Site Server-Side Bucket Replication.

This tutorial covers configuring Active-Active replication between two MinIO clusters. For a tutorial on multi-site replication between three or more MinIO clusters, see Enable Multi-Site Server-Side Bucket Replication.

Requirements

You must meet all of the basic requirements for bucket replication described in Bucket Replication Requirements.

In addition, to set up active-active bucket replication, you must meet the following additional requirements:

Access to Both Clusters

You must have network access and login credentials with required permissions to both deployment to set up active-active bucket replication.

You can access the deployments by installing mc and using the command line. Use the mc alias set command to create an alias for both MinIO deployments.

Alias creation requires specifying an access key for a user on the deployment. This user must have permission to create and manage users and policies on the deployment.

Specifically, ensure the user has at minimum:

• admin:CreateUser
• admin:ListUsers
• admin:GetUser
• admin:CreatePolicy
• admin:GetPolicy
• admin:AttachUserOrGroupPolicy

Considerations

Procedure

• Configure Two-Way Bucket Replication Using the Command Line

  • Create Replication Remote Targets
  • Create a New Bucket Replication Rule on Each Deployment
  • Validate the Replication Configuration

Configure Two-Way Bucket Replication Using the Command Line mc

This procedure creates two-way, active-active replication between two MinIO deployments.

This procedure assumes you have already defined an alias for each deployment as a user with the necessary replication permissions.

1) Create a New Bucket Replication Rule on Each Deployment

Use the mc replicate add command to add a new replication rule to each MinIO deployment.

mc replicate add ALIAS/BUCKET \
   --remote-bucket 'https://USER:PASSWORD@HOSTNAME:PORT/BUCKET' \
   --replicate "delete,delete-marker,existing-objects"

• Replace ALIAS with the alias of the origin MinIO deployment. The name must match the bucket specified when creating the remote target in the previous step.

• Replace BUCKET with the name of the bucket to replicate from on the origin deployment.

• Replace the --remote-bucket to specify the remote MinIO deployment and bucket to which the ALIAS/BUCKET replicates.

  The USER:PASSWORD must correspond to a user on the remote deployment with the necessary replication permissions.

  The HOSTNAME:PORT must resolve to a reachable MinIO instance on the remote deployment. The BUCKET must exist and otherwise meet all other replication requirements.

• The --replicate "delete,delete-marker,existing-objects" flag enables the following replication features:

  • Replication of Deletes
  • Replication of existing Objects

  See mc replicate add --replicate for more complete documentation. Omit any field to disable replication of that component.

Specify any other supported optional arguments for mc replicate add.

Repeat this step on the other MinIO deployment. Change the ALIAS and --remote-bucket values to correspond to the first deployment.

You should have two replication rules configured at the conclusion of this step - one created on each deployment that points to the bucket on the other deployment. Use the mc replicate ls command to verify the created replication rules.

2) Validate the Replication Configuration

Use mc cp to copy a new object to the replicated bucket on one of the deployments.

mc cp ~/foo.txt ALIAS/BUCKET

Use mc ls to verify the object exists on the destination bucket:

mc ls ALIAS/BUCKET

Repeat this test by copying another object to the second deployment and verifying the object replicates to the first deployment.

Once both objects exist on both deployments, you have successfully set up two-way, active-active replication between MinIO buckets.

4 - Enable Multi-Site Server-Side Bucket Replication

The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the Enable Two-Way Server-Side Bucket Replication procedure with additional considerations required to ensure predictable replication behavior across all sites.

• To configure replication between arbitrary S3-compatible services, use mc mirror.
• To configure one-way “active-active” replication between two MinIO deployments, see Enable Two-Way Server-Side Bucket Replication.
• To configure one-way “active-passive” replication between MinIO deployments, see Enable One-Way Server-Side Bucket Replication.

Multi-Site Active-Active replication configurations can span multiple racks, datacenters, or geographic locations. Complexity of configuring and maintaining multi-site configurations generally increase with the number of sites and size of each site. Enterprises looking to implement multi-site replication should consider leveraging MinIO SUBNET support to access the expertise, planning, and engineering resources required for addressing that use case.

Requirements

You must meet all of the basic requirements for bucket replication described in Bucket Replication Requirements.

In addition, to create multi-site bucket replication set up, you must meet the following additional requirements:

Access to All Clusters

You must have network access and log in credentials with correct permissions to all deployments to set up multi-site active-active bucket replication.

You can access the deployments by installing mc and using the command line. Use the mc alias set command to create an alias for each MinIO deployment.

Alias creation requires specifying an access key for a user on the deployment. This user must have permission to create and manage users and policies on the deployment.

Specifically, ensure the user has at minimum:

• admin:CreateUser
• admin:ListUsers
• admin:GetUser
• admin:CreatePolicy
• admin:GetPolicy
• admin:AttachUserOrGroupPolicy

Considerations

Click to expand any of the following:

Procedure

This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure before attempting to implement the documented steps.

• Configure Multi-Site Bucket Replication Using the Command Line

  • Create New Bucket Replication Rules
  • Validate the Replication Configuration

Configure Multi-Site Bucket Replication Using the Command Line mc

This procedure uses the placeholder ALIAS to reference the alias each MinIO deployment being configured for replication. Replace these values with the appropriate alias for each MinIO deployment.

This procedure assumes each alias corresponds to a user with the necessary replication permissions.

1) Create New Bucket Replication Rules

Use the mc replicate add command to add a new replication rule to each MinIO deployment.

mc replicate add ALIAS/BUCKET \
   --remote-bucket 'https://USER:PASSWORD@HOSTNAME:PORT/BUCKET' \
   --replicate "delete,delete-marker,existing-objects"

• Replace ALIAS with the alias of the origin MinIO deployment. The name must match the bucket specified when creating the remote target in the previous step.

• Replace BUCKET with the name of the bucket to replicate from on the origin deployment.

• Replace the --remote-bucket to specify the remote MinIO deployment and bucket to which the ALIAS/BUCKET replicates.

  The USER:PASSWORD must correspond to a user on the remote deployment with the necessary replication permissions.

  The HOSTNAME:PORT must resolve to a reachable MinIO instance on the remote deployment. The BUCKET must exist and otherwise meet all other replication requirements.

• The --replicate "delete,delete-marker,existing-objects" flag enables the following replication features:

  • Replication of Deletes
  • Replication of existing Objects

  See mc replicate add --replicate for more complete documentation. Omit any field to disable replication of that component.

Specify any other supported optional arguments for mc replicate add.

Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments minio1, minio2, and minio3 would require repeating this step on each deployment for each remote.

Specifically, in this scenario, perform this step twice on each deployment:

• On the minio1 deployment, once for a rule for minio2 and again for a separate rule for minio3.
• On the minio2 deployment, once for a rule for minio1 and again for a separate rule for minio3.
• On the minio3 deployment, once for a rule for minio1 and again for a separate rule for minio2.

2) Validate the Replication Configuration

Use mc cp to copy a new object to the replicated bucket on one of the deployments.

mc cp ~/foo.txt ALIAS/BUCKET

Use mc ls to verify the object exists on the destination bucket:

mc ls ALIAS/BUCKET

Repeat this test on each deployment by copying a new unique file and checking that the file replicates to each of the other deployments.

You can also use mc stat to check the file to check the current replication stage of the object.

5 - Resynchronize Bucket from Remote Replica

The procedure on this page resynchronizes the contents of a MinIO bucket using a healthy replication remote. Resynchronization supports recovery after partial or total loss of data on a MinIO deployment in a replica configuration.

For example, consider a MinIO active-active replication configuration similar to the following:

Resynchronization allows using the healthy data on one of the participating MinIO deployments as the source for rebuilding the other deployment.

Resynchronization is a per-bucket process. You must repeat resynchronization for each bucket on the remote which suffered partial or total data loss.

Requirements

MinIO Deployments Must Be Online

Resynchronization requires both the source and target deployments be online and able to accept read and write operations. The source must have complete network connectivity to the remote.

The remote deployment may be “unhealthy” in that it has suffered partial or total data loss. Resynchronization addresses the data loss as long as both source and destination maintain connectivity.

Resynchronization Requires Existing Replication Configuration

Resynchronization requires the healthy source deployment have an existing replication configuration for the unhealthy target bucket. Additionally, resynchronization only applies to those replication rules created with the existing object replication option.

Use mc replicate ls to review the configured replication rules and targets for the healthy source bucket.

Replication Requires Matching Object Encryption Settings

MinIO supports replication of objects encrypted using SSE-KMS and SSE-S3:

• For objects encrypted using SSE-KMS, MinIO requires that the target bucket support SSE-KMS encryption of objects using the same key names used to encrypt objects on the source bucket.
• For objects encrypted using SSE-S3, MinIO requires that the target bucket also support SSE-S3 encryption of objects regardless of key name.

As part of the replication process, MinIO decrypts the object on the source bucket and transmits the unencrypted object over the network. The destination MinIO deployment then re-encrypts the object using the encryption settings from the target. MinIO therefore strongly recommends enabling TLS on both source and destination deployments to ensure the safety of objects during transmission.

MinIO does not support replicating client-side encrypted objects (SSE-C).

Replication Requires MinIO Deployments

MinIO server-side replication only works between MinIO deployments. Both the source and destination deployments must run MinIO Server with matching versions.

To configure replication between arbitrary S3-compatible services, use mc mirror.

Replication Requires Versioning

MinIO relies on the immutability protections provided by versioning to support replication and resynchronization.

Use mc version info to validate the versioning status of both the source and remote buckets. Use the mc version enable command to enable versioning as necessary.

If you exclude a prefix or folder from versioning within the source bucket, MinIO cannot replicate objects within that folder or prefix.

Replication Requires Matching Object Locking State

MinIO supports replicating objects held under WORM Locking. Both replication buckets must have object locking enabled for MinIO to replicate the locked object. For active-active configuration, MinIO recommends using the same retention rules on both buckets to ensure consistent behavior across sites.

You must enable object locking during bucket creation as per S3 behavior. You can then configure object retention rules at any time. Configure the necessary rules on the unhealthy target bucket prior to beginning this procedure.

Considerations

Resynchronization Requires Time

Resynchronization is a background processes that continually checks objects in the source MinIO bucket and copies them to the remote as-needed. The time required for replication to complete may vary depending on the number and size of objects, the throughput to the remote MinIO deployment, and the load on the source MinIO deployment. Total time for completion is generally not predictable due to these variables.

MinIO recommends configuring load balancers or proxies to direct traffic only to the healthy cluster until synchronization completes. The following commands can provide insight into the resynchronization status:

• mc replicate resync status on the source to track the resynchronization progress.
• mc replicate status on the source and remote to track normal replication data.
• Run mc ls -r --versions ALIAS/BUCKET | wc -l against both source and remote to validate the total number of objects and object versions on each.

Resynchronize Objects after Data Loss

This procedure uses an existing MinIO replication configuration to restore missing data to one of the MinIO deployments participating in that configuration. Specifically, a healthy MinIO deployment (the SOURCE) synchronizes it’s existing data to the unhealthy MinIO deployment (the TARGET).

This procedure assumes an existing alias for the SOURCE that has the necessary permissions for configuring replication.

You can repeat this procedure for each bucket that requires resynchronization. You can have no more than one replication job running per bucket.

1) List the Configured Replication Targets on the Healthy Source

Run the mc replicate ls command to list the configured remote targets on the healthy SOURCE deployment for the BUCKET that requires resynchronization.

mc replicate ls SOURCE/BUCKET --json

• Replace SOURCE with the alias of the source MinIO deployment.
• Replace BUCKET with the name of the bucket to use as the source for resynchronization.

The output resembles the following:

{
   "op": "",
   "status": "success",
   "url": "",
   "rule": {
      "ID": "cer1tuk9a3p5j68crk60",
      "Status": "Enabled",
      "Priority": 0,
      "DeleteMarkerReplication": {
         "Status": "Enabled"
      },
      "DeleteReplication": {
         "Status": "Enabled"
      },
      "Destination": {
         "Bucket": "arn:minio:replication::UUID:BUCKET"
      },
      "Filter": {
         "And": {},
         "Tag": {}
      },
      "SourceSelectionCriteria": {
         "ReplicaModifications": {
            "Status": "Enabled"
         }
      },
      "ExistingObjectReplication": {
         "Status": "Enabled"
      }
   }
}

Each document in the output represents one configured replication rule. The Destination.Bucket field specifies the ARN for a given rule on the bucket. Identify the correct ARN for the Bucket from which you want to resynchronize objects.

2) Start the Resynchronization Procedure

Run the mc replicate resync start command to begin the resynchronization process:

mc replicate resync start --remote-bucket "arn:minio:replication::UUID:BUCKET" SOURCE/BUCKET

• Replace the --remote-bucket value with the ARN of the unhealthy BUCKET on the TARGET MinIO deployment.
• Replaced SOURCE with the alias of the source MinIO deployment.
• Replace the BUCKET with the name of the bucket on the healthy SOURCE MinIO deployment.

The command returns a resynchronization job ID indicating that the process has begun.

3) Monitor Resynchronization

Use the mc replicate resync status command on the source deployment to track the received replication data:

mc replicate resync status ALIAS/BUCKET

The output resembles the following:

mc replicate resync status /data
Resync status summary:
● arn:minio:replication::6593d572-4dc3-4bb9-8d90-7f79cc612f01:data
   Status: Ongoing
   Replication Status | Size (Bytes)    | Count
   Replicated         | 2.3 GiB         | 18
   Failed             | 0 B             | 0

The Status updates to Completed once the resynchronization process completes.

4) Next Steps

• If the TARGET bucket damage extends to replication rules, you must recreate those rules to match the previous replication configuration. See Enable Two-Way Server-Side Bucket Replication for additional guidance.
• Perform basic validation that all buckets in the replication configuration show similar results for commands such as mc ls and mc stat.
• After restoring any replication rules and verifying replication between sites, you can configure the reverse proxy, load balancer, or other network control plane managing connections to resume sending traffic to the resynchronized deployment.