Administration

• 1: Batch Replication
• 2: Managing Your Deployment
• 3: Batch Key Rotation
• 4: Managing Objects
• 5: Batch Expiration
• 6: Security and Access
• 7: SUBNET
• 8: Silo Console
• 9: Object Management
  • 9.1: Bucket Versioning
  • 9.2: Transition Objects to Remote MinIO Deployment
  • 9.3: Silo Object Locking
  • 9.4: Transition Objects from MinIO to S3
  • 9.5: Object Deletion
  • 9.6: Transition Objects from MinIO to GCS
  • 9.7: Object Lifecycle Management
  • 9.8: Transition Objects from MinIO to Azure
  • 9.9: Automatic Object Expiration
  • 9.10: Data Compression
• 10: Monitoring Bucket and Object Events
  • 10.1: Bucket notifications
  • 10.2: Publish Events to AMQP (RabbitMQ)
  • 10.3: Publish Events to MQTT
  • 10.4: Publish Events to NATS
  • 10.5: Publish Events to NSQ
  • 10.6: Publish Events to Elasticsearch
  • 10.7: Publish Events to Kafka
  • 10.8: Publish Events to MySQL
  • 10.9: Publish Events to PostgreSQL
  • 10.10: Publish Events to Redis
  • 10.11: Publish Events to Webhook
• 11: Identity and Access Management
  • 11.1: Silo Identity Management
  • 11.2: User Management
  • 11.3: Group Management
  • 11.4: OpenID Connect Access Management
  • 11.5: Active Directory / LDAP Access Management
  • 11.6: Silo External Identity Management Plugin
  • 11.7: Access Management
  • 11.8: Silo External Access Management Plugin
• 12: Server-Side Encryption of Objects
  • 12.1: Server-Side Encryption with Per-Bucket Keys (SSE-KMS)
  • 12.2: Server-Side Encryption Per-Deployment Key (SSE-S3)
  • 12.3: Server-Side Encryption with Client-Managed Keys (SSE-C)
• 13: Bucket Replication
  • 13.1: Requirements to Set Up Bucket Replication
  • 13.2: Enable One-Way Server-Side Bucket Replication
  • 13.3: Enable Two-Way Server-Side Bucket Replication
  • 13.4: Enable Multi-Site Server-Side Bucket Replication
  • 13.5: Resynchronize Bucket from Remote Replica
• 14: Batch Framework
• 15: Core Administration Concepts

1 - Batch Replication

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.

The replicate batch job replicates objects from one MinIO deployment (the source deployment) to another MinIO deployment (the target deployment). Either the source or the target must be the local deployment.

Batch Replication between MinIO deployments have the following advantages over using mc mirror:

• Removes the client to cluster network as a potential bottleneck
• A user only needs access to starting a batch job with no other permissions, as the job runs entirely server side on the cluster
• The job provides for retry attempts in event that objects do not replicate
• Batch jobs are one-time, curated processes allowing for fine control replication
• (MinIO to MinIO only) The replication process copies object versions from source to target

Starting with the MinIO Server RELEASE.2023-05-04T21-44-30Z, the other deployment can be either another MinIO deployment or any S3-compatible location using a realtime storage class. Use filtering options in the replication YAML file to exclude objects stored in locations that require rehydration or other restoration methods before serving the requested object. Batch replication to these types of remotes uses mc mirror behavior.

Behavior

Access Control and Requirements

Batch replication shares similar access and permission requirements as bucket replication.

The credentials for the “source” deployment must have a policy similar to the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget",
                "admin:ListBatchJobs",
                "admin:DescribeBatchJob",
                "admin:StartBatchJob",
                "admin:CancelBatchJob"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}

The credentials for the “remote” deployment must have a policy similar to the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}

See mc admin user, mc admin user svcacct, and mc admin policy for more complete documentation on adding users, access keys, and policies to a MinIO deployment.

MinIO deployments configured for Active Directory/LDAP or OpenID Connect user management can instead create dedicated access keys for supporting batch replication.

Filter Replication Targets

The batch job definition file can limit the replication by bucket, prefix, and/or filters to only replicate certain objects. The access to objects and buckets for the replication process may be restricted by the credentials you provide in the YAML for either the source or target destinations.

For example, you can use a batch job to perform a one-time replication sync to push objects from a bucket on a local deployment at minio-local/invoices/ to a bucket on a remote deployment at minio-remote/invoices. You can also pull objects from the remote deployment at minio-remote/invoices to the local deployment at minio-local/invoices.

Small File Optimization

Starting with RELEASE.2023-12-09T18-17-51Z, batch replication by default automatically batches and compresses objects smaller than 5MiB to efficiently transfer data between the source and remote. The remote MinIO deployment can check and immediately apply lifecycle management tiering rules to batched objects. The functionality resembles that offered by S3 Snowball Edge small file batching.

You can modify the compression settings in the replicate job configuration.

Replicate Batch Job Reference

The YAML must define the source and target deployments. If the source deployment is remote, then the target deployment must be local. Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.

replicate:
  apiVersion: v1
  source:
    type: minio
    bucket: mybucket
    prefix:
      - prefix1
      - prefix2
...

For the source deployment

• Required information

  type:	Must be minio.
  bucket:	The bucket on the deployment.

• Optional information

  prefix:	The prefix on the object(s) that should replicate. Beginning with MinIO Server RELEASE.2024-08-03T04-33-23Z, v2 of the Batch Job Replicate API allows you to list multiple prefixes. Specify replicate.apiVersion as v2 to replicate from multiple prefixes.
  endpoint:	Location of the deployment to use for either the source or the target of a replication batch job. For example, https://minio.example.net. If the deployment is the mc alias set specified to the command, omit this field to direct MinIO to use that alias for the endpoint and credentials values. Either the source deployment or the remote deployment must be the “local” alias. The non-“local” deployment must specify the endpoint and credentials.
  path:	Directs MinIO to use Path or Virtual Style (DNS) lookup of the bucket. - Specify on for Path style - Specify off for Virtual style - Specify auto to let MinIO determine the correct lookup style. Defaults to auto.
  credentials:	The accesskey: and secretKey: or the sessionToken: that grants access to the object(s). Only specify for the deployment that is not the local deployment.
  snowball	version added: RELEASE.2023-12-09T18-17-51Z Configuration options for controlling the batch-and-compress functionality.
  snowball.disable	Specify true to disable the batch-and-compress functionality during replication. Defaults to false.
  snowball.batch	Specify the maximum integer number of objects to batch for compression. Defaults to 100.
  snowball.inmemory	Specify false to stage archives using local storage or true to stage to memory (RAM). Defaults to true.
  snowball.compress	Specify true to generate compress batched objects over the wire using the S2/Snappy compression algorithm. Defaults to false or no compression.
  snowball.smallerThan	Specify the size of object in Megabits (MiB) under which MinIO should batch objects. Defaults to 5MiB.
  snowball.skipErrs	Specify false to direct MinIO to halt on any object which produces errors on read. Defaults to true.

For the target deployment

• Required information

  type:	Must be minio.
  bucket:	The bucket on the deployment.

• Optional information

  prefix:	The prefix on the object(s) to replicate.
  endpoint:	The location of the target deployment. If the target is the alias specified to the command, you can omit this and the credentials fields. If the target is “local”, the source must specify the remote deployment with endpoint and credentials.
  credentials:	The accesskey and secretKey or the sessionToken that grants access to the object(s).

For filters

For notifications

For retry attempts

If something interrupts the job, you can define how many attempts to retry the job batch. For each retry, you can also define how long to wait between attempts.

Sample YAML Description File for a replicate Job Type

Use mc batch generate to create a basic replicate batch job for further customization.

For the local deployment, do not specify the endpoint or credentials. Either delete or comment out those lines for the source or the target section, depending on which is the local.

replicate:
  apiVersion: v1
  # source of the objects to be replicated
  source:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX # 'PREFIX' is optional
    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted
    # Either the 'source' or 'remote' *must* be the "local" deployment
    endpoint: "http[s]://HOSTNAME:PORT" 
    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
    credentials:
      accessKey: ACCESS-KEY # Required
      secretKey: SECRET-KEY # Required
    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used
    snowball: # automatically activated if the source is local
      disable: false # optionally turn-off snowball archive transfer
      batch: 100 # upto this many objects per archive
      inmemory: true # indicates if the archive must be staged locally or in-memory
      compress: false # S2/Snappy compressed archive
      smallerThan: 5MiB # create archive for all objects smaller than 5MiB
      skipErrs: false # skips any source side read() errors

  # target where the objects must be replicated
  target:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX # 'PREFIX' is optional
    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted

    # Either the 'source' or 'remote' *must* be the "local" deployment
    endpoint: "http[s]://HOSTNAME:PORT"
    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
    credentials:
      accessKey: ACCESS-KEY
      secretKey: SECRET-KEY
    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used

  # NOTE: All flags are optional
  # - filtering criteria only applies for all source objects match the criteria
  # - configurable notification endpoints
  # - configurable retries for the job (each retry skips successfully previously replaced objects)
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "datetime" # match objects created after this date and time in RFC3339 format
      createdBefore: "datetime" # match objects created before this date and time in RFC3339 format

      ## NOTE: tags are not supported when "source" is remote.
      # tags:
      #   - key: "name"
      #     value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

      # metadata:
      #   - key: "content-type"
      #     value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

    notify:
      endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
      token: "Bearer xxxxx" # optional authentication token for the notification endpoint

    retry:
      attempts: 10 # number of retries for the job before giving up
      delay: "500ms" # least amount of delay between each retry

2 - Managing Your Deployment

You can use the MinIO Console to perform many of the deployment monitoring and management functions available in MinIO, such as:

• Monitor the deployment activity and health by viewing a dashboard of metrics, server or audit logs, trace history, S3 events, or drive health.
• Configure alerts by adding or managing a notification target.
• Set up site replication to synchronize datacenters for timely access across geographically dispersed workforces or for disaster preparedness.
• Configure deployment settings.

Monitoring

The Monitoring section provides an interface for monitoring the MinIO deployment.

The section contains the following subsections: Some subsections may not be visible if the authenticated user does not have the required administrative permissions.

Metrics

The Console Dashboard section displays metrics for the MinIO deployment. The default view provides a high-level overview of the deployment status, including the uptime and availability of individual servers and drives.

The Console also supports displaying time-series and historical data by querying a Prometheus service configured to scrape data from the MinIO deployment. Specifically, the MinIO Console uses the Prometheus query API to retrieve stored metrics data and display historical metrics See Monitoring and Alerting using Prometheus for more information on scraping MinIO metrics into Prometheus.

Logs

The Console Logs section displays server logs generated by the MinIO Deployment.

• Use the Nodes dropdown to filter logs to a subset of server nodes in the MinIO deployment.
• Use the Log Types dropdown to filter logs to a subset of log types.
• Use the Filter to apply text filters to the log results

Select the Start Logs button to begin collecting logs using the selected filters and settings.

Audit

The Audit Log section provides an interface for viewing audit logs collected by a configured PostgreSQL service.

Trace

The Trace section provides HTTP trace functionality for a bucket or buckets on the deployment. This section provides similar functionality to mc admin trace.

You can modify the trace to show only specific trace calls. The default is to show only S3 related HTTP traces.

Select Filters to open additional filters to apply to trace output, such as restricting the Path on which the trace applies to a specific bucket or bucket prefix.

Watch

The Watch section displays S3 events as they occur on the selected bucket. This section provides similar functionality to mc watch.

Encryption

The Encryption section allows you to view the status and metrics for configured Key Encryption Service providers.

Events

The Events section provides an interface to view, add, or remove Event Notification targets.

You can use this screen configure MinIO to push notification events to the one or more target destinations, including Redis, MySQL, Kafka, PostgreSQL, AMQP, MQTT, Elastic Search, NATS, NSQ, or a Webhook.

Select the Add Event Destination + button to add a new event target to the deployment.

You can select an existing notification target from the list to view its details or delete the target.

Site Replication

The Site Replication section provides an interface for adding and managing the site replication configuration for the deployment.

Configuring site replication requires that only a single site have existing buckets or objects (if any).

Encryption

The Encryption setting provides an interface for listing, creating, and deleting keys for use with MinIO Server-Side Encryption.

You can use keys created or listed in this view for object encryption operations, including setting a bucket-level default key.

Configuration

The Settings section provides an interface for viewing and retrieving configuration settings for all MinIO Servers in the deployment. Use the buttons to Export and Import the settings between deployments.

This section contains the following subsections.

• Region
• Compression
• API
• Heal
• Scanner
• Etcd
• Logger Webhook
• Audit Webhook
• Audit Kafka

Some subsections may not be visible if the authenticated user does not have the required administrative permissions.

The interface functionality mimics that of using mc admin config get or mc admin config set. Refer to those commands for details on how to define the many options.

Some configuration settings may require restarting the MinIO deployment to apply changes.

3 - Batch Key Rotation

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.

The keyrotate batch job type cycles the sse-s3 or sse-kms keys for encrypted objects on a MinIO deployment.

The YAML configuration supports filters to restrict key rotation to a specific set of objects by creation date, tags, metadata, or kms key. You can also define retry attempts or set a notification endpoint and token.

Key Rotate Batch Job Reference

Use the keyrotate job type to create a batch job that cycles the sse-s3 or sse-kms keys for encrypted objects.

Required Fields

type:	Either sse-s3 or sse-kms.
key:	Only for use with the sse-kms type. The key to use to unseal the key vault.

Optional Fields

For flag based filters

For notifications

For retry attempts

If something interrupts the job, you can define a maximum number of retry attempts. For each retry, you can also define how long to wait between attempts.

Sample YAML Description File for a keyrotate Job Type

Use mc batch generate to create a basic keyrotate batch job for further customization:

keyrotate:
  apiVersion: v1
  bucket: BUCKET
  prefix: PREFIX
  encryption:
    type: sse-s3 # valid values are sse-s3 and sse-kms
    key: <new-kms-key> # valid only for sse-kms
    context: <new-kms-key-context> # valid only for sse-kms

  # optional flags based filtering criteria
  # for all objects
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "date" # match objects created after this date and time in RFC3339 format
      createdBefore: "date" # match objects created before this date and time in RFC3339 format
      tags:
        - key: "name"
          value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
      metadata:
        - key: "content-type"
          value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
      kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
    notify:
      endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
      token: "Bearer xxxxx" # optional authentication token for the notification endpoint
    retry:
      attempts: 10 # number of retries for the job before giving up
      delay: "500ms" # least amount of delay between each retry

4 - Managing Objects

You can use the MinIO Console to perform several of the bucket and object management and interaction functions available in MinIO. Depending on the permissions and IAM policies for the authenticated user, you can:

• Browse, upload, revert, manage, and interact with objects.
• Browse, create, and manage buckets.
• Create or monitor remote tiers for object transition rules.

Object Browser

The Object Browser lists the buckets and objects the authenticated user has access to on the deployment.

After logging in or navigating to the tab, the object browser displays a list of the user’s buckets, which the user can filter. Select a bucket to show a list of objects in the bucket.

Select a specific object to display summary information about the object such as name, size, tags, holds, and retention policies that apply. The console also shows the object’s metadata.

The user can perform actions on the bucket’s objects, depending on the policies and permissions that apply. Example actions the user may be able to perform include:

• Rewind to a previous version
• Create prefixes
• View deleted objects
• Upload objects
• Download objects
• Share
• Preview
• Manage legal holds
• Manage retention
• Manage tags
• Inspect
• Display versions
• Delete

Buckets

The Console’s Bucket section displays all buckets to which the authenticated user has access. Use this section to create or manage these buckets, depending on your user’s access.

Creating Buckets

Select Create Bucket to create a new bucket on the deployment. MinIO validates bucket names. To see the rules for bucket names, select View Bucket Naming Rules.

MinIO does not limit the total number of buckets allowed on a deployment. However, MinIO recommends no more than 500,000 buckets per deployment as a general guideline.

While creating a bucket, you can enable versioning, object locking, bucket size (quota) limits, and retention rules (which require versioning).

You must configure replication, locking, and versioning options at the time of bucket creation. You cannot change these settings for the bucket later.

Managing Buckets

Use the Search bar to filter for specific buckets. Select the row for the bucket to display summary information about the bucket.

Form the summary screen, select any of the available tabs to further manage the bucket.

When managing a bucket, your access settings may allow you to view or change any of the following:

• The Summary section displays a summary of the bucket’s configuration.

  Use this section to view and modify the bucket’s access policy, encryption, quota, and tags.

• Configure alerts in the Events section to trigger notification events when a user uploads, accesses, or deletes matching objects.

• Copy objects to remote locations in the Replication section with Server Side Bucket Replication Rules.

• Expire or transition objects in the bucket from the Lifecycle section by setting up Object Lifecycle Management Rules.

• Review security in the Access section by listing the policies and users with access to that bucket.

• Properly secure unauthenticated access with the Anonymous section by managing rules for prefixes that unauthenticated users can use to read or write objects.

Tiers

The Tiering section provides an interface for adding and managing remote tiers to support lifecycle management transition rules. MinIO tiering supports moving objects from the deployment to the remote storage, but does not support automatically restoring them to the deployment.

The tiering tab allows users with the appropriate permissions to:

• Review the status and summary information for all configured remote tiers.
• Create a tier for a new remote target to storage on another MinIO deployment, Google Cloud Storage, Amazon’s AWS S3, or Azure.
• Cycle the access credentials for any of the configured tiers with the tier’s icon.

5 - Batch Expiration

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.

The expire batch job applies Automatic Object Expiration behavior to a single bucket. The job determines expiration eligibility based on the provided configuration, independent of any configured expiration rules.

Behavior

Immediate Expiration of Objects

Batch expiration occurs immediately as part of the batch job, as compared to the passive scanner-based application of expiration rules. Specifically, batch expiration does not yield to application I/O and may impact performance of regular read/write operations on the deployment.

Expiration Eligibility Determined at Batch-Run

The batch expiration works per-bucket and runs once to completion. The job determines expiration eligibility at the time the job runs, and does not rescan or recheck for new objects periodically.

To capture any new objects eligible for expiration, re-run the batch job.

Expiry Rules Check Latest Object Only

The batch expiration job only checks the latest or “current” version of each object against each batch expiration rule.

Expire Batch Job Reference

tags:
  - key: archive
    value: True

metadata:
  - key: content-type
    value: image/*

Sample YAML Description for an expire Job Type

Use mc batch generate to create a basic expire batch job for further customization.

expire:
  apiVersion: v1
  bucket: mybucket # Bucket where this job will expire matching objects from
  prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
  rules:
    - type: object  # objects with zero ore more older versions
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 70h # match objects older than this value
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before this date and time in RFC3339 format
      tags:
        - key: name
          value: pick* # match objects with tag 'name', all values starting with 'pick'
      metadata:
        - key: content-type
          value: image/* # match objects with 'content-type', all values starting with 'image/'
      size:
        lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
        greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object.

    - type: deleted # objects with delete marker as their latest version
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before this date and time in RFC3339 format
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object including delete markers.

  notify:
    endpoint: https://notify.endpoint # notification endpoint to receive job completion status
    token: Bearer xxxxx # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: 500ms # least amount of delay between each retry

6 - Security and Access

You can use the MinIO Console to perform several of the identity and access management functions available in MinIO, such as:

• Create child access keys that inherit the parent’s permissions.
• View, manage, and create access policies.
• Create and manage user credentials or groups with the built-in MinIO IDP, connect to one or more OIDC provider, or add an AD/LDAP provider for SSO.

Access Keys

The Access Keys or Service Accounts section displays all Access Keys associated to the authenticated user. The summary list of access keys that already exist for a particular user includes the access key, expiration, status, name, and description.

Access Keys support providing applications authentication credentials which inherit permissions from the “parent” user.

For deployments using an external identity manager such as Active Directory or an OIDC-compatible provider, access keys provide a way for users to create long-lived credentials.

• You can select the access key row to view its custom policy, if one exists.

  You can create or modify the policy from this screen. Access key policies cannot exceed the permissions granted to the parent user.

• You can create a new access key by selecting Create access key.

  The Console auto-generates an access key and password. You can select the eye icon on the password field to reveal the value. You can override these values as needed.

  You can set a custom policy for the access key that further restricts the permissions granted to users authenticating with that key. Select Restrict beyond user policy to open the policy editor and modify as necessary.

  Ensure you have saved the access key password to a secure location before selecting Create to create the access key. You cannot retrieve or reset the password value after creating the access key.

  To rotate credentials for an application, create a new access key and delete the old one once the application updates to using the new credentials.

Policies

The Policies section displays all policies on the MinIO deployment. The Policies section allows you to create, modify, or delete policies.

Policies define the authorized actions and resources to which an authenticated user has access. Each policy describes one or more actions a user, group of users, or access key can perform or conditions they must meet.

The policies are JSON formatted text files compatible with Amazon AWS Identity and Access Management policy syntax, structure, and behavior. Refer to Policy Based Action Control for details on managing access in MinIO with policies.

This section or its contents may not be visible if the authenticated user does not have the required administrative permissions.

• Select + Create Policy to create a new MinIO Policy.

• Select the policy row to manage the policy details.

  The Summary view displays a summary of the policy.

  The Users view displays all users assigned to the policy.

  The Groups view displays all groups assigned to the policy.

  The Raw Policy view displays the raw JSON policy.

Use the Users and Groups views to assign a created policy to users and groups, respectively.

Identity

The Identity section provides a management interface for MinIO-Managed users.

The section contains the following subsections. Some subsections may not be visible if the authenticated user does not have the required administrative permissions.

Users

The Users section displays all MinIO-managed users on the deployment.

This section is not visible for deployments using an external identity manager such as Active Directory or an OIDC-compatible provider.

• Select Create User to create a new MinIO-managed user.

  You can assign groups and policies to the user during creation.

• Select a user’s row to view details for that user.

  You can view and modify the user’s assigned groups and policies.

  You can also view and manage any Access Keys associated to the user.

Groups

The Groups section displays all groups on the MinIO deployment.

This section is not visible for deployments using an external identity manager such as Active Directory or an OIDC-compatible provider.

• Select Create Group to create a new MinIO Group.

  You can assign new users to the group during creation.

  You can assign policies to the group after creation.

• Select the group row to open the details for that group.

  You can modify the group membership from the Members view.

  You can modify the group’s assigned policies from the Policies view.

  Changing a user’s group membership modifies the policies that user inherits. See Access Management for more information.

OpenID

MinIO supports using an OpenID Connect (OIDC) compatible IDentity Provider (IDP) for external management of user identities.

Examples of OpenID providers include:

• Okta
• KeyCloak
• Dex
• Google
• Facebook

Configuring an external IDP enables Single-Sign On workflows, where applications authenticate against the external IDP before accessing MinIO.

Use the screens in this section to view, add, or edit OIDC configurations for the deployment. MinIO supports any number of active OIDC configurations.

LDAP

MinIO supports using an Active Directory or LDAP (AD/LDAP) service for external management of user identities. Configuring an external IDentity Provider (IDP) enables Single-Sign On (SSO) workflows, where applications authenticate against the external IDP before accessing MinIO.

Use the screens in this section to view, add, or edit an LDAP configuration for the deployment. MinIO only supports one active LDAP configuration.

MinIO queries the Active Directory / LDAP server to verify the client-specified credentials. MinIO also performs a group lookup on the AD/LDAP server if configured to do so.

7 - SUBNET

You can use the MinIO Console to perform several of the license and subscription related functions available in MinIO, such as:

• View the license you are currently using for your MinIO deployment.
• Subscribe to a commercial license, which includes access to MinIO SUBNET.
• Manage the deployment’s Enterprise license.
• Access Support tools for sharing with MinIO Engineering.
• Review differences between license options.

License

MinIO offers three licensing options:

1. Open source with the GNU AGPLv3 license
2. Enterprise Lite, a commercial license with included support direct from MinIO Engineers
3. Enterprise Plus, a commercial license with included support direct from MinIO Engineers, longer release cycle, shorter SLA, and other benefits

The License page shows the current license status of the deployment. You can also begin the registration process to sign up for a paid subscription or add the deployment to an existing subscription.

Deployments licensed under AGPLv3 must comply to the terms of the license. MinIO cannot make the determination as to whether your application’s usage of MinIO is in compliance with the AGPLv3 license requirements. You should instead rely on your own legal counsel or licensing specialists to audit and ensure your application is in compliance with the licenses of MinIO and all other open-source projects with which your application integrates or interacts.

MinIO Commercial Licensing is the best option for applications which trigger AGPLv3 obligations (for example, open sourcing your application). Applications using MinIO—or any other OSS-licensed code—without validating their usage do so at their own risk.

Health

The Health section provides an interface for running a health diagnostic for the MinIO Deployment. For clusters connected to the Internet, the report uploads automatically to SUBNET.

The resulting health report is intended for use by MinIO Engineering via MinIO SUBNET and may contain internal or private data points such as hostnames. Exercise caution before sending a health report to a third party or posting the health report in a public forum.

If desired, you can download the latest report from the page.

Performance

The Performance section provides an interface for running a performance test of the deployment. The resulting test can provide a general guideline of deployment performance under S3 GET and PUT requests.

For more complete performance testing, consider using a combination of load-testing using your staging application environments and the MinIO WARP tool.

Profile

The Profile section provides an interface for running system profiling of the deployment. The results can provide insight into the MinIO server process running on a given node.

The resulting report is intended for use by MinIO Engineering via MinIO SUBNET. Independent or third-party use of these profiles for diagnostics and remediation is done at your own risk.

Inspect

The Inspect section provides an interface for capturing the erasure-coded metadata associated to an object or objects. MinIO Engineering may request this output as part of diagnostics in MinIO SUBNET.

The resulting object may be read using MinIO’s debugging tool. Independent or third-party use of the output for diagnostics or remediation is done at your own risk. You can optionally encrypt the object such that it can only be read if the generated encryption key is included as part of the debugging toolchain.

Call Home

Call Home is an optional feature where a deployment registered for MinIO SUBNET can automatically send daily health diagnostic reports or real-time error logs to SUBNET. Having these reports equips engineering support with a record of diagnostics, logs, or both when responding to support requests.

MinIO installs with Call Home options disabled by default.

Use the Call Home section to enable or disable uploading either once-per-day health diagnostic reports or real-time error logs to SUBNET. The health reports and real-time logs are separate functions you can enable or disable separately. You can enable both diagnostics and logs at the same time, if desired.

8 - Silo Console

The MinIO Console is a rich graphical user interface that provides similar functionality to the mc command line tool.

This page provides an overview of the MinIO Console and describes configuration options and instructions for logging in.

Overview

You can use the MinIO Console for administration tasks like Identity and Access Management, Metrics and Log Monitoring, or Server Configuration.

SILO embeds the maintained Silo Console in the server. The Silo Console repository documents the downstream source, releases, and compatibility changes; standalone deployment is an advanced integration path and must use a Console version compatible with the target server release.

Supported Browsers

MinIO Console runs on a variety of current, stable release browsers.

For the best experience in the MinIO Console, use the latest stable release of your preferred browser. Some browsers that are supported include:

• Chrome
• Edge
• Safari
• Firefox
• Opera

This list is not exhaustive and is subject to change.

For a full list of browsers and versions for running MinIO Console, see the Browserslist website.

Configuration

The MinIO Console inherits the majority of its configuration settings from the MinIO Server. The following environment variables enable specific behavior in the MinIO Console:

Static vs Dynamic Port Assignment

MinIO by default selects a random port for the MinIO Console on each server startup. Browser clients accessing the MinIO Server are automatically redirected to the MinIO Console on its dynamically selected port. This behavior emulates the legacy web browser behavior while reducing the risk of a port collision on systems which were running MinIO before the embedded Console update.

You can select an explicit static port by passing the minio server --console-address commandline option when starting each MinIO Server in the deployment.

For example, the following command starts a distributed MinIO deployment using a static port assignment of 9001 for the MinIO Console. This deployment would respond to S3 API operations on the default MinIO server port :9000 and browser access on the MinIO Console port :9001.

minio server https://minio-{1...4}.example.net/mnt/drive-{1...4} \
      --console-address ":9001"

Deployments behind network routing components which require static ports for routing rules may require setting a static MinIO Console port. For example, load balancers, reverse proxies, or Kubernetes ingress may by default block or exhibit unexpected behavior with the dynamic redirection behavior.

You must also ensure that the host system firewall grants access to the configured Console port.

Logging In

The MinIO Console displays a login screen for unauthenticated users. The Console defaults to providing a username and password prompt for a MinIO-managed user.

For deployments configured with multiple identity managers, select the Other Authentication Methods dropdown to select one of the other configured identity providers. You can also log in using credentials generated using a Security Token Service (STS) API.

Documentation

The Documentation tab opens this documentation site in a separate browser window or tab.

Available Tasks

Once logged in to the MinIO Console, users can perform many kinds of tasks.

• Manage objects by browsing or uploading objects, managing bucket settings, or creating tiers.
• Review or modify identity and security with access keys, policies, and Identity Provider settings.
• Monitor the health and activities with metrics, notifications, or site replication
• Manage your deployment’s license

9 - Object Management

• Versioning overview
• Object locking and retention overview
• MinIO Object Lifecycle Management Part I
• MinIO Object Lifecycle Management Part II

An object is binary data, such as images, audio files, spreadsheets, or even binary executable code. The term “Binary Large Object” or “blob” is sometimes associated to object storage, although blobs can be anywhere from a few bytes to several terabytes in size. Object Storage platforms like MinIO provide dedicated tools and capabilities for storing, listing, and retrieving objects using a standard S3-compatible API.

MinIO Object Storage uses buckets to organize objects. A bucket is similar to a top-level drive, folder, or directory in a filesystem (/mnt/data or C:\), where each bucket can hold an arbitrary number of objects.

The structure of objects on the MinIO server might look similar to the following:

/ #root
/images/
   2020-01-02-MinIO-Diagram.png
   2020-01-03-MinIO-Advanced-Deployment.png
   MinIO-Logo.png
/videos/
   2020-01-04-MinIO-Interview.mp4
/articles/
   /john.doe/
      2020-01-02-MinIO-Object-Storage.md
      2020-01-02-MinIO-Object-Storage-comments.json
   /jane.doe/
      2020-01-03-MinIO-Advanced-Deployment.png
      2020-01-02-MinIO-Advanced-Deployment-comments.json
      2020-01-04-MinIO-Interview.md

With the example structure, an administrator would create the /images, /videos and /articles buckets. Client applications write objects to those buckets using the full “path” to that object, including all intermediate prefixes.

MinIO supports multiple levels of nested directories and objects using prefixes to support even the most dynamic object storage workloads. MinIO automatically infers the intermediate prefixes, such as /articles/john.doe from the full object path using / as a delimiter. Clients and administrators should not create these prefixes manually.

Neither clients nor administrators would manually create the intermediate prefixes, as MinIO automatically infers them from the object name.

Path vs Virtual Host Bucket Access

MinIO supports both path-style (default) or virtual-host bucket lookups.

For example, consider a MinIO deployment with an assigned Fully Qualified Domain Name (FQDN) of minio.example.net:

• With path-style lookups, applications specify the full path to a bucket, such as minio.example.net/mybucket.
• With virtual-host lookups, applications specify the bucket as a subdomain, such as mybucket.minio.example.net/.

Some applications may require or expect virtual-host lookup support when performing S3 operations against MinIO. To enable virtual-host bucket lookup, you must set the MINIO_DOMAIN environment variable to a FQDN that resolves to the MinIO Deployment.

If you configure MINIO_DOMAIN, you must consider all subdomains of the specified FQDN as exclusively assigned for use as bucket names. Any MinIO services which conflict with those domains, such as replication targets, may exhibit unexpected or undesired behavior as a result of the collision.

For example, if setting MINIO_DOMAIN=minio.example.net, you cannot assign any subdomains of minio.example.net (in the form of *.minio.example.net) to any MinIO service or target. This includes hostnames for use with bucket, batch, or site replication.

Object Organization and Planning

Administrators typically control the creation and configuration of buckets. Client applications can then use S3-compatible SDKs to create, list, retrieve, and delete objects on the MinIO deployment. Clients therefore drive the overall hierarchy of data within a given bucket or prefix, where Administrators can exercise control using policies to grant or deny access to an action or resource.

MinIO has no hard thresholds on the number of buckets, objects, or prefixes on a given deployment. The relative performance of the hardware and networking underlying the MinIO deployment may create a practical limit to the number of objects in a given prefix or bucket. Specifically, hardware using slower drives or network infrastructures tend to exhibit poor performance in buckets or prefixes with a flat hierarchy of objects. For other considerations, thresholds, or limitations to keep in mind, see Thresholds and Limits.

Consider the following points as general guidance for client applications workload patterns:

• Deployments with modest or budget-focused hardware should architect their workloads to target 10,000 objects per prefix as a baseline. Increase this target based on benchmarking and monitoring of real world workloads up to what the hardware can meaningfully handle.
• Deployments with high-performance or enterprise-grade hardware can typically handle prefixes with millions of objects or more.

MinIO SUBNET Enterprise accounts can utilize yearly architecture reviews as part of the deployment and maintenance strategy to ensure long-term performance and success of your MinIO-dependent projects.

For a deeper discussion on the benefits of limiting prefix contents, see the article on optimizing S3 performance.

Object Versioning

The specific client behavior on write, list, get, or delete operations on a bucket depends on the versioning state of that bucket:

See Bucket Versioning for more complete documentation.

Object Tagging

MinIO supports adding custom tags to an object. A tag is a key-value pair included in the metadata of an object. Tags can be used to control access with policies or locate an object with mc find --tags.

MinIO supports adding up to 10 custom tags to an object.

For more on setting tags, refer to mc tag set.

Object Retention

MinIO Object Locking (“Object Retention”) enforces Write-Once Read-Many (WORM) immutability to protect versioned objects from deletion. MinIO supports both duration based object retention and indefinite legal hold retention.

Delete operations against a WORM-locked object depend on the specific operation:

• Delete operations which do not specify a version ID result in the creation of a “Delete Marker”
• Delete operations which specify the version ID of a locked object result in a WORM locking error

You can only enable object locking when first creating a bucket. Enabling bucket locking also enables versioning.

MinIO Object Locking provides key data retention compliance and meets SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per Cohasset Associates.

See MinIO Object Locking and Object Deletion for more complete documentation.

Object Lifecycle Management

MinIO Object Lifecycle Management allows creating rules for time or date based automatic transition or expiry of objects. For object transition, MinIO automatically moves the object to a configured remote storage tier. For object expiry, MinIO automatically deletes the object.

MinIO applies lifecycle management rules on versioned and unversioned buckets using the same behavior as normal client operations. You can specify transition or lifecycle rules that handle the latest object versions, non-current object versions, or both.

MinIO lifecycle management is built for behavior and syntax compatibility with AWS S3 Lifecycle Management. MinIO uses JSON to describe lifecycle management rules. Conversion to or from XML may be required for importing rules created on S3 or similar compatible platforms.

See Object Lifecycle Management for more complete documentation.

Target Bucket Considerations

MinIO does not require that the target bucket match object management or versioning configurations with the source bucket. The target bucket may have its own set of object management rules, if defined with care.

Target buckets should not have their own rules for expiration or additional tiering. Expiration rules can result in removal of tiered data still in use by the source bucket. Tiering to an additional remote creates an additional network hop between the hot tier and it’s data while also increasing operational complexity.

You may configure object locking or versioning on the remote bucket.

Enabling versioning or object locking on the target bucket may have effects such as the following:

• Object locking set on the target bucket may prevent desired delete operations from the source bucket from completing.
• MinIO tiers objects with their own UUID, so versioning on the target bucket is redundant at best.
• Reduced storage efficiency on the target, as delete operations result in creation of a DeleteMarker rather than freeing space.
• Duplicate delete markers on source and target buckets.

Exclusive Access to Remote Data

MinIO must have exclusive access to the target bucket. No other user, process, application, or resource should have any access to or perform any actions against the target bucket.

All access to the transitioned objects must occur through MinIO via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.
• No lifecycle management rules (such as transition or expiration) on the remote storage bucket.

To facilitate this exclusive access, grant the lifecycle management user read, write, and delete access to the target bucket in its policy. All other policies should deny access to the target bucket.

Conflicting Objects

Applications must assign non-conflicting, unique keys for all objects. This includes avoiding creating objects where the name can collide with that of a parent or sibling object. MinIO returns an empty set for LIST operations at the location of the collision.

For example, the following operations create a namespace conflicts

PUT data/invoices/2024/january/vendors.csv
PUT data/invoices/2024/january <- collides with existing object prefix

PUT data/invoices/2024/january
PUT data/invoices/2024/january/vendors.csv <- collides with existing object

While you can perform GET or HEAD operations against these objects, the name collision causes LIST operations to return an empty result set at the /invoices/2024/january path.

9.1 - Bucket Versioning

• Versioning overview
• Versioning lab

Overview

MinIO supports keeping multiple “versions” of an object in a single bucket.

When enabled, versioning allows MinIO to keep multiple iterations of the same object. Write operations which would normally overwrite an existing object instead result in the creation of a new versioned object. MinIO versioning protects from unintended overwrites and deletions while providing support for “undoing” a write operation. Bucket versioning is a prerequisite for configuring object locking and retention rules.

For versioned buckets, a write operation results in a new version of that object with a unique version ID. MinIO marks the “latest” version of the object that clients retrieve by default. Clients can then explicitly choose to list, retrieve, or remove a specific object version.

Define object expiration rules to remove versions of objects no longer needed, such as by the number of versions or the date of versions.

Read Operations on Versioned Objects

Review each of the four images in this series to see how MinIO retrieves objects in a versioned bucket. Use the arrows on either side of the images to navigate from one to the next.

Object with Single Version

MinIO adds a unique version ID to each object as part of write operations.

Object with Multiple Versions

MinIO retains all versions of an object and marks the most recent version as the “latest”.

Retrieving the Latest Object Version

A read operation request without a version ID returns the latest version of the object.

Retrieving a Specific Object Version

Include the version ID to retrieve a specific version of an object during a read operation.

MinIO implicitly determines prefixes from object paths. Explicit prefix creation typically only occurs with Spark and similar workloads which apply legacy POSIX/HDFS directory creation behavior within the S3 context.

Versioning is Per-Namespace

MinIO uses the full namespace (the bucket and path to an object) for each object as part of determining object uniqueness. For example, all of the following namespaces are “unique” objects, where mutations of each object result in the creation of new object versions at that namespace:

databucket/object.blob
databucket/blobs/object.blob
blobbucket/object.blob
blobbucket/blobs/object.blob

While object.blob might be the same binary across all namespaces, MinIO only enforces versioning with a specific namespace and therefore considers each object.blob above as distinct and unique.

Versioning and Storage Capacity

MinIO does not perform incremental or differential-type versioning. For mutation-heavy workloads, this may result in substantial drive usage by older or aged object versions.

For example, consider a 1GB object containing log data. An application appends 100MB of data to the log and uploads to MinIO. MinIO would then contain both the 1GB and 1.1GB versions of the object. If the application repeated this process every day for 10 days, the bucket would eventually contain more than 14GB of data associated to a single object.

MinIO supports configuring configuring object lifecycle management rules to automatically expire or transition aged object versions and free up storage capacity. For example, you can configure a rule to automatically expire object versions 90 days after they become non-current (i.e. no longer the “latest” version of that object). See MinIO Object Expiration for more information.

You can alternatively perform manual removal of object versions using the following commands:

• mc rm --versions - Removes all versions of an object.

• mc rm --versions --older-than -

  Removes all versions of an object older than the specified calendar date.

Version ID Generation

MinIO generates a unique and immutable identifier for each versioned object as part of write operations. Each object version ID consists of a 128-bit fixed-size UUIDv4. UUID generation is sufficiently random to ensure high likelihood of uniqueness for any environment, are computationally difficult to guess, and do not require centralized registration process and authority to guarantee uniqueness.

MinIO does not support client-managed version ID allocation. All version ID generation is handled by the MinIO server process.

For objects created while versioning is disabled or suspended, MinIO uses a null version ID. You can access or remove these objects by specifying null as the version ID as part of S3 operations.

Versioned Delete Operations

Performing a DELETE operation on a versioned object creates a 0-byte DeleteMarker as the latest version of that object. For objects where the latest version is a DeleteMarker, clients must specify versioning flags or identifiers to perform GET/HEAD/LIST/DELETE operations on a prior version of that object. The default server behavior omits DeleteMarker objects from consideration for unversioned operations.

MinIO can utilize Lifecycle Management expiration rules to automatically remove versioned objects permanently. Otherwise, use manual DELETE operations to permanently remove non-current versioned objects or DeleteMarker objects.

To permanently delete an object version, perform the DELETE operation and specify the version ID of the object to delete. Versioned delete operations are irreversible.

Deleting an Object

Performing a DELETE operation on a versioned object produces a DeleteMarker for that object.

Reading a Deleted Object

Clients by default retrieve the “latest” object version. MinIO returns a 404-like response if the latest version is a DeleteMarker.

Retrieve Previous Version of Deleted Object

Clients can retrieve any previous version of the object by specifying the version ID, even if the “Latest” version is a DeleteMarker.

Delete a Specific Object Version

Clients can delete a specific object version by specifying the version ID as part of the DELETE operation. Deleting a specific version is permanent and does not result in the creation of a DeleteMarker.

The following mc commands operate on DeleteMarkers or versioned objects:

• Use mc ls --versions to view all versions of an object, including delete markers.
• Use mc cp --version-id=UUID ... to retrieve the version of the “deleted” object with matching UUID.
• Use mc rm --version-id=UUID ... to delete the version of the object with matching UUID.
• Use mc rm --versions to delete all versions of an object.

Tutorials

Enable Bucket Versioning

You can enable versioning using the MinIO Console, the MinIO mc CLI, or using an S3-compatible SDK.

Use the mc version enable command to enable versioning on an existing bucket:

mc version enable ALIAS/BUCKET

• Replace ALIAS with the alias of a configured MinIO deployment.
• Replace BUCKET with the target bucket on which to enable versioning.

Objects created prior to enabling versioning have a null version ID.

Exclude a Prefix From Versioning

You can exclude certain prefixes from versioning using the MinIO Client. This is useful for Spark/Hadoop workloads or others that initially create objects with temporary prefixes.

• Use mc version enable with the --excluded-prefixes option:

  mc version enable --excluded-prefixes "prefix1, prefix2" ALIAS/BUCKET

  • Replace ALIAS with the alias of a configured MinIO deployment.
  • Replace BUCKET with the name of the bucket you want to exclude prefixes for.

The list of --excluded-prefixes prefixes match all objects containing the specified strings in their prefix or name, similar to a regular expression of the form prefix*. To match objects by prefix only, use prefix/*.

For example, the following command excludes any objects containing _test or _temp in their prefix or name from versioning:

mc version enable --excluded-prefixes "_test, _temp" local/my-bucket

You can exclude up to 10 prefixes for each bucket. To add or remove prefixes, repeat the mc version enable command with an updated list. The new list of prefixes replaces the previous one.

To view the currently excluded prefixes, use mc version info with the --json option:

mc version info ALIAS/BUCKET --json

The command output resembles the following, with the list of excluded prefixes in the ExcludedPrefixes property:

$ mc version info local/my-bucket --json
{
 "Op": "info",
 "status": "success",
 "url": "local/my-bucket",
 "versioning": {
  "status": "Enabled",
  "MFADelete": "",
  "ExcludedPrefixes": [
   "prefix1, prefix2"
  ]
 }
}

To disable prefix exclusion and resume versioning all prefixes, repeat the mc version enable command without --excluded-prefixes:

mc version enable ALIAS/BUCKET

Exclude Folders from Versioning

You can exclude folders from versioning using the MinIO Client.

• Use mc version enable with the --exclude-folders option to exclude objects with names ending in / from versioning:

  mc version enable --exclude-folders ALIAS/BUCKET

  • Replace ALIAS with the alias of a configured MinIO deployment.
  • Replace BUCKET with the bucket you want to exclude folders for.

To check whether folders are versioned for a bucket, use the mc version enable command with the --json option. If the ExcludeFolders property is true, folders in that bucket are not versioned.

mc version enable --excluded-prefixes ALIAS/BUCKET --json

The command output resembles the following:

$ mc version info local/my-bucket --json
{
 "Op": "info",
 "status": "success",
 "url": "local/my-bucket",
 "versioning": {
  "status": "Enabled",
  "MFADelete": "",
  "ExcludeFolders": true
 }
}

To disable folder exclusion and resume versioning all folders, repeat the mc version enable command without --exclude-folders:

mc version enable ALIAS/BUCKET

Suspend Bucket Versioning

You can suspend bucket versioning at any time using he MinIO mc CLI or using an S3-compatible SDK.

Use the mc version suspend command to enable versioning on an existing bucket:

mc version suspend ALIAS/BUCKET

• Replace ALIAS with the alias of a configured MinIO deployment.
• Replace BUCKET with the target bucket on which to disable versioning.

Objects created while versioning is suspended are assigned a null version ID. Any mutations to an object while versioning is suspended result in overwriting that null versioned object. MinIO does not remove or otherwise alter existing versioned objects as part of suspending versioning. Clients can continue interacting with any existing object versions in the bucket.

9.2 - Transition Objects to Remote MinIO Deployment

The procedure on this page creates a new object lifecycle management rule that transitions objects from a bucket on a primary MinIO deployment to a bucket on a remote MinIO deployment. This procedure supports cost-management strategies such as tiering objects from a “hot” MinIO deployment using NVMe storage to a “warm” MinIO deployment using SSD.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias set command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required Source MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration
• s3:GetLifecycleConfiguration

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier
• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required Remote MinIO Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the remote tier credentials provide read, write, list, and delete permissions for the remote bucket.

For example, the following policy on the remote MinIO deployment provides the necessary permission for transitioning objects into and out of the remote tier:

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket"
            ],
            "Sid": ""
      },
      {
            "Action": [
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket/*"
            ],
            "Sid": ""
      }
   ]
}

Modify the Resource for the bucket into which MinIO tiers objects.

Refer to the Access Management documentation for more complete guidance on configuring the required permissions.

Remote Bucket Must Exist

Create the remote bucket prior to configuring lifecycle management tiers or rules using that bucket as the target.

If the remote bucket contains existing data, use the prefix feature to isolate transitioned objects from any other objects on that bucket.

Considerations

Lifecycle Management Object Scanner

MinIO uses a scanner process to check objects against all configured lifecycle management rules. Slow scanning due to high IO workloads or limited system resources may delay application of lifecycle management rules.

Exclusive Access to Remote Data

MinIO requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” MinIO source is strongly linked to the object data on the “warm/cold” remote tier. MinIO cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through MinIO via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.
• No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

MinIO stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. MinIO supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of Remote Data

MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. MinIO therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

MinIO creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the MinIO deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement MinIO Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the MinIO deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the MinIO deployment. Replace this value with the appropriate alias for the MinIO deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

wget -O - https://silo.pgsty.com/extra/examples/LifecycleManagementAdmin.json | \
mc admin policy create Alpha LifecycleAdminPolicy /dev/stdin
mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
mc admin policy attach Alpha LifecycleAdminPolicy --user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and MinIO Policy Based Access Control for more complete documentation on MinIO users and policies respectively.

2) Configure the Remote Storage Tier

Use the mc ilm tier add command to add the remote MinIO deployment as the new remote storage tier:

mc ilm tier add minio TARGET TIER_NAME  \
   --endpoint https://HOSTNAME       \
   --access-key ACCESS_KEY           \
   --secret-key SECRET_KEY           \
   --bucket BUCKET                   \
   --prefix PREFIX                   \
   --storage-class STORAGE_CLASS     \
   --region REGION

The example above uses the following arguments:

3) Create and Apply the Transition Rule

Use the mc ilm rule add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

mc ilm rule add ALIAS/BUCKET \
--transition-tier TIERNAME \
--transition-days DAYS \
--noncurrent-transition-days NONCURRENT_DAYS
--noncurrent-transition-tier TIERNAME

The example above specifies the following arguments:

4) Verify the Transition Rule

Use the mc ilm rule ls command to review the configured transition rules:

mc ilm rule ls ALIAS/PATH --transition

• Replace ALIAS with the alias of the MinIO deployment.
• Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.

9.3 - Silo Object Locking

• Object locking and retention overview
• Object locking and retention lab

Overview

MinIO Object Locking (“Object Retention”) enforces Write-Once Read-Many (WORM) immutability to protect versioned objects from deletion. MinIO supports both duration based object retention and indefinite legal hold retention.

MinIO Object Locking provides key data retention compliance and meets SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per Cohasset Associates.

Bucket Without Locking

MinIO versioning preserves the full history of object mutations. However, applications can explicitly delete specific object versions.

Bucket With Locking

Applying a default 30 Day WORM lock to objects in the bucket ensures a minimum period of retention and protection for all object versions.

Delete Operations in Locked Bucket

Delete operations follow normal behavior in versioned buckets, where MinIO creates a DeleteMarker for the object. However, non-Delete Marker versions of the object remain under the retention rules and are protected from any specific deletion or overwrite attempts.

Versioned Delete Operations in Locked Bucket

MinIO blocks any attempt to delete a specific object version held under WORM lock. The earliest possible time after which a client may delete the version is when the lock expires.

MinIO object locking is feature and API compatible with AWS S3. This page summarizes Object Locking / Retention concepts as implemented by MinIO. See the AWS S3 documentation on How S3 Object Lock works for additional resources.

You can only enable object locking during bucket creation as per S3 behavior. You cannot enable object locking on a bucket created without locking enabled. You can then configure object retention rules at any time. Object locking requires versioning and enables the feature implicitly.

Interaction with Versioning

Objects held under WORM locked are immutable until the lock expires or is explicitly lifted. Locking is per-object version, where each version is independently immutable.

If an application performs an unversioned delete operation on a locked object, the operation produces a delete marker. Attempts to explicitly delete any WORM-locked object fail with an error. Delete Markers are not eligible for protection under WORM locking. See the S3 documentation on Managing delete markers and object lifecycles for more information.

For example, consider the following bucket with GOVERNANCE Mode locking enabled by default:

$ mc ls --versions play/locking-guide

  [DATETIME]    29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
  [DATETIME]    32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
  [DATETIME]    23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv

Attempting to perform a delete on a specific version of data.csv fails due to the object locking settings:

$ mc rm --version-id 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 play/data.csv

  Removing `play/locking-guide/data.csv` (versionId=62429eb1-9cb7-4dc5-b507-9cc23d0cc691).
  mc: <ERROR> Failed to remove `play/locking-guide/data.csv`.
      Object, 'data.csv (Version ID=62429eb1-9cb7-4dc5-b507-9cc23d0cc691)' is
      WORM protected and cannot be overwritten

Attempting to perform an unversioned delete on data.csv succeeds and creates a new DeleteMarker for the object:

$ mc rm play/locking-guide/data.csv

  [DATETIME]     0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv
  [DATETIME]    29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
  [DATETIME]    32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
  [DATETIME]    23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv

Interaction with Lifecycle Management

MinIO object expiration respects any active object lock and retention settings for objects covered by the expiration rule.

• For expiration rules operating on only the current object version, MinIO creates a Delete Marker for the locked object.
• For expiration rules operating on non-current object versions, MinIO can only expire the non-current versions after the retention period has passed or has been explicitly lifted (e.g. legal holds).

For example, consider the following bucket with GOVERNANCE Mode locking enabled by default for 45 days:

$ mc ls --versions play/locking-guide

  [7D]    29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
  [30D]    32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
  [60D]    23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv

Creating an expiration rule for current objects older than 7 days results in a Delete Marker for the object:

$ mc ls --versions play/locking-guide

  [0D]     0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv
  [7D]    29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv
  [30D]    32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv
  [60D]    23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv

However, an expiration rule for non-current objects older than 7 days would only take effect after the configured WORM lock expires. Since the bucket has a 45 day GOVERNANCE retention set, only the v1 version of data.csv is unlocked and therefore eligible for deletion.

Tutorials

Create Bucket with Object Locking Enabled

You must enable object locking during bucket creation as per S3 behavior. You can create a bucket with object locking enabled using the MinIO mc CLI or using an S3-compatible SDK.

Use the mc mb command with the --with-lock option to create a bucket with object locking enabled:

mc mb --with-lock ALIAS/BUCKET

• Replace ALIAS with the alias of a configured MinIO deployment.
• Replace BUCKET with the name of the bucket to create.

Configure Bucket-Default Object Retention

You can configure object locking rules (“object retention”) using the MinIO mc CLI, or using an S3-compatible SDK.

MinIO supports setting both bucket-default and per-object retention rules. The following examples set bucket-default retention. For per-object retention settings, defer to the documentation for the PUT operation used by your preferred SDK.

Use the mc retention set command with the --recursive and --default options to set the default retention mode for a bucket:

mc retention set --recursive --default MODE DURATION ALIAS/BUCKET

• Replace MODE with either either COMPLIANCE or GOVERNANCE.
• Replace DURATION with the duration for which the object lock remains in effect.
• Replace ALIAS with the alias of a configured MinIO deployment.
• Replace BUCKET with the name of the bucket on which to set the default retention rule.

Enable Legal Hold Retention

You can enable or disable indefinite legal hold retention for an object using the MinIO mc CLI or using an S3-compatible SDK.

You can place a legal hold on an object already held under a COMPLIANCE or GOVERNANCE lock. The object remains WORM locked under the legal hold even when the retention lock expires. You or another user with the necessary permissions must explicitly lift the legal hold to remove the WORM lock.

Use the mc legalhold set command to toggle the legal hold status on an object.

mc legalhold set ALIAS/PATH

• Replace ALIAS with the alias of a configured MinIO deployment.
• Replace PATH with the path to the object for which to enable the legal hold.

Object Retention Modes

MinIO implements the following S3 Object Locking Modes:

GOVERNANCE Mode

An object under GOVERNANCE lock is protected from write operations by non-privileged users.

GOVERNANCE locked objects enforce managed-immutability for locked objects, where users with the s3:BypassGovernanceRetention action can modify the locked object, change the retention duration, or lift the lock entirely. Bypassing GOVERNANCE retention also requires setting the x-amz-bypass-governance-retention:true header as part of the request.

The MinIO GOVERNANCE lock is functionally identical to the S3 GOVERNANCE mode.

COMPLIANCE Mode

An object under COMPLIANCE lock is protected from write operations by all users, including the MinIO root user.

COMPLIANCE locked objects enforce complete immutability for locked objects. You cannot change or remove the lock before the configured retention duration has passed.

The MinIO COMPLIANCE lock is functionally identical to the S3 COMPLIANCE mode.

Legal Hold

An object under legal hold is protected from write operations by all users, including the MinIO root user.

Legal holds are indefinite and enforce complete immutability for locked objects. Only privileged users with the s3:PutObjectLegalHold permission can set or lift the legal hold.

Legal holds apply at the object level. If you enable legal hold for a group of objects, such as the contents of a bucket, subsequently created objects in that bucket are not affected.

Legal holds are complementary to both GOVERNANCE Mode and COMPLIANCE Mode retention settings. An object held under both legal hold and a GOVERNANCE/COMPLIANCE retention rule remains WORM locked until the legal hold is lifted and the rule expires.

For GOVERNANCE locked objects, the legal hold prevents mutating the object even if the user has the necessary privileges to bypass retention.

9.4 - Transition Objects from MinIO to S3

The procedure on this page creates a new object lifecycle management rule that transition objects from a MinIO bucket to a remote storage tier on the Amazon Web Services S3 storage backend or an S3-compatible service. This procedure supports use cases such as tiering objects to low-cost or archival storage after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias set command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration
• s3:GetLifecycleConfiguration

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier
• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required S3 Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the remote tier credentials provide read, write, list, and delete permissions for the remote bucket.

For example, the following policy provides the necessary permission for transitioning objects into and out of the remote tier:

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket"
            ],
            "Sid": ""
      },
      {
            "Action": [
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket/*"
            ],
            "Sid": ""
      }
   ]
}

Modify the Resource for the bucket into which MinIO tiers objects.

Refer to the Amazon S3 Permissions documentation for more complete guidance on configuring the required permissions.

Remote Bucket Must Exist

Create the remote S3 bucket prior to configuring lifecycle management tiers or rules using that bucket as the target.

Considerations

Lifecycle Management Object Scanner

MinIO uses a scanner process to check objects against all configured lifecycle management rules. Slow scanning due to high IO workloads or limited system resources may delay application of lifecycle management rules. See Lifecycle Management Object Scanner for more information.

Exclusive Access to Remote Data

MinIO requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” MinIO source is strongly linked to the object data on the “warm/cold” remote tier. MinIO cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through MinIO via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.
• No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

MinIO stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. MinIO supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of Remote Data

MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. MinIO therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

MinIO creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the MinIO deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement MinIO Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the MinIO deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the MinIO deployment. Replace this value with the appropriate alias for the MinIO deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

wget -O - https://silo.pgsty.com/extra/examples/LifecycleManagementAdmin.json | \
mc admin policy create Alpha LifecycleAdminPolicy /dev/stdin
mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
mc admin policy attach Alpha LifecycleAdminPolicy --user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and MinIO Policy Based Access Control for more complete documentation on MinIO users and policies respectively.

2) Configure the Remote Storage Tier

Use the mc ilm tier add command to add an Amazon S3 service as the new remote storage tier:

mc ilm tier add s3 TARGET TIER_NAME  \
   --endpoint https://HOSTNAME       \
   --access-key ACCESS_KEY           \
   --secret-key SECRET_KEY           \
   --bucket BUCKET                   \
   --prefix PREFIX                   \
   --storage-class STORAGE_CLASS     \
   --region REGION

The example above uses the following arguments:

3) Create and Apply the Transition Rule

Use the mc ilm rule add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

mc ilm rule add ALIAS/BUCKET \
--transition-tier TIERNAME \
--transition-days DAYS \
--noncurrent-transition-days NONCURRENT_DAYS
--noncurrent-transition-tier TIERNAME

The example above specifies the following arguments:

4) Verify the Transition Rule

Use the mc ilm rule ls command to review the configured transition rules:

mc ilm rule ls ALIAS/PATH --transition

• Replace ALIAS with the alias of the MinIO deployment.
• Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.

9.5 - Object Deletion

Overview

This page summarizes how a DELETE operation affects objects depending on the configuration of the bucket that contains the object.

Any combination of the following factors may impact how DELETE operations function:

• Bucket versioning
• Object locking rules
• Object Lifecycle Management rules
• Object tiering
• Site or bucket replication
• Scanner

Permissions

MinIO uses a policy based access control system for access management. The user or service account must provide the correct policy action and conditions to allow a DELETE for the bucket and object.

Unversioned Objects

When performing a DELETE operation on an object in a bucket that does not have versioning enabled, the operation is straightforward. After verifying the user or service account has permission to perform the DELETE operation, MinIO permanently removes the object.

The user or service account requesting the delete action the action must have the s3:DeleteObject action permission for the bucket and object.

Versioned Objects

DELETE operations work differently when an object is versioned.

The user or service account must have the s3:DeleteObjectVersion action permission for the bucket and object.

Delete operations on the current version

A DELETE operation on a versioned object that does not specify a version UUID results in the creation of a DeleteMarker placed as the head of the object.

In this scenario, MinIO does not actually remove the object or any of its versions from the disk. All existing versions of the object remain available to access by specifying the version’s UUID. When a DeleteMarker is the head for the object, MinIO does not serve the object for GET requests that do not specify a version ID. Instead, MinIO returns a 404-like response.

You can find the UUID of object versions with mc ls --versions.

To remove the current version of the object from the drive, find the UUID of the version, and then use mc rm --version-id=UUID ... to delete the current version. In this scenario, the immediately preceding version of the object then becomes the current version of the object served for GET requests of the object with no UUID specified.

Delete operations on a prior version

To delete prior versions of an object, specify the version’s UUID. You can retrieve the version UUID with mc ls --versions. When the DELETE request specifies a version-id and the user has the correct permissions to delete the object version, MinIO permanently removes the specified version from the drive.

Delete all versions

Use mc rm --versions to delete all versions of an object. This is irreversible.

Lifecycle Management Expiration

You can define one or more lifecycle management expiration rule(s) to expire objects after a certain version number count or a certain period of time. When more versions exist than the rule specifies, or when a version is older than specified, MinIO permanently removes the object version from the drive.

These rules rely on the scanner to process the rule on the bucket. The scanner operates as a lower priority continuous process where READ and WRITE actions are preferred. Because of this, object versions that meet the requirements for expiration may not immediately be removed from MinIO.

See the scanner page for more details on how the scanner works and configuration options.

DeleteMarkers are their own objects. Lifecycle rules can remove DeleteMarkers that are the only remaining versions of their objects.

With JSON, lifecycle rules can remove all versions of a deleted object after a specified number of days.

Retained Objects

MinIO protects objects subject to a locking rule from being overwritten or deleted. These rules require that objects be retained until either the rule expires or is removed.

DELETE operations on locked objects without a specified version result in the creation of a DeleteMarker for the object. However, the object versions themselves are retained as required by the lock.

DELETE operations that specify an object version are subject to the retention rules. MinIO protects object versions subject to a lock from being overwritten or deleted until the lock expires or is removed.

Replicated Objects

Replication duplicates objects from one location to another. MinIO supports replication at the bucket level or the cluster (“site”) level.

Delete operations may or may not replicate, depending on the type of replication and how the replication is configured.

Site Replication

For clusters with multi-site replication enabled, MinIO replicates all delete operations performed on any cluster to each of the other clusters in the peer group.

Delete behavior on any single peer follows the same processes as any MinIO deployment.

Bucket Replication

With bucket replication, MinIO supports replicating delete operations between a source bucket and a configured remote bucket. MinIO synchronizes deleting specific object versions and new delete markers. Delete operation replication uses the same replication process as all other replication operations.

MinIO requires explicitly enabling versioned deletes and delete marker replication. Use the mc replicate add --replicate field to specify either delete and delete-marker or both to enable versioned deletes and delete marker replication, respectively. To enable both, specify both strings using a comma separator: delete,delete-marker.

For delete marker replication, MinIO begins the replication process after a delete operation creates the delete marker. MinIO uses the X-Minio-Replication-DeleteMarker-Status metadata field for tracking delete marker replication status. In active-active replication configurations, MinIO may produce duplicate delete markers if both clusters concurrently create a delete marker for an object or if one or both clusters were down before the replication event synchronized.

For replicating the deletion of a specific object version, MinIO marks the object version as PENDING until replication completes. Once the remote target deletes that object version, MinIO deletes the object version on the source. While this process ensures near-synchronized version deletion, it may result in listing operations returning the object version after the initial delete operation. MinIO uses the X-Minio-Replication-Delete-Status for tracking delete version replication status.

MinIO only replicates explicit client-driven delete operations. MinIO does not replicate objects deleted by lifecycle management expiration rules. For active-active configurations, set the same expiration rules on all of the replication buckets to ensure consistent application of object expiration.

9.6 - Transition Objects from MinIO to GCS

The procedure on this page creates a new object lifecycle management rule that transition objects from a MinIO bucket to a remote storage tier on the Google Cloud Storage backend. This procedure supports use cases like moving aged data to low-cost public cloud storage solutions after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias set command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration
• s3:GetLifecycleConfiguration

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier
• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required GCS Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the GCS credentials provide read, write, list, and delete permissions for the remote bucket.

Refer to the GCS IAM permissions documentation for more complete guidance on configuring the required permissions.

Remote Bucket Must Exist

Create the remote GCS bucket prior to configuring lifecycle management tiers or rules using that bucket as the target.

If you set a default GCS storage class, MinIO uses that default if you do not specify a storage class when defining the remote tier. Ensure you document the settings of both your GCS bucket and MinIO tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes.

Considerations

Lifecycle Management Object Scanner

MinIO uses a scanner process to check objects against all configured lifecycle management rules. Slow scanning due to high IO workloads or limited system resources may delay application of lifecycle management rules. See Lifecycle Management Object Scanner for more information.

Exclusive Access to Remote Data

MinIO requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” MinIO source is strongly linked to the object data on the “warm/cold” remote tier. MinIO cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through MinIO via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.
• No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

MinIO stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. MinIO supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of Remote Data

MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. MinIO therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

MinIO creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the MinIO deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement MinIO Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the MinIO deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the MinIO deployment. Replace this value with the appropriate alias for the MinIO deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

wget -O - https://silo.pgsty.com/extra/examples/LifecycleManagementAdmin.json | \
mc admin policy create Alpha LifecycleAdminPolicy /dev/stdin
mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
mc admin policy attach Alpha LifecycleAdminPolicy --user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and MinIO Policy Based Access Control for more complete documentation on MinIO users and policies respectively.

2) Configure the Remote Storage Tier

Use the mc ilm tier add command to add a new Google Cloud Storage service as the remote storage tier:

mc ilm tier add gcs TARGET TIER_NAME \
   --bucket BUCKET \
   --prefix PREFIX \
   --credentials-file CREDENTIALS \
   --storage-class STORAGE_CLASS

The example above uses the following arguments:

3) Create and Apply the Transition Rule

Use the mc ilm rule add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

mc ilm rule add ALIAS/BUCKET \
--transition-tier TIERNAME \
--transition-days DAYS \
--noncurrent-transition-days NONCURRENT_DAYS
--noncurrent-transition-tier TIERNAME

The example above specifies the following arguments:

4) Verify the Transition Rule

Use the mc ilm rule ls command to review the configured transition rules:

mc ilm rule ls ALIAS/PATH --transition

• Replace ALIAS with the alias of the MinIO deployment.
• Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.

9.7 - Object Lifecycle Management

• MinIO Object Lifecycle Management Part I
• MinIO Object Lifecycle Management Part II
• MinIO Object Lifecycle Management Lab

Use MinIO Object Lifecycle Management to create rules for time or date based automatic transition or expiry of objects. For object transition, MinIO automatically moves the object to a configured remote storage tier. For object expiry, MinIO automatically deletes the object.

MinIO derives it’s behavior and syntax from S3 lifecycle for compatibility in migrating workloads and lifecycle rules from S3 to MinIO. For example, you can export S3 lifecycle management rules and import them into MinIO or vice-versa. MinIO uses JSON to describe lifecycle management rules and may require conversion to or from XML as part of importing S3 lifecycle rules.

Object Transition (“Tiering”)

MinIO supports creating object transition lifecycle management rules, where MinIO can automatically move an object to a remote storage “tier”. MinIO supports any of the following remote tier targets:

• MinIO
• Amazon S3
• Google Cloud Storage
• Microsoft Azure Blob Storage

MinIO object transition supports use cases like moving aged data from MinIO clusters in private or public cloud infrastructure to low-cost private or public cloud storage solutions. Directory objects, which are 0-byte objects with a name ending in /, do not tier. MinIO manages retrieving tiered objects on-the-fly without any additional application-side logic.

Use the mc ilm tier add command to create a remote target for tiering data to that target. You can then use the mc ilm rule add --transition-days command to transition objects to that tier after a specified number of calendar days.

You can verify the tiering status of an object using mc ls against the bucket or bucket prefix. The output includes the storage tier of each object:

$ mc ls play/mybucket
[2022-11-08 11:30:24 PST]    52MB  STANDARD log-data.csv
[2022-11-09 12:20:18 PST]    120MB WARM event-2022-11-09.mp4

• STANDARD marks objects stored on the MinIO deployment.
• WARM marks objects stored on the remote tier with matching name.

Exclusive Access to Remote Data

MinIO requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” MinIO source is strongly linked to the object data on the “warm/cold” remote tier. MinIO cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through MinIO via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.
• No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

MinIO stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. MinIO supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of Remote Data

MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. MinIO therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

MinIO creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the MinIO deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement MinIO Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Versioned Buckets

MinIO adopts S3 behavior for transition rules on versioned buckets. Specifically, MinIO by default applies the transition operation to the current object version.

To transition noncurrent object versions, specify the --noncurrent-transition-days and --noncurrent-transition-tier options when creating the transition rule.

Object Expiration

MinIO lifecycle management supports expiring objects on a bucket. Object “expiration” involves performing a DELETE operation on the object. For example, you can create a lifecycle management rule to expire any object older than 365 days.

Use mc ilm rule add --expire-days to expire objects after a specified number of calendar days.

For buckets with replication configured, MinIO does not replicate objects deleted by a lifecycle management expiration rule. See Replication of Delete Operations for more information.

Versioned Buckets

MinIO adopts S3 behavior for expiration rules on versioned buckets. MinIO has several default behaviors for versioned buckets:

• MinIO applies the expiration option to only the current object version by creating a DeleteMarker as is normal with versioned delete.

  To expire noncurrent object versions, specify the --noncurrent-expire-days option when creating the expiration rule.

• MinIO does not expire DeleteMarkers even if no other versions of that object exist.

  To expire delete markers when there are no remaining versions for that object, specify the --expire-delete-marker option when creating the expiration rule.

• To expire all versions of an object that does not have a delete marker after a specified period of days, use the --expire-all-object-versions flag with the --expire-days flag. This permits the permanent deletion of the object after the specified number of days pass.

  Note

  Changed: MinIO

  RELEASE.2024-05-01T01-11-10Z

  This flag applies only to objects that do not have a delete marker.

Lifecycle Management Object Scanner

MinIO uses a built-in scanner to actively check objects against all configured lifecycle management rules.

The scanner is a low-priority process that yields to high I/O workloads to prevent performance spikes triggered by rule timing. The scanner may therefore not detect an object as eligible for a configured transition or expiration lifecycle rule until after the lifecycle rule period has passed.

9.8 - Transition Objects from MinIO to Azure

The procedure on this page creates a new object lifecycle management rule that transition objects from a MinIO bucket to a remote storage tier on the Azure storage backend. This procedure supports use cases like moving aged data to low-cost public cloud storage solutions after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias set command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration
• s3:GetLifecycleConfiguration

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier
• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required Azure Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the Azure credentials provide read, write, list, and delete permissions for the remote storage account and container.

Refer to the Azure RBAC documentation for more complete guidance on configuring the required permissions.

Remote Storage Account and Container Must Exist

Create the remote Azure storage account and container prior to configuring lifecycle management tiers or rules using that resource as the target. When creating the Azure storage account, ensure the storage account corresponds to either Standard or Premium blob storage with the locally redundant storage (LRS) redundancy option. The Azure Go SDK API used by MinIO does not support any other redundancy options.

If you set a Storage Account default access tier, MinIO uses that default if you do not specify a storage class when defining the remote tier. Ensure you document the settings of both your Azure storage account and MinIO tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes.

For more information on Azure storage accounts, see Storage accounts.

Considerations

Exclusive Access to Remote Data

MinIO requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” MinIO source is strongly linked to the object data on the “warm/cold” remote tier. MinIO cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through MinIO via S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

• No external mutation, migration, or deletion of objects on the remote storage.
• No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

MinIO stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. MinIO supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of Remote Data

MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. MinIO therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

MinIO creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the MinIO deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement MinIO Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the MinIO deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the MinIO deployment. Replace this value with the appropriate alias for the MinIO deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

wget -O - https://silo.pgsty.com/extra/examples/LifecycleManagementAdmin.json | \
mc admin policy create Alpha LifecycleAdminPolicy /dev/stdin
mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
mc admin policy attach Alpha LifecycleAdminPolicy --user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and MinIO Policy Based Access Control for more complete documentation on MinIO users and policies respectively.

2) Configure the Remote Storage Tier

Use the mc ilm tier add command to add a new remote storage tier:

mc ilm tier add azure TARGET TIER_NAME \
   --account-name ACCOUNT \
   --account-key KEY \
   --bucket CONTAINER \
   --endpoint ENDPOINT \
   --prefix PREFIX \
   --storage-class STORAGE_CLASS

The example above uses the following arguments:

3) Create and Apply the Transition Rule

Use the mc ilm rule add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

mc ilm rule add ALIAS/BUCKET \
--transition-tier TIERNAME \
--transition-days DAYS \
--noncurrent-transition-days NONCURRENT_DAYS
--noncurrent-transition-tier TIERNAME

The example above specifies the following arguments:

4) Verify the Transition Rule

Use the mc ilm rule ls command to review the configured transition rules:

mc ilm rule ls ALIAS/PATH --transition

• Replace ALIAS with the alias of the MinIO deployment.
• Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.

9.9 - Automatic Object Expiration

Each procedure on this page creates a new object lifecycle management rule that expires objects on a MinIO bucket. This procedure supports use cases like removing “old” objects after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias set command to create an alias for the source MinIO cluster and the destination S3-compatible service. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying expiry operations.

Required Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration
• s3:GetLifecycleConfiguration

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier
• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Expire Objects after Number of Days

Use mc ilm rule add with --expire-days to expire bucket contents a number of days after object creation:

mc ilm rule add ALIAS/PATH --expire-days "DAYS"

• Replace ALIAS with the alias of the S3-compatible host.
• Replace PATH with the path to the bucket on the S3-compatible host.
• Replace DAYS with the number of days after which to expire the object. For example, specify 30 to expire the object 30 days after creation.

Expire Versioned Objects

Use mc ilm rule add to expiring noncurrent object versions and object delete markers:

• To expire noncurrent object versions after a specific duration in days, include --noncurrent-expire-days.
• To expire delete markers for objects with no remaining versions, include --expire-delete-marker.

mc ilm rule add ALIAS/PATH \
   --noncurrent-expire-days NONCURRENT_DAYS \
   --expire-delete-marker

• To expire all versions of an object, include --expire-all-object-versions. This expiration only applies to objects without a DeleteMarker as the latest or current version.

  mc ilm rule add ALIAS/PATH \
     --expire-all-object-versions

• Replace ALIAS with the alias of the S3-compatible host.

• Replace PATH with the path to the bucket on the S3-compatible host.

• Replace NONCURRENT_DAYS with the number of days after which to expire noncurrent object versions. For example, specify 30d to expire a version after it has been noncurrent for at least 30 days.

9.10 - Data Compression

Overview

MinIO Server supports compressing objects to reduce disk usage. Objects are compressed on PUT before writing to disk, and uncompressed on GET before they are sent to the client. This makes the compression process transparent to client applications and services.

Depending on the type of data, compression may also increase overall throughput. Write throughput for a production deployment is generally 500MB per second or greater per available CPU core in the system. Decompression is approximately 1 GB per second or greater for each CPU core.

For best results, review MinIO’s recommended hardware configuration or use MinIO SUBNET to work directly with engineers for analyzing compression performance.

Default File Types

Data compression is a global option, the configured settings apply to all buckets in a deployment. Enabling data compression compresses the following types of data by default:

You can control which objects are compressed by specifying the desired file extensions and media (MIME) types.

Excluded File Types

Some data cannot be effectively compressed. For example: video, already compressed data, or files less than 4KiB. MinIO does not compress common incompressible file types, even if they are specified in the compression configuration.

Objects of these types are never compressed:

Data Compression and Encryption

MinIO supports encrypting compressed objects but recommends against combining compression and encryption without a prior risk assessment. Before enabling encryption for compressed objects, carefully consider the security needs of your environment.

See Transparent Data Compression on MinIO for more about combining compression and encryption. MinIO SUBNET users can log in and engage with our engineering and security teams to review encryption options.

Tutorials

Enable Data Compression

To enable data compression, use mc admin config set to set the compression key enable option to on.

The following enables compression for new objects of the default types:

mc admin config set ALIAS compression enable=on

• Replace ALIAS with the alias of a configured MinIO deployment.

Existing uncompressed objects are not modified. To configure which extensions and types to compress, see Configure Which Objects to Compress.

To view the current compression settings:

mc admin config get ALIAS compression

Disable Data Compression

To disable data compression, use mc admin config set to set the compression key enable option to off:

The following disables data compression for new objects:

mc admin config set ALIAS compression enable=off

• Replace ALIAS with the alias of a configured MinIO deployment.

Existing compressed objects are not modified.

Configure Which Objects to Compress

Configure the objects to compress by specifying the desired file extensions and media types in extensions or mime_types arguments.

The default data compression configuration compresses the following types of data:

The sections below describe how to configure compression for the desired file extensions and media types.

Compress All Compressible Objects

To compress all objects except the default excluded types, use mc admin config set to set the compression key extensions and mime_types options to empty lists:

mc admin config set ALIAS compression extensions= mime_types=

• Replace ALIAS with the alias of a configured MinIO deployment.

Compress Objects by File Extension

To compress objects with certain file extensions, use mc admin config set to set the desired file extensions in an extensions argument.

The following command compresses files with the extensions .bin and .txt:

mc admin config set ALIAS compression extensions=".bin, .txt"

• Replace ALIAS with the alias of a configured MinIO deployment.

The new list of file extensions replaces the previous list. To add or remove an extension, repeat the extensions command with the complete list of extensions to compress.

The following adds .pdf to the list of file extensions from the previous example:

mc admin config set ALIAS compression extensions=".bin, .txt, .pdf"

• Replace ALIAS with the alias of a configured MinIO deployment.

Compress Objects by Media Type

To compress objects of certain media types, use mc admin config set to set the compression key mime_types option to a list of the desired types.

The following example compresses files of types application/json and image/bmp:

mc admin config set ALIAS compression mime_types="application/json, image/bmp"

• Replace ALIAS with the alias of a configured MinIO deployment.

The new list of media types replaces the previous list. To add or remove a type, repeat the mime_types command with the complete list of types to compress.

You can use * to specify all subtypes of a single media type. The following command adds all text subtypes to the list from the previous example:

mc admin config set ALIAS compression mime_types="application/json, image/bmp, text/*"

• Replace ALIAS with the alias of a configured MinIO deployment.

10 - Monitoring Bucket and Object Events

Bucket Notifications

MinIO bucket notifications allow administrators to send notifications to supported external services on certain object or bucket events. MinIO supports bucket and object-level S3 events similar to the Amazon S3 Event Notifications.

MinIO supports publishing bucket or object events to the following supported targets on certain supported events.

• Publish Events to AMQP (RabbitMQ)
• Publish Events to MQTT
• Publish Events to NATS
• Publish Events to NSQ
• Publish Events to Elasticsearch
• Publish Events to Kafka
• Publish Events to MySQL
• Publish Events to PostgreSQL
• Publish Events to Redis
• Publish Events to Webhook

See Bucket notifications for more complete documentation on MinIO Bucket Notifications.

Deployment Metrics

MinIO provides a Prometheus-compatible endpoint for supporting time-series querying of metrics.

Server Logs

MinIO provides the following interfaces for remotely reading server logs:

• The mc admin logs command returns the specified server’s console output.
• MinIO supports pushing server logs to an HTTP webhook for further ingestion. See Publish Server Logs to HTTP Webhook for more information.

10.1 - Bucket notifications

MinIO bucket notifications allow administrators to send notifications to supported external services on certain object or bucket events. MinIO supports bucket and object-level S3 events similar to the Amazon S3 Event Notifications.

Supported notification targets

MinIO supports publishing event notifications to the following targets:

Asynchronous vs synchronous bucket notifications

With asynchronous delivery, MinIO fires the event at the configured remote and does not wait for a response before continuing to the next event. Asynchronous bucket notification prioritizes sending events with the risk of some events being lost if the remote target has a transient issue during transit or processing.

With synchronous delivery, MinIO fires the event at the configured remote and then waits for the remote to confirm a successful receipt before continuing to the next event. Synchronous bucket notification prioritizes delivery of events with the risk of a slower event-send rate and queue fill.

To enable synchronous bucket notifications for all configured remote targets, use either of the following settings:

• Set the MINIO_API_SYNC_EVENTS environment variable to on and restart the MinIO deployment.
• Set the api.sync_events configuration setting to on and restart the MinIO deployment.

Supported S3 event types

MinIO bucket notifications are compatible with Amazon S3 Event Notifications. This section lists all supported events.

Object events

MinIO supports triggering notifications on the following S3 object events:

s3:ObjectAccessed:Get

data

s3:ObjectAccessed:GetLegalHold

data

s3:ObjectAccessed:GetRetention

data

s3:ObjectAccessed:Head

data

s3:ObjectCreated:CompleteMultipartUpload

data

s3:ObjectCreated:Copy

data

s3:ObjectCreated:DeleteTagging

data

s3:ObjectCreated:Post

data

s3:ObjectCreated:Put

data

s3:ObjectCreated:PutLegalHold

data

s3:ObjectCreated:PutRetention

data

s3:ObjectCreated:PutTagging

data

s3:ObjectRemoved:Delete

data

s3:ObjectRemoved:DeleteMarkerCreated

data

Specify the wildcard * character to select all events related to a prefix:

s3:ObjectAccessed:*

data

Selects all s3:ObjectAccessed -prefixed events.

s3:ObjectCreated:*

data

Selects all s3:ObjectCreated -prefixed events.

s3:ObjectRemoved:*

data

Selects all s3:ObjectRemoved -prefixed events.

Replication events

MinIO supports triggering notifications on the following S3 replication events:

s3:Replication:OperationCompletedReplication

data

s3:Replication:OperationFailedReplication

data

s3:Replication:OperationMissedThreshold

data

s3:Replication:OperationNotTracked

data

s3:Replication:OperationReplicatedAfterThreshold

data

Specify the wildcard * character to select all s3:Replication events:

s3:Replication:*

data

ILM transition events

MinIO supports triggering notifications on the following S3 ILM transition events:

s3:ObjectRestore:Post

data

s3:ObjectRestore:Completed

data

s3:ObjectTransition:Failed

data

s3:ObjectTransition:Complete

data

Specify the wildcard * character to select all events related to a prefix:

s3:ObjectTransition:*

data

Selects all s3:ObjectTransition -prefixed events.

s3:ObjectRestore:*

data

Selects all s3:ObjectRestore -prefixed events.

Scanner events

MinIO supports triggering notifications on the following S3 scanner transition events:

s3:Scanner:ManyVersions

data

Scanner finds objects with more than 1,000 versions.

s3:Scanner:BigPrefix

data

Scanner finds prefixes with more than 50,000 sub-folders.

Global events

MinIO supports triggering notifications on the following global events. You can only listen to these events through the ListenNotification API:

s3:BucketCreated

data

s3:BucketRemoved

data

Payload schema

All notification payloads use the same overall schema. Depending on the type of notification, some fields may be omitted or have null values.

{
    "eventVersion": "string",
    "eventSource": "string",
    "awsRegion": "string",
    "eventTime": "string",
    "eventName": "string",
    "userIdentity": {
        "principalId": "string"
    },
    "requestParameters": {
        "key": "value"
    },
    "responseElements": {
        "key": "value"
    },
    "s3": {
        "s3SchemaVersion": "string",
        "configurationId": "string",
        "bucket": {
            "name": "string",
            "ownerIdentity": {
                "principalId": "string"
            },
            "arn": "string"
        },
        "object": {
            "key": "string",
            "size": 10000,
            "eTag": "string",
            "contentType": "string",
            "userMetadata": {
                "key": "string"
            },
            "versionId": "string",
            "sequencer": "string"
        }
    },
    "source": {
        "host": "string",
        "port": "string",
        "userAgent": "string"
    }
}

Example

The following example is a notification for an s3:ObjectCreated:Put event:

{
  "EventName": "s3:ObjectCreated:Put",
  "Key": "test-bucket/image.jpg",
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "minio:s3",
      "awsRegion": "",
      "eventTime": "2025-02-06T01:04:31.998Z",
      "eventName": "s3:ObjectCreated:Put",
      "userIdentity": {
        "principalId": "access_key"
      },
      "requestParameters": {
        "principalId": "access_key",
        "region": "",
        "sourceIPAddress": "192.168.1.10"
      },
      "responseElements": {
        "x-amz-id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
        "x-amz-request-id": "182178E8B36AC9DF",
        "x-minio-deployment-id": "2369dcb4-348b-4d30-8fc9-61ab089ba4bc",
        "x-minio-origin-endpoint": "https://minio.test.svc.cluster.local"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "Config",
        "bucket": {
          "name": "test-bucket",
          "ownerIdentity": {
            "principalId": "access_key"
          },
          "arn": "arn:aws:s3:::test-bucket"
        },
        "object": {
          "key": "image.jpg",
          "size": 84452,
          "eTag": "eb52f8e46f60a27a8a1a704e25757f30",
          "contentType": "image/jpeg",
          "userMetadata": {
            "content-type": "image/jpeg"
          },
          "sequencer": "182178E8B3728CAC"
        }
      },
      "source": {
        "host": "192.168.1.10",
        "port": "",
        "userAgent": "MinIO (linux; amd64) minio-go/v7.0.83"
      }
    }
  ]
}

10.2 - Publish Events to AMQP (RabbitMQ)

MinIO supports publishing bucket notification events to a AMQP 0-9-1 service endpoint such as RabbitMQ.

MinIO relies on the https://github.com/streadway/amqp project for AMQP connectivity. The project is primarily tested against RabbitMQ deployments, though other AMQP 0-9-1-compatible services may also work. The procedures on this page assume a RabbitMQ deployment using the AMQP 0-9-1 protocol as the service endpoint.

Add an AMQP Endpoint to a MinIO Deployment

The following procedure adds a new AMQP service endpoint for supporting bucket notifications in a MinIO deployment.

Prerequisites

AMQP 0-9-1 Service Endpoint

MinIO relies on the https://github.com/streadway/amqp project for AMQP connectivity. The project is primarily tested against RabbitMQ deployments, though other AMQP 0-9-1-compatible services may also work. This procedure assumes a RabbitMQ deployment using the 0-9-1 protocol as the service endpoint.

If the AMQP service requires authentication, you must provide an appropriate username and password during the configuration process to grant MinIO access to the service.

MinIO mc Command Line Tool

This procedure uses the mc command line tool for certain actions. See the mc Quickstart for installation instructions.

1) Add the AMQP Endpoint to MinIO

You can configure a new AMQP service endpoint using either environment variables or by setting runtime configuration settings.

   set MINIO_NOTIFY_AMQP_ENABLE_<IDENTIFIER>="on"
   set MINIO_NOTIFY_AMQP_URL_<IDENTIFIER>="<ENDPOINT>"
   set MINIO_NOTIFY_AMQP_EXCHANGE_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_EXCHANGE_TYPE_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_ROUTING_KEY_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_MANDATORY_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_DURABLE_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_NO_WAIT_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_INTERNAL_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_AUTO_DELETED_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_DELIVERY_MODE_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_QUEUE_DIR_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_QUEUE_LIMIT_<IDENTIFIER>="<string>"
   set MINIO_NOTIFY_AMQP_COMMENT_<IDENTIFIER>="<string>"