Integrations

• 1: Using Silo with Veeam
• 2: Disaggregated HDP Spark and Hive with Silo
• 3: AWS CLI with Silo Server
• 4: Configure NGINX Proxy for Silo Server
• 5: Upload Files Using Pre-signed URLs
• 6: Generate a Let’s Encrypt Certificate for Silo with Certbot

1 - Using Silo with Veeam

When using Veeam Backup and Replication, you can use S3-compatible object storage such as Silo as a capacity tier for backups. This disaggregates storage for the Veeam infrastructure and allows you to retain control of your data. Silo’s straightforward setup and administration let a Veeam backup administrator deploy an object store for capacity tiering.

Prerequisites

• One or both of Veeam Backup and Replication with support for S3 compatible object store (e.g. 9.5.4) and Veeam Backup for Office365 (VBO)
• Silo object storage set up according to the deployment procedure
• Veeam requires TLS connections to the object storage. Configure TLS using the network encryption guide.
• The S3 bucket, Access Key and Secret Key have to be created before and outside of Veeam.
• Configure the Silo client for the Veeam Silo endpoint using the mc command reference.

Setting up an S3 compatible object store for Veeam Backup and Replication

Create a bucket for Veeam backups

Create a bucket for Veeam Backup, e.g.,

mc mb myminio/veeambackup

NOTE: For Veeam Backup with Immutability, create the bucket with object lock enabled, e.g.,

mc mb -l myminio/veeambackup

Object locking requires erasure coding on the silo server. See the erasure coding documentation for more information.

Add Silo as an object store for Veeam

Follow the Veeam documentation for adding S3-compatible object storage: Add Object Storage.

For Veeam Backup with Immutability, choose the amount of days you want to make backups immutable for

Creating the Scale-out Backup Repository

• Under the Backup Infrastructure view, click on Scale-out Repositories and click the Add Scale-out Repository button on the ribbon.
• Follow the on screen wizard
• On the Capacity Tier screen, check the box to Extend scale-out backup repository capacity with object storage checkbox and select the object storage. If you want to be able to test backup data immediately after a job is run, under the object storage selection, check the “Copy” box and uncheck the “Move” box.

Create a backup job

Backup Virtual Machines with Veeam Backup and Replication

• Under Home > Jobs > Backup in Navigation Pane, click on Backup Job button in the ribbon and choose Virtual Machine. Follow the on screen wizard.
• On the Storage screen, choose the Scale-out Backup Repository that was configured previously.
• Continue with the backup job creation. On the Summary screen, check the Run the Job when I click Finish checkbox and click the Finish button. The backup job will start immediately. This will create an Active Full backup of the VMs within the backup job.
• Since we selected Copy mode when creating the SOBR, the backup will be copied to the capacity tier as soon as it is created on the performance tier.
• For Veeam Backup with Immutability, you can choose a number of restore points or days to make backups immutable.

Backup Office 365 with VBO

• Create a new bucket for VBO backups

mc mb -l myminio/vbo

• Under Backup Infrastructure, right click on Object Storage Repositories and choose “Add object storage”

• Follow through the wizard as above for Veeam Backup and Replication as the steps are the same between both products
• Under Backup Infrastructure -> Backup Repositories, right click and “Add Backup Repository”
• Follow the wizard. Under the “Object Storage Backup Repository” section, choose the Silo object storage you created above

• When you create your backup job, choose the backup repository you created above.

Test the setup

The next time the backup job runs, you can use the mc admin trace myminio command and verify traffic is flowing to the Silo nodes. For Veeam Backup and Replication you will need to wait for the backup to complete to the performance tier before it migrates data to the Silo capacity tier.

20:09:10.216 [200 OK] s3.GetObject veeam-minio01:9000/vbo/Veeam/Backup365/vbotest/Organizations/6571606ecbc4455dbfe23b83f6f45597/Webs/ca2d0986229b4ec88e3a217ef8f04a1d/Items/efaa67764b304e77badb213d131beab6/f4f0cf600f494c3eb702d8eafe0fabcc.aac07493e6cd4c71845d2495a4e1e19b 139.178.68.158    9.789ms      ↑ 90 B ↓ 8.5 KiB
20:09:10.244 [200 OK] s3.GetObject veeam-minio01:9000/vbo/Veeam/Backup365/vbotest/RepositoryLock/cad99aceb50c49ecb9e07246c3b9fadc_bfd985e5deec4cebaf481847f2c34797 139.178.68.158    16.21ms      ↑ 90 B ↓ 402 B
20:09:10.283 [200 OK] s3.PutObject veeam-minio01:9000/vbo/Veeam/Backup365/vbotest/CommonInfo/WebRestorePoints/18f1aba8f55f4ac6b805c4de653eb781 139.178.68.158    29.787ms     ↑ 1005 B ↓ 296 B

2 - Disaggregated HDP Spark and Hive with Silo

1. Cloud-native Architecture

Kubernetes manages stateless Spark and Hive containers elastically on the compute nodes. Spark has native scheduler integration with Kubernetes. Hive, for legacy reasons, uses YARN scheduler on top of Kubernetes.

All access to MinIO object storage is via S3/SQL SELECT API. In addition to the compute nodes, MinIO containers are also managed by Kubernetes as stateful containers with local storage (JBOD/JBOF) mapped as persistent local volumes. This architecture enables multi-tenant MinIO, allowing isolation of data between customers.

MinIO also supports multi-cluster, multi-site federation similar to AWS regions and tiers. Using MinIO Information Lifecycle Management (ILM), you can configure data to be tiered between NVMe based hot storage, and HDD based warm storage. All data is encrypted with per-object key. Access Control and Identity Management between the tenants are managed by MinIO using OpenID Connect or Kerberos/LDAP/AD.

2. Prerequisites

• Install Hortonworks Distribution using this guide.

  • Setup Ambari which automatically sets up YARN
  • Installing Spark

• Install MinIO Distributed Server using one of the guides below.

  • Deployment based on Kubernetes
  • Deployment based on MinIO Helm Chart

3. Configure Hadoop, Spark, Hive to use MinIO

After successful installation navigate to the Ambari UI http://<ambari-server>:8080/ and login using the default credentials: [username: admin, password: admin]

3.1 Configure Hadoop

Navigate to Services -> HDFS -> CONFIGS -> ADVANCED as shown below

Navigate to Custom core-site to configure MinIO parameters for _s3a_ connector

sudo pip install yq
alias kv-pairify='yq ".configuration[]" | jq ".[]" | jq -r ".name + \"=\" + .value"'

Let’s take for example a set of 12 compute nodes with an aggregate memory of 1.2TiB, we need to do following settings for optimal results. Add the following optimal entries for core-site.xml to configure s3a with MinIO. Most important options here are

cat ${HADOOP_CONF_DIR}/core-site.xml | kv-pairify | grep "mapred"

mapred.maxthreads.generate.mapoutput=2 # Num threads to write map outputs
mapred.maxthreads.partition.closer=0 # Asynchronous map flushers
mapreduce.fileoutputcommitter.algorithm.version=2 # Use the latest committer version
mapreduce.job.reduce.slowstart.completedmaps=0.99 # 99% map, then reduce
mapreduce.reduce.shuffle.input.buffer.percent=0.9 # Min % buffer in RAM
mapreduce.reduce.shuffle.merge.percent=0.9 # Minimum % merges in RAM
mapreduce.reduce.speculative=false # Disable speculation for reducing
mapreduce.task.io.sort.factor=999 # Threshold before writing to drive
mapreduce.task.sort.spill.percent=0.9 # Minimum % before spilling to drive

S3A is the connector to use S3 and other S3-compatible object stores such as MinIO. MapReduce workloads typically interact with object stores in the same way they do with HDFS. These workloads rely on HDFS atomic rename functionality to complete writing data to the datastore. Object storage operations are atomic by nature and they do not require/implement rename API. The default S3A committer emulates renames through copy and delete APIs. This interaction pattern causes significant loss of performance because of the write amplification. Netflix, for example, developed two new staging committers - the Directory staging committer and the Partitioned staging committer - to take full advantage of native object storage operations. These committers do not require rename operation. The two staging committers were evaluated, along with another new addition called the Magic committer for benchmarking.

It was found that the directory staging committer was the fastest among the three, S3A connector should be configured with the following parameters for optimal results:

cat ${HADOOP_CONF_DIR}/core-site.xml | kv-pairify | grep "s3a"

fs.s3a.access.key=minio
fs.s3a.secret.key=minio123
fs.s3a.path.style.access=true
fs.s3a.block.size=512M
fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a
fs.s3a.committer.magic.enabled=false
fs.s3a.committer.name=directory
fs.s3a.committer.staging.abort.pending.uploads=true
fs.s3a.committer.staging.conflict-mode=append
fs.s3a.committer.staging.tmp.path=/tmp/staging
fs.s3a.committer.staging.unique-filenames=true
fs.s3a.connection.establish.timeout=5000
fs.s3a.connection.ssl.enabled=false
fs.s3a.connection.timeout=200000
fs.s3a.endpoint=http://minio:9000
fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem

fs.s3a.committer.threads=2048 # Number of threads writing to MinIO
fs.s3a.connection.maximum=8192 # Maximum number of concurrent conns
fs.s3a.fast.upload.active.blocks=2048 # Number of parallel uploads
fs.s3a.fast.upload.buffer=disk # Use drive as the buffer for uploads
fs.s3a.fast.upload=true # Turn on fast upload mode
fs.s3a.max.total.tasks=2048 # Maximum number of parallel tasks
fs.s3a.multipart.size=512M # Size of each multipart chunk
fs.s3a.multipart.threshold=512M # Size before using multipart uploads
fs.s3a.socket.recv.buffer=65536 # Read socket buffer hint
fs.s3a.socket.send.buffer=65536 # Write socket buffer hint
fs.s3a.threads.max=2048 # Maximum number of threads for S3A

The rest of the other optimization options are discussed in the links below

• https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
• https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/committers.html

Once the config changes are applied, proceed to restart Hadoop services.

3.2 Configure Spark2

Navigate to Services -> Spark2 -> CONFIGS as shown below

Navigate to “Custom spark-defaults” to configure MinIO parameters for _s3a_ connector

Add the following optimal entries for spark-defaults.conf to configure Spark with MinIO.

spark.hadoop.fs.s3a.access.key minio
spark.hadoop.fs.s3a.secret.key minio123
spark.hadoop.fs.s3a.path.style.access true
spark.hadoop.fs.s3a.block.size 512M
spark.hadoop.fs.s3a.buffer.dir ${hadoop.tmp.dir}/s3a
spark.hadoop.fs.s3a.committer.magic.enabled false
spark.hadoop.fs.s3a.committer.name directory
spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads true
spark.hadoop.fs.s3a.committer.staging.conflict-mode append
spark.hadoop.fs.s3a.committer.staging.tmp.path /tmp/staging
spark.hadoop.fs.s3a.committer.staging.unique-filenames true
spark.hadoop.fs.s3a.committer.threads 2048 # number of threads writing to MinIO
spark.hadoop.fs.s3a.connection.establish.timeout 5000
spark.hadoop.fs.s3a.connection.maximum 8192 # maximum number of concurrent conns
spark.hadoop.fs.s3a.connection.ssl.enabled false
spark.hadoop.fs.s3a.connection.timeout 200000
spark.hadoop.fs.s3a.endpoint http://minio:9000
spark.hadoop.fs.s3a.fast.upload.active.blocks 2048 # number of parallel uploads
spark.hadoop.fs.s3a.fast.upload.buffer disk # use disk as the buffer for uploads
spark.hadoop.fs.s3a.fast.upload true # turn on fast upload mode
spark.hadoop.fs.s3a.impl org.apache.hadoop.spark.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.max.total.tasks 2048 # maximum number of parallel tasks
spark.hadoop.fs.s3a.multipart.size 512M # size of each multipart chunk
spark.hadoop.fs.s3a.multipart.threshold 512M # size before using multipart uploads
spark.hadoop.fs.s3a.socket.recv.buffer 65536 # read socket buffer hint
spark.hadoop.fs.s3a.socket.send.buffer 65536 # write socket buffer hint
spark.hadoop.fs.s3a.threads.max 2048 # maximum number of threads for S3A

Once the config changes are applied, proceed to restart Spark services.

3.3 Configure Hive

Navigate to Services -> Hive -> CONFIGS-> ADVANCED as shown below

Navigate to “Custom hive-site” to configure MinIO parameters for _s3a_ connector

Add the following optimal entries for hive-site.xml to configure Hive with MinIO.

hive.blobstore.use.blobstore.as.scratchdir=true
hive.exec.input.listing.max.threads=50
hive.load.dynamic.partitions.thread=25
hive.metastore.fshandler.threads=50
hive.mv.files.threads=40
mapreduce.input.fileinputformat.list-status.num-threads=50

For more information about these options please visit https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html

Once the config changes are applied, proceed to restart all Hive services.

4. Run Sample Applications

After installing Hive, Hadoop and Spark successfully, we can now proceed to run some sample applications to see if they are configured appropriately. We can use Spark Pi and Spark WordCount programs to validate our Spark installation. We can also explore how to run Spark jobs from the command line and Spark shell.

4.1 Spark Pi

Test the Spark installation by running the following compute intensive example, which calculates pi by “throwing darts” at a circle. The program generates points in the unit square ((0,0) to (1,1)) and counts how many points fall within the unit circle within the square. The result approximates pi.

Follow these steps to run the Spark Pi example:

• Login as user ‘spark’.
• When the job runs, the library can now use MinIO during intermediate processing.
• Navigate to a node with the Spark client and access the spark2-client directory:

cd /usr/hdp/current/spark2-client
su spark

• Run the Apache Spark Pi job in yarn-client mode, using code from org.apache.spark:

./bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn-client \
    --num-executors 1 \
    --driver-memory 512m \
    --executor-memory 512m \
    --executor-cores 1 \
    examples/jars/spark-examples*.jar 10

The job should produce an output as shown below. Note the value of pi in the output.

17/03/22 23:21:10 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.302805 s
Pi is roughly 3.1445191445191445

Job status can also be viewed in a browser by navigating to the YARN ResourceManager Web UI and clicking on job history server information.

4.2 WordCount

WordCount is a simple program that counts how often a word occurs in a text file. The code builds a dataset of (String, Int) pairs called counts, and saves the dataset to a file.

The following example submits WordCount code to the Scala shell. Select an input file for the Spark WordCount example. We can use any text file as input.

• Login as user ‘spark’.
• When the job runs, the library can now use MinIO during intermediate processing.
• Navigate to a node with Spark client and access the spark2-client directory:

cd /usr/hdp/current/spark2-client
su spark

The following example uses log4j.properties as the input file:

4.2.1 Upload the input file to HDFS:

hadoop fs -copyFromLocal /etc/hadoop/conf/log4j.properties
          s3a://testbucket/testdata

4.2.2 Run the Spark shell:

./bin/spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m

The command should produce an output as shown below. (with additional status messages):

Spark context Web UI available at http://172.26.236.247:4041
Spark context available as 'sc' (master = yarn, app id = application_1490217230866_0002).
Spark session available as 'spark'.
Welcome to


      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.2.6.0.0-598
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

• At the scala> prompt, submit the job by typing the following commands, Replace node names, file name, and file location with your values:

scala> val file = sc.textFile("s3a://testbucket/testdata")
file: org.apache.spark.rdd.RDD[String] = s3a://testbucket/testdata MapPartitionsRDD[1] at textFile at <console>:24

scala> val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
counts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:25

scala> counts.saveAsTextFile("s3a://testbucket/wordcount")

Use one of the following approaches to view job output:

View output in the Scala shell:

scala> counts.count()
364

To view the output from MinIO exit the Scala shell. View WordCount job status:

hadoop fs -ls s3a://testbucket/wordcount

The output should be similar to the following:

Found 3 items
-rw-rw-rw-   1 spark spark          0 2019-05-04 01:36 s3a://testbucket/wordcount/_SUCCESS
-rw-rw-rw-   1 spark spark       4956 2019-05-04 01:36 s3a://testbucket/wordcount/part-00000
-rw-rw-rw-   1 spark spark       5616 2019-05-04 01:36 s3a://testbucket/wordcount/part-00001

3 - AWS CLI with Silo Server

AWS CLI is a unified tool to manage AWS services. It is frequently the tool used to transfer data in and out of AWS S3. It works with any S3 compatible cloud storage service.

In this recipe we will learn how to configure and use AWS CLI to manage data with MinIO Server.

1. Prerequisites

Install MinIO Server using the SILO installation guide.

2. Installation

Install AWS CLI from https://aws.amazon.com/cli/

3. Configuration

To configure AWS CLI, type aws configure and specify the MinIO key information.

Access credentials shown in this example belong to https://play.min.io:9000. These credentials are open to public. Feel free to use this service for testing and development. Replace with your own MinIO keys in deployment.

aws configure
AWS Access Key ID [None]: Q3AM3UQ867SPQQA43P2F
AWS Secret Access Key [None]: zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
Default region name [None]: us-east-1
Default output format [None]: ENTER

Additionally enable AWS Signature Version ‘4’ for MinIO server.

aws configure set default.s3.signature_version s3v4

4. Commands

To list your buckets

aws --endpoint-url https://play.min.io:9000 s3 ls
2016-03-27 02:06:30 deebucket
2016-03-28 21:53:49 guestbucket
2016-03-29 13:34:34 mbtest
2016-03-26 22:01:36 mybucket
2016-03-26 15:37:02 testbucket

To list contents inside bucket

aws --endpoint-url https://play.min.io:9000 s3 ls s3://mybucket
2016-03-30 00:26:53      69297 argparse-1.2.1.tar.gz
2016-03-30 00:35:37      67250 simplejson-3.3.0.tar.gz

To make a bucket

aws --endpoint-url https://play.min.io:9000 s3 mb s3://mybucket
make_bucket: s3://mybucket/

To add an object to a bucket

aws --endpoint-url https://play.min.io:9000 s3 cp simplejson-3.3.0.tar.gz s3://mybucket
upload: ./simplejson-3.3.0.tar.gz to s3://mybucket/simplejson-3.3.0.tar.gz

To delete an object from a bucket

aws --endpoint-url https://play.min.io:9000 s3 rm s3://mybucket/argparse-1.2.1.tar.gz
delete: s3://mybucket/argparse-1.2.1.tar.gz

To remove a bucket

aws --endpoint-url https://play.min.io:9000 s3 rb s3://mybucket
remove_bucket: s3://mybucket/

4 - Configure NGINX Proxy for Silo Server

The following documentation provides a baseline for configuring NGINX to proxy requests to MinIO in a Linux environment. It is not intended as a comprehensive approach to NGINX, proxying, or reverse proxying in general. Modify the configuration as necessary for your infrastructure.

This documentation assumes the following:

• An existing NGINX deployment
• An existing MinIO deployment
• A DNS hostname which uniquely identifies the MinIO deployment

There are two models for proxying requests to the MinIO Server API and the MinIO Console:

upstream minio_s3 {
   least_conn;
   server minio-01.internal-domain.com:9000;
   server minio-02.internal-domain.com:9000;
   server minio-03.internal-domain.com:9000;
   server minio-04.internal-domain.com:9000;
}

upstream minio_console {
   least_conn;
   server minio-01.internal-domain.com:9001;
   server minio-02.internal-domain.com:9001;
   server minio-03.internal-domain.com:9001;
   server minio-04.internal-domain.com:9001;
}

server {
   listen       80;
   listen  [::]:80;
   server_name  minio.example.net;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      proxy_pass https://minio_s3; # This uses the upstream directive definition to load balance
   }

   location /minio/ui/ {
      rewrite ^/minio/ui/(.*) /$1 break;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;

      # This is necessary to pass the correct IP to be hashed
      real_ip_header X-Real-IP;

      proxy_connect_timeout 300;

      # To support websockets in MinIO versions released after January 2023
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      # Some environments may encounter CORS errors (Kubernetes + Nginx Ingress)
      # Uncomment the following line to set the Origin request to an empty string
      # proxy_set_header Origin '';

      chunked_transfer_encoding off;

      proxy_pass https://minio_console; # This uses the upstream directive definition to load balance
   }
}

upstream minio_s3 {
   least_conn;
   server minio-01.internal-domain.com:9000;
   server minio-02.internal-domain.com:9000;
   server minio-03.internal-domain.com:9000;
   server minio-04.internal-domain.com:9000;
}

upstream minio_console {
   least_conn;
   server minio-01.internal-domain.com:9001;
   server minio-02.internal-domain.com:9001;
   server minio-03.internal-domain.com:9001;
   server minio-04.internal-domain.com:9001;
}

server {
   listen       80;
   listen  [::]:80;
   server_name  minio.example.net;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      proxy_pass http://minio_s3; # This uses the upstream directive definition to load balance
   }
}

server {

   listen       80;
   listen  [::]:80;
   server_name  console.example.net;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;

      # This is necessary to pass the correct IP to be hashed
      real_ip_header X-Real-IP;

      proxy_connect_timeout 300;

      # To support websocket
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      chunked_transfer_encoding off;

      proxy_pass http://minio_console/; # This uses the upstream directive definition to load balance
   }
}

5 - Upload Files Using Pre-signed URLs

Using pre-signed URLs, a client can upload files directly to an S3-compatible cloud storage server (S3) without exposing the S3 credentials to the user.

This guide describes how to use the presignedPutObject API from the MinIO JavaScript Library to generate a pre-signed URL. This is demonstrated through a JavaScript example in which an Express Node.js server exposes an endpoint to generate a pre-signed URL and a client-side web application uploads a file to MinIO Server using that URL.

• Upload Files Using Pre-signed URLs

  • 1. Create the Server
  • 2. Create the Client-side Web Application

1. Create the Server

The server consists of an Express Node.js server that exposes an endpoint called /presignedUrl. This endpoint uses a Minio.Client object to generate a short-lived, pre-signed URL that can be used to upload a file to MinIO Server.

// In order to use the MinIO JavaScript API to generate the pre-signed URL, begin by instantiating
// a `Minio.Client` object and pass in the values for your server.
// The example below uses values for play.min.io:9000

const Minio = require('minio')

var client = new Minio.Client({
    endPoint: 'play.min.io',
    port: 9000,
    useSSL: true,
    accessKey: 'Q3AM3UQ867SPQQA43P2F',
    secretKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG'
})

// Instantiate an `express` server and expose an endpoint called `/presignedUrl` as a `GET` request that
// accepts a filename through a query parameter called `name`. For the implementation of this endpoint,
// invoke [`presignedPutObject`](https://github.com/minio/minio-js/blob/master/docs/API.md#presignedputobjectbucketname-objectname-expiry)
// on the `Minio.Client` instance to generate a pre-signed URL, and return that URL in the response:

// express is a small HTTP server wrapper, but this works with any HTTP server
const server = require('express')()

server.get('/presignedUrl', (req, res) => {
    client.presignedPutObject('uploads', req.query.name, (err, url) => {
        if (err) throw err
        res.end(url)
    })
})

server.get('/', (req, res) => {
    res.sendFile(__dirname + '/index.html');
})

server.listen(8080)

2. Create the Client-side Web Application

The client-side web application’s user interface contains a selector field that allows the user to select files for upload, as well as a button that invokes an onclick handler called upload:

<input type="file" id="selector" multiple>
<button onclick="upload()">Upload</button>

<div id="status">No uploads</div>

<script type="text/javascript">
  // `upload` iterates through all files selected and invokes a helper function called `retrieveNewURL`.
  function upload() {
        // Get selected files from the input element.
        var files = document.querySelector("#selector").files;
        for (var i = 0; i < files.length; i++) {
            var file = files[i];
            // Retrieve a URL from our server.
            retrieveNewURL(file, (file, url) => {
                // Upload the file to the server.
                uploadFile(file, url);
            });
        }
    }

    // `retrieveNewURL` accepts the name of the current file and invokes the `/presignedUrl` endpoint to
    // generate a pre-signed URL for use in uploading that file: 
    function retrieveNewURL(file, cb) {
        fetch(`/presignedUrl?name=${file.name}`).then((response) => {
            response.text().then((url) => {
                cb(file, url);
            });
        }).catch((e) => {
            console.error(e);
        });
    }

    // ``uploadFile` accepts the current filename and the pre-signed URL. It then uses `Fetch API`
    // to upload this file to S3 at `play.min.io:9000` using the URL:
    function uploadFile(file, url) {
        if (document.querySelector('#status').innerText === 'No uploads') {
            document.querySelector('#status').innerHTML = '';
        }
        fetch(url, {
            method: 'PUT',
            body: file
        }).then(() => {
            // If multiple files are uploaded, append upload status on the next line.
            document.querySelector('#status').innerHTML += `<br>Uploaded ${file.name}.`;
        }).catch((e) => {
            console.error(e);
        });
    }
</script>

Note: This uses the File API, QuerySelector API, fetch API & Promise API.

6 - Generate a Let’s Encrypt Certificate for Silo with Certbot

Let’s Encrypt is a new free, automated, and open source, Certificate Authority.

Certbot is a console based certificate generation tool for Let’s Encrypt.

In this recipe, we will generate a Let’s Encypt certificate using Certbot. This certificate will then be deployed for use in the MinIO server.

1. Prerequisites

• Install MinIO Server using the Red Hat Linux deployment guide.
• Install Certbot from the Certbot website.

2. Dependencies

• Port 443 for https needs to be open and available at time of executing certbot.
• Certbot needs root access while executing because only root is allowed to bind to any port below 1024.
• We will be using our own domain myminio.com as an example in this recipe. Replace with your own domain under your setup.

3. Recipe Steps

Step 1: Install Certbot

Install Certbot by following the documentation at https://certbot.eff.org/.

Step 2: Generate Let’s Encrypt cert

# certbot certonly --standalone -d myminio.com --staple-ocsp -m test@yourdomain.io --agree-tos

Step 3: Verify Certificates

List your certs saved in /etc/letsencrypt/live/myminio.com directory.

$ ls -l /etc/letsencrypt/live/myminio.com
total 4
lrwxrwxrwx 1 root root  37 Aug  2 09:58 cert.pem -> ../../archive/myminio.com/cert4.pem
lrwxrwxrwx 1 root root  38 Aug  2 09:58 chain.pem -> ../../archive/myminio.com/chain4.pem
lrwxrwxrwx 1 root root  42 Aug  2 09:58 fullchain.pem -> ../../archive/myminio.com/fullchain4.pem
lrwxrwxrwx 1 root root  40 Aug  2 09:58 privkey.pem -> ../../archive/myminio.com/privkey4.pem
-rw-r--r-- 1 root root 543 May 10 22:07 README

Step 4: Set up SSL on MinIO Server with the certificates.

The certificate and key generated via Certbot needs to be placed inside user’s home directory.

$ cp /etc/letsencrypt/live/myminio.com/fullchain.pem /home/user/.minio/certs/public.crt
$ cp /etc/letsencrypt/live/myminio.com/privkey.pem /home/user/.minio/certs/private.key

Step 5: Change ownership of certificates.

$ sudo chown user:user /home/user/.minio/certs/private.key
$ sudo chown user:user /home/user/.minio/certs/public.crt

Step 6: Start MinIO Server using HTTPS.

If you are not going to run MinIO with root privileges, you will need to give MinIO the capability of listening on ports less than 1024 using the following command:

sudo setcap 'cap_net_bind_service=+ep' ./minio

Now, you can start MinIO Server on port “443”.

$ ./minio server --address ":443" /mnt/data

If you are using dockerized version of MinIO then you would need to

$ sudo docker run -p 443:443 -v /home/user/.minio:/root/.minio/ -v /home/user/data:/data minio/minio server --address ":443" /data

Step 7: Visit https://myminio.com in the browser.