PGSTY SILO Blog

• 1: Posts
  • 1.1: MinIO Fork, Promise Kept
  • 1.2: MinIO Is Dead, Long Live MinIO
  • 1.3: MinIO Is Dead. Which Next?
  • 1.4: MinIO is Dead
• 2: Release Notes
  • 2.1: Silo Console 2.1.0 Released
  • 2.2: Silo Console 2.0.0 Released
  • 2.3: Silo Pkg 3.11.0 Released
  • 2.4: mcli 20260806 Released
  • 2.5: Silo 20260806 Released
  • 2.6: Silo 20260804 Released
  • 2.7: mcli 20260804 Released
  • 2.8: Silo 20260618 Released
  • 2.9: Silo 20260417 Released
  • 2.10: Silo 20260325 Released
  • 2.11: Silo 20260321 Released
  • 2.12: Silo 20260314 Released
  • 2.13: Silo 20260214 Released
  • 2.14: Silo 20251203 Released
• 3: SILO Security Chronicle
  • 3.1: CVE-2026-32285: The jsonparser Advisory That Required No Patch
  • 3.2: CVE-2026-33322: OIDC JWT Algorithm Confusion
  • 3.3: CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain
  • 3.4: CVE-2026-34204: Replication Metadata Injection
  • 3.5: CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass
  • 3.6: CVE-2026-40344: Snowball Auto-Extract Authentication Bypass
  • 3.7: CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass
  • 3.8: CVE-2026-42600: ReadMultiple Storage-REST Path Traversal
  • 3.9: Internode Path Containment Audit: Paying Off What CVE-2026-42600 Left Owing
  • 3.10: The Parser Knew, the Schema Didn't: Config Keys That Could Take Every Notification Down
  • 3.11: Object Grant, Bucket Reach: When 'bucket/*' Could Rewrite the Bucket Itself
  • 3.12: Absent Is Not Empty: A Blank versionid and the Fail-Open It Invites
  • 3.13: Three Headers, One Lie: Making the Client Source Address Mean Something
  • 3.14: Sorted Is Not Increasing: How One Duplicate Part Number Doubled an Object
• 4: Design Records
  • 4.1: CopyObject Checksums Must Cover Logical Object Bytes
  • 4.2: DSN-Only Database Notifications: A Compatibility Boundary for #53
  • 4.3: Preview Text, Never Execute It: SILO Console Text Preview PRD
  • 4.4: When the Total Is Unknown: Folder Download Progress

1 - Posts

Essays and analysis about MinIO, S3-compatible object storage, and the SILO community fork.

1.1 - MinIO Fork, Promise Kept

Two months ago in “MinIO is Dead, Long Live MinIO,” I promised I’d keep the MinIO fork patched. The recurring objection on HN is fair: can one person actually maintain something like this? The real answer isn’t clicking fork. It’s what happens when CVEs start landing.

Between April 15 and 17, pgsty/minio shipped RELEASE.2026-04-17, closing four CVEs and a handful of related vulnerabilities disclosed in the same window.

The scope I committed to originally was narrow: no new features, keep the supply chain running, handle reproducible bugs and security issues as they come in. This release is what it looks like when that promise gets tested.

What happened upstream

In December 2025, MinIO moved the open-source repo to maintenance mode. The README said security fixes would be “evaluated case by case.” In February 2026, the repository was archived and the landing page became “this repository is no longer maintained.”

The SECURITY.md in that same archived repo still says: “we will always provide security updates for the latest release.”

Over the past month, four high-severity and two medium-severity vulnerabilities have been disclosed against the final open-source release.

It’s been 184 days since the last upstream release. Vulnerabilities get disclosed; fixes ship only in the commercial build. The guidance for OSS users is a single line: upgrade to AIStor.

AIStor starts around $100k/year for 400 TiB — roughly S3 pricing, for software you install and operate yourself.

It’s a clean arrangement: archive the repo so there’s no obligation to patch, keep publishing CVE advisories for visibility, and route everyone who reads them toward the commercial product.

Someone still has to patch the old one.

What this release fixes

Full write-ups, CVSS arithmetic, and PoCs are in the release notes. The short version:

• CVE-2026-33322 (OIDC JWT algorithm confusion, CVSS 9.8): under certain IdP configurations, an attacker who knows the OIDC ClientSecret can mint a token claiming any identity — including consoleAdmin — and MinIO will accept it. Vulnerable window: November 2022 through March 2026. About three and a half years.
• CVE-2026-33419 (LDAP STS enumeration and brute-force): the login endpoint leaks which usernames are real, and there’s no rate limiting on the subsequent password guessing. End of the chain is an STS credential.
• CVE-2026-34204 (replication-header metadata injection): a regular PUT or COPY with certain X-Minio-Replication-* headers can write an object into a permanently unreadable state. The data is still on disk; you just can’t read it back out.
• CVE-2026-39414 (S3 Select memory exhaustion): one request, one OOM.
• GHSA-hv4r-mvr4-25vw / GHSA-9c4q-hq6p-c237: two signature-verification bypasses on the unsigned-trailer path. Anonymous or forged-signature requests can successfully write objects on certain routes.

Plus the usual dependency cleanup from go-jose, go.opentelemetry.io, and the Go 1.26.2 upgrade itself — about twenty security items in total counting transitive dependencies.

How it got fixed

I said in the earlier post that I’d rely on AI coding agents, and that’s how this round went. My role was closer to “review and decide” than “write code.”

Per-issue flow, roughly:

1. Codex drafts first. Given the CVE description and relevant code paths, it produces an initial patch.
2. Claude Code reviews adversarially. Picks holes from the attacker’s side.
3. Back to Codex. If it agrees with Claude Code’s critique, it reworks. If not, it has to write out why. No silent overrides.
4. Another round of review by Claude Code, with both sides’ reasoning on the table. Iterate until they converge.
5. Tests. Codex proposes cases, Claude Code adds more, Codex runs them, Claude Code reviews the results.
6. I decide. Read the diff, run the tests, merge or send it back with comments.

I didn’t write any of the code in this round. My job was to define the problem, set constraints, pick between approaches, read diffs, run tests, and merge. The GitHub log shows Vonng, Codex, and Claude Code as co-authors — that’s just who did the work.

A few things I noticed about how this runs in practice.

Two heterogeneous agents in opposition catch more than one agent alone. A single agent patching a security bug tends toward confident-sounding fixes that quietly miss a boundary condition. Having a second agent argue against the first filters out most of those.

It forces the tradeoffs into writing. When two implementations diverge, someone has to say why A over B. That exchange is the thing I can actually act on as the person deciding what to merge.

Real maintenance is patch-on-patch, not one-shot. The LDAP STS fix is a good example. The first version landed, and then we realized: successful requests shouldn’t count against the rate limit; X-Forwarded-For shouldn’t be trusted by default; the limiter should key on source IP plus normalized username, not just one. Three follow-up commits before it settled. Iterating through that by hand would have cost a lot more time.

Why this fork exists

Because I use MinIO myself.

MinIO is a production dependency for Pigsty. I need working binaries, a complete console, packages that keep shipping, and someone actually handling CVEs. That keeps the scope narrow. No new features, no turning the repo into a playground. Compatibility, supply chain, fixes when they’re needed.

— Chainguard also ships MinIO container images that track upstream’s post-archive commits, a useful option if you use their images. This fork is a different shape: source tree, RPM/DEB packages, restored console, and doesn’t depend on upstream continuing to push patches somewhere.

The fork is at about 1,300 stars on GitHub and 50,000+ pulls on Docker Hub now. Not remarkable numbers, but enough to tell me I’m not the only one who needed this fork to keep shipping.

If you’re already running OSS MinIO, migration is cheap:

• Docker: swap minio/minio for pgsty/minio.
• RPM / DEB: on GitHub Releases, or via pig.
• Source: pgsty/minio
• Docs: silo.pgsty.com

You don’t need to replace anything around it or relearn the API. In most cases, you’re just pointing a compatible binary at the same deployment. If you want a full HA production setup, Pigsty ships one for free.

Something I use broke; I’m fixing it.

What’s different in 2026 is the cost of “I’m fixing it.” With two coding agents and someone to referee between them, the maintenance load of a mid-sized Go codebase is tractable for one person in a way it wasn’t a year or two ago. That’s about it — not a grand theory about open-source resilience, just the current operating point.

If you’re running OSS MinIO, the migration is cheap and the patches are current. If another CVE drops, I’ll still be here.

• MinIO Is Dead, Long Live MinIO
• From AGPL to Apache: Reflections on Pigsty’s License Change
• Originally published in Chinese

1.2 - MinIO Is Dead, Long Live MinIO

MinIO’s open-source repo has been officially archived. No more maintenance. End of an era — but open source doesn’t die that easily.

I created a MinIO fork, restored the admin console, rebuilt the binary distribution pipeline, and brought it back to life.

If you’re running MinIO, swap minio/minio for pgsty/minio. Everything else stays the same. (CVE fixed, and the console GUI is back)

The Death Certificate

On December 3, 2025, MinIO announced “maintenance mode” on GitHub. I wrote about it in MinIO Is Dead.

On February 12, 2026, MinIO updated the repo status from “maintenance mode” to “no longer maintained”, then officially archived the repository. Read-only. No PRs, no issues, no contributions accepted. A project with 60k stars and over a billion Docker pulls became a digital tombstone.

If December was the clinical death, this February commit was the death certificate.

Today (Feb 14), a widely circulated article titled How MinIO went from open source darling to cautionary tale laid out the full timeline.

Percona founder Peter Zaitsev also raised concerns about open-source infrastructure sustainability on LinkedIn. The consensus in the international community is clear:

MinIO is done.

Looking back at the timeline over the past years, this wasn’t a sudden death. It was a slow, deliberate wind-down:

A company that raised $126M at a billion-dollar valuation spent five years methodically dismantling the open-source ecosystem it built.

But Open Source Endures

Normally this is where the story ends — a collective sigh, and everyone moves on.

But I want to tell a different story. Not an obituary — a resurrection.

MinIO Inc. can archive a repo, but they can’t archive the rights that the AGPL grants to the community.

Ironically, AGPL was MinIO’s own choice. They switched from Apache 2.0 to AGPL to use it as leverage in their disputes with Nutanix and Weka — keeping the “open source” label while adding enforcement teeth. But open-source licenses cut both ways — the same license now guarantees the community’s right to fork.

Once code is released under AGPL, the license is irrevocable. You can set a repo to read-only, but you can’t claw back a granted license. That’s the beauty of open-source licensing by design: a company can abandon a project, but it can’t take the code with it.

So — MinIO is dead, but MinIO can live again.

That said, forking is the easy part. Anyone can click the Fork button. The real question isn’t “can we fork it” but “can someone actually maintain it as a production component?”

Why would I do that?

I didn’t set out to take this on. But after MinIO entered maintenance mode, I waited a couple of weeks for someone in the community to step up.

But I didn’t find one. So I did it myself.

Some background: I maintain Pigsty — a batteries-included PostgreSQL distribution with 460+ extensions, cross-built for 14 Linux distros. I also maintain build pipelines for 290 PG extensions, several PG forks, and dozens of Go Projects (Victoria, Prometheus, etc.) packaging across all major platforms. Adding one more to the pipeline was a piece of cake.

I’m not new to MinIO either. Back in 2018, we ran an internal MinIO fork at TanTan (back when it was still Apache 2.0), managing ~25 PB of data — one of the earliest and largest MinIO deployments in China at the time.

More importantly, MinIO is an optional module in Pigsty. Many users run it as the default backup repository for PostgreSQL in production. We did consider several alternatives, but none were a drop-in replacement for MinIO-based workflows.

We use MinIO ourselves, so keeping the supply chain alive was not optional — it had to be done. As early as December 2025, when MinIO announced maintenance mode, I had already built CVE-patched binaries and switched to them.

pgsty/minio RELEASE.2025-12-03T12-00-00Z

What We’ve Done

As of today, three things.

1. Restored the Admin Console

This was the change that frustrated the community the most.

In May 2025, MinIO stripped the full admin console from the community edition, leaving behind a bare-bones object browser. User management, bucket policies, access control, lifecycle management — all gone overnight. Want them back? Pay for the enterprise edition. (~$100,000)

We brought it back.

The ironic part: this didn’t even require reverse engineering. You just revert the minio/console submodule to the previous version. They swapped a dependency version to replace the full console with a stripped-down one. The code was always there.

We put it back.

2. Rebuilt Binary Distribution

In October 2025, MinIO stopped distributing pre-built binaries and Docker images, leaving only source code. “Use go install to build it yourself” — that was their answer.

For the vast majority of users, the value of open-source software isn’t just a copy of the source — supply chain stability is what matters. You need a stable artifact you can put in a Dockerfile, an Ansible playbook, or a CI/CD pipeline — not a requirement to install a Go compiler before every deployment.

We rebuilt the distribution:

Docker Images

pgsty/minio is live on Docker Hub. docker pull pgsty/minio and you’re good.

RPM / DEB Packages

Built for major Linux distributions, matching the original package specs.

CI/CD Pipeline

Fully automated build workflows on GitHub, ensuring ongoing supply chain stability.

If you’re using Docker, just swap minio/minio for pgsty/minio.

For native Linux installs, grab RPM/DEB packages from the GitHub Release page. You can also use pig (the PG extension package manager) for easy installation, or configure the pigsty-infra APT/DNF repo to install from it:

curl https://repo.pigsty.io/pig | bash; 
pig repo add infra -u; pig install minio

Just works as usual.

3. Restored Community Edition Docs

MinIO’s official documentation was also at risk — links had started redirecting to their commercial product, AIStor.

We forked minio/docs, fixed broken links, restored removed console documentation, and deployed it as the SILO documentation site.

The docs use the same CC Attribution 4.0 license as the original, with necessary maintenance.

Commitments

Some things worth stating up front to set expectations.

No New Features — Just Supply Chain Continuity

MinIO as an S3-compatible object store is already feature-complete. It’s a finished software. It doesn’t need more bells and whistles — it needs a stable, reliable, continuously available build. (I already have PostgreSQL for these, so I don’t need something like S3 table or S3 vector. A stable S3 core is all I need)

What we’re doing: making sure you can get a working, complete MinIO binary, with the admin console included and CVE fixed. RPM, DEB, Docker images — built automatically via CI/CD, drop-in compatible with your existing minio. We keep the existing minio naming and behavior where legally and technically feasible.

This Is a Production Build, Not an Archive

We run these builds ourselves and have been dogfooding them in production for three months. If something breaks, we detect it early and patch it quickly.

I build this primarily for Pigsty and our own usage, but I hope it helps others too.

I’m willing to Track CVEs and Fix Bugs

If you run into issues, feel free to report them at pgsty/minio. I’ll do my best to fix these — but please don’t treat this as a commercial SLA.

Given that AI coding tools have made bug fixing dramatically cheaper, and that we’re explicitly not adding any new features, I believe the maintenance workload is manageable. (how often do you see one?)

Trademark Is Tricky, But We’ll Cross That Bridge When We Come to It

Disclaimer

Trademark Notice: MinIO® is a registered trademark of MinIO, Inc. This project (pgsty/minio) is an independently maintained community fork under the AGPL license. It has no affiliation with, endorsement by, or connection to MinIO, Inc. Use of “MinIO” in this post refers solely to the open-source software project itself and implies no commercial association.

AGPLv3 gives us clear rights to fork and distribute, but trademark law is a separate domain. We’ve marked this clearly everywhere as an independent community-maintained build.

If MinIO Inc. raises trademark concerns, we’ll cooperate and rename (probably something like silo or stow). Until then, we think descriptive use of the original name in an AGPL fork is reasonable — and renaming all the minio references doesn’t serve users.

AI Changed the Game

You might ask: can one person really maintain this?

It’s 2026. Things are different now.

AI coding tools are changing the economics of open-source maintenance.

With tools like Claude Code & Codex, the cost of locating and fixing bugs in a complex Go project has dropped by more than an order of magnitude. What used to require a dedicated team to maintain a complex infra project can now be handled by one experienced engineer with an AI copilot.

Maintaining a MinIO build without adding new features is a manageable task. The key requirement is testing and validation. and we already have that scenario, which lets us verify compatibility, reliability, and security in practice.

Consider: Elon cut X/Twitter’s engineering team down to ~30 people and the system still runs. Maintaining a MinIO fork without new features is considerably less daunting

Just Fork It

MinIO Inc. can archive a GitHub repo, but they can’t archive the demand behind 60k stars, or the dependency graph behind a billion Docker pulls. That demand doesn’t disappear — it just finds its way out.

HashiCorp’s Terraform got forked into OpenTofu, and it’s doing fine. MinIO’s situation is actually more favorable — AGPL is more permissive for forks than BSL, with no legal gray area for community forks. A company can abandon a project, but open-source licenses are specifically designed so the code can’t die.

Fork is the most powerful spell in open source. When a company decides to shut the door, the community only needs two words:

Fork it.

Reference

• MinIO Is Dead
• MinIO Is Dead, Are There Alternatives?
• From AGPL to Apache: Reflections on Pigsty’s License Change
• MinIO: Promise made, Promise kept

Originally published in Chinese

1.3 - MinIO Is Dead. Which Next?

MinIO announced maintenance mode two days ago. I ranted in “MinIO Is Dead” and immediately got flooded with “so what now?”

The usual suspects: Ceph, RustFS, SeaweedFS, Garage. I packaged all of them for Linux (RPM/DEB) and ran them through the grinder.

Short version: there’s no perfect substitute. Ceph is powerful but overkill; SeaweedFS rocks tiny files but needs an external metadata DB; Garage is cute but too barebones; RustFS targets the MinIO niche but is still alpha.

Quick scan of the field

MinIO is the open-source S3 clone. If all you need is basic object CRUD, any S3-compatible store works. But parity with MinIO means more than APIs—it’s about reliability, operability, tooling, documentation, SOPs. Replacing it cleanly is hard.

Ignoring commercial clouds, here’s the OSS menu:

• Ceph – arguably the best choice for enterprises, but brutally complex. Most folks don’t need block + file + object in one, and it requires extras like Podmon. MinIO’s single binary spoiled us.
• SeaweedFS – optimized for oceans of small files; O(1) disk seeks make it absurdly fast there. But it relies on an external metadata store. If you want a general-purpose object store, that dependency is annoying.
• Garage – built by Deuxfleurs with NGI funding. Delightfully light (10 MB), great for self-hosters and edge nodes. But S3 compatibility is thin: no versioning, no cross-region replication, no IAM. Enterprises will laugh.
• RustFS – the only project explicitly chasing “drop-in MinIO,” but it’s still alpha.

RustFS vs. MinIO

RustFS looked the most promising, so I wired it into Pigsty as a MinIO replacement. Most logic carried over, but a few differences popped up:

• Certificates must follow specific naming rules.
• Health checks differ from MinIO’s endpoints.
• mc admin doesn’t work; you can’t push fine-grained IAM policies. That’s a deal-breaker for many teams.

It ran, but I’m not shipping alpha software into production, so I shelved the branch. I’ll revisit when RustFS hits GA.

Will RustFS repeat MinIO’s mistakes?

RustFS has potential, but I worry it’ll retrace MinIO’s path. I asked the AI big three (GPT‑5 Pro, Claude 4 Opus, Gemini 3 Pro) to audit the project. Gemini leveled some serious accusations; Claude corroborated.

The red flags match MinIO’s history: Apache 2.0 license + copyright assignment CLA + single commercial gatekeeper. With that risk profile, I’m downgrading RustFS from “optimistic” to “cautious wait-and-see.”

So what now?

Pigsty bundles MinIO as an optional module for PostgreSQL backups or as an on-prem S3 for apps like Supabase. After surveying the alternatives, I’m not eager to swap it out. I might add a pgBackRest-native backup server option, but ripping out MinIO today feels premature.

Best plan: stay on the latest MinIO release, lock the version, isolate it on the network, and wait a few months. Maybe the community forks it; maybe RustFS matures. Adjust when reality changes.

RustFS still has a golden window to seize MinIO’s niche with a safer, community-friendly fork. That window is measured in months, not years.

If you stick with MinIO

Use the latest build, not the April 22, 2025 edition with the GUI. There’s a serious CVE in the interim:

• CVE-2025-62506 – privilege escalation via session-policy bypass (HIGH). Low-privilege users can mint new accounts and escalate.

In a locked-down intranet the risk is manageable, but you still want the fix, which landed in the 2025‑10‑15 release. MinIO pulled the prebuilt binaries starting with that version, offering source only. Annoying, but it’s Go—go build and you’re done. I forked MinIO, ran their packager, and produced RPM/DEBs for 2025‑12‑03 so I’m not deploying vulnerable bits: https://github.com/pgsty/minio

Security patches still need humans. MinIO claims they’ll fix critical issues, but if the community wants a maintained fork, now’s the moment. Start from 2025‑04‑22, cherry-pick critical bug/security fixes, and keep a community LTS alive.

MinIO is “done” software. It doesn’t need the latest S3 gimmick (Vector/Table); it needs steady bugfixes. That’s perfect for a community branch. Plenty of storage vendors rely on MinIO; maintaining a fork beats writing a new object store from scratch.

2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio: pgsty/minio / Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.

1.4 - MinIO is Dead

December 3, 2025 was a day to mark in open-source software history. MinIO’s team updated the project status on GitHub, announcing the MinIO open-source project was entering “maintenance mode.” This basically declared the death of MinIO as an open-source project.

MinIO the company has finally completed its transformation from a dragon-slaying hero into the very dragon it once sought to slay.

From Dragon-Slayer to Dragon

Democratization Era (2014–2019): The Apache of Object Storage

MinIO was founded in 2014 with a highly idealistic vision – to be “the Apache of object storage.” In an era dominated by AWS S3, MinIO’s ultra-lightweight design (a single static binary) and 100% S3 API compatibility quickly won developers’ hearts.

During this phase, MinIO was licensed under the liberal Apache 2.0 license, encouraging developers to integrate it into all kinds of applications. Its core pitch: “turn any hardware into AWS S3.” This open strategy was wildly successful. MinIO claimed its Docker image had been pulled over 1 billion times, making it the world’s most widely deployed object storage service. At this point, MinIO was a darling of the cloud-native stack – the default storage backend in many Kubernetes setups.

License Weaponization (2019–2025): The AGPL War

The first major crack in community relations appeared around 2019–2021. MinIO announced it was changing its core license from Apache 2.0 to GNU AGPLv3.

The official explanation was that this move aimed to prevent cloud providers (like AWS, Azure) from “freeloading” the code and repackaging it as proprietary services — a common defensive tactic in open source. During this period, MinIO shifted from being a community guardian to an aggressive defender of its IP. In 2022, MinIO publicly accused Nutanix Objects of violating its license and revoked Nutanix’s right to use MinIO; in 2023, MinIO sued high-performance filesystem vendor Weka on similar grounds. These legal actions, though legally contentious, sent a clear signal: MinIO no longer welcomed commercial use without paying up. This set the legal and psychological stage for the full lockdown that would come in 2025.

Control Plane Neutered (May 2025)

In May 2025, MinIO decided to strip the MinIO Console out of the community edition. The console was a critical GUI for bucket management, IAM, monitoring, and audit logging. After this removal, the open-source MinIO was left with only a basic “object browser” GUI – essentially just a file viewer/downloader.

Meanwhile, key admin features like policy management, site replication configuration, and lifecycle management were moved entirely into the commercial enterprise edition. This change downgraded the open-source MinIO from a full-featured storage management system into a mere data-plane component, robbing it of the control-plane capabilities needed to run as a standalone product in production.

Cutting Off Binary Distribution (Oct 2025)

On October 15, 2025 – right as a critical security vulnerability (CVE-2025-10-15T17-29-55Z / GHSA-jjjj-jwhf-8rgr) was disclosed – MinIO stopped publishing updated Docker images to Docker Hub and Quay.io. The timing of this move was highly strategic. By cutting off binaries during a major security incident, MinIO effectively used security as a bargaining chip.

This decision directly broke the automated deployment pipelines for countless users. Helm charts, Ansible playbooks, and Terraform scripts expecting minio/minio (or Bitnami’s minio) image suddenly failed to find updates. Auto-scaling groups trying to pull new nodes hung due to missing images. For teams without a Go build environment or an internal container registry, MinIO instantly became unusable.

Maintenance Mode (Dec 2025)

On December 3, 2025, MinIO, Inc. officially updated its channels and GitHub repo to announce that the open-source project is now in “maintenance mode.” The README stated that there will be no further feature additions or improvements, issues and PRs will no longer be reviewed, and even critical security fixes would be provided “as appropriate.” No more RPM/DEB packages or Docker images will be released. Essentially, anyone needing updates or support is advised to switch to the commercial AIStor product.

Technical Impact: Damage to the Open-Source Ecosystem

MinIO’s move to maintenance mode dealt an immediate and far-reaching blow to many tech stacks.

Broken CI/CD Pipelines and an Automation Crisis

Thousands of Helm charts, Ansible playbooks, and Terraform scripts depend on the minio/minio (or Bitnami’s minio) container image. With official images no longer published, third-party packagers like Bitnami — who can’t get a stable upstream release — also had to stop updates.

• Cascade effect: Deployments in fresh environments started failing outright. Auto-scaling groups, upon launching new instances, would hang or error out when the MinIO image couldn’t be pulled.
• Cost of fixes: Companies now have to rewrite their deployment scripts to point to a self-hosted image, and set up internal build pipelines to compile and package MinIO from source.

Security Vacuum: CVE Patches Go Private

The most lethal consequence of halting binary distribution is delayed security patches. In the October 2025 incident, for example, MinIO effectively withheld the patched binaries for the vulnerability.

• Risk exposure: Companies without dedicated security teams are forced to keep running older, vulnerable versions with known critical flaws.
• Compliance nightmare: For organizations under PCI-DSS, HIPAA, SOC2, etc., not being able to obtain vendor-signed security updates is a compliance disaster. Lacking official patches, they technically fall out of compliance.

Exponentially Higher Ops Complexity

Removing the UI wasn’t just a hit to user experience – it increased operational burden. Tasks that used to be a few clicks in the Console (configuring bucket policies, setting user permissions) now require ops engineers to master the mc CLI or hand-craft complex JSON policy docs. This raises the skill floor and makes MinIO far less friendly as a lightweight internal tool.

Underlying Reasons: Pressure from Capital and Commercialization

The driving force behind MinIO’s decisions is the logic of venture capital. By 2025, MinIO had raised a total of $126 million in funding. The most significant was a $103 million Series B in January 2022 led by Intel Capital, SoftBank Vision Fund II, and General Catalyst, which crowned MinIO a unicorn (valued over $1 billion).

In VC terms, a $1B valuation means the company must show a clear path to IPO — typically demanding $100M+ in Annual Recurring Revenue (ARR) and rapid growth. In Feb 2025, MinIO announced its ARR had grown 149% over the past two years businesswire.com. Impressive growth, but to live up to a sky-high valuation, organic conversion alone wasn’t enough.

Cutting off the free open-source offering is the most direct way to force a huge user base into paid customers.

In 2025, MinIO underwent a full rebrand and launched “MinIO AIStor,” styling itself as “the data backbone for enterprise AI.” Management recognized that general-purpose object storage (for backups, file servers, etc.) was a red-ocean market with thin margins, whereas generative AI’s appetite for high-throughput data (the exascale AI era) promised the next big surge. By tuning its product for AI workloads and focusing on Fortune 500 enterprises linkedin.com, MinIO essentially decided to cut loose its low-value open-source user base. The move to maintenance mode signaled MinIO’s official pivot from a broad open-source project into a vertical, high-end AI software vendor.

MinIO isn’t a garage hobby project by a few geeks anymore; it’s a company that took $126M in VC and is valued at over $1B. Backed by Intel Capital and SoftBank, once you take that money, your boss is no longer the users — it’s the investors. And what do investors want? ARR, growth, IPO. You tell them, “We have a billion Docker pulls!” and they’ll ask, “How many dimes did those pulls pay us?”

The reality is brutal. To the VCs, those small businesses and individual devs using free MinIO are low-value assets. They open issues and ask for support — consuming expensive engineer time, bandwidth, and servers — yet will never convert to paying customers. MinIO’s leadership knows their real cash cows are the Fortune 500 firms doing generative AI. The ones training GPT models or running self-driving pipelines need AIStor, ultra-high performance, and 24/7 enterprise SLAs.

So flipping the project into “maintenance mode” is essentially an asset carve-out. MinIO is cutting away the “dead weight” (free users) and concentrating on the milkable “cash cows” (enterprise AI clients). In business strategy this is called focus. To the investors, it’s being responsible. But from the perspective of open source, it’s simply betrayal.

Personal Reflections

I started using MinIO around 2018 (back when it was Apache-licensed). We built a few multi-petabyte object storage clusters for videos, images, backups — probably one of the largest MinIO deployments in China at the time. I wrote deployment/monitoring playbooks for MinIO (still open-sourced in Pigsty).

As an open-source startup founder, I can understand the motivation behind these moves. But as an open-source contributor and user — I also know many folks right now have one phrase in their minds: “I have never seen such shamelessness.”

An open-source license isn’t a shackle, but it is a social contract. Developers contribute code, users contribute testing, feedback, and reputation; together, they make a project successful. MinIO enjoyed a decade of community goodwill and parlayed the bragging rights of “#1 in global downloads” into venture funding. Then it turned around and told the very users who propped it up: “You free-riders, get lost.” This kind of move breaks the fundamental trust that open source is built on.

This “bait and switch” tactic is even more nauseating than a crypto rug pull. A rug pull only takes your money — MinIO is pulling the rug out from under the tech stacks of thousands of companies. Adopting a technology isn’t just picking up a binary; it’s buying into an ecosystem and a design philosophy. They got everyone onboard, let the switching costs pile up sky-high, and then suddenly kicked away the ladder. In fact, as open-source expert Tison thoroughly discussed in his article The Bait-and-Switch Open-Source Strategy, the core issue with this model is deception.

MinIO betrayed the community, so the community may abandon it as well. Alternatives like Garage, SeaweedFS, or the new RustFS are ready to step in.

If I have to sum up my feelings, I’d borrow a line from The Hitchhiker’s Guide to the Galaxy:

—— “So long, and thanks for all the fish.”

2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio: pgsty/minio / Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.

2 - Release Notes

Each published SILO version has its own page with the release date, major changes, security fixes, dependency updates, and related commits.

2.1 - Silo Console 2.1.0 Released

Published: 2026-08-06 · Version: v2.1.0 · Repository: pgsty/silo-console

SILO Console 2.1.0 is the first feature release after the independent 2.0.0. It does three things:

1. Speaks two languages — every console screen, help topic, and documentation link now renders in English or Chinese, behind a toggle on every page, with zero new runtime dependencies;
2. Reads the right metrics — the dashboard moves off the MinIO Metrics V2 names onto V3, with explicit handling for the semantics V3 changed underneath it;
3. Stops lying in edge cases — a select-all that matched what a bulk action would delete, placeholders that survive object names containing $&, timestamps that carry a timezone, and empty metrics that read “no data” instead of a fabricated 0.

This is a minor release. No environment variable, module path, API contract, binary name, or data layout changes. Upgrading is a binary or image swap.

A Bilingual Console

The console is an administration surface for an object store, and a large share of its operators read Chinese first. 2.1.0 makes the interface bilingual without importing an i18n framework — the embedded delivery model means every kilobyte is paid for in the binary. This is issue #6, which proposed i18next; the dependency-free substitution is the one deliberate deviation from it.

How it works

The design constraint was: no new dependency, no build step, no extraction pipeline, and partial coverage must never break the page.

• English source strings are the dictionary keys. t("Create Bucket") looks up the Chinese entry; a missing key returns the English string unchanged. Coverage can therefore grow incrementally, and a typo degrades to English rather than to a raw key like console.bucket.create.
• Three dictionaries, one merge. zh.ts (165 chrome entries), zhHelp.ts (247 help-topic entries), and zhScreens.ts (1,373 screen entries) merge with chrome taking precedence — about 1,785 entries in total.
• The language preference mirrors dark mode: localStorage → systemSlice → setLanguage. There is no browser-locale detection; the default is English, and the choice is explicit.
• Central interception points rather than per-callsite edits: the page-header wrapper, confirm dialogs, help items, route definitions, and the dashboard’s panel renderer each translate on the way out. This is why 220 screen files could be localized without touching their business logic.
• Module split matters. i18n/lang.ts holds pure primitives (translate, localizeUrl) and imports no store — systemSlice depends on it, so importing the store back would form a cycle. The hooks (useT, useLanguage, useLocalizedLink) and interpolate() live in i18n/index.tsx.

The toggle is a stroke-drawn 文/A icon mounted in the page header on every page and reused on the login page.

What it covers

Login and SSO flows, navigation and the command palette, the dashboard and every metrics panel, buckets and the full object browser (uploads, previews, sharing, versioning, rewind), users/groups/policies/access keys, configuration and event destinations, IDP and KMS, logs, health reports, speedtest, profiling, inspect, trace, watch, and the license page.

Beyond visible strings:

• Documentation links localize. silo.pgsty.com links gain a /zh prefix in Chinese; the Pigsty site swaps domains (pigsty.io ↔ pigsty.cc). GitHub, MinIO, AWS, and YouTube links are left alone.
• The help blog feed is per-language, fetching /zh/blog/index.xml in Chinese, with an independent cache per language.
• The command palette stays searchable in both languages. Menu entries translate for display but keep their English originals as keywords, so “桶” and “buckets” both match.
• Chart legends translate only their static prefix. translateLegend preserves instance suffixes like [server:drive], and the data layer keeps raw legends so components that match on them for arithmetic (capacity summing) keep working.
• Timestamps are unified, not merely translated — see below.

What it costs

Roughly +61 KB on the embedded payload (2.79 MB → 2.85 MB, +2.2%), zero new dependencies, and the dictionaries land in their own lazily-loaded chunk. The English rendering path is byte-stable: with the default language, output is identical to 2.0.0.

What stays English

Backend error strings (182 of them) are produced by the Go server and are not translatable from the frontend. A handful of strings hardcoded inside the vendored mds component library — the collapsed-menu “Sign Out” tooltip, and the data table’s “Columns”, “Loading…”, and ON/OFF toggles — remain English; two of them (“Sign Out”, “Actions:”) are swapped via a scoped CSS rule, but the rest would require patching the vendor.

Metrics V3 Migration

The dashboard queried MinIO Metrics V2 names. SILO deployments scrape V3 (/minio/metrics/v3), so the dashboard depended on an endpoint the monitoring pipeline no longer collected. 2.1.0 rewrites all 26 widgets onto the V3 catalog — 31 queries over 29 distinct metric names — and drops three widgets (51/61/62) that no layout ever referenced. This is issue #7; the Info-page half is #8.

The decision is V3-only: no runtime fallback, no probing, no version-selection knob. SILO Console targets SILO deployments, where the server, the scrape pipeline, and the console ship together. The SILO server keeps serving V2 endpoints for external consumers; the console simply stopped using them. A fallback would have been actively harmful — a metrics store retaining 15 days of V2 series would let an or-fallback silently read stale data.

The semantics V3 changed

Three properties of V3 break a naive name-for-name rewrite, and each needed a deliberate answer:

1. Cluster groups are exported identically by every node. /cluster/* metrics carry no server label and are not leader-gated, so an N-node scrape yields N duplicate series. Queries aggregate with max()/min() — never sum(), which would multiply cluster totals by the node count.
2. Zero values are not exported at all. Any metric whose value is ≤ 0 is skipped. Offline drive counts, healing-drive counts, and erasure-set health simply vanish rather than reporting 0, which a stat card renders as an empty panel. Every affected query carries a companion guard so the panel reads a real 0.
3. There is no minio_heal_* namespace. The V2 heal activity signal was in-memory anyway — it reset on restart and bumped on any scan. It is replaced by two cards with defensible semantics: Erasure Health (baselined on write quorum) and Usage Data Age (how stale the scanner’s usage snapshot is).

Zero-state semantics

An adversarial review of the migration produced eight findings, all fixed before release. They share one theme — the difference between zero, no data, and not yet scanned:

• Capacity free/used baselines on the always-present total, so a full cluster reads 0 free instead of vanishing.
• Online Drives is guarded against the all-offline case, where the zero-skip would erase the panel exactly when it matters most.
• Bucket and object counts guard on the usage group’s own freshness gauge, so a cluster that has not completed its first scan reads no data rather than a fabricated 0.
• Empty single-value results render as —, not 0.
• An empty size distribution no longer fabricates seven zero-height bins.
• Fractional rates stay visible (parseFloat axis domain, two-decimal CPU formatter) instead of collapsing to 0.
• Sub-second Usage Data Age clamps to “1 second” instead of rendering blank.

A regression suite (api/admin_info_metrics_test.go) now pins every widget query to the V3 catalog, asserts widget-ID uniqueness, and enforces the per-widget guard taxonomy: health and traffic widgets need a nodes-online companion, usage counts need the usage-group freshness companion, and capacity needs the total baseline. The full mapping is documented in docs/metrics-v3.md.

Also fixed

• Widget 17 queried sent_bytes twice and widget 11 queried syscall_read twice — both internode/syscall pairs were transposed into duplicates.
• Label-less matrices (the result of max() aggregation) serialize with no metric field at all, which crashed the frontend’s label extraction and produced a 0 B capacity donut and an empty usage-growth chart. Guarded.
• An unused per-widget Prometheus label-values prefetch stalled every widget request by up to a second. Deleted.
• The dashboard’s usage cards, chart controls, and dense Traffic/Resources panels were rebuilt on one grammar and now reflow through tablet widths.

Two server-side bugs were identified during this work and are tracked upstream rather than worked around here: minio_cluster_usage_buckets_since_last_update_seconds emits nanoseconds (the objects variant is correct), and V3 bucket-level sent/received traffic are transposed.

Correctness Fixes

Placeholders that survive real object names

String.prototype.replace interprets $&, $', $`, and $1 in the replacement value as directives. S3 keys legally contain $. So an object named report$&.csv did not render as itself — it re-injected the matched placeholder text into the output and corrupted the message. All 37 dictionary placeholder substitutions now pass the value through a function replacement, where no such interpretation happens. This was a latent bug in the original English UI, not something i18n introduced; the i18n audit is simply what found it.

A select-all that means what it shows

The vendored data table renders a plain untranslatable “Select” header whenever onSelectAll is absent — which was the case on all seven selectable tables. Worse, the naive fix is wrong: a select-all that replaces the whole selection drops rows hidden by an active filter, so the header checkbox and a subsequent bulk action can target different sets. The implementation toggles only the currently visible rows and preserves filter-hidden selections, so the header state can no longer imply a different set than the action would touch.

Timestamps with a timezone

Bucket, object, version, rewind, and access-key timestamps rendered as a mix of verbose English forms and — in several places — a 12-hour clock without AM/PM, which is simply ambiguous. All of them now render as yyyy-MM-dd HH:mm[:ss] (ZZZZ) in both languages.

A translation runtime that survives live data

t() also receives runtime strings: user agents, RSS titles, object names. Two hardening changes followed:

• misses return unchanged, unconditionally — the implicit @context suffix stripping is gone, because it silently mutated live data that happened to contain @;
• dictionary lookups are guarded with hasOwnProperty, so a hostile input naming an inherited Object.prototype member (constructor, toString) cannot leak a function into the UI.

Interaction and accessibility

• An expired session opening a deep link bounced through /login and back, accumulating a redirect chain instead of landing on the form once (#1).
• Collapsed sidebar buttons carried no accessible name; screen readers announced them as unlabelled (#4). Access Key inputs now declare their autocomplete intent instead of letting password managers guess (#5).
• Mobile metrics and bucket panels scroll instead of clipping (#3).
• The speedtest control row wraps instead of overflowing its card, its duration accepts seconds or minutes, and its size defaults to MiB to match its own unit list.
• Sidebar bucket rows use a virtual row pitch matching the 44px item, so selected and hovered highlights no longer overlap.
• Unit chips render the selected unit’s label rather than its raw value.

No SUBNET, No Telemetry

Upstream removed Subnet, Registration, and Call Home; this fork inherited that state but still carried three traces. 2.1.0 removes them:

• the health websocket’s subnetResponse field never addressed a subnet — it is a sentinel meaning “the report was assembled” — and is now reportStatus: "ok";
• two help topics claimed the health report “uploads automatically to SUBNET” and that inspect output is “transmitted to SILO SUBNET”. Neither was true. They now describe what happens: the report is generated on the deployment and downloaded by the browser;
• the unreferenced CONSOLE_SUBNET_PROXY constant is deleted.

For the record, 2.1.0’s outbound network posture is unchanged and remains: no analytics, no telemetry, no beacons, no external scripts or fonts. silo-console update is still disabled. The release catalog is contacted only if SILO_RELEASE_SERVICE_HOST (or RELEASE_SERVICE_HOST) is explicitly set — there is no default. The only automatic outbound request the browser makes is the help panel’s blog feed, and only after a user opens the Blog tab.

Upgrade Guide

There is nothing to migrate. No environment variable, module path, protocol field, systemd unit, binary name, or data layout changes between 2.0.0 and 2.1.0.

install -m 0755 silo-console-linux-amd64 /usr/local/bin/silo-console

Two things are worth knowing:

• The dashboard now requires Metrics V3. If your Prometheus scrapes only the V2 endpoints, dashboard panels will read no-data. Point the scrape at /minio/metrics/v3; Pigsty-managed deployments already do.
• The language default is English, chosen per browser and stored in localStorage. There is no server-side default and no browser-locale detection, so no existing deployment changes appearance on upgrade.

Verification Scope

Before tagging, the full change set was reviewed and the following gates were run against the final tree: go build, go vet, golangci-lint (0 issues), the Go unit suite across all packages, gofmt, TypeScript type checking, the frontend production build, Prettier across all sources, dictionary duplicate-key checks, and a debug-leftover scan of the complete diff.

The 29 intermediate commits were restructured into 20 logical ones by pure tree operations, and the rebuilt tip was verified byte-identical to the pre-rewrite tree. The embedded payload was rebuilt twice from a clean directory and confirmed byte-identical, which is the property the release pipeline’s zero-diff gate depends on. The pre-rewrite history is retained in a backup ref.

The Metrics V3 migration was additionally reviewed adversarially by an independent model, and all eight findings were fixed (see Zero-state semantics); its queries were validated against a live metrics store with real cluster data.

Known Limitations

• The SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle; the OIDC code paths are covered by unit tests.
• Backend error strings and several vendored mds component strings remain English (see What stays English).
• Chinese translation covers the console’s own surfaces; help-topic bodies are translated, but the documentation pages they link to follow the docs site’s own language coverage.
• Two server-side V3 metric bugs (nanosecond bucket-usage age, transposed bucket traffic) are tracked upstream and are not worked around in the console.
• Automatic self-update remains disabled; upgrades are explicit.

Issues Closed

2.1.0 closes every issue filed against 2.0.0. Each carries a comment on the tracker describing the fix, the commits, and the coverage added.

Three acceptance criteria are recorded as unmet rather than quietly ticked: web-app has no unit-test runner, so the i18n test suite (#6) and the focused constructLabelNames test (#2) would require introducing test tooling first, and #6’s contributor documentation for adding translation keys is not yet written.

Related Commits and Links

The complete v2.1.0 change set consists of 20 logical commits. The v2.1.0 tag additionally carries three later documentation commits that rewrote the repository README; they change no shipped behavior.

• 8764f5d — fix(web): stop recursive login redirects
• 437c56c — fix(ui): make the dashboard and bucket list usable on narrow screens
• 85fc0c6 — fix(a11y): name collapsed sidebar controls and credential fields
• e3fed07 — fix(metrics): rebuild dashboard cards, chart controls, and layout
• fa11576 — feat(login): polish controls and legal attribution
• 9fc17c1 — feat(i18n): add hand-rolled EN/ZH core, dictionaries, and language toggle
• 622c02e — feat(i18n): localize login, navigation, and the help system
• 6a03719 — feat(i18n): localize dashboard and metrics screens
• 14b1c2d — feat(i18n): localize bucket and object browser screens
• 0298062 — feat(i18n): localize identity, configuration, and event destinations
• 41094f6 — feat(i18n): localize observability, admin tools, and shared components
• e964992 — feat(metrics): migrate the dashboard to MinIO Metrics V3
• 0b2251f — fix(i18n): harden the translation runtime for live data and chart legends
• 9b60148 — fix(console): unify timestamps on a timezone-carrying standard format
• bf110ae — fix(console): give selectable tables a visible-rows select-all
• 5fc8f22 — fix(i18n): escape-proof all placeholder substitutions
• fef8fab — fix(console): polish speedtest, sidebar, and help chrome
• c4911e8 — chore(console): drop SUBNET remnants from health reporting
• 1d631c4 — docs: record the SILO Console v2.1.0 changelog
• 912d847 — build: regenerate optimized embedded web assets

Links:

• SILO Console source · Releases
• SILO website · Documentation
• Licensing · Attribution · Trademark

2.2 - Silo Console 2.0.0 Released

Published: 2026-08-04 · Version: v2.0.0 · Repository: pgsty/silo-console

SILO Console 2.0.0 is the first major release of this object-storage administration console as an independent project. Continuing from the georgmangold/console v1.9.1 maintenance line, it accomplishes three things:

1. An independent identity — product name, visual system, documentation entry points, source attribution, and the release pipeline all move into the SILO project, while the Go module path, environment variables, and other compatibility contracts are deliberately retained;
2. A redesigned interface — the login page, theme system, dashboard, and console details are reworked under one design language, backed by a regenerated brand icon set;
3. Hardened engineering — the embedded frontend payload shrinks from roughly 10MB to 3.5MB, known dependency vulnerabilities drop to zero, and a batch of inherited defects — including a real runtime data race — is fixed.

Before publication this release went through two independent review passes: a full code review with commit-history restructuring, followed by an adversarial re-verification (exhaustive asset validation, HTTP semantics probing, full routing regression, and smoke tests against the published artifacts themselves).

Why 2.0.0

This console originated as MinIO Console and was carried forward by the Alevsk/console and georgmangold/console community maintenance lines. SILO Console continues from there, maintained by the Pigsty community as the browser-based administration interface for SILO.

The version jumps from v1.9.1 to v2.0.0 because these public contracts change together:

• the product is now uniformly SILO Console, with the primary repository at pgsty/silo-console;
• the release binary changes from console to silo-console, and the container image moves to ghcr.io/pgsty/silo-console;
• release assets, checksums, package metadata, CLI descriptions, and project links all switch to SILO;
• in-product identity, help entry points, copyright attribution, source offers, and trademark notices are re-established.

The migration strategy is “clear external identity, restrained internal compatibility”: operators must take notice, but the underlying compatibility interfaces are not mechanically renamed.

Naming and Delivery Contracts

CLI authorship, usage text, and project descriptions now identify Pigsty and SILO Console. DEB/RPM/APK vendor, maintainer, homepage, description, and license metadata are updated accordingly; the executable installs to /usr/local/bin/silo-console.

Deliberately Retained Compatibility Identifiers

The following names still contain minio or the old console, but they are interface, protocol, or installation compatibility layers — not leftover branding:

Upgrade scripts therefore must not run repository-wide minio → silo or console → silo-console replacements. Migrating these compatibility interfaces in the future will require aliases, deprecation windows, and an explicit dual-read strategy; 2.0.0 does none of that.

A Redesigned Interface

2.0.0 is not a logo swap — the interface was redesigned end to end.

Login page

The login page is rewritten from scratch. The left brand panel renders a slowly drifting sine-mesh animation generated purely on Canvas (zero external dependencies, honors prefers-reduced-motion, pauses in background tabs), states the project’s proposition — “Keep the S3 Interface / Own the Object Store” — and keeps the full MinIO trademark notice at the bottom. The right-hand form is functionally untouched, preserving every existing automation selector. The Chakra Petch typeface used by the SILO wordmark ships as a ~20KB locally bundled subset with no external requests.

A unified theme system

All console colors converge into one light/dark theme layer: neutral greys for text and borders, the brand steel blue for primary actions and selection, and a sidebar that uses the same night palette as the login panel in both modes. Controls and cards share consistent radii and transitions, inputs get a keyboard focus ring, and modals animate in (also honoring reduced motion). Server-provided customStyles keep full precedence.

Console polish

• Dashboard (Metrics): stat cards rebuilt under one grammar — muted labels, tabular numerals, aligned status dots; charts and info strips are theme-driven; the upstream absolute-positioning layout is gone.
• Unified empty states: placeholder text in Watch, Trace, bucket Events/Replication/Lifecycle, and every other data panel is now centered and de-emphasized instead of raw top-left text.
• Vertical tabs: detail-page tabs change from bordered grey blocks to a quiet pill list, eliminating the stray empty cell at the bottom of the rail.
• License page: a new VERSION section shows both the connected server’s release and the Console’s own version; accounts without admin:ServerInfo never issue the request and the row stays hidden. The page also consolidates AGPLv3 licensing, the AGPL section-13 source offer, lineage, and trademark boundaries.
• A batch of interaction fixes: the sidebar now collapses on initial load at mobile widths (previously it waited for a resize event); the bottom navigation no longer lags window-height changes; the bucket accordion highlight spans the full row; the dashboard no longer overflows horizontally on narrow screens; and the help panel is now truly lazy — the login page makes no external requests at all.

Brand icon set

The favicon, PWA, and Apple Touch icons still carried a previous-generation hand-drawn emblem. 2.0.0 re-rasterizes every size (ico 16+32, favicon 16/32/96, apple 180, manifest 192/512) from the official silo.svg vector emblem, with safe-area margins on home-screen sizes, and trims the Web App Manifest to the modern icon set, dropping the 2014-era legacy density entries. The icon payload drops from 473KB to 160KB, and the browser tab icon finally matches the in-product brand.

Smaller and Faster

Embedded delivery is this console’s core form factor — the frontend ships inside the binary via go:embed. 2.0.0 optimizes that path systematically:

• Embedded payload: ~9.6MB → 3.5MB. Text assets (JS/CSS/SVG/…) are precompressed at build time with deterministic gzip and embedded compressed-only; legacy WOFF fonts (~1.25MB that no supported browser ever downloads) and a set of entirely unreferenced orphan images are removed.
• First-load transfer: ~5.7MB → ~1.7MB. Static assets previously shipped uncompressed on the wire; they are now emitted directly with Content-Encoding: gzip at zero runtime cost, with on-the-fly decompression for the rare client that does not accept gzip.
• Correct HTTP semantics. Accept-Encoding is parsed with full RFC 9110 q-values (gzip;q=0 gets identity bytes), responses carry Vary: Accept-Encoding, and non-GET/HEAD requests to static paths and the SPA entry receive 405 with an Allow header.
• Reproducible builds. Compression uses a pure-JS implementation (fflate) for byte-identical output across platforms, and the release pipeline enforces a hard gate: rebuilding the embedded assets in a clean environment must produce zero diff against the commit.

Release binaries (all frontend assets included, stripped) weigh roughly 35–40MB; for the downstream SILO server, embedding this console now costs about 3.5MB instead of about 10MB.

Security and Dependencies

Go: the build baseline moves to Go 1.26.5 and the golang.org/x family is fully refreshed. Every reachable vulnerability reported by govulncheck is resolved:

The single remaining advisory sits in golang.org/x/crypto, has no upstream fix yet, and is unreachable from this codebase; it is tracked as a known item.

Frontend: the full dependency-tree audit (production and tooling) is clean, covering the high-severity form-data CRLF injection and the DOMPurify and qs advisories; React Router is migrated to 7.18.2 (keeping the v6-compatible declarative API, with full routing regression). The only explicitly ignored advisory affects an unstable API this project does not use.

Runtime correctness: a real data race between HTTP log-target initialization and shutdown is fixed, along with shared-mock races in the test suite; supported Go packages pass -race across the board. As a side benefit, the go-m1cpu upgrade fixes the local go run cgo crash on recent macOS.

Update Checks and Default Network Behavior

This release keeps conservative defaults for upgrade tooling:

• automatic self-update in silo-console update is disabled — the command prints guidance and never downloads or replaces the binary;
• the release catalog gains SILO_RELEASE_SERVICE_HOST, with the previous RELEASE_SERVICE_HOST as a compatibility fallback; with neither set, no remote release service is contacted;
• the help panel’s blog content loads only when opened, and its links accept https://silo.pgsty.com exclusively.

Automatic updates will be reconsidered once signed release assets and a tested rollback path are in place.

Release Artifacts and Platform Matrix

The release ships 16 assets:

The pipeline triggers on tag pushes, pins third-party Actions to commit SHAs, and enforces the clean-checkout and zero-diff asset-rebuild gates before GoReleaser runs.

Upgrade Guide

Standalone binary

install -m 0755 silo-console-linux-amd64 /usr/local/bin/silo-console
/usr/local/bin/silo-console server

When building from source, make console still produces ./console; install it under the release name before wiring it into a production service.

DEB/RPM/APK and systemd

Packages continue to install /etc/systemd/system/minio-console.service, whose unit starts /usr/local/bin/silo-console. EnvironmentFile=/etc/default/console, console-user, and existing CONSOLE_* variables are unchanged. This retention lets package upgrades keep acting on the existing service instead of creating a parallel one.

Configuration and integrations

• do not rename CONSOLE_MINIO_SERVER or CONSOLE_MINIO_REGION;
• do not touch github.com/minio/console in Go imports;
• prefer SILO_RELEASE_SERVICE_HOST for self-hosted release catalogs;
• replace any reliance on console update with explicit download, verification, and deployment;
• update process-path-based monitoring to /usr/local/bin/silo-console.

This release does not change the object data layout and requires no bucket or object migration.

Dual Review and Validation Scope

2.0.0 went through two independent review passes before publication. The first pass performed a full code review, fixed the defects described above, restructured 13 intermediate commits into 8 logical ones, and ran Go -race across supported packages, go vet, golangci-lint, govulncheck, frontend type checks, production builds, Prettier, dead-code checks, and the full dependency audit. The second, adversarial pass independently re-ran the core gates and added:

• all 184 embedded files fetched three ways each (gzip client, identity client, HEAD) with per-file hash comparison against the embedded sources;
• RFC semantics probes (including combined q-values such as gzip;q=0, *;q=0.5), method restrictions, the OIDC callback, and SPA deep links;
• full React Router 7 regression: deep links, client-side navigation, bucket-detail tab switching, and browser history back;
• mobile first-load sidebar behavior, login-page external-request monitoring, and light/dark full-site tours;
• downloaded release assets verified byte-for-byte against checksums, binary self-reported version confirmed, and a smoke test of the published binary against a live server;
• zero-diff asset rebuilds confirmed on both macOS and Linux.

The complete pre-rewrite history is preserved in backup refs for rollback.

Known Limitations

• automatic self-update is disabled; upgrades are explicit;
• the SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle (the OIDC code paths are covered by unit tests and HTTP-level checks);
• one golang.org/x/crypto advisory has no upstream fix yet and is unreachable from this codebase;
• SILO does not yet maintain its own video library; videos in the help panel are clearly labeled upstream compatibility material;
• administrative features depend on the MinIO-compatible Admin API — SILO Console is not a generic browser for arbitrary S3 services;
• retained Go module paths, environment variables, protocol fields, and the systemd unit name still appear in code, configuration, and process listings.

Related Commits and Links

The complete v2.0.0 change set consists of 8 logical commits:

• 50797de — feat: establish SILO Console identity and compatibility
• 23ae6e8 — feat: redesign and harden the SILO Console web app
• 7a83a77 — build: update Go toolchain and dependencies
• 1330d25 — fix: eliminate logger shutdown and test mock races
• 06b3a34 — docs: publish the SILO Console v2.0.0 guide
• 4b24372 — build: regenerate optimized embedded web assets
• c38eb64 — ci: package and publish SILO Console v2 releases
• b952a12 — brand: regenerate the icon set from the official silo.svg emblem

Links:

• SILO Console source · Releases
• SILO website · Documentation
• Licensing · Attribution · Trademark

2.3 - Silo Pkg 3.11.0 Released

Release date: 2026-08-04 · Version: v3.11.0 · Commit: d8b1fa7 · Repository: pgsty/silo-pkg

This is the fork’s first pinned release. It restores the IAM bucket/object resource boundary reported as upstream minio/minio#20449: a policy condition-key bypass fix, three LDAP connection defects, a certificate watcher leak, a seeded-RNG defect, and the module’s real minimum Go version.

What This Repository Is

silo-pkg is a maintained fork of minio/pkg, carrying fixes needed by community MinIO forks that the now commercially driven upstream no longer accepts. The repository was renamed from pgsty/minio-pkg on 2026-08-02.

The module path intentionally remains unchanged as github.com/minio/pkg/v3. Existing import "github.com/minio/pkg/v3/..." statements do not change; only the right-hand side of the replace directive does:

replace github.com/minio/pkg/v3 => github.com/pgsty/silo-pkg/v3 v3.11.0

The /v3 suffix is the module’s major version, not a directory name, and must not be omitted. It is also why this release is numbered v3.11.0 rather than v4.0.0: Go requires the major version of a tag to match the major-version suffix declared in go.mod, so a v4.0.0 tag on a .../v3 module is rejected by the toolchain. Publishing a real v4 would mean changing the module path and rewriting roughly 395 import sites across the server, mc and Console — abandoning the drop-in property that is the point of keeping upstream’s path.

The IAM Bucket/Object Boundary

Every bucket-level S3 operation authorizes with an empty object name. The IAM matcher turned that into a resource string and, for the empty-object case, appended a trailing slash:

resource.WriteString(args.BucketName)
if args.ObjectName != "" {
    // "bucket/object"
} else {
    resource.WriteByte('/') // "bucket/"  <-- the defect
}

"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. A policy granting s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in this bucket” — therefore also authorized bucket-level actions. In a multi-tenant cluster, a tenant holding only that grant could call PutBucketPolicy and install {"Principal":"*"}, making the bucket publicly readable or writable, or grant itself bucket-level control. It could also delete the bucket outright, which is the reproduction in the upstream issue.

The bucket-policy evaluation path used for anonymous access never had this slash and was already reference-correct. Only the IAM path was wrong, in exactly one place.

Why not correct the whole boundary

Removing the slash for every bucket-level request is the obvious fix, and upstream tried it: the change was reverted the same day for breaking policies that relied on the old behaviour. Two properties make the full correction a migration rather than a patch.

It revokes grants real deployments depend on. It does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation and ListBucketMultipartUploads when granted through bucket/*. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the project that wrote the server writes it that way, production policies do too.

It cuts both directions. The matcher builds the same resource string for Allow and Deny, so removing the slash tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements. An administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection.

How the protected set was chosen

The scope was decided by one question: does reaching this action give the caller something its object-scoped grant does not already provide?

That question is the right one because of how the defect fires. Resource matching runs after action matching, so the bug only bites when the statement already grants the bucket-level action — which in practice means s3:*. The affected principal therefore already holds full read, write and delete over every object in the bucket. The useful question is not how dangerous an action sounds in the abstract, but what reaching it adds to a position that already includes all of the data.

Withheld from object-only grants (twelve actions):

Deliberately not withheld, and asserted by a test so that adding one is a deliberate act with a visible cost rather than an edit to a list:

• PutBucketTagging, PutBucketEncryption, PutBucketNotification. These are bucket-level writes and an earlier draft did withhold them. None gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handed s3:* on bucket/* and told the bucket is theirs may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release.
• CreateBucket. It targets a bucket that does not exist yet, so there is nothing to mutate or destroy, and provisioning flows commonly create a tenant’s bucket with that tenant’s own bucket/* credentials.
• The read/list family (ListBucket, GetBucketLocation, the configuration reads). Breaking these is what got upstream’s own attempt reverted. They wait for a migration-gated release.

Only Allow statements are affected. Deny statements keep the historical resource string, so no bucket lock is ever weakened, and NotResource exclusions keep their full reach.

Monotonicity, and the claim that was wrong twice

All of the above rests on one property: this change may remove permissions and must never add one. That property was asserted twice from reasoning rather than from tests, and was false both times. Recording how is more useful than recording only the final state.

The first attempt let the withheld slash reach the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected.

The second attempt fixed that and shipped saying the result was provably monotone. An independent adversarial review of that release produced a counterexample. Withholding the slash does not merely remove a match — it changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". A fixed-width wildcard is the clean case:

Allow s3:PutBucketPolicy on arn:aws:s3:::mybucke?

? matches exactly one character. Against the nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it does, so the hardening granted something the buggy matcher refused.

The fix is not another special case. On the protected path the matcher now requires both forms to match — the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction: there is no pattern it can newly satisfy, and no argument left to get wrong. mybucket* still grants (it matched both all along), mybucket/* is still withheld, and mybucke? is refused exactly as it always was.

Two lessons are worth carrying forward. A correctness fix in an authorization path must never make anything newly allowed — and the only way to know is to test both directions, because the reasoning felt airtight in both cases where it wasn’t. And when a security property is load-bearing, build it out of an operation that cannot violate it rather than out of a case analysis believed to be complete.

Evidence

The property is verified rather than argued. A decision corpus of 27,000 authorization outcomes — 15 resource patterns × 3 buckets × 5 object names × 20 actions × 6 statement forms — was generated against both the pre-hardening baseline and this release and compared entry by entry:

Every one of the 144 narrowed outcomes falls inside the design intent, with nothing outside it: exactly the twelve protected actions; only the three Allow statement forms, with zero transitions for Deny, NotResource-excluded or deny-NotResource forms; only four object-only resource patterns; and only bucket-level requests, with object-level requests entirely untouched. 12 × 4 × 3 = 144, fully accounted for.

Regression coverage exists at both layers. In this repository, twelve matcher tests pin each direction, including an invariant test that every protected action really is bucket-only — ResetBucketReplicationState, despite its name, is an object action and stays out. In the server, three end-to-end tests drive the real handlers at the client, inline-session-policy and S3-router levels; all three fail against the pre-fix build and pass against this one.

What to change

You are affected only if a stored policy grants one of the twelve actions — or s3:* — on a resource pattern containing /, with no bare bucket ARN for the same bucket. The fix is to add the bare ARN alongside the object pattern:

"Resource": ["arn:aws:s3:::bucket", "arn:aws:s3:::bucket/*"]

That pairing is the conventional form, is what upstream’s own tests use, and worked before this release as well. Built-in canned policies are unaffected — readwrite, readonly, writeonly and diagnostics all use Resource: "*".

MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup, restores the historical matching in full — both the over-granting and the over-blocking. It is a single global switch; per-action scoping is deferred.

Policy Condition-Key Lookup Order

getValuesByKey() previously looked up a policy condition key by its canonical MIME spelling (http.CanonicalHeaderKey) before trying the original name. The map it reads mixes values calculated by the server for the current request (SourceIp, SecureTransport, CurrentTime, username and others, stored under condition-key spellings) with HTTP headers supplied by the request (stored under canonical MIME spellings).

Checking the canonical spelling first allowed a client header to override a value calculated by the server.

For a MinIO server this is a policy bypass. The simplest example is s3:prefix: a Prefix request header could satisfy a home-directory prefix condition while the real ?prefix= query parameter still listed the entire bucket. The same path reached aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:username, aws:userid, aws:principaltype, aws:UserAgent, aws:groups, ldap:username, ldap:groups, jwt:groups, s3:versionid, s3:signatureversion, s3:signatureAge, s3:authType and s3:LocationConstraint. Anonymous bucket policies were directly exposed. SigV4 did not prevent the attack because a client can add a header that is not listed in SignedHeaders.

There was a second consequence: when the server stored a value under one spelling and the policy key resolved another, the wrong entry won. s3:object-lock-mode could resolve to the caller’s X-Amz-Object-Lock-Mode header rather than the retention mode the server would actually apply.

The fix reverses the lookup order: match the condition key’s exact name first, then use the canonical spelling only as a fallback for condition keys that genuinely name request headers, such as the s3:x-amz-* family. This ports minio/pkg#226 and adds regression coverage the upstream change did not carry.

At the library’s raw-map layer, if a producer stores one logical field under both the exact condition name and its canonical MIME name, the exact name now wins. This is a library lookup rule, not an S3 wire-protocol rule that says query parameters take precedence. The Silo server first normalizes condition values by their real source. For storage class and upload tagging, where both Header and query forms remain compatible, Header presence wins, including an empty value; query is only the fallback.

LDAP Connection Path

Three defects in connect(). Two were introduced by this fork in b0c08a7 and shipped in v3.6.2 and v3.6.3. Users of either release should upgrade promptly.

StartTLS was skipped when ServerInsecure was enabled. Upstream called StartTLS in an outer block controlled only by ServerStartTLS, so enabling both options created a plaintext connection and then upgraded it. b0c08a7 moved the call into an else branch, making StartTLS unreachable whenever ServerInsecure was true. The connection stayed plaintext and the following bind sent credentials over it. MinIO exposes MINIO_IDENTITY_LDAP_SERVER_INSECURE and MINIO_IDENTITY_LDAP_SERVER_STARTTLS independently and Validate() rejects no combination, so this state was reachable.

This release restores the upstream semantics: the two switches are additive, not mutually exclusive. ServerInsecure disables implicit ldaps://; ServerStartTLS still performs the upgrade. The exposure window is limited to v3.6.2 and v3.6.3.

A Config without a TLS section could panic on the ldaps:// path. After l.TLS.Clone() moved outside the StartTLS branch, ordinary ldaps:// connections also called it. Clone() returns nil for a nil receiver, but the next line assigned ServerName. The MinIO server always supplies TLS settings, but this is a library and mc also consumes it. The code now falls back to an empty tls.Config, matching what DialURL would have built.

StartTLS had no deadline. go-ldap only starts its request timer when requestTimeout > 0, while StartTLS itself has no timeout. A server that completed TCP setup and then stopped responding to the extension request could hold the connect goroutine forever. The timer is now armed before StartTLS.

A failed StartTLS leaked the connection. Inherited from upstream. Dial failures do not return a connection, making StartTLS failure the only connect() path that could return both a connection and an error. Callers only took ownership when the error was nil, leaving a socket behind for every login attempt against a server with a broken upgrade. The failure path now closes the connection and returns nil.

Other Fixes

• certs: file watchers were never stopped. Manager.AddCertificate() registered two notify.Watch() calls and stopped neither: if the second failed, the first leaked, and both survived until process exit after the manager closed. Certificate.Watch() and watchFile() had the same problem. All four paths now use watchDirSafe(), which returns a stop function invoked on errors and ctx.Done(). This ports the certs/ part of minio/pkg#228. On Windows the function replaces filesystem notification with polling rather than using polling only as a failure fallback, so certificate reload can lag by one symlinkReloadInterval (10 seconds). This fork has no Windows CI; that platform was only cross-compiled.
• rng: reader subkeys came from a zeroed local variable. init() read 32 bytes of entropy into r.tmp but derived four subkeys from a same-named zeroed local, collapsing four per-block streams into one. Reset() and ResetSize() then replayed the previous stream byte for byte. MinIO creates a new reader for each randreader.New() call and never resets it, so the practical server impact is limited; warp exposed the defect. This ports minio/pkg#230.
• xtime: Duration implemented UnmarshalJSON but not MarshalJSON. Encoding produced an integer number of nanoseconds while decoding unconditionally stripped the first and last byte and expected a quoted string, so neither direction could round-trip. It now encodes using time.Duration’s string form. This ports minio/pkg#242.

Compatibility Impact

• Twelve bucket-level write actions are no longer authorized through an object-only resource pattern. See What to change. Object access, ListBucket, CreateBucket, bucket tagging, default encryption and event notification are all unaffected, as are Deny statements and NotResource exclusions.
• The minimum Go version moves from 1.26.1 down to 1.25.0. A patch number in the go directive is a hard minimum for every consumer, not a record of the toolchain used to build the module. The conventional split is a language version on the go line and a development version on a separate toolchain line. 1.25.0 is what the dependency graph actually requires and what upstream declares. CI builds the complete test suite with Go 1.25 under GOTOOLCHAIN=local, so the minimum is proven rather than aspirational.
• The JSON wire format of xtime.Duration changes from a nanosecond integer to a duration string such as "2h" or "30m". Persisted numeric values can no longer be read back. No such use was found in MinIO or mc: batch job definitions persist as YAML and the msgp path remains int64.
• Deployments with both ServerInsecure and ServerStartTLS enabled whose LDAP server does not support StartTLS connected successfully in plaintext on v3.6.2/v3.6.3 and now fail to connect. That is the correct result, but it surfaces during connection rather than configuration validation. Disable ServerStartTLS for such a server.
• Policy.IsAllowedActions can disagree with a direct decision for the twelve protected actions. It enumerates SupportedActions, which includes the s3:* pattern itself, so the returned set can contain s3:* — and therefore appear to permit a protected action — while the direct evaluation denies it. Nothing in the server calls it, and Console calls it with an empty bucket name, which never reaches the hardened branch. Recorded rather than changed, because altering a public API’s output in a maintenance release is the larger risk.

Divergence from Upstream v3.11.0

The version number follows upstream’s line and makes no claim of identical content. The measured delta, comparing action-string constants across policy/:

24 actions exist only upstream: six s3:*ObjectAnnotation* actions, five admin: actions (DistJobStatus, Get/SetBucketCompression, two TablesReplication*), and thirteen s3tables: actions covering function CRUD and tagging. These belong to the AIStor vocabulary this fork deliberately does not carry, because the community server does not implement them.

Three actions are named differently on each side. Upstream renamed and split these; this fork retains the earlier names:

A policy naming any of these six action strings therefore validates on exactly one of the two. Nothing in the Silo server, mc or Console references them, so there is no impact inside this ecosystem — but a consumer swapping upstream v3.11.0 for this release should know the vocabulary is not interchangeable.

rng has no arm64 assembly. Upstream added rng/xor_arm64.{go,s} after this fork’s divergence point; this release falls back to the pure-Go xor_noasm.go path on arm64. The result is correct and cross-compiles cleanly, but slower than upstream on that architecture. It is a clean candidate for a future sync, being a pure performance change with no vocabulary entanglement.

Companion Server Behavior

• The condition-key change in this release must be paired with the server changes that reserve internal condition-key names and populate values by semantic source, as noted at the top.
• s3:signatureAge is exposed only after the SigV4 presigned-request verifier calculates it. A client-supplied x-amz-signature-age Header is ignored on every other request type.
• s3:prefix, s3:delimiter and s3:max-keys come only from query parameters. Content hash, copy source, metadata directive, SSE and object-lock conditions come only from the corresponding headers. The X-Amz-Content-Sha256 query value consumed while verifying a presigned request does not become a policy condition.
• s3:x-amz-storage-class retains its compatible query form, as do request tags on PutObject and CreateMultipartUpload. For both fields, Header presence wins and query is used only when the Header is absent.
• s3:ExistingObjectTag/* comes only from tags loaded from the stored object, so a request’s own X-Amz-Tagging can no longer impersonate existing object state. PutObject, CreateMultipartUpload and PutObjectTagging bind s3:RequestObjectTag/* to the tag input those handlers consume. Other action paths retain the historical X-Amz-Tagging Header fallback for compatibility, so treat request-tag conditions as constraints only where the API actually consumes tags.
• aws:SourceIp is calculated from forwarding headers. Whether it is enforceable depends on the server’s trusted-proxy configuration; see the server’s own release notes for MINIO_API_TRUSTED_PROXIES.

Verification

Everything below was run against the tagged commit, with the working tree clean and the tag pointing at HEAD:

• make test — golangci-lint plus go test -race -tags kqueue ./..., all packages passing.
• go mod tidy -diff clean; gofmt -l empty; go vet ./... clean.
• Cross-compilation for linux/amd64, linux/arm64, darwin/arm64 and windows/amd64.
• govulncheck ./... — zero reachable vulnerabilities. One module-level notice remains, GO-2026-5932 in x/crypto/openpgp; that package is unmaintained, has no fixed version, and this repository does not import it.
• Resolution from an empty module cache through the public proxy, confirming the release is fetchable as published.
• The 27,000-outcome authorization corpus described above.

Dependencies and Tooling

Dependency updates clear nine reachable findings previously reported by govulncheck: seven x/crypto/ssh issues reached through sftp, GO-2026-6061 in gRPC reached through etcd, and GO-2026-4945 in go-jose reached through oidc.

Five dependencies — minio-go, minio/mux, etcd client/v3, go-oidc and lestrrat-go/jwx — were deliberately not upgraded. MinIO consumes this module through replace, and Minimal Version Selection chooses the highest version in the entire graph, so upgrading them here would also pull the server forward. None has a reported vulnerability requiring that change.

All three workflows previously asked setup-go for a Go version lower than go.mod required and failed on the first Go command; they are now aligned. The linter also fetched an installer from the master branch and reinstalled it on every run. The URL and version are now pinned to v2.11.3, and a matching installed version skips the download.

Changes Deliberately Not Taken from Upstream

• AIStor policy vocabulary (Memory/cortex, Tables/Iceberg, KMS, compression and annotations) and the typed action-constant refactor, none of which the community server implements. This is the source of the action-vocabulary delta.
• securityAuditAdmin, which grants admin:ExportIAM and therefore exposes every secret key despite what the name suggests.
• rng AVX2/NEON assembly. Revisiting the arm64 half is noted above as a future sync candidate.
• net.BandwidthBytesPerSec (declared but never read upstream), replicationAdmin and DistJobStatusAction.
• Two changes initially taken and removed after review: the consolereadonly built-in policy and GetAllGlobalCertificates. Neither has a consumer. Once operators bind a built-in policy name to users, withdrawing it is particularly unsafe: policy mappings persist by name, and an unresolved name merges into an empty policy that denies everything. Its inherited admin:CreateUser Deny also cannot be combined with iamAdmin. The certificate helper inventoried a cache the community server never populates.
• Upstream’s golangci-lint tool directive, which would add roughly 200 linter dependencies to every downstream consumer’s module graph.

Deliberately Deferred

The general problem in minio/minio#20449 — that bucket/* still reaches ListBucket, GetBucketLocation, the configuration reads, CreateBucket and the three tenant-plausible writes — is not closed here. Closing it means revoking grants real deployments depend on, so it belongs to a release that carries a migration path.

What that release owes operators is more than a longer action list, because no one can enumerate every deployment’s stored policies — which puts a hard ceiling on any approach that picks the protected set by guessing. Three things raise it:

• A startup policy audit that walks stored policies and names each one whose meaning changes, in both the grant and the deny direction. It is read-only and can ship before the enforcement change rather than with it, turning an upgrade surprise into a pre-upgrade checklist.
• A denial that explains itself. When a request is refused because only an object-scoped grant matched, say so and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
• A switch with a scope. MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH is all-or-nothing today, so an operator who needs one action back must reopen the self-escalation path along with it.

Related Commits

• d8b1fa7: fix(policy): settle the bucket-write hardening’s scope and monotonicity
• 1f97549: fix(policy): extend the bucket-write hardening to every bucket-only write
• 3c24ad1: fix(policy): withhold object-only grants from sensitive bucket writes
• da6a22a: docs: say what this fork is and how to depend on it
• 4055b2f: fix(xtime): marshal Duration as a duration string
• 13c26cd: fix(rng): initialize the reader subkeys from the seeded entropy
• 88b37ac: fix(certs): stop file watchers on every exit path
• 74dd36e: fix(ldap): keep StartTLS when ServerInsecure is also set
• 424c3d0: fix(ldap): close the connection when StartTLS fails
• 045d10f: fix(ldap): guard a nil TLS config and arm the StartTLS deadline
• 5c4bf50: fix(policy): prefer the exact key name over the canonical header form
• 802539f: chore(deps): refresh the dependency set and declare the real minimum Go
• e4ec64a: ci: build on the Go version go.mod requires, and prove the declared minimum
• 747d8b8: build: pin the golangci-lint installer and skip a matching install

2.4 - mcli 20260806 Released

Published: 2026-08-06 · Version: RELEASE.2026-08-06T00-00-00Z

Two days after mcli 20260804, this release completes the client’s transition to the Silo identity. It is deliberately a pure rebranding and lockdown release: --version and --help now present the Silo client, every remaining path to MinIO’s SUBNET service is disabled at build time, the embedded vendor encryption key is removed from the diagnostics tooling, and the contribution policy moves to no-CLA with a mandatory DCO sign-off. There are no dependency changes and no protocol changes in this cycle — go.mod is byte-for-byte identical to 20260804 — so the regression surface is confined to text, command gating, and CI.

Major Changes

• Silo identity across the CLI: the client introduces itself as “Silo client for object storage and filesystems”. Roughly 220 help texts were reworked: usage lines that refer to the managed server now say “Silo/MinIO server”, example aliases moved from myminio/play to mysilo, example LDAP DNs moved to dc=example,dc=com, and example tier names to SILOTIER-*. Factual references stay factual: the minio tier type, protocol headers, and third-party interop mentions are untouched.
• SUBNET disabled at build time: connectivity is compiled out behind a single guard, and the one HTTP choke point that every SUBNET request funnels through refuses with the stable error above. Command entry points gate early, diagnostics force local mode, and the AGPL license notice shown by mcli license info no longer carries a commercial-subscription pitch. A dedicated regression suite (cmd/subnet-disabled_test.go) pins all of this, so an upstream merge cannot silently reconnect anything.
• Governance — no CLA, DCO required: contributions are accepted inbound=outbound under AGPL-3.0-or-later; contributors keep their copyright, and the maintainers collect no rights beyond the project license. Every commit must carry a Signed-off-by trailer, enforced by a new CI workflow that matches the trailer against the commit author’s email and exempts only GitHub-issued bot addresses. CONTRIBUTING.md, the PR template, and both READMEs document the policy, and the code-of-conduct contact now points at the fork’s maintainer.
• Dual copyright attribution: runtime output and help now credit both lineages — Copyright (c) 2015-2025 MinIO, Inc. and Copyright (c) 2025-2026 PGSTY — with source builds computing the end year dynamically. NOTICE states the fork relationship, and the non-affiliation with MinIO, Inc., explicitly.
• Release line renamed to main: workflow branch filters, documentation, and contributor instructions now target main; the legacy master references are gone.

Hardening

• Vendor encryption key removed: mcli support inspect used to fall back to encrypting its output with an embedded MinIO RSA public key whenever no key was supplied — producing archives only the vendor could decrypt. The embedded key is gone: inspect now relies on the server-generated per-request key that is printed to the caller (or an operator-supplied key), and any encrypted-upload path with no configured recipient fails closed instead of silently borrowing a third-party key. Diagnostic output an operator produces is now always decryptable by that operator.
• Brand-policy gate: buildscripts/check-branding.sh runs in make verifiers and in CI. It fails the build if MinIO-operated endpoints, commercial upsell URLs, the upstream product identity, or any embedded MII… public key reappear in the command tree — while explicitly allowlisting the preserved compatibility identifiers (environment variables, protocol headers, module path, legacy-migration defaults, and original copyright headers).

Engineering and Delivery

• CI moved to the Node 24 Actions line: actions/checkout v7, actions/setup-go v7, goreleaser-action v7, and the Docker action family — all still pinned to commit SHAs, with dependabot keeping the pins current.
• Functional tests run against controlled servers only: the suite defaults to a local server (localhost:9000) instead of MinIO’s public demo cluster, and CI downloads the pinned SILO server release RELEASE.2026-08-04T00-00-00Z from pgsty/silo, verified by SHA-256, before running the suite.
• Zero dependency changes: no module updates this cycle; the 20260804 security baseline (Go 1.26.5, zero known reachable vulnerabilities) carries over unchanged.
• Audited before tagging: the release was gated by an independent adversarial review — a full read of the 245-file diff, brand/compatibility grep sweeps, call-graph verification that no command or flag combination can reach subnet.min.io / play.min.io / dl.min.io, and smoke tests confirming every disabled path returns its stable error with exit status 1.

Compatibility

Everything scripts and integrations depend on is deliberately unchanged: the mc command name and the mcli package/binary name; the ~/.mc / ~/.mcli configuration directories (derived from the invoked name); the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go SDK user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job Prometheus scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme. The client remains fully compatible with MinIO servers and other S3-compatible endpoints.

Related Commits

• 8a883ca: ci: move the branch filters to main and fetch the server from pgsty/silo
• 8c304dd: ci: move the pinned actions onto the Node 24 runtime
• 02b1c11: docs: name the release line main, not master
• 810bbd2: ci: pin the functional-test server to a release whose watch API works
• d145647: fix: disable SUBNET connectivity and licensing upsell paths
• 5061c4f: rebrand: adopt Silo identity in CLI help and examples
• c7f7706: docs: align governance files and package metadata with the fork
• 65c71b2: test: default functional tests to a local server and add brand gate
• c62a64d: fix: credit both MinIO and PGSTY in copyright notices
• d205f88: docs: adopt no-CLA plus DCO contribution policy
• 95326ce: docs: add related-projects table and polish contribution wording
• d2c0db7: fix: remove the vendor encryption key and close the proxy-set path
• 0c6704d: fix: repair a link and help text damaged by the brand sweep

2.5 - Silo 20260806 Released

Version: RELEASE.2026-08-06T00-00-00Z · Commit: 3be10fcc1a44f6620ded0bd303461f9d688cca23

SILO 20260806 is the first release published under the Silo name. The previous release, 20260804, was the last one delivered as pgsty/minio; this release completes the cutover to github.com/pgsty/silo and renames every delivery surface — binary, packages, container images, systemd unit, Helm chart — while deliberately preserving every wire and configuration surface a MinIO deployment depends on. On top of the rename it adds native health checking (silo healthcheck), a single-binary distroless container image pilot, complete license-compliance materials in every artifact, and a release pipeline gated on compatibility snapshots and build provenance.

The release covers 28 commits after RELEASE.2026-08-04T00-00-00Z, changing 396 files with 27,188 insertions and 19,561 deletions. It passed a six-phase pre-release acceptance, including a real four-node TLS cluster migration from MinIO to Silo — with byte-verified data integrity, maintenance-gated rolling restarts, fault injection, and a full rollback rehearsal.

Highlights

• The rebrand is complete, and compatibility is the contract. Repository, binary (/usr/bin/silo), packages (silo rpm/deb/apk), images (docker.io/pgsty/silo), and service (silo.service) are renamed; the S3 and admin APIs, /minio/* routes, MINIO_* environment variables, x-minio-* headers, on-disk .minio.sys format, and Go module paths are all preserved and frozen by a CI compatibility guard.
• Native health checking: silo healthcheck [live|ready|cluster|cluster-read] probes the server’s own health API with correct exit codes, decoded quorum diagnostics, TLS auto-detection, and a --maintenance pre-drain gate — no shell, curl, or mc required in the container.
• Distroless image pilot: pgsty/silo:<RELEASE>-distroless ships exactly one program — the silo binary — on gcr.io/distroless/static, with an exec-form HEALTHCHECK baked in and /data created writable in the image layer.
• The classic image does not change behavior: same entrypoint, same bundled tools, mc ready local keeps working, and no HEALTHCHECK was added to it. It now bundles mcli 20260806.
• Compliance completed: LICENSE and NOTICE ship in every package and image, CREDITS is regenerated from the actually-linked module set (291 modules) and guarded in CI, and the project adopts a no-CLA, DCO-based contribution policy.
• Components refreshed: embedded SILO Console 2.1.1, silo-pkg 3.11.0, mcli 20260806, Go 1.26.5.
• Provenance-gated releases: container images are built only from published, checksum- and attestation-verified release archives; image SBOMs and provenance attestations now cover the distroless variant too.

The rename

What changed, and what deliberately did not:

The server presents its own identity — silo --version reports the AGPL-3.0 license, MinIO’s 2015-2025 copyright, PGSTY’s modification copyright, and the “based on MinIO technology” attribution — and every inherited connection to MinIO-operated services (the update feed and its signing key, SUBNET, telemetry) is severed rather than redirected. The container entrypoint translates the legacy minio argv token, so docker run pgsty/silo minio server /data keeps working.

A snapshot-based rebrand guard runs in CI: it fails on any drift, in either direction, across 334 route literals, 437 environment tokens, 84 headers, and 9,014 exported symbols.

Native health checking

The server binary can now probe its own health endpoints, which makes container health checks possible without any second binary — and is what the distroless image relies on:

silo healthcheck [FLAGS] [live|ready|cluster|cluster-read]

• The check vocabulary maps 1:1 onto /minio/health/<path>; live (the default) answers “is this process serving,” ready adds KMS/etcd reachability when configured, and the cluster pair evaluates write/read quorum across every erasure set.
• Exit codes are 0 (healthy) and 1 (anything else) — never the Docker-reserved 2. One diagnostic line decodes the server’s x-minio-server-status and quorum headers for docker inspect; --json emits a machine-readable verdict.
• The probe target is derived the way the server derives its own listen address: --address / MINIO_ADDRESS, with HTTPS auto-detected from public.crt + private.key in the certs directory, or overridden wholesale with --url / MINIO_HEALTHCHECK_URL. The environment form exists because a probe process cannot see the server’s command line — if the server’s address or TLS comes from CLI arguments, one environment variable redirects the baked-in probe.
• silo healthcheck --maintenance cluster answers the pre-drain question: exit 0 means the node can be taken down without losing HA; HTTP 412 (exit 1) means it cannot.
• Certificate verification is skipped, matching the kubelet’s documented behavior for HTTPS probes, and the transport ignores HTTP_PROXY so loopback probes never route through a proxy.

Kubernetes needs none of this — kubelet httpGet probes hit /minio/health/live and /minio/health/ready from outside the container — and the cluster checks should stay out of per-container probes: they reflect cluster-wide quorum, not one process. The full design rationale, including verified endpoint semantics, is recorded in the health-check design note.

Distroless image pilot

Alongside the classic image, this release publishes a distroless variant: pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distroless, plus a rolling distroless tag.

• Base is gcr.io/distroless/static-debian12: CA certificates, tzdata, /tmp, and an /etc/passwd with a nonroot (65532) entry — no shell, no package manager, no libc. On top of it, exactly one program: /usr/bin/silo (plus the license set under /licenses/). The image is 128 MB versus the classic 199 MB.
• The binary is the ENTRYPOINT; an exec-form HEALTHCHECK running silo healthcheck ready is baked in (interval 30s, timeout 10s, start-period 2m, retries 3), so Compose users get working depends_on: condition: service_healthy with zero configuration.
• /data is created in the image layer, world-writable — there is no entrypoint left to repair volume ownership at runtime, and this is what makes every privilege mode work, --user included. This fixes, for the distroless variant, the non-root failure documented in #55.
• Not supported in this variant: the deprecated MINIO_USERNAME/MINIO_GROUPNAME privilege-drop path (use --user or Kubernetes runAsUser), docker exec <c> sh debugging (use ephemeral-container tooling), and in-image mc (use the released mcli or the client image).
• TLS: mount certificates at /tmp/.silo/certs (the container’s default certs directory) and both the server and the baked-in probe derive HTTPS from the same location; for CLI-configured servers, set MINIO_HEALTHCHECK_URL.

The classic image remains the default and is unchanged. If the pilot proves out, the distroless variant becomes the recommended image later; the decision record lives in the design note above.

Container images

The classic image was diffed field by field against pgsty/minio:RELEASE.2026-08-04T00-00-00Z: entrypoint, exposed ports, volumes, working directory, user, and (absent) health-check configuration are identical. Exactly three differences exist, all deliberate: Cmd is ["silo"] instead of ["minio"], the upstream update-verification key variable MINIO_UPDATE_MINISIGN_PUBKEY is removed (updates through upstream channels are permanently disabled), and HOME=/tmp is declared to match the entrypoint’s writable-home guarantee.

The bundled client is upgraded to mcli RELEASE.2026-08-06T00-00-00Z (with the mc alias preserved), pinned by per-architecture SHA-256 digests and verified against the published checksums at build time. Interoperability of the released mcli 20260806 against this server — multipart, versioning, presigned URLs, metadata/tags, user and policy administration — was verified as part of release acceptance.

Helm chart

The chart ships as silo 7.0.1, preserving rendered resource identity with the legacy chart across a simulated upgrade (verified by the migration guard over 7 rendered resources). Its default image tag now points at this release — docker.io/pgsty/silo is a fresh repository, so the inherited default could never have pulled. The chart still ships no liveness/readiness/startup probes; adding them is planned, and documented, in the design note’s follow-up phase.

Packaging and migration

RPM, DEB, and APK packages install exactly six files: /usr/bin/silo, silo.service, a sysusers definition (creating the silo system user), /etc/default/silo (marked config/noreplace), LICENSE, and NOTICE. RPMs are GPG-signed with the PGSTY maintainer key (9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20). RPM and DEB now carry a unified, PGDG-style 1PGSTY release segment — silo-<version>-1PGSTY.<arch>.rpm and silo_<version>-1PGSTY_<arch>.deb — replacing the inherited bare -1 on RPM and the missing revision on DEB; APK names stay bare because Alpine pkgrel admits only -r<integer>.

silo.service is designed for takeover: Type=notify (readiness is signaled by the server itself), Conflicts=minio.service + After=minio.service (starting Silo stops a running MinIO unit), and two environment files — /etc/default/minio is read first and /etc/default/silo overrides it — so an existing MinIO configuration is inherited without editing. For existing deployments whose data is owned by the minio user, the documented drop-in keeps ownership untouched:

# /etc/systemd/system/silo.service.d/10-legacy-user.conf
[Service]
User=minio
Group=minio

Migration troubleshooting, from the acceptance run: if Silo starts as the packaged silo user against a deployment whose TLS certificates live under the minio user’s home, it fails with HTTPS specified in endpoints, but no TLS certificate is found and restart-loops until the systemd start limit — the legacy-user drop-in above is the fix. Keep the MinIO package and unit installed (disabled) during the migration window: the rollback path — stop Silo, start MinIO — was rehearsed and reads all data written during the Silo window, because the migration touches neither data ownership nor format.

Components and dependencies

• SILO Console 2.1.1 — the embedded console, selected from pgsty/silo-console while preserving the github.com/minio/console import path.
• silo-pkg 3.11.0 — retains the policy/LDAP/certificate fixes including the LDAP-over-TLS repair tracked in #15.
• mcli 20260806 — bundled in the image and released separately; see its release notes.
• Go 1.26.5 — toolchain unchanged from 20260804.

Build, CI, and release pipeline

• Compatibility as a CI gate: the rebrand guard snapshots routes, environment tokens, headers, metrics, storage/policy identifiers, and exported symbols, and fails on any unreviewed drift; companion scripts assert the delivery surface (binary path, unit contents, image layout) and that no live upstream endpoint remains in runtime code.
• Release-image gate: every release-pipeline run builds both container images and asserts, among others: the distroless HEALTHCHECK survives into the image config (it is a Docker extension outside the OCI spec), /data ships world-writable, no shell and no /usr/bin/minio exist, Docker’s health state turns healthy from the baked probe alone, and SIGTERM still stops the server gracefully as root and as --user 1001:1001.
• Provenance chain: images are built from the published release archives after checksum verification and gh attestation verify against the exact tag; per-architecture SBOMs and provenance attestations are pushed for the classic and distroless images; the distroless health-check gate runs before the multi-arch manifests are promoted.
• Workflow runtime moved to Node 24 across CI actions.

Compatibility and upgrade notes

1. Package upgrades are a takeover, not an in-place update. Install silo, keep /etc/default/minio as is (it is inherited), enable silo.service; starting it stops minio.service via the conflict relation. Data is untouched.
2. Keep data ownership stable with the legacy-user drop-in above; do not chown storage or move certificates during migration.
3. Distributed clusters: full-stop switchover only. See the warning above — mixed Silo/MinIO nodes do not form a cluster.
4. Container users: the image is now docker.io/pgsty/silo; docker.io/pgsty/minio stays frozen at 20260804 as an archive. The classic image’s behavior is unchanged — including mc ready local health checks — and the distroless variant is strictly opt-in.
5. Distroless differences are deliberate: no shell, no in-image mc, no MINIO_USERNAME path; health is native; servers configured via CLI arguments need MINIO_HEALTHCHECK_URL for the baked-in probe.
6. Helm users: chart 7.0.1’s defaults now pull this release; override image.tag explicitly if you pin versions.
7. Known and unchanged: the classic image still does not create /data in the layer, so fully non-root docker run against a Docker-managed volume fails as before (#55, fixed in the distroless variant); the inherited Postgres/MySQL legacy notification-migration limitation from the 20260804 notes still applies (#53).
8. Pair with mcli 20260806 for the client side; older clients continue to work over the unchanged wire protocol.

Verification

This release was verified in stages, each with recorded evidence:

• unit and end-to-end matrices for the health-check command: target derivation and precedence (flag/env/derived), real-TLS auto-detection, exit-code contract, JSON schema, timeout bounds, usage errors;
• cluster-semantics verification on a four-node cluster: with 2 of 4 nodes stopped, cluster reports 503 with write-quorum=5 while cluster-read and live stay 200 — the write/read quorum split observed live, matching the erasure math;
• image acceptance: the classic image diffed field-by-field against the 20260804 baseline; the distroless image asserted down to file inventory, exact health-check configuration, and root/non-root/TLS/env-override runtime scenarios;
• an adversarial model-based code review of the new code, with every confirmed finding fixed and re-verified;
• a six-phase pre-release acceptance concluding in a real migration: a Pigsty-deployed four-node TLS MinIO 20260804 cluster (16 drives, EC:4) was migrated to Silo via the packaged takeover path — reference data (multipart, versioned, tagged objects) read back byte-identical, four maintenance-gated rolling restarts, kill -9 fault injection with the load balancer serving 23/24 continuous IO rounds (the only failure in the kill second), Prometheus metrics continuity, and a full rollback to MinIO and back, proving the migration reversible.

Validation boundaries

Not proven by this release and not to be inferred: external LDAP/OIDC/KMS/etcd services (the only case where ready diverges from live was not exercised against a live KMS); amd64 packages were cross-built and payload-checked but not installed on a physical x86-64 host; the renamed Docker publish workflow (including the new SBOM/attestation lanes) has its first production run at this release’s publication; Windows and Intel macOS were not tested.

Artifacts

• GitHub release RELEASE.2026-08-06T00-00-00Z at pgsty/silo, with checksummed platform archives, provenance attestations, and RPM/DEB/APK packages (GPG-signed RPMs);
• docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Z and latest; docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distroless and distroless — published on demand from the finished release;
• companion releases: mcli 20260806, silo-pkg 3.11.0, embedded SILO Console 2.1.1;
• design record: Native Health Checks and the Distroless Image.

Selected changes

• 15def34dc, 77bdc4c0c: drop upstream delivery residue; present Silo identity and close inherited upstream services
• 15ab10833: rename the delivery artifacts to silo and complete the package payload
• 30749911b: ship the silo binary in the image and translate the legacy argv command
• e071bb77e: replace the minio chart with a silo chart that preserves identity
• bd8df5166: gate the rebrand on compatibility, packaging, and provenance evidence
• 6613c2a3c: pin the external test fixtures and run the suites against the silo binary
• fd2ca1c6d, c46b16ec6, c47733abc, f1c77d5a2: cut over to pgsty/silo and main; document the archived branch
• 6740e6978: move the workflow actions onto the Node 24 runtime
• b57275be3: adopt the no-CLA plus DCO policy and fix copyright terms
• 62717d7bf, a6d6d9b02: update the embedded Console to 2.1.0, then 2.1.1
• 6bd9cf77e: regenerate CREDITS from the linked module set and guard it in CI
• 219670d31: ship LICENSE and NOTICE in every package and image
• 2ff594f4b: add the native silo healthcheck subcommand
• 4c34d2309: add the distroless image variant as a pilot
• b6d47b739, 9462cce16: harden both per adversarial review; lint cleanup
• 16b78eb4e: bundle mcli 20260806 and point the Helm defaults at this release
• 062a91bee: pin the CREDITS module closure to the shipped linux target
• 467931455: unify the rpm and deb release segment as 1PGSTY
• b14ea22aa: match checksum manifest entries exactly in the image publish lane
• 3be10fcc1: add a manual finalize lane refreshing SBOMs and checksums for signed Draft packages

Acknowledgments

Four contributors have code merged into this fork, and the Git history carries their authorship: @ZouhairCharef patched CVE-2026-34986 in go-jose (#18), @mfredenhagen patched CVE-2026-39883 in OpenTelemetry (#19), @pinginfo implemented Flush on trackingResponseWriter to repair bucket notification streaming (#34), and @waterkip repointed the documentation links to the Silo portal (#41).

A first release under a new name is also the right moment to thank everyone who has filed issues against this fork — bug reports, compatibility findings, and proposals alike, resolved and still open:

@mosesdd (#1), @Xavier-777 (#2, #17), @jiadzh (#3), @TLINDEN (#4), @AntonOfTheWoods (#5), @zylpsrs (#6), @nsanitate (#7), @makinikm (#9), @magicxor (#10), @spaceg00se-r (#11, #14), @heroes1412 (#13), @vampywiz17 (#15), @davinkevin (#20), @chalukyaj (#30), @cbornet (#31, #32), @jvasile (#33), @Kesavaambati (#35), @redfoxfox (#38), @kuldeep-link11 (#39, #40), @meesudzu (#42), @pmezhuev (#43), and @kh0mka (#51).

Several of this release’s headline items trace directly back to those reports: the bundled-client guarantee to #4 and #9, the LDAP-over-TLS repair to #15, the completed package payload to #33, GPG-signed RPMs to #43, the migration guide to #42, and the distroless /data fix to #55.

Pull requests still in flight deserve a mention too. @davinkevin’s distroless image PR (#21) anticipated this release’s pilot months in advance — the shipped variant supersedes that PR with the native health check built in, but the direction was proposed there first. Conformance PRs from @magicxor (#12) and @ycjlin (#37) are queued for review immediately after this release.

Everyone who has contributed to this fork is recorded in CONTRIBUTORS.md, which is now the project’s attribution record — GitHub generates no contributor graph for forks.

2.6 - Silo 20260804 Released

Version: RELEASE.2026-08-04T00-00-00Z · Commit: d88f46ccee345a9c2fabe2d221d9a9e56bc11aec

SILO 20260804 is a security, correctness, and release-engineering update to the pgsty/minio community fork. It completes the internode storage-containment work begun with CVE-2026-42600, prevents request-controlled values from impersonating server-calculated S3/IAM policy conditions, restores streaming flush behavior, fixes several multipart and versioning edge cases, hardens notification configuration migration, moves the build baseline to Go 1.26.5, and connects the server to the SILO-maintained Console, shared package, and mcli releases. The release pipeline was rebuilt to produce reproducible binaries and GPG-signed packages.

The release covers 50 commits after the pre-2026-06-18 baseline, changing 155 files with 9,241 insertions and 981 deletions. Every change was reviewed against the tagged commit and verified on macOS ARM64 and Linux AMD64, with GitHub CI green on the released HEAD.

Highlights

• Internode containment completed: validates storage-REST message bodies, storage Grid frames, and peer-S3 Grid requests at the storage boundary, closing the remaining path, volume, erasure-metadata, panic, and unbounded-allocation defects left after removing ReadMultiple.
• S3/IAM decisions now use effective values: client input can no longer shadow internal condition values; request tags and existing-object tags are separated; s3:signatureAge is confined to verified presigned requests; and s3:versionid follows the version the server actually acts on.
• Bucket and object resources are separated: twelve sensitive bucket-level writes are no longer authorized through an object-only bucket/* resource pattern. A documented compatibility switch is available for migration.
• Multipart compatibility and correctness improved: full-object checksum completion works without per-part checksums when the protocol permits it, zero-length multipart checksums are preserved, and duplicate part numbers are rejected instead of assembling duplicated data.
• Streaming reliability restored: trackingResponseWriter now implements Flush correctly and records implicit HTTP 200 responses, repairing mcli watch, bucket-notification listeners, and S3 Select keep-alives affected by the inherited regression documented in the 20260618 release.
• Notification configuration hardened: NATS and AMQP keys used by parsers and legacy migration are registered and round-trip correctly; libpq connection parameters are quoted safely; invalid-key errors no longer echo secret values.
• Reproducible, signed release pipeline: binaries no longer embed the build machine’s paths, packages install under the canonical systemd path, and RPMs are GPG-signed. The container entrypoint now shuts down gracefully on every privilege path.
• Release baseline refreshed: Go 1.26.5, klauspost/compress 1.18.7, Apache Thrift 0.24.0, SILO Console 2.0.0, silo-pkg 3.11.0, and mcli 20260804.

Security Hardening

Internode storage and Grid containment — SN-2026-002

Removing the obsolete ReadMultiple endpoint in 20260618 closed one reachable path but did not close the underlying defect class. Storage-REST request bodies and Grid RPC frames do not pass through the HTTP query-validation middleware, and peer-S3 RPCs can bypass the storage-REST wrapper entirely.

This release moves containment to the storage boundary and validates every caller-controlled path, volume, erasure parameter, part size, shard length, and allocation length before use. The fixes include:

• reject traversal on both path and volume axes, including Windows volume-root aliases;
• cover peer-S3 bucket RPCs that reach drives without the storage-REST wrapper;
• reject zero or unusable data/parity/block-size combinations before shard arithmetic;
• reject negative part sizes and truncated shards instead of reporting them healthy;
• cap storage-REST ReadFile allocations at 5 GiB;
• bound other allocations derived from internode declarations;
• contain panics in deadline-bounded storage work without blocking the caller;
• preserve ReadParts errors across keep-alive responses and avoid the empty-part trace panic.

These routes require cluster-root or internode credentials and are registered only in distributed-erasure deployments. Single-node S3 behavior is unchanged. See Internode Path Containment Audit for the protocol-surface analysis.

Effective policy-condition values — SN-2026-003

The policy condition map historically mixed values calculated by the server with raw request entries. A client-controlled spelling could therefore shadow or synthesize an internal condition value. SILO 20260804 pairs silo-pkg 3.11.0’s exact-key lookup rule with server-side source normalization:

• internal condition names cannot be supplied as arbitrary client values;
• s3:prefix, s3:delimiter, and s3:max-keys come from their effective query inputs;
• header-backed x-amz-* conditions do not accept unrelated query substitutes;
• when storage class or upload tagging supports both forms, an explicitly present Header wins, including an empty Header;
• s3:ExistingObjectTag/* comes only from stored object metadata;
• s3:RequestObjectTag/* is bound to the tag input consumed by the relevant operation;
• s3:signatureAge is exposed only after verified SigV4 presigned authentication calculates it;
• s3:versionid is absent when no version is named and is rebound per DeleteObjects entry to the effective resolved version.

The version-ID behavior closes the fail-open trap that a superficial “omit empty values” fix would have created for Multi-Delete. See Absent Is Not Empty.

Bucket/object resource boundary — SN-2026-004

The IAM matcher used to append a slash to a bucket-level request, allowing an object-only resource such as arn:aws:s3:::bucket/* to authorize selected bucket-level operations. This release withholds twelve sensitive writes from that pattern on Allow statements:

PutBucketPolicy, DeleteBucketPolicy, PutBucketObjectLockConfiguration, PutBucketVersioning, PutReplicationConfiguration, PutBucketLifecycle, DeleteBucket, ForceDeleteBucket, PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration.

Deny and NotResource behavior is unchanged. Read/list operations, CreateBucket, bucket tagging, default encryption, and notification configuration remain compatible. Built-in policies use Resource: "*" and are not affected.

"Resource": ["arn:aws:s3:::bucket", "arn:aws:s3:::bucket/*"]

Trusted client-address boundary

MINIO_API_TRUSTED_PROXIES provides an enforceable, opt-in boundary for aws:SourceIp, audit remotehost, event notification Host, and the client address shown by mcli admin trace:

• set it to an address/CIDR list to trust forwarding headers only from those peers and walk forwarding chains from right to left;
• set it to none to ignore all forwarding headers;
• leave it unset to preserve historical behavior exactly.

The old _MINIO_API_XFF_HEADER=off switch still suppresses only X-Forwarded-For; it does not protect against X-Real-IP or RFC 7239 Forwarded. If IP-based policy is part of your security boundary, configure trusted proxies explicitly and prevent direct access to the S3 API port. Multi-node deployments should allow their own node addresses. See Client Source Address Trust.

S3 and Storage Correctness

Multipart upload

• CompleteMultipartUpload accepts the S3 full-object checksum mode when the completed request supplies no per-part checksums and the upload metadata does not require them.
• The checksum of a zero-length multipart object is retained instead of being discarded as empty metadata.
• Part numbers must be strictly increasing. Duplicate entries such as [1,1] now return InvalidPartOrder instead of consuming the upload and assembling the same part twice. Legal part lists with gaps or a non-1 start remain accepted. See Duplicate Part Numbers.

Object reads and buffer ownership

• erasure reads again pool buffers only where ownership permits reuse;
• update downloads return caller-owned buffers instead of exposing data that can be overwritten after return;
• the old HTTP streaming helpers orphaned by ReadMultiple removal are deleted after reference and platform-tag checks.

HTTP response tracking and S3 Select

• trackingResponseWriter.Flush() delegates to the underlying flusher and commits the response state correctly;
• the first implicit write records HTTP 200, preserving audit and metric accuracy;
• S3 Select tests no longer race a client parser against response-body ownership;
• CSV, JSON, and Parquet selection paths remain covered, including range/error and keep-alive behavior.

The inherited silent-flush regression called out in SILO 20260618 is therefore fixed in this release.

IAM, Versioning, and Audit Behavior

• DeleteObject and each entry in DeleteObjects evaluate s3:versionid against the effective version selected by the server.
• Request tags can no longer impersonate existing-object tags during policy evaluation.
• The merrs tag is restored when dangling-object deletion records are emitted, preserving the intended audit classification.
• Bucket-policy and IAM paths share the hardened condition-source rules while retaining their established S3 routing and error behavior.

Notification Configuration

• registers the NATS user_credentials, nkey_seed, and tls_handshake_first keys read by the parser;
• separates the legacy NATS environment-variable spelling from the stored config key;
• repairs NATS migration round trips and the AMQP immediate/internal mapping;
• adds a mechanical audit that compares keys read and written by notification code with each subsystem’s registered schema;
• quotes libpq connection-string parameters so whitespace, quotes, and backslashes retain their intended value;
• prevents invalid-key diagnostics from echoing secret values.

See Notify Keyspace Registration.

Components and Dependencies

• Go 1.26.5: includes security fixes in crypto/tls and os plus compiler, runtime, networking, and syscall corrections.
• klauspost/compress 1.18.7: refreshes the compression stack used by object and archive paths.
• Apache Thrift 0.24.0: updates the dependency compiled through Parquet support.
• go-systemd 22.6.0: deliberately retained instead of 22.7.0 because the later version introduced a NetBSD clock dependency incompatible with the supported cross-build matrix.
• SILO Console 2.0.0: the embedded console is selected from pgsty/silo-console while preserving the compatible github.com/minio/console import path.
• silo-pkg 3.11.0: provides the companion policy, LDAP, certificate, RNG, and time-format fixes while preserving the github.com/minio/pkg/v3 module path.
• mcli 20260804: the embedded client comes from pgsty/mc; release images expose it as mcli and keep the mc compatibility alias.

See the companion release notes for silo-pkg 3.11.0, mcli 20260804, and SILO Console 2.0.0.

Build, CI, and Packaging

This release rebuilt the release pipeline for reproducibility and supply-chain integrity:

• Graceful container shutdown on every path. The entrypoint’s custom UID/GID branches now exec into the server so it runs as PID 1 and receives SIGTERM directly; previously those branches left an intermediate shell as PID 1 and the server was killed at the container stop timeout. A CI smoke test builds the release runtime image and asserts graceful shutdown on both the default and drop-privilege paths.
• Reproducible binaries. Release binaries no longer embed the build machine’s GOPATH/GOROOT, so -trimpath holds and a third party rebuilding the tag gets matching bytes. The published Linux binary contains no build-host path.
• Hardened release workflow. The release tag is passed through the environment and whitelisted rather than spliced into the shell, the build is checked out at the tag being released, and an untracked shadow GoReleaser config that could publish or move latest out of band was removed.
• Honest gates. CI gates build, vet, unit tests, lint, generation drift, race tests, and cross-compilation; the cross-compile matrix is aligned to the exact set of published targets; and the lint and dependency-install steps now fail on real errors instead of masking them.
• Signed, canonical packages. RPM, DEB, and APK packages are produced with nFPM under the PGSTY identity, the systemd unit installs at /usr/lib/systemd/system/minio.service with Type=notify, and RPMs are GPG-signed offline with the PGSTY maintainer key (fingerprint 9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20).
• Release and container publication remain separate gates. GoReleaser produces the platform archives, checksums, and packages; the multi-architecture image is published on demand from the finished release. A local snapshot does not prove a public release or image exists.

Compatibility and Upgrade Notes

1. Keep every node on one release during a cluster rollout. Internode validation changed across storage-REST and Grid surfaces; mixed binaries were not production-tested.
2. Audit custom IAM policies. Add the bare bucket ARN for the twelve protected bucket writes. Use MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on only as a temporary migration control.
3. Configure client-address trust deliberately. If aws:SourceIp or audit attribution matters, set MINIO_API_TRUSTED_PROXIES and close direct network paths around the proxy.
4. Review legacy database notification settings. Convert Postgres/MySQL host/user/password fields to the supported connection-string format before restart.
5. Expect duplicate multipart completion entries to fail. Clients sending the same part number more than once now receive InvalidPartOrder instead of a corrupted successful object.
6. Use the matching mcli. The 20260804 client disables self-update and must be upgraded through packages or GitHub Releases; mcli update remains as a compatibility command but exits non-zero.
7. RPM users can enable signature verification. Packages are signed with the maintainer key above; import it before enabling gpgcheck for the SILO packages.

Verification

Changes were reviewed against the tagged commit and re-verified rather than trusted from prior reports:

• git diff --check, gofmt, module verification, and YAML/shell syntax;
• go build ./..., go vet ./..., project lint, and govulncheck ./...;
• full go test ./..., the complete race suite, and repeated race tests over storage, policy, notification, HTTP tracking, and S3 Select changes;
• generator idempotence plus deliberate stale-source and untracked-output counterexamples;
• cross-compilation across every published target;
• Linux AMD64 native full tests, targeted race tests, live S3/mcli smoke tests (create/upload/download/copy, range, versioning, delete markers, health checks, graceful shutdown, restart persistence), and systemd notify behavior;
• release-artifact verification: GitHub CI green on the released HEAD, reproducible binaries with no build-host path, the systemd unit installed at /usr/lib/systemd/system, and RPM signatures validated with rpmkeys --checksig.

govulncheck found no vulnerability reachable from the Server or mcli code. One module-level notice remains for the unmaintained golang.org/x/crypto/openpgp package (GO-2026-5932); that package is not imported into these binaries.

Validation boundaries

The following were not proven by this release and must not be inferred from cross-compilation or unit tests:

• native Windows execution and Windows filesystem semantics;
• Intel macOS and physical Linux ARM64 hosts;
• a production multi-node rolling upgrade, site replication, or lifecycle expiration run;
• real reverse-proxy chains and direct-ingress isolation;
• external LDAP, OIDC, KMS, STS, Postgres, MySQL, NATS, and AMQP services;
• installing and upgrading the signed package under a real systemd host.

Artifacts

• GitHub release RELEASE.2026-08-04T00-00-00Z with checksummed platform archives for Linux, Darwin, and Windows on amd64 and arm64;
• RPM, DEB, and APK packages under the PGSTY identity, with GPG-signed RPMs;
• docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Z and the release-selected latest tag, published on demand from the release;
• matching SILO Console 2.0.0, silo-pkg 3.11.0, and mcli 20260804 references.

Selected Changes

• ca7baa670, 80e8eaa42, b6f70ab08: validate internode paths, erasure metadata, and allocation sizes
• a36fd8fff: contain panics in deadline-bounded storage work
• 2f55347f7: bind S3/IAM policy conditions to effective request values
• 744a9dcd7: bind s3:versionid to the effective object version
• 97b7d2804: enforce the bucket/object resource boundary
• fe6dc4780: add the trusted-proxy client-address boundary
• 22c1e41fd: reject duplicate multipart part numbers
• c8590413f, 3e14733f1: restore full-object and zero-length multipart checksum behavior
• 8069a32ac, 65795ee1f: restore response commit and streaming flush semantics
• 162ded343, 0c14d8151: repair notification key registration and libpq quoting
• 924717926, 89d346bf5: restore safe buffer pooling and returned-buffer ownership
• 3b8a55dee: exec into the dropped-privilege process so signals reach the server
• 2ca4971d9: stop stamping the build machine’s paths into the binary
• 4c185d5a6, e064b5555: harden the release workflow and remove the shadow config
• aa5139369: install the systemd unit under /usr/lib
• 11d79fddc, ca674a696, 021110b45, d88f46cce: gate build, vet, tests, lint, generation, race, and cross-compilation, and smoke-test the release image

2.7 - mcli 20260804 Released

Published: 2026-08-04 · Version: RELEASE.2026-08-04T00-00-00Z

This is the first release of the pgsty/mc community fork since 20260417. It fixes a credential leak in debug logging, severs every remaining connection between the client and upstream release channels, moves containers and packages onto artifacts this fork builds itself, and migrates packaging from MinIO’s pkger to standard nFPM — with GPG-signed RPMs for the first time.

Upstream minio/mc was archived in July 2026. Its final commit, 77f82e18, is exactly this fork’s base, and upstream never cut a release containing it — so this build is strictly newer than any official mc binary ever published.

Major Changes

• Self-update disabled, upstream release channels severed: the minio/selfupdate and aead.dev/minisign dependencies and all binary-replacement logic are gone, along with the update notifier and the FIPS/non-FIPS update paths. The update command survives as a compatibility shell, and the runtime helpers (Docker / DCOS / Kubernetes / source-build detection) moved to a dedicated cmd/runtime-info.go. The client previously reached out to upstream release feeds on every invocation to print an upgrade hint; there is now no outbound release probing at all.
• Containers and artifacts fully localized: the default image is built from the checked-out fork source, and hotfix binaries are copied from the local build context — no upstream prebuilt binaries are downloaded. The upstream publishing files Dockerfile.release, Dockerfile.release.old_cpu, and docker-buildx.sh were removed, and the obsolete MinIO hotfix upload target is disabled.
• Packaging migrated to nFPM: replaced MinIO’s pkger with standard nFPM. Artifact layout and install path are unchanged (/usr/local/bin/mcli, package name mcli, YYYYMMDDHHMMSS.0.0 version scheme), but the vendor is now PGSTY, the license uses the SPDX identifier AGPL-3.0-or-later, and the Debian Section moved from empty to utils.
• RPMs are now GPG-signed: RPMs are signed offline with the maintainer key (fingerprint 9592A7BC7A682E7333376E09E7935D8DB9BD8B20). All package metadata is asserted before signing, and the signature is re-verified with checksums regenerated afterwards. DEB and APK packages carry no package-level signature; their trust anchor lives at the repository layer.
• Build provenance hardened: every previously published binary was stamped by the Go toolchain as built from a modified working tree (vcs.modified=true), which broke the link between an artifact and its Git tag. This release fixes that and adds an enforcing check to both the release and test pipelines, so every binary is traceable to an exact commit.

Security Fixes

• SUBNET credentials redacted in debug logs: with --debug enabled, SUBNET HTTP exchanges are printed in full. Previously the api-key / api_key query parameters, authentication headers, and response bodies all reached the log in clear text — and SUBNET’s authentication and registration endpoints return API keys, licenses, and tokens in their responses. Both parameter spellings and duplicate values are now masked uniformly, sensitive response headers are redacted, and SUBNET response bodies are excluded from debug dumps entirely. The leak is inherited from upstream and present in every previous release, upstream mc included: if you have ever shared --debug output of SUBNET commands (mcli license ... / mcli support ...), treat the API keys and licenses in it as exposed and rotate them.
• Redaction isolated from caller state: debug tracing now dumps copies of the request and response, so redaction cannot mutate objects the caller still holds. Zero-length, fixed-length, and unknown-length response bodies are all covered, and callers can still read the response normally.

Dependency Updates

This cycle’s dependency work is security maintenance, not routine hygiene: every bump below except the term / mod / sync / tools refresh closes at least one published advisory in the Go vulnerability database, and govulncheck reports zero known vulnerabilities reachable from this release’s code. No security advisory has ever been published for minio/mc, minio-go, madmin-go, or minio/pkg themselves.

• Go build baseline upgraded from 1.26.2 to 1.26.5 (the newest 1.26.x at release time), picking up the 1.26.3–1.26.5 security batches — including GO-2026-4970 (symlink-based root escape in os) and GO-2026-5856 (Encrypted Client Hello privacy leak in crypto/tls), the two most relevant to an S3 client that writes local files and speaks TLS.
• github.com/klauspost/compress from v1.18.5 to v1.18.7 (closes GO-2026-5841).
• github.com/prometheus/prometheus from v0.310.0 to v0.311.3 (closes GO-2026-5264, GO-2026-5381, GO-2026-5710).
• google.golang.org/grpc from v1.79.3 to v1.82.1 (closes GO-2026-6061), with the genproto family refreshed alongside.
• The golang.org/x/* family refreshed across the board: crypto v0.49.0 → v0.53.0 (the 14-advisory GO-2026-5005…5033 batch), net v0.52.0 → v0.56.0 (GO-2026-5025…5030 and GO-2026-5942), sys v0.42.0 → v0.46.0 (GO-2026-5024), text v0.35.0 → v0.39.0 (GO-2026-5970), plus term, mod, sync, and tools.
• Removed aead.dev/minisign and github.com/minio/selfupdate, and synchronized the third-party credits file.

Engineering and Delivery

• Integration test dependencies pinned: CI no longer downloads the MinIO server from a mutable upstream URL. It now uses a versioned pgsty/minio release archive verified by its SHA-256 digest, with Go pinned to 1.26.5.
• Release pipeline verification: a packaging validation workflow compares the binary inside all three package formats byte-for-byte against the build output, and checks package names, checksums, architecture fields, and every metadata field. The expected RPM metadata is sourced from the signing script itself, so configuration drift cannot strand a release part-way through signing.
• CI supply-chain hardening: every GitHub Action is pinned to a commit SHA with dependabot keeping them current, workflow permissions are narrowed to read-only, and a stale workflow pointing at the upstream organization’s project board was removed.
• Documentation: the English and Chinese READMEs now state this fork’s distribution channels and self-update policy explicitly, and installation instructions that would silently install upstream mc were removed.

Related Commits

• 9603ee3: fix: redact SUBNET secrets in HTTP debug logs
• f6ae2b0: fix: disable self-update in Pigsty builds
• c05a6e4: build: update Go deps and toolchain to 1.26.5
• 1f105aa: build: use local fork artifacts for containers
• 1182da5: ci: pin fork integration test dependencies
• 9ee207f: docs: clarify Pigsty fork distribution channels
• 0686cd8: fix: isolate SUBNET debug redaction
• ad10a2a: build: complete local Docker context isolation
• 5f54221: docs: update mc README and cn version
• 02c0305: build: migrate release packaging to nFPM
• 4c4dcc4: build: harden release provenance and package metadata

2.8 - Silo 20260618 Released

Published: 2026-06-18 · Version: RELEASE.2026-06-18T00-00-00Z

This release is a security and dependency-maintenance update for the pgsty/minio fork. It hardens LDAP STS throttling, completes S3 Select oversized-record enforcement, removes the obsolete ReadMultiple internode storage-REST API, upgrades the Go build baseline to 1.26.4, and refreshes Go module dependencies to pick up additional third-party security fixes.

Major Changes

• Remove the obsolete ReadMultiple storage-REST API: the legacy /rmpl internode endpoint is removed rather than patched in place, including its route, handler, client wrapper, storage interfaces, xlStorage methods, generated datatypes, and related metric. No production caller is expected after upstream multipart handling moved to ReadParts, but clusters should still run a consistent release during rolling upgrades.
• Complete S3 Select oversized-record enforcement: JSON Lines input now uses the bounded reader path, so oversized records are rejected consistently instead of bypassing limits on SIMD-capable CPUs. S3 Select stream errors now preserve the intended error code and wrap JSON parser failures as JSONParsingError.
• Harden LDAP STS rate-limit source bucketing: throttling is now keyed only by source IP, avoiding username-shared buckets that could be drained by one client to lock out a legitimate user. Trusted-proxy handling now resolves X-Forwarded-For from right to left, rejects catch-all trusted-proxy CIDRs, ignores RFC 7239 Forwarded, and documents the X-Real-IP deployment contract.
• Refresh the Go runtime and module baseline: release, hotfix, goreleaser, and old-CPU Docker builds now use golang:1.26.4-alpine; go.mod is updated to Go 1.26.4; and dependencies are refreshed across NATS, Prometheus, Azure SDK, Apache Thrift, gRPC, OpenTelemetry, Google API/auth, Go x/*, and related transitive libraries.

Direct Security Fixes

• CVE-2026-42600: remove the obsolete ReadMultiple storage-REST API to close the legacy internode file-read path exposed through /rmpl.
• CVE-2026-39414: complete oversized S3 Select record enforcement for JSON Lines inputs and preserve correct S3 Select error semantics.
• CVE-2026-33419: further harden LDAP STS rate-limit accounting and trusted-proxy source-IP handling.

Dependency Security Updates

• Update github.com/Azure/go-ntlmssp from v0.1.0 to v0.1.1, fixing CVE-2026-32952, where malformed NTLM challenges could panic a Go process.
• Update github.com/apache/thrift from v0.22.0 to v0.23.0, fixing CVE-2026-41602 in the Go TFramedTransport implementation.
• Update github.com/nats-io/nats-server/v2 from v2.11.1 to v2.11.15, absorbing the NATS 2.11.x security patch line. Notable fixes include pre-auth WebSocket and leafnode denial-of-service issues, MQTT authorization issues, JetStream management API authorization hardening, credential exposure fixes, and request identity-spoofing fixes, including CVE-2026-27889, CVE-2026-29785, CVE-2026-33217, CVE-2026-33218, CVE-2026-33222, and CVE-2026-33247.
• Update github.com/prometheus/prometheus from v0.310.0 to v0.311.3, absorbing Prometheus security fixes for remote-read denial of service, stored XSS in UI surfaces, and remote-write configuration secret exposure, including CVE-2026-42154, CVE-2026-44903, CVE-2026-42151, and CVE-2026-40179.
• Upgrade the release build baseline through Go 1.26.4 and refresh supporting Go module families, including golang.org/x/crypto, golang.org/x/net, golang.org/x/sys, golang.org/x/text, google.golang.org/grpc, and OpenTelemetry. These updates keep the fork aligned with patched upstream dependency baselines even where the previously pinned version was already past the specific public advisory range.

Related Commits

• 5e40665: fix: harden LDAP STS rate-limit source bucketing
• fd69c89: fix: complete CVE-2026-39414 S3 Select record limit enforcement
• 73ac524: fix: CVE-2026-42600 remove ReadMultiple storage-REST API
• df627ff: fix: bump Go toolchain to 1.26.4
• 3e61b1d: chore: update Go module dependencies

2.9 - Silo 20260417 Released

Published: 2026-04-17 · Version: RELEASE.2026-04-17T00-00-00Z

This release focuses on security hardening and compatibility tightening. It bundles fixes across OIDC, LDAP STS, S3 Select, replication metadata handling, unsigned-trailer flows, the Snowball upload path, and multiple dependency- and Go toolchain-related security issues, while also incorporating the LDAP TLS regression fix and a cleanup of community-fork documentation.

Major Changes

• Tighten the identity-authentication flow: OIDC / WebIdentity now accepts only asymmetrically signed ID Token values backed by the IdP JWKS; symmetrically signed tokens such as HS256 are no longer accepted. LDAP STS also now hides the distinction between unknown-user and bad-password failures to reduce username-enumeration risk.
• Update LDAP STS rate limiting: limits now apply to both source IP and normalized username, and successful requests no longer consume quota incorrectly. By default MinIO now uses only the socket peer address as the source and no longer trusts X-Forwarded-For, X-Real-IP, or Forwarded; to rate-limit by real client IP, configure MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES explicitly.
• Make upload and write paths stricter: presigned query parameters can no longer be combined with unsigned-trailer PUT or multipart uploads. Snowball auto-extract now also performs full signature validation on the unsigned-trailer path and rejects anonymous or forged-signature requests.
• Prevent replication metadata spoofing: internal X-Minio-Replication-* headers attached to ordinary PUT / COPY requests are now rejected or ignored, and only trusted replication flows may write the related internal metadata.
• Clarify S3 Select error semantics: oversized CSV and line-delimited JSON records now return OverMaxRecordSize directly instead of the generic InternalError; clients or alerting rules that depend on the old error code should be adjusted.
• Upgrade the runtime and dependency baseline: fix the regression where ldaps:// did not correctly apply TLS settings, replace minio/pkg/v3 with pgsty/minio-pkg/v3, and pin several critical dependencies that are prone to breaking changes. The release also upgrades go-jose, go.opentelemetry.io, and Go 1.26.2 to unify the build and release baseline.
• Refresh documentation and security guidance: update SECURITY.md, VULNERABILITY_REPORT.md, docs/sts/ldap.md, and related documents, add a security advisory index, and switch upstream minio/minio references in the security guidance over to pgsty/minio.

Fixed CVEs

• CVE-2026-34986: upgrade go-jose to v4.1.4 and fix known security issues in the JWT / JOSE dependency chain.
• CVE-2026-39883: upgrade the go.opentelemetry.io dependency stack to fix the PATH-hijacking risk.
• CVE-2026-33322: restore the strict JWKS-only OIDC JWT verification path to block keyring injection and algorithm-confusion risk.
• CVE-2026-33419: systematically harden LDAP STS authentication, rate limiting, source-address identification, and accounting logic across four follow-up fixes.
• CVE-2026-34204: reject injection of X-Minio-Replication-* metadata by untrusted requests to prevent objects from being written with invalid replication state.
• CVE-2026-39414: reject oversized S3 Select records early to avoid continued buffering and parsing of abnormal inputs.
• GHSA-hv4r-mvr4-25vw: close the unsigned-trailer query-auth bypass.
• GHSA-9c4q-hq6p-c237: harden unsigned-trailer authentication and signature validation in Snowball auto-extract scenarios.
• CVE-2026-32280, CVE-2026-32281, and CVE-2026-32283: upgrade Go to 1.26.2 and absorb the upstream toolchain and stdlib security fixes.

Related Commits

• c878ca0: fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
• e970ec5: fix: upgrade go-jose to v4.1.4 to patch CVE-2026-34986
• a206510: fix: CVE-2026-39883 upgrade go.opentelemetry.io
• fd65f11: merge: PR #18 upgrade go-jose to v4.1.4 for CVE-2026-34986
• bc087e4: merge: PR #19 upgrade go.opentelemetry.io for CVE-2026-39883
• f1f2239: fix: CVE-2026-33322 restore JWKS-only OIDC JWT verification
• 6619d0c: fix: CVE-2026-33419 harden LDAP STS auth
• fcb8f24: fix: CVE-2026-34204 reject untrusted replication metadata
• c5765dc: fix: CVE-2026-39414 reject oversized S3 Select records
• fa7c579: fix: GHSA-hv4r-mvr4-25vw block unsigned-trailer query auth bypass
• b50ab58: fix: GHSA-9c4q-hq6p-c237 harden Snowball unsigned-trailer auth
• 9a4b3cd: fix: CVE-2026-32280/CVE-2026-32281/CVE-2026-32283 upgrade Go to 1.26.2
• c55b52c: fix: CVE-2026-33419 preserve LDAP STS rate limits on success
• 817a457: fix: CVE-2026-33419 harden LDAP STS rate-limit source IP
• 084a154: fix: CVE-2026-33419 tighten LDAP STS rate-limit accounting
• 16e34f9: docs: refresh security guidance and fork references

2.10 - Silo 20260325 Released

Published: 2026-03-25 · Version: RELEASE.2026-03-25T00-00-00Z

This is a maintenance release centered on packaging, stability, and security disclosure. It improves the shipping artifacts, fixes an LDAP TLS regression, and explicitly documents the secure dependency set carried by the release.

Major Changes

• Bundle mcli/mc into the Docker image and add checksum verification for a better out-of-the-box image experience.
• Fix the LDAP TLS regression affecting ldaps:// deployments so TLS settings are correctly honored.
• Remove inherited upstream CI/CD workflows that are no longer used in the community-maintained fork.
• Pin several critical dependencies to avoid further fallout from upstream breaking changes.

Fixed CVEs

• CVE-2026-24051: the release notes explicitly call out go.opentelemetry.io/otel/sdk v1.42.0, which avoids the macOS PATH-hijacking arbitrary code execution issue.
• CVE-2025-10543: the release notes explicitly ship github.com/eclipse/paho.mqtt.golang v1.5.1, fixing incorrect MQTT packet encoding for oversized UTF-8 strings.
• CVE-2025-58181: the release notes explicitly ship golang.org/x/crypto v0.49.0, fixing unbounded memory consumption in ssh GSSAPI authentication handling.

Related Commits

• f2f9a40: add mcli/mc from pgsty/mc to Docker image
• ce1c537: fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
• ee55e53: remove upstream CI/CD workflows inherited from minio/minio

2.11 - Silo 20260321 Released

Published: 2026-03-21 · Version: RELEASE.2026-03-21T00-00-00Z

This maintenance release is built around the Go 1.26.1 upgrade and a broad dependency refresh. Beyond stricter compiler and linter compatibility fixes, it also delivers the most substantial security dependency refresh in the current release line.

Major Changes

• Upgrade the build environment from Go 1.26.0 to Go 1.26.1.
• Refresh direct and indirect dependencies to converge on the newer toolchain.
• Fix linter and test issues exposed by the stricter Go 1.26.1 checks.

Fixed CVEs

• CVE-2026-27137: Go stdlib 1.26.0 -> 1.26.1 fixes incomplete email-constraint enforcement in crypto/x509.
• CVE-2026-27138: Go stdlib 1.26.0 -> 1.26.1 fixes a crypto/x509 panic triggered by malformed certificates.
• CVE-2026-25679: Go stdlib 1.26.0 -> 1.26.1 fixes insufficient validation of IPv6 host literals in net/url.
• CVE-2026-27139: Go stdlib 1.26.0 -> 1.26.1 fixes FileInfo metadata escaping the Root boundary in os.
• CVE-2026-27142: Go stdlib 1.26.0 -> 1.26.1 fixes missing URL escaping in html/template for meta refresh content.
• CVE-2026-26958: filippo.io/edwards25519 v1.1.0 -> v1.2.0 fixes incorrect or undefined MultiScalarMult behavior.
• CVE-2025-10543: github.com/eclipse/paho.mqtt.golang v1.5.0 -> v1.5.1 fixes incorrect MQTT packet encoding for oversized UTF-8 strings.
• CVE-2026-24051: go.opentelemetry.io/otel/sdk v1.38.0 -> v1.42.0 fixes the macOS PATH-hijacking arbitrary code execution issue.
• CVE-2026-33186: google.golang.org/grpc v1.77.0 -> v1.79.3 fixes authorization bypass caused by a missing leading slash in the HTTP/2 :path pseudo-header.

Related Commits

• 5abd9a8: bump golang to 1.26.1 and update deps
• 377fc61: fix: satisfy stricter Go 1.26.1 linter checks

2.12 - Silo 20260314 Released

Published: 2026-03-14 · Version: RELEASE.2026-03-14T12-00-00Z

This release switches the project to the community-maintained Console fork and performs a sizeable dependency refresh to establish the base for the later Go 1.26.x maintenance releases.

Major Changes

• Switch to the community-maintained georgmangold/console v1.9.1 fork in place of the unmaintainable upstream Console dependency.
• Refresh a large portion of the direct and indirect dependency graph so the new Console and toolchain combination builds cleanly.
• Fix the go vet format directive issue in grid_test.go and adjust tests for the HTTP behavior changes in Go 1.26.

Fixed CVEs

• CVE-2025-47913: golang.org/x/crypto v0.37.0 -> v0.46.0 fixes a panic in ssh/agent when handling malformed responses.
• CVE-2025-58181: golang.org/x/crypto v0.37.0 -> v0.46.0 fixes unbounded memory consumption in ssh GSSAPI authentication parsing.
• CVE-2025-47914: golang.org/x/crypto v0.37.0 -> v0.46.0 fixes a panic in ssh/agent caused by malformed identity messages.
• CVE-2025-47911: golang.org/x/net v0.39.0 -> v0.48.0 fixes quadratic parsing complexity in html.Parse for crafted inputs.
• CVE-2025-58190: golang.org/x/net v0.39.0 -> v0.48.0 fixes an infinite parsing loop in golang.org/x/net/html.

Related Commits

• 68521b3: add github ci/cd pipeline
• 00f3cf7: RELEASE.2026-03-14T12-00-00Z with go 1.26.0

2.13 - Silo 20260214 Released

Published: 2026-02-14 · Version: RELEASE.2026-02-14T12-00-00Z

This early infrastructure-focused community release restores the embedded Console, introduces GitHub CI/CD, and lifts the Go baseline to 1.26.0, which also absorbs a batch of security fixes from the older toolchain generation.

Major Changes

• Restore the embedded Console and refresh the README to clarify the community fork position.
• Add GitHub CI/CD workflows as the base for automated builds and multi-platform delivery.
• Add quick links for docs, Docker, the GitHub repository, and installation through the pig package manager.

Fixed CVEs

These issues were absorbed as part of the Go 1.25.5 -> 1.26.0 upgrade:

• CVE-2025-68121: crypto/tls could incorrectly accept mutated CA configuration during session resumption.
• CVE-2025-61730: TLS 1.3 could process handshake messages incorrectly across encryption-level boundaries.
• CVE-2025-61726: net/url query parsing could be abused for memory exhaustion.
• CVE-2025-61728: archive/zip could consume excessive CPU while building archive indexes.
• CVE-2025-68119: cmd/go could trigger unexpected code execution when invoking external VCS tooling.
• CVE-2025-61731: the #cgo pkg-config: directive could be abused for arbitrary file writes.
• CVE-2025-61732: cmd/cgo comment parsing discrepancies could enable code smuggling.

Related Commits

• 8630937: Restore embedded console and update README for community fork
• 5d57938: add github ci/cd pipeline

2.14 - Silo 20251203 Released

Published: 2025-12-15 · Version: RELEASE.2025-12-03T12-00-00Z

This is the earliest traceable community release. Its purpose is to establish the community packaging and distribution baseline rather than to deliver incremental fixes over an earlier community release.

Major Changes

• Build the community packaging flow around minio/pkger.
• Choose a maintenance-mode upstream MinIO baseline as the starting point for the community-maintained fork.
• Produce the first apk, deb, and rpm artifacts for ongoing community releases.

Fixed CVEs

• This is the first community release. The GitHub Release does not provide a delta-style security-fix list against an earlier community version, and this page does not attempt to reconstruct the full historical CVE delta against the upstream maintenance baseline.

Related Commits

• d4cd4b4: RELEASE.2025-12-03T12-00-00Z with go 1.25.5

3 - SILO Security Chronicle

This is the security chronicle of the SILO community fork, listed from newest to oldest. Each CVE has its own article: the original threat model, the turns taken during review, the rejected alternatives, the final invariant, the evidence, and the compatibility cost all stay with that incident.

3.1 - CVE-2026-32285: The jsonparser Advisory That Required No Patch

Status: Closed without a code change
GitHub issue: pgsty/minio#26

Security maintenance is not always a sequence of “find a vulnerability, then ship a patch.” The initial assessment of CVE-2026-32285 was that the repository might still carry a vulnerable jsonparser; replacing the dependency or maintaining another fork was even considered. Checking the resolved module version and actual reachability changed the conclusion: the tree already used v1.1.2, which contained the fix, and govulncheck found no reachable vulnerable symbol.

The right final action was not to manufacture an upgrade. It was to record the evidence and close the issue.

What was wrong with the initial assumption

The issue was first understood as “this dependency has no fixed version.” Acting on that premise without verification could have produced several changes that looked proactive but made the project worse:

• changing the dependency graph for no security benefit;
• introducing compatibility regressions in the name of a nonexistent fix;
• adding another fork that would need long-term maintenance;
• implying that previous SILO releases were demonstrably exposed when that had not been established.

Security work cannot be measured by whether it produces a diff. Leaving correct code unchanged is itself a security decision, and it needs evidence.

Investigation

The investigation narrowed the question through four layers of evidence:

1. Confirm the version actually selected in the current go.mod and go.sum graph.
2. Check the upstream release and establish that v1.1.2 already contained the relevant fix.
3. Run and inspect govulncheck; it reported no reachable vulnerable symbol.
4. Attribute the discrepancy in the issue to stale advisory or vulnerability-database information, not to a vulnerability still present in the source tree.

Four claims must remain separate: a version was once listed as affected, a package is imported, a vulnerable symbol is reachable in the program, and remote input can actually exploit that path. None of them proves the others.

Why there was no “just in case” upgrade

If the selected version already includes the fix, bumping to an arbitrary newer version does not make the system safer. It only expands the change surface and makes later regressions harder to attribute. That is especially risky in a large Go module graph.

The final decision was therefore to:

• avoid committing a fictitious fix;
• preserve the version and reachability evidence in the issue;
• keep version gates and govulncheck in place to detect a future dependency rollback;
• treat “no change required now” as a dated conclusion, not a permanent exemption.

Verification boundary

This incident established that the checkout examined on 2026-04-15 did not require a code change for CVE-2026-32285. It does not establish that every future branch, module graph, or release will remain unaffected. A dependency downgrade or a change in module selection requires the version and reachability checks to be repeated.

This article records the investigation and the basis for closure. The original govulncheck was not rerun while preparing this chronicle.

The principle this incident left behind

The objective of security maintenance is an accurate risk conclusion, not a patch for every issue. For a dependency CVE, ask in order: which version is actually selected, whether the vulnerable code enters the program, whether the symbol is reachable, and whether a deployed entry point makes it exploitable. Only when those answers require a source change should the investigation produce a diff.

3.2 - CVE-2026-33322: OIDC JWT Algorithm Confusion

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Affected entry points: AssumeRoleWithWebIdentity, AssumeRoleWithClientGrants
GitHub issue: pgsty/minio#22

The old implementation placed the OIDC client secret in the JWT verifier keyring while also accepting HMAC signing methods. An attacker who knew that client secret could therefore mint an HS-signed token and exchange it through STS for temporary credentials. The final fix restored asymmetric, JWKS-only verification. It deliberately broke HS256/384/512 compatibility instead of keeping an option that would reintroduce ambiguous trust semantics.

The vulnerability was not the absence of signature verification

At first glance, the old code did verify JWT signatures. The boundary that failed was more specific: which kind of key the verifier would accept, and whether the token header could select an algorithm with semantics that did not match that key’s intended role.

The attack chain required several conditions:

• the attacker obtained the OIDC client secret;
• the attacker constructed an HMAC-signed ID token;
• the verifier treated the client secret as an HMAC signing key;
• the token reached the WebIdentity or ClientGrants STS flow and was exchanged for temporary credentials.

Disclosure of a client secret is already serious, but it should not automatically confer the power to issue arbitrary user ID tokens. Combining those capabilities in one keyring created the algorithm-confusion vulnerability.

A compatibility path was implemented, then deliberately removed

During the fix, an allow_hmac-style compatibility path was implemented. It appeared reasonable: keep the secure default while letting users with a real requirement opt in. But retaining a shared secret in the general verifier keyring meant administrators would need to understand that the option expanded the entire STS trust boundary. Any future drift in the method allowlist could reopen the flaw.

The trade-off became clear:

The most important decision was not what code was added, but that a completed compatibility implementation was removed.

Final invariants

The fix was concentrated in the OIDC JWT verification path and established four rules:

• verifier keys come only from the identity provider’s JWKS;
• the OIDC client secret never enters the JWT verification keyring;
• HS256, HS384, and HS512 are always rejected;
• the ordinary RS256 flow and JWKS refresh/retry use the same method allowlist.

The CVE was not used as a pretext for expanding JOSE support. PS256 and EdDSA remained out of scope.

Verification and release

The development record includes HS256 rejection, RS256 acceptance, JWKS refresh/retry regression tests, and focused go test ./internal/config/identity/openid. Temporary compatibility helpers, configuration, and tests were all removed from the final diff.

The public fix is f1f2239, released with SILO 2026-04-17. This article records the historical verification; those tests were not rerun while preparing the chronicle.

Compatibility cost

This is an explicit breaking change. Identity providers that still issue HS256/384/512 tokens must migrate to JWKS-backed RSA or ECDSA before upgrading SILO. The project chose a narrower trust model that is easier to explain and audit over preserving a configuration that happened to work before.

3.3 - CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain

Status: Released, followed by two rounds of corrections
First containing release: RELEASE.2026-04-17T00-00-00Z
Complete correction: RELEASE.2026-06-18T00-00-00Z
GitHub issue: pgsty/minio#23

The core vulnerability was straightforward: LDAP STS returned different results for “user does not exist” and “password is wrong,” creating a username oracle. The first fix unified the external authentication failure and added limits by source IP and username. Continued review then showed that success refunds, spoofable source headers, reservation accounting, and the shared username bucket could turn the security control itself into a new attack surface.

The final June design removed the username bucket that enabled precise account lockout, retained only the source-IP bucket, and made proxy attribution an explicit deployment contract.

Initial threat model

The entry point is AssumeRoleWithLDAPIdentity. An attacker needs no existing MinIO account. Access to the LDAP STS endpoint is enough to compare the code, status, or message returned for an unknown user and a wrong password, enumerate valid usernames, and combine that knowledge with password spraying, guesses about organizational naming, or social engineering.

The fix could not simply disguise every error as “wrong password.” LDAP connection, lookup-bind, and directory-service failures still needed to appear as infrastructure errors, or operators would lose the ability to diagnose the service.

First round: uniform responses and a limiter

The initial fix on 2026-04-15 did three things:

• unknown user and bad password returned the same external STS authentication error;
• LDAP infrastructure failures still returned 500, with the real cause retained in the server log;
• a new in-memory limiter initially created buckets for both source IP and normalized username.

This closed the content side channel and raised the cost of brute-force attempts, but the limiter state machine and source attribution exposed more problems.

Second round: success, attribution, and accounting

The follow-up changes on April 16 addressed three classes of defects:

1. Successful authentication must not consume the failure allowance; the reserve/commit/cancel/refund lifecycle had to be explicit.
2. The socket peer must be used by default; X-Forwarded-For, X-Real-IP, and Forwarded cannot be trusted merely because a request supplies them.
3. Refund and capacity need hard bounds so cancel logic cannot mint tokens.

A proxy must be placed on an explicit trusted allowlist before it can influence source attribution.

Third round: remove the username bucket

Adversarial review in June overturned the intuition that “source plus username must be stronger than source alone.” A shared username bucket could be exhausted continuously from arbitrary origins. With only a low request rate, an attacker could lock a targeted account before the legitimate user ever reached an LDAP bind.

The final fix therefore:

• removed the per-username bucket;
• peeled trusted hops from XFF right to left and selected the first untrusted address;
• rejected trusted-proxy footguns such as 0.0.0.0/0 and ::/0;
• stopped using Forwarded for security-sensitive bucketing;
• allowed X-Real-IP only under a contract in which the proxy overwrites rather than forwards client input.

This turn in the review shows that a security control needs its own threat model. More dimensions of throttling do not automatically mean more security.

Rejected alternatives

Verification and release

The historical record covers limiter reserve/commit/cancel/refund behavior, concurrency, success and infrastructure failures, external equivalence of unknown-user and bad-password responses, and RemoteAddr, spoofed-header, trusted-proxy, multi-hop, and catch-all-CIDR cases. Focused package tests and builds were recorded as passing.

The LDAP security end-to-end test skips when _MINIO_LDAP_TEST_SERVER is absent, so an outer ok cannot be presented as proof of the full LDAP scenario.

The first public fix was 6619d0c. Follow-up corrections include c55b52c, 817a457, 084a154, and 5e40665.

Final cost and residual risk

• The limiter now keys only on source IP and gives up a hard per-account throttle across different origins.
• It is per-node and in-memory, not a cluster-wide password defense.
• Botnets, distributed origins, IPv6 address rotation, and LDAP bind timing remain.
• Incorrect trusted-proxy configuration can still destroy source attribution.
• A Forwarded-only deployment falls back to the peer bucket and loses granularity.

Rate limiting can reduce attempts from one source. The uniform external authentication response is what actually conceals whether a username exists.

3.4 - CVE-2026-34204: Replication Metadata Injection

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub issue: pgsty/minio#24

Ordinary PUT and COPY requests could smuggle X-Minio-Replication-* headers into internal X-Minio-Internal-* SSE metadata, creating objects whose replication state did not match the authorized path and could even make them unreadable. The final fix stopped accepting replication-only metadata by default, restored it only in a trusted flow authorized for ReplicateObjectAction, and sanitized CopyObject before any header consumer ran.

Threat model

An attacker needed only ordinary object-write permission, not internode credentials. The input came entirely from client-controlled X-Minio-Replication-* headers, but metadata extraction converted it into internal replication or SSE state.

Later read paths interpreted the object according to that false internal state. The result could be an unreadable object: an integrity and availability failure. Almost every production server accepting untrusted writes needed to be treated as affected.

The root problem was not the header name. It was that data from an untrusted source acquired internal semantics without passing replication authorization.

Reject the whole request, or sanitize precisely?

Rejecting an ordinary request whenever it contains a replication header is the most obvious fix. It would also turn a header clients were previously allowed to send and have ignored into a hard failure. The final design was more precise:

• the default extraction path does not accept replication-only metadata;
• ordinary PUT and COPY strip those fields first;
• only a path authorized for ReplicateObjectAction restores them;
• replica-status writes use the same trusted condition;
• legitimate multipart and Snowball replication flows explicitly restore the SSE metadata they require.

That keeps the compatibility change inside internal semantics instead of expanding it to every client carrying an extraneous header.

Why CopyObject had to sanitize early

CopyObject headers are not used only for the final metadata map. They can be consumed earlier by precondition logic and SSE-C source handling. Removing them immediately before the object write is too late: earlier consumers have already been contaminated.

The final sanitization occurs before those consumers. “Untrusted replication headers never enter internal semantics” becomes one invariant instead of a convention every downstream function must remember to enforce.

Implementation and verification

The change covered handler utilities, object handlers, and multipart handlers, with tests at several layers:

• trusted and untrusted metadata extraction at the helper layer;
• malicious PUT and COPY cases at the handler layer;
• CopyObject header sanitization;
• red/green comparison between the vulnerable parent and the patched tree;
• live-server before/after behavior showing that a malicious header no longer made an object unreadable;
• continued operation of legitimate replication, multipart, and Snowball flows.

The public fix is fcb8f24. This article preserves the historical verification boundary; no live server was started again while preparing the chronicle.

Cost and residual risk

• Internal replication headers supplied by ordinary clients are now ignored or stripped.
• Replication-only metadata must be restored explicitly inside an authorized branch.
• If a future replication entry point forgets to restore it, the result should be a functional regression rather than another untrusted write path.
• The audit focused on replication headers; it does not establish that every X-Minio-Internal-* field has undergone the same trust review.

This incident leaves a simple review question: a field that looks “internal” is not necessarily trusted. Ask where it came from and which authorization decision allowed it to acquire internal meaning.

3.5 - CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass

Status: Released; the second-round fix was completed in June
Initial fix release: RELEASE.2026-04-17T00-00-00Z
Complete fix release: RELEASE.2026-06-18T00-00-00Z
GitHub issue: pgsty/minio#25

The first fix in April reused the existing 1 MiB maxCharsPerRecord limit for both CSV and ordinary JSON Lines. This prevented unbounded buffering while waiting for a delimiter and returned the explicit OverMaxRecordSize error to clients. A June review then found that CPUs with SIMD support took a different simdjson fast path that bypassed the limit completely.

The final solution sent JSON Lines through the bounded reader on every CPU. The same review also corrected error mapping, parser errors, and the flush of completed records before a terminal error. SILO temporarily gave up the SIMD fast path in exchange for consistent security semantics.

Threat model

An attacker can submit or query an object containing an extremely long single record. The reader continues buffering until it sees a record delimiter, allowing memory and CPU denial of service. More subtly, the same input can select a different implementation according to the machine’s CPU features. Safe behavior on a test machine does not necessarily prove safe behavior in production.

Error semantics are part of the fix. If an oversized record appears only as a generic InternalError, clients and monitoring systems cannot distinguish an enforced security limit from a server failure.

First round: reuse the existing 1 MiB invariant

The first patch did not invent a new configuration knob. It applied the existing maxCharsPerRecord = 1 MiB rule:

• the CSV splitter and line-delimited JSON rejected oversized records before buffering or parsing them further;
• the earliest splitter error was preserved instead of being overwritten by a partial decode;
• the error propagated as OverMaxRecordSize rather than collapsing into InternalError.

This was a deliberate compatibility contraction. Clients with lines or records larger than 1 MiB now had to split their input.

Second round: a hardware-dependent bypass

Following the call chain again in June exposed this path:

JSON Lines -> simdj.NewReader -> simdjson.ParseNDStream

When simdjson.SupportedCPU() returned true, JSON Lines bypassed the bounded json.PReader. The third-party parser kept reading past a chunk boundary until it found a newline. A generic reader wrapper could not simultaneously preserve already completed records and guarantee a bound on the next record.

The final choice was not another wrapper. JSON Lines temporarily stopped using the SIMD path and always used the bounded PReader. If SIMD support returns, that implementation must enforce the same record limit itself and pass the same CPU-independent regression suite.

Stream semantics corrected in the same round

The review also fixed several adjacent behaviors:

• use errors.As to pass through errors implementing SelectError, not just one concrete type;
• have the JSON worker wrap parser failures as JSONParsingError;
• flush completed records still waiting below the batch threshold before emitting a terminal error event;
• preserve error priority in input order instead of letting a later oversized record overwrite an earlier parse error.

Those details determine whether a client sees the correct failure or a resource-limit fix that quietly broke the streaming protocol.

Deliberately left outside this CVE

• The historical mismatch between CSV AllowQuotedRecordDelimiter and the outer physical-newline splitter.
• Whether \r in CRLF counts toward the record length.
• Restoring SIMD performance without an equivalent bound.

These questions may be real, but they require independent AWS-compatibility evidence or a more complex quote-aware splitter. They did not belong in a security patch based on guesses.

Verification and release

The historical record includes oversized JSON Lines, error-code preservation, and behavior tests that do not depend on the local machine’s SIMD capabilities. go test ./internal/s3select/... -count=1 and git diff --check were recorded as passing.

The initial public fix was c5765dc; the complete June correction is fd69c89. Those tests were not rerun while preparing this article.

Final trade-offs

• JSON Lines performance may decrease; this incident did not produce a benchmark that quantifies it.
• The 1 MiB per-record limit rejects oversized input accepted by previous releases.
• Quoted, multiline CSV semantics still need separate work.
• Any future CPU-specific fast path must share the slow path’s security tests.

The second fix leaves the central lesson: a security invariant must hold across hardware-dependent paths. A green test on one CPU does not prove that another execution engine is protected.

3.6 - CVE-2026-40344: Snowball Auto-Extract Authentication Bypass

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub advisory: GHSA-9c4q-hq6p-c237

Snowball’s PutObjectExtractHandler omitted the streaming unsigned-trailer authentication case. A tar stream with a forged signature could enter untar() before authentication completed, and one request could fan out into many object writes. The final fix initialized the correct reader, handled the decoded length, and completed SigV4 verification before any tar byte reached the extractor.

Why the identifier changed

The official CVE had not been assigned when the fix was written, so the commit subject used the temporary identifier fake CVE-2026-40028. The final identifier is CVE-2026-40344. The historical commit was not rewritten; the advisory and this article use the official number.

From one missing authentication case to bulk object writes

The entry point was Snowball / PutObjectExtract auto-extraction. The request used unsigned-trailer streaming, an authentication type the old handler did not cover as ordinary PUT did.

The danger was larger than one incorrectly authorized request. Once the tar stream entered untar(), that request could create multiple attacker-chosen objects. An authentication omission was therefore amplified into a bulk-write problem.

Final invariant: the extractor sees zero bytes on failure

The key statement in the fix was:

If authentication ultimately fails, untar() must have seen zero bytes.

That rule excludes “extract first, then roll back if authentication fails.” Object writes travel through several paths, and proving a complete rollback is much harder than proving that input never crossed the boundary. Authentication had to close before data entered the extractor.

Implementation

The final change:

• recognized authTypeStreamingUnsignedTrailer;
• read X-Amz-Decoded-Content-Length;
• used newUnsignedV4ChunkedReader();
• performed complete SigV4 request verification before entering untar();
• preserved valid signed Snowball requests and CRC32 trailer flows.

Verification

The historical commit and investigation record cover:

• rejection of a forged-signature Snowball unsigned-trailer request;
• rejection of anonymous Snowball writes to a non-public bucket;
• successful extraction with a valid signature and trailing CRC32;
• red/green comparison between the vulnerable parent and the patched tree;
• containerized before/after smoke tests.

The public fix is b50ab58. The container tests were not rerun while preparing this article.

Compatibility and residual risk

• Clients that relied on an unsigned-trailer Snowball combination whose signature was never really verified will fail after upgrading.
• Authentication now closes before extraction, but tar paths, archive-size limits, and object-count limits remain separate security surfaces.
• Snowball and ordinary unsigned-trailer requests now share a reader; future changes must regress both paths together.

The essence of the fix was not another if. It moved the authentication decision in front of the actual amplification boundary.

3.7 - CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub advisory: GHSA-hv4r-mvr4-25vw

Query-string SigV4 credentials could enter a STREAMING-UNSIGNED-PAYLOAD-TRAILER data flow, while the old code verified the signature only when an Authorization header was present. A request carrying a valid access-key identifier could therefore complete a write without a correct signature.

The final fix moved presigned rejection and SigV4 verification into newUnsignedV4ChunkedReader(), making every caller consuming that stream share one authentication boundary.

Identifier note

The official CVE had not been assigned when the patch was written, so its commit subject used fake CVE-2026-40027. The final identifier is CVE-2026-41145. The historical commit remains unchanged; public material uses the official identifier.

Root cause: authentication was coupled to transport form

The affected entry points included PutObject and PutObjectPart. The request selected STREAMING-UNSIGNED-PAYLOAD-TRAILER, with its credentials and signature in the query string rather than the Authorization header.

The old handler used header presence to decide whether to verify a signature. The body reader still consumed the data normally, silently degrading query authentication into something close to an anonymous write. The attacker needed to know a valid access-key identifier, but did not need to produce a correct signature.

The problem was not failure to parse the query parameters. It was that authentication depended on how credentials were transported instead of the trust boundary at which the stream was consumed.

Why the patch did not live in each handler

Anonymous unsigned-trailer requests were not prohibited wholesale. If bucket policy explicitly permits anonymous writes, they can still follow the anonymous authorization path. The forbidden state is the mixture of query credentials with no verification of those credentials.

Implementation and verification

The fix performs presigned rejection and SigV4 verification at the reader entry in cmd/streaming-v4-unsigned.go, while removing the gates in the PutObject and multipart handlers that depended on header presence.

New tests cover forged query PUT, multipart, mixed authentication, and anonymous policy. The historical record also includes a vulnerable-parent write that succeeded, failure on the patched tree, and live-server before/after smoke tests showing that header-authenticated and valid anonymous flows continued to work.

The public fix is fa7c579. The live exploit was not rerun while preparing this article.

Compatibility and residual risk

• Presigned/query unsigned-trailer is now explicitly unsupported, an intentional breaking change.
• Moving the fix into the reader significantly reduces the chance that a sibling handler omits the check again.
• Other streaming authentication modes still need their own audits; this reader fix does not establish that every SigV4 streaming combination is safe.

The shape of this fix matters as much as its payload: when several handlers share an authenticated data stream, authentication belongs to the reader rather than to optional checks in each caller.

3.8 - CVE-2026-42600: ReadMultiple Storage-REST Path Traversal

Status: Released
First containing release: RELEASE.2026-06-18T00-00-00Z
GitHub advisory: GHSA-xh8f-g2qw-gcm7
Affected scope: Distributed erasure only; cluster-root / internode JWT required

The msgpack body of /rmpl carried Bucket, Prefix, and Files. The old code joined those values into filesystem paths without a containment check. The initial fix implemented full preflight validation. Continued call-chain review then found that this API had had no production caller since 2024. The final solution changed from “retain and harden” to removing the route, handler, client, interface, and generated code.

Deleting roughly a thousand lines was a larger source diff than a local validation guard, but it left a smaller long-term attack surface.

Threat model

The vulnerable route was registered only in distributed erasure mode; single-node deployments were unaffected. An attacker needed an internode JWT derived from the root secret, control of a node, or the ability to intercept unencrypted traffic between nodes.

The dangerous fields were inside the msgpack body, not the URL or form data, so upper HTTP path middleware never saw them. xlStorage.ReadMultiple joined and read the resulting paths directly, allowing them to escape the drive root.

This was not an anonymous S3 vulnerability. It crossed the boundary from “cluster root or peer” to “any file readable by the node process.”

First design: retain the API and validate it completely

The initial patch in xlStorage.ReadMultiple:

• rejected absolute paths, . and .. segments, backslashes, Windows drive prefixes, and NUL bytes;
• checked final containment across drive, volume, prefix, and file;
• preserved the historical contract for an empty Bucket and .minio.sys/multipart where possible;
• returned an error before any read or streaming began.

That design could close the known traversal, but review quickly exposed an early-return gap.

MaxResults exposed the danger of validate-as-you-use

The first patch validated each Files item inside the read loop. For Files=[good, bad] with MaxResults=1, the function returned after reading the first item and never validated the second.

It did not read the second malicious file, but it violated the intended invariant that the entire msgpack request must be valid before execution. Validation was therefore moved into a complete preflight pass, with path-length checks also completed before streaming.

The general rule is useful beyond this endpoint: when a request can return early or stream a partial response, checking an element immediately before use is not equivalent to validating the whole request.

Final decision: delete the API

Further call-chain audit established that:

• upstream removed the last production caller in September 2024;
• multipart had moved to ReadParts;
• the current tree had no in-tree production consumer;
• upstream’s final remediation also deleted ReadMultiple.

The final change removed the route, handler, client wrapper, StorageAPI / xlStorage method, metric, datatype, and generated code. storageRESTVersion retained the existing compatibility strategy.

Verification and release

The in-place validation phase ran focused tests for xlStorage, the storage-REST client, msgpack encode/decode, and path edge cases; adversarial review exposed the MaxResults flaw. The deletion phase checked the route, client, interface, generated surfaces, and absence of callers.

The public fix is 73ac524, released with SILO 2026-06-18. The post-deletion full suite was not rerun while preparing this article.

Compatibility and claim boundary

• The external S3 API is unchanged.
• Third-party implementations that privately called the internal /rmpl endpoint will stop working.
• A mixed-version rolling upgrade may encounter a protocol mismatch, so cluster nodes should be kept on the same version during the upgrade.
• Removing this endpoint proves only that ReadMultiple no longer exists; it does not establish that every internal node request carrying body paths has completed a containment audit.

This CVE reached the right final fix, but it also leaves an important distinction: closing one endpoint is not the same claim as closing an entire vulnerability class.

3.9 - Internode Path Containment Audit: Paying Off What CVE-2026-42600 Left Owing

Status: Fixed on the local pgsty/minio branch, unreleased and not disclosed (no CVE/GHSA requested; the upstream repository is archived) Affected scope: Distributed erasure only; cluster-root / internode JWT required Prerequisite reading: CVE-2026-42600 · ReadMultiple

This article contains complete exploitation vectors and measurements. Publishing it constitutes disclosure. Hold it until the fixed release ships.

The previous entry closed with this sentence:

Deleting the endpoint proves only that ReadMultiple no longer exists. It cannot be extrapolated into a completed containment audit of every internode body path.

That was an IOU, written down in plain sight. This is the record of paying it — and a not-very-flattering construction log.

Conclusions first

• Twelve defects, all inherited from upstream. Verified by per-function md5 comparison: the fork’s diff against upstream on the affected files is pure deletion, zero added lines.
• This is not a new vulnerability. It is the remainder of CVE-2026-42600 — three more protocol surfaces under the same root cause.
• Our failure is not in the code. It is in the record: a point fix was written up as a closure.
• While fixing it we introduced four regressions of our own, every one of them in a rule we invented rather than reused.

“N endpoints” was the wrong frame

Earlier audits kept counting endpoints, arriving at 19, then 21, then 22 — and missing an entire protocol surface each time. The real structure is four surfaces:

The first row matters as much as the rest: it overturns the earlier “all 21 endpoints escapable” claim. Those audits grepped for the validation helper inside handler bodies, found nothing, and concluded there was no protection — missing that the protection lives in the middleware layer.

The fourth row is the one no amount of hardening in storage-REST handlers can reach.

The root cause is three layers, not one bug

Three design facts, none wrong on its own:

1. Validation happens only at the HTTP surface. r.Form is populated from url.ParseQuery(RawQuery) and never from a body (introduced 2017).
2. Grid RPC bypasses the middleware. /minio/grid/v1 upgrades once; subsequent msgpack frames never re-enter the HTTP chain (introduced 2023).
3. The storage layer performs no containment. getVolDir rejects a volume only when it is exactly ""/./.., and pathJoin runs Clean (settled 2018).

In one sentence: the upper layer assumes the lower one validates, the lower assumes the upper already did, and neither can see the channel in between.

ReadMultiple was merely one endpoint that exercised that structure. Removing it left the structure intact.

One line of the timeline deserves singling out. The divide-by-zero in ShardFileSize has been present since 2020, but only when it moved inside xioutil.WithDeadline in 2024-10 — a change meant to fix large-object timeouts — did it escalate from “one failed request” to “the whole process exits”, because WithDeadline runs its work function on a bare goroutine that no recover() can reach. That escalation was not visible at the time.

Vectors, confirmed by execution

Every one reproduced against a real xlStorage through the real REST/grid client, with planted sentinel files. None of this is static inference.

Two of these had never been found before and are worth calling out.

RenameFile with an empty source path hits the volume-root alias and relocates the whole volume. Aimed at .minio.sys, one ordinary S3 GET afterwards yields the cluster’s IAM and configuration. It requires no traversal sequence at all — so any audit that greps for .. misses it by construction.

A negative part size floors both terms of ShardFileSize to zero. checkPart’s only integrity test is st.Size() < expectedSize, so every file that exists is reported intact, including a truncated shard. Worse, this holds whether or not the erasure parameters are valid: metadata that passes FileInfo.IsValid() — the very check healing trusts — is affected. That is not an input-validation problem but a data-integrity one: a legitimate heal reading poisoned metadata concludes the shard is fine and skips the repair.

The fix: two chokepoints, not twenty patches

The invariant to restore is one sentence: a path from an internode payload must resolve inside the volume it names, and a volume must resolve inside the drive root.

Only .. can break the first half (absolute and backslash-prefixed paths are folded under volumeDir by pathJoin’s Clean), and the second half has one independent break: paths that alias the volume root. Two rules, therefore — not a policy matrix.

Under 40 lines of core logic. No handler is modified, no call site in xl-storage.go is touched, and the local erasure path is left alone.

Two details worth recording:

• The check must run before the join, on the raw argument. pathJoin runs Clean against an absolute drivePath, which erases a leading .. entirely — /drive/../../etc becomes /etc, so a check placed after the join reads clean and passes everything. This is the most likely way a future refactor silently undoes the fix.
• NSScanner is the one method that reaches the filesystem without getVolDir. Its guard line is load-bearing, not decorative.

Among the rejected alternatives, the notable one is adding containment at all 33 pathJoin(volumeDir, …) sinks. That would be genuine defence in depth, but it means 33 edits in the most performance-sensitive file in the tree, each needing its own judgement about whether the volume root is a legitimate target. The guard rails buy most of the same resistance to drift for a fraction of the risk. This is an explicitly recorded IOU: if a code path is ever added that reaches the filesystem without getVolDir, the decision must be revisited.

Construction log: four regressions we caused ourselves

This section is unflattering and more informative than the fix.

First: whitespace. The initial rule treated whitespace as a separator, refusing " " and " " — legal S3 object keys. PutObject commits through RenameData, so such a key would fail on every remote drive simultaneously and break write quorum. The irony: the vulnerability needs root credentials; this bug needs a user to send a space.

Second: backslash-only keys. Same function, same root cause. path.Clean never treats \ as a separator, so on Unix "\\" is an ordinary filename. Refusing it made a distributed cluster reject a write a single-node server accepts — the same S3 API behaving differently by deployment topology.

Third: spaces and periods on Windows. The Win32 normalisation layer strips trailing spaces and periods from a path component, so a component made only of those vanishes and the path resolves to its parent. That makes both " " and "..." volume-root aliases on Windows — and "..." was sitting in our own list of legal object names at the time. We had not merely missed the vector; we had asserted it was safe.

Fourth: negative part sizes. The new guard rejected only “positive size with unusable parameters”, equating “non-zero” with “positive”. Negative values take a different route to the same zero.

Two false greens

Writing the AppendFile acceptance test produced two meaningless green runs in a row:

1. Driving it through the REST client — which special-cases *bytes.Reader and derives Content-Length from it, silently overriding the forged value.
2. Switching to an opaque reader — at which point Go’s own HTTP client refuses to send a request whose body is shorter than the declared length.

Only driving the handler directly through httptest reproduced it. The lesson: a client’s self-protection is not a server’s defence, and an attacker with a raw socket has no such scruples.

A third was a design failure. We built a per-field reflection poisoner, then discarded it: it cannot distinguish “should have rejected but delegated” from “correctly allowed a non-path field” (ETag, Algorithm, …), so it reports correct behaviour as failure.

The subtlest lived in the fuzzer. The first property test treated separator-only strings as an exception with an early return. That is not an exception, it is a blind spot — the fuzzer had been shut out of the entire category by hand and could never have found the backslash key in a million executions. A wrong exception is more dangerous than no fuzzer at all, because it creates the impression the space has been searched.

A very concentrated pattern

The reused semantics produced zero regressions; the invented rules produced all of them.

This is not coincidence. hasBadPathComponent is already the object layer’s own rule via IsValidObjectPrefix, validated by real S3 traffic for years, and structurally cannot reject anything creatable through the S3 API. An invented rule has nothing behind it but the author’s imagination.

The actionable form: reuse rather than invent; and when you must invent, write the property test by exclusion rather than enumeration, express exceptions with a predicate independent of the implementation, and keep them as few as possible.

The guard rails matter more than the patch

The final test suite pulls in two directions, and neither alone is enough:

• Falsifiability — remove each guard in turn and confirm the tests actually go red (191 failing subtests with the traversal guards removed; 64 GiB and 10.4 GB reappearing with the allocation guards removed). This is precisely what the rejected community PR lacked: its test asserted err != nil against a target that did not exist, so it passes with the vulnerability fully intact.
• Legal-traffic fuzzing — asserting that any key IsValidObjectName accepts, the guards accept (1.96M executions, no violations), and that any legal bucket name survives getVolDir (810K). This is what our own first two attempts lacked.

Plus a method-level reflection rail that fails by name when a path-taking method is added to StorageAPI unguarded.

History states the case for these rails bluntly: CVE-2026-39414 was also point-fixed on 2026-04-15 and only received a fix: complete ... two months later. Counting this one, “point fix → recorded as closure → completed months later” has now happened twice in this fork. The problem is not that someone was careless. It is that nothing in the tree could tell you a class was still open. Guard rails turn “someone must remember” into “CI fails”.

On adversarial review

This fix went through five rounds of independent adversarial review. Each round found one missed defect, and all five stood: whitespace keys → backslash keys → the AppendFile allocation → Windows spaces and periods → negative part sizes.

Our own review did find two in the same period (the WithDeadline log amplification and ReadParts using the wrong rule), but only after being pushed to that standard.

The hit rate says something plain: the last gate before merge should be independent acceptance, not the author’s own conclusion. During this work the author judged the change ready to ship four times and was overturned three.

Follow-up status

The first draft listed two implementation gaps. Both are now closed on the local branch, but none of these follow-up commits is in a published server release as of 2026-08-03:

• ReadFileHandler is bounded. Commit b6f70ab08 rejects a declared read length above 5 GiB, the maximum size of the S3 part represented by this legacy whole-file bitrot path. Legitimate GiB-scale reads can still allocate on that scale; the change removes caller-controlled allocation above the format’s real ceiling rather than pretending large reads are cheap.
• Negative part sizes cannot be persisted or trusted. Commit 80e8eaa42 rejects them at the AddVersion write funnel and again in CheckParts and VerifyFile, so both new poison and already-written metadata are covered. The internode boundary check uses the same predicate.
• Non-positive erasure block sizes are rejected at construction. Commit 80e8eaa42 validates blockSize in NewErasure, covering the other offset and decode divisions that a single downstream ShardFileSize guard could not. Rebalance’s separate division is guarded at its own boundary.

Two limitations remain and should not be folded into a stronger claim:

• No Windows CI. Windows builds are published; tests run on Ubuntu only. The Windows rule is reasoned from documented Win32 behaviour and has not been verified on the platform.
• Symlinks. The containment check is lexical, as upstream’s is.

Closing

The previous entry said that closing an endpoint and closing a defect class are two different conclusions. This time the known sinks are closed on the local branch, at the cost of four self-inflicted regressions and three overturned declarations that it was ready to ship. Publication remains a separate gate: the fixes above are not in a released server build yet.

If only one sentence survives: the vulnerability was upstream’s; our mistake was treating a point fix as a closure. And what prevents a third occurrence is not a more careful person — it is a test that fails.

3.10 - The Parser Knew, the Schema Didn't: Config Keys That Could Take Every Notification Down

Status: Fixed on the local pgsty/minio branch as 162ded343, unreleased Classification: Configuration-schema consistency and availability, not a vulnerability; includes one defensive hardening (credential values no longer echoed in validation errors) Affected scope: notify_nats JWT/NKey/TLS-handshake-first options, notify_amqp immediate, and any pre-2020 config migrated with an enabled NATS target — whose failure then silences every notification backend Tracking: pgsty/minio issue #39

This article names two unfixed availability defects in neighbouring code (the Postgres/MySQL migration writes, and kvFields typo folding). Neither is exploitable — both break the operator’s own configuration, loudly or not at all — and both are already named in the committed audit test’s allowlist. Publication needs no hold beyond the release itself.

Conclusions first

• Three notify_nats options — user_credentials, nkey_seed, tls_handshake_first — and one notify_amqp option — immediate — were read by the parser, written by the legacy migration, and registered nowhere. CheckValidKeys rejected exactly what GetNotifyNATS required.
• One constant meant two things. target.NATSUserCredentials held the string "MINIO_NOTIFY_NATS_USER_CREDENTIALS", sat in the environment-variable const block, and was used both as an env var name and as a config key. The snake_case config key for creds-file auth did not exist anywhere in the program.
• The reporter’s error did not come from their command. It came from the legacy migration: the pre-fix migration reproduces the issue’s error text byte for byte, including the notify_nats:ONE target name their command never mentioned. The migration wrote the store once; validation rejects it at every boot thereafter.
• The blast radius is the amplifier: FetchEnabledTargets fails fast on the first bad subsystem, its only caller just logs, and the global target list stays nil — so one broken NATS entry silently switches off Kafka, webhook, MQTT, and everything else.
• Inherited from upstream. Three feature PRs — #19139 (2024-02, user_credentials), #21008 (2025-04, tls_handshake_first), #21231 (2025-04, nkey_seed) — each added the parser and the env var, and each skipped the schema. Upstream is archived; the fork inherits both the defect and the duty.
• The fix registers the keys, splits the two-faced constant, corrects the migration — including a sibling bug that silently wrote immediate’s value under the internal key — tolerates the legacy on-disk spelling on the load path only, stops echoing values in invalid-key errors in both CheckValidKeys forms, and installs an AST audit that mechanically forbids this defect class across all ten notify subsystems.
• The audit found the next instance before the ink dried: the Postgres/MySQL legacy migrations write five unregistered keys, one of which is a plaintext database password. Recorded, allowlisted shrink-only, tracked for follow-up.

The error that named a target nobody asked about

The report (issue #39, by kuldeep-link11, against a NATS cluster using JWT operator/accounts auth) is a clean reproduction: configure notify_nats with a credentials file, watch it bounce.

$ mc admin config set us notify_nats:FITCHECK \
    address=nats-1:4222 subject=events.object.created \
    MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds \
    jetstream=off queue_dir=/data/queue-fitcheck queue_limit=100000

mc: <ERROR> ... found invalid keys
    (MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds
     nkey_seed= tls_handshake_first=off ) for 'notify_nats:ONE' sub-system,
    use 'mc admin config reset myminio notify_nats:ONE' to fix invalid keys

Two things in that error are wrong in ways the command cannot explain. The invalid-key list contains nkey_seed= and tls_handshake_first=off — keys the user never passed. And the rejected sub-system is notify_nats:ONE, while the command configured notify_nats:FITCHECK.

The second oddity is the whole case. Our reviewer established that the mc admin config set path cannot even carry an unregistered key: the server-side tokenizer, kvFields, splits the input line by searching for registered key names, so an unknown token never becomes a key at all — it is absorbed into the preceding key’s value. Probed directly:

input:  subject=s MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/x.creds
stored: subject="s MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/x.creds"

So the rejection could not have been about the command line. It was validateConfig sweeping the whole subsystem and tripping over a different, already-stored target named ONE that carried all three keys. Only one code path in the tree writes those key names into a store: the legacy config migration. Driving the pre-fix migration on an enabled NATS target named ONE reproduces the issue’s error text character for character — including the empty nkey_seed=, which is just what migration writes when the legacy config had no NKey.

That reframes the incident. This was not “the server rejected my command.” It was: an old config was migrated once, the migration wrote three keys the validator does not accept, and the store has been failing validation at every boot since — taking every other notification target down with it, silently, because the failure is logged and swallowed. The reporter’s command merely walked into the blast radius and got handed someone else’s error.

One constant, two meanings

The declaration, as inherited (internal/event/target/nats.go, pre-fix):

const (
    NATSAddress  = "address"
    NATSSubject  = "subject"
    NATSUsername = "username"
    NATSPassword = "password"
    NATSNKeySeed = "nkey_seed"            // config key — correct shape
    // ...
    EnvNATSUsername     = "MINIO_NOTIFY_NATS_USERNAME"
    NATSUserCredentials = "MINIO_NOTIFY_NATS_USER_CREDENTIALS"  // ← in the Env block
    EnvNATSPassword     = "MINIO_NOTIFY_NATS_PASSWORD"
)

NATSUserCredentials is named like a config key, valued like an env var, and shelved with the env vars. The parser used it as both: once as the env var to look up, once as the config key to read from the stored KVS. The migration used it as a key to write. There was no "user_credentials" string anywhere in the program — the config key for creds-file auth simply did not exist, which is why the reporter, finding no documented key, resorted to passing the env var name as one.

A name that means two things will eventually be wrong in one of them. Here it was wrong in both directions at once: as a key it was unregistered garbage; as the only spelling available it taught users and the migration to write garbage.

Four surfaces, no handshake

A notify option in this codebase lives on four surfaces that must agree: the defaults (DefaultNATSKVS — what validation accepts and mc admin config get displays), the help (HelpNATS — what mc admin config documents), the parser (GetNotifyNATS — what the server actually reads), and the migration (SetNotifyNATS — what upgrades write). Nothing ties them together. Three upstream feature PRs each updated the parser and the env plumbing, and each forgot the first two surfaces:

The AMQP row hides the quieter sibling. The AMQP migration did not skip immediate — it wrote immediate’s value under the internal key, and dropped cfg.Internal entirely:

config.KV{
    Key:   target.AmqpInternal,          // wrong key
    Value: config.FormatBool(cfg.Immediate),  // right value
},
// cfg.Internal: written nowhere

Because internal is registered, this one passes validation. The NATS gaps break a migrated config loudly enough to be found eventually; the AMQP gap corrupts it silently — a migrated broker config carries the wrong flag with a clean bill of health. One defect class, two presentations: the unregistered key fails closed, the misrouted value fails wrong.

The amplifier

None of this would deserve the word “outage” without the aggregation semantics. FetchEnabledTargets iterates the ten notify subsystems and returns (nil, err) on the first failure; its only caller logs the error and moves on, leaving the global notification target list nil; every later lookup nil-guards into an empty list. One rejected notify_nats target therefore turns off all bucket notifications — Kafka, webhook, AMQP, MQTT, the lot — with nothing but one line in the server log.

We considered changing this to per-subsystem isolation and decided not to, in this fix. Skip-the-broken-subsystem is a real behavioural change to how operators experience a bad config: today it fails loudly-in-aggregate (everything stops), and configurations that operators have already reasoned about depend on validation being all-or-nothing. Rewiring that is a compatibility decision that deserves its own change, not a rider on a registration fix — and once registration is correct, legal configs no longer trigger the cascade at all. The decision is recorded as a doc comment on FetchEnabledTargets and pinned by a characterization test, so the next person to touch it changes it on purpose or not at all.

The fix

About a hundred lines of production change, carried by nine hundred lines of tests (162ded343: 8 files, +1029/−7).

Registration. All four keys enter their default KVS and help schema, placed where an operator would look for them (user_credentials beside username, nkey_seed after token, tls_handshake_first after tls_skip_verify, immediate beside mandatory). Registration is also what makes a key visible: all four now appear in mc admin config get output where they previously did not.

The constant, split. NATSUserCredentials becomes a real config key, "user_credentials"; a new EnvNATSUserCredentials carries the env string. Every env var name involved — MINIO_NOTIFY_NATS_USER_CREDENTIALS, _NKEY_SEED, _TLS_HANDSHAKE_FIRST, MINIO_NOTIFY_AMQP_IMMEDIATE, and their _TARGET-suffixed forms — is frozen byte-for-byte: they are public interface, they worked throughout (the env route was always the workaround), and a test now pins them as raw string literals, so no rename of a Go constant can drift them silently.

Help flags, by precedent. Both new NATS values are file paths (a .creds file; an NKey seed file), so they are marked Sensitive but not Secret, mirroring cert_authority/client_cert/client_key rather than password/token. Secret would additionally redact them from mc admin config get — hiding an operator’s own configured path from them, which is why the private-key path client_key never had it either.

Migration, corrected. SetNotifyNATS now writes the real key; SetNotifyAMQP writes immediate = cfg.Immediate and internal = cfg.Internal.

If you are affected today, on a pre-fix build: the env var route works and always did, and mc admin config reset myminio notify_nats:<target> un-wedges a poisoned store at the cost of its settings. On the fixed build, poisoned stores simply load again — next section.

Living with what the old migration already wrote

Fixing the migration helps the next upgrade. It does nothing for stores the broken migration already wrote, which contain the literal key MINIO_NOTIFY_NATS_USER_CREDENTIALS — still unregistered, still fatal at every boot. Telling those operators to hand-reset their config would mean punishing them for our write.

So the load path tolerates it, narrowly. Validation accepts the legacy spelling for the NATS subsystem only — a test asserts AMQP still rejects it, so the tolerance cannot become a general escape hatch — and the parser falls back to it only when the real key is empty. Precedence is env > user_credentials > legacy key, and it holds by construction rather than by convention: the fallback result is passed as the default argument of the env lookup. All three orderings are tested. The legacy key stays out of the defaults and the help on purpose: it is tolerated, never advertised, never newly settable (kvFields sees to that).

The constant for it is a package-local literal, not an alias of EnvNATSUserCredentials — deliberately. It names bytes already on disk, so it must not follow any future rename of the env constant. The comment says so.

One trap discovered while wiring this, worth its own paragraph because it will bite someone eventually: the codebase has two CheckValidKeys — a free function and a method — and their deprecatedKeys parameters mean opposite things. The free function tolerates the listed keys (skips them); the method subtracts them from the valid set (rejects them). Refactoring this call from one form to the other would silently invert the tolerance into a ban. That asymmetry is now documented at the call site, which is the best one can do short of renaming an exported API.

The tolerance is written to be retired: the clean end state is to rewrite the legacy key into user_credentials once at load, then delete both the tolerance and the fallback. That is follow-up #2 below — and it also closes a small hole the tolerance leaves open: an unregistered key carries no Sensitive flag, so a tolerated legacy key ships its value (a path) unredacted in health-diagnostics bundles while user_credentials shows *redacted*.

Secrets in error messages

The invalid-keys error that started all this printed the rejected pairs with their values: found invalid keys (MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds ...). Those paths are mild. The mechanism is not: whatever value rides on a rejected key — a mistyped nkey_sed=<seed>, a bind password on a stale LDAP key — lands in the server log and in the mc client’s terminal.

Both CheckValidKeys forms now print key names only, keeping the shape and the mc admin config reset hint. The second site was one step beyond the written task scope — the method form serves LDAP, OpenID, and the policy plugin, where a rejected value can be an actual bind password — and the independent review, asked to judge that extension, said it would have demanded it: fixing one of two identical leaks is a half-fix. Repo-wide, nothing parsed values out of that string and no test asserted the old text; the change is global and intended.

There is a converse worth recording: this redaction is what stands between the next defect of this class and a credential in the logs. The follow-up below found the Postgres/MySQL migrations writing a plaintext database password under an unregistered key — on a pre-redaction build, the resulting rejection prints that password.

A guard that makes the class extinct

Registering four keys fixes four keys. The class — four surfaces, no handshake — stays open unless something ties the surfaces together mechanically. The fix therefore ships an AST-based audit test that parses parse.go and legacy.go, resolves the constants (from the target package sources, so there is no hand-maintained list to rot), and asserts, for all ten notify subsystems:

• every key the parser reads is registered in that subsystem’s defaults;
• every key the migration writes is registered (minus an explicit, shrink-only allowlist — next section);
• every help entry names a registered key.

Run against the pre-fix tree, it fails on exactly the four known gaps and nothing else — which is the red proof that it measures the right thing.

The adversarial review then attacked the audit itself with a mutation harness, on the theory that a guard you cannot watch fail is a guess — the discipline the previous article argued for. Seven of its nine mutations were caught. Two were not, and both blinded the audit silently: rename the parser’s loop variable (the read-collector pattern-matched the receiver name kv), or switch a migration entry to Go’s idiomatic elided composite-literal form (the write-collector demanded a typed config.KV{...}). In both cases the collector returns an empty map, the assertion loop iterates zero keys, and the test passes vacuously. Both are refactors a maintainer would make without a second thought; one of them is what gofumpt nudges you toward.

Two hardenings closed this, each verified in both directions — with the hardening the mutation is caught; with the hardening removed (the counterfactual) the vacuous pass returns:

• A floor assertion in the reverse direction: every registered key must be seen being read. This holds for all ten subsystems today — measured, not assumed, including the deprecated streaming_* keys read inside a nested conditional — so it costs nothing, and a blinded collector now produces one loud error per registered key (22 of them for NATS) instead of a green run.
• A widened literal guard: typed literals that are neither config.KV nor config.KVS are skipped; untyped (elided) literals have no type to check and are now inspected rather than ignored.

Final score: ten mutations, ten caught — the harness gained one variant along the way, and the closure round swept the full suite. The audit also enforces its own allowlist in both directions — removing an entry that is still needed fails, and an entry that goes stale (the migration no longer writes that key) fails too, so the allowlist can neither grow silently nor lie about the present.

What the audit found next

The write-side check refused to go green on two subsystems that had nothing to do with issue #39. SetNotifyPostgres and SetNotifyMySQL write five keys — host, port, username, password, database — that no default KVS registers and no parser reads. These are relics of the pre-DSN configuration shape, and the migration still emits them. Driving the real helpers confirms it: a migrated Postgres or MySQL notify target is rejected on the next load with found invalid keys (host, port, username, password, database) — the same failure mode as NATS, the same every-boot persistence, the same all-notifications blast radius through the fail-fast. And password there is a plaintext database password, which is exactly the value the redaction above now keeps out of the logs.

It is deliberately not fixed in this change. The scope was locked to the NATS and AMQP gaps, and the right treatment (register the five as deprecated, or stop writing them, or both) is a judgment call that deserves its own red/green cycle. It is pinned in the audit’s knownUnregisteredWrites allowlist with a shrink-only comment, so it cannot be quietly forgotten: the day someone fixes it, the stale allowlist entry fails the test and demands its own deletion.

Open items, none in a released build as of 2026-08-04:

1. Postgres/MySQL migration unregistered writes — major, live at every boot for anyone migrating a pre-KV config with those targets enabled.
2. Rewrite-on-load for the legacy NATS key, then retire the F5 tolerance and fallback; also closes the health-bundle redaction gap for tolerated keys.
3. kvFields typo folding — an unknown key name in mc admin config set is silently absorbed into the preceding key’s value instead of erroring. Pre-existing upstream wart; it protected nobody here and will corrupt someone’s subject eventually.

Review record

The change went through three gates before commit:

The honest accounting, in the house tradition: neither demand was a defect in the production fix. One was a lint gate (two British spellings that would have failed make test — and while fixing them, the implementer’s rewritten comment introduced a third, dialled, which the same gate caught; the demand vindicated itself in real time). The other was the audit-blindness pair above — durability of the guard, not correctness of the change. What the review did overturn was the incident’s origin story: the migration-path reproduction, the ONE target, and the kvFields absorption proof all came from the reviewer, and they change what operators should conclude — this was a boot-time outage lying in wait in stored configs, not a CLI validation quirk.

The implementer’s red phase also surfaced three bugs in its own new tests before the fix landed, recorded rather than smoothed over: a fixture that assumed stored targets are layered over defaults when config.Merge actually passes them through verbatim; a characterization test that segfaulted on a nil HTTP transport (FetchEnabledTargets dereferences it unconditionally — hostile to testing, noted, unfixed); and an early draft keyed to the very constant the fix renames, which made it pass green pre-fix — rewritten against the literal string so it pins the on-disk schema rather than the Go symbol.

Declined, and left open

Declined, deliberately:

• Per-subsystem error isolation in FetchEnabledTargets — a compatibility decision, not a rider (above).
• Registering or advertising the legacy key — tolerated on load, absent from defaults and help, impossible to set anew.
• Renaming EnvNatsTLSHandshakeFirst’s odd casing — an aesthetic rename in a fork is diff noise that buys nothing.
• Fixing the Postgres/MySQL migration here — scope-locked, pinned in the allowlist instead (above).

Left open: the three follow-ups above, and one cosmetic consequence — a store that still carries the tolerated legacy key will show it verbatim in mc admin config get until rewrite-on-load lands.

Closing

Every one of these keys worked perfectly through the environment variable, which is why three feature PRs could ship, get reviewed, get used, and never notice that the config-file half of the interface was stillborn. The parser and the schema are two descriptions of the same contract, maintained by hand, four surfaces wide — and for two and a half years nothing in the build checked that they agree.

If only one sentence survives: when two artifacts must stay identical and only convention binds them, the divergence is not a risk but a schedule — put a machine between them, then mutate the machine until you have watched it catch the drift you fear.

3.11 - Object Grant, Bucket Reach: When 'bucket/*' Could Rewrite the Bucket Itself

Status: Fixed on pgsty/silo-pkg main (3c24ad1, extended by 1f97549, scoped to its final twelve actions in d8b1fa7), released as silo-pkg v3.11.0; consumed by pgsty/minio Classification: Access-control hardening — a privilege boundary, narrowly restored Affected scope: IAM users/roles/service accounts granted only object-scoped (arn:aws:s3:::bucket/*) access, in deployments that share a cluster across tenants Tracking: upstream minio/minio issue #20449 (public since 2024, still open)

Conclusions first

• In IAM policy matching, a bucket-level request carries an empty object name, and the matcher built its resource string as "bucket/". An object-only policy pattern — "arn:aws:s3:::bucket/*" — then matched that string, so a grant that should cover only objects also authorized bucket-level actions.
• The dangerous one is PutBucketPolicy. A tenant holding only s3:* on bucket/* could install a bucket policy with Principal:"*" — making the bucket publicly readable or writable — or grant itself bucket-level control. Same mechanism, same class: DeleteBucket/ForceDeleteBucket (the issue’s own reproduction), PutReplicationConfiguration (exfiltration), PutBucketLifecycle (mass deletion), PutBucketVersioning, PutBucketObjectLockConfiguration, and the rest of the bucket-configuration writes.
• The full correction is a two-directional behavior change: it tightens over-granting Allow statements and loosens over-blocking Deny statements, and it would revoke ListBucket/GetBucketLocation grants that many real deployments write as bucket/* today. That is a compatibility break, not a clean patch.
• So we shipped a narrow fix — first six sensitive bucket-configuration writes, then, in a second pass, twelve: the bucket-level writes that hand the caller something its object access does not already give it, plus four that no handler implements. Only on Allow statements, so no Deny and no NotResource exclusion is ever weakened, with an environment-variable escape hatch. The compatibility-sensitive read/list family, CreateBucket, and three bucket writes with plausible tenant use are left unchanged, by decision.
• Twice we claimed the change could only remove permissions, and twice an untested case said otherwise — the second time found by an independent review of a shipped release. The protected path now requires the resource to match both the bare and the historical form, which makes the property hold by construction rather than by argument.
• The fix is red/green proven at the matcher layer and end to end through the real handlers; the object-scoped hot path is untouched.

The slash, and the empty object name

Every bucket-level S3 operation authorizes with an empty object name — checkRequestAuthType(ctx, r, policy.PutBucketPolicyAction, bucket, ""). The IAM matcher turned that into a resource string, and for the empty-object case it appended a trailing slash:

resource.WriteString(args.BucketName)
if args.ObjectName != "" {
    // "bucket/object"
} else {
    resource.WriteByte('/') // "bucket/"  <-- the defect
}

"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. So a policy that grants s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in bucket” — was evaluated as granting bucket-level actions too. The bucket-policy evaluation path (for anonymous/public access) never had this slash and is the reference-correct behavior; only the IAM path was wrong, and there was exactly one place it went wrong.

This is upstream minio/minio #20449, filed in 2024. An early upstream attempt deleted the slash outright and was reverted the same day for breaking policies that relied on the old behavior. The lesson we took from that revert shaped the fix below.

What it actually enables

PutBucketPolicyHandler has a single authorization gate and nothing behind it. Once the IAM check passes, the caller may store any well-formed bucket policy for that bucket.

The concrete chain, in a multi-tenant cluster:

1. An administrator grants tenant A the policy Allow s3:* on arn:aws:s3:::bucket-a/*, intending “A may work with the objects in bucket-a, nothing more.”
2. Because of the slash, A may call PutBucketPolicy on bucket-a.
3. A installs { "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::bucket-a/*" }. Every object in bucket-a is now readable by the anonymous internet. s3:* makes it world-writable. Pointing Principal at an account A controls exfiltrates the data; granting itself bucket-level actions in that policy is self-escalation.

The same object-only grant reaches other bucket-configuration writes with comparable consequences: replication to an attacker’s target, a one-day lifecycle expiry that deletes the bucket’s contents, disabling versioning, tampering with object-lock retention. None of these should be reachable from a grant scoped to objects.

This is not remotely exploitable and requires no missing credential — the caller is an authenticated principal you deliberately gave a scoped policy to. In a single-tenant deployment, that principal is your own trusted user and the practical risk is low. In a shared, multi-tenant cluster it is a real cross-tenant boundary failure.

Why a narrow fix, not the whole boundary

The obvious fix is to stop appending the slash for every bucket-level request. We did not do that, for two reasons that matter more than the one-line diff suggests.

It breaks common, benign usage. The correction does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation, and ListBucketMultipartUploads when they were granted through bucket/*. Many deployments write exactly that and rely on it. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the projects that wrote the server write it this way, production policies do too. A maintenance upgrade that turns those into AccessDenied is precisely the kind of surprise we refuse to ship.

It cuts both directions. The matcher builds the same resource string for Allow and Deny. So the full correction tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements: an administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection for bucket-level actions. A clean-looking fix that moves security in two directions at once is not a maintenance patch — it is a migration.

So we narrowed the change to where it is unambiguously right and effectively free of compatibility cost:

• Only bucket-level writes are protected. The first pass covered six sensitive configuration writes: PutBucketPolicy, DeleteBucketPolicy, PutReplicationConfiguration, PutBucketLifecycle, PutBucketVersioning, PutBucketObjectLockConfiguration. The second pass (below) extended that to twelve. Almost nobody grants these through an object-only pattern on purpose — you do not accidentally rely on an object grant being able to rewrite a bucket’s policy or delete the bucket — so revoking that path breaks essentially no one.
• Only on Allow statements. Deny statements keep the historical resource string, so no existing Deny is ever weakened. The narrow fix only ever adds a denial.
• The read/list family is left exactly as it was. ListBucket on bucket/* still works. That is the compatibility-sensitive part, and it waits.

The fix

The matcher keeps the trailing slash in every case except one: a bucket-level Allow statement being evaluated for a protected action, with the compatibility shim off.

resource.WriteString(args.BucketName)
if args.ObjectName != "" {
    // "bucket/object" — unchanged
} else if args.BucketName == "" {
    resource.WriteByte('/') // KMS two-phase sentinel — unchanged
} else if legacyBucketResourceMatch.Load() ||
    statement.Effect != Allow ||
    !isSensitiveBucketMutation(args.Action) {
    resource.WriteByte('/') // historical behavior for Deny / non-sensitive / shim-on
}
// else: bare "bucket" — an object-only "bucket/*" no longer authorizes it

Because args.Action is the concrete request action, a wildcard grant (s3:*) is covered too: the wildcard matches at the action step, and by the time the resource string is built the action is the specific PutBucketPolicy. A bare-bucket resource (arn:aws:s3:::bucket) and the * resource still match, so correctly scoped grants — including the built-in readwrite policy — are untouched.

The escape hatch is MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup. It restores the full historical behavior — both the over-grant and the over-block — for any operator who needs the old semantics while they adjust their policies.

The second pass, and the question that decided its size

The first round protected six configuration writes and stopped. Reviewing it against the original issue showed that was not enough: the action reproduced in #20449 itself — DeleteBucket — was still reachable through an object-only grant. An end-to-end test against the first-pass build confirmed it: a user holding nothing but s3:* on arn:aws:s3:::bucket/* called RemoveBucket and the bucket was gone.

Extending the set raised the real question — how far? The first instinct was “every bucket-only write except CreateBucket,” fifteen actions. That was the wrong instinct, and the reason is a detail of how the bug fires.

The bug only triggers when the statement already grants the bucket-level action. Resource matching runs after action matching, so a read-only tenant holding s3:GetObject on bucket/* never reaches DeleteBucket — the action never matched. In practice the affected principal holds s3:*, which means they already have full read, write, and delete over every object in the bucket. That reframes the severity of each candidate action, because the question is not “how dangerous is this action in the abstract” but “what does reaching it add to a position that already includes all the data?”

By that test, three groups fall out:

Protected — reaching these grants something the object access does not. PutBucketPolicy and DeleteBucketPolicy hand access to other principals, anonymous included, and can grant the caller bucket-level actions it was never given: self-escalation and public exposure. PutBucketObjectLockConfiguration and PutBucketVersioning defeat protections that exist precisely to stop a holder of write access from destroying data. PutReplicationConfiguration and PutBucketLifecycle act under server credentials and keep acting after the caller’s access is revoked. DeleteBucket and ForceDeleteBucket destroy the bucket entity and its configuration irreversibly.

Protected at zero cost. PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration have no MinIO server behavior attached today — no handler at all, or a handler that returns NotImplemented after the authorization check. Withholding them changes nothing that works, and covers them in advance if a handler is ever wired.

Deliberately not protected. PutBucketTagging, PutBucketEncryption, and PutBucketNotification are bucket-level writes, and the first draft of this pass did protect them. They came back out. None of the three gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handed s3:* on bucket/* and told “this bucket is yours” may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release. They keep the historical matching, and a test now asserts that they are unprotected, so putting any of them back is a deliberate act with a visible cost rather than an edit to a list.

That leaves twelve actions, shipped as silo-pkg v3.11.0. Two older boundaries stand unchanged: CreateBucket keeps the historical matching (it targets a bucket that does not exist yet, and provisioning flows commonly create a tenant’s bucket with that tenant’s own credentials), and the read/list family still waits for the migration-gated change.

The choice of what to break, in other words, was made by asking would an administrator ever write this on purpose — not by ranking the actions by how dangerous they sound. The first question predicts which upgrades break; the second only sets urgency.

The claim that was wrong twice

Everything above rests on one property: this change may remove permissions and must never add one. Both times we asserted it, we were asserting it about a mechanism we had reasoned through rather than tested through. Both times it was false.

The first pass withheld the slash from the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected. Restoring the historical form for NotResource fixed that, and the second pass shipped saying the result was “provably monotone.”

An independent adversarial review of that release produced a counterexample within the hour. Withholding the slash does not merely remove a match — it changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". The clean case is a fixed-width wildcard:

Allow s3:PutBucketPolicy on arn:aws:s3:::mybucke?

? matches exactly one character. Against the historical nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it matches, so the hardening granted something the buggy matcher refused. Small in reach — you have to write a length-sensitive pattern — but it is precisely the class of defect the property was supposed to exclude, shipped in a release whose notes claimed the property held.

The fix is not another special case. On the protected path the matcher now requires both forms to match: the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction — there is no pattern it can newly satisfy, and no argument to get wrong next time. mybucket* still grants (it matched both all along); mybucket/* is still withheld; mybucke? is refused exactly as it always was. That shipped as silo-pkg v3.11.0.

Two things are worth taking from this beyond the patch itself. A correctness fix in an authorization path must never make anything newly allowed — and the only way to know is to test both directions, because the reasoning feels airtight in both cases where it wasn’t. And when a security property is load-bearing, build it out of an operation that cannot violate it rather than out of a case analysis you believe is complete.

Regression tests now pin each direction: grants narrowed, Deny untouched, NotResource exclusions untouched, fixed-width wildcards not broadened, the three unprotected writes still reachable, plus an invariant test that every protected action really is bucket-only (ResetBucketReplicationState, despite its name, is an object action and stays out). In the server they run end to end through the real handlers — client, inline session policy, and the S3 router — and every one of them fails against the release that had the bug.

What you will notice

For nearly everyone: nothing. Object access is unchanged, ListBucket via bucket/* is unchanged, and correctly written bucket policies are unchanged.

The one visible change: a request that tries to delete the bucket, or change its policy, replication, lifecycle, versioning, or object-lock configuration using credentials whose only matching grant is an object-only bucket/* pattern now returns AccessDenied. Bucket tagging, default encryption, and event notification are not affected. That is the boundary being enforced. If a deployment genuinely depends on the old behavior, set MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on and grant those actions on the bare bucket ARN (arn:aws:s3:::bucket) at your own pace.

What we deliberately left open

The general problem in #20449 — that bucket/* reaches the remaining bucket-level actions: ListBucket, GetBucketLocation, the configuration reads, CreateBucket, and the three tenant-plausible writes above — is not fixed here. Closing it fully means revoking grants that real deployments depend on, so it belongs to a future release that carries a migration path.

What that release owes operators is more than a wider action list, because no one can enumerate every deployment’s policies — which means shrinking or growing the protected set by guessing is an exercise with a hard ceiling. Three things raise it:

• A startup policy audit. Walk the stored policies and name each one whose meaning changes, in both the grant and the deny direction. That turns an upgrade surprise into a pre-upgrade checklist, it is read-only, and it can ship before the enforcement change rather than with it.
• A denial that explains itself. When a request is refused because only an object-scoped grant matched, say exactly that, and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
• A switch with a scope. MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH is all-or-nothing today: an operator who needs one action back has to reopen the self-escalation path along with it. Per-action scoping is what makes the change safe to adopt.

Recording the boundary rather than implying it: today twelve bucket-level writes are corrected. Everything else — the read/list family, CreateBucket, and bucket tagging, encryption, and notification — still honors bucket/* as a bucket-level grant, by decision, until that migration-gated change lands.

Closing

A single appended slash turned “only the objects” into “and the bucket too.” The tempting fix removes the slash everywhere and, in doing so, breaks a listing pattern half the world relies on and quietly weakens every Deny written against bucket/*. The fix we shipped removes it in exactly the place where an object-scoped grant should never have reached — the writes that can make a bucket public, and the ones that can delete it — and nowhere else. The rest is written down, waiting for a release where breaking it is something users are told to expect rather than something that happens to them.

3.12 - Absent Is Not Empty: A Blank versionid and the Fail-Open It Invites

Status: Fixed on the local pgsty/minio branch as 744a9dcd7, unreleased Classification: Policy-enforcement correctness — a fail-closed report, a fail-open trap avoided, and one narrow trim bypass closed. Not a headline CVE — see How we classify this Affected scope: Any deployment with a bucket/IAM policy using Null or StringEquals on s3:versionid; the reported break is on DeleteObject/DeleteObjects Tracking: upstream minio/minio issue #21735 (reporter iTrooz, 2026-01-10); upstream repository archived read-only since 2026-04-25

This article documents an unreleased fix and two unfixed same-class residuals in neighbouring paths (governance-bypass and Snowball). Hold publication until the fix ships and the residuals are triaged.

Conclusions first

• The policy engine decides Null by slice length, not by content. MinIO wrote "versionid": {""} into the condition map unconditionally, so a request that named no version still presented a length-1 slice. Null:{s3:versionid:true} — “match only when the key is absent” — could therefore never match, and Null:false always matched. The reporter’s “allow deletes only of the current object” policy denied every current-object delete (HTTP 200 envelope, per-object AccessDenied).
• The one-line fix is a trap. “Write the key only when it is non-empty” fixes the report and simultaneously opens something worse. DeleteObjects carries each object’s version in the XML body; the condition builder reads only the query string. Drop the empty key and a body version simply vanishes from the map — read as absent, i.e. as null — so a policy meant to protect old versions would authorize deleting a specific one. Fail-closed defect, meet fail-open bypass.
• The real fix has two parts: write the key only when a version is named, and bind it, for DeleteObject, to the effective server-resolved version (ReqInfo.VersionID) — the per-entry body value that the DeleteObjects loop already resolves — rather than to whatever the query string happened to carry.
• A third, adjacent hole closed on the way: the builder read the version untrimmed while the object layer trims it, so a padded ?versionId=V%20 let a Deny StringEquals s3:versionid "V" be sidestepped on the read/tag/copy paths.
• Inherited from upstream, and unfixable there. minio/minio is archived read-only, so the fix lives in the fork; this is the same getConditionValues we hardened in the condition-source work.

Absent is not empty

A condition key in a MinIO policy resolves to a lowercase name in a map[string][]string, and the engine answers Null by asking how long that slice is (silo-pkg .../policy/condition/nullfunc.go):

func (f nullFunc) evaluate(values map[string][]string) bool {
	rvalues := getValuesByKey(values, f.k)
	if f.value { // Null:true — "the key must be absent"
		return len(rvalues) == 0
	}
	return len(rvalues) != 0 // Null:false — "the key must be present"
}

The content of the strings is never read. A slice {""} has length 1. To this function, a present-but-empty value is indistinguishable from a real version ID, and both are the opposite of absent.

Now the value that fed it, as inherited (cmd/bucket-policy.go, getConditionValues):

args := map[string][]string{
	// ...
	"versionid": {vid}, // vid == "" for any request that names no version
	// ...
}

vid is the request’s ?versionId, empty on the overwhelming majority of calls. So every request, versioned or not, arrived at the engine carrying versionid: [""] — permanently length-1, permanently “present.”

The two Null directions then invert:

The reporter wrote the canonical “let clients delete current objects but not roll back versions” policy — Allow s3:DeleteObject with Condition {"Null": {"s3:versionid": "true"}} — and watched every version-less delete return AccessDenied. The Allow never fired because its condition tested “no version named” and the map insisted a version was always named. StringEquals cannot see the difference either ({""} and absent both fail to intersect a non-empty policy value); only Null and ForAllValues:* are sensitive to it, which is why Null is where it surfaced.

The fail-open next door

The obvious fix writes the key only when it is non-empty, and for a single DeleteObject that is completely correct: no version → absent → Null:true matches. Ship that alone, though, and Multi-Delete turns it into an authorization bypass.

DeleteObjects (POST /{bucket}?delete) does not put versions in the query. Each object carries its own optional version in the request body:

<Delete>
  <Object><Key>photo.jpg</Key><VersionId>a1b2…</VersionId></Object>
  <Object><Key>notes.txt</Key></Object>
</Delete>

The condition builder reads r.Form — the query string — and nothing merges an XML body into it. So under the naive fix, an entry that names version a1b2… in the body produces an empty query version, the key is omitted, and the engine sees absent — null. A policy written to allow only null-version deletes now matches, and the specific old version the operator meant to protect is deleted. The fail-closed nuisance from the report has become a fail-open on exactly the operation that most needs to be scoped per object.

This is the crux the reporter’s simple case hides: the condition value must be the version the server will actually act on for this object, and for Multi-Delete that value lives on a channel the condition builder never looked at.

The fix: the effective version, not a convenient one

Two mechanisms, because either alone is wrong.

1 — Represent absence honestly (cmd/bucket-policy.go). Write the key only when the request names a version, so “no version” becomes a length-0 read:

if vid != "" {
	args["versionid"] = []string{vid}
}

2 — Bind DeleteObject to the effective version (cmd/auth-handler.go, authorizeRequestWithTags). The DeleteObjects loop already resolves each entry’s body version into ReqInfo.VersionID (via checkRequestAuthTypeWithVID, cmd/bucket-handlers.go:502, a sequential loop — no shared-state race). Authorization rebinds the condition value to that server-resolved string, and deletes the key when it is empty:

conditionValuesForAuth := func(lc string, cred auth.Credentials) map[string][]string {
	values := getConditionValuesWithTags(r, lc, cred, existingTags, requestTags)
	if action == policy.DeleteObjectAction {
		// DeleteObjects carries the effective version in each XML object,
		// not in the request query. Keep authorization scoped to that entry.
		if versionID == "" {
			delete(values, "versionid")
		} else {
			values["versionid"] = []string{versionID}
		}
	}
	return values
}

An end-to-end test drives a &versionId=query-level-decoy on the DeleteObjects URL and asserts it never reaches any entry’s decision — the per-entry body value wins, the decoy is stripped.

Why DeleteObjectAction only, and not a blanket ReqInfo rebind. The tempting simplification — “always use ReqInfo.VersionID” — breaks copy. For a CopyObject, the source read is authorized as GetObject against the source’s version, which travels in the x-amz-copy-source header, and getConditionValues already extracts it there; ReqInfo.VersionID for a copy holds the destination query (usually empty). A blanket rebind would overwrite the correct copy-source version with the wrong one. Every non-delete version-aware operation (Get, Head, tagging, retention, copy-source read) carries its version in the query or the copy-source header, both of which the builder reads, and both of which are the effective version for a single object. Only Multi-Delete diverges. So the override is precisely as wide as the divergence, and no wider.

The version the server acts on is the trimmed one

One gap remained once the delete paths were correct. The builder read the version raw:

vid := r.Form.Get(xhttp.VersionID) // untrimmed

while every path that actually uses the version trims it first — newContext (cmd/utils.go:806) and getOpts (cmd/object-api-options.go:101) both strings.TrimSpace. So on non-delete version-aware actions, a padded ?versionId=V%20 presented "V " to the policy engine while the object layer read, tagged, or retained version "V". A Deny keyed on StringEquals s3:versionid "V" — “protect this exact version” — saw "V ", failed to match, and did not fire; the operation on "V" proceeded. A narrow bypass (the attacker must know the version and that a space changes nothing downstream), but a real one.

The fix trims both reads, aligning the condition value with the effective version:

vid := strings.TrimSpace(r.Form.Get(xhttp.VersionID))
// ... and the copy-source fallback likewise

DeleteObjectAction was already immune, because it uses the already-trimmed ReqInfo.VersionID. Trimming introduces no new allow: it can only make the condition value equal the version actually operated on, which tightens Deny and corrects Allow in the same direction. We proved it is load-bearing by removing only the trim and watching the padded test case go red.

What it affected

The reported break is on delete, but the underlying key is read by many actions. After the fix, every version-aware chain evaluates s3:versionid against the version the server resolves for that operation:

A forgery route was already closed by the earlier condition-source work and is worth restating: versionid is a reserved internal key (both the versionid and canonical Versionid spellings), so a client cannot inject a second copy through the header/query merge loops. The wire parameter is spelled versionId (capital I) and lands in an inert args["versionId"] the engine never reads.

Two directions of impact, kept distinct because they have different severities:

• Functional (the report): version-less deletes were wrongly denied. Fail-closed — an availability and usability defect, not a grant.
• Security (the trap and the trim): the naive fix would have granted deletes of protected versions on Multi-Delete (fail-open); and the untrimmed value permitted a narrow Deny bypass on read/tag/copy. The fix closes the first before it can exist and the second where it already did.

How we classify this

We are not minting a CVE for this, and the honest reasons are worth stating.

The behaviour the reporter filed is fail-closed: MinIO denied operations the policy meant to allow. A system that is too strict leaks nothing and grants nothing; it is a correctness and usability defect, and inflating a false-deny into a vulnerability would cheapen every real entry in this chronicle, whose neighbours are authentication bypasses and path traversals.

What carries genuine security weight is not the report but its vicinity. The fail-open on Multi-Delete is real, but it is a hazard we would have introduced, not one that shipped — the value of the two-part design is that the dangerous version never existed in a build. The trim bypass did exist, but it is narrow: it requires a Deny keyed on an exact s3:versionid, an attacker who knows the version, and it only ever affected non-delete paths. We closed it because it was in reach, not because it was a headline.

So: policy-enforcement correctness, filed here because that is where we keep silent enforcement failures, with the security interest recorded plainly rather than dressed up.

The boundaries we did not cross

Two same-class residuals remain, recorded rather than silently left:

• Governance-bypass in Multi-Delete. When an entry carries object-lock, enforceRetentionBypassForDelete re-authorizes under BypassGovernanceRetentionAction (cmd/bucket-object-lock.go:153). That action is not DeleteObjectAction, so the effective-version override does not apply, and its s3:versionid is still the query value — absent in a normal Multi-Delete — rather than the per-entry version whose lock is being bypassed.
• Snowball tar extraction. PutObjectExtract takes each member’s version from the tar PAX record minio.versionId after the per-file authorization, so a named version can be written that never appeared in any condition value.

Both are narrow, both are pre-existing, and both would widen the change from “fix the reported key” into “re-plumb every action’s version into ReqInfo.” We scoped to the reported surface and wrote the IOUs down here, for the same reason the previous article recorded its object-layer omission: a deliberate omission that is not written down is indistinguishable from an oversight six months later.

A related decision, declined: the sibling keys username, userid, signatureversion, and authType are still written unconditionally empty, carrying exactly the present-but-empty defect versionid just shed — Null:{aws:username:true} is always false, including for the anonymous caller it should match. Fixing them is a one-liner each and a forty-caller blast radius, and some (principaltype is never empty) do not share the bug at all. We did not bundle a broad presence sweep into a versionid fix; it is named here as the next thread to pull.

Falsification

Three experiments, in the discipline that a test you have not watched fail is not yet a test.

• Revert both source files to HEAD. The end-to-end DeleteObjects test turned red with every version-less entry returning AccessDenied — a faithful reproduction of issue #21735 — and the unit test caught the {""} key directly (“an absent versionId was exposed to policy evaluation”). Reapply, green.
• Remove only the TrimSpace. The padded case went red on the exact assertion — got [7f4b6b5f-…dd8 ] — proving the trim is not decoration. Restore, green.
• The decoy. The Multi-Delete test appends &versionId=query-level-decoy to the URL and asserts it reaches no entry’s decision, which is what distinguishes “reads the query” from “reads the effective per-entry version.”

The change touched only five files (cmd/bucket-policy.go, cmd/auth-handler.go, two tests, one doc example), committed with explicit paths in a working tree that had concurrent unrelated work in it, so nothing from the neighbouring efforts was swept in.

Source and lineage

The report is upstream minio/minio#21735, opened 2026-01-10 against RELEASE.2025-09-07T16-13-09Z: a Null:{s3:versionid:true} policy denying version-less DeleteObjects. The upstream repository went archived and read-only on 2026-04-25, so there is no upstream fix to wait for and no maintainer to coordinate with — the fork is the only venue, and the record here is the resolution.

The defect is old and inherited. getConditionValues has written versionid unconditionally for as long as the key has existed; the length-based Null semantics are upstream’s, in the policy package the fork consumes via silo-pkg. This is the same function and the same lineage as the earlier condition-source hardening that stopped client input from shadowing server-derived condition values — a related read of “what a policy condition is allowed to believe about a request,” continued here into “and it must believe the version the server will actually act on.”

Closing

Absent is not empty. A map that cannot say “no version” by leaving the key out will say it by leaving the value blank, and a Null that counts length will believe a version was named on every request that named none.

If one sentence survives: a fail-closed bug is the dangerous kind to fix, because the obvious repair flips it to fail-open — so bind the condition to the value the server actually acts on, from the same channel the operation reads, not the channel that was convenient; and when you stop at the reported surface, write down the versions you left on the wrong channel, rather than trusting the next person to find them.

3.13 - Three Headers, One Lie: Making the Client Source Address Mean Something

Status: Landed on pgsty/minio master as fe6dc4780, unreleased Classification: Opt-in hardening plus a documentation defect, not a vulnerability and not a regression; no CVE assigned. The underlying weakness is inherited from upstream and its default behaviour is unchanged here Affected scope: aws:SourceIp policy conditions, the audit log remotehost field, S3 event notification Host, and the client shown by mc admin trace — on any deployment whose S3 API port is reachable without passing through a header-sanitising proxy Upstream: nothing to file — minio/minio is archived. Prior art there: PR #4736 (2017, the concern raised and half-addressed), discussion #17878 (2023, maintainer marks it working as intended), PR #20977 (2025, the partial switch)

This article states plainly that an IpAddress policy condition is not enforceable on a directly-reachable MinIO deployment, and that this remains true by default after the change. That is a property of upstream MinIO as shipped, not a defect introduced by the fork, and it has never been documented anywhere. Publishing it is the point.

Conclusions first

• MinIO reads the client’s address out of three interchangeable headers — X-Forwarded-For, X-Real-IP, RFC 7239 Forwarded — and never from the TCP connection unless all three are absent. That address becomes aws:SourceIp and the audit log’s client field, so whoever controls it controls both IP-based access control and the attribution of every logged action.
• The one switch that existed, _MINIO_API_XFF_HEADER=off, suppresses one of those three. An attacker’s response is to send X-Real-IP instead. Our own code comment was recommending it as the mitigation.
• “Put MinIO behind a reverse proxy” is not sufficient, for two independent reasons: on Kubernetes an Ingress and a ClusterIP Service routinely coexist so the proxy is not the only way in; and the stock nginx recipe appends to X-Forwarded-For, leaving a client-supplied entry in the left-most position — which is exactly where MinIO reads.
• The fix is a new opt-in setting, MINIO_API_TRUSTED_PROXIES, generalising a trusted-proxy mechanism this fork already built for LDAP STS rate limiting. Set to a list, forwarded headers are believed only from those peers and chains are walked right-to-left. Set to none, nothing is believed.
• The most consequential decision was one we reversed. The first implementation widened _MINIO_API_XFF_HEADER=off to suppress all three headers. That was the only part of the change that could alter an existing deployment’s behaviour, and it was backed out. The switch keeps its exact upstream semantics, and upstream’s TestXFFDisabled is retained unmodified as the proof.
• Net compatibility impact: none for any deployment that does not opt in.
• Adversarial review found four defects in the first implementation, including one that made the new setting silently ineffective for every deployment configured through an environment file.

What the address is actually used for

The value comes from a single function, handlers.GetSourceIPFromHeaders. Tracing its consumers is what turns this from a logging curiosity into a security question:

Two of these are security-relevant in different ways. A forged aws:SourceIp is a live access-control bypass: an IpAddress condition meant to confine a principal to an office CIDR is satisfied by asserting an address in that CIDR, and a NotIpAddress deny is evaded by asserting one outside it. A forged audit address is quieter and arguably worse — it corrupts the record retroactively, it applies even where no IP-based policy exists, and nobody notices until they need the logs.

Note also that the value is never validated as an IP address in the default path. Forwarded: for="_gazonk" is accepted and returned verbatim; upstream’s own test asserts it.

The setting that looked like a mitigation

internal/handlers/proxy.go gates exactly one header:

if enableXFFHeader {
    if fwd := r.Header.Get(xForwardedFor); fwd != "" {
        // ... left-most entry
    }
}
if addr == "" {
    if fwd := r.Header.Get(xRealIP); fwd != "" {
        addr = fwd                       // not gated
    } else if fwd := r.Header.Get(forwarded); fwd != "" {
        // ... first for= element        // not gated
    }
}

Setting _MINIO_API_XFF_HEADER=off costs an attacker one line: send X-Real-IP instead of X-Forwarded-For. Worse, disabling X-Forwarded-For moves the trust to a header the operator has not thought about, so the switch can leave a deployment in a state its owner has not modelled.

Where did it come from? Upstream PR minio/minio#20977, whose entire stated motivation is:

Customer request to disable all XFF header handling, ping me in Slack for more details.

No security rationale, no mention of X-Real-IP or Forwarded, no public discussion of why one header was gated and two were not. The narrowness is an oversight, not a considered scope. That mattered for the design, because it meant nobody had decided the other two should stay trusted — but as we will see, it did not end up justifying a change to the switch.

Upstream knew, in 2017

The most interesting thing found while writing this up is that none of it is news to upstream. The history is a small lesson in how a security decision decays.

August 2017. IpAddress / NotIpAddress condition support is added in PR #4736. During review the maintainer, @harshavardhana, raises exactly the concern this article is about: X-Forwarded-For is trivially spoofed, the left-most entry is the client’s own, and using it for a security decision would let a malicious client reach objects. The contributor accepts it and removes X-Forwarded-For support entirely, leaving only X-Real-IP, on the stated reasoning that a proxy sets it and a client cannot manipulate it. Merged five days later.

That reasoning is half right, and its unstated half is the whole problem: X-Real-IP is untamperable only if the proxy in front overwrites it. Nothing enforced that, and nothing told operators it was load-bearing.

Today. X-Forwarded-For is read first, ahead of X-Real-IP. The 2017 decision did not survive; it dissolved across later refactors of the condition-value plumbing rather than being reversed on purpose. There is no commit that says “we are re-admitting the spoofable header into policy decisions” — which is precisely how this class of decay happens.

August 2023. In discussion #17878 an operator reports that source IPs behind a load balancer are unreliable. The maintainer’s answer is unambiguous: without reliable source-IP visibility, IP-based restrictions are impractical, and the recommendation is to compartmentalise by tag or namespace instead. Marked working as intended.

So upstream’s own position — stated by a maintainer, in public — is do not rely on aws:SourceIp. That is a defensible engineering stance. What is missing is anywhere an operator would encounter it: it is not in the policy documentation, not in the condition-key reference, and not near the setting that appears to make it safe. An IpAddress condition is accepted without complaint and behaves as though it works.

That gap is the actual defect being fixed here, and it reframes the change. The allow-list is not overturning an upstream judgement; it is offering the mechanism that would make the 2017 concern answerable, to the deployments that want it. The documentation is doing the heavier lifting: writing down a contract that has been implicit since 2017 and contradicted by its own switch since 2025.

Why “put it behind a proxy” is not the answer

This is the standard advice, and it fails in two common, independent ways.

The proxy is not the only way in

On Kubernetes an Ingress and a ClusterIP Service routinely coexist. The Ingress sanitises headers; the Service does not, and any pod in the cluster can reach it. The security boundary is assumed to be the Ingress but is actually the pod network. The same shape appears in Pigsty deployments, where a load balancer fronts MinIO while the service ports remain reachable on the internal network.

The canonical nginx recipe preserves attacker input

The near-universal snippet is:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

$proxy_add_x_forwarded_for expands to $http_x_forwarded_for, $remote_addr — it appends to whatever the client sent. A client sending X-Forwarded-For: 1.2.3.4 causes nginx to forward 1.2.3.4, <real client>. MinIO takes the left-most element, which is the attacker’s.

So a correctly-proxied, hardened, no-direct-access deployment is still forgeable, because left-most parsing and append-style proxies are mutually incompatible. Only proxy_set_header X-Forwarded-For $remote_addr; (overwrite) is safe under the default mode, and that is not what operators copy from the documentation.

This is the finding that rules out a documentation-only fix. Deployment discipline cannot close it; the chain has to be read from the other end, which requires code.

The design

Rather than inventing a mechanism, we generalised one this fork already has. MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES — added during the LDAP STS throttling work — already implements CIDR allow-list parsing, catch-all rejection, and a right-to-left chain walk, scoped to rate-limit bucketing. The parser moved to internal/config and both paths now share it.

One setting selects the mode:

Under the allow-listed mode, X-Forwarded-For and Forwarded are read right-to-left, stepping over entries that name a listed proxy, and the first remaining address wins. Each proxy appends the peer it actually saw, so an entry the client injected sits to the left of the one its proxy wrote and the walk stops before reaching it. Appending proxies become safe.

GetSourceScheme is deliberately untouched. It feeds the Location URL in S3 responses rather than a policy decision, and suppressing it would hand http:// URLs to every deployment terminating TLS at a proxy.

Trade-offs

Whether to change the default

Changing the default to distrust forwarded headers would make every existing reverse-proxy deployment’s aws:SourceIp and audit addresses become the proxy’s address overnight. Policies could fail closed; audit continuity would break.

Not changing it leaves directly-reachable deployments exposed.

Decision: do not change it. But “do not change” is not the same as “stay silent”. The consequence is now written into the code comment and the operator documentation in as many words: under the default mode, an IpAddress condition is not access control and the audit address is not evidence. Making the cost visible so operators can choose is better than making a choice for them that detonates during a release window.

Whether to widen the existing switch

This is the decision we got wrong first and reversed, and it is the part of the story most worth recording.

The original brief asked that when an operator explicitly disables forwarded-header trust, clients must not be able to forge through an equivalent header. The obvious reading is “fix _MINIO_API_XFF_HEADER=off so it covers all three”, and that is what the first implementation did.

The case for widening was decent. Upstream’s PR title says “disable all X-Forwarded-For header handling”; its described scope is audit logs and IP-based access control, both of which are about not believing client-claimed addresses; the switch is undocumented, so its audience is small; and the failure direction is safe, since the fallback is the real TCP peer rather than an attacker-controlled value.

The case against turned out to be decisive. Widening it changes behaviour for a real population: operators whose proxy appends to X-Forwarded-For (polluted) but overwrites X-Real-IP (clean) may have discovered off as a way to get the correct address. Upstream’s TestXFFDisabled asserts exactly that behaviour — with the switch off and both headers present, X-Real-IP wins — so the behaviour is not merely incidental, it is pinned by a test. Those operators would have seen audit addresses silently flip from the real client to their proxy, and IpAddress conditions potentially start denying.

The reversal came from separating the goal from the mechanism. The goal was “a complete, enforceable way to turn this off exists”. Nothing required that the existing variable be the thing that provides it. Expressing it as MINIO_API_TRUSTED_PROXIES=none achieves the identical guarantee with zero effect on anyone who has not opted in.

Decision: leave _MINIO_API_XFF_HEADER exactly as upstream defined it. It gates X-Forwarded-For only, within whichever trust mode is in force. Upstream’s TestXFFDisabled is retained unmodified and still passes. The documentation now says plainly what the switch is not: it is a parsing switch, not a trust boundary, and a client refused one header simply sends another.

A secondary benefit: this collapses two interacting variables into one policy setting, so there is no longer a precedence rule ("off outranks the allow-list") for operators to learn and for us to get wrong.

Environment variable or config subsystem

Registering trusted_proxies as an api subsystem key would give mc admin config visibility, help text, and hot reload, matching how sts_trusted_proxies is done.

It would also introduce a window. Config subsystems load after the object layer initialises, so between process start and config application the trust policy would be empty — which under a “list is empty means trust everyone” reading is fail-open. A security boundary must not have a fail-open window. Separately, a trust boundary that can be changed at runtime is not obviously desirable.

Decision: environment variable. The cost is discoverability, and one bug described below.

Loopback is always trusted as a peer

The FTP and SFTP front-ends connect to the S3 layer over 127.0.0.1 and declare their session’s client with X-Forwarded-For (cmd/sftp-server-driver.go). An allow-list that does not exempt loopback attributes every FTP and SFTP request to the server itself.

The cost is that any process on the same host can forge. That is acceptable: an attacker who can open connections from localhost already has code execution on the host, and the threat model is lost well before this point. The FTP/SFTP regression, by contrast, would be certain and would affect everyone using those front-ends.

Decision: exempt loopback as a peer. Adversarial review then caught that the first implementation also treated loopback as a skippable chain entry, which is unnecessary for the FTP/SFTP case and actively harmful — see below. The two are now separate checks.

X-Forwarded-For versus X-Real-IP: genuinely unresolvable

Under the allow-listed mode, which header wins when both are present?

• A proxy that authors only X-Real-IP and relays the client’s X-Forwarded-For (some nginx configurations) → preferring X-Forwarded-For takes the forged value.
• A proxy that authors only X-Forwarded-For and relays the client’s X-Real-IP (AWS ALB) → preferring X-Real-IP takes the forged value.

Both are common, and the server cannot tell which situation it is in from the request. This is not an undecided question; it is undecidable without the operator telling us what their proxy authors.

Decision: prefer X-Forwarded-For. It is the only one of the two that carries a chain that can be checked against the allow-list, and this path decides access control rather than rate-limit bucketing, so the value that can be validated should win. It also keeps header precedence identical to the default mode, so switching modes does not silently change precedence as well.

This deliberately diverges from getSTSLDAPTrustedProxySourceIP, which prefers X-Real-IP. Two contradicting implementations of the same question in one codebase is a hazard in itself, so both sites now carry a comment naming the divergence and its reason, so that nobody “unifies” them without re-deciding. The operator documentation states the mitigation for both directions: strip whichever header your proxy does not author.

A broad allow-list is worse than no allow-list

This is the most counter-intuitive property and the one most likely to bite.

Entries on the list are skipped during the chain walk. So MINIO_API_TRUSTED_PROXIES=10.0.0.0/8, configured because the load balancer is 10.0.0.1, makes every client inside 10/8 skippable as well. A client at 10.5.5.5 sending X-Forwarded-For: 8.8.8.8 produces a chain of 8.8.8.8, 10.5.5.5; the walk steps over 10.5.5.5 as a “trusted hop” and returns 8.8.8.8.

A broad list therefore does not merely trust more peers — it lets those peers forge. nginx’s set_real_ip_from has the same property with real_ip_recursive.

There is no algorithmic fix: the list is doing double duty as “who may forward” and “whose address may be discarded”, and separating them would mean two lists to keep in sync. Decision: keep one list, and make the constraint prominent — a callout block in the operator documentation, and the rule stated in the code comment where the walk happens. Only /0 is rejected as a catch-all, and the documentation says explicitly that this is a guardrail rather than a proof, since 0.0.0.0/1,128.0.0.0/1 covers the same ground.

Multi-node forwarding

MinIO forwards requests between nodes for bucket-DNS routing, listing continuation, heal-by-token, batch jobs and pool decommissioning. The receiving node’s TCP peer is the forwarding node, not the client.

This is not an obscure path: a ListObjectsV2 continuation token carries the node index, so any client can cause its own request to be forwarded. Under none, that request is then evaluated with aws:SourceIp set to an internal node address — which an IpAddress condition allowing internal ranges would treat as a pass.

Decision: document it, and steer multi-node clusters to the allow-list. none cannot be corrected for this case, because it believes nothing by definition. Automatically seeding the cluster’s own addresses was considered and rejected: it needs DNS resolution at startup and re-resolution as node addresses change, which is more machinery and more failure modes than the explicit configuration it replaces.

Compatibility

The change was deliberately structured so that risk is not spread evenly across it. Every piece is either opt-in or dead code in the default configuration.

Evidence for the default path. unverifiedSourceIP is a verbatim copy of the original function body, including its quirks: the ", " separator, the fall-through when the left-most element is empty, and the acceptance of non-IP values such as _gazonk. An independent review verified behavioural parity against HEAD over 21 cases — empty X-Forwarded-For, a bare comma, a leading ", ", "," versus ", " separators, " , ", IPv4-mapped addresses, bracketed IPv6, non-IP junk, and all three Forwarded forms. Upstream’s TestGetSourceIP and TestXFFDisabled are both retained unmodified and pass.

A subtlety self-review caught. The new setting is read after MINIO_CONFIG_ENV_FILE is loaded, which is what makes it work in packaged deployments. The obvious tidiness move is to read _MINIO_API_XFF_HEADER in the same place — and that would have been a behaviour change, because upstream reads it at package initialisation, before environment files exist. An operator who wrote it into an environment file has it silently ignored today; picking it up would make an already-deployed setting suddenly start working, flipping their source addresses from the left-most X-Forwarded-For entry to X-Real-IP. The old switch therefore keeps upstream’s read timing along with upstream’s semantics, and a test pins that so nobody tidies it later. The quirk is documented instead: set it in the process environment if you want it honoured.

The defensive code that was not defending anything. Sharing the parser initially came with two extras: allow-list entries written in IPv4-mapped form were rewritten to the IPv4 prefix they denote, and the address being matched was unmapped and de-zoned. Both looked like corrections — an entry written ::ffff:192.168.1.10 is otherwise accepted and then matches nothing, which is a silent failure worth removing.

They were removed anyway, and the reason is worth recording. Both call paths reduce the address through net.ParseIP(...).String() before matching, and that already collapses ::ffff:10.0.0.1 to 10.0.0.1; a dual-stack listener reports an IPv4 peer in plain form regardless. So neither extra could be reached by a real request. Their only observable effect was on what the shared function meant for the LDAP STS allow-list that had been using it first — 18 differences that a test could see by calling the function directly and no deployment could.

Worse, one of them manufactured the fail-open described below: rewriting ::ffff:0:0/96 turned a /96 into 0.0.0.0/0. Deleting the rewrite removes the bug’s cause rather than ordering around it. What remains is pure code motion, verified identical to the previous implementation across every combination of 37 allow-list values and 21 peer addresses — zero parse differences, zero match differences. The wart it declined to fix (a mapped-form entry matches nothing) is the pre-existing behaviour, fails closed, and is now stated in the function’s own comment so the next person does not re-derive the same tempting fix.

The one residual risk worth naming. The default mode’s code path did change: there is now a switch and a function call in front of the original body. If that plumbing were wrong it would affect everyone, not just opt-in users. The parity testing above is why we believe it is not, but “verified equivalent over 21 cases” is a different claim from “provably identical”, and the honest version is the former.

What adversarial review found

An independent agent was tasked with breaking the first implementation. It found four real defects, all since fixed and covered by regression tests.

A silent fail-open, and the worst of the four. The trust policy was read in the package’s init(). But loadEnvVarsFromFiles() calls os.Setenv for everything in MINIO_CONFIG_ENV_FILE long afterwards — which is how MinIO is configured in essentially every packaged deployment. An operator putting MINIO_API_TRUSTED_PROXIES in /etc/default/minio would have got the historical trust-any-peer mode, with no error reported, and a malformed value would have been silently ignored rather than fatal. The policy is now applied in serverHandleEnvVars, which runs after the file load and before any listener.

Loopback skipped as a chain entry. Described above: the peer exemption was being reused as a hop exemption, which is a needless instance of the broad-list problem. Now two separate checks.

Two fail-closed correctness bugs. A zoned IPv6 peer (fe80::1%eth0) canonicalised to nothing, because net.ParseIP rejects zones — so such a peer could never be a trusted proxy. And an allow-list entry written in IPv4-mapped form (::ffff:192.168.1.10) was accepted at startup and then matched nothing at all, since netip.Prefix.Contains is false across differing bit widths.

Two further defects were found while writing the documentation rather than the code, which is its own small lesson:

Repeated header lines. Header.Get returns only the first header line. HAProxy’s option forwardfor adds a second X-Forwarded-For line rather than extending the first, so Get would hand back the client’s line and put the forged value right back where the right-to-left walk exists to avoid it. Now flattened across all lines with Header.Values.

The internal forwarder relayed client claims. internal/handlers/forwarder.go set X-Real-IP only when absent, so a client’s value passed between nodes unchanged. Under an allow-list that includes the cluster’s own nodes — the configuration we recommend — a client could thereby borrow a peer node’s authority. The forwarder now drops X-Real-IP and Forwarded when the incoming peer is not entitled to have set them. X-Forwarded-For needs no such handling, because Go’s ReverseProxy appends the true peer and the receiving node’s walk reaches that entry first.

A second round, after the rework

Reworking the setting warranted a second adversarial pass, which was worth running: differential testing found zero behavioural differences against HEAD across 4,745,520 source-IP resolutions and 345,600 forwarder rewrites, but it also found three more ways to fail open — one of them introduced by the first round’s own fix.

A catch-all smuggled in as an IPv4-mapped prefix. MINIO_API_TRUSTED_PROXIES=::ffff:0:0/96 is a /96 as written, so it passed the catch-all check; the rewrite that unmapped IPv4-mapped entries then turned it into 0.0.0.0/0, trusting every peer. The first fix moved the breadth check to the far side of the rewrite. The eventual fix deleted the rewrite, once it became clear it was unreachable by any real request — which removes the cause instead of guarding its output.

This is the one most worth dwelling on. The fail-open was manufactured by a fix for an unrelated fail-closed bug, and the fix for the fail-open was a reordering that left the manufacturing step in place. Two rounds of correction, both defensible, neither addressing the fact that the code should not have been there. Hardening changes deserve the same adversarial treatment as the code they harden, and “is this reachable at all?” belongs near the front of that treatment.

A deliberate value naming nobody. MINIO_API_TRUSTED_PROXIES="," parsed to an empty list and fell back to the permissive default. An empty unset variable must mean “default”, but a value the operator actually typed which names no proxy is a mistake, and answering it with trust-everyone is the one behaviour they cannot have wanted. It is now a startup error. Whitespace-only remains equivalent to unset, since that is what an empty shell variable expands to.

A remote value that could not be read. MinIO supports env:// indirection, where a variable’s value is fetched from a remote webhook. env.Get discards the error from that fetch and returns the empty string — which this code would have read as “unset”, reinstating trust-any-peer at exactly the moment the operator’s intent could not be determined. The setting is now read through env.LookupEnv so the error is surfaced and startup stops. This is a general hazard for any security-relevant setting read via env.Get, and worth remembering beyond this change.

A third pass, attacking from angles the first two shared

Both earlier passes attacked the resolver as a unit. Two things that neither could see:

Nothing had tested that the trust policy reaches a decision. Every test to that point checked what the resolver returned, and the one test at the policy layer only asserted that aws:SourceIp equalled the resolver’s output — in the default mode. So a version where the resolver was correct but the policy engine read something else would have passed everything. There is now a test that drives a forged X-Forwarded-For through getConditionValues into a real IpAddress evaluation under each mode: believed by default, ignored under none, ignored from an unlisted peer, and still honoured from the listed proxy. It passes, but it should have existed before the change was called done.

The allow-listed mode had a resource amplification the default mode does not. The chain was flattened into a slice before being walked, so a client behind a trusted proxy could turn the 1 MiB header allowance into roughly 33 MB of slice headers per request — around thirty-fold — plus a million-iteration walk. The default path never had this, because it uses strings.Index on the raw header. The walk now scans backwards over the header text in place, allocating nothing, and stops after 100 hops; real chains are a handful, the answer sits at the right-hand end, and running out of budget yields no address, which falls back to the peer. A test pins the zero-allocation property, because it is the kind of thing an innocent-looking refactor would undo.

Also corrected in this pass: the deployment contract was stated too narrowly. “The proxy must overwrite whichever headers it sets” misses the case that actually bites — a proxy that correctly authors only X-Real-IP or only Forwarded still relays the client’s X-Forwarded-For, and that is the header read first. The general rule, now stated as such, is to strip every source-address header the proxy does not itself write.

One reported finding was reviewed and not treated as a defect: the catch-all guard rejects /0 and nothing else, so 0.0.0.0/1,128.0.0.0/1 covers the same ground and is accepted. Tightening it would mean rejecting broad-but-not-/0 prefixes in the parser now shared with the LDAP allow-list, newly failing configurations that are valid today, to defend against a value no deployment realistically holds. The documentation states plainly that the check is a guardrail rather than a proof, and the callout about naming proxies instead of subnets is where the real defence lives.

Recommendations

By topology:

• Proxy you control, API port genuinely unreachable otherwise. Set nothing. But verify whether your proxy overwrites or appends: if the config says $proxy_add_x_forwarded_for, you are forgeable today. Switch to $remote_addr, or adopt the allow-list.
• Direct exposure, no proxy. MINIO_API_TRUSTED_PROXIES=none.
• Kubernetes or Pigsty, Ingress plus reachable Service. The allow-list, containing the proxy addresses and the MinIO node addresses. This is the configuration that makes an IpAddress condition mean anything.
• Any multi-node cluster. The allow-list with node addresses, not none.

Two rules apply to every allow-list deployment. Name proxies, not the subnet they occupy. And strip at the edge every source-address header your proxy does not itself write — listing a peer means believing all three headers from it, they are consulted in a fixed order, and a header your proxy leaves alone is entirely the client’s. A proxy that correctly authors only X-Real-IP, or only Forwarded, still relays the client’s X-Forwarded-For, which is read first.

What was not done

• Automatic seeding of cluster node addresses. Feasible via EndpointServerPools, but it needs DNS resolution and re-resolution on address changes. Explicit configuration was judged the smaller risk.
• Registration as an api config key. See the fail-open window above. Revisitable if observability turns out to matter more than the startup guarantee.
• ExistingObjectTag/*. The sibling defect from the condition-value hardening — it carries the request’s own tags rather than the object’s stored tags — remains open by decision, and is unaffected by any of this.

Verdict on severity

Default behaviour matches upstream, and MinIO has never documented aws:SourceIp as trustworthy on a directly-reachable deployment, so “the default is unsafe” is closer to a documentation defect than a vulnerability. One thing is squarely a defect, though: an operator set an explicit security switch and it did not do what its name and its only public description implied, and it failed silently. That is worth a record, scoped to the incompleteness of upstream’s _MINIO_API_XFF_HEADER rather than to anything the fork introduced.

minio/minio is archived, so there is no upstream to coordinate with — the same position as CVE-2026-42600.

3.14 - Sorted Is Not Increasing: How One Duplicate Part Number Doubled an Object

Status: Fixed on the local pgsty/minio branch as 22c1e41fd, unreleased Classification: Data correctness, not a vulnerability — see Why this is not a CVE Affected scope: All backends, any authenticated S3 client, on its own upload Tracking: pgsty/minio issue #49

One section of this article describes an unfixed process-level panic in a neighbouring code path. Hold publication until that is fixed and released.

Conclusions first

• sort.SliceIsSorted with a < predicate does not test strict increase. It tests the absence of an inversion. Equal neighbours contain no inversion, so [1,1] was accepted.
• Upload one 5 MiB part, complete with [1,1], and the server returns HTTP 200 and a 10 MiB object. The upload is then consumed: a corrected retry gets NoSuchUpload. The client cannot recover.
• Inherited from upstream, and old. The check has had this shape since 2016-08. Two refactors — 2017 and 2023 — rewrote it faithfully, because each preserved the predicate, and the predicate was never the problem.
• The fix is one loop at the handler layer. The object layer is left undefended by decision, and that IOU is written down here rather than left implicit.
• Three independent reviews found no defect in the fix. What they found was a comment that misstated why a neighbouring guard exists — and, through that comment, an unrelated node-level panic.

The verb, not the predicate

The code, as inherited:

if !sort.SliceIsSorted(complMultipartUpload.Parts, func(i, j int) bool {
	return complMultipartUpload.Parts[i].PartNumber < complMultipartUpload.Parts[j].PartNumber
}) {
	writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartOrder), r.URL)
	return
}

It reads as “reject unless the part numbers strictly increase.” It does not do that. IsSorted evaluates the predicate in the reversed direction only — for each neighbouring pair it asks less(i, i-1), i.e. “is this element smaller than the one before it,” and reports unsorted the moment one such inversion appears. For a pair of equal elements that question is false. No inversion, therefore sorted.

The consequence is worth stating precisely, because it is what makes the misuse survive review: no strict predicate can make IsSorted reject duplicates. The only spelling that works is the non-strict one — passing <= as the less function, so that equal neighbours register as an inversion. To ask for strictly increasing you would have to write the operator that reads as not strict. Every reviewer who checked that the predicate said < was checking the right character in the wrong function.

Our replacement drops IsSorted rather than trying to spell it correctly:

for i := 1; i < len(complMultipartUpload.Parts); i++ {
	if complMultipartUpload.Parts[i-1].PartNumber >= complMultipartUpload.Parts[i].PartNumber {
		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartOrder), r.URL)
		return
	}
}

The rejection-set delta is exactly one class: lists containing an adjacent equal pair. Everything previously rejected is still rejected; everything previously accepted, except duplicates, is still accepted. Non-adjacent duplicates come along for free — a strictly increasing sequence is globally distinct, so [1,2,1] and [1,3,2,3] are caught by the inversion they are forced to contain.

Two refactors preserved it faithfully

The archaeology is the most transferable part of this incident.

Both refactors were correct as refactors: they preserved behaviour exactly, which is what a refactor is supposed to do. The 2023 commit was a repository-wide cleanup with no bearing on multipart semantics at all. It carried the < across unchanged, and the < was never wrong — CompletedParts.Less needs < to be a valid sort.Interface.

The defect lived in the relationship between the predicate and the function it was handed to, and a refactor that moves the predicate cannot see that relationship. A decade, three shapes, one behaviour: an ordering check that answers a question adjacent to the one it appears to answer.

What it actually did

Measured against both erasure backends, through the real signed HTTP handler:

The ETag suffix is the part count the server believes it assembled. There is no internal disagreement to detect: the metadata, the size, and the ETag are all mutually consistent and all wrong. The object simply is not what was uploaded.

Two properties make this worse than a bad error code.

The upload is consumed. Assembly runs to completion and cleans up the multipart upload, so the corrected retry returns NoSuchUpload. A client that notices the wrong size cannot fix it by resending the right list; it has to start the whole upload over, if it still has the data.

It is reachable by accident. No adversary is required. Any client that appends a part to its completion list twice — a plausible bug in a resumable-upload wrapper, a retry path, or a list built by concatenation — silently gets a doubled part instead of a 400.

Why this is not a CVE

It belongs in this chronicle because it is a silent server-side correctness failure, and this is where we keep those. It is not a vulnerability, and we are not going to inflate it into one.

The request must carry the caller’s own credentials, address the caller’s own upload, and the damaged object is the caller’s own. There is no cross-tenant effect, no privilege change, no disclosure, and no path to another account’s data. What breaks is the guarantee that a completed multipart object equals the bytes you uploaded — serious, but a correctness guarantee, not an access-control boundary.

The entries around this one in this chronicle are authentication bypasses and path traversals. Filing this beside them under the same label would make every label in the table mean less.

The boundary decision, and what it costs

The object layer has no duplicate defence at all. erasureObjects.CompleteMultipartUpload sizes its output slice to the request (cmd/erasure-multipart.go:1249) and then resolves each requested part number against current metadata (:1255). The same number resolves twice, writes two identical ObjectPartInfo entries, and adds its size twice. AddObjectPart does deduplicate by part number, but it deduplicates the metadata slice, not the request. The 5 MiB minimum-size rule cannot help either, because the duplicated part is individually legal.

We fixed the handler and left that alone. The reasoning:

• It is the only entrance where a client-controlled list exists. The other four callers — batch, restore, decommission, rebalance — build their lists server-side from oi.Parts or 1..n, and are strictly increasing by construction.
• The required output is an S3 error code, which is an API-layer concern. The object layer’s error vocabulary maps to a different code, so intercepting lower would hand clients a less accurate diagnosis.
• Minimality. This fork ships narrow fixes, and a change in the assembly loop is not narrow.

The cost, recorded rather than implied: the uniqueness invariant now has exactly one enforcement point, and nothing enforces the enforcement. No compiler error and no test failure will greet the person who adds a fifth caller to the object layer; they will get a silently corrupted object. That is the same species of IOU the previous article recorded about getVolDir, and it is written down for the same reason: an unrecorded deliberate omission is indistinguishable from an oversight six months later.

What we deliberately did not add

Part numbers need not start at 1 and need not be consecutive. [1,3], [5,9] and [3] are all legal S3, and all still complete successfully.

This matters more than it sounds. “Also require the list to start at part 1” is a one-line addition that looks like tightening, would pass a casual review, and would break legal clients — anything that abandons a part after a failed upload and completes with what it has. The temptation is real precisely because the fix next door is about validating the same list.

So two test cases exist for no purpose other than to make that change fail. We verified they do their job by injecting the constraint and confirming that exactly those two cases went red and nothing else did. A guard rail nobody has fired once is a guess.

The one behaviour change we did not intend

A 14-input differential against the pre-fix build turned up exactly one behavioural change beyond duplicate rejection: [0,0] and [-1,-1] — lists that are both duplicated and out of range — moved from InvalidPart to InvalidPartOrder. Both are HTTP 400.

We accepted it, on the principle that a format error should outrank a state error: an ordering violation is decidable without reading any storage, while part existence is not. It also only affects requests that were going to fail regardless, so no client that previously succeeded can now fail.

On S3 fidelity itself we are making a documented inference, not a measurement. AWS defines InvalidPartOrder as the parts list not being in ascending order, and documents that part numbers may be non-consecutive; duplicates are not ascending. We did not verify this against a live AWS endpoint, and two independent reviewers reached the same conclusion by the same documentary route, which is agreement, not evidence.

Falsification, and a comment that was wrong

Two mutation experiments, in the discipline the previous article argued for — a test you have never watched fail is not yet a test.

Inject “must start at part 1.” Exactly the two gap cases went red; the four lists starting at 1 stayed green. The guard rail is targeted, not incidental.

Delete the neighbouring len(Parts) == 0 guard. The expected result was that an empty completion would produce some wrong-but-orderly error. The actual result was that the process panicked: the empty list reaches a storage decorator that indexes element zero of the part-path slice without a length check, on a goroutine that no recover can reach. The S3 face is masked by that one guard line, which has been there since 2022 and is not documented as load-bearing. It is tracked separately as an unfixed node-level defect, which is why this article is held.

And the part worth publishing at our own expense: the comment we wrote about that guard was wrong. It said dropping the length check would let an empty completion succeed — the opposite direction of the truth, and specifically the direction that understates danger. It was caught in review and corrected before the commit. A comment that misstates why a check exists is exactly how the check gets deleted three years later by someone tidying up.

Three acceptances, zero blocking findings

The change went through three independent gates before commit:

Stated plainly, because the honest version is less flattering than the table: none of the three found a defect in the fix. What review produced was the corrected comment and, through the mutation it prompted, the discovery of the unrelated panic. That is still a good return, but it is not the same as catching a bug in the patch, and the record should say which one happened.

The rebuilt-red-state detail is the one worth copying. A reviewer who reruns the author’s tests is checking the author’s arithmetic; a reviewer who reconstructs the broken state independently is checking the author’s claim.

Declined, and left open

Declined, deliberately:

• Two test additions — completing onto a pre-existing object, and giving each part distinct content so ordering is verified rather than just total size. Both are real improvements. Both were declined under a standing rule that this fork ships correctness and security fixes rather than test expansion, and the core invariant is already pinned by “the rejected request left no object and the upload still works.”
• XML root element name is not validated. A document with the wrong root but correct <Part> children is accepted. This is not a bypass — the same list still goes through the same check — it is pre-existing, and tightening it risks breaking real SDKs over namespace handling. Recorded, not fixed.

Left open, none of it in a released build as of 2026-08-03:

• Object-layer defence in depth for part uniqueness (see above).
• The empty-list panic in the storage decorator, tracked as a node-level defect.
• XML strictness, including <PartNumber>abc</PartNumber> returning 500 where 400 MalformedXML is correct.

The concurrent checksum work on completion (#46, #48, #50) was fenced off from this change entirely and shares no code with it.

Closing

The predicate was strict. The verb was not. A decade of review read the predicate — including the two commits that rewrote the line.

If only one sentence survives: check what the function does with the comparison, not just what the comparison says, and when you decide to leave the layer underneath undefended, write it down where the next person will trip over it, rather than trusting that they will re-derive your reasoning.

4 - Design Records

Design records capture the reasoning behind SILO maintenance decisions: the problem being solved, the compatibility boundary, rejected alternatives, implementation requirements, and the evidence required before release.

4.1 - CopyObject Checksums Must Cover Logical Object Bytes

This is the design and verification record for SILO #63.

Status: the checksum-domain fix was merged through PR #66; public release pending.
Related fixes: metadata-only transform state #67 through PR #69, and CopyObjectResult checksum fields #68 through PR #70; both merged, public release pending. Upstream client: minio-go #2295.
Release boundary: a merge does not prove that a release artifact, package, or container image already contains the fix.

The defect

CopyObject reads the source as logical object data, then may compress and encrypt the destination storage stream. The old handler installed a requested server-side checksum on a reader that already represented compressed bytes:

logical object -> S2 compression -> checksum -> optional encryption -> storage

The digest was valid but covered the wrong byte domain. A client downloading and independently hashing the object therefore obtained a different value. The API reproduction was deterministic:

stored CRC32 before the fix: hN7ytg==
logical object CRC32:         1WxbLg==

All five algorithms implemented by this SILO baseline were affected: CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. Compression combined with encryption made the wrong value nondeterministic because encrypted-stream S2 padding is randomized.

Accepted invariant

The logical checksum reader is now separate from storage transformation readers:

logical object
    -> server-side checksum
    -> optional S2 compression
    -> storage hash
    -> optional server-side encryption
    -> erasure coding and commit

The handler installs the hasher before starting the compression goroutine. PutObjReader retains the logical reader even when its active storage reader is replaced. At EOF, the object layer requires the checksum to exist, be valid, and match the expected base algorithm before committing metadata.

This reuses the checksumReader contract introduced for multipart upload. It adds no second abstraction, no second object read, and no new on-disk representation.

Verification boundary

The permanent API suite covers all five algorithms and default CRC64NVME; uncompressed, compressed, encrypted-only, and compressed-plus-encrypted destinations; SSE-C and SSE-S3; encrypted and compressed sources; versioned buckets; full and multipart-composite source checksums; in-place copy; zero-length and threshold data; indexed S2 streams; ETag; body round trip; HEAD/GET checksum mode; and internal invariant failures.

The regression is red on the unfixed baseline and green on the repaired tree. Focused race tests, shuffled repeated runs, full cmd tests, the CGO-disabled kqueue/dev CI shape, lint, vet, cross-compilation, compatibility guards, and remote CI were also required before merge.

Adjacent defects kept separate

Adversarial review found two inherited defects in nearby code:

1. A metadata/reference-only self-copy could change compression markers without rewriting referenced data. Versioned SSE-C key rotation could also fall into an invalid rewrite. This is isolated in #67.
2. Successful CopyObject XML omitted checksum elements after the checksum was committed. The server response fix is #68; minio-go also discarded those fields and is followed in #2295.

Legacy federated UploadPartCopy checksum recovery is a different API and remains #64.

The archived upstream minio/minio tree retains the original placement. silo-pkg does not own this reader chain. MCLI switches from server-side copy to download/upload when –checksum is requested, and SILO Console only passes CopyObject through minio-go, so neither required a duplicate server fix.

Existing objects

The repair affects future CopyObject operations. It does not scan or rewrite checksum metadata already stored by an affected version.

Objects are candidates for verification when they were created by CopyObject, destination compression matched their key or content type, and they carry an additional S3 checksum. Retrieve the object with checksum mode enabled, independently hash the downloaded logical bytes with the reported algorithm, and compare the Base64 values.

To repair an object, copy it to a new key while explicitly selecting the checksum algorithm. An in-place copy is possible with x-amz-metadata-directive: REPLACE, but it rewrites the object and replaces the current value on an unversioned bucket; a versioned bucket receives a new version. Validate retention, legal hold, metadata, tags, encryption keys, free capacity, and rollback requirements before bulk remediation.

SILO does not perform automatic online backfill because that would read and rewrite user data outside an explicit S3 operation.

4.2 - DSN-Only Database Notifications: A Compatibility Boundary for #53

This document is the product requirements and design record for SILO issue #53. It records the accepted compatibility boundary for PostgreSQL and MySQL bucket-notification targets before implementation begins.

Decision

SILO will retain PostgreSQL and MySQL notification targets, but support exactly one current configuration form for each:

• PostgreSQL requires a complete connection_string.
• MySQL requires a complete dsn_string.

The old five-field form — host, port, username, password, and database — remains unsupported by the current KV configuration system. SILO will not re-register those keys and will not synthesize a DSN from them during legacy migration.

The legacy migration contract is deliberately narrow:

This is a configuration-boundary decision, not removal of the database-notification feature.

Status: accepted design; implementation pending.
Owner: SILO server repository.
Tracking: pgsty/silo#53.
Target: the next SILO patch release after implementation and verification.

Context

SILO inherited two generations of database-notification configuration from MinIO.

The pre-KV JSON configuration could describe a database connection either as a complete string or as five fields:

host
port
username
password
database

The current KV configuration exposes only the driver-native form:

notify_postgres  -> connection_string
notify_mysql     -> dsn_string

This direction is not new. MinIO deprecated the five discrete fields in RELEASE.2020-04-10T03-34-42Z and instructed operators to move to connection_string or dsn_string. SILO’s current help tables, environment-variable documentation, and examples already present the complete string as the supported interface.

SILO is a new community fork with an explicit migration step. Its compatibility contract prioritizes the S3 and Admin APIs, current MINIO_* settings, on-disk data, and current KV configuration. It does not need to perpetuate every pre-2020 configuration spelling when a supported canonical form has existed for years.

The defect

The current legacy migration helpers, SetNotifyPostgres and SetNotifyMySQL, write both forms into the new KV configuration. Even when the old target already has a complete connection string, the helpers also emit all five discrete keys, usually with empty values.

The new parser rejects those keys because neither DefaultPostgresKVS nor DefaultMySQLKVS registers them. Key validation checks key presence, not whether the corresponding value is empty. Both legacy source forms therefore fail:

old complete string -> canonical string + five empty unknown keys -> rejected
old discrete fields -> empty canonical string + five populated unknown keys -> rejected

The failure is amplified by notification initialization. FetchEnabledTargets is fail-fast across notification subsystems: the first invalid subsystem returns an error and a nil target list. The caller logs the error and continues starting the object server, leaving healthy Webhook, Kafka, NATS, and other targets unavailable as well.

Merely returning an error from the two migration helpers does not fix that behavior. The error propagates through readConfigWithoutMigrate and initConfig, but initConfigSubsystem currently logs non-retriable configuration errors as “some features may be missing” and returns success. The server then starts without assigning globalServerConfig; notification failure is only one consequence, because region, storage class, compression, identity, and other stored settings may also be absent. The implementation must therefore carry a typed database-migration error to the startup boundary and make that error fatal. Classifying it as retriable is also wrong because the server would retry forever without any state change that could repair the configuration.

The resulting behavior is especially dangerous because object I/O still works. Operators can see a healthy S3 service while every configured event pipeline has stopped. Targets are never constructed, so delivery or later replay of events produced during the outage must not be assumed.

There is also a diagnostic-exposure issue. The unregistered password key has no sensitivity metadata and may be copied verbatim into health or diagnostic material. The registered connection_string and dsn_string keys are already treated as sensitive values.

Why the first fix was reverted

The first repair registered the five discrete keys and taught the parser to read them. That made migrated targets pass CheckValidKeys, and it appeared attractive because the target argument structures and constructors still contain code for the old fields.

It also broke the documented connection-string path.

The shared mc admin config set tokenizer discovers field boundaries by looking for registered key names. It is not fully quote-aware. Once port became a registered key, this valid input contained what looked like a second top-level field:

connection_string="host=db port=5432 dbname=events user=app"

The tokenizer split at the port= inside the quoted value, truncated connection_string, and handed the remainder to the port parser. The command then failed with invalid port.

Under the current tokenizer, registering common words such as host, port, and password creates a direct conflict between the connection-string grammar and the top-level KV grammar. The attempted registration fix was therefore reverted. Re-registering those keys is not an acceptable solution.

Product judgment

Database notification targets are a specialized but useful capability. They provide a direct database-backed namespace view or access journal without requiring an external event bus. That remains valuable for small deployments and for users already operating PostgreSQL or MySQL.

The legacy spelling of their connection parameters has much less value. A five-field model cannot represent the useful range of driver options: TLS modes and certificates, connection timeouts, application names, Unix sockets, multi-host PostgreSQL settings, MySQL driver parameters, and future driver capabilities. Supporting both forms also creates precedence, merging, redaction, and testing questions that do not exist with one canonical value.

The complete string is the better abstraction boundary: SILO owns notification semantics, while the database driver owns connection syntax.

The product decision is therefore to keep the capability and remove the compatibility illusion. An unsupported legacy target must be rejected clearly; it must not be accepted and transformed into a configuration that later disables unrelated targets.

Goals

1. Establish connection_string and dsn_string as the only supported live configuration interfaces for database notifications.
2. Allow a legacy JSON target that already contains the canonical string to cross the migration boundary without modification to its connection semantics.
3. Reject enabled discrete-only legacy targets before a partial or invalid KV configuration is activated.
4. Replace the current silent runtime failure mode of #53 — healthy targets disabled while the server appears healthy — with an explicit startup-time failure that operators must resolve before the server runs.
5. Ensure no migration error, log line, health report, or diagnostic bundle exposes a database password.
6. Remove the ten Postgres/MySQL exceptions from the source-level unregistered-write audit.
7. Make the compatibility boundary and operator remediation explicit in release and migration documentation.

Non-goals

• Supporting both DSN and discrete database fields in the current KV interface.
• Automatically synthesizing a DSN from old discrete fields.
• Rewriting the shared KV tokenizer.
• Changing FetchEnabledTargets fail-fast semantics in this patch.
• Silently skipping an enabled database target and continuing with partial notification coverage.
• Removing PostgreSQL or MySQL notification targets.
• Deleting the legacy struct fields needed to decode and identify unsupported input. They remain on shared target argument structs that are also used by live constructors, whose discrete-field connection-string synthesis is unreachable from current KV configuration; those fields must not become supported configuration keys.
• Correcting ignored errors from the other eight legacy notification setters. Their pre-existing silent-skip behavior remains unchanged in this narrowly scoped database-migration patch and requires a separate audit and design decision.

Functional requirements

Current configuration

1. notify_postgres accepts connection_string; notify_mysql accepts dsn_string.
2. The five discrete keys remain unregistered and rejected by current configuration commands.
3. Existing full strings must continue to support the database driver’s syntax, including parameters whose names contain host, port, user, password, or database.
4. No new public environment variables or KV keys are introduced.
5. The declared legacy variables MINIO_NOTIFY_POSTGRES_HOST/PORT/USERNAME/PASSWORD/DATABASE and their MySQL equivalents are not wired into current parsing and remain unsupported. They must not be documented as working alternatives to the complete-string variables.

Legacy migration

1. SetNotifyPostgres must return without emitting a target when the legacy target is disabled.
2. For an enabled target, SetNotifyPostgres must require a non-empty ConnectionString and write only registered Postgres keys. If both a canonical string and discrete fields are present, the canonical string wins and every discrete value is discarded.
3. SetNotifyMySQL must apply the equivalent rule to DSN.
4. Neither helper may emit host, port, username, password, or database.
5. A missing canonical string must return a typed or wrapped migration error identifying the subsystem and target name.
6. cmd/config-migrate.go must check and propagate both helper errors. Ignoring them is forbidden.
7. No partially migrated configuration may be activated or persisted after either helper fails.
8. Error text may name the required key and remediation, but must not include any connection-field value.
9. The propagated typed migration error must abort server startup. It must not be downgraded to the non-fatal “some features may be missing” path in initConfigSubsystem, and it must not enter the retriable-error loop.
10. Validation errors for a supplied canonical string follow the same startup-fatal and secrecy rules; wrapping must add target context without repeating the DSN or its components.

Recommended error shape:

notify_postgres:archive uses unsupported legacy discrete connection fields;
set connection_string before migrating to SILO

Operator remediation

An operator encountering the error must choose an explicit remediation path. This applies both before an initial switch to SILO and when upgrading a deployment that is already running SILO: legacy migration output is not persisted, so the same old JSON source can re-enter migration on every start. A deployment that currently starts with notifications silently broken can therefore fail to start after this repair until the source configuration is corrected.

1. On a compatible intermediate MinIO release, replace the old fields with connection_string or dsn_string, verify the target, and then migrate to SILO.
2. Disable or remove the legacy database target, migrate the server, and recreate the target with the canonical string afterward.
3. For a fresh SILO installation, create the target directly with the canonical string; no legacy migration is involved.
4. For an existing SILO deployment that still reads a legacy JSON file, stop on the previous working release, back up the source configuration, then convert, disable, or remove the database target before starting the fixed release. Do not delete or rewrite unrelated configuration.

Documentation must not suggest that a discrete-only target will be converted automatically.

Availability trade-off

This decision intentionally turns one unsupported configuration from a degraded startup into a hard startup failure. The immediate availability cost is real: a server that previously served objects while all notifications were silently dead may refuse to start after the repair.

That cost is accepted because an object server that appears healthy while configured event sinks are absent creates silent, potentially unrecoverable downstream data loss. SILO is a new fork with an explicit migration boundary, and the discrete form has been deprecated since 2020. A fatal, actionable precondition is preferable to an upgrade that reports success with reduced notification coverage. The release note must make this startup behavior prominent; it must not be buried as an internal migration cleanup.

Security requirements

1. The unsupported-input error must never format the legacy argument structure or its values.
2. Tests must use a sentinel password and assert that it is absent from returned errors and captured logs.
3. Migrated output must contain the registered sensitive connection-string key and no standalone password key.
4. If a diagnostic bundle was exported from an affected deployment before this repair, operators should treat the database password as potentially disclosed and rotate it.

Alternatives considered

Register and parse the discrete fields

Benefit: preserves the old source form and uses already existing argument fields.
Rejected because: registration makes common field names visible to the shared tokenizer and corrupts quoted connection strings. It also expands the supported public configuration surface after the fields were deprecated in 2020.

Synthesize a canonical string during migration

Benefit: preserves discrete-only legacy installations.
Rejected because: it creates permanent code and test ownership for an obsolete input form, including PostgreSQL quoting, MySQL DSN formatting, socket and IPv6 behavior, defaults, and future driver drift. For a new fork with an explicit migration boundary, the benefit does not justify the continuing surface.

Skip only the unsupported target

Benefit: keeps the object server and other notification targets running.
Rejected because: silently discarding a configured event sink can cause unobservable and unrecoverable event loss. A clear migration failure is safer than an apparently successful upgrade with reduced notification coverage.

Change global notification fail-fast behavior

Benefit: limits the blast radius of future invalid targets.
Rejected for this change because: it neither repairs the database target nor closes the credential-exposure path, and it changes system-wide error semantics. It may be evaluated independently with its own operational contract.

Remove database notification targets

Benefit: removes the complete database-specific maintenance surface.
Rejected because: the targets remain useful and self-contained. The defect belongs to an obsolete configuration form, not to the notification capability itself.

Implementation scope

The server change should remain narrow:

1. Update internal/config/notify/legacy.go so the two database setters emit only canonical registered keys and reject enabled targets without a canonical string.
2. Update cmd/config-migrate.go to propagate the two database-helper errors with subsystem and target context.
3. Define a typed database-migration error and update cmd/server-main.go so initConfigSubsystem returns it as fatal instead of logging and ignoring it. It must remain non-retriable.
4. Leave ignored errors from the other eight legacy notification setters unchanged in this patch; record them for a separate audit rather than expanding #53 implicitly.
5. Remove all ten Postgres/MySQL entries from knownUnregisteredWrites; the ratchet should become empty unless another independently justified legacy exception exists.
6. Add focused migration, startup, validation, secrecy, and coexistence tests.
7. Update database-notification and migration documentation in silo.pgsty.com.

The patch must not register the old keys, change the generic tokenizer, or refactor unrelated notification targets.

Acceptance criteria

The implementation is complete only when all of the following are demonstrated:

1. A legacy PostgreSQL target with a complete connection string migrates, passes CheckValidKeys, and is returned by GetNotifyPostgres unchanged.

2. A legacy MySQL target with a complete DSN does the equivalent.

3. Discrete-only enabled targets for both databases fail before target initialization with an actionable error containing the subsystem and target name, and server startup aborts.

4. Missing-string and malformed-string errors contain none of the sentinel host, username, password, database, or DSN values.

5. Disabled discrete legacy targets do not create configuration entries and do not block migration.

6. Migrated KVS output contains none of the ten discrete keys, including empty ones.

7. When a legacy target contains both a canonical string and conflicting discrete values, only the canonical string is migrated and no discrete sentinel appears in any output KVS value.

8. A SetKVS regression test using the real DefaultPostgresKVS and DefaultMySQLKVS key sets accepts a quoted connection string containing port=, host=, or password=.

9. A configuration containing healthy Webhook, Kafka, or NATS targets cannot reach FetchEnabledTargets with an invalid migrated database target because readConfigWithoutMigrate fails without yielding, persisting, or activating a partial configuration, and startup aborts on that typed error.

10. initConfigSubsystem returns the typed migration error; it neither logs-and-continues nor enters the retriable loop.

11. knownUnregisteredWrites no longer contains Postgres or MySQL exceptions.

12. The following verification passes:

    go test ./internal/config/notify ./internal/config ./internal/event/target -count=1
    go test -v ./cmd -run 'Test(ReadConfigWithoutMigrate|InitConfigSubsystem)' -count=1
    git diff --check

    The verbose cmd output must show that tests with both prefixes actually ran; a zero-match warning is a failed acceptance check. The normal server CI suite must also pass. In the documentation checkout, run make check.

Release and compatibility statement

The release note must describe this as an enforced compatibility boundary:

SILO database notification targets require connection_string for PostgreSQL and dsn_string for MySQL. The pre-2020 discrete host/port/username/password/database form is not migrated. Convert or recreate such targets before switching the deployment to SILO.

Deployments already running SILO with an old-format source configuration are equally affected: after this release the server will not start until each enabled legacy database target is converted, disabled, or removed.

The issue should close only after the repair is present in a published server tag. A merged patch, a local site build, and a published release are separate completion gates.

Review record

Claude Fable 5 reviewed the first draft at xhigh effort on 2026-08-23 and returned approve with required changes. The required calibration was incorporated: startup-fatal propagation now extends through initConfigSubsystem; already-running SILO deployments are covered; the availability trade-off is explicit; canonical-string precedence, dead legacy environment variables, other ignored helper errors, and executable tests are specified.

The same model then completed a final source-backed verification pass. Final verdict: approve, with no blocking findings. It confirmed that the English and Chinese records are aligned, the requirements are implementable against the current server tree, and the acceptance criteria cover the startup, migration, parser-regression, and secrecy boundaries.

4.3 - Preview Text, Never Execute It: SILO Console Text Preview PRD

Status: accepted design; implementation pending · Owner: pgsty/silo-console · Tracking: pgsty/silo#17 · Review: consensus of product, security, and frontend architecture reviews

SILO Console can preview images, PDFs, audio, and video, but not the small logs, text files, JSON documents, and XML documents that operators inspect every day. A correctly stored Content-Type does not help: these objects are classified as unsupported before the preview renderer is selected.

Restoring the old browser-native behavior would be easy. It would also be the wrong fix. An object in storage is controlled by the user who uploaded it. Loading that object as a same-origin HTML or XML document would turn a convenience feature into an execution boundary.

The accepted design therefore makes a stronger promise:

SILO previews eligible objects as bounded UTF-8 text. It never asks the browser to interpret their markup, MIME type, or file contents as a document.

This record fixes the product boundary, the resource limit, the security invariants, the implementation shape, and the evidence required before the feature can ship.

Decision

The first release will add a dedicated text preview type and a PreviewText component.

The contract is:

1. Preserve every existing image, PDF, audio, and video classification.
2. Only when the existing classifier returns none, consider a text fallback.
3. Admit the four target extensions or four exact passive text MIME types.
4. Fetch bytes through the ordinary authenticated download path, without preview=true.
5. Enforce a hard application read limit of 1 MiB.
6. Decode only strict UTF-8 and reject binary-looking content.
7. Render one React text node inside a scrollable <pre>.
8. Never use an iframe, HTML parser, XML parser, or HTML injection API.
9. Show the complete object or no object; do not show a truncated JSON or XML document.
10. Keep download available for files that are too large, invalidly encoded, or otherwise unavailable.

No Console API or S3 API change is required. The backend inline MIME allowlist is not expanded.

Current behavior

The defect is present in SILO Console v2.1.1, the version currently pinned by SILO when this design was written.

The frontend preview union contains only:

image | pdf | audio | video | none

Its extension table contains media formats, but not .log, .txt, .json, or .xml. Its MIME classifier likewise ignores text/plain, application/json, application/xml, and text/xml.

Runtime verification produced this split: