In 2021 I launched GMK Arch, a keycap set built around Arch Linux visual language. It sold approximately 4,900 line items across roughly 10 regions in 30 days for approximately $432K. Supplier coordination, demand forecasting, pricing, and international fulfillment were one person's job, with no prior hardware shipping experience.
It shaped how I think about software systems more than any single engineering project. Here's why.
Demand forecasting is a systems problem
When you run a group buy, you don't know the final demand until orders close. You commit to manufacturing quantities weeks before you know the real number. I started with an expectation of approximately 15 units for WITF (a later project), watched interest climb toward approximately 100, then watched it settle back to approximately 50 as timing and market conditions shifted.
This is the same problem as capacity planning in distributed systems. You don't know the real load until production traffic arrives. You commit resources based on projections. The systems that work are the ones that degrade gracefully when projections are wrong — not the ones that require perfect forecasts.
When I built Substrate's provider routing layer, I applied the same principle: design for the capacity you expect, but make the degradation path explicit when reality diverges from the plan. Circuit breakers, fallback chains, and budget enforcement are the software equivalent of renegotiating with your supplier mid-production.
Supplier coordination is dependency management
GMK Arch required coordinating with GMK (the manufacturer), 10+ regional vendors across US, Canada, South America, EU, Oceania, Southeast Asia, UK, Korea, China, and Norway, plus designers, material suppliers, and logistics providers. Each had its own timeline, constraints, and failure modes.
The software parallel is dependency management in a multi-service system. Each provider (Anthropic, OpenAI, Gemini, Groq) has different rate limits, different failure modes, different cooldown semantics. When I built OmniRoute's routing intelligence, the lessons from supplier coordination were immediately applicable: map the failure modes of each dependency, build explicit fallback paths, and don't assume one provider's behavior generalizes to all.
Cost reduction is architecture optimization
Through supplier negotiation and process optimization I cut the WITF Board's per-unit cost from over $500 to approximately $350 all-in, more than 40%. The savings came from knowing the manufacturing process well enough to find the costs that could move without hurting quality.
This maps directly to performance work in software. The biggest gains come from knowing the system deeply enough to find the real bottlenecks, not from generic optimizations. When I optimized ShareCLI's process observation layer, the 40% overhead reduction came from knowing the Linux kernel's process accounting well enough to avoid redundant syscalls — the same principle as knowing a supplier's cost structure well enough to negotiate.
Fulfillment is deployment
Getting 4,900 units from a factory in China to 10 countries with different customs regulations, shipping carriers, and delivery expectations is a logistics problem that parallels software deployment. You need rollback paths (what happens when a shipment is delayed?), monitoring (tracking numbers, delivery confirmations), and graceful degradation (what happens when one region's customs process blocks imports?).
The WITF Board taught me this more viscerally. When our external pick-and-pack provider changed terms mid-fulfillment, I had to adapt the distribution model in real time. That is the same as a cloud provider changing its API mid-deployment — the architecture has to absorb the change without failing the release.
The durable lesson
The durable lesson from physical product work is that systems engineering isn't just about code. It's about understanding constraints, mapping failure modes, building explicit fallback paths, and designing for graceful degradation — whether the system is a keycap group buy or a multi-provider AI routing layer.
The 4,900 line items shipped. The 10-region retail network held. The $432K arrived. The engineering lessons from that process still shape how I build software.