How to Buy Transparency Before You Sign the Contract

If you’re considering contracting a vendor to build your software, make root-cause analysis a contractual deliverable for any customer-affecting incident. Expect pushback, but understand why you’re getting it: when a vendor admits a defect, that admission has liability consequences for them. You have to give a protected path to be honest. Write the root-cause-analysis (RCA) as a blameless post-mortem — what was supposed to happen, what actually happened, where the gap was, and what changes. If you remove the blame, you remove the reason to defend.

I came to that conclusion after owning integration for a system where air-quality sensors reported to a supervisory hub, the hub decided when the air in a zone needed remediation, and it commanded an automation controller, which drove the HVAC through the thermostat. Four subsystems, owned by four different organizations, in one closed loop. Before we put our hub in the middle of it, the HVAC had run fine. After our integration, the complaints started: the property was too cold.

The hub’s software had been built by a contracted development vendor, which we licensed it from; the vendor owned the code, the API, and the IP. As the systems integrator, I asked for architecture documentation and didn’t get it — the reason given was that the software was proprietary. When the problem surfaced I asked for help diagnosing it, and the answer from the vendor was that everything looked good on their end. No logs came with that answer, and there were none I could pull myself. On the customer’s site, the platform was a black box.

Inside the company, the first explanation was the integration. I understand why: when a coupled system fails, the integrator is the nearest person standing. But the customer doesn’t care who is at fault, so I made an offer — if the integration was the problem, let’s have the vendor’s engineer on a working session and I would go back on site and redo the whole thing in the field, following their lead.

We scheduled working sessions with the vendor, and I asked for the developer who had actually built the module to join. Every session was the same manager instead, someone removed from the codebase, who would take the issue to his internal team and come back with one of two answers: the system is working fine, or we’ll get back to you — which later became the system is working fine. So I started collecting logs from the other nodes in the loop, from the automation-controller engineer and the firmware developer, and passing them to our product manager, who passed them to the vendor’s project manager.

The controller and firmware logs showed both of those layers correctly executing what the hub was sending them, which meant I could account for every node in the chain except one. The second piece of evidence was that the same fault appeared in other deployments. A defect that follows the product from site to site is not an integration problem.

In one of our internal working sessions I briefed our CTO and COO, and I made a deliberate choice not to make a technical argument. I told them what the customer was experiencing, that our customers sit in close proximity and talk to each other, and that my stopgap of taking the hub offline stopped the discomfort but also meant we were no longer delivering the product in full. Leadership took it to the vendor’s CEO, and the issue got resolved. Then the vendor’s openness slowly drifted back to where it had started. Executive pressure works, but it fades, because pressure is temporary, but incentives are permanent.

Initially I read the vendor’s behavior as evasion. I’ve come to think it was rational — admitting fault carried liability for them. Our contract bought us working software. It didn’t buy us visibility, documentation, or the whole truth about that software. The opacity wasn’t the vendor breaking the deal; as far as I can tell, it was the deal working exactly as written.

This particular loop ran inside one building, but swap that building for a plant floor or a facility on a base and the shape holds: sensing, decision, and actuation owned by different organizations, and every failure landing in the seams between them.

My own rule from those years is to take psychological ownership of the whole product regardless of where you sit in the org chart or which slice of delivery is formally yours, because the product only scales if the customer becomes its advocate, and the customer doesn’t experience your org chart. That ownership got me evidence at every node I could reach; it could not get me inside the one node that mattered.

None of that gets fixed by escalating harder. It gets fixed where the contract is written. So alongside the RCA obligation, I would name the engineer who built the module as the primary voice in any critical working session, because a manager relaying between you and the code loses information in both directions. And I would require that evidence, meaning logs, be retrievable by the people in the field, because without evidence every incident turns into a negotiation instead of a diagnosis. A failure that never gets examined is a failure that repeats. If you’re the one signing, buy the transparency up front — the person in the field can’t retrofit it.