The Challenges of Deploying a NEW Coupled System - How to Deal with Field Edge Cases

While deploying Delos’ DARWIN™ product, I came to realize that any new coupled system will inevitably encounter field edge cases regardless of how carefully it is designed. In these systems, each node owner understands their specific layer, and every individual component operates as intended; however, because the integrated system itself is brand new, a dedicated overarching expert rarely exists. The real test is whether a single individual takes accountability when an unforeseen issue arises, possesses sufficient visibility to diagnose the problem, and can swiftly adapt the system’s behavior across all units before severe impacts occur. When unexpected edge cases stalled our deployments, I stepped into that accountability role, bridging the seams between disparate technical disciplines.

DARWIN™ was a new coupled system with multiple technologies operating sequentially to execute an air quality remediation algorithm. Sensors reported data to an on-premise supervisory hub. The hub ran a remediation algorithm against that data, and when it judged that a zone had bad air quality, it sent a command to the automation controller. The controller activated thermostats to run the HVAC fan in the affected zone. Sensing, decision, actuation, across four subsystems built by four different organizations, running on four different software platforms.

During deployments, we encountered a unique edge case where spaces were being continuously heated or cooled. My first step was to decouple the system by investigating the layers I could observe. I analyzed logs from the firmware developer, which showed that the code between the on-premise hub and the system controller was passing along exactly what it had received. I then reviewed the automation controller with the integrator; it was correctly executing its commands. The sensor platform also confirmed that the air quality events were real. Every node in the chain was accounted for except the supervisory hub responsible for the decision-making. Since the platform had been built by a contracted vendor, its logs were unavailable to me, and they maintained that the issue was not on their end. I couldn’t open the “black box,” so I made the rest of the system observable until it was the only remaining variable.

In the absence of a definitive root cause analysis, I deduced that when the supervisory hub signaled an HVAC fan to operate, it blew air over coils remaining hot or cold from a cycle that had wrapped up moments prior. To be clear, this was purely my inference and was not officially confirmed. This was not a flaw in the algorithm itself, which explicitly contained a guard preventing remediation while a heating or cooling cycle was active. Rather, it lacked a mechanism to detect if a cycle had recently concluded, since that historical state was exposed by the controller to the hub. Consequently, blowing air across a cold coil pulled the zone below its set temperature target, prompting the thermostat to correctly trigger another cycle. The thermostat and remediation logic were both functioning as designed, yet operating in direct conflict.

To address the thermal discomfort, I recommended temporarily disabling the remediation feature fleet-wide while we investigated. Once leadership approved, the development vendor accessed the hubs remotely and shut down the algorithm individually on each unit. While this manual process succeeded for our current footprint, it clearly would not scale; we lacked a centralized mechanism to update system behavior across the entire fleet simultaneously. Had our operations been ten times larger, this limitation would have created a critical bottleneck. Eventually, a definitive fix was deployed, permanently resolving the issue.

Here are my key takeaways for managing coupled systems:

1. Guard State Transitions, Not Just Current State: The most critical risks in an actuation decision occur immediately following a mode shift, rather than during steady-state operation.
2. Audit Historical Context Before Commanding Third-Party Hardware: Always determine past actions in addition to current status before issuing commands to external equipment. If historical state data is missing from the interface, treat its integration as essential project scope rather than an optional future enhancement.
3. Enforce Vendor Observability Contracts: Require any vendor whose software controls physical equipment to provide clear logging that answers “what decision was made, using what data, and at what time.” Without this transparency, resolving incidents devolves into debate rather than systematic diagnosis.
4. Establish Rapid Fleet-Wide Control: Develop automated, centralized mechanisms to alter system behavior across the entire fleet in minutes before critical situations demand it.
5. Connect Field Feedback Directly to Design: When novel systems lack established experts, acknowledge that true expertise is forged directly in the field. Build robust channels to feed frontline operational insights straight back into engineering and design.

Reflecting on this and similar experiences, I believe that regardless of how carefully a novel system is designed, field edge cases are inevitable. The true test is never whether we prevented every issue, but rather: Is someone accountable when they arise? Can those on-site gather sufficient visibility to explain what occurred? And can system behavior be updated fleet-wide quickly enough to make a difference? Someone must bridge the gaps across disciplines—a critical role.