Introduction

Imagine a fleet of embedded devices deployed across the country, running smoothly for weeks—until, without warning, they stop communicating.

That was the mystery we faced when a customer reported a baffling issue in their widely deployed product. After extended uptime, devices would suddenly and silently stop communicating with the central server. The communication relied on a third-party driver, and while everything seemed stable on the surface, something was quietly going wrong under the hood.

This invisible failure severely impacted system reliability, forcing manual intervention and disrupting operations. Here’s how we diagnosed and resolved a memory leak that took its time to reveal itself—and nearly slipped through unnoticed.

Investigation and Analysis

We reviewed the integration of customer firmware with the third-party driver, which used dynamic memory allocation for various internal structures. In most cases, the driver internally managed memory cleanup. However, in certain specific usage scenarios, the responsibility to free allocated memory was passed to the developer—a detail that was easy to overlook.

Credit: https://medium.com/@dhleee0123/stack-heap-memory-86daeb1b48f7

Through detailed code analysis, we identified a couple of instances where the driver allocated memory, but the customer’s firmware did not properly free it.

To confirm the presence and scope of the memory leak, we tracked the allocated and freed memory at important locations in the code. Our testing showed a slow and consistent increase in memory usage over time, confirming the leak.

Eventually, memory exhaustion led to the failure of new memory allocations—causing the firmware to silently fail to maintain or establish communication with remote devices.

Solution

We suggested changes to the code to ensure memory was properly managed in all cases where the developer was responsible for cleanup. This included:

Ensuring deallocation was done where required.
Adding assertions and logging to catch any future memory mismanagement early.

After the customer implemented the changes, the memory usage stabilized, and communication with remote devices remained uninterrupted.

Credit: www.deusinmachina.net

Outcome

Remote devices now maintain continuous and stable communication over long periods.
Memory usage remains consistent, even after weeks of operation.
The customer regained full confidence in their deployment’s reliability, with no further manual resets required.

Key Takeaway

When working with third-party drivers that abstract away memory management, it is critical to understand the boundary of responsibility. In this case, inconsistent handling of dynamically allocated memory led to a subtle but severe memory leak. A combination of careful code review, targeted instrumentation, and a solid understanding of the driver’s behavior allowed for a precise and effective fix.