Simplifying intermediate layers means consolidating what stands between an MQTT device and the enterprise data platform down to a single component.

Not one more middleware to govern, but rather a direct path. The broker writes messages directly to where the data already lives: Apache Kafka, Confluent, Redpanda, IBM Event Streams, or Microsoft Azure Event Hubs.

In enterprise scenarios, event streaming infrastructure is typically already established and managed, while MQTT endpoints (field sensors, plants, vehicular telemetry, and IoT devices) scale exponentially. The architectural bottleneck does not lie so much in the connectivity between these two worlds, but in the additive integration pattern: stacking a standalone broker on top of an integration layer (e.g., Kafka Connect) and multiple source/sink connectors. This cumulative stack generates technical debt that inevitably clashes with infrastructure cost reviews.

The engineering challenge is therefore no longer just the functional validation of the data flow. What is increasingly necessary is the optimization of operational overhead: minimizing the nodes, configurations, and components required to guarantee uptime.

The Traditional Architecture is a File with Too Many Layers

Anyone who has built an MQTT-Kafka pipeline in production recognizes the pattern: a third-party MQTT broker, an integration layer (Kafka Connect or a custom pipeline), and a pair of source/sink connectors. Three distinct components with three release cycles, three configurations to version control, and three maintenance windows to negotiate with operations.

If you were to use a comparison, you might think of a Photoshop file layered over months. Every layer has a historical reason, but none can be deleted without first checking what lies underneath; as a result, the total file size takes its toll every single time you open it.

Stratification becomes an operational problem: each layer adds latency, storage cost, configuration surface area, and one more way for things to break. In enterprise MQTT architectures, the bill always arrives at the exact same point: the moment you need to scale from tens of thousands to millions of connected clients, and the pipeline begins to require a complete redesign.

Waterstream’s Infrastructure Simplification

The alternative works like developing a RAW file directly: no intermediate layers, no alteration, just ready-to-use MQTT data.

Waterstream is an MQTT-native streaming integration for the enterprise data ecosystem. It is the broker itself, writing directly into the streaming platform. One process, one persistence layer, one thing to monitor.

The technical condition that enables this compression is the complete absence of local state. Waterstream retains nothing on the node: sessions, subscriptions, retained messages — every byte of MQTT state resides in the streaming platform. From this stems horizontal scaling behind a load balancer, the absence of leader elections and split-brain scenarios (which are delegated to Kafka), and the fact that there is no data to back up that isn’t already inside the platform.

The point, therefore, is not to add a more sophisticated component: it is to remove the ones that were already there.

The Layers You Can Remove

Every MQTT-to-Kafka setup adds layers. These are the ones rendered redundant by an MQTT-native integration:

• Maintenance: One component to patch and upgrade instead of three, with only a single version compatibility check required for each upgrade.
• Configuration: No connectors to map, no Kafka Connect topologies to keep aligned with topic naming conventions (only Waterstream’s configuration remains, which is significantly leaner).
• Double Storage: MQTT messages are not persisted first in the broker and then in the platform: there is only one place where the data lives.
• Duplication Risk: With a single persistence layer, there is no possibility of misalignment between two copies of the same truth.
• Unvalidated Data: Validation takes place right at the ingestion point, not downstream from a chain of transformations that has already propagated the payload.

Layered Pipeline and Direct Path

A traditional pipeline requires operating three components — broker, integration layer, and connectors — whereas an MQTT-native integration requires only one.

Persistence layers drop from two (the broker and the platform) down to one: the platform. MQTT state is no longer local to the broker requiring replication, but lives directly in the streaming layer. This also changes fault behavior: where a layered pipeline must rebuild the local state of a crashed node, a stateless node restarts empty and immediately resumes service.

Regarding scalability, the traditional pipeline forces a redesign beyond certain thresholds of concurrent connections; the direct path scales simply by adding nodes. Furthermore, the target destination platform is no longer constrained by available connectors: Kafka, Confluent, Redpanda, IBM Event Streams, and Azure Event Hubs are all reachable using the exact same product and architecture.

Scalability, TCO Reduction, and Real-World Impact

An architectural simplification is measured by concrete operational metrics. By choosing direct integration, the advantages cluster around three main fronts:

• Simplification: One component instead of three means less to deploy, less to monitor, and less to update. It is the single metric an architect can present in a review without having to explain a complex benchmark.
• Scalability: Scaling is linear up to millions of concurrent clients: over 1,000,000 simultaneous connections on 12 modestly sized nodes (e.g. CPU e RAM) — the exact number varies with traffic patterns and payload size, but it illustrates the order of magnitude achievable without altering the underlying design. Adding capacity simply means adding nodes, not questioning the architecture.
• TCO Reduction: Eliminating intermediate layers means no longer paying licenses for unnecessary middleware and driving cloud storage costs tied to “double persistence” logs down to zero. A lighter infrastructure footprint doesn’t just reduce the bill; it causes a steep decline in the man-hours spent by the team managing and troubleshooting extra components.

Specifically, removing intermediate layers translates into tangible operational benefits in high-criticality environments where latency, connectivity, and data integrity dictate service effectiveness.

In predictive monitoring for industrial maintenance, for example, sensor telemetry must reach anomaly detection models while the intervention window is still open: every layer traversed steals time from that window, and every layer represents a single point where the stream could break during scheduled maintenance.

In energy efficiency and smart grid management, data arrives from hundreds of distributed sites with unstable connectivity. Core and Edge deployment allows resilient buffering at remote sites and convergence toward a central cluster without introducing a second architecture to maintain.

Finally, in digital health and medical monitoring, data duplication isn’t just a cost issue, but it’s a matter of clinical reliability: a single authoritative copy of a vital sign, written directly into the platform, eliminates any question as to which of the two versions is the correct one.

Conclusion

Returning to the initial metaphor: it is not about adding a more sophisticated development layer, but about eliminating those that have accumulated over time.

Waterstream supports the full MQTT 3.1 and 5 specifications, WebSockets, and bridge modes. Consequently, migration doesn’t need to start as a massive standalone project: you can route a single class of devices onto the new path, observe the simplification, scalability, and TCO impact on real-world data from that subset, and only then decide whether to extend it to the rest of your device fleet. In the meantime, the remainder of your existing pipeline continues to function without modification.

The use cases outlined above are just three examples of environments where the ratio between managed components and generated value shifts radically. To explore reference architectures and technical documentation, or to evaluate a free trial in other scenarios, visit waterstream.io. 

To learn how to adapt Waterstream to your infrastructure or dive deeper into a specific application scenario, reach out directly to our technical team.