
<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Confluent Archives - Waterstream.io | Simplify MQTT Data Integration</title>
	<atom:link href="https://waterstream.io/category/confluent/feed/" rel="self" type="application/rss+xml" />
	<link>https://waterstream.io/category/confluent/</link>
	<description>High performance MQTT Broker</description>
	<lastBuildDate>Tue, 30 Sep 2025 12:38:59 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	

<image>
	<url>https://waterstream.io/wp-content/uploads/2022/06/waterstreamfavicon-01.png</url>
	<title>Confluent Archives - Waterstream.io | Simplify MQTT Data Integration</title>
	<link>https://waterstream.io/category/confluent/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Waterstream Joins the Connect with Confluent Partner Program</title>
		<link>https://waterstream.io/2023/07/18/waterstream-joins-the-connect-with-confluent-partner-program/</link>
		
		<dc:creator><![CDATA[Waterstream]]></dc:creator>
		<pubDate>Tue, 18 Jul 2023 20:53:59 +0000</pubDate>
				<category><![CDATA[Confluent]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[MQTT]]></category>
		<category><![CDATA[News]]></category>
		<category><![CDATA[Press release]]></category>
		<guid isPermaLink="false">https://waterstream.io/?p=5481</guid>

					<description><![CDATA[<p>The easiest way to directly integrate data streaming with the Internet of Things (IoT) leveraging the standard MQTT protocol  [Milan] – July 18, 2023 – Waterstream today announced it has joined the Connect with Confluent partner program. This new program helps organizations accelerate the development of real-time applications through a native integration with Confluent Cloud. [&#8230;]</p>
<p>The post <a href="https://waterstream.io/2023/07/18/waterstream-joins-the-connect-with-confluent-partner-program/">Waterstream Joins the Connect with Confluent Partner Program</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></description>
			<style id="elementor-post-5481">.elementor-widget-image .widget-image-caption{color:var( --e-global-color-text );font-family:var( --e-global-typography-text-font-family ), Sans-serif;font-weight:var( --e-global-typography-text-font-weight );}.elementor-widget-text-editor{font-family:var( --e-global-typography-text-font-family ), Sans-serif;font-weight:var( --e-global-typography-text-font-weight );color:var( --e-global-color-text );}.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:var( --e-global-color-primary );}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap, .elementor-widget-text-editor.elementor-drop-cap-view-default .elementor-drop-cap{color:var( --e-global-color-primary );border-color:var( --e-global-color-primary );}</style>							<content:encoded><![CDATA[		<div data-elementor-type="wp-post" data-elementor-id="5481" class="elementor elementor-5481" data-elementor-post-type="post">
						<section class="elementor-section elementor-top-section elementor-element elementor-element-d6d3dd2 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="d6d3dd2" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-ee2301e" data-id="ee2301e" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-123956f elementor-widget elementor-widget-image" data-id="123956f" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
															<img fetchpriority="high" decoding="async" width="900" height="600" src="https://waterstream.io/wp-content/uploads/2023/07/Confluent-Partner-Program.jpg" class="attachment-large size-large wp-image-5975" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/Confluent-Partner-Program.jpg 900w, https://waterstream.io/wp-content/uploads/2023/07/Confluent-Partner-Program-300x200.jpg 300w, https://waterstream.io/wp-content/uploads/2023/07/Confluent-Partner-Program-768x512.jpg 768w" sizes="(max-width: 900px) 100vw, 900px" />															</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-1268ed4 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="1268ed4" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-eb5d7a1" data-id="eb5d7a1" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-87ef47c elementor-widget elementor-widget-text-editor" data-id="87ef47c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<h4 dir="ltr" style="line-height: 1.681547; margin: 17pt 8pt 0pt; text-align: center;"><span style="font-family: Montserrat, sans-serif; font-size: 10pt; font-style: italic; font-variant-caps: normal; white-space: pre-wrap;">The easiest way to directly integrate data streaming with the Internet of Things (IoT) leveraging the standard MQTT protocol </span></h4><p dir="ltr" style="line-height: 1.681548; margin-right: 7pt; text-indent: 2pt; margin-top: 18pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: bold; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">[Milan] – July 18, 2023 – </span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream today announced it has joined the </span><a style="text-decoration: none;" href="https://www.confluent.io/partners/connect"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Connect with Confluent</span></a><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> partner program. This new program helps organizations accelerate the development of real-time applications through a native integration with </span><a style="text-decoration: none;" href="https://www.confluent.io/confluent-cloud/?utm_campaign=tm.pmm_cd.2023_partner_cwc_waterstream_generic&amp;utm_source=waterstream&amp;utm_medium=partnerref"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Confluent Cloud</span></a><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Organizations now have the best experience for working with data streams within MQTT clients, paving a faster path to powering next generation customer experiences and business operations with real-time data. </span></p><p dir="ltr" style="line-height: 1.44; margin-left: 38pt; margin-top: 15pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">&#8220;Joining the Connect with Confluent partner program is an exciting milestone for Waterstream. This partnership allows us to directly integrate Waterstream Cloud with Confluent Cloud, opening up endless possibilities for our customers. </span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By leveraging Confluent Cloud&#8217;s powerful data streaming platform, we provide a cost  efficient solution to connect millions of MQTT clients to the Apache Kafka</span><span style="font-size: 11pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">®</span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ecosystem with just a few clicks. Waterstream brings exciting new features like replaying MQTT messages on the client or data validation with Confluent’s fully managed Schema Registry. </span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We are thrilled to be a part of the initial launch of this program and we are confident this partnership will deliver great value to customers.&#8221;</span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">– Paolo Castagna, Strategic and Business Advisor at Waterstream</span></p><p dir="ltr" style="line-height: 1.681549; margin: 16pt 4pt 0pt 1pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Connect with Confluent gives organizations direct access to Confluent Cloud, the cloud-native and complete data streaming platform that processes more than an exabyte of data per yearIt’s now easier than ever for organizations to stream data from anywhere with MQTT clients like cars, smart bands, scooters or even browsers with a fully managed Kafka service that spans hybrid, multi-cloud, and on-premises environments. Moreover, the program supercharges partners’ go-to-market efforts with access to Confluent engineering, sales, and marketing resources. This ensures customer success at every stage from onboarding through technical support. </span></p><p dir="ltr" style="line-height: 1.681547; margin: 15pt 3pt 0pt 1pt; text-indent: 1pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: italic; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“</span><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Data streaming is now a critical business requirement as companies shift toward a digital-first approach to everything,” said Paul Mac Farland, vice president, partner and innovation ecosystem, Confluent. “However, many companies don’t have the resources needed to successfully bring a complete set of data streaming capabilities to their applications and end users. Connect with Confluent solves this problem, helping Confluent technology partners accelerate their customers’ data-driven ambitions so they can win in the modern digital era.” </span></p><p dir="ltr" style="line-height: 1.44; margin-left: 1pt; margin-top: 15pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>Get started with Waterstream.</b></span></p><p dir="ltr" style="line-height: 1.681531; margin: 16pt 26pt 0pt 1pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The easiest way to get started with using Waterstream is with </span><a style="text-decoration: none;" href="https://waterstream.cloud/"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream Cloud</span></a><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, with Waterstream Cloud you can create an MQTT Broker connected with Confluent and Kafka data streams with just a few clicks. If you prefer a self-managed approach with greater control and flexibility, Waterstream also provides this option. To get started with the self-managed solution, </span><a style="text-decoration: none;" href="https://try.waterstream.io/"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">follow this link</span></a><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></p><p dir="ltr" style="line-height: 1.44; margin-top: 15pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: italic; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">About Waterstream</span></p><p dir="ltr" style="line-height: 1.44; margin-top: 15pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: italic; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream is a leading technology company specializing in data streaming solutions. Waterstream innovative MQTT Broker expands the Kafka ecosystem with IoT use cases, supporting the most popular IoT protocol with simple and efficient integration. Waterstream is available on edge, on-prem, public and private clouds, and as a fully managed solution with Waterstream Cloud.</span></p><p dir="ltr" style="line-height: 1.44; margin-left: 1pt; margin-top: 15pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: italic; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream has formed a strategic partnership with Confluent, the creators of Apache Kafka, to leverage the power of Confluent Cloud. This integration combines the most popular streaming platform with MQTT allowing businesses to process and analyze real-time data seamlessly from any device.</span></p><p dir="ltr" style="line-height: 1.44; margin-left: 1pt; margin-top: 15pt; margin-bottom: 0pt;"><span style="font-size: 10pt; font-family: Montserrat, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: italic; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream has received investment and support from CDP Venture Capital and Digital Magics. These partnerships provide Waterstream with resources and expertise to continue to enhance and evolve its platform. </span></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				</div>
		<p>The post <a href="https://waterstream.io/2023/07/18/waterstream-joins-the-connect-with-confluent-partner-program/">Waterstream Joins the Connect with Confluent Partner Program</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Future of Decision-Making: Collaborative Intelligence with Waterstream Cloud, Confluent Cloud, and ChatGPT</title>
		<link>https://waterstream.io/2023/07/10/the-future-of-decision-making-collaborative-intelligence-with-waterstream-cloud-confluent-cloud-and-chatgpt/</link>
		
		<dc:creator><![CDATA[Waterstream]]></dc:creator>
		<pubDate>Mon, 10 Jul 2023 16:17:21 +0000</pubDate>
				<category><![CDATA[Confluent]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[MQTT]]></category>
		<category><![CDATA[Use cases]]></category>
		<guid isPermaLink="false">https://waterstream.io/?p=5456</guid>

					<description><![CDATA[<p>  This blog post is co-authored with Donnie Kerr, Principal Technology Architect at Centric Consulting   Introduction In today&#8217;s fast-paced technological landscape, organizations are continuously exploring opportunities to harness the power of real-time data streaming and artificial intelligence (AI) to drive productivity and streamline processes. This blog post presents an innovative concept called the Collaborative [&#8230;]</p>
<p>The post <a href="https://waterstream.io/2023/07/10/the-future-of-decision-making-collaborative-intelligence-with-waterstream-cloud-confluent-cloud-and-chatgpt/">The Future of Decision-Making: Collaborative Intelligence with Waterstream Cloud, Confluent Cloud, and ChatGPT</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></description>
			<style id="elementor-post-5456">.elementor-5456 .elementor-element.elementor-element-46fd974{--display:flex;--flex-direction:column;--container-widget-width:100%;--container-widget-height:initial;--container-widget-flex-grow:0;--container-widget-align-self:initial;--flex-wrap-mobile:wrap;}.elementor-widget-image .widget-image-caption{color:var( --e-global-color-text );font-family:var( --e-global-typography-text-font-family ), Sans-serif;font-weight:var( --e-global-typography-text-font-weight );}.elementor-widget-text-editor{font-family:var( --e-global-typography-text-font-family ), Sans-serif;font-weight:var( --e-global-typography-text-font-weight );color:var( --e-global-color-text );}.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:var( --e-global-color-primary );}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap, .elementor-widget-text-editor.elementor-drop-cap-view-default .elementor-drop-cap{color:var( --e-global-color-primary );border-color:var( --e-global-color-primary );}.elementor-5456 .elementor-element.elementor-element-847cc27 > .elementor-widget-container{padding:1px 1px 020px 1px;}.elementor-5456 .elementor-element.elementor-element-847cc27{column-gap:0px;}.elementor-5456 .elementor-element.elementor-element-06575d6 .elementor-wrapper{--video-aspect-ratio:1.77777;}.elementor-5456 .elementor-element.elementor-element-3181115 > .elementor-widget-container{padding:0px 0px 30px 0px;}.elementor-5456 .elementor-element.elementor-element-ffc679f > .elementor-widget-container{padding:20px 0px 0px 0px;}.elementor-5456 .elementor-element.elementor-element-ffc679f{text-align:justify;}.elementor-5456 .elementor-element.elementor-element-36641ef > .elementor-widget-container{padding:0px 0px 0px 0px;}.elementor-5456 .elementor-element.elementor-element-36641ef{text-align:justify;}.elementor-5456 .elementor-element.elementor-element-0ca070e > .elementor-widget-container{padding:10px 0px 0px 0px;}.elementor-5456 .elementor-element.elementor-element-0ca070e{text-align:justify;}.elementor-5456 .elementor-element.elementor-element-230722b{text-align:justify;}.elementor-5456 .elementor-element.elementor-element-d83c2b3 .elementor-wrapper{--video-aspect-ratio:1.77777;}</style>							<content:encoded><![CDATA[		<div data-elementor-type="wp-post" data-elementor-id="5456" class="elementor elementor-5456" data-elementor-post-type="post">
				<div class="elementor-element elementor-element-46fd974 e-flex e-con-boxed e-con e-parent" data-id="46fd974" data-element_type="container" data-e-type="container">
					<div class="e-con-inner">
				<div class="elementor-element elementor-element-c4d351a elementor-widget elementor-widget-image" data-id="c4d351a" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
															<img decoding="async" width="900" height="600" src="https://waterstream.io/wp-content/uploads/2023/07/The-Future-of-Decision-Making.jpg" class="attachment-large size-large wp-image-5980" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/The-Future-of-Decision-Making.jpg 900w, https://waterstream.io/wp-content/uploads/2023/07/The-Future-of-Decision-Making-300x200.jpg 300w, https://waterstream.io/wp-content/uploads/2023/07/The-Future-of-Decision-Making-768x512.jpg 768w" sizes="(max-width: 900px) 100vw, 900px" />															</div>
				</div>
					</div>
				</div>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-725cd0c elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="725cd0c" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-91172f3" data-id="91172f3" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-847cc27 elementor-widget elementor-widget-text-editor" data-id="847cc27" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;"> </p><h1 dir="ltr" style="text-align: justify; line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;"><span style="color: #000000; font-variant-caps: normal; white-space: pre-wrap; font-weight: normal;"><span style="font-family: Muli, sans-serif; font-size: medium;"><i>This blog post is co-authored with Donnie Kerr, Principal Technology Architect at Centric Consulting </i></span></span></h1><p><span style="color: #000000; font-family: Arial;"><span style="caret-color: #000000; font-size: 26.666666px; white-space: pre-wrap;"> </span></span></p><p><span style="color: #000000; font-family: Arial;"><span style="caret-color: #000000; font-size: 26.666666px; white-space: pre-wrap;">Introduction</span></span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In today&#8217;s fast-paced technological landscape, organizations are continuously exploring opportunities to harness the power of real-time data streaming and artificial intelligence (AI) to drive productivity and streamline processes. This blog post presents an innovative concept called the Collaborative Intelligence Platform (CIP) by our partner Centric Consulting, which revolutionizes decision-making by seamlessly integrating AI, private enterprise data, and human guidance. Central to CIP are cutting-edge technologies such as Waterstream Cloud, <a href="https://www.confluent.io/confluent-cloud/?utm_campaign=tm.pmm_cd.2023_partner_cwc_waterstream_generic&amp;utm_source=waterstream&amp;utm_medium=partnerref">Confluent Cloud</a>, and advanced language models (LLMs) like ChatGPT.</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></p><h2><span style="font-size: 20pt; font-family: Arial; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From Waterstream to Waterstream Cloud</span></h2><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream was created as a straightforward yet sophisticated solution to bridge the gap between the Internet of Things (IoT) world, which utilizes MQTT, and the Apache Kafka®</span><span style="font-family: Muli, sans-serif; font-size: 12pt; font-style: normal; font-variant-caps: normal; white-space: pre-wrap;"> data streaming ecosystem. Despite its conceptual simplicity, Waterstream offers a multitude of configurations that users can customize. These configurations include mapping MQTT topics to Kafka topics and establishing authorization and authentication settings. Even seasoned DevOps professionals may require time to grasp and manage the details.</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To address this, Waterstream has evolved from being solely available as a Docker image with accompanying scripts to becoming a software-as-a-service platform called Waterstream Cloud. Currently, Waterstream Cloud is accessible only by invitation (contact us if you want to be included in the waiting list), but it will soon transition to the General Availability status. Through a user-friendly interface, users can effortlessly create one or multiple Waterstream clusters in a specific region on AWS and connect them with existing Confluent or open source Kafka clusters with just a few clicks.</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once the connection to Kafka is set up, users can focus more on defining the business case, determining how Waterstream maps MQTT topics to Kafka topics, and establishing the appropriate authorization and authentication rules. Waterstream Cloud further enhances the user experience by integrating a straightforward UI, enabling users to send and receive messages from the Waterstream Cluster or any MQTT broker, provided that the cluster has WebSocket functionality enabled. Additionally, if the Kafka cluster is temporarily unavailable or inaccessible, users can still test Waterstream by creating a single-node sidecar Kafka cluster that allows Waterstream to operate. However, it&#8217;s important to note that the sidecar Kafka cluster is not designed for production use and cannot be accessed by external producers or consumers.</span></p><h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;"><span style="font-size: 20pt; font-family: Arial; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Waterstream Cloud and Confluent Cloud, a perfect match</span></h1><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;">Waterstream Cloud simplifies the creation of a Waterstream MQTT broker, and it pairs perfectly with Confluent Cloud. Confluent Cloud is a fully managed, cloud-native, and complete dataevent streaming platform offered by Confluent, the company founded by the original creators of Apache Kafka. It provides a scalable and reliable infrastructure for developing real-time data streaming applications and implementing event-driven architectures in the cloud.</p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By leveraging Confluent Cloud, users can harness the power of Kafka without the burden of managing the underlying infrastructure. </span><span style="font-size: 12pt; font-style: normal; font-variant-caps: normal; font-family: Muli, sans-serif; font-variant-ligatures: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; white-space: pre-wrap;">To facilitate the seamless integration of Waterstream Cloud and Confluent Cloud, a dedicated configuration form is available. This form includes the necessary Kafka parameters from Confluent Cloud, making it effortless for users to connect the two platforms. Additionally, Waterstream offers the option to validate the payload of incoming MQTT messages, formatted as JSON, using the fully managed Schema Registry included in Confluent Cloud. </span><span style="font-size: 12pt; font-style: normal; font-variant-caps: normal; color: var( --e-global-color-text ); font-family: Muli, sans-serif; white-space: pre-wrap;">It is worth mentioning that the integration of Waterstream with the Confluent Schema Registry is a powerful combination that brings unified validation for MQTT messages and Kafka records. This integration simplifies development because it ensures consistent validation rules, and enhances interoperability between MQTT-based devices and Kafka-based services.</span></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-026710b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="026710b" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-444df71" data-id="444df71" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-beb1d57 elementor-widget elementor-widget-image" data-id="beb1d57" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
												<figure class="wp-caption">
										<img decoding="async" width="1024" height="518" src="https://waterstream.io/wp-content/uploads/2023/07/ConfluentCloudOptions-1024x518.png" class="attachment-large size-large wp-image-5458" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/ConfluentCloudOptions-1024x518.png 1024w, https://waterstream.io/wp-content/uploads/2023/07/ConfluentCloudOptions-300x152.png 300w, https://waterstream.io/wp-content/uploads/2023/07/ConfluentCloudOptions-768x389.png 768w, https://waterstream.io/wp-content/uploads/2023/07/ConfluentCloudOptions-1536x778.png 1536w, https://waterstream.io/wp-content/uploads/2023/07/ConfluentCloudOptions-2048x1037.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />											<figcaption class="widget-image-caption wp-caption-text">Confluent Cloud options in Waterstream Cloud</figcaption>
										</figure>
									</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-d413d70 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="d413d70" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-94bfbb5" data-id="94bfbb5" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-3d96f8c elementor-widget elementor-widget-text-editor" data-id="3d96f8c" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p><span id="docs-internal-guid-d057fa4e-7fff-6f80-030a-7b3a9deb8363" style="font-style: normal; font-variant-caps: normal;"><span style="font-size: 12pt; font-family: Muli, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; white-space: pre-wrap;">Here is a short </span><a href="https://www.youtube.com/watch?v=yVmVUKzdlGk"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: rgb(17, 85, 204); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; white-space: pre-wrap;">video</span></a><span style="font-size: 12pt; font-family: Muli, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; white-space: pre-wrap;"> showcasing the quick and straightforward process of starting a Waterstream Broker with Waterstream Cloud. It highlights the seamless connection to Confluent Cloud, enabling users to send MQTT messages directly from their browser, which are then written into Kafka.</span></span><br></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-2a542d0 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="2a542d0" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d97b965" data-id="d97b965" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-06575d6 elementor-widget elementor-widget-video" data-id="06575d6" data-element_type="widget" data-e-type="widget" data-settings="{&quot;youtube_url&quot;:&quot;https:\/\/youtu.be\/yVmVUKzdlGk&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default">
				<div class="elementor-widget-container">
							<div class="elementor-wrapper elementor-open-inline">
			<div class="elementor-video"></div>		</div>
						</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-8c5173c elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="8c5173c" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-091a268" data-id="091a268" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-3181115 elementor-widget elementor-widget-text-editor" data-id="3181115" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;"><span style="font-size: 20pt; font-family: Arial; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Collaborative Intelligence Platform</span></h1><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At this pivotal moment, generative AI technologies offer the potential for remarkable productivity gains that organizations desire. However, only 27% of companies currently use AI tools, despite the 90% higher productivity </span><a style="text-decoration: none;" href="https://slack.com/blog/news/state-of-work-2023"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">reported</span></a><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> by AI users. To foster the success of generative AI in the workplace, it must operate in a trusted environment where users have control of their own data and unwavering confidence in the relevance of its outputs.</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Donnie Kerr, Principal Architect at Centric Consulting, has been exploring the capabilities of this approach and introduced the paradigm of Collaborative Intelligence as a new methodology:</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 16px; font-style: normal; font-variant-caps: normal; text-align: start;"> </span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: italic; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Collaborative Intelligence leverages AI to mature a business process from little-to-no automation up to fully autonomous or somewhere in between. The end state of each process will vary depending on the goal. In some cases, a process can become fully autonomous, not requiring any human intervention. In other cases, a process may be very manual at first.”</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 16px; font-style: normal; font-variant-caps: normal; text-align: start;"> </span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Collaborative Intelligence Platform (CIP) seamlessly incorporates cutting-edge technologies essential for the comprehensive implementation of the CI methodology. The platform architecture harmoniously integrates prominent cloud-native technologies, as depicted below:</span></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-bb13813 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="bb13813" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-42f335d" data-id="42f335d" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-e78864d elementor-widget elementor-widget-image" data-id="e78864d" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
												<figure class="wp-caption">
										<img loading="lazy" decoding="async" width="1024" height="743" src="https://waterstream.io/wp-content/uploads/2023/07/CollaborativeIntelligencePlatform-1024x743.png" class="attachment-large size-large wp-image-5459" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/CollaborativeIntelligencePlatform-1024x743.png 1024w, https://waterstream.io/wp-content/uploads/2023/07/CollaborativeIntelligencePlatform-300x218.png 300w, https://waterstream.io/wp-content/uploads/2023/07/CollaborativeIntelligencePlatform-768x557.png 768w, https://waterstream.io/wp-content/uploads/2023/07/CollaborativeIntelligencePlatform-1536x1114.png 1536w, https://waterstream.io/wp-content/uploads/2023/07/CollaborativeIntelligencePlatform.png 1908w" sizes="(max-width: 1024px) 100vw, 1024px" />											<figcaption class="widget-image-caption wp-caption-text">The Collaborative Intelligence Platform, courtesy of Centric Consulting</figcaption>
										</figure>
									</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-350d84b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="350d84b" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c07cca1" data-id="c07cca1" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-ffc679f elementor-widget elementor-widget-text-editor" data-id="ffc679f" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p><span style="font-weight: 400;">To illustrate a practical application of CIP, let&#8217;s consider an example. Imagine an operator working in a logistics center who receives a notification through a conversational user interface powered by ChatGPT. In real-world data, the shipment system detects a delay of 30 minutes. The notification informs the operator that a shipment they are responsible for will be late. The operator is then prompted to decide whether they will handle the delayed shipment or if it should be reassigned to the next shift. As time progresses, the AI learns from patterns and identifies that parcels originating from a specific location often encounter delays, leading to automatic reassignment to the subsequent shift.</span></p><p><span style="color: var( --e-global-color-text ); font-size: 16px;">The core essence of CIP lies in the ability to seamlessly combine an AI platform with private enterprise data in real time, along with user decisions. In this scenario, Confluent Cloud plays a vital role by facilitating the merging of real-world events, user actions, and AI-generated outcomes. This integration enables a smooth and dynamic exchange of information, leading to improved collaboration and decision-making within the CIP framework letting the human control the full process.</span></p><p><span style="font-size: 16px; color: var( --e-global-color-text );">Let&#8217;s delve into the technical workflow of the Collaborative Intelligence Platform (CIP) by examining a simple Proof of Concept (PoC) that highlights its essential components. This PoC leverages Waterstream Cloud and Confluent Cloud. Both technologies utilize the concept of topics, MQTT or Kafka.  MQTT topics and Kafka topics share the same name to facilitate seamless mapping.</span></p><p><span style="font-size: 16px; color: var( --e-global-color-text );">The process initiates with user interactions on the UI, where they make information requests using ChatGPT prompts. To streamline testing, the UI offers some examples to get the user started.</span></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-bd095cc elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="bd095cc" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-1730652" data-id="1730652" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-11871f8 elementor-widget elementor-widget-image" data-id="11871f8" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
												<figure class="wp-caption">
										<img loading="lazy" decoding="async" width="1024" height="490" src="https://waterstream.io/wp-content/uploads/2023/07/ChatGPTCC-1024x490.png" class="attachment-large size-large wp-image-5460" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/ChatGPTCC-1024x490.png 1024w, https://waterstream.io/wp-content/uploads/2023/07/ChatGPTCC-300x144.png 300w, https://waterstream.io/wp-content/uploads/2023/07/ChatGPTCC-768x367.png 768w, https://waterstream.io/wp-content/uploads/2023/07/ChatGPTCC-1536x735.png 1536w, https://waterstream.io/wp-content/uploads/2023/07/ChatGPTCC-2048x980.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />											<figcaption class="widget-image-caption wp-caption-text">The ChatGPT Prompt</figcaption>
										</figure>
									</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-dc7da21 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="dc7da21" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-91031e4" data-id="91031e4" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-36641ef elementor-widget elementor-widget-text-editor" data-id="36641ef" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p>Within the UI front end, an MQTT client communicates with Waterstream Cloud to create new chats, send the user&#8217;s prompt, and remember the conversation history. Anyone experimenting with Chat GPT APIs knows it doesn&#8217;t maintain context. Your application will need to manage that, and we think of this in terms of state.</p><p>Usually, the app will have to send the call with the initial prompt payload (plus some hidden prompt engineering) and get the response, then send the initial prompt, plus the response, plus the new prompt for ChatGPT. This means incrementally larger payloads and slower response times as you iterate. <span style="font-size: 16px; color: var( --e-global-color-text );">Not to mention, the history is trapped in that session or state. But, if we use Confluent Cloud and <a href="https://www.confluent.io/product/ksqldb/">ksqlDB</a>, for example, we can preserve both state and history.</span></p><p>In our case, ksqlDB is used to load the user&#8217;s chat history and the ongoing conversation flow within each chat. This also allows for opening previous chats and continuing the conversation in real time. You could add a tumbling window so that the stream presented by ksqlDB only has the most recent chats or for a specific retention window. You could also use ksqlDB to present a single materialized view of the user&#8217;s prompt template. For example, a control operator would ask different questions than a diagnostician or machine programmer. In any of these cases these same streams are persisted in Kafka for learning and training the back end models, the results of which which can then be merged back in to these contexts, and get better over time.</p><p><span style="font-weight: 400;">Waterstream Cloud enables the direct saving of new Chats into a “Chats” topic for each user and user prompts as MQTT messages into a Confluent Cloud Kafka topic named &#8220;chatMessages”, eliminating the need for intermediaries. From there, the platform leverages ksqlDB stream processing to orchestrate the flow of events.</span></p><p><span style="font-size: 16px; color: var( --e-global-color-text );">ksqlDB and MQTT topics are a match made in heaven.  MQTT topic structure is ideal for routing flows of events from devices, UI, and the AI assistant into various ksqlDB stream pathways depending on the topic and/or value of the message. For example, the ChatGPT app supports Azure OpenAI GPT-3.5-turbo or OpenAI GPT-4.  A single prompt gets routed from the “chatMessages/{chatId}” MQTT topic to either Azure OpenAI or OpenAI Azure Function Sink connectors.  “ChatMessages_stream_aoai” and “ChatMessages_stream_oai” are the input streams that trigger the api calls. </span></p><p><span style="font-size: 16px; color: var( --e-global-color-text );">Similarly, to make the prompt aware of the enterprise context, chatMessages can be joined with data obtained from private data streams using ksqlDB. The MQTT topic effectively steers the prompt to be appended with the correct private data source(s). These private streams can be generated using connectors available in Confluent Cloud.  The augmented prompt topic is then forwarded to ChatGPT for processing.</span></p><p> </p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-383ffb5 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="383ffb5" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c50b94f" data-id="c50b94f" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-8811b5b elementor-widget elementor-widget-image" data-id="8811b5b" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
												<figure class="wp-caption">
										<img loading="lazy" decoding="async" width="1024" height="512" src="https://waterstream.io/wp-content/uploads/2023/07/TopicLineage-1024x512.png" class="attachment-large size-large wp-image-5461" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/TopicLineage-1024x512.png 1024w, https://waterstream.io/wp-content/uploads/2023/07/TopicLineage-300x150.png 300w, https://waterstream.io/wp-content/uploads/2023/07/TopicLineage-768x384.png 768w, https://waterstream.io/wp-content/uploads/2023/07/TopicLineage-1536x767.png 1536w, https://waterstream.io/wp-content/uploads/2023/07/TopicLineage-2048x1023.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />											<figcaption class="widget-image-caption wp-caption-text">The PoC within Confluent Cloud’s Stream Lineage</figcaption>
										</figure>
									</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-97d5346 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="97d5346" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-daddf17" data-id="daddf17" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-0ca070e elementor-widget elementor-widget-text-editor" data-id="0ca070e" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p><span style="font-weight: 400;">ChatGPT generates responses in the form of chunks, which are groups of one or more words. These chunks are stored in a stream called &#8220;chatgptResponseStream&#8221; as they are generated.</span></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-ac5147e elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="ac5147e" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d917146" data-id="d917146" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-7792fb6 elementor-widget elementor-widget-image" data-id="7792fb6" data-element_type="widget" data-e-type="widget" data-widget_type="image.default">
				<div class="elementor-widget-container">
												<figure class="wp-caption">
										<img loading="lazy" decoding="async" width="1024" height="507" src="https://waterstream.io/wp-content/uploads/2023/07/chatGPTResponseStream-1024x507.png" class="attachment-large size-large wp-image-5462" alt="" srcset="https://waterstream.io/wp-content/uploads/2023/07/chatGPTResponseStream-1024x507.png 1024w, https://waterstream.io/wp-content/uploads/2023/07/chatGPTResponseStream-300x149.png 300w, https://waterstream.io/wp-content/uploads/2023/07/chatGPTResponseStream-768x380.png 768w, https://waterstream.io/wp-content/uploads/2023/07/chatGPTResponseStream-1536x760.png 1536w, https://waterstream.io/wp-content/uploads/2023/07/chatGPTResponseStream-2048x1014.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" />											<figcaption class="widget-image-caption wp-caption-text">The chatGptResponseStream in Confluent Cloud</figcaption>
										</figure>
									</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-c63a4b9 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="c63a4b9" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-a0d2643" data-id="a0d2643" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-230722b elementor-widget elementor-widget-text-editor" data-id="230722b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p><span id="docs-internal-guid-ae15833d-7fff-3b1d-7fac-33031a9b9f86" style="font-style: normal; font-variant-caps: normal;"><span style="font-size: 12pt; font-family: Muli, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; white-space: pre-wrap;">At this stage, Waterstream Cloud plays a role once again by transmitting the answer chunks from the Kafka topic as messages to a specific MQTT topic. The frontend client subscribes to the MQTT topic &#8220;chatgptResponseStream.&#8221; As ChatGPT generates responses, the UI dynamically updates to display the words in real time. This allows users to observe the gradual construction of the response, with new words appearing on the screen, similar to the familiar OpenAI application. It is worth mentioning that other topics are used in this PoC to save chat history and user information, but aren’t&nbsp; discussed in detail here.</span></span><br></p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-b79e15a elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="b79e15a" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f307586" data-id="f307586" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-d83c2b3 elementor-widget elementor-widget-video" data-id="d83c2b3" data-element_type="widget" data-e-type="widget" data-settings="{&quot;youtube_url&quot;:&quot;https:\/\/youtu.be\/zV1bnn6v2mM&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default">
				<div class="elementor-widget-container">
							<div class="elementor-wrapper elementor-open-inline">
			<div class="elementor-video"></div>		</div>
						</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				<section class="elementor-section elementor-top-section elementor-element elementor-element-292603c elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="292603c" data-element_type="section" data-e-type="section">
						<div class="elementor-container elementor-column-gap-default">
					<div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e72a751" data-id="e72a751" data-element_type="column" data-e-type="column">
			<div class="elementor-widget-wrap elementor-element-populated">
						<div class="elementor-element elementor-element-6c4165b elementor-widget elementor-widget-text-editor" data-id="6c4165b" data-element_type="widget" data-e-type="widget" data-widget_type="text-editor.default">
				<div class="elementor-widget-container">
									<p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the advantages offered by Waterstream is the ability to write directly from the UI into Kafka and vice versa. ChatGPT chunks and other events can be easily communicated to the UI as notifications. Utilizing an MQTT broker instead of plain WebSockets enables the organization of data flow into multiple streams, simplifying the creation of a reactive UI. Moreover, Waterstream Cloud provides authentication and authorization capabilities, which are crucial for enterprise applications. User information can also be utilized to organize MQTT topics, leading to a well-structured data system.</span></p><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For further details about the Collaborative Intelligence Platform (CIP), please refer to the “</span><a style="text-decoration: none;" href="https://centricconsulting.com/resources/collaborative-intelligence-an-ai-powered-system-that-puts-humans-in-control/"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Collaborative Intelligence: An AI-Powered System That Puts Humans in Control</span></a><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">” white paper available on the Centric Consulting website. </span><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Another interesting reading on these themes is a blog post titled &#8220;</span><a style="text-decoration: none;" href="https://www.confluent.io/en-gb/blog/chatgpt-and-streaming-data-for-real-time-generative-ai/"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #1155cc; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">GPT-4 + Streaming Data = Real-Time Generative AI</span></a><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">&#8221; by Michael Drogalis, Principal Technologist at Confluent. </span></p><h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;"><span style="font-size: 20pt; font-family: Arial; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></h1><p dir="ltr" style="line-height: 1.38; text-align: justify; margin-top: 0pt; margin-bottom: 0pt;"><span style="font-size: 12pt; font-family: Muli, sans-serif; color: #374151; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-alternates: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In conclusion, in this blog post we discussed the journey from Waterstream to Waterstream Cloud, the integration with Confluent Cloud, and the idea and implementation of a Collaborative Intelligence Platform (CIP). Together, these technologies form a unified system that combines user prompts, actions, and private enterprise data to form Collaborative Intelligence. Although CIP is still in its early stages, it holds great potential by utilizing a streaming data approach to merge real-world events, user knowledge, control, and AI assistance. This integrated approach shows promise for enhancing collaboration and decision-making processes with the human still in the center.</span></p><p> </p>								</div>
				</div>
					</div>
		</div>
					</div>
		</section>
				</div>
		<p>The post <a href="https://waterstream.io/2023/07/10/the-future-of-decision-making-collaborative-intelligence-with-waterstream-cloud-confluent-cloud-and-chatgpt/">The Future of Decision-Making: Collaborative Intelligence with Waterstream Cloud, Confluent Cloud, and ChatGPT</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Kafka Native MQTT at Scale with Confluent Cloud and Waterstream</title>
		<link>https://waterstream.io/2020/07/21/apache-kafka-native-mqtt-at-scale-with-confluent-cloud-and-waterstream/</link>
		
		<dc:creator><![CDATA[Admin]]></dc:creator>
		<pubDate>Tue, 21 Jul 2020 02:09:00 +0000</pubDate>
				<category><![CDATA[Confluent]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[MQTT]]></category>
		<category><![CDATA[fleet management iot]]></category>
		<category><![CDATA[kafka iot]]></category>
		<category><![CDATA[kafka mqtt]]></category>
		<category><![CDATA[mqtt broker confluent]]></category>
		<guid isPermaLink="false">https://waterstream.io/?p=3120</guid>

					<description><![CDATA[<p>With billions of Internet of Things (IoT) devices, achieving real-time interoperability has become a major challenge. Together, Confluent, Waterstream, and MQTT are accelerating Industry 4.0 with new Industrial IoT (IIoT) and consumer IoT (CIoT) use cases, revolutionizing the way companies manufacture and use machines, devices, and other connected components. This blog post takes a look [&#8230;]</p>
<p>The post <a href="https://waterstream.io/2020/07/21/apache-kafka-native-mqtt-at-scale-with-confluent-cloud-and-waterstream/">Apache Kafka Native MQTT at Scale with Confluent Cloud and Waterstream</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">With billions of Internet of Things (IoT) devices, achieving real-time interoperability has become a major challenge. Together, Confluent, Waterstream, and MQTT are accelerating Industry 4.0 with new Industrial IoT (IIoT) and consumer IoT (CIoT) use cases, revolutionizing the way companies manufacture and use machines, devices, and other connected components.</p>



<p class="wp-block-paragraph">This blog post takes a look at IoT, its relation to the MQTT standard, and options for integrating MQTT with Apache Kafka<sup>®</sup>&nbsp;and Confluent Cloud, including a new Kafka-native implementation of an MQTT broker.</p>



<h2 class="wp-block-heading">Use Cases for IoT and event streaming with Apache Kafka</h2>



<p class="wp-block-paragraph">Let’s first overview a variety of CIoT and IIoT/Industry 4.0 examples to see how it works:</p>



<ul class="wp-block-list"><li>Lyft: Correlation of events from drivers, guests, and backend systems like CRM, weather service, traffic information, and the payment provider to calculate routes, estimated time of arrival, estimated cost, and more</li><li>Audi: Connected car infrastructure for real-time ingestion, processing, and analysis of events for after-sales and other customer-related use cases</li><li>Bosch Power Tools: Real-time alerting and new real-time dashboards that merge and present data from manufacturing plants, dealers, tool owners, and other sources across the company.</li><li>Deutsche Bahn: Connectivity to different technologies, such as messaging systems, databases, and files to correlate events for calculation and display of real-time train information, delays, and cancellations on mobile apps, train station displays, and other interfaces</li><li>E.ON: IoT cloud platform to integrate and correlate data from internal applications, smart homes, smart grids, and partner systems, providing a real-time infrastructure for energy distribution</li><li>Severstal: Real-time analytics at the edge for quality improvements and predictive maintenance in factory production lines.</li></ul>



<p class="wp-block-paragraph">All of these share the following requirements:</p>



<ul class="wp-block-list"><li>Real-time data integration and data processing</li><li>Mission-critical, 24/7 deployments without downtime</li><li>Large-scale processing of events from hundreds of thousands or even millions of users and machine and device interfaces</li></ul>



<p class="wp-block-paragraph">These examples use different architectures and infrastructures for edge computing, hybrid deployments, and cloud computing. Kafka and its ecosystem are the heart of all these deployments. We see this trend across many industries, including automotive, manufacturing, energy, oil and gas, logistics, and more.</p>



<p class="wp-block-paragraph">To understand the deployment architecture for Kafka and IoT in more detail, please check out Architecture Patterns for Distributed, Hybrid, Edge, and Global Apache Kafka Deployments and Apache Kafka is the New Black at the Edge in Industrial IoT, Logistics, and Retailing. For a general overview of use cases and architectures for event streaming in IoT use cases, please check out the following presentation: Introduction to IoT with Apache Kafka and Event Streaming. Standards and Protocols for IoT Scenarios.</p>



<h2 class="wp-block-heading">Standards and protocols for IoT scenarios</h2>



<p class="wp-block-paragraph">Several standards exist to support IoT projects:</p>



<ul class="wp-block-list"><li><strong>MQTT</strong>: The most relevant standard supporting plenty of use cases in CIoT and IIoT. Built for unreliable networks and millions of devices.</li><li><strong>OPC-UA</strong>: For Industrial IoT only. Most new machines, devices, and middleware in IIoT environments support this standard.</li><li><strong>HTTP</strong>: Synchronous communication with REST web services. Well understood, simple, and supported by almost all frameworks and products. Not an IoT standard, but often used in IoT projects if limited latency and scalability requirements are given.</li><li><strong>WebSocket</strong>: A modern, full-duplex communication channel over a single TCP connection. Used more and more in CIoT for real-time and scalable applications to replace HTTP communications, which do not scale and perform well in large deployments.</li><li><strong>Syslog, SNMP, and more</strong>: While standards like MQTT and OPC-UA are built to solve specific IoT challenges, most infrastructures complement these standards with other established technologies for logging, monitoring, and troubleshooting.</li></ul>



<p class="wp-block-paragraph">MQTT is one of the most important standards for IoT projects, both in IIoT and CIoT.</p>



<h2 class="wp-block-heading">MQTT and Apache Kafka</h2>



<p class="wp-block-paragraph">Most IoT projects combine MQTT and Kafka for good reasons. The high-level architecture typically looks something like this:</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/mqtt-apache-kafka-architecture.png" alt="" class="wp-image-3124"/></figure>



<p class="wp-block-paragraph">From a technical perspective, MQTT is just another producer and/or consumer. Kafka decouples different clients, applications, and backends from each other.</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/kafka-mqtt-system-schema-1024x228.png" alt="" class="wp-image-3125"/></figure>



<p class="wp-block-paragraph">The pros and cons of MQTT (as IoT standard technology) and the pros and cons of Kafka (as the de facto standard for an event streaming platform) map to one another in a very complementary way, making it clear why both are combined so often.</p>



<h2 class="wp-block-heading">Pros and cons of MQTT</h2>



<p class="wp-block-paragraph"><strong>MQTT pros:</strong></p>



<ul class="wp-block-list"><li>Lightweight</li><li>Supports all programming languages</li><li>Built for poor connectivity/high-latency scenarios (e.g., mobile networks)</li><li>High scalability and availability*</li><li>ISO standard</li><li>Most popular IoT protocol</li></ul>



<p class="wp-block-paragraph"><strong>MQTT cons:</strong></p>



<ul class="wp-block-list"><li>Only pub/sub, not stream processing</li><li>Asynchronous processing (clients can be offline for a long time)</li><li>No reprocessing of events</li></ul>



<h2 class="wp-block-heading">Pros and cons of Apache Kafka (from an IoT perspective)</h2>



<p class="wp-block-paragraph"><strong>Kafka pros</strong>:</p>



<ul class="wp-block-list"><li>Stream processing, not just pub/sub</li><li>High throughput</li><li>Large scale</li><li>High availability</li><li>Long-term storage and buffering</li><li>Reprocessing of events</li><li>Strong integration of other enterprise technology</li></ul>



<p class="wp-block-paragraph"><strong>Kafka cons</strong>:</p>



<ul class="wp-block-list"><li>Not built for tens of thousands connections</li><li>Requires stable network and good infrastructure</li><li>No IoT-specific features like Keep Alive, Last Will, or Testament</li></ul>



<p class="wp-block-paragraph">As you can see, MQTT and Kafka work together perfectly. You can learn more about this combination in Best Practices for Streaming IoT Data with MQTT and Apache Kafka.</p>



<h2 class="wp-block-heading">Predictive maintenance in real-time for 100,000 connected cars with MQTT and Kafka</h2>



<p class="wp-block-paragraph">MQTT and event streaming combined allow real-time integration, processing, and analytics of IoT data. A use case for MQTT and Kafka can look similar to this:</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/kafka-mqtt-broker-predictive-maintenance-schema.png" alt="" class="wp-image-3127"/></figure>



<p class="wp-block-paragraph">In this example, we see a real-time integration pipeline to stream data from millions of connected cars via MQTT to the event streaming platform for streaming ETL, machine learning, digital twin, big data analytics, and other use cases.</p>



<h2 class="wp-block-heading">MQTT integration options for Apache Kafka, Confluent Platform, and Confluent Cloud</h2>



<p class="wp-block-paragraph">So what are the different options for MQTT implementation and integration with Apache Kafka?</p>



<p class="wp-block-paragraph">Plenty of integration options exist for MQTT to Kafka integration:</p>



<ul class="wp-block-list"><li><strong>Kafka Connect + Confluent MQTT source and sink connectors + MQTT broker:</strong>&nbsp;A common option. Leverage the benefits of Kafka Connect for bidirectional communication with any standard compliant MQTT broker, including the reference implementation Mosquitto (for Hello World and non-critical deployments) or a scalable, battle-tested solution like Waterstream</li><li><strong>Confluent MQTT Proxy:</strong>&nbsp;A very lightweight option allowing direct integration between MQTT devices and a Kafka cluster. No need for an MQTT broker at all. Drawbacks are the limited feature set and (at the time of this writing) no bidirectional communication but just ingestion from MQTT to Kafka.</li><li><strong>Vendor-specific integration:</strong>&nbsp;Some vendors provide their own integrations between their MQTT broker and Kafka. For instance, Waterstream has its own Kafka plugin for bidirectional communication.</li><li><strong>Kafka-native implementation of MQTT:</strong>&nbsp;This option does not require integration because the MQTT broker is running as a Kafka application connecting to Kafka via native Kafka consumers and producers.</li></ul>



<p class="wp-block-paragraph">The trade-offs between using Kafka Connect and MQTT Broker, Confluent MQTT Proxy, or the Confluent REST Proxy are discussed in more detail in this presentation:&nbsp;<a href="https://www.confluent.io/kafka-summit-sf18/processing-iot-data-from-end-to-end/" target="_blank" rel="noreferrer noopener">Processing IoT Data from End to End with MQTT and Apache Kafka</a>.</p>



<h2 class="wp-block-heading">Finding the right option for your problem and use case</h2>



<p class="wp-block-paragraph">When evaluating your MQTT solution in conjunction with Kafka, there are many factors to consider as many offerings have limitations. For instance, cloud offerings from the major cloud providers typically are not fully compliant MQTT brokers, have limited feature sets, and have limited scalability and performance characteristics.</p>



<p class="wp-block-paragraph">Here are a few other aspects to check for:</p>



<ul class="wp-block-list"><li>MQTT compliance</li><li>Support of MQTT version 3.x or 5.x (the two main versions used in the industry)</li><li>Full implementation or a limited feature set that misses any of the following: Quality of Service (QoS), retained messages, Last Will and Testament, Persistent Sessions, Keep Alive, Client Take-Over, etc.</li><li>Uni or bidirectional communication between MQTT and Kafka</li><li>Security features like authentication, authorization, and encryption</li><li>Kafka integration (connector, plugin, and Kafka native)</li></ul>



<p class="wp-block-paragraph">For large-scale use cases, a dedicated, full MQTT broker implementation is the best option. The key question to ask yourself is whether you want to deploy a separate MQTT cluster or use a Kafka-native implementation.</p>



<p class="wp-block-paragraph">Streaming Machine Learning at Scale from 100,000 IoT Devices with Waterstream, Confluent, and TensorFlow&nbsp;is an example of a large-scale deployment with separated clusters for MQTT and Kafka. This architecture is well understood and works well in the real world.</p>



<p class="wp-block-paragraph">Having said this, let’s now focus on a new option to evaluate: an implementation of MQTT on top of Kafka. This adds one more option to compare to find the right architecture for your use case. The key advantage of this option is that you have to operate, maintain, and monitor just one distributed infrastructure for your mission-critical IoT projects because the MQTT broker runs Kafka natively as a Kafka Streams application, with all the benefits of Kafka under the hood (like high availability, high throughput, low latency, etc).</p>



<h2 class="wp-block-heading">Introducing Waterstream: A Kafka-native MQTT broker</h2>



<p class="wp-block-paragraph">Now you have a new option:&nbsp;Waterstream, which is Confluent verified and can turn your Kafka cluster into a full-featured MQTT broker. It works as a thin, bidirectional layer between Kafka and IoT devices. It doesn’t have intermediate persistence, and messages from MQTT clients are immediately written to Kafka and vice versa. As soon as the message is retrieved from Kafka, it’s sent to MQTT clients. All the necessary MQTT state (i.e., subscriptions, at-least-once and exactly-once QoS message status, and retained messages) is also stored in Kafka—no need for additional storage.</p>



<p class="wp-block-paragraph">Besides required MQTT functionality, Waterstream offers optional features (such as WebSockets) and features that are beyond the scope of MQTT specification (such as X.509 authentication and flexible authorization rules based on X.509 identity).</p>



<p class="wp-block-paragraph">Waterstream scales out linearly. For most operations, its nodes don’t depend on each other, so you can add more machines if you have more clients to handle. With 5–10 machines, it can handle hundreds of thousands of clients (see&nbsp;<a href="https://www.confluent.io/blog/iot-streaming-use-cases-with-kafka-mqtt-confluent-and-waterstream/#scalability-and-performance">scalability and performance</a>&nbsp;for more details).</p>



<p class="wp-block-paragraph">The reference architecture for Kafka-based IoT infrastructure with Waterstream might look like this:</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/mqtt-kafka-best-broker-1024x235.png" alt="" class="wp-image-3129"/></figure>



<p class="wp-block-paragraph">Devices send the telemetry to Waterstream through the load balancer. Waterstream nodes can be added or removed dynamically according to the current needs. Then this data is transferred to the Kafka topic and can be consumed by any Kafka tool—this may be custom code using a Kafka consumer, Kafka Connect, or a Kafka Streams application. ksqlDB can be leveraged to enrich the data as it arrives and feed some suggestions back to the devices. An example is alerting you of the nearest gas stations when you’re in a moving vehicle with low fuel. Producers (like some kind of management console) can send commands to the devices through Kafka and Waterstream.</p>



<p class="wp-block-paragraph">Of the pre-existing solutions, Confluent MQTT Proxy is closest to Waterstream. But unlike MQTT Proxy, Waterstream supports bidirectional communication and persists MQTT sessions. Therefore, if an MQTT client reconnects to Waterstream after a single node failure, then the load balancer can send the request to any other healthy node, and that node can pick an unfinished message transmission with QoS at least once or exactly-once.</p>



<p class="wp-block-paragraph">Compared to the MQTT Kafka connector, Waterstream has fewer moving parts—it doesn’t need a separately deployed, external MQTT broker because it is a full-featured broker itself (combined with Kafka, of course). Also, you wouldn’t need to deal with message loops or the unfortunate situation of a topic mapping a message from Kafka sent to an MQTT topic from which the connector reads again and sends back in the same Kafka topic.</p>



<p class="wp-block-paragraph">What are the best use cases for Waterstream? The obvious one is tight integration of your devices with Kafka. A thin layer between Kafka and MQTT Waterstream ensures the small latency on MQTT to a Kafka path. Another reason to use Waterstream is simpler deployment. As its nodes are stateless, you have less moving parts to configure and you can easily scale out or scale in as your traffic goes up or down. Nodes don’t depend on each other for message processing, so you can scale it horizontally to handle typical IoT use cases, which need hundreds of thousands of devices.</p>



<p class="wp-block-paragraph">Nothing comes without a cost, of course. All MQTT communication in Waterstream flows through Kafka, even if it’s supposed to go from one device to another, and there is no additional processing in centralized facilities like ksqlDB as expected. This means it takes longer for the message roundtrip. If you only need to send a message from one device to another, without integration with Kafka, Waterstream may have higher latency compared to traditional solutions.</p>



<h2 class="wp-block-heading">How Waterstream works</h2>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/waterstream-mqtt-broker-1024x401.png" alt="" class="wp-image-3130"/></figure>



<p class="wp-block-paragraph">According to configuration, Waterstream reads/writes MQTT messages from/to one or many Kafka topics using the MQTT topic name as the Kafka message key and the MQTT message payload as the Kafka message value. For each MQTT connection, an in-memory session is maintained, which keeps track of stateful aspects, such as at-least-once or exactly-once message transmission status, subscriptions, Last Will message, offsets of the last consumed messages, etc.</p>



<p class="wp-block-paragraph">If the session is persistent (the MQTT client flags the “Clean Session” as false), then the changes are persisted into the session topic in Kafka. This makes it possible to hand the session over to another Waterstream instance if the current one fails or if the MQTT client reconnects and the load balancer directs it to the other instance. When the client connects again, the Kafka Streams component picks the latest session state from this node or another one. Persisting the MQTT state in Kafka and reading it back in case of failure allows MQTT clients to achieve end-to-end, exactly-once integration if you use QoS 2 for&nbsp;<a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349264" target="_blank" rel="noreferrer noopener">publishing the messages and subscribing to the topics</a>, in which case, the reconnected, unfinished MQTT message will finish publishing and any missed messages from the subscriptions will be delivered in order.</p>



<p class="wp-block-paragraph">Since topic naming rules in Kafka are much stricter than in MQTT, not every MQTT topic may map to a Kafka topic. Therefore, an MQTT topic is represented by a message key in Kafka, and a single Kafka topic may contain messages for multiple MQTT topics. Waterstream supports&nbsp;<a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349843" target="_blank" rel="noreferrer noopener">topic mapping rules with MQTT-style wildcards</a>.</p>



<p class="wp-block-paragraph">Beyond that, a default Kafka topic must be configured in Waterstream, which contains messages from topics that don’t match any of the mapping rules. For example, with the rules<span style="color:#a33a00" class="has-inline-color">&nbsp;</span><span class="has-inline-color has-accent-color"><code>kafkaTopic1:cars/speed/+</code>&nbsp;and&nbsp;<code>kafkaTopic2:cars/#</code></span>, the MQTT topic&nbsp;<code><span class="has-inline-color has-accent-color">cars/speed/1</span></code>&nbsp;is mapped to&nbsp;<span class="has-inline-color has-accent-color"><code>kafkaTopic1</code>,&nbsp;<code>cars/location/1</code></span>, then to&nbsp;<span class="has-inline-color has-accent-color"><code>kafkaTopic2</code>,&nbsp;<code>cars/speed/1/2</code>,&nbsp;<code>trains/speed/1</code></span>, and the default topic.</p>



<p class="wp-block-paragraph">Whenever an MQTT client subscribes to some MQTT topic’s pattern, the Kafka consumer in the corresponding Waterstream node starts reading from the Kafka topics that are necessary to cover this pattern. Using the mapping from our previous example, if the client subscribes to&nbsp;<code><span class="has-inline-color has-accent-color">cars/speed/+</span></code>, then only&nbsp;<code><span class="has-inline-color has-accent-color">kafkaTopics1</span></code>&nbsp;is consumed. But if it subscribes to&nbsp;<code><span class="has-inline-color has-accent-color">cars/#</span></code>, then both&nbsp;<code><span class="has-inline-color has-accent-color">kafkaTopic1</span></code>&nbsp;and&nbsp;<code><span class="has-inline-color has-accent-color">kafkaTopic2</span></code>&nbsp;are consumed, and if&nbsp;<code><span class="has-inline-color has-accent-color">trains/#</span></code>, then only the default topic.</p>



<p class="wp-block-paragraph">Some caveats for an MQTT subscription still remain. If a Waterstream node consumes some Kafka topic, it has to read messages from all partitions, because from the MQTT topic wildcard (such as&nbsp;<span class="has-inline-color has-accent-color"><code>car/speed/+</code>&nbsp;or&nbsp;<code>trains/#</code></span>), you can’t always tell which exact Kafka keys to watch. MQTT topics are dynamic; they may come and go as the new messages arrive, so Waterstream should check every new message in the Kafka topics that match the MQTT subscription pattern. Hence, MQTT topics, which are meant to be consumed separately, should be mapped to the different Kafka topics. This is the key principle for configuring MQTT to a Kafka topic mapping in order to achieve the best possible read performance.</p>



<h2 class="wp-block-heading">Scalability and performance</h2>



<p class="wp-block-paragraph">We want to evaluate how many clients with a reasonable IoT message rate that Waterstream can support per node and how well it scales out to handle a larger number of clients. To simulate the load, we’ve developed a&nbsp;<a href="https://github.com/simplematter/simplematter-mqtt-load-simulator" target="_blank" rel="noreferrer noopener">simple but flexible tool</a>&nbsp;that is easy to run in different cloud environments (Kubernetes, GCP, etc.). It is configured to publish one message with a random text every 10 seconds. Message length is randomly picked between 400 and 600 bytes.</p>



<p class="wp-block-paragraph">Both Waterstream and the load simulator are deployed on GCP—Waterstream on&nbsp;<code><span class="has-inline-color has-accent-color">n1-standard-1</span></code>&nbsp;and&nbsp;<code><span class="has-inline-color has-accent-color">n1-standard-2</span></code>&nbsp;(2 CPU, 7.5 GB RAM) machines, and the load simulator on&nbsp;<code>n1-standard-1</code>&nbsp;(1 CPU, 3.75 GB RAM) machines. For more details, check out the following:</p>



<ul class="wp-block-list"><li><a href="https://cloud.google.com/compute/docs/machine-types#n1_standard_machine_types" target="_blank" rel="noreferrer noopener">Machine types</a></li><li><a href="https://github.com/simplematter/waterstream-gcp-terraform" target="_blank" rel="noreferrer noopener">Waterstream deployment Terraform scripts</a></li><li><a href="https://github.com/simplematter/simplematter-mqtt-load-simulator/tree/master/toolbelt/terraform_gcp" target="_blank" rel="noreferrer noopener">Load simulator, including Terraform scripts</a></li></ul>



<p class="wp-block-paragraph">For Kafka, we used&nbsp;Confluent Cloud, which is the easiest way to get it running and operated without any efforts on our side. It comes with tools to visualize and monitor the underlying topics and data flow from producers to consumers.</p>



<div class="wp-block-image"><figure class="aligncenter size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/mqtt-broker-data-flow.png" alt="" class="wp-image-3133"/></figure></div>



<p class="wp-block-paragraph">With each Waterstream deployment configuration, we gradually increase client count in the load simulator and watch the Grafana dashboard. Looking at the connections count and Kafka publish rate, we’ll decide if the current Waterstream configuration is sufficient for such a number of clients.</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/waterstream-mqtt-broker-config-1024x502.png" alt="" class="wp-image-3134"/></figure>



<figure class="wp-block-table"><table><tbody><tr><td class="has-text-align-center" data-align="center">Waterstream</td><td class="has-text-align-center" data-align="center">Load simulator</td><td class="has-text-align-center" data-align="center">Results</td></tr></tbody></table></figure>



<figure class="wp-block-table"><table><tbody><tr><td>Nodes Count</td><td>Heap</td><td>Nodes Count</td><td>Clients Per Node</td><td>Successful Connections</td><td>Kafka Publish Rate (msg/sec)</td></tr><tr><td>3</td><td>6000m</td><td>10</td><td>20K</td><td>190K</td><td>19K</td></tr><tr><td>5</td><td>6000m</td><td>10</td><td>20K</td><td>200K</td><td>20K</td></tr><tr><td>5</td><td>6000m</td><td>20</td><td>20K</td><td>400K</td><td>40K</td></tr><tr><td>5</td><td>6000m</td><td>30</td><td>20K</td><td>579K</td><td>50K</td></tr><tr><td>10</td><td>6000m</td><td>30</td><td>20K</td><td>600K</td><td>60K</td></tr><tr><td>10</td><td>6000m</td><td>40</td><td>20K</td><td>800K</td><td>80K</td></tr><tr><td>10</td><td>6000m</td><td>50</td><td>20K</td><td>993K</td><td>97K</td></tr><tr><td>12</td><td>6000m</td><td>50</td><td>20K</td><td>1,000K</td><td>100K</td></tr><tr><td>12</td><td>6000m</td><td>60</td><td>20K</td><td>1,180K</td><td>110K</td></tr></tbody></table></figure>



<p class="wp-block-paragraph">The results in green indicate cases where the system handled the load well. Those in yellow show where the system was on the verge of falling behind (up to 5% fewer connections than expected).</p>



<p class="wp-block-paragraph">After building a graph from this data, we see nearly linear scalability:</p>



<div class="wp-block-image"><figure class="aligncenter size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/mqtt-broker-scalability-graph.png" alt="" class="wp-image-3136"/></figure></div>



<p class="wp-block-paragraph">During the last evaluations, 12&nbsp;<code><span class="has-inline-color has-accent-color">n1-standard-2</span></code>&nbsp;nodes successfully handled 1.18 million of simulated devices. That means that one&nbsp;<span class="has-inline-color has-accent-color"><code>n1-standard-2</code>&nbsp;node</span> can handle about 98,000 MQTT connections.</p>



<h2 class="wp-block-heading">MQTT, Waterstream, and Kafka at the edge</h2>



<p class="wp-block-paragraph">While Waterstream and Kafka scale out well to support large loads in big datacenters, they can also scale down to tiny deployments at the edge. You can have a smaller deployment closer to the devices, such as in a factory building). This reduces the latency between the devices and Waterstream, helps to survive temporary internet connection outages, and reduces storage costs. You may not need to send all the data to the datacenter, just some pre-filtered or pre-aggregated records. Confluent Replicator can pump the data from these small deployments at the edge to the bigger cluster, which could be managed by Confluent Cloud.</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/mqtt-waterstream-kafka-schema-1024x407.png" alt="" class="wp-image-3137"/></figure>



<p class="wp-block-paragraph">To take it to the extreme, we’ve tested a deployment on a single&nbsp;<code>n1-standard-1</code>&nbsp;node in GCP (1 CPU, 3.7 GB RAM), which contains Apache ZooKeeper, Kafka, and Waterstream. The same&nbsp;<a href="https://github.com/simplematter/simplematter-mqtt-load-simulator" target="_blank" rel="noreferrer noopener">tool</a>&nbsp;used for generating the load for the scalability tests was used, but with only one node simulating 20,000 clients, each sending one message per every 10 seconds. Of those clients, 14,000 managed to connect successfully, with Waterstream writing 1,400 messages/second to Kafka. Below is a Grafana dashboard running locally, scraping data from Waterstream, which runs on a GCP node.</p>



<figure class="wp-block-image size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/waterstream_grafana-1024x503.png" alt="" class="wp-image-3138"/></figure>



<p class="wp-block-paragraph">Of course, with just a single node, you don’t have high availability and may suffer some downtime. Actual edge deployments will rarely go that far and should choose the size and number of nodes according to availability and performance needs.</p>



<h2 class="wp-block-heading">Live demo of Waterstream, Confluent Cloud, and ksqlDB</h2>



<p class="wp-block-paragraph">We’ve built a&nbsp;<a href="https://fleetdemo.waterstream.io/" target="_blank" rel="noreferrer noopener">live demo</a>&nbsp;of Waterstream connecting thousands of (virtual) devices and integrating it with Confluent Cloud and ksqlDB. It simulates a fleet of 15,000 trucks driving through Italy, which continuously send their position using MQTT messages.</p>



<p class="wp-block-paragraph">Starting from the current position, each truck is assigned to a random destination. The route is computed with OpenStreetMap using a profile compatible with a heavy vehicle. Upon reaching the destination, the truck stops for a random amount of time before starting again, thus simulating driver rest and new load preparation.</p>



<p class="wp-block-paragraph">The map displays a subset of the vehicles so that you can follow them as they move. The overall effect is pretty realistic with a smooth, fluctuating flow of messages presented in the embedded Grafana dashboard.</p>



<p class="wp-block-paragraph">Each device is simulated with an MQTT client that sends the following data to Waterstream in JSON format:</p>



<ul class="wp-block-list"><li>The plate that is used as the truck identifier</li><li>The current position</li><li>The next waypoint</li><li>The current speed</li><li>A timestamp</li></ul>



<p class="wp-block-paragraph">Each truck uses its MQTT topic, which consists of the concatenation of a fixed prefix together with the plate. Those topics are mapped into a single Kafka topic to make it easier to analyze them.</p>



<p class="wp-block-paragraph">The UI presents two kinds of information extracted from the Kafka topic. On the map, there are 15 markers showing moving vehicles, while on the sidebar, we present an aggregation of truck direction partitioned by cardinal points.</p>



<div class="wp-block-image"><figure class="aligncenter size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/waterstream-demo-mqtt-fleet-1024x488.png" alt="" class="wp-image-3139"/></figure></div>



<p class="wp-block-paragraph">This is how the data travels across this demo:</p>



<div class="wp-block-image"><figure class="aligncenter size-large"><img decoding="async" src="https://waterstream.io/wp-content/uploads/2021/04/waterstream-mqtt-ksqldb.png" alt="" class="wp-image-3140"/></figure></div>



<p class="wp-block-paragraph">The vehicles simulator sends data to Waterstream via MQTT, which in turn streams the data into Kafka for tracking the vehicles. ksqlDB processes this information and writes the results into another Kafka topic. Waterstream has mappings for both these topics so it can read them. The UI uses MQTT over WebSocket to connect to the Waterstream without any intermediaries. Authorization rules in Waterstream are configured in such a way that allows read-only access to the UI and write permissions to the vehicles simulator.</p>



<p class="wp-block-paragraph">The source code of the demo is available on&nbsp;<a href="https://github.com/simplematter/waterstream-fleet-demo" target="_blank" rel="noreferrer noopener">GitHub</a>.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">This blog post introduced a new option for combining MQTT and Kafka: Waterstream, a Kafka-native MQTT broker implementation leveraging Kafka Streams. This simplifies your event streaming architecture, no matter if your deployment is in the cloud with Confluent Cloud or at the edge with Confluent Platform.</p>



<p class="wp-block-paragraph">As proven by its design and benchmarks, this solution is scalable, able to connect to hundreds of thousands of MQTT devices, and fully MQTT compliant with bi-directional communication.</p>



<p class="wp-block-paragraph">The blog post first appeared on the <a href="https://www.confluent.io/blog/iot-streaming-use-cases-with-kafka-mqtt-confluent-and-waterstream/" target="_blank" rel="noreferrer noopener nofollow">Confluent Blog</a>.</p>
<p>The post <a href="https://waterstream.io/2020/07/21/apache-kafka-native-mqtt-at-scale-with-confluent-cloud-and-waterstream/">Apache Kafka Native MQTT at Scale with Confluent Cloud and Waterstream</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Waterstream is a verified Confluent integration</title>
		<link>https://waterstream.io/2020/06/10/waterstream-verified-integration/</link>
		
		<dc:creator><![CDATA[Waterstream]]></dc:creator>
		<pubDate>Wed, 10 Jun 2020 19:05:13 +0000</pubDate>
				<category><![CDATA[Confluent]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[News]]></category>
		<category><![CDATA[Technology]]></category>
		<category><![CDATA[confluent]]></category>
		<category><![CDATA[connector]]></category>
		<category><![CDATA[integration]]></category>
		<category><![CDATA[Waterstream]]></category>
		<guid isPermaLink="false">https://the7.io/beauty-studio/?p=898</guid>

					<description><![CDATA[<p>Waterstream is an verified Confluent integration We are very excited to announce from today Waterstream is officially Kafka integration verified by Confluent. In other words Waterstream is a fully supported partner integrations that meet a standard level of functional compatibility with the Confluent Platform ecosystem. We are in good company with other important vendors like Mongo [&#8230;]</p>
<p>The post <a href="https://waterstream.io/2020/06/10/waterstream-verified-integration/">Waterstream is a verified Confluent integration</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></description>
										<content:encoded><![CDATA[<h1>Waterstream is an verified Confluent integration</h1>
<p>We are very excited to announce from today Waterstream is officially Kafka integration <a href="https://www.confluent.io/hub/simplemattersrl/waterstream-kafka-mqtt">verified by Confluent</a>. In other words Waterstream is a fully supported partner integrations that meet a standard level of functional compatibility with the Confluent Platform ecosystem.</p>
<p>We are in good company with other important vendors like Mongo and HiveMQ. However, Waterstream is not a simply source or sink connector. It&#8217;s a full fledge MQTT brokers that wraps Kafka and turns it into an IoT platform with no other external integration.</p>
<p>To learn more how Waterstream differs from other solutions, check our <a href="/product/comparison">comparison page</a>.</p>
<p>The post <a href="https://waterstream.io/2020/06/10/waterstream-verified-integration/">Waterstream is a verified Confluent integration</a> appeared first on <a href="https://waterstream.io">Waterstream.io | Simplify MQTT Data Integration</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
