We are proud to announce that Waterstream version 1.3.0 brings support of the bridge mode. A flexible way to utilize an existing MQTT data source for testing and data ingesting in Kafka.
With the bridge you can establish a bidirectional or unidirectional connection between Waterstream and another MQTT broker and get your messages replicated between them – like on this picture:
One of the machines of the Waterstream cluster can be configured as a bridge – it connects to another MQTT broker as a regular MQTT client and according to the configuration reads or writes messages to make sure they’re replicated in Kafka.
On the Waterstream side the bridge doesn’t use the network layer, but other than that it also looks like the normal MQTT client – therefore it needs its own Client ID. A single Waterstream machine may have bridges to multiple remote brokers. Both MQTT v 3.1.1 and MQTT v 5.0, but the use of the 5.0 is recommended because it supports No Local subscription option which simplifies avoiding message loops in a bidirectional connection.
We also made an example bridging the data from the Lufthansa Open API notification service, if you are interested you can chek the Demo here.
You may use a bridge mode as an alternative to the Source/Sink MQTT Connectors for Kafka Connect , as a means of a gradual transition to Waterstream from another MQTT broker or to make some tests on your architecture without compromising the “live” system.
To run Waterstream in the bridge mode you need to create a bridge configuration file, make it available to Waterstream and specify its location with the environment variable MQTT_BRIDGES_CONFIG_FILE.
Following a step by step configuration setup example:
docker run -it --rm \
-e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
-e MQTT_PORT=1883 \
-e SESSION_TOPIC=mqtt_sessions \
-e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
-e CONNECTION_TOPIC=mqtt_connections \
-e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
-e MQTT_BRIDGES_CONFIG_FILE=/etc/bridge.conf \
-v bridge.conf:/etc/bridge.conf:ro
-v waterstream.license:/etc/waterstream.license:ro \
--network host \
--name waterstream-kafka simplematter/waterstream-kafka-minified:1.3.7
Here is an example configuration file:
connectionBackOffInitial: 5000
connectionSuccessMinDuration: 60000
bridges: [
{
name: "First Bridge"
host: "broker1"
port: 1883
protocolLevel: 5
cleanSession: true
localClientId: "s1"
remoteClientId: "bridge1"
topics: [
{
pattern: "area1/#"
localPrefix: "a/"
remotePrefix: "b/"
direction: "OUT"
qos: 2
},
{
pattern: "#"
localPrefix: "allIncoming/"
direction: "IN"
}
]
}
]
This instructions will define a bridge to the MQTT v5 broker on broker1:1883 which has one inbound pattern and one outbound pattern which do some prefix re-mapping.
The definition of patterns allow us to select which topic we would like to use.
For instance, with the previous configuration, remote messages from MQTT topic “foo” will be replicated locally into the topic “allIncoming/foo”.
Local messages from topic “a/area1/bar” will be replicated to the remote topic “b/area1/#”. Local messages from topics “c/area1/bar” or “a/area2/bar” will not be replicate to the remote machine because the don’t match the pattern and prefix from the configuration.
You can find more details about Waterstream bridge configuration here.
If you wish to know more details or you need some clarifications please do not hesitate to contact us.
