1. You need to ingest tons of data from your on premises devices.
2. You have a low bandwidth connection to the internet / to your datacenter.
3. You have tons and tons of data that need to be transmitted in near real-time.
4. You have to transmit this data securely and near real time using your low bandwidth compression.

Solution (Use Compression for data, Use Kafka or RabbitMQ On Edge, for ingestion)

1. Step 1 – Compression – Since you have large data volumes to ingest and transfer, how about compressing the data? The compression can happen natively at the OS level – using Unix Compress or Windows GZIP.
2. Step 2 – Edge Devices with RabbitMQ  – These can serve dual purposes – ingesting and processing data as it comes in.  The processing can include the COMPRESSION.
3. Step 3  – Transmitting the data  – This needs to be a reliable messaging protocol

IoT and MQTT based communication – RabbitMQ

Microservices and RabbitMQ (or any messaging system)

The post Messaging System Architecture – Low Bandwidth, High Data Volume, Disconnected Systems appeared first on Anuj Varma, Hands-On Technology Architect, Clean Air Activist.