Problem Statement

  1. You need to ingest tons of data from your on premises devices.
  2. You have a low bandwidth connection to the internet / to your datacenter.
  3. You have tons and tons of data that need to be transmitted in near real-time.
  4. You have to transmit this data securely and near real time using your low bandwidth compression.

Solution (Use Compression for data, Use Kafka or RabbitMQ On Edge, for ingestion)

  1. Step 1 – Compression – Since you have large data volumes to ingest and transfer, how about compressing the data? The compression can happen natively at the OS level – using Unix Compress or Windows GZIP.
  2. Step 2 – Edge Devices with RabbitMQ  – These can serve dual purposes – ingesting and processing data as it comes in.  The processing can include the COMPRESSION.
  3. Step 3  – Transmitting the data  – This needs to be a reliable messaging protocol

IoT and MQTT based communication – RabbitMQ

Related image

Microservices and RabbitMQ (or any messaging system)

 

Related image

Anuj holds professional certifications in Google Cloud, AWS as well as certifications in Docker and App Performance Tools such as New Relic. He specializes in Cloud Security, Data Encryption and Container Technologies.

Initial Consultation

Anuj Varma – who has written posts on Anuj Varma, Hands-On Technology Architect, Clean Air Activist.

 Twitter Linkedin