Representatives from South Africa’s most prominent companies recently met at an event organised by Synthesis Software Technologies in partnership with Confluent to discuss their experiences with Apache Kafka®.
Apache Kafka® is an event streaming platform initially developed at LinkedIn before being released as open-source software.
“Event streaming” doesn’t refer to live streaming video of conferences, but rather providing large quantities of data as a real-time stream of events. Kafka can handle trillions of events a day.
Around 70% of Fortune 500 companies use Kafka event streaming to transform their businesses.
Among the organisations represented at the roundtable were FNB, Absa, Capitec, Vitality Group, Vodacom, Cell C, Discovery Insure, BankservAfrica, TransUnion, JDGroup, ABINBEV, and Hollard.
Synthesis uses innovative technology solutions to make businesses more competitive today. They have helped several of these institutions with their Kafka deployments.
Confluent was founded by the original creators of Apache Kafka® and offers a managed cloud solution that enhances the technology with an ecosystem of Data in Motion services that enable a modern data infrastructure. Synthesis became a Confluent partner in October 2019.
“Synthesis is Confluent’s trusted partner, having successfully established a data streaming platform that sets Data in Motion across organisations in South Africa,” said Confluent UKI & Africa Enterprise Account Executive, Jack Bingham.
Confluent Data in Motion with Apache Kafka® is widely adopted for use cases ranging from real-time inventory, logging for omnichannel banking or IoT device instrumentation.
Its strength is its ability to make high-volume data available as a real-time stream for consumption in systems with different requirements — from batch systems like Hadoop and real-time systems that require low-latency access, to stream processing engines that transform the data streams as it arrives.
The infrastructure provided by Kafka lets you build around a single central nervous system that decouples systems and applications to liberate data across an organisation for real-time capture and processing.
First National Bank (FNB) selected Kafka to guarantee the scale of the data infrastructure. Currently, all core banking mainframe data is available through Kafka and generates over a gigabit per second of throughput.
Once data is available in Kafka, new innovative experiences and applications can be built on the real-time stream. For example, FNB explained it uses Kafka to handle click and tap events streamed from its smartphone app that helps the bank perform real-time fraud detection to save more than R7m per month.
BankservAfrica, a non-profit clearinghouse owned by South Africa’s big banks that facilitates interbank transactions, is using Kafka in a new payments system it plans to launch this year.
The project is intended to be a solution for large volumes of low-value transactions to boost financial inclusion in South Africa.
Currently called the Rapid Payments Programme, Bankserv said the system will offer real-time switching capability between banks for instant clearing.
BankservAfrica anticipates clearing 1,500 transactions per second over this system, and Bankserv believes its chosen technology will get it there.
Another major player using Apache Kafka® is Vitality Group.
“As the event streaming capability was a new technology being introduced into my team, my existing developers had very little practical exposure to the product,” said Vitality Group Head of Integrations, Nic Mouyis.
“Through our arrangement with Synthesis, we were able to very quickly onboard very effective engineers who were able to assist us with defining the solution, rolling it out, as well as optimising the throughput of the solution while still being able to deliver the project on time.”
Getting buy-in from your architect and developers
A major topic of discussion was how organisations can onboard Kafka and get buy-in from their system architects and developers.
One suggestion to get developers interested was to build a small proof-of-concept system at the organisation to demonstrate Kafka’s power and prove value in the approach.
Confluent’s Bingham also suggested that putting together a team to spearhead a Kafka project is often a great way to get the attention of the rest of the developers in an organisation. This forms the foundation of a centre of excellence to own the architecture, governance and operational best practices to scale new use cases.
FNB suggested pointing them to a powerful Kafka feature like Exactly-Once to convince architects to adopt Kafka.
In simple terms, Exactly-Once Semantics is a framework and transaction model that allows Kafka to process data without loss or duplication, ensuring that computed results are always accurate.
By showing architects and developers the functionality they will get without sacrificing the utility they are used to, their seniors and teammates can convince them that Apache Kafka® is worth looking into.
Synthesis offers Kafka assessments where a Principal Kafka Engineer, the Head of the Synthesis Data Streaming Practice, and optionally other specialists look at your organisation’s needs.
The specialist technology consultancy also offers in-depth and customised real-world training on Kafka.
For more information, visit the Synthesis website and contact the team.