Observability Done Right: Best Practices and Anti-Patterns for Effective System Monitoring

-
Image
  WHAT Observability is a concept that refers to the ability to gain insights into the behavior and performance of complex systems. In the context of software engineering, observability involves the collection, analysis, and visualization of data from software applications, infrastructure, and other components of a system. In the animal kingdom, observability plays a critical role in survival, allowing animals to monitor their surroundings, detect threats, and find food. Dolphins use echolocation to observe their surroundings. They emit high-frequency sounds that bounce off objects, allowing them to create a 3D map of their environment. Thanks for reading Knowledge Cafe! Subscribe for free to receive new posts and support my work. Subscribed WHY In today's era, architectures are becoming increasingly large, complex, and fast-paced due to the faster development and deployment of software by distributed teams with the help of DevOps, continuous delivery, and agile development methodo...

Apache Kafka Use cases

-


Kafka was created first in the tech labs of LinkedIn – the world’s biggest network of professionals. Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale. Most known for its excellent performance, low latency, fault tolerance, and high throughput, it's capable of handling thousands of messages per second.

Website activity tracking(E commerce)

Website activity (product views, product searches) is published to customer activity topics and becomes sourced to real-time processing, offline analytics to tool like Google’s BigQuery, Azure CosmosDB, RedShift.

Real time Fraud detection

Juniper Research estimates online FDP (fraud detection and prevention) spending is all set to touch $9.3 billion by 2022. Real-time data streaming helps to spot anomalies – that is peculiar and abnormal instances that deviate from the usual way of operation. These anomalies can be classified as fraud or errors. Apache Kafka helps in large volume data ingestion. It is easily scalable which helps manage any volume of data for real-time fraud detection.

Online Gaming

A key challenge in infrastructure monitoring for online gaming is the required elasticity. Kafka allows for the number of partitions to be increased on the fly, so more Brokers can be added to handle extra partitions for a Topic. Each additional Broker and partition brings a performance increase. Kafka also maintains replicas of each partitions data across the nodes.

Storage for back pressure and decoupling of services

 It is not just a messaging system! Kafka also stores data as long as you want. For instance, a Kafka topic is stored for a few days for log analytics. Another Kafka topic is stored for years to analyze customer and payment transactions of the past.

Log aggregation

Log aggregation systems typically collects physical log files from servers and puts them in a central server (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of logs or event data as a stream of messages. Application can push data on specific Kafka topics which can be aggregated and processed later by consumer.

Stream processing

Apache Kafka is so good for real-time data streaming that it even used by giant corporations like Twitter, Uber, Pinterest, Netflix, Tumblr, PayPal and many others. Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing.

Commit Log

Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage.


Popular posts from this blog

Chain of responsibility using Spring @Autowired List

-
Image
There is a way in Spring 3.1 to auto populate a typed List which is very handy when you want to push a bit the decoupling and the cleaning in your code. To show you how it works, I will implement a simple chain of responsibility that will take care of printing some greetings for a passed User. Let start from the (only) domain class we have, the User: package in.softcaretech.springchain; public class User { private final String name; private final char gender; public User(String name, char gender) { super(); this.name = name; this.gender = gender; } public String getName() { return name; } public char getGender() { return gender; } } Then we create an interface that defines the type for our command objects to be used in our chain: package in.softcaretech.springchain; public interface Printer { void print(User user); } This is the generic class (the template) for a Printer implementation. The  org.springframework.core.Ordered  is used to tell the AnnotationAwareOrderCo...
Explore more »

Iterate Through a HashMap

-
The  HashMap  is one of the most useful data structures in the Java programming language. Once you have a HashMap filled with data, you may want to iterate through its keys and values. Below are three different ways to iterate through a Java HashMap. Sample HashMap Here is our sample HashMap. The key is an Integer and the value is a String: HashMap <Integer, String> hm = new HashMap<Integer, String>(); hm.put(0, "zero"); hm.put(1, "one"); hm.put(2, "two"); In the above Java code, we first declare the HashMap. Then we add the values "zero", "one" and "two" with the keys 0, 1 and 2 respectively. Now that the HashMap has data, we can try to iterate over the keys and values. Iteration Example 1 for (int i=0; i < hm.size(); i++) { Integer key = hm.keySet().toArray()[i]; String val = hm.values().toArray()[i]; System.out.println("key,val: " + key + "," + val); } In this example, we use a  for  loop ...
Explore more »

Under the Hood: Understanding the Gossip Protocol in Apache Cassandra

-
Image
  Gossip Protocol Apache Cassandra is a highly scalable and distributed NoSQL database management system designed to handle large amounts of data across many commodity servers. It was developed at Facebook and later became an Apache Software Foundation project. Cassandra offers a number of key features, including: Linear scalability, high availability, Predictive data consistency, flexible data modeling and high performance. What is Gossip Protocol The Gossip protocol is a key component of the Apache Cassandra distributed database system. It is used for node communication and failure detection within the cluster. The following is how it works: Each node in the Cassandra cluster maintains a list of all other nodes in the cluster and information about their status. The Gossip protocol operates in rounds, where each node sends its state information to a randomly selected set of other nodes. The receiving nodes update their state information based on the information received from their...
Explore more »