Observability Done Right: Best Practices and Anti-Patterns for Effective System Monitoring

-
Image
  WHAT Observability is a concept that refers to the ability to gain insights into the behavior and performance of complex systems. In the context of software engineering, observability involves the collection, analysis, and visualization of data from software applications, infrastructure, and other components of a system. In the animal kingdom, observability plays a critical role in survival, allowing animals to monitor their surroundings, detect threats, and find food. Dolphins use echolocation to observe their surroundings. They emit high-frequency sounds that bounce off objects, allowing them to create a 3D map of their environment. Thanks for reading Knowledge Cafe! Subscribe for free to receive new posts and support my work. Subscribed WHY In today's era, architectures are becoming increasingly large, complex, and fast-paced due to the faster development and deployment of software by distributed teams with the help of DevOps, continuous delivery, and agile development methodo...

Chaos Engineering | Type of Attacks

-

 


Today’s advance distributed software systems must be tested for potential weaknesses and faults. Chaos engineering is the process of testing a distributed computing system to ensure that it can tolerate unexpected disruptions. It relies on concepts underlying chaos theory, which focus on random and unpredictable behavior. If you are interested in knowing more about Chaos Engineering and History please refer this article from Gremlin 

In this article we will discuss about various categories of attack and some usecases. 



Resource Attack

Generate load across CPU, Memory and Storage devices
Help in preparation for sudden load change, validating auto scaling, test monitoring and alerting config. Its like preparing our system for Black Friday sale in advance. 

CPU Attack

CPU attack sends heavy traffic on system which can help to identify stability and performance undrer stress. We can also validate auto scaling and alerting mechanism. 

Memory Attack

Memory leak is top reason for "Out Of Memory" in production. Memory leaks happens when application consume more memory resources than release. This attack will help to validate hypothesis for memory intensive work load like in-memory cache, machine learning model. It will also help in cloud migration by simulating auto-scaling configuration. 

Disk Attack

Disk attacks are often used to simulate reading or writing a large data set, such as a restored backup, replicated database. It can also help in identifying loop holes in automatic disc cleanup process. 

I/O Attack

An IO attack can help you prepare for slower storage solutions by simulating their performance. This attack help to validate disk heavy work load (batch process which read/write from disk) and effectiveness of in-memory cache. 

State Attack

State attacks change the state of your environment by terminating processes, shutting down or restarting hosts, and changing the system clock. This lets you prepare your systems for unexpected changes in your environment such as power outages, node failures, clock drift, or application crashes.

Process Killer Attack

Process killer attacks allow teams to terminate a specific process or set of processes. This will ensure watch-dog effectiveness for application/service restart and testing leader re-election in clustered work load.

Shutdown Attack 

This is similar to chaos monkey where entire host is shutdown which enable team to build highly resilient system. This will help to validate DR scenarios like automatic work load migration, replication and high availability of clustered workload.   

Time Travel Attack

Time travel attacks allow you to change the system clock. This lets you prepare for scenarios such as Daylight Savings Time (DST), clock drift between hosts, and expiring SSL/TLS certificates.

Network Attack

Network attacks let you simulate unhealthy network conditions including dropped connections, high latency, packet loss, and DNS outages. This lets you build applications that are resilient to unreliable network conditions.

Blackhole Attack

Blackhole attacks help you simulate outages by dropping network traffic between services. This lets you uncover hard dependencies, test fallback and failover mechanisms, and prepare your applications for unreliable networks. We can also validate monitoring and alerting mechanism for cluster. 

Latency Attack

Latency is the amount of time taken for a network request to travel from one network endpoint to another. The Latency attack injects a delay into outbound network traffic, letting you validate your system’s responsiveness under slow network conditions. This will also help in circuit breaker configuration for retry and timeout threshold. 

DNS Attack

Recently we have seen Akmai DNS failure caused many popular becoming un-reachable. More info here The DNS attack simulates a DNS outage by blocking network access to DNS servers. This lets you prepare for DNS outages, test your fallback DNS servers, and validate DNS resolver configurations. 

Packet Loss Attack

This attack is very helpful for streaming services, such as live video or multiplayer gaming which rely on a high throughput of data. When there is network congestion, many packets are queued and some packages may loss due to queue capacity threshold on your hardware. Packet Loss attacks let you replicate this condition and simulate the end user experience and configuration of replay meachanism for better user experience.


In next article we will discuss about other Chaos Engineering concepts. 


Popular posts from this blog

Chain of responsibility using Spring @Autowired List

-
Image
There is a way in Spring 3.1 to auto populate a typed List which is very handy when you want to push a bit the decoupling and the cleaning in your code. To show you how it works, I will implement a simple chain of responsibility that will take care of printing some greetings for a passed User. Let start from the (only) domain class we have, the User: package in.softcaretech.springchain; public class User { private final String name; private final char gender; public User(String name, char gender) { super(); this.name = name; this.gender = gender; } public String getName() { return name; } public char getGender() { return gender; } } Then we create an interface that defines the type for our command objects to be used in our chain: package in.softcaretech.springchain; public interface Printer { void print(User user); } This is the generic class (the template) for a Printer implementation. The  org.springframework.core.Ordered  is used to tell the AnnotationAwareOrderCo...
Explore more »

Iterate Through a HashMap

-
The  HashMap  is one of the most useful data structures in the Java programming language. Once you have a HashMap filled with data, you may want to iterate through its keys and values. Below are three different ways to iterate through a Java HashMap. Sample HashMap Here is our sample HashMap. The key is an Integer and the value is a String: HashMap <Integer, String> hm = new HashMap<Integer, String>(); hm.put(0, "zero"); hm.put(1, "one"); hm.put(2, "two"); In the above Java code, we first declare the HashMap. Then we add the values "zero", "one" and "two" with the keys 0, 1 and 2 respectively. Now that the HashMap has data, we can try to iterate over the keys and values. Iteration Example 1 for (int i=0; i < hm.size(); i++) { Integer key = hm.keySet().toArray()[i]; String val = hm.values().toArray()[i]; System.out.println("key,val: " + key + "," + val); } In this example, we use a  for  loop ...
Explore more »

Under the Hood: Understanding the Gossip Protocol in Apache Cassandra

-
Image
  Gossip Protocol Apache Cassandra is a highly scalable and distributed NoSQL database management system designed to handle large amounts of data across many commodity servers. It was developed at Facebook and later became an Apache Software Foundation project. Cassandra offers a number of key features, including: Linear scalability, high availability, Predictive data consistency, flexible data modeling and high performance. What is Gossip Protocol The Gossip protocol is a key component of the Apache Cassandra distributed database system. It is used for node communication and failure detection within the cluster. The following is how it works: Each node in the Cassandra cluster maintains a list of all other nodes in the cluster and information about their status. The Gossip protocol operates in rounds, where each node sends its state information to a randomly selected set of other nodes. The receiving nodes update their state information based on the information received from their...
Explore more »