Observability Done Right: Best Practices and Anti-Patterns for Effective System Monitoring

-
Image
  WHAT Observability is a concept that refers to the ability to gain insights into the behavior and performance of complex systems. In the context of software engineering, observability involves the collection, analysis, and visualization of data from software applications, infrastructure, and other components of a system. In the animal kingdom, observability plays a critical role in survival, allowing animals to monitor their surroundings, detect threats, and find food. Dolphins use echolocation to observe their surroundings. They emit high-frequency sounds that bounce off objects, allowing them to create a 3D map of their environment. Thanks for reading Knowledge Cafe! Subscribe for free to receive new posts and support my work. Subscribed WHY In today's era, architectures are becoming increasingly large, complex, and fast-paced due to the faster development and deployment of software by distributed teams with the help of DevOps, continuous delivery, and agile development methodo...

Chaos Engineering : Game Day

-

 





What is chaos engineering:

Chaos engineering is a methodology that helps developers attain consistent reliability by hardening distributed services against failures in production. Another way to think about chaos engineering is that it's about embracing the inherent chaos in complex systems and, through experimentation, growing confidence in your solution's ability to handle it. A common way to introduce chaos is to deliberately inject faults that cause system components to fail. The goal is to observe, monitor, respond to, and improve your system's reliability under adverse circumstances.

Why Chaos Engineering?

Contrary to what the name may indicate, chaos events are not performed in a chaotic fashion. The goal of chaos engineering is to identify weakness in a system through controlled experiments that introduce random and unpredictable behavior. A main benefit of chaos engineering is that organizations can use it to identify vulnerabilities before a hacker does or before a system failure. In simple words Game day or Chaos day is like fire drill. 

Chaos Day/ Game Day : Planned Failure

About a week prior to the event, our champion sent a company-wide email on behalf of the chaos team. 
In any Game Day, an exact target or targets should be specified. Without it, it’s impossible to bring in the right people to run and observe the Game Day. It could be as simple as “Cassandra cluster” or “Inventory service” in which case at least those that either run or use the service can make a decision whether they can or want to attend.

Game Day Participants and Roles

Chaos General: This person owns the experiments that are going to be run
Chaos Commander: This person is the one in charge of implementing these Chaos Engineering experiments
Chaos Scribe: chaos scribe is observing and writing down all the notes as this game day happens
Chaos Observer: chaos observer is the one testing that user experience, looking at monitoring, alerting dashboards.


What is a Chaos Schedule?

  • Start in war room or over zoom call with all participants
  • White boarding architecture session to clear assumption and create hypothesis 
  • Defining test cases and scope (define blast radius)
  • Execution of test and monitoring
key questions to ask as tests are being conducted:
    • Do we have enough information?
    • Is the behavior what we expected?
    • What is the customer seeing if this were to happen?
    • What’s happening to systems upstream or downstream?
  • Recap : 
What happened? Was that expected? What do we do next? 
After tests are run, it’s good to take some time to wind down, then have a follow up recap. This should be done relatively soon after the GameDay (days, not weeks), as the experience is still fresh for everyone.

 


Summary:
While a single Game Day can provide some insights into the behavior of an application or system, continuous implementation will help increase security in the operation of distributed applications and components in the long run. Hopefully, this has given you some insight into how companies like Netflix, Amazon and Walmart can benefit from the concept of game days. In future articles we will discuss about various types of attacks and initiatives for chaos engineering. 

References:

Popular posts from this blog

Chain of responsibility using Spring @Autowired List

-
Image
There is a way in Spring 3.1 to auto populate a typed List which is very handy when you want to push a bit the decoupling and the cleaning in your code. To show you how it works, I will implement a simple chain of responsibility that will take care of printing some greetings for a passed User. Let start from the (only) domain class we have, the User: package in.softcaretech.springchain; public class User { private final String name; private final char gender; public User(String name, char gender) { super(); this.name = name; this.gender = gender; } public String getName() { return name; } public char getGender() { return gender; } } Then we create an interface that defines the type for our command objects to be used in our chain: package in.softcaretech.springchain; public interface Printer { void print(User user); } This is the generic class (the template) for a Printer implementation. The  org.springframework.core.Ordered  is used to tell the AnnotationAwareOrderCo...
Explore more »

Iterate Through a HashMap

-
The  HashMap  is one of the most useful data structures in the Java programming language. Once you have a HashMap filled with data, you may want to iterate through its keys and values. Below are three different ways to iterate through a Java HashMap. Sample HashMap Here is our sample HashMap. The key is an Integer and the value is a String: HashMap <Integer, String> hm = new HashMap<Integer, String>(); hm.put(0, "zero"); hm.put(1, "one"); hm.put(2, "two"); In the above Java code, we first declare the HashMap. Then we add the values "zero", "one" and "two" with the keys 0, 1 and 2 respectively. Now that the HashMap has data, we can try to iterate over the keys and values. Iteration Example 1 for (int i=0; i < hm.size(); i++) { Integer key = hm.keySet().toArray()[i]; String val = hm.values().toArray()[i]; System.out.println("key,val: " + key + "," + val); } In this example, we use a  for  loop ...
Explore more »

Under the Hood: Understanding the Gossip Protocol in Apache Cassandra

-
Image
  Gossip Protocol Apache Cassandra is a highly scalable and distributed NoSQL database management system designed to handle large amounts of data across many commodity servers. It was developed at Facebook and later became an Apache Software Foundation project. Cassandra offers a number of key features, including: Linear scalability, high availability, Predictive data consistency, flexible data modeling and high performance. What is Gossip Protocol The Gossip protocol is a key component of the Apache Cassandra distributed database system. It is used for node communication and failure detection within the cluster. The following is how it works: Each node in the Cassandra cluster maintains a list of all other nodes in the cluster and information about their status. The Gossip protocol operates in rounds, where each node sends its state information to a randomly selected set of other nodes. The receiving nodes update their state information based on the information received from their...
Explore more »