Mesos - Automate your Data Center with Containers
These days, the containers are getting more and more popular. To know more about the problems with them and getting familiar with clustering the containers, I’ve attended a talk (slides) about an open-source Apache project – Mesos, given by David Greenberg, author of the book “Building Applications on Mesos”.
As a container, for example the docker container, we can imagine an application code with operating system and isolated resources. The containers are popular mainly for these 3 benefits: easy to deploy, easy to reproduce and easy to build. However, using containers may be challenging when we need to answer these questions:
How to automate operations?
How to migrate from the existing infrastructure?
How to manage database/file storage? The containers perfectly manage cpu and memory, but not the disc space.
To help solving challenges with containers, there are container orchestration systems like Mesos, Kubernetes or Docker Swarm. They are supposed to manage cluster of containerized nodes. The main advantage of Mesos over its alternatives, is the great scaling (e.g. 10k+ nodes in Mesos vs. hundred of nodes in Kubernetes).
Mesos comes with two useful features: offers and volumes. By offer it is meant that Mesos offers some resources and then applications are allowed to react to what’s available. For example, if available resources (cpu, memory) are lower than required, it may be better that application runs slowly than not running at all. Volumes are used to store data. Mesos supports two types: internal volumes (local discs) and external volumes (network disc). It follows the mental model: “Cattle, not pets!” Basically the pets are not fun to be on-call.
Problematic data store description directly in the Docker documentation
To get started with Mesos, there are two options:
Hard way – do it yourself (e.g. use and configure Mesos subsystems like Marathon, Mesos DNS and Marathon-LB).
Easy way – use commercial product Mesosphere DC/OS
The Asynchronous Uncoordinated Continuous Delivery of 35+ uServices
In the next talk (slides), we discussed practical experience with Continuous Delivery in Ocado Technology, a company, where I worked before joining Davinci software. The Ocado’s state-of-the-art software solutions run a pure online supermarket emphasizing on maximum efficiency and customer satisfaction. Something like Amazon in the world of groceries.
They have recently moved from classical to continuous delivery way of developing webshop applications. The main change is to perform dozen (10-20) of deployments a day instead of a single deployment in a few weeks with couple of hotfixes. With more deployments, the number and severity of bugs should be limited and easier to spot. To decrease risks, it is unnecessary to provide fully automated testing with high coverage. Also developers must be more responsible and careful what they are developing and deploying. Technical debt should be minimized.
To be able to control, what is active in the deployment or release, there are used feature flags. Basically the feature source code is present but its feature flag says if the feature is active or not. That makes a clear border between the deployments and release. Also rollback is simpler, because it is just about changing the feature flag to false. One drawback is that feature flags add some complexity to manage the flags and logic to be able to work with and without the active feature.
Backward compatibility is achieved by both old version and new version working in a same time. If the new version is working properly in some period, then the old version may be switched off or removed. This helps to avoid big deployments of multiple services and databases at once, but make deployments sequential and limit the impact of the changes.
The developers work on small subtasks of a story with short living branches (1-2 days). After a quick code review, the branch is merged to the master branch and deployed (with feature flag). Developer then can do a quick test by activating the feature flag on. With such a small changesets, bugs can be easily spotted and the feature may be deactivated quickly. Also, short living branches prevent having big conflicts if two developers work on the same resource.
There were few whole-day workshops at JAX2016 conference from which I've chosen the "Developing microservices" one given by microservice pioneer Chris Richardson to get familiar with the microservices architecture. The whole conference was mainly about the buzz words as microservices, containers or docker. Why there is so much talk about the microservices? Maybe because of a great success of disrupting companies like Netflix or Uber. For example, Netflix architecture consists of 600 microservices.
The classic approach is to start with monolithic application. Everything is working fine at the beginning. But as the service or application is successful, big problems start with growing, complexity, scaling, lots of coordination, obsolete technology stack and everything leads toward the "monolithic hell". Microservices are here to solve those problems.
But... Are the microservices useful for everybody?
The correct answer is NO. Basically the microservices are usually best for large and complex applications, where the benefits win over the drawbacks. We would need to carefully consider drawbacks of the microservices architecture, mainly complexity in architecture, multiple databases with transaction management, complexity of testing and deploying, and the risk of high latency.
Taxi application as a microservices example
We have learned a lot of aspects about the microservice architecture:
How to partition the monolithic application
Strategies how to deploy
Internal communication (between the microservices themselves)
External communication (between the microservices application and the world)
Service discovery (both client and server side)
Ways to move from a monolithic application toward microservices
One of the biggest issues to be solved in some microservice-based application are the distributed transactions. Simple example are two microservices with separate databases. The first service updates an entity in its db, call the second service to update the second database. This needs to be transactional, i.e. if the second database update fails, we need to revert also the first database. With distributed transactions, we need to move from ACID to BASE (Basic Availability, Soft-state, Eventual consistency) view.
One solution could be two-phase commit (2PC). But this may not help us, because 2PC has several drawbacks (single point of failure, chatty, reduced throughput due to locks, low NoSQL support). Another solution is the global database. But that approach suffers from lack of encapsulation/tight coupling and might not satisfy the data access requirements of multiple microservices. The third solution is event sourcing.
The basic principle of the event sourcing is to store events on every data/entity change instead of storing states. Then service can reconstruct the domain object in memory from the events. Events are stored in the „Event store“ that is a component combining database with the message broker. This is a different paradigm from classical domain objects with their states stored. It can help us to fix problems with distributed transactions, but source code may be more complicated and harder to read. For example Eventuate or Lightbend.
Empathy - The Missing Ingredient in Effective Software Development
I found it interesting to listen if and how empathy should be used for software development. Daniel Bryant from UK has explained to us that we are typically developing software for other people with other people to create an (emotional) experience (slides). And therefore empathy is really important, because everything is about people. We can use approach with empathy for example for describing use cases or test cases, we describe real persons with background, goals, feelings and values as our users. Or another example may be that architect must do some coding to better understand developer’s point of view.
Very important aspect is to seek rapid feedback. No one gets things right all the time and feedback is a way to help improve things immediately together with our personal improvements. Empathy is like a muscle, we need to exercise to be more empathetic people.
Last advice that I have learned is: "Do not be afraid to be empathetic, because then your software can be more successful and the people happier!"