I love the idea of everything as code. It enables use of version control for all aspects of software development. These days we use everything as code approach for documentation, infrastructure provisioning, deployment automation, continuous integration build jobs, pull request and issue templates, and many other things. The beauty of this approach is that it enables us to use the same tools as we use use for software development.
One aspect of the software development where we are lacking the as code approach is in creating architecture diagrams.
As software engineers and architects we all have to create architecture diagrams to convey our ideas with other fellow engineers. For this discussion it does not matter whether you create architecture diagrams before or after or during development. The fact of the matter is that we all have to create diagrams in our software engineering life.
Since the last couple of years I was on the look out for a free and open-source diagram tool that allow me to save the diagram source in the version control. One tool that I used heavily in the last couple of years draw.io. We can use either the web or desktop editor to create diagrams. The diagram source in XML format can be backed to Google drive or Git repository. This tool has been great help for me and I really love using it. But to me draw.io is still not the pure as code tool. You use it like just any other diagram tool. It does not have code first approach. Code is the second class citizen in draw.io.
Recently I discovered a pure diagram as code tool called Diagrams. With Diagrams, you create diagrams by writing Python code. You can use it to build cloud and on-premise architecture diagrams. It supports AWS, Azure, GCP, Kubernetes, Alibaba Cloud, and Oracle Cloud.
To get started with Diagrams, you need to run following commands. The below are instructions for running it on Mac. Create a directory where you want to store diagrams and navigate to it.
$ brew install graphviz $ virtualenv -p python3 venv $ source venv/bin/activate $ pip install diagrams
You don’t have to type
$. It signifies command prompt.
Most tools stand on the shoulders of other powerful tools. Diagrams uses Graphviz its backbone. In the commands shown above, we first installed Graphviz. It is an open source graph visualisation software that makes use of abstract graphs and network to represent structural information.
Next, we installed diagrams library. You will notice that I am using Python Virtualenv to avoid polluting my global Python installation.
Now that we have met our prerequisites. Let’s learn how to use Diagrams by creating a few diagrams.
I will use examples mentioned in the talk: Scaling Up to Your First 10 Million Users.
It is one of the best conference talk on building scalable systems I have seen. Since this talk is about AWS we will be building diagrams that use AWS services.
Please note that I will be deviating a bit from the architecture presented in the video. Also, I am still learning Diagrams library so the code shown below might not be the best.
Diagram 1: Application built for 1 user
In the talk author starts by defining architecture for a single user application. For a single user application, we can use following:
- AWS Lightsail: Virtual private server
- Database and application server running on single machine
- Elastic IP
- Amazon Route 53
The Python code for the diagram is shown below.
from diagrams import Cluster, Diagram, Edge from diagrams.onprem.client import User from diagrams.aws.compute import Lightsail from diagrams.aws.network import Route53 with Diagram("Single User Architecture", show=True, direction="TB"): with Cluster(""): route_53 = Route53("dns") with Cluster("AWS"): route_53 - Edge(label="Elastic IP", color="orange", forward=True) >> Lightsail("Lightsail Server") User("User") >> route_53
The above code will produce following diagram.
Diagram 2: Users between 1 and 100
Now, we will install database on its own server. We will use AWS RDS service.
from diagrams import Cluster, Diagram from diagrams.onprem.client import User from diagrams.aws.compute import EC2 from diagrams.aws.network import Route53 from diagrams.aws.database import RDS with Diagram("100 User Architecture", show=True): route_53 = Route53("dns") with Cluster("AWS"): route_53 >> EC2("Web Server") >> RDS("Database") User("User") >> route_53
The above code will produce following diagram.
Diagram 3: Between 100 users and 1000
Add a cache and use bigger machines.
from diagrams import Cluster, Diagram from diagrams.onprem.client import User from diagrams.aws.compute import EC2 from diagrams.aws.network import Route53, ElasticLoadBalancing from diagrams.aws.database import RDS, ElastiCache with Diagram("1000 User Architecture", show=True): with Cluster("AWS"): route_53 = Route53("dns") lb = ElasticLoadBalancing("lb") vm_group = [EC2("Web1"), EC2("Web2")] cache = ElastiCache("Redis Cache") route_53 >> lb >> vm_group >> cache >> RDS("Database") User("User") >> route_53
The above will create following diagram.
Diagram 4: Between 1000 and 10000 users
Now, we will use running application instances in multiple AZs. Also, we will use horizontally scalable database Aurora in multiple AZ. We are rendering static assess using Cloud Front and S3
The Python code is shown below.
from diagrams import Cluster, Diagram from diagrams.onprem.client import User from diagrams.aws.compute import EC2 from diagrams.aws.network import Route53, ElasticLoadBalancing, Cloudfront from diagrams.aws.database import Aurora, ElastiCache from diagrams.aws.storage import S3 with Diagram("10000 User Architecture", show=True): with Cluster("AWS"): route_53 = Route53("DNS") cloud_front = Cloudfront("CDN") s3 = S3("S3 Object Storage") lb = ElasticLoadBalancing("LB") cache = ElastiCache("Redis Cache") aurora_db = Aurora("Aurora DB") with Cluster("Availablity Zone 1"): vm_group1 = [EC2("Web1"), EC2("Web2")] lb >> vm_group1 >> cache >> aurora_db with Cluster("Availablity Zone 2"): vm_group2 = [EC2("Web3"), EC2("Web4")] lb >> vm_group2 >> cache >> aurora_db route_53 >> lb route_53 >> cloud_front >> s3 User("User") >> route_53
In this blog we covered how to use Diagrams library to create diagrams by writing Python code. I think Diagrams is a good start in the right direction. It still does not give all the freedom and you will be constrained by it. I will keep following this library and hopefully contribute back.