In my current role as Chief Technology Officer (CTO), I am responsible for the engineering quality of our software delivery teams. We are an IT service organization where every now and then, we get to work with different kinds of customers in different domains and each at a different maturity level in software delivery.
One thing that I often struggle with is how to unite the individual engineering teams working for different customers with a common shared belief system that is actionable, pragmatic, and less abstract.
You may ask, isn’t that what organization values stand for? Before we answer, let’s define the three common terms: Values, Principles, and Practices.
Values, Principles, and Practices
- Values are abstract and desirable behaviors that make an organization’s belief system. Examples are courage, communication, quality, knowledge sharing, to name a few.
- Principles are basic truths that explain or control how something happens or work. Examples are principles of Newtonian physics, software engineering principles like Abstraction, Modularity, etc.
- Practices are principles applied on the ground. Example, test driven development, continuous integration, infrastructure as code, etc.
They build on top of each other and push in both the directions as shown below.
Let’s again come back to the question why organizational values are not sufficient. In my view, organizational values are powerful but they are general and they don’t help in engineering decision making. They don’t provide the specifics and guard rails necessary to drive decision making on a team level.
Engineering Team Principles
Engineering team principles provide a framework on how engineering teams should make decisions and carry out their work when faced with a difficult situation. The situation could be related to architecture and design, or prioritization, or soft issues like work behavior. They help set checks and balances in the system that help teams look at the bigger picture and make decisions, keeping larger goals in mind.
Engineering team principles can help transform a bunch of software programmers to a high performing autonomous engineering team.
This post is an aggregation of useful Engineering Team Principles that I have read(refer resources section) on the web and Engineering Team Principles that I have discovered while building systems.
Principle #1: Strive for simplicity
Most software over time tends to become complex. The complexity in software is of two types: essential complexity and accidental complexity. Essential complexity is a property of the problem you are trying to solve. Accidental complexity is the complexity that we bring into the software while building it. We can’t reduce essential complexity but our goal must be to keep accidental complexity to minimum.
Simplicity is the opposite of complexity. Simple code is well organized, logically minimal, and readable.
We prefer readability over cleverness. The code we write captures the essence of the problem domain and uses the language of the domain.
We write code for humans. The code that we write will be owned by a team and not just by the author. Hence, simple and readable code is preferred.
We apply simplicity to not only the code we write but also when we debug problems. We write simple possible tests to reproduce the problem and then solve the problem using tests as the guide. Tests help us write code that is simple to understand.
Principle #2: Build iteratively in small increments
We build software iteratively in small increments:
- To keep code changes small so that we can do quality code reviews
- To keep risk minimal
- To deliver value to the customers faster
- To learn from the customer behavior and make informed decisions
Principle #3: Minimize technology variation
We put effort to minimize the variety of technologies in our system to drive efficiency, and, at the same time, have a managed program of innovation. We believe we only have limited innovation tokens and we should use them effectively.
To achieve this, we monitor current technologies in use, gravitate to those that most appropriately address the given challenges and selectively experiment with new technologies to find those that will improve our products.
Principle #4: Fix problems, even when they are not created by you
We fix not only the bug we are trying to solve but also the code around it. We respect individual and team boundaries, but when we see bad things, we make an effort to work with others to resolve them.
- We improve README by adding certain missing steps/details.
- We build small tools that can save time for others.
- We apply the boy scouts rule, “Always leave the campground cleaner than you found it” to keep improving things each day. We take collective responsibility for our code.
These small improvements compound over time.
Principle #5: Automate stuff that you do manually the third time
We believe automation is the key to becoming a productive and efficient software developer. If we do something multiple times manually, then we find creative ways to automate it. We challenge the status-quo.
Automating repetitive tasks makes them less error prone and we achieve better predictability, reproducibility, resilience, and security.
The code we write for automation is of the same quality as the code we run in production. We keep improving and enhancing it.
Principle #6: Don’t reinvent the wheel
We try to build solutions on top of an existing solution. We create systems or tools that give us the business leverage so that we code what we absolutely need and maintain it for the longer term.
We understand that there is more cost involved in maintaining software than writing it. So, we favor existing solutions and prefer to build on top of existing solutions.
Principle #7: Ask in the open
We believe engineers learn best from each other. We promote our engineers to ask questions in group channels rather than one-on-one basis. This way they become comfortable sharing their views, opinions, doubts, and questions in open. This creates a healthy culture where more people are benefitted than the one who asked the question.
Principle #8: Document architecture and design decisions.
We document our decisions for better understanding, helping our future self, and helping others learn how we arrived at them. Design documents help us to learn from each other, get feedback, and deliver better products. We end up delivering better solutions because we build using the collective wisdom of the team.
We keep updating the wiki so that we can find answers when we need them.
Principle #9: Leverage your impact [1]
We believe in building once selling twice. We convert our local work impact to a larger communal impact.
Consider ways in which you can expand the scope of impact for your time:
- Avoid private messaging when you could communicate in a relevant public channel
- Internal presentations could be re-worked for public consumption
- Internal documents could be turned into blog posts
- Documentation on teams and processes could be open for others to refer and learn from
- A common reusable code artifact can be made open source
Principle #10: Think beyond running software on your developer machine
We think before we start coding and take input from different sources and then design the solution.
We just don’t build software on our local machine and hand others to run it. We understand how our code runs in the production. We look at monitoring dashboards to understand the performance and reliability of our software. This way we build continuous feedback loop that help us improve software over time. We add structured logs, comments, traces, and contextual information so that we can fix code when it fails at 3 A.M. [2]
We believe in testing our own code and act as our own QA. We write automated unit and integration tests.
Principle 11: Spend an afternoon each week to learn something new
We believe in continuous learning and experimentation. These two are essential for becoming a good and effective software engineer. We build expertise in tools and technologies that we use daily so that we are more productive.
We learn with curiosity. We keep a list of technologies we want to try handy and try them for a couple of hours each Friday. These broaden our minds and help us solve problems in innovative ways.
We understand that we can’t just use a shiny new project in our products. We evaluate them with rigor and understand that when building production level software we must be cautious of untested new technologies.
Principle #12: Use timeboxed spikes to do comparative analysis of technologies
We use experiments and write throwaway code to understand the problem better. We use timeboxed spikes as a means to evaluate technologies. For example, to choose between Golang and Java, we will build a small service in Java and Go and then compare them qualitatively. This helps us take data-driven decisions.
Principle #13: Apply Hanlon’s Razor
Never attribute to malice that which can be adequately explained by neglect.
We apply Hanlon razor to help us overcome the bias which comes when things go wrong. Most of us tend to blame the other person for wrong doing and assume they had malicious intentions. When we assume another person has malicious intent, we don’t give people the benefit of the doubt of things happening because of neglect.
We all make stupid mistakes but we never think that we made mistakes of bad intent. But, for others we don’t keep an open mind. Life will become much easier and we will have better relationships if we start giving people benefit of doubt in such situations. More often than not inability or neglect is the cause than malice.
Principle #14: Have bias for action
We know that just sitting on the problem will not solve the problem. We bias towards getting value into the hands of our customers quickly. We take actions and shake things up.
Principle #15: Strive for stateless, immutable, and idempotent components
We strive to build loosely coupled components that are:
- Stateless: A stateless process or application can be understood in isolation. There is no stored knowledge of or reference to past transactions. Each transaction is made from scratch for the first time. Stateless applications provide one service or function and use a content delivery network (CDN), web, or print servers to process these short-term requests.
- Immutable: An object is immutable if its state cannot be modified. Immutable things are automatically thread-safe, without requiring synchronization. Overall, immutability tends to result in fewer bugs and makes it easier to prove a program correct.
- Idempotent: An idempotent operation produces the same result even when it’s executed multiple times. This allows clients to safely retry operations in case of timeouts due to service processing or network failures.
Principle #16: If you can’t show it’s a bottleneck, don’t optimise it.[3]
Correctness is nearly always more important than performance. Because optimisation generally increases code complexity, only go after performance when you are sure a program works correctly and is running at a sufficiently large scale that the gains will be significant. Telemetry and profiling are the only sure ways to know where the bottlenecks are: intuition is often misleading.
Principle #17: Teach what you learn
We teach others what we know. We are not knowledge hoarders. We understand that we learn when we teach others. We use internal knowledge sharing sessions, blogging, conference talks, Teams chat, to share what we have learned. Knowledge grows when we share with others.
When we teach others, we empower them to make their own decisions and solve problems on their own. This also makes them teach others.
Principle #18: Unblock others
We help our team members when they are stuck. We consider this as a high value activity. We prioritize tasks like code reviews ahead of writing new features as it unblock others and reduce stress. We expect our engineers to ask questions in channels where multiple people can provide the answer instead of asking questions one-on-one.
Principles #19: Consistency matters
Consistency has a compounding effect on the software we write.
- We format our code in a consistent manner
- We follow coding standards and use static analysis and linting tools to enforce them
- We follow consistent process so that there is a less cognitive overhead required
Principle #20: Have opinions with open mind
We believe engineers need to have an opinion. We share our opinion freely but we don’t treat it as gospel. We have an open mind to change our opinions based on data and new information. We consider it healthy to challenge each other’s opinions. This helps us grow and improve our decision making process.
Principle #21: Engineering is all about trade-offs
There is no perfect solution, we make trade offs all the time. Sometimes the trade offs are obvious, but sometimes they are layers away from the thing we can see in front of us. Always think about where the trade offs could be, if they’re not immediately obvious.
Shekhar Gulati 
Excellent read. Thanks Shekhar.