Last week I had to create a visualisation for organization entity chart. The complexity was that an entity can be created by merging of two entities. So, I was looking for a solution that gives the flexibility to have two parent nodes. I started with D3.js but quickly figured out that tree charts in D3.js can’t have two parent nodes. After a lot of googling and trying many different libraries I decided to create that using Graphviz. The only problem was that I wanted to do it on the client side.
Below is the chart that I wanted to create. As you can see below – Dummy Org ABC Limited has two parents Dummy Org India Ltd and Dummy Org East Africa DMCC. Similarly, Dummy Org Devs Project has two parents Dummy Org Infrastructure Limited and Dummy Org Infrastructure & Dummy Org Project
I am doing software development for the last 15 years. Since I heard the term “Full Stack Developer” few years back I had a strong dislike for this term. To most people full stack developer is someone who can write frontend code, build backend APIs, and deploy applications to cloud. They can code in multiple programming languages(JS/Typescript, Golang/Java/C#/Python), knows enough to design and store data in either relational and NoSQL databases, follow DevOps practices (CI, Configuration management, etc.), knows about multiple architecture styles – Microservices, Monolithic, Serverless, and can at least deploy to one public cloud. Along with all this they also know and practice automation testing and can write clean, maintainable code.
The 1% rule states that within an internet community only 1% of the users of a website add content, while the other 99% of the participants only lurk. Another variant of this law is 90-9-1 rule which states that in a collaborative website such as a wiki, 90% of the participants of a community only consume content, 9% of the participants change or update content, and 1% of the participants add content. – Link
The broken windows theory is a criminological theory that states that visible signs of crime, anti-social behavior, and civil disorder create an urban environment that encourages further crime and disorder, including serious crimes. Social psychologists and police officers tend to agree that if a window in a building is broken and is left unrepaired, all the rest of the windows will soon be broken. This is as true in nice neighborhoods as in rundown ones. This theory has been applied to software development, suggesting that poor quality code (or Technical Debt) can lead to a perception that efforts to improve quality may be ignored or undervalued, thus leading to further poor quality code. This effect cascades leading to a great decrease in quality over time. – Link
Today, I was working on an application that uses Microservices based architecture. Each Microservice had its own schema and they talk to each other using their published contracts.
We wanted to keep database migration script with each Microservice rather than keeping a common module to manage database Migrations. We might later migrate to common module for database migration but we want to start by keeping schema for each Microservice separate.
Since last couple of years I have not learnt any new programming language so this week I decided to change that. I decided to learn Swift programming language.
Swift is a modern, multi-paradigm, compiled, statically typed, safe, open-source programming language by Apple. Most developers use it to build apps for Apple platforms – macOS, iOS, watchOS, tvOS and beyond.
I don’t build Apple apps for living. For most of software engineering life I have built web applications and backend services. So,the best way for me to learn a new language is to build a web application using it. In this tutorial series I will be building a web application from scratch using Swift. We will be using Vapor web framework. It is the most used web framework for Swift. Vapor is non-blocking and event-driven built on top of Apple’s SwiftNIO.
Below is the list of management and leadership lessons that I find useful. I collected them from various books and articles I read in the last few years. It is a work in progress so I will keep updating this list as I learn more on this subject.
The greatest leader is not necessarily the one who does the greatest things. He is the one that gets the people to do the greatest things.
Today, I was helping a team that was getting a database deadlock in their Java Spring Boot application using Microsoft SQL Server. The stacktrace of the error was as shown below.
Exception is org.springframework.dao.CannotAcquireLockException: could not execute query; SQL ; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute query] with root cause
com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 132) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
The database deadlock was caused when user uploads a file that was being written to the database. While the file was being uploaded user navigated to another page where data from the same table was queried.
When we debugged SQL Server traces we discovered that row-level lock was escalated to table lock. This was causing SQL Server to stop other transactions from accessing the table.
To solve this issue we enabled READ_COMMITTED_SNAPSHOT option for the database so that it uses READ_COMMITTED with row versioning isolation strategy. The default database isolation level in SQL Server is READ_COMMITTED which means a query in the current transaction cannot read data modified by another transaction that has not yet committed, thus preventing dirty reads. The isolation level uses shared locking or row versioning to prevent dirty reads, depending on whether the READ_COMMITTED_SNAPSHOT database option is enabled. Since wee enabled READ_COMMITTED_SNAPSHOT SQL Server will use versioning instead of locks.
ALTER DATABASE <database name>
SET READ_COMMITTED_SNAPSHOT ON
WITH ROLLBACK IMMEDIATE;
These days I am working on a multi-tenant SaaS application where we are using SQL Server as our main transactional database. Using SQL Server is fine when you run the application but for unit and integration tests you want to use an in-memory database for quick turn around, better isolation, free from any external service to be available. Last thing you want is developers to not write tests because they take too much time to run. I love the experience of running a build on a clean machine and it works without any setup or configuration.
Lately I have to use SQL Server for a Microservices based system that I am currently building. This is the first time in my 15 years of software development life I had to use SQL Server. I was looking for a tool that can help me generate database schema from the already created database schema. I wanted to do store schema in my version control system. The usual tool to do this kind of work is SQL Server Management Studio but it works only on Windows operating system. Since I use Mac I was looking for a tool that is cross platform. My search led me to discover mssql-scripter, a multi-platform command line experience for scripting SQL Server databases.
This week I had to work on a performance issue. Performance issues are always fun to work with. They give an opportunity to get into the depth of the technology we are using. We learn how much we don’t know about the technology we are using everyday. These days we too quickly think about changing the database or underlying library when faced with the performance bottleneck. When what we really need to do is learn about the technology we are using in depth.
The performance issue I am talking about was related to bulk insertion of data in to the database. We were using Spring Data JPA with SQL Server. In our use case, end user was uploading an excel file that the application code first parse, then process, and finally store in the database. The NFR for this requirement was that we should be able to process and store 100,000 records in less than 5 minutes.
Before my changes we were processing 10,000 records in 47 minutes. This certainly looked bad.
After making the changes that I will discuss in this post we were able to process 10,000 records in 20 seconds. We were able to process 100,000 in less that 4 minutes which is well below our NFR.