Welcome to the third post of lessons I learnt(LIL) series. I had a good week and I am content with what I achieved. Telling yourself repeatedly that life is not a race and you can take your time and be at peace with yourself is a powerful feeling. There were two thoughts that repeatedly came to my mind this week that I wanted to share with you this week.
A couple of weeks back I learnt about DuckDB while going over DB Weekly newsletter. It immediately caught my attention as I was able to quickly understand why need for such a database exist. Most developers are used to working with an embedded file based relational database in their local development environment. Most popular choice among embeddable RDBMS is SQLite. Developers use embeddable databases because there is no set up required and they can get started quickly in a couple of minutes. This enables quick prototyping and developers can quickly iterate on business features.
DuckDB is similar to SQLite in the sense it is also designed to be used as an embeddable database. Developers can easily include it as a library in their code and start using it. Later in this post, I will cover how we can use DuckDB with Python.
If you are building REST APIs you will understand that HTTP status codes at times are not sufficient to convey enough information about an error. There is a specification RFC 7807 that defines how you can return error information back to the client. The way it works is by identifying a specific type of problem with a URI.
To make it more clear let’s suppose you are building a todo list application where you want to build API to move items from one list to other. If the target list has a cap on maximum items it can hold and you try to move an item from source list to target list the list then you should get an exception.
If all you have is HTTP status code then you can just return HTTP status code 403 forbidden. However, it does not give the API client enough information about why the request was forbidden. Different developers have solved this problem differently by building their application specific error objects. Problem Details for HTTP APIs specification solves this problem by defining a standard format in which API error response should be returned.
Welcome to the second post of lessons I learnt(LIL) series. This week I went from being stressed, to meh, to content, and finally to the state of happiness. It is immensely powerful to know your feelings so that you can take corrective actions if required. Each week give us an opportunity to correct ourselves, learn from our mistakes, and brings a ray of hope that we can do better.
I find hope in the darkest of days, and focus in the brightest. I do not judge the universe. – Dalai Lama
This week I only have one lesson to share with you. I thought multiple times about in the last one week and I think if we can master it we can build great teams.
Few years back I was working on an application where I had to pull data from an event table(populated using database triggers) and update my application specific data stores . This is a common problem that most software web developers need to solve. At that time I was not aware that this problem has a name. Sometime later I learnt that this is called Change Data Capture (CDC). As per wikipedia article on change data capture,
In databases, change data capture (CDC) is a set of software design patterns used to determine (and track) the data that has changed so that action can be taken using the changed data.
The key benefit of CDC is that you can identify the changed data in your source database which you can then incrementally apply to your target system. In absence of CDC, we are left with doing bulk loading of the data which is both time consuming and costly.
Since last three months I am running a course in my office where I am teaching junior developers how to write clean and maintainable code. In my experience, you can’t make people write clean and maintainable code until you make them write automated unit tests. For the purpose of this discussion, I don’t care much whether you write test first or last. The goal should be to write quality tests.
Writing unit tests is easy but writing clean and maintainable unit tests is difficult. Over time I have realised there are properties of the test that if satisfied can help you write maintainable tests. I will not cover FIRST(Fast, Isolated, Repeatable, Self-validating, and Timely) properties as they are covered at a lot of places.
Osquery is a an awesome host instrumentation framework from Facebook. It can instrument Mac, Linux, and Windows servers. It organises system data in tables that you can query using your favourite query language – SQL. It is SQL for your infrastructure. You can query for system intruders, system information, compliance, installed apps, running processes, and many more data points.
Osquery uses SQLite syntax for SQL. So, if you need more information about SQL syntax outside of what is covered in osquery documentation then you should give SQLite documentation a read.