Linux Tip: How to create a list of directories inside a directory and place an empty file in each of them

Today, I wanted to create a list of directories inside a directory and place an empty file in each of them. In this quick post, I will share two commands that can do that for you.

$ mkdir $(printf "%02i-day " $(seq 1 50))
$ find . -type d -exec touch {}/.gitkeep \;

You don’t have to type $. It is only to signify command-prompt.

Tools Software Engineers Should Know: 6 Things You Can Do with NetCat

In the second post of the tools series, I am going to cover netcat utility. It comes prepackaged on most of the linux distributions. The netcat utility is often referred to as a Swiss army knife of networking tools. It has a long list of features. Few of them are mentioned below.

  1. Outbound or inbound connections, TCP or UDP, to or from any ports
  2. Full DNS forward/reverse checking, with appropriate warnings
  3. Ability to use any local source port
  4. Built-in port-scanning capabilities

Continue reading

Mental Models for Software Engineers: Regret Minimization Framework

I became aware of Regret Minimization Framework quite recently while reading a random post on the web. With in that timeframe it has helped me take a couple of big decisions with clarity. Had I known this earlier I could have avoided taking few bad decisions.

Regret Minimization Framework is a mental model that wants you to project into the future and look back at your decision so that you can quantify which of the options could lead to minimal regret. Then, you should go with the option that will lead to minimum regret.

Continue reading

Issue #31: 10 Reads, A Handcrafted Weekly Newsletter For Software Developers

The time to read this newsletter is 200 minutes.

A liar will not be believed, even when he speaks the truth – Aesop

  1. How to remove duplicate lines from files keeping the original order: 15 mins read. Finally learnt something about awk. The post explains how you can remove duplicate lines in a file while preserving their order. This deduplication on steroids. It is in my todo list to learn awk one day.
  2. Google’s Chrome Becomes Web ‘Gatekeeper’ and Rivals Complain: 15 mins read. I have read this multiple times. Chrome is at the core of Google’s digital strategy. Google needs to track us to show ads and make money. This is the reason they are coming up with updated Chrome Extension API that will limit what ad blockers can do. In my view, the big problem is not Chrome or Google. We have ads because people want to earn money from their content. Google does not put ads magically; site owners add Google ad tracking scripts that share information with Google. Till the time, we don’t create a better financial model for content creators. This problem can’t be solved. Brave browser by Brendan Eich, co-founder of Mozilla and the current CEO of Brave Software Inc. is trying to do some work on it but it is still early days for it.
  3. Tests that sometimes fail: 30 mins read. Author makes following valid points:
    1. Flaky tests are useful at finding underlying flaws in our application. In some cases when fixing a flaky test, the fix is in the app, not in the test
    2. Common patterns of flaky tests
      1. Flaky tests caused by hard coded ids because they rely on database sequences
      2. Making bad assumptions about DB ordering. Result returned by SQL query is unordered.
      3. Incorrect assumptions about time
      4. Bad assumptions about the environment
    3. Mitigation patterns
      1. Run test suite in a tight loop, over and over again on a cloud server. Each time tests fail we flag them and at the end of a week of continuous running we mark flaky specs as “skipped” pending repair.
      2. One big issue with flaky tests is that quite often they are very hard to reproduce. To accelerate a repro I tend to try running a flaky test in a loop.
      3. Invest in fast test suite
      4. Add purpose built diagnostic code to debug flaky tests you can not reproduce
  4. You need neither PWA nor AMP to make your website load fast: 10 mins read. Author writes, “why was AMP needed? Well, basically Google needed to lock content providers to be served through Google Search. But they needed a good cover story for that. And they chose to promote it as a performance solution”. I kind of agree with author that AMP hurts the web community more than it helps. I have disabled AMP in my blog.
  5. Fast key-value stores: An idea whose time has come and gone: 30 mins read. Interesting paper by Google on building stateful services instead of stateless. I also went with stateful service architecture in my last application. It has its own challenges but in some cases it is the only viable option.
  6. 6 new ways to reduce your AWS bill with little effort: 10 mins read. This post can help you save some $$$ in your monthly AWS bill. The author suggests 6 ways we can reduce AWS bill. Out of the 6, I found following two ways worth a try:
    1. Use EC2 AMD instances
    2. Use VPC endpoints instead of NAT gateways
  7. Disaster Tolerance Patterns Using AWS Serverless Services: 30 mins read. Just read it if you are using AWS.
  8. How Far Out is AWS Fargate?: 15 mins read. This is a good post comparing AWS Fargate and AWS Lambda.
    1. With Lambda you pay per invocation and the price is based on the memory you allocate for your function (up to 30GB) and its execution time. The amount of compute available to your Lambda function is based on it’s memory allocation. This pricing model is ideal for workloads that have spikes and/or long periods of downtime.
    2. Fargate, on the other hand, lets you configure how many VCPUs (up to 8) and GBs of memory (up to 3GB) you want your Fargate tasks to have independently, priced by the secondrounded up to one minute.
  9. Learning to Listen to one’s own Boredom: 15 mins read. All of us need to learn to develop a ‘late style’ – ideally as early on in our lives as possible: a way of being wherein we shake off the dead hand of habit and social fear and relearn to listen to what entertains us
  10. How We Built a Content-Based Filtering Recommender System For Music with Python: 30 mins read. I love these kind of tutorial that help you learn by building an application in step by step manner. Give it a try and you will learn something about building a content-based recommender system for music.

Video of the week

Tools Software Engineers Should Know: MTR: A network diagnostic tool

We software developers are good at debugging code related issues but when it comes to issues that require fighting with infrastructure or network then we find ourselves in a difficult position. We can solve these issues if we know the right tool to use. I faced a similar position this week. I am starting a new series where every week or two I will write about a new tool that can help us debug these kind of issues.

This week I was debugging an issue where few requests to the destination server were timing out. These types of issues typically fall under networking errors and require you to use a network diagnostic tool. Most developers start diagnosing the issue using ping and traceroute tool. Both these tools are useful but you have to run them both together to debug the issue. Recently, I discovered MTR which combines ping and traceroute tools in a single tool. I found that most developers that I work with are not unaware of this tool so I decided to document it for future me and others.

mtr stands for My traceRoute. It is useful when you need to figure out number of hops to the destination server or latency at each hop. It also help you see packet loss at each hop so that you can narrow down the place where you might be facing issue. MTR collects information regarding the state, connection, and latency of the intermediate hosts. Thus giving a complete overview of the connection between two hosts on the network.

Continue reading

Building Copy as Plain Text Google Chrome Extension in 15 mins

Today, I wanted to make it easy for me to copy text from the web in plain text format. I read a lot of stuff on the web. When I find good articles I take notes and store them in Evernote. Evernote provides a rich text editor to compose notes. The default behaviour provided by Chrome is to copy the text along with the page style. This is not what I want most of the time. To avoid this, I have to first copy the text in a plain editor like Notepad or browser address bar and then copy it again and paste in Evernote. I am doing this for long time. I thought there has to be a better way. It came to my mind that this can be easily solved by writing a Google Chrome Extension. A Google Chrome extension that adds copy as plain text context menu option.

Continue reading

Mental Models for Software Engineers: Occam’s Razor

Occam’s Razor helps us choose between two or more explanations of a problem. It provides a useful mental model for problem-solving. A razor is a principle or rule of thumb that allows one to eliminate unlike explanations for a phenomenon, or avoid unnecessary actions.

One popular definition of Occam’s Razor is:

If we face two possible explanations which make the same predictions, the one based on the least number of unproven assumptions is preferable, until more evidence comes along.

This mental model help us look for explanations that are least complicated. It does not mean explanation has to be easy so that anyone can grasp it with limited effort but it means explanation can be logically reasoned without making too many assumptions.

Continue reading