TIL #3: Using xbar to build ArgoCD deployment monitor

This week I was going over the latest edition(Volume 27) of Thoughtworks Technology Radar and found the addition of xbar in their Tools section. xbar lets you put output from any script/program in your macOS menu bar. I first wrote about it in October 2021 when I showed how you can use it show WordPress page views analytics in your macOS menu bar.

From the Thoughtworks Radar entry on xbar

On remote teams, we sorely lack having a dedicated build monitor in the room; unfortunately, newer continuous integration (CI) tools lack support for the old CCTray format. The result is that broken builds aren’t always picked up as quickly as we’d like. To solve this problem, many of our teams have started using xbar for build monitoring. With xbar, one can execute a script to poll build status, displaying it on the menu bar.

Continue reading “TIL #3: Using xbar to build ArgoCD deployment monitor”

TIL #2: Kafka poison pill message and CommitFailedException

Yesterday I was working with a team that was facing issue with their Kafka related code. The Kafka consumer was failing with the exception

[] ERROR [2022-11-22 08:32:52,853] com.abc.MyKakfaConsumer: Exception while processing events
! java.lang.NullPointerException: Cannot invoke "org.apache.kafka.common.header.Header.value()" because the return value of "org.apache.kafka.common.header.Headers.lastHeader(String)" is null
! at com.abc.MyKakfaConsumer.run(MyKakfaConsumer.java:83)
! at java.base/java.lang.Thread.run(Thread.java:833)

The consumer code looked like as shown below.

Continue reading “TIL #2: Kafka poison pill message and CommitFailedException”

TIL #1 : Using liquibase validCheckSum to solve a deployment issue

Taking inspiration from Simon Willison[1] I will start posting TIL (Today I learned) posts on something new/interesting I learn while building software. Today, I was working with a colleague on a problem where in our database migration script was working in the dev environment but failing in the staging environment. The customer platform team has mandated that we can’t access database directly and the only way to fix things is via liquibase scripts. In this post I will not discuss if I agree with them or not. That’s a rant for another day.

In our staging environment we were getting following exception

changelog-main.xml::2::author1 was: 8:a67c8ccae76190339d0fe7211ffa8d98 but is now: 8:d76c3d3a528a73a083836cb6fd6e5654
changelog-main.xml::3::author2 was: 8:0f90fb0771052231b1ax45c1x8bdffax but is now: 8:a25ca918b2eb27a2b453d6e3bf56ff77

If you have worked with Liquibase or any other similar database migration tool you will understand that this happens when developer has changed an existing changeset. This causes checksum to change for an existing changset. So, when next time liquibase tries to apply changset it gives validation error and fails.

Developer should never change an existing changeset and this is one thing we make sure we don’t miss during our code reviews.

Continue reading “TIL #1 : Using liquibase validCheckSum to solve a deployment issue”

Sending Outlook Calendar Invite using Java Mail API

Today I had to implement a functionality related to sending Outlook Calendar invite. The app backend was written in Java. As it turns out the simple task of sending a calendar invite is much more complicated than I expected. You have to construct the message with all the right parameters set else your calendar invite will not behave as you expect. I wasted an hour or two figuring out why RSVP buttons are not coming in the invite. As it turned out it was one of the missing parameters that caused the issue. Also, I wanted to send calendar invite to both outlook and Google calendar.

Continue reading “Sending Outlook Calendar Invite using Java Mail API”

Building a Sentiment Analysis Python Microservice with Flair and Flask

Flair delivers state-of-the-art performance in solving NLP problems such as named entity recognition (NER), part-of-speech tagging (PoS), sense disambiguation and text classification. It’s an NLP framework built on top of PyTorch.

In this post, I will cover how to build sentiment analysis Microservice with flair and flask framework.

Continue reading “Building a Sentiment Analysis Python Microservice with Flair and Flask”

How Docker uses cgroups to set resource limits?

Today, I was interested to know how does Docker uses cgroups to set resource limits. In this short post, I will share with you what I learnt.

I will assume that you have a machine on which Docker is installed.

Docker allows you to pass resource limits using the command-line options. Let’s assume that you want to limit the IO read rate to 1mb per second for a container. You can start a new container with the device-read-bps option as shown below

$ docker run -it --device-read-bps /dev/sda:1mb centos

In the above command, we are instantiating a new centos container. We specified device-read-bps option to limit the read rate to 1mb per second for /dev/sda device.

Continue reading “How Docker uses cgroups to set resource limits?”

TIL #8: Installing and Managing PostgreSQL with Brew

This post documents how to install and manage PostgreSQL using brew package manager for macOS.

To install PostgreSQL using brew, run the following command.

$ brew install postgresql

If you wish to start and stop PostgreSQL, then you do that using following brew commands.

$ brew services stop postgresql
$ brew services start postgresql

Create a database for your username

createdb `whoami`

To fix the error role "postgres" does not exist, run the following command

createuser -s postgres

Now, you will be able to log into psql using both your username and postgres

psql
psql -U postgres

To get information about your PostgreSQL installation, you can run following brew command.

brew info postgresql

There are time you might forget if you installed PostgreSQL using brew. You can check if PostgreSQL was installed by brew by running following command.

brew list |grep postgres

The brew package manager installs PostgreSQL under following location.

/usr/local/Cellar/postgresql/

The configuration files as inside following directory.

/usr/local/var/postgres

TIL #7: Java Lambda Puzzler

Today, a colleague asked me how they can pass a java.util.Stream to a function that accept an java.lang.Iterable.

Let’s suppose we have a following function that accepts an Iterable.

public static void doSth(Iterable<String> iterable){
        iterable.forEach(System.out::println);
 }

The calling function has a Stream that it want to pass to the doSth function.

public static void main(String[] args) throws IOException {
        Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
}

One way we could easily achieve this is by collecting the Stream into a Collection like List. As List is an Iterable so we can pass it. This is should below

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(lines.collect(Collectors.toList()));

This works but what if Stream is big and collecting into an in-memory data structure like List leads to java.lang.OutOfMemoryError: Java heap space.

I googled around and found a nice Lambda hack.

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(lines::iterator);

I have worked a lot on Java 8 but first time I looked at it I couldn’t figure out how it works. If you know Java 8, give it a shot.

The magic here is that Iterable interface has a single abstract method iterator.

public interface Iterable<T> {
    Iterator<T> iterator();
}

This means we can write it as following Lambda function.

Iterable<T> iterable = () -> iterator();

In our case, Stream has an iterator method. So, we can convert Stream to Iterable by defining a lambda function as shown below.

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(() -> lines.iterator());

You can refactor the Lambda to a method reference.

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(lines::iterator);

Can we use () -> iterator() in the for-each loop

I wondered if we can use the lambda expression in the for-each loop

for (String str : () -> lines.iterator()) {
    System.out.println(str);
}

This looks like a valid use case. As for-each loop works with types that implement Iterable interface. But, it turns out the code does not compile. It gives Lambda expression not implemented here error.

The answer for this is mentioned in the JSR335

Deciding what contexts are allowed to support poly expressions 
is driven in large part by the practical 
need for such features:

The expression in an enhanced for loop is not in a 
poly context because, as the construct is currently defined, 
it is as if the expression were a receiver: 
exp.iterator() (or, in the array case, exp[i]). 
It is plausible that an Iterator could be wrapped 
as an Iterable in a for loop via a 
lambda expression (for (String s : () -> stringIterator)), 
but this doesn't mesh very well with the semantics of Iterable.

Another Interesting thing to note is that Iterable is not marked as @FunctionalInterface. I don’t know the exact reason why Iterable is not marked as @FunctionalInterface. My guess is that because Iterable has special semantics in Java so they didn’t explicitly marked it @FunctionalInterface.

TIL #6: Top 10 Commands That You Use On Your Command-line Terminal

I found a command that you can use to list down top commands that you use in your terminal

$ history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl |  head -n10

Following is my list

     1  1244  12.4412%   cd
     2  1056  10.5611%   git
     3  827   8.27083%   docker
     4  575   5.75058%   ls
     5  523   5.23052%   gdw
     6  447   4.47045%   docker-compose
     7  378   3.78038%   pass
     8  221   2.21022%   gs
     9  187   1.87019%   open
    10  184   1.84018%   rm

gdw is shortcut for Gradle wrapper.

TIL #5: Representing Response Object in REST API

I was thinking what is the best way to represent response object in REST API. So, I started thinking about all the things response object should provide to work effectively. Below is the list of things I want my response object to have:

  1. A single object structure to handle both successful and error scenarios
  2. Ability to handle multiple types of objects
  3. Ability to easily figure out if request failed or succeeded
  4. Store the error code and error message

The code in this post uses Java as the language of choice. As I use Spring Boot to build REST APIs so Jackson is the default JSON serialisation library.

The code below shows the ApiResponse object that captures both the success and error payload.

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonValue;

public final class ApiResponse<T> {

    private ErrorResponse error;
    private T data;
    private Status status;

    private ApiResponse(Status status, ErrorResponse error) {
        this.error = error;
        this.status = status;
    }

    private ApiResponse(Status status, T data) {
        this.data = data;
        this.status = status;
    }

    @JsonCreator
    public static <T> ApiResponse<T> success(
            @JsonProperty("status") Status status,
            @JsonProperty("data") T data) {
        return new ApiResponse<>(status, data);
    }

    @JsonCreator
    public static <T> ApiResponse<T> error(
            @JsonProperty("status") Status status,
            @JsonProperty("error") ErrorResponse error) {
        return new ApiResponse<>(status, error);
    }

    public enum Status {
        SUCCESS("success"), ERROR("error");

        private final String status;

        Status(String status) {
            this.status = status;
        }

        @JsonValue
        public String getStatus() {
            return this.status;
        }
    }

    public ErrorResponse getError() {
        return this.error;
    }

    public T getData() {
        return this.data;
    }

    public Status getStatus() {
        return this.status;
    }
}

The key points in the class shown above are:

  1. Status enum will hold the status of the request. User can just look at this request to figure out whether request succeeded or failed.
  2. The class is generic so it can be used to store any type of object in the data.
  3. User can get more details about the error from the error object.

The ErrorResponse class is shown below.

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

public final class ErrorResponse {
    public final String code;
    public final String message;

    @JsonCreator
    public ErrorResponse(
            @JsonProperty("code") String code,
            @JsonProperty("message") String message) {
        this.code = code;
        this.message = message;
    }
}

The ErrorResponse class has two fields code and message. The code can be HTTP code or any business specific code. The message variable gives details about the error.

I found the above response classes work great for the REST APIs that I recently built.

Example of successful response is shown below.

{
    "status": "success",
    "data": {
        "task": "Write a post",
        "taskStatus":"in_progress",
        "tags":["writing"]
    }
}

Example of failure response is show below.

{
    "status": "error",
    "error": {
        "code" "409",
         "message" : "User with username xyz already exists"
    }
}

One thing that you should ensure is that your JSON serialisation library exclude null values. For example, in case of error scenario you don’t want data field to be null. If you are using Spring Boot, you can instruct Jackson to exclude null fields by specifying a property shown below.

spring.jackson.default-property-inclusion=NON_NULL

If you are using Java 8 or Google’s Gauva and want to exclude Optional type as well then you should use NON_ABSENT value.

spring.jackson.default-property-inclusion=NON_ABSENT