TIL #7: Java Lambda Puzzler

Today, a colleague asked me how they can pass a java.util.Stream to a function that accept an java.lang.Iterable.

Let’s suppose we have a following function that accepts an Iterable.

public static void doSth(Iterable<String> iterable){
        iterable.forEach(System.out::println);
 }

The calling function has a Stream that it want to pass to the doSth function.

public static void main(String[] args) throws IOException {
        Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
}

One way we could easily achieve this is by collecting the Stream into a Collection like List. As List is an Iterable so we can pass it. This is should below

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(lines.collect(Collectors.toList()));

This works but what if Stream is big and collecting into an in-memory data structure like List leads to java.lang.OutOfMemoryError: Java heap space.

I googled around and found a nice Lambda hack.

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(lines::iterator);

I have worked a lot on Java 8 but first time I looked at it I couldn’t figure out how it works. If you know Java 8, give it a shot.

The magic here is that Iterable interface has a single abstract method iterator.

public interface Iterable<T> {
    Iterator<T> iterator();
}

This means we can write it as following Lambda function.

Iterable<T> iterable = () -> iterator();

In our case, Stream has an iterator method. So, we can convert Stream to Iterable by defining a lambda function as shown below.

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(() -> lines.iterator());

You can refactor the Lambda to a method reference.

Stream<String> lines = Files.lines(Paths.get("src", "main", "resources", "names.txt"));
doSth(lines::iterator);

Can we use () -> iterator() in the for-each loop

I wondered if we can use the lambda expression in the for-each loop

for (String str : () -> lines.iterator()) {
    System.out.println(str);
}

This looks like a valid use case. As for-each loop works with types that implement Iterable interface. But, it turns out the code does not compile. It gives Lambda expression not implemented here error.

The answer for this is mentioned in the JSR335

Deciding what contexts are allowed to support poly expressions 
is driven in large part by the practical 
need for such features:

The expression in an enhanced for loop is not in a 
poly context because, as the construct is currently defined, 
it is as if the expression were a receiver: 
exp.iterator() (or, in the array case, exp[i]). 
It is plausible that an Iterator could be wrapped 
as an Iterable in a for loop via a 
lambda expression (for (String s : () -> stringIterator)), 
but this doesn't mesh very well with the semantics of Iterable.

Another Interesting thing to note is that Iterable is not marked as @FunctionalInterface. I don’t know the exact reason why Iterable is not marked as @FunctionalInterface. My guess is that because Iterable has special semantics in Java so they didn’t explicitly marked it @FunctionalInterface.

Issue #6: 10 Reads, A Handcrafted Weekly Newsletter for Humans

Hey y’all,
Here are 10 reads I thought were worth sharing this week. The total time to read this newsletter is 130 minutes.
Half of everything you hear in a classroom is crap. Education is figuring out which half is which. — Larrabee’s Law
  1. The 5 Lies We Love to Tell : 10 mins read. The author makes the point that we all lie to ourselves. We should stop fooling ourselves that we don’t lie. The biggest problem with lies is that they consume a lot of your mental power in the background. You have to expend your mental energy to keep reminding yourself what lie you made to your inner self so that you don’t deviate from it.
  2. Please Stop Using Adblock (But Not Why You Think) : 10 mins read.  The key point in this post is that Adblock is making a lot of money by making advertisers like Google to whitelist their ads. I was shocked to read the dark side of AdBlock. The author recommends that people use free and open source uBlock Origin.
  3. Tech’s Two Philosophies : 15 mins read. The author makes the point that there are two main philosophies in the tech industry. The first philosophy shared by Google and Facebook is that computers should do work for humans. The second philosophy shared by Microsoft and Apple is that computers empower humans and help them do their work efficiently.
  4. It’s about time to design in the real world. Introducing Hadron! : 10 mins read. Hadron is a tool aimed to make designing through code visual, fast and easy by embracing the web platform. Even though you will use code, the great thing is that not only very little writing is needed to get started, but also your designs can be progressively enhanced. Meaning that you can start designing with only simple HTML and CSS, and later make your design do more by adding behaviour through other Web Components or even writing JS yourself.
  5. The Economics of Writing a Technical Book: 15 mins read. This is a good post that will help you understand economics of writing a technical book. In my opinion, author made good money from writing his first book. Part of it has to do with the fact that he wrote for O’Reilly Media. Writing a book is tiring and cumbersome so kudos to the author on publishing his first book.
  6. Three-day no-meeting schedule for engineers: 5 mins read. It is great to see organisations understanding the need for focussed time. I strongly believe what Paul Graham wrote in his essay on Manager vs Maker schedule. Software development is a creative endeavour that requires an undistracted, peaceful environment for good work. I hope my organization also does the same one day.
  7. High availability and scalable reads in PostgreSQL : 20 mins read. A detailed primer on streaming replication, complete with performance measurements.
  8. How Postgresql analysis helped to gain 290 times performance boost for a particular request : 10 mins read. This is an interesting read as the guy tried many difficult solutions before figuring out a simple change to improve performance of his query by 290 times. Simple solutions are difficult to find. This post shows  how query and data model design minor mistakes can lead to performance bottlenecks and how extremely useful explain analyze command can be.
  9. Experiences with running PostgreSQL on Kubernetes : 20 mins read. Kubernetes is not aware of the deployment details of Postgres. A naive deployment could lead to complete data loss. Here’s a typical scenario when that happens. You set up streaming replication and let’s say the first master is up. All the writes go there and they asynchronously replicate to the standby. Then suddenly the current master goes down but the asynchronous replication has a huge lag caused by something like a network partition. If the naive failover leader election algorithm kicks in or the administrator who doesn’t know the state manually triggers failover, the secondary becomes the master. That becomes the source of truth. All of the data during that period is lost because all of the writes that were not replicated disappear. Whenever the admin recovers the first master it’s no longer the master any more and it has to completely sync the state from the second node which is now the master.

TIL #6: Top 10 Commands That You Use On Your Command-line Terminal

I found a command that you can use to list down top commands that you use in your terminal

$ history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl |  head -n10

Following is my list

     1  1244  12.4412%   cd
     2  1056  10.5611%   git
     3  827   8.27083%   docker
     4  575   5.75058%   ls
     5  523   5.23052%   gdw
     6  447   4.47045%   docker-compose
     7  378   3.78038%   pass
     8  221   2.21022%   gs
     9  187   1.87019%   open
    10  184   1.84018%   rm

gdw is shortcut for Gradle wrapper.

TIL #5: Representing Response Object in REST API

I was thinking what is the best way to represent response object in REST API. So, I started thinking about all the things response object should provide to work effectively. Below is the list of things I want my response object to have:

  1. A single object structure to handle both successful and error scenarios
  2. Ability to handle multiple types of objects
  3. Ability to easily figure out if request failed or succeeded
  4. Store the error code and error message

The code in this post uses Java as the language of choice. As I use Spring Boot to build REST APIs so Jackson is the default JSON serialisation library.

The code below shows the ApiResponse object that captures both the success and error payload.

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonValue;

public final class ApiResponse<T> {

    private ErrorResponse error;
    private T data;
    private Status status;

    private ApiResponse(Status status, ErrorResponse error) {
        this.error = error;
        this.status = status;
    }

    private ApiResponse(Status status, T data) {
        this.data = data;
        this.status = status;
    }

    @JsonCreator
    public static <T> ApiResponse<T> success(
            @JsonProperty("status") Status status,
            @JsonProperty("data") T data) {
        return new ApiResponse<>(status, data);
    }

    @JsonCreator
    public static <T> ApiResponse<T> error(
            @JsonProperty("status") Status status,
            @JsonProperty("error") ErrorResponse error) {
        return new ApiResponse<>(status, error);
    }

    public enum Status {
        SUCCESS("success"), ERROR("error");

        private final String status;

        Status(String status) {
            this.status = status;
        }

        @JsonValue
        public String getStatus() {
            return this.status;
        }
    }

    public ErrorResponse getError() {
        return this.error;
    }

    public T getData() {
        return this.data;
    }

    public Status getStatus() {
        return this.status;
    }
}

The key points in the class shown above are:

  1. Status enum will hold the status of the request. User can just look at this request to figure out whether request succeeded or failed.
  2. The class is generic so it can be used to store any type of object in the data.
  3. User can get more details about the error from the error object.

The ErrorResponse class is shown below.

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

public final class ErrorResponse {
    public final String code;
    public final String message;

    @JsonCreator
    public ErrorResponse(
            @JsonProperty("code") String code,
            @JsonProperty("message") String message) {
        this.code = code;
        this.message = message;
    }
}

The ErrorResponse class has two fields code and message. The code can be HTTP code or any business specific code. The message variable gives details about the error.

I found the above response classes work great for the REST APIs that I recently built.

Example of successful response is shown below.

{
    "status": "success",
    "data": {
        "task": "Write a post",
        "taskStatus":"in_progress",
        "tags":["writing"]
    }
}

Example of failure response is show below.

{
    "status": "error",
    "error": {
        "code" "409",
         "message" : "User with username xyz already exists"
    }
}

One thing that you should ensure is that your JSON serialisation library exclude null values. For example, in case of error scenario you don’t want data field to be null. If you are using Spring Boot, you can instruct Jackson to exclude null fields by specifying a property shown below.

spring.jackson.default-property-inclusion=NON_NULL

If you are using Java 8 or Google’s Gauva and want to exclude Optional type as well then you should use NON_ABSENT value.

spring.jackson.default-property-inclusion=NON_ABSENT

TIL #4: Downloading a zip from S3 to local directory using AWS CLI

Today, I had a need to download a zip file from S3 . I quickly learnt that AWS CLI can do the job. The AWS CLI has aws s3 cp command that can be used to download a zip file from Amazon S3 to local directory as shown below.

 $ aws s3 cp s3://my_bucket/myzip.zip ./

If you want to download all files from a S3 bucket recursively then you can use the following command

$ aws s3 cp s3://my_bucket/  ./ -- recursive

You can specify your AWS profile using the profile option shown below.

$ aws s3 cp s3://my_bucket/myzip.zip ./ -- profile test

To download all the files from a folder you can use following command:

$ aws s3 cp s3://my_bucket/my_folder  ./ -- recursive

You can also use include and exclude options to filter files based on wildcards. For example, let’s suppose you only want to download files with zip extension from a S3 bucket my_bucket then you can use the following command.

$ aws s3 cp s3://pcl-caps ./ --recursive --exclude "*" --include "*.zip"

There is another great option dryrun that you can use to see the actions that will be performed without running the command. In the above command, if we add —dryrun flag then we can see which all files will be downloaded to local directory.

$ aws s3 cp s3://pcl-caps ./ --recursive --exclude "*" --include "*.zip" --dryrun

Issue #5: 10 Reads, A Handcrafted Weekly Newsletter for Humans

Welcome to the fifth  issue of 10 Reads weekly newsletter. Below are the 10 posts that I found good to read this week. Total time to read this newsletter is 145 minutes.

The only true wisdom is in knowing you know nothing – Socrates

  1. Stack Overflow Isn’t Very Welcoming. It’s Time for That to Change. : 10 mins read: Stack Overflow has realized that new users are not finding it welcoming so they are taking steps to improve it. I think for me the bigger lesson is that when an organization like Stack Overflow can work towards recognising mistakes and work on fixing them why can’t other companies do the same. In most oragnizations,  senior management try to hide problems under the carpet never recognising their wrong doings. I am not expecting organisations to go public with their problems but instead their should be open communication.
  2. Is There A Fix For Impostor Syndrome15 mins read. Reading this post I realized I also suffer from impostor syndrome. The post author describes 5 kinds of impostor and their behaviour that she learnt from book The Secret Thoughts of Successful Women: Why People Suffer from Impostor Syndrome and How to Thrive Inspite of it.
    1. The Expert: This manifests as a state of cringing denial when called an expert.
    2. The Perfectionist: Perfectionism underlies a feeling that one could have (and should have) done better.
    3. The Superwoman/man: Some people can’t stop working, taking on every task they can. Young argues that this kind of workaholism is the expression of a need for external validation and can be countered only by focusing on setting one’s own metrics for personal success.
    4. The Natural Genius: This behavior involves judging one’s worth on the basis of raw ability as opposed to effort.
    5. The Rugged Individualist: Rugged individualism demands that all tasks be performed alone, and little to no help is sought.
  3. How To Get Your Coworker To Agree With You : 30 mins read. This is not a long read but I spent a lot of time thinking about times when I behaved arrogantly and tried to force people to think my way. Many times it just doesn’t work. The key points for me in this post are:
    1. If you say someone that they are wrong, they will become defensive straightaway and try hard to prove why their decision is correct. It does not matter if your intentions were good other person will not be able to see the point. The author suggests that we should avoid arguments at all cost. In one of the books that I read recently, author suggests The only way to get best of an argument is to avoid it.
    2. Before you expect others to listen to you build a strong relationship and trust. People will only listen if they consider you worth listening to. There is no magic solution here. You have to work hard on yourself before you can expect people to agree with you.
    3. Don’t force your decisions on others. Give them options & let them choose the best based on their expertise.
    4. Don’t distrust expertise of your coworkers. The wisest people are the ones who always empty their cups.
  4. Tips for High Availability by Netflix15 mins  read. In this post, Netflix engineers share how they are able to keep their services up. Netflix has built a continuous integration and delivery tool called Spinnaker that imbibes all the best practices Netflix engineers have learnt over the year. Some of the best practices that we can use to make our systems highly available are:
    1. Prefer regional deploys over global deploys
    2. Use Red/Black deployment strategy  for production deploys
    3. Use deployment windows
    4. Enable Chaos Monkey
    5. Know how to roll back your deploy quickly
  5. Why Our Brains Fall for False Expertise, and How to Stop It 30 mins read. If the people who offer the most valuable contributions to your organization aren’t appropriately recognized for it, they won’t stay long. Or, possibly worse, they will stay and stop trying.
  6. Rust in production at Figma : 10 mins read. The author mentions a lot of compelling reasons why people should consider Rust for writing performance sensitive back-ends. The main benefits of Rust are Low memory usage, Awesome performance, and Solid toolchain. Rust is new so there are rough edges as well. Read the post to learn about the pros and cons of using Rust.
  7. Bitcoin is the greatest scam in history :10 mins read: I am glad someone finally put it. Every other day I hear people talking about Bitcoin and BlockChain without understanding what they are getting into. Cryptocurrency is a scam because:
    1. Bitcoins are accepted almost nowhere, and some cryptocurrencies nowhere at all. Even where accepted, a currency whose value can swing 10 percent or more in a single day is useless as a means of payment.
    2. Extreme price volatility also makes bitcoin undesirable as a store of value. And the storehouses — the cryptocurrency trading exchanges — are far less reliable and trustworthy than ordinary banks and brokers.
    3. A bitcoin has no intrinsic value. It only has value if people think other people will buy it for a higher price — the Greater Fool theory.
  8. PostgreSQL 11 will finally have procedures5 mins read. Many people have asked for this feature for years and PostgreSQL 11 will finally have it. I am of course talking about CREATE PROCEDURE. Traditionally PostgreSQL has provided all the means to write functions (which were often simply called “stored procedures”). However, in a function you cannot really run transactions – all you can do is to use exceptions, which are basically savepoints. Inside a function you cannot just commit a transaction or open a new one. CREATE PROCEDURE will change all that and provide you with the means to run transactions in procedural code.
  9. pg_wal is too big… what’s going on?15 mins read. Here are two ways to go about it: the first – is to take an emergency action, this is the last resort when there is only 2% or less of free disc space, lack of time to find the cause of the problem and the main goal of your actions is to avoid the risk of the database crash halt. The second – is set of actions directed to find out the cause of the space consumption – something you can only do when there is no risk of the database’s emergency stop.
  10. Keep The Degree Of Difficulty Down5 mins read: So the better approach is to pick something simple to execute, nail it, then build on it with another relatively simple move, nail that too, and keep going.

TIL #3: Exclude null fields in Spring Boot REST JSON API, Serializing Enum value with Jackson, and Change remote for a branch in Git

The three things that I learned today are mentioned below.

Learning 1: Exclude null fields in Spring Boot REST JSON API response

Spring Boot uses Jackson to convert to JSON. Spring Boot allows you to configure through a configuration property whether you want to include null values or not. By default, serialised JSON will include null values as well. To remove null values, you should use following property add it to your application.properties.

spring.jackson.default-property-inclusion=NON_NULL

If you are using Java 8 or Google’s Gauva and want to exclude Optional type as well then you should use NON_ABSENT value.

spring.jackson.default-property-inclusion=NON_ABSENT

To learn about all the values, you should look at Jackson’s com.fasterxml.jackson.annotation.JsonInclude.Include enumeration.

Learning 2: Serializing Enum value with Jackson

The second learning that I had today was around how to properly serialize enum values in Jackson. I had enum shown below

public enum Status {
    SUCCESS("success"), ERROR("error");

    private final String status;

    Status(String status) {
        this.status = status;
    }
}

I wanted my JSON structure to be as shown below.

{
  "status": "success"
}

To achieve that you have to Jackson’s @JsonCreator and @JsonValue annotation as shown below.

public enum Status {
    SUCCESS("success"), ERROR("error");

    private final String status;

    @JsonCreator
    Status(String status) {
        this.status = status;
    }

    @JsonValue
    public String getStatus() {
        return this.status;
    }
}

Learning 3: Change remote of a branch in Git

$ git branch develop --set-upstream-to=upstream/develop

You can view remote tracked by local branch using the following command.

$ git branch -lvv