Using boto3 with Jython

Few days back I had a requirement that I had to use boto3 with Jython. boto3 is AWS EC2 python SDK that you can use to work with various Amazon Cloud API’s. Jython is the JVM implementation of Python. We were packaging our Jython scripts and boto3 and its dependencies inside a JAR. boto3 and Jython work great together when you use them in a normal way i.e. when boto3 can load its data model files from file system. This does not work when you package your script and its dependencies inside a JAR as the model files are then not available on the filesystem but are available on the classpath. In this blog, I will show you how we used boto3 to overcome this limitation. Continue reading

Finatra Tutorial — Build Beautiful REST API The Twitter Way

Finatra is an open-source project by Twitter that can be used to build REST APIs in Scala programming language. Finatra builds on top of Twitter’s Scala stack — twitter-server, finagle, and twitter-util.

  1. Finagle: It can be used to construct high performance servers.
  2. Twitter Server: It defines a template from which servers at Twitter are built. It uses finagle underneath.
  3. Twitter-Util: A bunch of idiomatic, small, general purpose tools for Scala.

In this step-by-step tutorial, we will cover how to build a Scala REST API using Finatra version 2. Finatra version 2 is a complete rewrite of finatra and is significantly faster(50 times according to documentation) than version 1.x.

This blog is part of my year long blog series 52 Technologies in 2016

You can read full blog here https://github.com/shekhargulati/52-technologies-in-2016/blob/master/01-finatra/README.md

JUnit Rule + Java 8 Repeatable Annotations == Clean Tests

Last couple of months I am spending most of my free time writing Docker Java REST API client using RxJava and OkHttp. I have been following TDD for developing this API. Some of the test cases in RxDockerClientTest have to first create a docker container and then they perform other operations. For example, shouldStartCreatedContainer test case will test that API can start a created container. Similarly, there are test cases that need a container. One common solution to achieve this is to use @Before setUp and @After tearDown methods that take care of creating a container before test case is executed and removing the container after test execution. The problem with this solution is that container will be created for every test case in the test class. I only wanted to create a container for test cases that need it. Continue reading

Let’s master Java 8 Date Time API

So far in this series we have focussed on functional aspects of Java 8 and looked at how to design better API’s using Optional and default and static methods in Interfaces. In this blog, we will learn about another new API that will change the way we work with dates — Date Time API. Almost all Java developers will agree that date and time support prior to Java 8 is far from ideal and most of the time we had to use third party libraries like Joda-Time in our applications. The new Date Time API is heavily influenced by Joda-Time API and if you have used it then you will feel home.

Continue reading

Write Infinite FizzBuzz Sequence in Java 8

One way to write FizzBuzz program in Java 8 using Stream API.

Write a program that prints the integers from 1 to 100. But for multiples of three print “Fizz” instead of the number, and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

import java.util.function.IntPredicate;
import java.util.function.Supplier;
import java.util.stream.IntStream;
import java.util.stream.Stream;

public class FizzBuzz {

    public static void main(String[] args) {
        IntStream numbers = IntStream.iterate(1, el -> el + 1);

        IntPredicate divBy3 = number -> number % 3 == 0;
        IntPredicate divBy5 = number -> number % 5 == 0;
        IntPredicate divBy3And5 = divBy3.and(divBy5);

        Stream<String> fizzBuzzStream = numbers
                .mapToObj(String::valueOf)
                .map(number -> parse(number, divBy3And5, () -> "FizzBuzz"))
                .map(number -> parse(number, divBy3, () -> "Fizz"))
                .map(number -> parse(number, divBy5, () -> "Buzz"));

        fizzBuzzStream.limit(30).forEach(System.out::println);
    }

    private static String parse(String numberStr, IntPredicate predicate, Supplier<String> supplier) {
        try {
            int number = Integer.parseInt(numberStr);
            if (predicate.test(number)) {
                return supplier.get();
            }
            return String.valueOf(number);
        } catch (NumberFormatException e) {
            return numberStr;
        }
    }
}

The above program will run into integer overflow. You can use BigInteger to overcome that as shown below.

import java.math.BigInteger;
import java.util.function.Predicate;
import java.util.function.Supplier;
import java.util.stream.Stream;

public class FizzBuzzBigInteger {

    public static void main(String[] args) {
        BigInteger one = BigInteger.ONE;
        Stream<BigInteger> numbers = Stream.iterate(one, el -> el.add(one));

        Predicate<BigInteger> divBy3 = number -> number.remainder(BigInteger.valueOf(3)) == BigInteger.ZERO;
        Predicate<BigInteger> divBy5 = number -> number.remainder(BigInteger.valueOf(5)) == BigInteger.ZERO;
        Predicate<BigInteger> divBy3And5 = divBy3.and(divBy5);

        Stream<String> fizzBuzzStream = numbers
                .map(String::valueOf)
                .map(number -> parse(number, divBy3And5, () -> "FizzBuzz"))
                .map(number -> parse(number, divBy3, () -> "Fizz"))
                .map(number -> parse(number, divBy5, () -> "Buzz"));

        fizzBuzzStream.limit(100).forEach(System.out::println);
    }

    private static String parse(String numberStr, Predicate<BigInteger> predicate, Supplier<String> supplier) {
        try {
            BigInteger number = new BigInteger(numberStr);
            if (predicate.test(number)) {
                return supplier.get();
            }
            return numberStr;
        } catch (NumberFormatException e) {
            return numberStr;
        }
    }
}

The better and more functional approach suggested(in the comments) by Dominic is shown below.

import java.math.BigInteger;
import java.util.Objects;
import java.util.Optional;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.function.Supplier;
import java.util.stream.Stream;

public class Fizzbuzz {

    public static void main(String[] args) {
        BigInteger one = BigInteger.ONE;

        Predicate<BigInteger> divBy3 = number -> Objects.equals(number.remainder(BigInteger.valueOf(3)), BigInteger.ZERO);
        Predicate<BigInteger> divBy5 = number -> Objects.equals(number.remainder(BigInteger.valueOf(5)), BigInteger.ZERO);
        Predicate<BigInteger> divBy3And5 = divBy3.and(divBy5);

        Stream<BigInteger> numbers = Stream.iterate(one, el -> el.add(one));

        Stream<String> fizzBuzzStream = numbers
                .map(FizzBuzzMatcher.matching(divBy3And5, () -> "FizzBuzz")
                        .otherwise(FizzBuzzMatcher.matching(divBy3, () -> "Fizz")
                                .otherwise(FizzBuzzMatcher.matching(divBy5, () -> "Buzz")
                                        .otherwise(BigInteger::toString))));

        fizzBuzzStream.limit(100).forEach(System.out::println);
    }
}

interface FizzBuzzMatcher extends Function<BigInteger, Optional<String>> {

    static FizzBuzzMatcher matching(Predicate<BigInteger> matcher, Supplier<String> resultMsgSupplier) {
        return number -> matcher.test(number) ? Optional.of(resultMsgSupplier.get()) : Optional.empty();
    }

    default Function<BigInteger, String> otherwise(Function<BigInteger, String> next) {
        return number -> apply(number).orElseGet(() -> next.apply(number));
    }
}

Docker REST API wget request and responses

This blog will list Docker wget requests. This assumes you are using boot2docker with https connection.

Create an Image

POST /images/create

$ wget --method=POST --header="Content-Type:application/json" --no-check-certificate 
--certificate=$DOCKER_CERT_PATH/cert.pem --private-key=$DOCKER_CERT_PATH/key.pem 
https://192.168.99.100:2376/images/create\?fromImage\=busybox -O - -v

Response

--2015-10-03 14:15:07--  https://192.168.99.100:2376/images/create?fromImage=busybox
Connecting to 192.168.99.100:2376... connected.
WARNING: cannot verify 192.168.99.100's certificate, issued by '/O=shekhargulati':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: 'STDOUT'

-                                                [<=>                                                                                            ]       0  --.-KB/s             {"status":"Pulling from library/busybox","id":"1-ubuntu"}
{"status":"Already exists","progressDetail":{},"id":"6003abefd7b2"}{"status":"Already exists","progressDetail":{},"id":"8171cf9d0131"}{"status":"Digest: sha256:e51c3b513c0b04603c32d6961858c5d380c94c1eb03ad8f66685ef3ddf280114"}
{"status":"Pulling from library/busybox","id":"1.21-ubuntu"}
-                                                [ <=>                                                                                           

List images with dangling true filter

GET /images/json

Docker REST API documentation does not give an example of how to pass filters . You have to JSON encode them. This means you have to first convert them to JSON like {"dangling":["true"]} and then use utility classes like Java’s URLEncoder.encode(json, UTF_8.name()) to encode json.

$ wget -i --header="Content-Type:application/json" --no-check-certificate 
--certificate=$DOCKER_CERT_PATH/cert.pem --private-key=$DOCKER_CERT_PATH/key.pem 
https://192.168.99.100:2376/images/json??all=false&filters=%7B%22dangling%22%3A%5B%22true%22%5D%7D -O - -v

Tag an image into a repository

POST /images/(name)/tag

$ wget --method=POST --header="Content-Type:application/json" --no-check-certificate 
--certificate=$DOCKER_CERT_PATH/cert.pem --private-key=$DOCKER_CERT_PATH/key.pem https://192.168.99.100:2376/images/openshift/hello-openshift/tag?repo=shekhargulati/hello-openshift&tag=v42 -O - -v

Get Container Logs

GET /containers/(id)/logs

$ wget --method=GET --header="Accept: application/vnd.docker.raw-stream" 
--no-check-certificate --certificate=$DOCKER_CERT_PATH/cert.pem 
--private-key=$DOCKER_CERT_PATH/key.pem 
https://192.168.99.100:2376/containers/05c49c850f83/logs?stderr=1&stdout=1&timestamps=1&follow=1&tail=10 -O - -v

Exec Start

POST /exec/(id)/start

wget --method=POST --header="Accept:application/vnd.docker.raw-stream" --header="Content-Type:application/json" --body-data '{"Detach": false,"Tty": false}' --no-check-certificate --certificate=$DOCKER_CERT_PATH/cert.pem --private-key=$DOCKER_CERT_PATH/key.pem https://192.168.99.100:2376/exec/c481eeb18bf58cd5b1375fea57b7449f50f823ce3724a1eb344fb3fe5c1b6cf9/start -O - -v