10 Things I Learnt By Reading Retrofit Source Code

Welcome to the first post of X-things-I-learnt-reading-Y’s-source-code series. In this blog series, I will share what I learnt after reading source code of popular open-source projects. Each week I plan to spend at least couple of hours reading source code of an open-source project and then sharing it on my blog. I plan to cover aspects such as design patterns project use, coding practices , API design, how they solved particular problem with their design, or any other aspect that I found useful to share. The purpose of this series is to learn how good developers design and build software. Then, I can apply these learnings to my work.

This week we will look at Retrofit, an open-source Java library built by folks at Square. Retrofit makes it easy to write type-safe HTTP clients for Android and Java. If you have written more than one HTTP client library then you will know that most of the code to write HTTP client is boilerplate and similar in nature. I, myself, find writing HTTP client libraries boring and tedious.

Retrofit in 5 minutes

Retrofit ease the process of creating HTTP clients. You write an interface specifying all the operations your client supports and Retrofit generates a proxy that implements the actual code of making HTTP API calls. It will also take care of converting objects to and from their representation in HTTP. For example, Retrofit will convert custom request object to JSON and convert JSON response to custom object. Let’s look at the example shown in the documentation:

public interface GitHubService {
  @GET("users/{user}/repos")
  Call<List<Repo>> listRepos(@Path("user") String user);
}

In the code snippet shown above, we are writing client for Github REST API.

1. First, we declared that we want to implement method in our client that will list all the repositories for a user.
2. Next, we used annotations @GET and @Path to specify HTTP method and path variable respectively.
3. Finally, listRepos method returns a Call<List<Repo>>, which only prepares the HTTP request but does not execute it. Once you have the Call object, you can decide whether you want to make synchronous or asynchronous request to the server.

Repo is our custom domain class representing Github Repository.

import com.google.gson.annotations.SerializedName;

public class Repo {
    String id;
    String name;
    @SerializedName("full_name")
    String fullname;
    String description;
    public Repo(String id, String name, String fullname, String description) {
        this.id = id;
        this.name = name;
        this.fullname = fullname;
        this.description = description;
    }
}

Finally, you can use your client as shown below.

import retrofit2.Call;
import retrofit2.Response;
import retrofit2.Retrofit;
import retrofit2.converter.gson.GsonConverterFactory;

public class RetrofitExample {

    public static void main(String[] args) throws IOException {
        Retrofit retrofit = new Retrofit.Builder()
                .baseUrl("https://api.github.com/")
                .addConverterFactory(GsonConverterFactory.create())
                .build();
        GitHubService gitHubService = retrofit.create(GitHubService.class);
        Call<List<Repo>> reposCall = gitHubService.listRepos("shekhargulati");
        Response<List<Repo>> response = reposCall.execute();
        response.body().forEach(System.out::println);
    }
}

In the code shown above, we made GET request to fetch list of my public repositories.

1. We started by creating an instance of Retrofit using the builder API. We specified we want to use GSON based GsonConverterFactory to convert JSON response to our domain object Repo.
2. Next, we created an instance of our service by calling create method on the retrofit instance passing it our service interface.
3. Then, we made call to Github REST API using our client and listed all the repositories. We executed the call synchronously by calling the execute method.
4. Finally, we printed all the repositories to the console.

10 Things I learnt

Now that we understand what is Retrofit and how we can use it I will share what I learnt by reading its source code. Below is the list of 10 things that I learnt:

1. I start reading source code by looking at the project build file. I believe build file can tell you a lot about the project quality and there are tricks that can be easily applied to our own projects. Retrofit uses Apache Maven as its choice of build tool. The first thing that caught my attention is how cleanly they specified dependency versions in the properties sections of their parent pom.xml. As a developer, you only have to look at one section to get understanding of all the libraries used by the project.

   <properties>
   
       <!-- Dependencies -->
       <android.version>4.1.1.4</android.version>
       <okhttp.version>3.8.0</okhttp.version>
       <animal.sniffer.version>1.14</animal.sniffer.version>
   
       <!-- Adapter Dependencies -->
       <rxjava.version>1.3.0</rxjava.version>
       <rxjava2.version>2.0.0</rxjava2.version>
       <guava.version>19.0</guava.version>
   
       <!-- Converter Dependencies -->
       <gson.version>2.7</gson.version>
       <protobuf.version>3.0.0</protobuf.version>
       <jackson.version>2.7.2</jackson.version>
       <wire.version>2.2.0</wire.version>
       <simplexml.version>2.7.1</simplexml.version>
       <moshi.version>1.4.0</moshi.version>
     // removed couple of sections
     </properties>
   

   In the <properties> section above, they cleanly separated dependencies of different modules by the use of comments. You can easily figure out which module use which dependency along with its version. I know it is a very small thing but in most projects it is a nightmare to figure out all the dependencies used by different modules of the project without firing mvn dependency:tree or related command.

   Retrofit make use of dependencyManagement section to declare all the dependencies of the project in the parent pom.xml. This avoids duplicating dependency version information in child modules and source of truth regarding dependency versions is maintained at one place. I already do that in my projects but many times I have worked with teams that don’t use dependencyManagement feature.

2. Retrofit API makes use of Builder pattern to construct Retrofit instance as shown below. Builder pattern allow you to construct a complex object by separating construction from its representation. There is no direct way to create instance of the Retrofit class. You have to use Builder, a static member class of Retrofit

   Retrofit retrofit = new Retrofit.Builder()
                   .baseUrl("https://api.github.com/")
                   .addConverterFactory(GsonConverterFactory.create())
                   .build();
   

   The use of Builder keeps the API clean by avoiding use of telescoping constructor pattern to create objects. The use of builder also help create immutable Retrofit instances, thus giving us all the immutability benefits.

   There are couple of benefits of making Builder a static inner class:

   1. You don’t have to learn about any other class apart from Retrofit. Many times I have used Builder pattern in my work, I made builder a separate class. Making builder a separate class defeats the purpose as now you expect developer to know about multiple classes. Making builder a static inner class provides all the flexibility of the builder at the same time frees developer from learning multiple classes in the API. This keeps the surface area of the API minimal.
   2. The other benefit is that you don’t have to name builder as RetrofitBuilder. You will end up doing it if you make builder a top level class. By making Builder a static inner class, every builder can have name Builder as their name. Thus, keeping code clean and uniform.

   If you have read Effective Java book, then you will notice that this is Item 2: Consider a builder when faced with many constructor parameters in action.

3. Retrofit is designed to work on Java 7, Java 8, and Android runtimes. There are differences in each of the three platform. For example, in Java 8, Retrofit has to handle default methods (a JDK 8 feature). Retrofit library has an abstraction retrofit2.Platform that encapsulates the platform code is running on. When you construct the Builder instance, first thing Retrofit does is create a Platform instance. Platform is a singleton so next time you need it you can just call the static get method on the Platform class and it will return you the same instance. retrofit2.Platform make use of Class.forName to check the platform code is running as shown below.

   private static Platform findPlatform() {
       try {
           Class.forName("android.os.Build");
           if (Build.VERSION.SDK_INT != 0) {
           return new Android();
           }
       } catch (ClassNotFoundException ignored) {
       }
       try {
           Class.forName("java.util.Optional");
           return new Java8();
       } catch (ClassNotFoundException ignored) {
       }
       return new Platform();
   }
   

   This is an example of how you can build libraries that work on different platforms by encapsulating differences in respective platform classes.

4. Next thing I liked in the Retrofit library is the usage of flat package hierarchy. There are only two packages retrofit2 and retrofit2.http. There is no need to include groupId in the package like com.square.retrofit2. In Java community, the common practice is to name base package as $groupId.$artifactId. So, if your groupId is com.square and artifactId is retrofit2, then base package is named as com.square.retrofit2. This leads to long package names, which does not convey much. You can strip down groupId from your package name to keep package names short and easy to remember. Your import statements will also start looking much more readable.
   

   You are not required to include groupId in package name. The groupId information is used for identifying your project uniquely across all the projects. You need them when other needs to import your library in their project.

   I have also started using flat package hierarchy in my projects since last one year. I believe this is something Java community learnt from Python and JavaScript communities.

5. I don’t like writing *Util classes. The reason is Util classes tend to grow and we use them as excuse for not separating concerns in their respective classes. I have seen projects where most of the business code is in single Util class. A utility class should only have static utility methods that take an input and give an output back. In most Java projects you will have at least one Util class. I am yet to work on a project that does not have a class that contains Util in its name. Like most projects, Retrofit also has a class named Utils. Retrofit library ensures the correct use of Utils class by making the class and all its methods package private. This means Utils class can’t be used outside the library. Also, Utils class is marked final with its constructor private so that you should only write static methods in it.

   final class Utils {
   
     private Utils() {
       // No instances.
     }
     // static methods removed for brevity
   }
   

   If you have read Effective Java book, then you will notice that this is Item 4: Enforce noninstantiability with a private constructor in usage. 

6. Retrofit library takes care of implementing the service interface so that we don’t have to write code to make HTTP calls in our code. This is achieved by using JDK’s java.lang.reflect.Proxy class. When you call the retrofit.create(GithubService.class) method, then retrofit will create a dynamic proxy that implements the GithubService interface. The dynamic proxy is created using Proxy.newProxyInstance static method method that takes three parameters — the class loader to define the proxy class, the list of interfaces to implement, and the invocation handler to dispatch method invocations to. The below is the code in the create method of Retrofit class.

   return (T) Proxy.newProxyInstance(service.getClassLoader(), new Class<?>[] { service },
       new InvocationHandler() {
           private final Platform platform = Platform.get();
   
           @Override public Object invoke(Object proxy, Method method, @Nullable Object[] args)
               throws Throwable {
           // If the method is a method from Object then defer to normal invocation.
           if (method.getDeclaringClass() == Object.class) {
               return method.invoke(this, args);
           }
           if (platform.isDefaultMethod(method)) {
               return platform.invokeDefaultMethod(method, service, proxy, args);
           }
           ServiceMethod<Object, Object> serviceMethod =
               (ServiceMethod<Object, Object>) loadServiceMethod(method);
           OkHttpCall<Object> okHttpCall = new OkHttpCall<>(serviceMethod, args);
           return serviceMethod.callAdapter.adapt(okHttpCall);
           }
       });
   

   In the code shown above:

   • If the method is from the Object class, we execute the method directly.
   • If the method is Java 8 default method, then it is invoked by the Java 8 specific platform class. To invoke the default method, they use MethodHandles.Lookup class as shown below.

     @Override Object invokeDefaultMethod(Method method, Class<?> declaringClass, Object object,
         @Nullable Object... args) throws Throwable {
         // Because the service interface might not be public, we need to use a MethodHandle lookup
         // that ignores the visibility of the declaringClass.
         Constructor<Lookup> constructor = Lookup.class.getDeclaredConstructor(Class.class, int.class);
         constructor.setAccessible(true);
         return constructor.newInstance(declaringClass, -1 /* trusted */)
             .unreflectSpecial(method, declaringClass)
             .bindTo(object)
             .invokeWithArguments(args);
     }
     

   • One that we are interested in is invoked next. First the java.lang.reflect.Method is converted to ServiceMethod. ServiceMethod is Retrofit library class . It reads all the metadata of the method from the annotations and convert it to an HTTP call.
7. The service interface can return a variety of return types depending on the configured CallAdapter. The default CallAdapter allows you to return Call<SomeType>. The GithubService created above returns Call<List<Repo>>. Retrofit provides an extensible mechanism to adapt Call return type to different return types like CompletableFuture<List<Repo>> or RxJava’sObservable<Repo> etc. For example, if you want to use Java 8 CompleteableFuture as your return type, then you have to add adapter-java8 module to your classpath and configure Retrofit instance to use Java8CallAdapterFactory as shown below.

   Retrofit retrofit = new Retrofit.Builder()
           .baseUrl("https://api.github.com/")
           .addConverterFactory(GsonConverterFactory.create())
           .addCallAdapterFactory(Java8CallAdapterFactory.create())
           .build();
   

   Then, you can change the service interface to as shown below.

   CompletableFuture<List<Repo>> listRepos(@Path("user") String user)
   

   As mentioned above, the extensibility is provided by the use of CallAdapter<R,T>. CallAdpater takes a Call of R type and return a T type. Below is the snippet of CallAdapter interface.

   public interface CallAdapter<R, T> {
       Type responseType();
       
       T adapt(Call<R> call);
     
       abstract class Factory {
               public abstract CallAdapter<?, ?> get(Type returnType, Annotation[] annotations,
           Retrofit retrofit);
       }
   }
   

   As shown above, you can register multiple call adapters by passing factory instances to the addCallAdapterFactory method. A CallAdapter.Factory will be responsible for deciding whether it can handle the return type or not. If it can handle the return type, then it will create a specific instance of CallAdapter else it will return null. A CallAdapter receives a Call<R> and wraps it inside the T. For example in Java 8 case, we will wrap Call<R> inside CompleteableFuture<R>.

   The decision regarding which adapter is used is made during the creation of ServiceMethod object. Retrofit iterates over all the registered adapters to find which adapter can serve the return type. This information is then used during the Proxy creation. The invocation handler adapts the Call type to the expected return type by calling adapt method of the adapter. You can see this in the Proxy.newProxyInstance code shown above.

   I think another way to build this extensible mechanism could be to use Java ServiceLoader mechanism. This way you don’t have to manually register adapters. You will send a request and any adapter that can fullfil that request will respond. This is what JUnit 5 does to figure out which engine can handle the test. JUnit 5 is the next version of JUnit.

   Jake Wharton from Retrofit team clarified why they didn’t use Service loader mechanism We don’t use service loader because order matters. One converter can delegate to another (same for call adapters) and for that to work correctly we need ordering guarantees.

   This is a good example to learn how to build extensible library.

8. Another example of extensibility is the way Retrofit handle request and response conversion. This allow you to specify your own convertor by passing a Convertor that tells how to convert a type to another type. This allows Retrofit to support variety of conversion mechanisms like Gson, Moshi, Jackson, Java 8 Optional factory, protobuffers, etc.The interesting thing to notice is that Converter interface also has similar philosophy. You have interface factory pair. Factory will tell if it can handle the request. If it can handle the request, then Convertor object will be created.

   public interface Converter<F, T> {
     T convert(F value) throws IOException;
   
     abstract class Factory {
       public @Nullable Converter<ResponseBody, ?> responseBodyConverter(Type type,
           Annotation[] annotations, Retrofit retrofit) {
         return null;
       }
   }
   

9. Retrofit code base is also good for learning how to name your test cases. They don’t prefix test cases with should ortest . Name of the tests represent behaviour they are testing like interfaceWithExtendIsNotSupported.
10. The last thing that I found in their code is use of MockWebServer JUnit@Rule for testing HTTP interactions. MockWebServer is part of Square’s another library mockwebserver. Using MockWebServer, you can supply canned responses and the server replays them upon request in sequence. This avoids hitting the real API servers and your test become more predictable. Very useful when you are working with code that makes HTTP calls.

    @Rule public final MockWebServer server = new MockWebServer();
    

Conclusion

There is a lot that we can learn from open source code base. I hope you learnt something from this post. I learnt a lot. Please share your feedback.