Finatra Tutorial: Building Scalable Services The Twitter Way


Finatra is an open-source project by Twitter that can be used to build REST APIs in Scala programming language. Finatra builds on top of Twitter’s Scala stack — twitter-server, finagle, and twitter-util.

  1. Finagle: It can be used to construct high performance servers.
  2. Twitter Server: It defines a template from which servers at Twitter are built. It uses finagle underneath.
  3. Twitter-Util: A bunch of idiomatic, small, general purpose tools for Scala.

In this step-by-step tutorial, we will cover how to build a Scala REST API using Finatra version 2.13.

Prerequisite

  1. Scala 2.12.1
  2. IntelliJ Idea Community Edition
  3. JDK 8

Building an application from scratch

In this blog, we will build a simple application called fitman. The goal of this application is to track weight of an individual. Every week, a user enter his/her weight and a status message describing how they are feeling. This will allow them to view a timeline of their body weight.

Step 1: Create a Scala SBT project using IntelliJ Idea

Open IntelliJ Idea and select Scala > SBT project. You will see screen as shown below.

After selecting, press Next button. Enter the project details and press Finish button.

This will create a Scala based SBT project that we will use.

You can use any other IDE or tool as well to scaffold a Scala SBT project.

Step 2: Adding required dependencies to build.sbt

Your build.sbt should look like following:

name := "fitman"

version := "1.0"

scalaVersion := "2.12.1"

lazy val versions = new {
  val finatra = "2.13.0"
  val logback = "1.1.7"
  val guice = "4.0"
}

libraryDependencies += "com.twitter" %% "finatra-http" % versions.finatra
libraryDependencies += "ch.qos.logback" % "logback-classic" % versions.logback

Please be patient it takes time to download all the dependencies.

Step 3: Fitman says Hello

A finatra app consists of an http server, a list of controllers, and zero or more filters. Let’s create a simple Scala class that extends finatra’s com.twitter.finatra.http.HttpServer as shown below.

import com.twitter.finatra.http.HttpServer

object FitmanApp extends FitmanServer

class FitmanServer extends HttpServer

In the code shown above, we created a server FitmanServer that extends com.twitter.finatra.http.HttpServer. HttpServer extends TwitterServer and adds configuration specific to an http server. TwitterServer is a template using which other types of servers can be created. FitmanApp is an object that is used to launch the server. The above code will not compile as we have to override configureHttp method of HttpServer

From the documentation,

The reason for having a separate object is to allow server to be instantiated multiple times in tests without worrying about static state persisting across test runs in the same JVM.

Also, according to documentation Finatra convention is to create a Scala object with a name ending in Main. I prefer to use convention where name ends with App so I am using that.

Let’s write our first controller — HelloController in the FitmanApp.scala file as shown below.

import com.twitter.finagle.http.Request
import com.twitter.finatra.http.routing.HttpRouter
import com.twitter.finatra.http.{Controller, HttpServer}

object FitmanApp extends FitmanServer

class FitmanServer extends HttpServer {
  override protected def configureHttp(router: HttpRouter) {
    router.add[HelloController]
  }
}

class HelloController extends Controller {

  get("/hello") { request: Request =>
    "Fitman says hello"
  }

}

In the code shown above, we created HelloController which extends finatra Controller abstract class. HelloController is defined with one endpoint – /hello. When an HTTP GET request is made to /hello then the associated callback function will be called. The callback function has callback: RequestType => ResponseType signature. It accepts a com.twitter.finagle.http.Request and returns back a response of any type that can be converted to com.twitter.finagle.http.Response. The callback function in this case just returns a string.

To make server aware of the controller, we registered HelloController with FitmanServer by overriding its configureHttp method. The configureHttp exposes HttpRouter that is used to register an instance of HelloController.

You can run the application just like you will run any Scala main program.

This will launch the netty based server at port 8888. As there is no route configured, so you will not be able to do anything useful.

If you want to use any other port that 8888 then you can use -http.port flag and set the it to your preferred value like -http.port=:8080

Now, when you make an HTTP GET request to http://localhost:8888/hello you will receive Fitman says Hello message in the response.

$ curl -i http://localhost:8888/hello
HTTP/1.1 200 OK
Content-Length: 17
Fitman says hello

Admin Interface

Every Finatra by default exposes an admin interface at http://localhost:9990/admin that you can use to get system level details like CPU usage profile, heap profile, server information, and many other. To learn about all admin features refer to Twitter Server documentation.

Overriding default server configuration

There are two ways you can override default server configuration values. One way is to use flags as discussed previously with admin and http ports. The other way you can override default server configuration is by overriding fields in the FitmanServer as shown below.

class FitmanServer extends HttpServer {

  override protected def defaultFinatraHttpPort: String = ":8080"
  override protected def defaultHttpServerName: String = "FitMan"

  override protected def configureHttp(router: HttpRouter) {
    router.add[HelloController]
  }
}

Step 4: Let’s write feature test for HelloController

One of the feature of Finatra that impressed me most was its inbuilt support for feature testing. Feature testing is a form of blackbox testing that tests a particular feature from outside.

Let’s add dependencies to build.sbt file.

libraryDependencies ++= Seq(
  "com.twitter" %% "finatra-http" % versions.finatra,
  "ch.qos.logback" % "logback-classic" % versions.logback,

  "com.twitter" %% "finatra-http" % versions.finatra % "test",
  "com.twitter" %% "finatra-jackson" % versions.finatra % "test",
  "com.twitter" %% "inject-server" % versions.finatra % "test",
  "com.twitter" %% "inject-app" % versions.finatra % "test",
  "com.twitter" %% "inject-core" % versions.finatra % "test",
  "com.twitter" %% "inject-modules" % versions.finatra % "test",
  "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test",

  "com.twitter" %% "finatra-http" % versions.finatra % "test" classifier "tests",
  "com.twitter" %% "finatra-jackson" % versions.finatra % "test" classifier "tests",
  "com.twitter" %% "inject-server" % versions.finatra % "test" classifier "tests",
  "com.twitter" %% "inject-app" % versions.finatra % "test" classifier "tests",
  "com.twitter" %% "inject-core" % versions.finatra % "test" classifier "tests",
  "com.twitter" %% "inject-modules" % versions.finatra % "test" classifier "tests",

  "org.mockito" % "mockito-core" % "1.9.5" % "test",
  "org.scalacheck" %% "scalacheck" % "1.13.4" % "test",
  "org.scalatest" %% "scalatest" % "3.0.0" % "test",
  "org.specs2" %% "specs2-mock" % "2.4.17" % "test"
)

Let’s write our first feature test that will test the /hello endpoint. To create a feature test, you have to extend a trait called FeatureTest. You have to provide implementation for server definition as shown below. We created an instance of EmbeddedHttpServer passing it our application server — FitmanServer.

import com.twitter.finagle.http.Status._
import com.twitter.finatra.http.EmbeddedHttpServer
import com.twitter.inject.server.FeatureTest

class HelloControllerFeatureTest extends FeatureTest {
  override protected def server = new EmbeddedHttpServer(new FitmanServer)

  test("Server#Say hello") {
    server.httpGet(path = "/hello", andExpect = Ok, withBody = "Fitman says hello")
  }
}

This test will start an embedded http server and will make an actual HTTP GET request to the /hello endpoint. We asserted that HTTP status code returned by our service is 200 i.e. OK and response body contains text Fitman says hello. If you change the body text to something other than Fitman says hello then test will fail with detailed message outlining the difference between texts as shown below.

"Fitman says hello[]" did not equal "Fitman says hello[123]"

This allows you to test the externally visible features of the API.

Step 5: Let’s capture weight

Let’s first write the feature test for our WeightResource. The feature test will test that when an HTTP POST request is made to /weights endpoint then weight will be stored in some database. In today’s blog, we will use a mutable Map to act as a database. Later in this series, we will cover how to work with databases in Scala. We will update our blog then.

import com.twitter.finagle.http.Status
import com.twitter.finatra.http.EmbeddedHttpServer
import com.twitter.inject.server.FeatureTest

class WeightResourceFeatureTest extends FeatureTest {
  override val server = new EmbeddedHttpServer(
    twitterServer = new FitmanServer
  )

  test("WeightResource should save user weight when POST request is made"){
    server.httpPost(
      path = "/weights",
      postBody =
        """
          |{
          |"user":"shekhar",
          |"weight":85,
          |"status":"Feeling great!!!"
          |}
        """.stripMargin,
      andExpect = Status.Created,
      withLocation = "/weights/shekhar"
    )
  }

}

When you will run the test case then this test will fail as we have not yet added functionality for WeightResource.

Our data model looks like as shown below. It should have same field names as JSON.

case class Weight(
                   user: String,
                   weight: Int,
                   status: Option[String],
                   postedAt: Instant = Instant.now()
                 )

Now, let’s write our WeightResource.

import com.twitter.finatra.http.Controller

import scala.collection.mutable

class WeightResource extends Controller {

  val db = mutable.Map[String, List[Weight]]()

  post("/weights") { weight: Weight =>
    val weightsForUser = db.get(weight.user) match {
      case Some(weights) => weights :+ weight
      case None => List(weight)
    }
    db.put(weight.user, weightsForUser)
    response.created.location(s"/weights/${weight.user}")
  }

}

Update FitmanServer with new Controller

    router.add(new WeightResource)

In the code shown above, we did the following:

  1. We created a mutable Map to store weight for a user.
  2. In the post("/weights") callback, we are directly using our case class Weight instead of using Finagle request. Finatra automatically converts the request body to the case class.
  3. In the post method callback, we first check whether the user exists in the db or not. If user exists, then we add weight to its existing weights collection else we create new List with weight.
  4. Finally, we return the response back to the user. response.created makes sure that HTTP status 201 is set. We also set the location header to point to a new resource.

Run the test case and it should be green now.

Step 6: View user weight

Now, let’s write our second operation that will return all captured weights for a user. We will start with a feature test as shown below.

test("WeightResource should list all weights for a user when GET request is made") {
  val response = server.httpPost(
    path = "/weights",
    postBody =
    """
          |{
          |"user":"test_user_1",
          |"weight":80,
          |"posted_at" : "2017-09-14T23:16:06.871Z"
          |}
        """.stripMargin,
    andExpect = Status.Created
  )

  server.httpGetJson[List[Weight]](
    path = response.location.get,
    andExpect = Status.Ok,
    withJsonBody =
    """
          |[
          |  {
          |    "user" : "test_user_1",
          |    "weight" : 80,
          |    "posted_at" : "2017-09-14T23:16:06.871Z"
          |  }
          |]
        """.stripMargin
  )
}

Add the following method to WeightResource

import com.twitter.finagle.http.Request
get("/weights/:user") { request: Request =>
  db.getOrElse(request.params("user"), List())
}

Step 7: Getting logging right

In step 2, we added logback-classic dependencies to the classpath so that we can effectively log in our application. finatra uses SLF4J api for framework logging. SLF4J provides an API abstraction for various logging framework like log4j, logback, etc. Developers are free to choose their favorite logging library that works with SLF4J. finatra documentation recommends to use Logback as an SLF4J binding as it is much more superior and performant than other logging libraries.

To log in your application, you have to mixin com.twitter.inject.Logging trait into your application’s object or class. Let’s add some log statements to WeightResource.

import com.twitter.finagle.http.Request
import com.twitter.finatra.http.Controller
import com.twitter.inject.Logging
import org.joda.time.Instant

import scala.collection.mutable

class WeightResource extends Controller with Logging {

  val db = mutable.Map[String, List[Weight]]()

  post("/weights") { weight: Weight =>
    val r = time(s"Total time take to post weight for user '${weight.user}' is %d ms") {
      val weightsForUser = db.get(weight.user) match {
        case Some(weights) => weights :+ weight
        case None => List(weight)
      }
      db.put(weight.user, weightsForUser)
      response.created.location(s"/weights/${weight.user}")
    }
    r
  }

  get("/weights/:user") { request: Request =>
    info( s"""finding weight for user ${request.params("user")}""")
    db.getOrElse(request.params("user"), List())
  }

}

case class Weight(
                   user: String,
                   weight: Int,
                   status: Option[String],
                   postedAt: Instant = Instant.now()
                 )

Step 8: Validations

Finatra comes with validation annotations that can be used to add validation support. Out of the box Finatra comes with following validation annotations.

  1. CountryCode
  2. FutureTime
  3. Max
  4. Min
  5. NotEmpty
  6. OneOf
  7. PastTime
  8. Range
  9. Size
  10. TimeGranularity
  11. UUID

All these annotations are defined in finatra-jackson module. It is already in our class path so we don’t have to do much to use it.

Let’s write a test case

"Bad request when user is not present in request" in {
  server.httpPost(
    path = "/weights",
    postBody =
      """
        |{
        |"weight":85
        |}
      """.stripMargin,
    andExpect = Status.BadRequest
  )
}

If you run the test now, it will fail with Http status code 500 i.e. Internal server error.

To make it work first we have to register a filter in the FitmanServer.

class FitmanServer extends HttpServer {

  override protected def defaultFinatraHttpPort: String = ":8080"

  override protected def defaultHttpServerName: String = "FitMan"

  override protected def configureHttp(router: HttpRouter) {
    router
      .filter[CommonFilters]
      .add[HelloController]
      .add[WeightResource]
  }
}

Now test will pass.

"Bad request when data not in range" in {
  server.httpPost(
    path = "/weights",
    postBody =
      """
        |{
        |"user":"testing12345678910908980898978798797979789",
        |"weight":250
        |}
      """.stripMargin,
    andExpect = Status.BadRequest,
    withErrors = Seq(
      "user: size [42] is not between 1 and 25",
      "weight: [250] is not between 25 and 200"
    )
  )
}

Update the Weight case class

case class Weight(
                   @Size(min = 1, max = 25) user: String,
                   @Range(min = 25, max = 200) weight: Int,
                   status: Option[String],
                   postedAt: Instant = Instant.now()
                 )

That’s it for this post.

Conclusion

You saw that Finatra has many powerful features that will make it easy for you to build your applications. For more information, you should refer to the documentation.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s