Using Spray to mock 3rd party APIs in your tests

If you need to test/mock 3rd party API’s / a crawler / external web service, and you dont want to use some fancy DI framework (I hate the magic in DI frameworks, I use constructor DI instead..), anyway, here’s a helper class I put togather to that end

If you haven’t got a chance to try out Spray, you totally should, its a high performance web server built on top of Netty and Akka

Spray has a very nice HTTP API. Few things I liked:

  • HTTP method mapping spray.http.HttpMethods - GET, POST etc..
  • HttpRequest and HttpResopnse are basically wrappers around data you would expect to have in a request and response data
  • spray.http.Uri wrapper class around what you would expect to find in the URI
  • it really is just an Akka Actor, with just the right amount of control exposed and the rest well hidden behind its nice APIs

So, for testing/mocking all I really need is:

  • HTTP method
  • URI
  • body(content)

Here’s an implementation of the helper class

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
import akka.actor._
import akka.io.IO
import java.util.concurrent.atomic.AtomicInteger
import spray.can.Http
import spray.http.{Uri, HttpMethod, HttpResponse, HttpRequest}

object Helpers {

lazy implicit val testSystem = ActorSystem("helpers-test-system")
// used to track the last assigned port
val lastPort = new AtomicInteger(20000)
// a single expected triplet of method+uri+body
type Replay = (HttpMethod,Uri.Path,String)

// factory method to get a replater to play around with
def webReplayer(replay : List[Replay]) : (ActorRef,Int) = {
// everytime the factory is used a new port is assigned,
// so many instances can co-exist and tests can run in parallel without getting
// on eachother's toes
val newPort = lastPort.addAndGet(1)
// instantiate a new Spray server. Yes, its that simple!
val newServer = testSystem.actorOf(Props(new Helpers.ReplayerActor(replay)))
IO(Http) ! Http.Bind(newServer, interface = "127.0.0.1", port = newPort)
// return a pair of the server instance and the assigned port
// the newPort should be used by client code to construct the remote URI with
(newServer,newPort)
}

// object to use to tell the server to shutdown..
object ShutTestServer

// a concrete implementation for a Spray Actor to listen to incoming HttpRequest(..) messages
// provided a list of messages to be matched against arrived and content to be sent back
class ReplayerActor(var replay :List[Replay]) extends Actor {
var ioServer : ActorRef = null
override def receive: Receive = {
case _ : Http.Connected =>
sender ! Http.Register(self)
ioServer = sender()
// this is basically a "catch all" HttpRequest pattern
case HttpRequest(method, uri,_,_,_) =>
// a test to see that this is indeed expected to happen at this point
if (!replay.isEmpty && replay.head._1 == method && uri.path == replay.head._2){
sender ! HttpResponse(entity = replay.head._3)
replay = replay.tail
} else {
sender ! HttpResponse(status = 400, entity = "was expecting a different request at this point")
}
case ShutTestServer => ioServer ! PoisonPill
case _ => //ignore
}
}
}

And now a cocrete client code example to use this tool

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// some random content I would like to receive in this scenario (or test..)
val content =
s"""
|<html>
|<head>
|<title>sometitle</title>
|</head>
|<body>some body</body>
|</html>
""".stripMargin

// we use the factory method to get an instance of "replayer" to play around with
val (_testWebServer,port) = Helpers.webReplayer(
List(
(GET,Uri.Path("/somesite"),content)
)
)

// and where ever in your actual code, use the port we got back above and have it "hit" the server
val whatEver = YourHTTPGetter.fetch(s"http://127.0.0.1:$port/somesite")
assert(whatEver == content)

Profit!

Making The Case For Kotlin

The Kotlin project page has this to say about Kotlin: “is a statically typed programming language that compiles to JVM byte codes and JavaScript.” project page

Kotlin is a very interesting project, I wanted to check it out, so I decided to write this blog post exploring some design aspects behind Kotlin, and specifically how it compares to the leading JVM “hacker” language - Scala.

Who Is Behind Kotlin?

Even though there are several decent free IDE’s available today (Eclipse, NetBeans, Visual Studio Express..), it (could be) surprising that JetBrains is able to rack in between 199$ for a personal license and up to 699$ for a commercial license (for companies) including 1 year update subscription (I have indeed bought one personal license for my development needs). If you are developing on top of VisualStudio you are (most) probably familiar with the 250$+ plugin ReSharper.

I would argue that this makes them a good candidate, with insight into good and bad language features, but lets not get ahead of ourselves and set the expectations too high.

Why Another JVM Language?

JVM is a well known platform, its weaknesses and scalability strategies are available and documented well on the web, its being used as classic Java run-time VM and for some other popular languages like Scala, Groovy, Clojure and others. Many modules have been already written in Java, normally JVM run languages are compatible with the existing Java ecosystem.

Why Static Typing?

In the old (and still ongoing) battle of dynamic and static languages, the consensus is that dynamic languages are faster and easier way to get things done, on the static languages side they allow programmers harness the power of intelligent IDEs to scale up and maintain their code. Type-inferred languages try to find a middle ground approach so that you don’t need to write so much boilerplate code (cue JAVA) to support the static typing, while having the benefits of a full compile time type enforcement.

Finally, Kotlin Vs. Scala - Fight!

Scala today is the fastest growing JVM language and is probably competing for the same crowd as Kotlin, Kotlin put up a comparison page link so we will simply go over the omissions part and try to make sense of it.

OMITTED - Implicit conversions, parameters

What is it? Scala implicits is (IMHO) a single biggest weakness in the language, once they are used, along with the ability to boast symbols for syntax, it quickly stops being human readable and more like a DSL. Which requires many trips to the docs or even worse - the source code.

SBT (Scala Build Tool) is the Maven/Ant of the Scala world link, its very difficult to get things done with it without many trips to google/documentation

1
2
3
4
5
lazy val pricing = Project (
"pricing",
file ("cdap2-pricing"),
settings = buildSettings ++ Seq (libraryDependencies ++= pricingDeps)
) dependsOn (common, compact, server)

Lift Is a popular web framework, here’s a snippet I took from one of its documentation pages link (see the "#result *" #> x part)

1
2
3
4
5
6
7
8
9
def render = inputParam match {
case Full(x) =>
println("Input is: "+x)
"#result *" #> x

case _ =>
println("No input present! Rendering input form HTML")
PassThru
}

Thoughts - Indeed for many people this is the part that makes Kotlin a better “scalable” language, by not allowing programmers and libraries hinder the code unreadable without an IDE

OMITTED - Overridable Type Members

What is it? Type members in Scala allow you to manage local generic types by assign a type to a class/abstract class/trait member

1
2
3
abstract class HasTypedMember {
type typeMe
}

An extending class will provide a specific type for this type type member

1
2
3
class ExtendingClass extends HasTypedMember {
type typeMe = Int
}

Thoughts - Frankly, its not really clear why Kotlin decided to omit having typed members, I could be missing something.. Never the less Kotlin usually takes the minimalistic and simplistic route, my guess is that if this is indeed useful, the door is open to add it later..

OMITTED - Path-dependent types

What is it? Pointing to a class under another class in Java would mean a specific type (Parent.Child), well not so in Scala. Scala makes the nested class of two same class instances different, much like you would expect two members of two different instances of a single class. Confused? here is some code to explain this better

Java:

1
2
3
4
5
6
7
8
9
10
11
12
public class Outer {
public class Inner {}
public void useInner(Outer.Inner inner){...}
}
//.. somewhere in the code
o1 = new Outer()
o2 = new Outer()
i1 = new o1.inner()
i2 = new o2.inner()
// this would work fine
o1.useInner(i2) // works
o2.useInner(i1) // works

Scala:

1
2
3
4
5
6
7
8
9
10
11
12
class Outer {
class Inner {}
def useInner(inner : Inner){...}
}
//.. somewhere in the code..
val o1 = new Outer
val o2 = new Outer
val i1 = new o1.Inner
val i2 = new o2.Inner
o1.useInner(i1) // works fine
o2.useInner(i2) // works fine
o1.useInner(i2) // THROWS type mismatch exception!

Thoughts - It makes some sense that the path to a type that can only exist in an instance is enforced as such.. It would be interesting to know why Kotlin decided to implement this feature in a similar way to Java (conservative?)

OMITTED - Existential types

What is it? This is yet another trade-off for Scala to be able to inter-op with Java
Let’s see an example:

1
2
3
4
5
// Create a case class with generic type
case class Z[T]( blah:T )
// create an instance and cast it to a type created with existential type annotation
Seq( Z[Int]( 5 ), Z[String]("blah") ) : Seq[Z[X] forSome { type X >: String with Int }]
res1 = Seq[Z[_ >: String with Int]] = List(Z(5), Z(blah))

Thoughts - Scala supports type variance to describe generic type bounds, Scala designer (Odersky) chose to add existential types support due to three main reasons link: erasure, raw types (lack of generics) and Java’s “wildcard” types.

Kotlin took the more “purist” approach (also known as reified types) of dropping support for erasure, raw types, wildcard types and finally existential types.

OMITTED - Complicated logic for initialization of traits

What is it? Traits are similar to abstract classes, except that you may use several traits on the same class at the same time (like interfaces).

In Scala, traits are constructed before the extended classes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
trait Averager {
val first : Int
val second : Int
val avg : Int = first / second
}
// if we were to naively construct a class like so:
class Naive extends Averager {
val first = 20
val second = 10
}
// create instance
> new Naive()
java.lang.ArithmeticException: / by zero
// what happened was that our val "avg" was computed at construction time before seeing the values we put in Naive

Alternatively the trait can be used with a pre-initialized anonymous class val res = new { val first = 20 ; val second = 10 } with Averager, also the val which depends on other vals (avg in our case) can be made “lazy” with the (obvious) lazy keyword lazy val avg : Int = first / second.

As a side note, using def for “avg” could work, but it would mean evaluating whatever is defined under “avg” every-time this method is called.

Thoughts - Traits in Scala can have state which could be made tricky. If you read this blog post link you will see Kotlin developers thoughts and a nice insight into what went into designing the inheritance in Kotlin. (don’t forget to read the comments!)

OMITTED - Custom symbolic operations

What is it? Scala does not have a built in handling of symbols (like =,+,/ etc), they are actually method calls, as you can have symbols used for method names.

Thoughts - This may sound useful at first, but can and (arguably) will create very confusing looking code, like we saw before in the Lift and SBT examples.

OMITTED - Built-in XML

What is it? Exactly as it sounds, Scala built in XML native support, so you can do stuff like this

1
val holdsXml = <outer><inner>some text</inner></outer>

Thoughts - Having a built in support for XML as a language feature is interesting, and sometimes useful. An interview link with Odersky reveals that he would consider removing the native XML support in Scala if had written Scala now.

Bottom Line

Kotlin is an idiomatic language that draws from the many good features in todays leading languages, specifically from Scala which is a leading language on the JVM platform today. We’ve looked into omissions on Kotlin’s part versus Scala

I’ve very much enjoyed writing this blog post and learning about the evolution that the design of Java/Scala/Kotlin languages, and the choices the respective authors made along the way.

To get a better feeling of Kotlin I might pick it up for a small project, at which point I will surely blog about it some more..

Setting Up IntelliJ For Android And Scala On Ubuntu

Prerequisites

Android SDK

  1. get the latest SDK

  2. after extracting, run ./tools/android

  3. from the list select the Android version your afer, right now its Android 4.0.3 (API 15) (I also installed Android 2.3.3), click on Install packages to get the installation started

  4. setup your env echo "export ANDROID_HOME=~/Applications/android-sdk-linux/" >> ~/.bashrc

  5. refresh your currect session source ~/.bashrc

Adding SBT support to IntelliJ

  1. open File > Settings > Plugin

  2. click on Browser repositories

  3. find SBT, right click and select Download and install

Setup SBT

  1. create folder ~/bin/ (if you dont have ~/bin in your PATH, then you can do this: echo "PATH=$PATH:~/bin" >> ~/.bashrc (refresh your currect session source ~/.bashrc)

  2. download sbt-launcher.jar and place it under ~/bin

  3. create launcher: touch ~/bin/sbt and then echo 'java -Xmx512M -jar `dirname $0`/sbt-launch.jar "$@"' >> ~/bin/sbt

  4. make it executable: chmod u+x ~/bin/sbt

Setup android-plugin for Scala support

  1. install giter8 curl https://raw.github.com/n8han/conscript/master/setup.sh | sh

  2. run it ~/bin/cs n8han/giter8

Setup sbt-idea

  1. create folder (if it doesnt exist) mkdir -p ~/.sbt/plugins/

  2. create the file ~/.sbt/plugins/build.sbt and add the following lines to it:

    resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
    addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.0.0")
    

Finally, now that you have all the tools you need, proceed to creating a new project

  1. run the android plugin project setup tool: ~/bin/g8 jberkel/android-app, follow the onscreen questions (or press return for defaults)
    this will create the project folder and create the files needed to build an android project

  2. open the new folder cd <project name>

  3. setup IntelliJ project support sbt gen-idea

  4. open the project in IntelliJ