In the last episode, we started fly, an OTP application for on-the-fly file processing. Today we'll build a plug so that we can wire it into a web application. Let's get started.

Project

We're starting out with the fly project, tagged before this episode. We'd like to build a plug, and I'd like to do it with tests. Let's bring in plug as a dependency:

  def application do
    [
      applications: [
        # ...
        :plug
      ],
      # ...
    ]
  end

  defp deps do
    [
      # ...
      {:plug, "~> 1.3.0"},
      # ...
    ]
  end
mix deps.get

Let's add a new test for Fly.Plug:

mkdir test/fly
vim test/fly/plug_test.exs

We'll start with a basic plug test, calling the plug directly and confirming our response:

defmodule Fly.PlugTest do
  use ExUnit.Case, async: true
  use Plug.Test

  @opts Fly.Plug.init([])
  @fly_jpg_url "https://raw.githubusercontent.com/dailydrip/fly/after_episode_283/test/fixtures/fly.jpg"

  test "gets static text" do
    # Create a test connection
    conn =
      :get
        |> conn("/static?url=#{@fly_jpg_url}")
        |> Fly.Plug.call(@opts)

    # Assert the response and status
    assert conn.status == 200
    assert conn.resp_body == "giggity"
  end
end

We know it won't pass without the module existing, so let's start building the plug:

vim lib/fly/plug.ex
defmodule Fly.Plug do
  @behaviour Plug
  import Plug.Conn
  alias Plug.Conn
  require Logger

  def init([]), do: []

  def call(%Conn{path_info: [config_atom_string]} = conn, []) do
    try do
      config_atom = String.to_existing_atom(config_atom_string)
      result = Fly.run(config_atom, "fake_input")
      conn
        |> resp(200, result)
    rescue
      e in ArgumentError ->
        Logger.error (inspect e)
        Logger.error "Zomg no good: #{inspect config_atom_string}"
        Logger.error "here's conn: #{inspect conn}"
        conn
    end
  end
  def call(conn, []), do: conn
end

OK, so this is a bit unfair since it took me a while to build it like I wanted and it's not good enough since it won't be setting the response type appropriately - but it works! The next thing we'd like to do is fetch the file specified by the URL and pass its data through the worker. We'll mock an actual fetch for our tests eventually, but for now let's just do it wrong and dumb:

defmodule Fly.PlugTest do
  # ...
  @root_dir File.cwd!
  @test_dir Path.join(@root_dir, "test")
  @fixtures_dir Path.join(@test_dir, "fixtures")

  @opts Fly.Plug.init([])
  @fly_jpg_url "https://raw.githubusercontent.com/dailydrip/fly/after_episode_283/test/fixtures/fly.jpg"

  # ...
  test "handles pngify" do
    expected =
      @fixtures_dir
        |> Path.join("fly.png")
        |> File.read!

    # Create a test connection
    conn =
      :get
        |> conn("/pngify?url=#{@fly_jpg_url}")
        |> Fly.Plug.call(@opts)

    # Assert the response and status
    assert conn.status == 200
    assert conn.resp_body == expected
  end
end

This won't work because we don't fetch the specified URL. Let's move on to fetching it in our Plug and passing the data we fetch into the worker. We'll begin by faking it:

defmodule Fly.Plug do
  @behaviour Plug
  import Plug.Conn
  alias Plug.Conn
  require Logger

  def init([]), do: []

  def call(%Conn{path_info: [config_atom_string]} = conn, []) do
    try do
      config_atom = String.to_existing_atom(config_atom_string)
      conn = fetch_query_params(conn)
      url = conn.query_params["url"]
      file = get(url)
      result = Fly.run(config_atom, file)
      conn
        |> resp(200, result)
    rescue
      e in ArgumentError ->
        Logger.error (inspect e)
        Logger.error "Zomg no good: #{inspect config_atom_string}"
        Logger.error "here's conn: #{inspect conn}"
        conn
    end
  end
  def call(conn, []), do: conn

  # Fake fetching the URL for now
  defp get(url) do
    File.cwd!
      |> Path.join("test")
      |> Path.join("fixtures")
      |> Path.join("fly.jpg")
      |> File.read!
  end
end

So this test passes. Let's go ahead and actually fetch the file with tesla using the hackney adapter.

  def application do
    [
      applications: [
        # ...
        :hackney,
      ],
      # ...
    ]
  end

  defp deps do
    [
      # ...
      {:tesla, "~> 0.5.0"},
      {:hackney, "~> 1.6.3"},
      # ...
    ]
  end
mix deps.get

We'll build an HTTP client module:

vim lib/fly/http.ex
defmodule Fly.Http do
  use Tesla
  adapter Tesla.Adapter.Hackney
end

Now we'll use that in our plug:

defmodule Fly.Plug do
  # ...
  defp get(url) do
    %{body: body} = Fly.Http.get(url)
    body
  end
end

So now we've got our plug doing almost everything our workers can do. The last thing is to be able to pass options.

For now we'll just strip the "url" key out of the query string and pass it along, but we're definitely going to have to protect this a bit so that people can't request arbitrary work from the server. We want to be able to give people a URL that allows them to perform this sort of work against our plug, but we don't necessarily want it to be wide open.

Let's ignore that for now, and write a test for resizing:

  # We'll extract a `read_fixture` function while we're at it.
  defp read_fixture(filename) do
    @fixtures_dir
      |> Path.join(filename)
      |> File.read!
  end

  test "handles resize" do
    # Create a test connection
    conn =
      :get
        |> conn("/resize?url=#{@fly_jpg_url}&size=100x")
        |> Fly.Plug.call(@opts)

    # Assert the response and status
    assert conn.status == 200
    assert conn.resp_body == read_fixture("fly_resize_100x.jpg")
  end

If we run this, it fails because the Resize worker doesn't know how to handle not being passed a size. Let's pass everything but the url through as arguments:

defmodule Fly.Plug do
  # ...
  def call(%Conn{path_info: [config_atom_string]} = conn, []) do
    try do
      config_atom = String.to_existing_atom(config_atom_string)
      conn = fetch_query_params(conn)
      url = conn.query_params["url"]
      file = get(url)

      # Here's how we'll build our options, filtering out the "url" key and
      # turning each argument into an existing atom since our workers expect
      # atoms in the options.
      options =
        conn.query_params
        |> Map.delete("url")
        |> Enum.map(fn({k, v}) -> {String.to_existing_atom(k), v} end)
        |> Map.new

      result = Fly.run(config_atom, file, options)
      conn
        |> resp(200, result)
    rescue
      e in ArgumentError ->
        Logger.error (inspect e)
        Logger.error "Zomg no good: #{inspect config_atom_string}"
        Logger.error "here's conn: #{inspect conn}"
        conn
    end
  end
  def call(conn, []), do: conn
  # ...
end

Run the tests, and they pass.

Summary

That's all that I think we can reasonably do today. I haven't tested it in a router yet, but we should be able to forward a route to this plug and have a pretty decent means of building new workers that can be routed to by their configured atom. The next step will be - I think - to build in some basic URL generation and caching / duplicate-matching.

We don't want 20 simultaneous requests for a given job to yield 20 exact duplicates of the corresponding work. Instead, we'd like to start the work with the first request and then block them all until the result has returned. Then we'd like to provide caching so that future requests are essentially memoized.

Hope you're enjoying this. See you soon!

Resources