IPv6-only network in Elixir

I recently had to deal with deployment to AWS. I don’t know why it has to be like this, but AWS belongs in its own category of hell. The amount of effort it took to get to a production-ready infrastructure for Elixir running was mind-boggling. From my blood, sweat, and tears, I’ve shared a repo to help others save weeks or months of their lives dealing with this.

The IPv6-only story

One aspect of this infrastructure is IPv6-only networks. Our instances use an IPv6 address to communicate with the outside world. This eliminates the need for a NAT. However, the world still deals with IPv4, and IPv6 is not compatible with IPv4.

So due to backward compatibility requirements, we’re living in a world of dual-stack IPv4/IPv6.

Dual-stack works by having both an IPv4 and IPv6 address available. Programming will decide what address to use to connect to a hostname. Happy Eyeballs algorithm will connect to both addresses in parallel, and whichever is the fastest to respond will be the one used.

In OTP/Elixir we don’t have any built-in algorithm deciding what to connect with. Neither is there a way to globally configure what IP format to use. Instead, we have to explicitly set an :inet6 flag when resolving the address:

iex(1)> :inet.getaddr('google.com', :inet) 
{:ok, {142, 251, 34, 46}}

iex(2)> :inet.getaddr('google.com', :inet6)      
{:ok, {9735, 63664, 16391, 2067, 0, 0, 0, 8206}}

This means that every single library that resolves a hostname to an address will have its particular way of resolving it.

Honeybadger and :hackney

This became a problem with Honeybadger. Under the hood, Honeybadger uses :hackney. :hackney resolves the address like this:

is_ipv6(Host) ->
  case inet_parse:address(Host) of
    {ok, {_, _, _, _, _, _, _, _}} ->
      true;
    {ok, {_, _, _, _}} ->
      false;
    _ ->
      case inet:getaddr(Host, inet) of
        {ok, _} ->
          false;
        _ ->
          case inet:getaddr(Host, inet6) of
            {ok, _} ->
              true;
            _ ->
              false
          end
      end
  end.

If the host is not already an IP address it’ll first resolve the IPv4 address, and if that fails it’ll resolve the IPv6 address.

This logic is problematic, but :hackney will skip the above logic when the :inet6 flag is set in the connect options. Unfortunately, Honeybadger didn’t allow for passing in options to :hackney. I opened a pull request that allows for passing connect options. It has now been merged to main, and we can globally configure the options with:

config :honeybadger,
  # ...
  hackney_opts: [connect_options: [:inet6]]

My instances are now successfully connecting to honeybadger over IPv6!

Good HTTP client practice

The above highlights an issue I see from time to time with libraries.

When building a library you want to make things work out of the box. Developers shouldn’t have to implement their own HTTP client just to use Honeybadger. The problem is when you don’t give developers any control over the dependencies as seen above.

At a minimum, it must be possible to pass on options to the underlying HTTP client. However, why force the use of a certain HTTP client dependency? What if I need to pool connections together across my system? Why can’t I just use my Finch process for all outgoing requests?

That’s why I recommend setting up HTTP client adapters. It is what I’m doing with Assent, defining a behaviour and using it for the built-in adapter(s):

@type method :: :get | :post
@type body :: binary() | nil
@type headers :: [{binary(), binary()}]
@callback request(method(), binary(), body(), headers(), Keyword.t()) :: {:ok, map()} | {:error, any()}

I’m doing the same with TestServer HTTP servers to support Bandit, :cowboy, and :httpd:

@type scheme :: :http | :https
@type instance :: pid()
@type port_number :: :inet.port_number()
@type tls_options :: keyword()
@type server_options :: keyword()

@callback start(instance(), port_number(), scheme(), tls_options(), server_options()) :: {:ok, pid(), server_options()} | {:error, any()}
@callback stop(instance(), server_options()) :: :ok | {:error, any()}

Parker Selbert is currently working on introducing this to the Honeybadger library!

This helps developers manage their dependency graphs, and it allows developers in unique situations to solve the problems without having to fork, patch, and then wait for an upstream release with a fix.