Metaprogramming is a scary word: it sounds like voodoo for programmers, and in some manners it is. It is used by most popular Elixir libraries (Ecto, Phoenix and Elixir itself) and as I leveled up as an Elixir programmer, I needed to understand how it worked under the hood.

I personally felt intimidated by the topic, and I owned the Metaprogramming Elixir book for more than a year before actually reading it 😱

But for any seasoned Elixir programmer, it is an excellent handbook and very pleasant to read. It is also pragmatic and I concur 100% with Chris McCord’s introductory advice:

Rule 1 : Don't Write Macros - Chris McCord

Be cautious with metaprogramming as it is hardly maintainable and a pain to debug; don't even think about writing your application core features with it. It is more relevant for technical libraries, custom DSL and any other specific subjects.

It's a kind of magic

As many of you, I came from a Ruby on Rails development background and I remember that while learning Rails, I was so impressed by the language expressivity and its magic features. Coming then from a Java background, it was a blessing to me!

irb> 2.years.from_now + 5.days.from_now 
#=> Mon, 28 Nov 2022 13:38:36 UTC

irb> Person.find_by_user_name_and_password(user_name, password)
#=> #<Person id: 10, user_name: "Ryan">

Rails made indeed an intensive usage of Ruby dynamic features to extend the language at runtime: the find_by_user_name_and_password call  just above is an example of dynamically generated class method based on the Person model attributes.

But there are caveats to Ruby magic:

• too much magic may hurt as you lack the understanding of what really happens under the hood. Same logic also applies to Elixir metaprogramming (remember Chris McCord's motto? "Don't Write Macros" 😅)
• Ruby magic is dynamic by nature which means it is performed at runtime, at the expense of performance. Whereas Elixir macros are evaluated at compile time and inserted in your bytecode as if it were manually written code.

What is metaprogramming?

Before going deeper in the subject, it’s worth reminding that metaprogramming is the art of writing code that will generate code. Metaprogramming features are first class members of the Elixir language which provides macro API to rapidly extend the core language.

Did you know that if-then-else or unless block syntax are written in Elixir through its macro mechanism? It provides syntaxic sugar over a binary case pattern matching structure.

defmacro macro_unless(clause, do: expression) do
  quote do
    if(!unquote(clause), do: unquote(expression))
  end
end

You could also be using meta-programmed features everyday without knowing it:

• ExUnit test definitions leverages on macros
• Ecto querying DSL can perfectly mimic SQL language thanks to metaprogramming
• Of course Chris McCord’s Phoenix framework uses macro a lot, like the Routing DSL

By the way, did you know that more than 90% of the Elixir sourcecode is written in Elixir? Meta enough for you? 🤯

Elixir Macros 101

Your first Elixir macros will be a bit tedious to write. I won’t give a full course about Elixir metaprogramming (buy Chris McCord’s book!) but I will instead give you a few keys to understand it.

The main thing you need to understand is that all code you write in Elixir is represented in memory as an Abstract Syntax Tree (AST). The compiler will parse your code and turn it into an AST, then the BEAM will evaluate the AST in order to run it.

Metaprogramming, which we previously defined as « code to generate code » could be achieved by generating raw Elixir code (as strings) but it would be unefficient and hardly maintainable.

That’s why Elixir provides API’s to generate and evaluate AST. Here comes quote: an Elixir primitive which can convert any Elixir code into its AST.

iex(1)> quote do 5 + 8 * 3 end
#=> {:+, [context: Elixir, import: Kernel],
#=>   [5, {:*, 
#=>     [context: Elixir, import: Kernel], 
#=>     [8, 3]
#=>   ]}]
#=> }

This code can be evaluated the same way you would have evaluated code in a string (think eval in Javascript)

iex(1)> Code.eval_string("5 + 8 * 3")
#=> {29, []}
iex(2)> Code.eval_quoted(quote do 5 + 8 * 3 end)
#=> {29, []}

Now what happens if you try to evaluate code with dynamic parts?

iex(1)> a = 5
iex(2)> Code.eval_string("3 * a")
#=> ** (CompileError) nofile:1: undefined function a/0

Of course a is unknown to your evaluated code and a simple way to fix this would be to use text interpolation.

iex(1)> a = 5
iex(2)> Code.eval_string("3 * #{a}")
#=> {15, []}

Well the good news is you can interpolate within your quoted code as well: use the unquote primitive.

iex(1)> a = 5
iex(2)> Code.eval_quoted(quote do 3 * unquote(a) end)
#=> {15, []}

Simple as that! So whenever you think you are lost between your quote / unquote blocks, just look at it as if it were text interpolation.

Then you need to know about defmacro which you probably already encountered in your Phoenix project. It's a special construct able to produce code, from quoted expressions and parameters.

Here is an example that generates a few functions :

defmodule MyMacro do
  defmacro generate_functions(name_and_values) do
    for {name, value} <- name_and_values do
      quote do
        def unquote(name)(), do: unquote(value)
      end
    end
  end
end

defmodule MyModule do
  require MyMacro
  
  MyMacro.generate_functions([{:one, 1}, {:two, 2}, {:three, 3}])
end

iex(1)> MyModule.two + MyModule.three 
#=> 5

There are quite a few other things to know when it comes to Elixir metaprogramming, but quote, unquote and defmacro are the basics and can already help you to build powerful stuff.

Ok, but what for?

I will end this post with some examples of the way we use metaprogramming in our Elixir projects.

Extends Ecto query API

Ecto provides a fully featured querying API that mimics SQL. It relies on Elixir Macros and will be able to trigger compilation errors whenever your query is malformed.

iex(1)> query = from u in User, where: u.age > 0, select: u.name
iex(2)> Repo.all(query)

But every now and then you'll want to use some specific database functions not available in Ecto API. You can then use fragment Ecto function which let you insert raw SQL in your Ecto Query:

query = 
  from u in User,
  where: is_nil(fragment("?->>?", u.metadata, "phone"))

But the proper way is to leverage on macros to extend the Ecto query API:

defmodule MyQueryMacros do
  defmacro jsonb_get(column, key) do
    quote do
      fragment("?->>?", unquote(column), unquote(key))
    end
  end
 end
 
 query = 
   from u in User,
   where: is_nil(jsonb_get(u.metadata, "phone"))

Aliases & Imports

As your Elixir codebase grows, you can see that you repeat over and over the same alias and import blocks at the head of your files. Not really DRY...

Phoenix already offers a solution you can use through the use / __using__ macro. To declare a controller or a view, phoenix suggests a bunch of default aliases and imports declared in my_app_web.ex file. Here is an example of changelog.com *_web.ex file :

thechangelog/changelog.com

Hacker to the 💚. Contribute to thechangelog/changelog.com development by creating an account on GitHub.

GitHubthechangelog

You can then use the macro with the following snippet, which will alias and import a few Phoenix modules and even generate some authorization related functions:

defmodule MyAppWeb.MyController do
  use MyAppWeb, :controller
  
  # ...
end

In our own project, we also brewed our own aliasing system that we can use like this to prevent us from declaring the same aliases over and over.

defmodule MyApp.Contracts.SomeContractService do
  use MyApp,
    aliases: [:campaigns, :contracts, :users],
    private_aliases: [:contracts]
    
  # ...
end   

I may write further about these specific macros in another post 😉

changix

A final example of how we use macros in our main Elixir application is our application changelog. We present new features, bug fixes and improvement to our users with a changelog sidebar (and a blinking badge to draw their attention).

Instead of feeding this changelog from database, we use markdown files with a YAML front matter header.

At first, we used to parse these files at runtime everytime a user triggered the sidebar, then we cached the HTML output in memory in order to improve performance.

But after reading Chris McCord's book, we realized that these markdown files could be considered as source code and be compiled as well (or at least processed at compile-time). Here comes changix: a library relying on macros to provide compile-time changelog features.

cblavier/changix

Compile-time changelog features for Elixir. Contribute to cblavier/changix development by creating an account on GitHub.

GitHubcblavier

Final words

I hope that this post will help you feel more confortable with Elixir metaprogramming and give you enough courage to hack yours; but remember Chris's advice and always be cautious 😅