A deep dive into HEEx compilation from LiveView 1.2

This article is a technical deep dive. You don’t need to understand any of it to use colocated CSS (or LiveView in general). It is published alongside our announcement post for LiveView 1.2.

HEEx is based on EEx and has been built as a custom EEx.Engine.

A HEEx template starts with text and ends up as Elixir code that describes how assigns flow through the template, using change tracking to ensure that dynamic parts are only sent to the client when they change.

As an example, starting with the following template:

<p>Hello World</p>
<.my_component />
Lorem ipsum dolor sit amet

LiveView compiles it into something like this:

dynamic = fn track_changes? ->
  ...

  v0 =
    case changed do
      %{} ->
        nil

      _ ->
        Phoenix.LiveView.Engine.live_to_iodata(
          Phoenix.LiveView.TagEngine.component(
            &my_component/1,
            ...
          )
        )
    end

  [v0]
end

%Phoenix.LiveView.Rendered{
  static: [
    "<p>Hello World</p>\n",
    "\nLorem ipsum dolor sit amet\n"
  ],
  dynamic: dynamic
}

The %Rendered{} struct is the basis for change tracking in LiveView. To render it into HTML, the static parts are interspersed with the result of calling the dynamic function and recursively converting dynamic parts into HTML. When change tracking, LiveView only needs to call the dynamic function. Any unchanged parts are nil and omitted from the diff. You can read more about this in José’s post about LiveView’s rendering optimizations on the Dashbit blog.

The anatomy of an EEx.Engine

To understand the problem we faced with colocated CSS, we first need to understand how an EEx.Engine is implemented. Let’s look at a simple example.

defmodule MyEngine do
  @behaviour EEx.Engine

  def init(_opts) do
    # initial state
  end

  def handle_text(state, meta, text) do
    # called for every text chunk in the template
  end

  def handle_expr(state, marker, expr) do
    # called for every expression in the template, e.g. <%= ... %>
  end

  def handle_begin(state) do
    # called whenever a nesting starts, e.g. <% if @foo do %>
  end

  def handle_end(state) do
    # called whenever a nesting ends, e.g. <% end %>
  end

  def handle_body(state) do
    # invoked at the very end of a template, returns the final code
  end
end

When fed with this text

<p>Hello World</p>
<%= if @foo do %><.my_component /><% end %>
Lorem ipsum dolor sit amet

our MyEngine would be invoked in the following way:

1. init is called and initializes the state
2. handle_text is called with "<p>Hello World</p>\n"
3. handle_begin is called because a do-block (nesting) starts
4. handle_text is called with "<.my_component />"
5. handle_end is called because the do-block (nesting) ends
6. handle_expr is called with the expression if @foo do ... end, where ... corresponds to the code returned by handle_end
7. handle_text is called with "\nLorem ipsum dolor sit amet\n"
8. handle_body is called with the accumulated state

Notice that handle_end must return the Elixir code that is then injected into the if expression and passed to handle_expr.

How LiveView uses EEx.Engine

In LiveView, the change tracking part of template compilation is handled by Phoenix.LiveView.Engine. It is responsible for analyzing which assigns are used to then generate the appropriate code and emit a %Phoenix.LiveView.Rendered{} struct.

All the HTML specifics are handled one layer above, by Phoenix.LiveView.TagEngine. For the Engine, the text <p>Hello World</p> is just a chunk of text - a static part of the template that can be emitted as-is. For the TagEngine, however, it is important to understand the structure of the HTML, for example to warn you if you forgot a closing tag.

To do all that, LiveView includes code that tokenizes the raw HTML text into a list of tokens, representing the HTML:

[
  {:tag, "p", [], %{...}},
  {:text, "Hello World", %{...}},
  {:close, :tag, "p", %{...}},
  {:text, "\n", %{...}},
  {:local_component, "my_component", [], %{closing: :self, ...}},
  {:text, "\nLorem ipsum dolor sit amet\n", %{...}}
]

We can now trace what actually happens when the TagEngine compiles the following template:

<p>Hello World</p>
<%= if @foo do %><.my_component /><% end %>
Lorem ipsum dolor sit amet

1. init is called. We initialize our state and also initialize Phoenix.LiveView.Engine, keeping its state in our state as substate.
2. handle_text is called with "<p>Hello World</p>\n". We start to tokenize the text into tokens: [{:tag, "p", [], %{...}}, {:text, "Hello World", %{...}}, {:close, :tag, "p", %{...}}, {:text, "\n", %{...}}] and keep it in our state.
3. handle_begin is called because a do-block (nesting) starts. We initialize a new empty list of tokens. We also invoke handle_begin on the Engine.
4. handle_text is called with "<.my_component />". We tokenize the text into tokens: [{:local_component, "my_component", [], %{closing: :self, ...}}]
5. handle_end is called because the do-block (nesting) ends. At this point, we know that the tokens for the block have to be complete. So we check for any missing closing tags and then reduce over our token list: For each :tag and :text, we invoke the Engine with the appropriate HTML text, which becomes the template’s static. Any expressions are forwarded to the Engine‘s handle_expr. Finally, we call the Engine‘s handle_end. At this point, the nesting is fully compiled into change tracking code.
6. handle_expr is called. We inject the fully compiled code from handle_end as {:expr, code} into our token list.
7. handle_text is called with "\nLorem ipsum dolor sit amet\n". We tokenize the text into tokens: [{:text, "\nLorem ipsum dolor sit amet\n", %{...}}] and add it to our token list.
8. handle_body is called. We know that the template ended, so we check for any missing closing tags again. Then we reduce over the token list to handle any remaining components and expressions. This is very similar to step 5. Finally, we invoke the Engine‘s handle_body, at which point we have the final code for the template.

Importantly, compilation of nestings happens inside-out. At the point where we iterate over the tokens (handle_end and handle_body), any inner nestings are already fully compiled.

This was not a problem for colocated hooks or colocated JS, because those did not need to modify inner parts of the template. However, for colocated CSS, we need to inject phx-css-* and phx-r attributes, which, especially for slots, could also be in a deeply nested part of the template.

Let’s revisit the code from before and modify it slightly to use colocated CSS itself and include a nesting:

<style :type={MyApp.ColocatedCSS}>
  p {
    color: red;
  }
</style>
<%= if @show do %>
  <p>Hello World</p>
  <.my_list description="My List">
    <:item>
      <p>Item 1</p>
    </:item>
    <:item>
      Item 2
    </:item>
  </.my_list>
<% end %>

If we kept the same compilation strategy as before, we would end up in handle_body with code like this:

[
  {:tag, "style",
   [
     {":type", {:expr, "MyApp.ColocatedCSS", %{...}}, %{...}}
   ], %{...}},
  {:text, "\n  p {\n    color: red;\n  }\n", %{...}},
  {:close, :tag, "style", %{...}},
  {:expr, code}
]

At this point, it’s too late to process the tokens and see that we have a colocated CSS block. The inner part was already compiled, so we cannot inject the required phx-css-* attributes any more.

How we solved it

Now, a solution to this problem would be to move the handling of macro components - which is what we call colocated JS and CSS internally - to the HTML tokenizer and handle them directly in handle_text. This would technically work, but it has a different set of issues. For example, one requirement for colocated CSS is that it must be at the very beginning of a template, not part of a nesting, and it also must not contain any expressions.

To handle those requirements, the code would have to be spread across different checks in the TagEngine and could not live in a single handle_text tokenization pass. We decided that this would make the implementation unnecessarily complex and hard to maintain and instead took a different approach: introduce a separate tokenization and parsing pass for HEEx.

If we built a complete list of tokens from the raw template, including EEx expressions, we could process macro components without intermingling them with the rest of the compilation. In fact, there was already another place in LiveView code that did something similar: the HTMLFormatter.

The formatter needs to build a complete view of the template without compiling any code. To do that, it calls EEx.tokenize/2, which produces a list of EEx-specific tokens:

iex(1)> EEx.tokenize("""
        <p>Hello World</p>
        <%= if @foo do %><.my_component /><% end %>
        Lorem ipsum dolor sit amet
        """)
{:ok,
 [
   {:text, ~c"<p>Hello World</p>\n", %{line: 1, column: 1}},
   {:start_expr, ~c"=", ~c" if @foo do ", %{line: 2, column: 1}},
   {:text, ~c"<.my_component />", %{line: 2, column: 18}},
   {:end_expr, [], ~c" end ", %{line: 2, column: 35}},
   {:text, ~c"\nLorem ipsum dolor sit amet\n", %{line: 2, column: 44}},
   {:eof, %{line: 4, column: 1}}
 ]}

The formatter then uses the HTML tokenizer to further tokenize the text tokens into HTML tokens and rewrites the EEx tokens slightly:

[
  {:tag, "p", [], %{...}},
  {:text, "Hello World", %{...}},
  {:close, :tag, "p", %{...}},
  {:text, "\n", %{...}},
  {:eex, :start_expr, "if @foo do", %{line: 2, opt: ~c"=", column: 1}},
  {:local_component, "my_component", [],
   %{closing: :self, ...}},
  {:eex, :end_expr, "end", %{line: 2, opt: [], column: 35}},
  {:text, "\nLorem ipsum dolor sit amet\n", %{column_end: 1, line_end: 4}}
]

For formatting, it further processes tokens into a tree structure, which allows it to catch obvious mistakes like missing closing tags, mismatched tags, etc. - which we also handle separately in the TagEngine:

[
  {:tag_block, "p", [], [{:text, "Hello World", %{newlines: 0}}],
   %{mode: :block}},
  {:text, "\n", %{newlines: 1}},
  {:eex_block, "if @foo do",
   [{[{:tag_self_close, ".my_component", []}], "end"}],
   %{line: 2, opt: ~c"=", column: 1}},
  {:text, "\nLorem ipsum dolor sit amet\n", %{newlines: 1}}
]

By re-using this tree structure (slightly modified), we were actually able to simplify some parts of the TagEngine substantially, while keeping macro component handling in a separate step that can manipulate the full tree without worrying about compilation details.

In LiveView 1.2, compiling HEEx templates looks like this:

1. Tokenize the raw template into a mixed HTML+EEx token list.
2. Generate a tree structure from the token list. At this step we validate that the HTML is well-formed and process any macro components.
3. Compile the tree structure recursively by visiting each node and compiling it into change-tracked code using Phoenix.LiveView.Engine.

While this means that we have an extra step in the compilation process, and we measured template compilation to be about 5% slower, we find it well worth the effort because of the simplification and maintainability improvements.

Happy coding!

-Steffen