Internal Messages
An Elixir message is sent by sending an arbitrary term to a process identifier. This is the Elixir address of any process in a single Erlang virtual machine.
Example: send(some_process, :hello_world)
Within the Anoma node implementation engines communicate using these Elixir messages, which can contain any valid Elixir term (e.g., functions, integers, etc.).
Sending a message to, e.g., the mempool is a message send of any valid Elixir term.
Sending a message to the mempool looks something like this: send(mempool_pid, {:add_transaction, %Transaction{..}})
Because Elixir messages can be an arbitrary Elixir term, it is difficult to translate them directly into a general-purpose message format (e.g., JSON, CSV,…).
Engine Registry
Within a single Erlang virtual machine (BEAM), an engine is registered with the Registry. The registry maps names to process identifiers. This allows any process to look up the process identifier for a process for which it has its name.
Assuming the node executing the below request has node id “dead”, sending a message to its mempool is as simple as follows.
pid = Registry.whereis("dead", Mempool)
send(pid, :hello_mempool)
This mechanism is elegant because it allows us to write communication logic by assuming that a node id exists, and its mempool engine exists.
In the context of distributed nodes, we want to keep this logic the same. Sending a message to the local mempool of node “dead” should be exactly the same as sending a message to a mempool in a cave in Switzerland of node “beef.”
Assuming the node executing the below request has node id “dead”, sending a message to the mempool of node “beef” should be as simple as follows.
pid = Registry.whereis("beef", Mempool)
send(pid, :hello_mempool)
We do not want to care about the physical location of this mempool. All we care about is that we have the same guarantees the BEAM gives us for a local message send.
External Messages
To send messages to a node that’s not within the same Erlang virtual machine (BEAM) Elixir terms are not a viable message type, and thus there will have to be a translation from Elixir terms to “wire messages.” Ray has discussed “ur-messages” to this end here.
By translating plain Elixir messages into a common format, we gain the flexibility to have any piece of software understand these messages, on the condition that interpreting the “ur format” is reasonably simple.
Relaying internal messages to the outside
Assuming that the local node has id “dead”, and the remote node has id “beef”, the diagram below explains how messages are routed across the network.
-
Node “beef” announces itself via a GRPC call to “dead”. This message contains the relevant information to communicate with this node, as per the specs. Based on this information the following things happen.
-
Node “dead” assumes that “beef” has a Mempool, Storage and other engines. For each of these, a proxy process is created (e.g., Mempool Proxy). This processes register themselves in the local Registry as the engines for node “beef.” When any process on the “dead” virtual machine sends a message to, e.g., the mempool, these processes will receive it.
-
Node “dead” creates a “Transport Protocol” for every protocol “beef” announced it supports. E.g., GRPC and TCP. These processes translate ur messages into GRPC requests or TCP messages, and so on.
-
When a local process sends a message to the mempool of node “beef”, the following happens.
- The message arrives in the mailbox of the Mempool Proxy process.
- The Mempool Proxy process translates these messages into the “ur-message” format. Every message this process receives is actually intended for the remote mempool, so they all have to go over the wire.
- The Mempool Proxy process takes any matching Transport Protocol process to handle the message and sends it the ur message format.
- The Transport Protocol knows its protocol and translates the ur message into a proper wire message. For GRPC these are GRPC calls, for TCP these are binary serializations of the ur message.
From the perspective of the remote node, the following happens.
- The GRPC endpoint (or TCP endpoint) receives a
CallRequestorCastRequest. These contain the target engine and the ur message. The message is decoded into a proper Elixir message and sent to the local mempool for processing.
I wrote up this post because it clarified some architectural decision I made to D. I will explain this post and its diagram during the specs meeting as well.
