Network Subsystem Overview & Plans

The Network Subsystem consists of the following engines.
The Future plans subsections contains plans not in the specs yet.

For now this is just a rough overview with lots of context/details missing.

Registry

The Registry stores advertisements that arrive from the network,
such as NodeAdvert, TopicAdvert, DomainAdvert,
which provide the necessary information to connect/subscribe/join these.

Router

The Router is responsible for routing messages between local engines and remote nodes
via a Transport protocol, which is selected by the Router for outgoing connections.
It maintains a session for each remote node that may use multiple transport connections.
Currently this is a simple per-node message sequence number that allows detecting a lost message
when one connection times out and switching to a new one.

Future plans

The Router should be able to support asynchronous encrypted sessions
via an asynchronous key agreement protocol such as X3DH or PQXDH by Signal,
via multiplexed underlying transport connections.
It requires the use of prekeys, which is an input to the DH key exchange.
Such prekeys would be advertised via the NodeAdvert for the initial connection
and later updated by direct messages.

Once this async session setup is in place,
we can add store-and-forward relays that enable asynchronous communication in the system.

Transport

Transport opens and maintains connections to remote nodes via various transport protocols with different guarantees.
Currently this is done via QUIC and TLS.

Future plans

Add more transport options such as Tor and a mix network,
as well as censorship-resistant obfuscated transport protocols or proxies.

Add local peer discovery protocols for LANs to be able to discover nodes on the local network and connect to them directly.

PubSub Topic

A Pub/Sub topic has a set of publishers and subscribers where the publishers can publish events,
and the subscribers can read them.
The Topic ID is a public key and allows the topic owner to designate publshiers.
It is backed by a Blocklace data structure
that is a CRDT that stores a causal DAG of events as an append-only log,
and gives a partial order of events.

Future plans

The partial order of events given by the Blocklace can be extended to total order
by an additional protocol that can be enabled for the topic if needed
using the Cordial Miners (or similar/improved) protocol.

Based on the Blocklace CRDT, which is an append-only log,
add more complex CRDT types, such as Set (for the membership protocol).

Add epochs that allow garbage collection of old events that are not relevant anymore.

Add fine-grained permissions for publishers based on the type of operation in the event
(e.g. some publishers may send only ACKs)

Add end-to-end event encryption which allows relays that help forwarding messages but they cannot read them.
This would be a simple per-topic symmetric key to start with.

Storage

The distributed storage protocol stores immutable objects split up into chunks in the network.
An object is stored as a Merkle tree of chunks,
where the chunk size is a parameter of each object and usually a few MB in size.
The object ID is derived from the root chunk ID.

Objects may become mutable by sending an updated version to a pub/sub topic.

Objects may expire by specifying an expiry timestamp.

A Storage Commitment signed by a node allows nodes to commit to storing an object for a given time period.
These commitments are then sent to one or more pub/sub topics.
Each node stores a local database of storage commitments it knows about,
which allows building a local index of which object is stored by which nodes in the network.

Future plans

Connect object expiry to pub/sub topics epochs, as another option next to absolute timestamps.

Access control for object requests based on domain membership and/or access control lists.

Domain

Future plans

A Domain has a set of members and optionally a set of public nodes that are advertised in a DomainAdvert
that can be used to join the domain and to send external requests to it.

Join requests are approved or rejecetd based on an authentication protocol specified in the DomainAdvert,
this can be none for public domains, a shared key, membership certificate, ZK proof, etc.

External requests are used to request information from the domain by non-members
and may require an authentication token.

The domain consists of a set of CRDTs each backed by a pub/sub topic.

Two CRDTs are present in all domains:

  1. The set of members:
    each member updates its own entry continuously,
    and advertises their NodeAdvert and topic subscriptions in the domain.
    The maintenance of the members data structure also serves as a membership protocol,
    and the topic subscriptions of each member is used for clustering members in the domain
    based on their subscriptions.

  2. The index of CRDTs:
    The set of top-level CRDTs in the domain is maintained in an index that is also a CRDT.

2 Likes

Thank you for this write-up. Do you by chance have a diagram that shows how these network subsystems relate to each other?

something like this?

image

1 Like

yes it’s a good approximation, i’m going to make more precise diagrams later

1 Like

I would appreciate, if you actually could make a short sequence of diagrams, in analogy to Pokémon evolution

  • what’s the baby p2p
  • what’s the first level of evolution
  • what is the final evolution

… ideally with a “change log” of features added (and for which reason/functionality/…).

yes i have a rough idea for a roadmap already,
right now i’m writing down a more detailed version of the architecture overview in the ART
and then make a more precise roadmap how to get there.

a rough version of the roadmap / evolution per component in the order of priorities is the following:

  1. pubsub: permissions, epochs, content-based pubsub, total ordering optionally
  2. storage: permissions, mutable CRDT objects via pubsub, incentives/payments
  3. domains: membership & dissemination protocol, authentication, external requests
  4. router: async encrypted sessions, store+fwd relays, causal delivery
  5. transport: additional transports (tor, mixnet), LAN discovery
1 Like

One thing that would be helpful for system design would be a list verifiable properties and constraints induced by primitives on the network layer.

One example:

  • Causal relations between messages enable us to construct and verify partial orders of messages.
  • We can not do better than partial orders at this layer.
1 Like

another point that we touched on and i should write a separate post about is ordering & dissemination requirements for CRDTs and comparing that to the requirements of total order consensus that we already have, to understand better the differences & similarities

another topic to expand on is the relationship between physics, causal message DAGs, and service commitments

1 Like

Thank you for the write-up!

In the future could / would we replace this with hash backreferencing?

What does a local peer discovery protocol specifically require (that a “non-local” peer discovery protocol wouldn’t provide)? Are you talking about IP address scanning / IP-layer LAN advertisements, something like this? Is there prior art we can draw on here?

Can I interpret “public key” to mean “external identity” in the sense used in the Anoma specs (i.e. that general type)? A particular advantage of generality here is that it would allow us to have topics which are managed by resources (and the regular state machine).

This link doesn’t work.

This link also doesn’t work.

Do we need to do this in a special way for pub/sub, or can data in the event itself just be encrypted?

Other questions related to pub/sub:

  • How do we currently organize dissemination (of events)?
  • How do we track subscribers?

In that case, how are object IDs created and tracked? How would ordering conflicts be handled?

we have both, for different purposes:

  1. per-sender sequence numbers per sender allow simple and efficient detection and replay of lost messages and feed into the encryption algorithm thus unencrypted
  2. causal dependencies of messages are hash references, these are part of the message and encrypted

here I meant IP layer LAN protocols indeed, like using IP multicast or mDNS or similar mechanisms
that we can use to discover local peers that run anoma, and establish a direct connection with, and do a private set intersection to determine the common set of domains both nodes are members of, join the domain and participate in the p2p protocols (pubsub, storage, crdt sync, etc)

yes, we should represent them as external identities in the specs, here i was purely thinking in terms of cryptography when i wrote this

it’s a bit more complex than “just encrypting” the event, here i assume you mean all events in the topic with the same pre-shared key (PSK) which works but not the most secure, at least a nonce that is derived from a per-publisher sequence number would be needed.
dependending on security requirements we might even want to use a double ratchet or similar encryption algorithm that also prevents members who join later to see earlier messages, in case that’s a requirement for the topic (e.g. secure chat).

For now each publisher’s node tracks their direct subscribers, and sends messages directly to all direct subscribers. Thus there’s no multi-hop dissemination, it’s a simple star topology.

Later we’ll have a more sophisticated protocol for this that uses subscription clustering of domain members to build efficient dissemination trees between them, i.e. connect to members who share similar subscriptions.

You mean for mutable objects?
To obtain the latest state, one has to join the pubsub topic associated with the CRDT object, and read the latest state after applying operations, or request such state from another node who subscribes to it and can share the latest state.

fixed

I see. Makes sense, looking forward to seeing this all spelled out.

Aye, makes sense - this would be awesome!

:+1:

Good points. It does seem like different approaches will be necessary in different scenarios.

I’m trying to understand the relationship between “immutable objects split up into chunks in the network” in the distributed storage protocol and “mutable objects” as you discuss here. Are these “the same kind of object”, or just the word “object” used in two different places to mean two different things? As I understood, for example, a Merkle tree of chunks, the root of the Merkle tree would be dependent on the data, so an object ID derived from the root would change if the data changed. Are you talking about a different kind of “CRDT object” here?

yes, we have different kind of objects (perhaps we can come up with better terms avoid mixing them up):

  1. Immutable objects stored as a Merkle tree

  2. Mutable CRDT data structures. These are modified using operations sent to pubsub channels. These mutable data structures can reuse immutable objects, e.g. a CRDT set or append-only log of immutable objects. Also, a snapshot of a mutable data structure can also be stored as an immutable object for quick sync or sharing.

1 Like