[Proposal] Anoma Network Architecture

Anoma Research Topics
1

Network Architecture - ART Report

Introduction

  • motivation / angle of approach / models

    • compositional cryptographic identities
      • the external identity / internal identity system
    • actor model
    • heterogeneous trust & heterogeneous preferences
      • people want a subset of messages
      • both trust and preferences vary over time
        • which we want to be able to adapt to
    • heterogenous / sovereign domains
      • different requirements for different domains
        • authentication, protocols, etc
    • message delivery guarantees & semantics

  • models to architecture & protocols

    • how these models lead to the proposed sw architecture & network protocols
    • e.g.
      • cryptographic identities + actor model
        • engines/nodes with cryptographic identities
        • associating network addresses with external identities
        • message routing based on cryptographic identities
      • compositional identities
        • unicast/multicast routing depending on the number of identities
    • overview of the network & sw architecture

Requirements

Design requirements for:

  • messaging patterns

    • unicast/multicast/anycast

  • message delivery semantics

    • expressed per-message/node/domain
    • delivery semantics
      • unreliable
        • best-effort, unordered delivery
      • reliable
        • fifo/causal
        • exactly once

  • Network architecture & protocols

    • privacy/security/efficiency/etc considerations
    • node trust/reputation/measurements
    • sovereign domains architecture
    • intra-node & inter-node protocols
      • message routing, addressing, transport
    • intra-domain protocols
      • protocols/services provided (pub/sub, storage, etc)
      • join/authentication methods
      • topology maintenance/membership protocol requirements
    • inter-domain protocols
      • domain lookup/routing protocols
      • clustering/efficiency considerations
      • privacy considerations

  • Software architecture

    • engines & message passing requirements
    • interoperability, protocol versioning
    • language-independent serialization
    • modular transport system
      • various network transport protocols
      • local transport for local user interfaces (i.e. User actors)
    • measurements
    • allow network/protocol testing/evaluation with simulated network topology & nodes

Related work

  • mention other p2p libraries/frameworks with similar scope (e.g. libp2p, gnunet)

Network Architecture

  • describe overall network architecture
    • identities for addressing & message authentication
    • unicast/multicast messaging
    • sovereign domains
  • describe each network protocol
    • messages types & protocol logic
    • how different protocols interact with / rely on each other

Software Architecture

  • describe engines & message passing, motivated by the actor model
    • unicast/multicast/anycast comm patterns
  • message preferences
    • privacy/security/reliability/ordering
  • describe each engine of the networking machine
    • what network protocols they implement
    • message flows between engines
  • protocol testing & evaluation
    • launch many nodes on the same machine
      • multiple nodes (set of actors) directly connected
      • network transport for simulation:
        • add delay + message loss

Future work

  • open problems/questions

  • improvements necessary

  • future research directions

  • future ART reports

    • inter-domain protocols
      • peer sampling & clustering
      • small-world routing
        • for anycast requests to domains
    • intra-domain protocols
      • topology & pubsub
      • decentralized secure group messaging
    • trust & reputation
4 Likes
4

Sketch of high-level mathematical model

  • We assume an opaque, cryptographic, compositional identity system (as defined here, although rather than citing the specs we should include these definitions directly in the report).
  • We assume a dynamic underlying physical network topology which is constantly changing - nodes (identified by cryptographic identity) are moving between networks, switching IP addresses, even switching physical models completely (e.g. Bluetooth, LoRa).
  • We assume that - at any network time slice - each node has a specific subscription preference function, which can be expressed as a predicate Payload -> Boolean, which specifies which sorts of message payloads they would like to receive. This predicate is also expected to be constantly changing (although not too quickly).
  • We want to provide a generalized primitive Send
    • Send takes:
      • an opaque payload ([]byte)
      • a target external identity (which may be decomposable)
      • routing preferences and constraints (e.g. cost vs latency vs bandwidth)
      • information flow preferences and constraints (exact form TBD, see here for inspiration)
    • The implementation of Send should (across this distributed network)
      • deliver the payload to all nodes who:
        • are part of the target external identity, and
        • are interested in the payload (determined by their preference predicate)
      • … while optimizing for the specified routing preferences as best as possible,
      • … not violating any of the routing constraints,
      • … respecting the specified information flow preferences as best as possible,
      • … and not violating any of the information flow constraints.
  • In general, the metadata gossiped around the network is designed to facilitate this implementation of Send, both from a completeness perspective (the message payloads reaching the right destinations) and from an optimization perspective (the routing and information flow preferences being optimized for as best as possible).
  • In general, routing is a Bayesian inference problem, which can be split into two parts:
    1. Given known information (observed network traffic in the past), infer the probability distribution of underlying network topology states.
    2. Given the probability distribution of underlying network topology states and the routing preferences/constraints, route the message in a way which optimizes for the preferences and does not violate any of the constraints.
1 Like
5

In general good direction/starting points for defining the underlying model, see some of my notes below.

each node has a specific subscription preference function

To be precise: each engine instance

We assume that - at any network time slice - each node has a specific subscription preference

Again, each engine instance.
This assumption seems to be specific to pub/sub, which takes topic subscriptions, and can possibly be further extended with content-based filters, such as tags or other specific attributes of received events

For unicast messages the assumption is that any valid message according to the protocol an engine implements is accepted, however engines may specify a routing scope where an incoming/outgoing message may go to, such as local to the node, a topic, or a domain.
Constraining certain messages to the local node only is an important security consideration that constrains access to e.g. local storage, config, etc engines.

Send takes:

Here we need to separate two kinds of preferences:

  • next-hop node selection by a routing protocol (done by an engine implementing a routing protocol, e.g. onion routed relay, multicast routing for topic-based pub/sub, anycast routing for domain requests)
  • transport selection when sending a message to a specific node (done by the Transport engine)

These preferences & constraints are by default only influence local choice,
however a subset of these may be forwarded along with a message by a multi-hop routing protocol (e.g. message expiry)

This way we can talk about multi-hop routing as recursive Send calls by different nodes until it reaches the destination.

The implementation of Send should (across this distributed network)

Here I would point out that Send is implemented in a layered way:
a high-level Send (provided by the Router) is responsible for sending direct unicast messages between nodes via Transport, and other types of messages (relay, multicast, anycast) the Router hands off to the appropriate protocol implemented by another engine, which have their own Send implementation that selects a next-hop node

metadata gossiped around

We refer to these as authenticated node/topic/domain advertisements, and they can be delivered via various protocols not just gossip, i.e. direct messages, pubsub, or an actual p2p gossip protocol, etc.
Indeed, these facilitate routing by associating node/transport addresses to identities, which the routing and transport protocols use for next-hop node/transport selection.

Next to Send , we should also specify a Deliver primitive, which delivers a message to a local engine instance once it has been received. This takes into considerations delivery semantics specified in the message by the sender:

  • unordered
    • delivered immediately
  • FIFO order
    • delivered once previous messages for the same sender-destination pair has been delivered
    • implemented using sequence numbers
  • causal order
    • delivered once all causal dependencies of the message has been delivered
    • implemented using causal barriers, i.e. listing direct deps of a message in the message header

In case of FIFO or Causal order, when a missing message is detected, these should be requested from the network by the receiver, i.e. the routing protocol that is delivering the message, e.g. the Router for unicast messages, or PubSub/Relay/DomainRouting/etc for other protocols.

Related recent discussions around the Ordering Machine:

  • the Ordering Machine assumes reliable delivery of messages, and it needs causal delivery semantics which it implements on its own
  • however, this is a generic useful primitive not just by the OM that the Network Machine should be able to provide, and once it does, OM can rely on those directly
  • this should be eventually specified in the specs under properties

Related work to transport selection algorithms and p2p software & network architecture:

Automatic Transport Selection and Resource Allocation for Resilient Communication in Decentralised Networks
We begin by detailing the overall requirements for an algorithm for transport selection and resource allocation, and then compare three different solutions using (1) a heuristic, (2) linear optimisation, and (3) machine learning

The GNUnet System
Describes network & software architecture, and P2P protocols
Chapter 3.1 Transport underlay abstraction: summary of the paper above

1 Like
6

Related to the system model, verification considerations:
https://martin.kleppmann.com/2022/10/12/verifying-distributed-systems-isabelle.html

7

:+1:

:+1:

I think in the mathematical model this can be unified - “any valid message according to the protocol an engine implements” is still a subscription preference, just a simple one. Additionally, even for unicast messages, receiving engines may have restrictions on which other nodes they will accept incoming messages from.

:+1: