This segmentation proposal would split all of the engines into five groups (called “machines” here):
The message machine (corresponding to the current P2P engine / P2P layer work)
The ordering machine (corresponding to the current mempool, consensus, and execution engine work ~ Typhon)
The strategy machine, including
Solver engine (makes decisions about how and when to do solving)
Automation engine (automatically makes decisions about what consensi to run, who to store data or do computation for, intents to periodically send, etc. based on user preferences)
Interaction engine (handles I/O with the user)
The identity machine, which stores all private key material in this system, including:
Decryption engine (decrypts messages addressed to this agent)
Commitment engine (makes commitments ~ signatures from this agent)
The hardware abstraction machine (HAM), which abstracts the expensive hardware operations:
Computational search (compute engine)
Durable storage (storage engine)
Rough functionality for these engines is as described in the specs. It needs to be explicated in much more detail, but in case you were wondering about what a particular engine does, you might find the associated page there helpful.
Outstanding questions / notes:
I’m not sure about the use of the word “machine”. Alternatively, we could call everything here an engine, and some engines are just composed of other engines (which is fine). Thoughts?
Sometimes the private key material will lie out of control of the Anoma implementation, we’ll need to think about how to abstract this properly (but I don’t think it’s super difficult).
Did I miss any engines?
Is everyone OK with using “Typhon” and “Taiga” as the names for the implementations? (and keeping all of the abstractions as “XX machine” or “XX engine”)
Does this all make sense? Any alternative proposals or concerns?
I feel like “engine” is a bit more descriptive and “machine” creates a nice consistent naming hierarchy, but maybe we don’t need it that much: at first it felt a bit weird to have many engines and just one machine, but bc “machine” in “resource machine” carries a meaningful reference it is actually fine, so I think I like “engine” more
Notes from sync session with @cwgoes & @isheff . Due to the length of the document will keep in the HackMD. Links to the relevant topics discussed are provided below.
Personally, I don’t really like the term “message machine”. I think reducing the responsibility of the P2P layer to just message-passing is a mistake. To me it makes more sense to generalize it a bit more and use something like “networking machine”. However, if you want to stick to something more close to message machine, “communication machine” would be a better name (especially from a distributed systems perspective).
Sounds like we could model this with CSPs for hard constraints/policies + some heuristics/ML for soft preferences/policies
As long as sign and decrypt operations are available by the underlying OS/HSM and we can call them + get the correct return objects, the worst case scenario would be needing to implement one wrapper per platform.
I would like to re-open this discussion (because I am rewriting the highest-level pages in the spec, in particular the ordering machine), because I wish I had both “ordering machine” and “ordering engine” as different, but related terms—which I would like to explain using the following picture.
.
In one sentence, we can have two different but equivalent implementations of the (abstract) ordering machine—one that uses a single ordering engine and one that consists of a set of communicating ordering engines—and these behave “the same” (for the right notion of same).
Note that in this sentence I am using
singular ordering engine for a “composed” engine (aka as engine group in our discussions in the Typhon department)
plural ordering engines for the set of communicating ordering engines of the distributed implementation.
This is compatible enough with the terminology for replicated state machines and copies of the state machine (aka as replicas) could be implemented using replica engines.
While there is no need to make ‘Machine’ automatically the abstract thing, it would make things so much more consistent, in particular if we keep the idea that a bunch of engines together are “just” a composed engine. WDYT @cwgoes@degregat@vveiln ?
What I am actually missing is a definitive picture for what should all be on Anoma agents (which also are called Anoma nodes colloquially?) and what other entities we want to talk about in the operational specs.
I think that I like this distinction, and it fits our current concepts of “intent machine” and “resource machine” (both of which are designed to be compositional) as well. One nit: can we also distinguish between engine types and instances (as I think you may do already in the written specs)? So, in your example sentence:
This would be “one that uses a single ordering engine instance and one that consists of a set of communicating ordering engine instances”.
Or should I read “engine” without modifiers as “engine instance”? Either way, we should clarify.
I want to think about this a bit more though. Other opinions welcome.
I think we should rather talk about some concept of roles, where, say:
a role is a configuration of the Anoma node software to run a particular set of engine instances, wired in a particular way, and configured (as applicable) with a specific configuration
roles, in general, define what subset of messages a particular node will receive, process, and send
roles can be composed, where composing two roles A and B will result in the node processing both messages that A would process and messages that B would process (there may be some overlap, and corresponding efficiency gains compared to two separate nodes)
roles may be changed dynamically at runtime, and may even be reified in order to e.g. share relevant metadata to inform P2P message routing
Is this in the direction of what you’re looking for?
In general I agree with the machine/engine distinction, just have one clarifying question:
In this model, does an ordering engine (I did not find a conclusive definition in the spec after using the mdbook search for the term) ever decompose into ordering engines, or would it only ever decompose into mempool, consensus and execution engine?
If the latter, we would call several composed ordering engines an ordering machine, correct?
The ordering engine type (= ordering machine) could be instantiated either “monolitically” or “composed”, typically on a single “physical” host. One could imagine that even on a single physical host cluster, we have some kind of replication mechanism (within a single trust zone). Indeed, I would propose (if not yet written elsewhere) that the idea of composition of engines in a single trust zone is to be differentiated from the “properly” decentralized distribution across trust zones.
The idea was that the ordering machine is essentially the mathematical model and/or the spec while, on the other hand, every engine has a unique ID and is an interactable entity (in presence of a suitable networking machine implementation). So, while we cannot compose engine instances to get a specification, a set of engines composed (or otherwise put together) might in fact operate as if they were a single engine—sharing the same behavior (as specified by the machine), i.e.‚ being “the same” up to implementation details.
For additional context, the cause of confusion comes from general ideas of composition (on the level of machine/math) and this (to me) rather vague idea of replication in the area of distributed systems, mainly concerned with implementations running on computers. So far we don’t have any counterpart of the notion of distribution planned in the denotational spec (or have we @isheff ?).
If we eventually address this topic, could it be together with a game description where it makes sense to have “communicating” machines (in math), each of which is controlled by a rational actor. For example, by use of a “correlation device”, or something better. @nikete what could we do, e.g., to make validators communicate within a closed consensus system with staking rewards, if we want to approximate the game that is being played (say in the cøomos hub)?
In fact this idea of roles makes total sense, in particular for expository purposes. It would help the reader imagine themselves to take some role A and then we can describe how the system looks for A-agents, i.e., what are the (typical) interactions of A with the rest of Anoma and which response guarantees we have. What would definitively help is a ranked list of roles which we should all cover on the top level page of the operational specs, ideally, in that order. Then, we might also just cut the list.
User, Solver, Validator are my top three candidates for the list.
The only nit picking I could do is that a user certainly is not a bunch of bits and bytes in common English, but the configuration would definitely exactly define the ways a role could interact with the system as you write in the next bullet:
…in the hopes that we don’t forget about too many side channels for observations.
… and if we get things right, then composing roles is tantamount to “perfect” collusion of two role agents in game theory.
Dynamic reconfiguration in the sense of adding and removing capabilities of a specific (external) identity seems rather harmless; the more challenging topics are the creation of new IDs and key change mechanisms for IDs. Interestingly, the only TLA+ stab at the actor model I know (ReActor: A notation for the specification of actor systems
and its semantics) does assume that all IDs are eternal and given at “genesis”.
To be clear, here I was only talking about you (the interactive user, e.g. via CLI) interacting with your node at runtime (e.g. by sending a CLI command) to change what role it is running (e.g. instruct your node to additionally try to solve a new type of intents with a specific algorithm). This shouldn’t require any complex modelling or key change, just clear abstractions and good engineering.
