Informal's ongoing experiments with p2p layers for Malachite

Hey all,

Context

Some context here before I describe ongoing work:

  1. We have made Malachite open-source, a flexible BFT consensus engine in Rust. See here for brief announcement and here as a good starting point for architecture.
  2. We have finished implementing a major improvement to the CometBFT p2p layer in the shape of the DOG protocol, announced here.
  3. We have passed the baton for maintaining CometBFT to the Interchain Labs team, announced here.

Given the above, we are shifting core engineering focus in Informal Systems towards Malachite, while also working with a few select teams to improve and specialize CometBFT for specific requirements.

Ongoing work

Now for the ongoing work –

Searching higher performance alternative to GossipSub

We are adapting DOG protocol as an alternative to GossipSub. This is because initial experiments with Malachite (which builds on GossipSub) were very promising, however, we are not confident we are able to push the implementation to its limits given potential limitations in the networking layer. We would like a p2p protocol for both mempool (user traffic) and vote/parts (consensus traffic) dissemination. The two networks may be separated, but p2p performance requirements are similar for both, at a high level:

  • path redundancy should be dynamic and adjust to network conditions: if the peers are connected via too many paths – a parameter – , then some of those paths should not be used anymore, but if it is below, then more paths should be used
  • log(n) latency for disseminating a message

The DOG protocol is the ongoing exploration we’re doing in this regard.

Separating replication from ordering

This is an old idea that took many forms, will describe here in the context of a potential design as part of Malachite we would like to explore.

We would like to explore next a protocol design where the consensus engine relies on an external network for dissemination of user transactions. Roughly, this would work as follows. Let’s call the external network a data provider network (DPN). User transactions go to a specific validator. The validator calls on the DPN to disseminate a batch of these transactions. The batch has an identifier, say id(b). The DPN provides the following guarantees: within a latency envelope – say X milliseconds, a system parameter – it guarantees that 2/3rds of validators will receive the batch with id id(b). We can think of this property as a Quality of Service promise (QoS). The DPN replies with a certificate cert(id(b)). The validators then propose and decide on identifiers of these batches, avoiding the dissemination of bulky block parts that comprise user transactions.

Some open questions:

The DPN network is of interest in the P2P party category. This is similar to a content delivery network, but it’s unclear to me what guarantees can the cert(id(b)) provide. Should the DPN network have BFT guarantees? Not sure. The minimal semantics I can think of is the QoS promise. We have not explored further if liveness in consensus (block building) can be affected by the fact that the DPN can only guarantee 2/3rd of validators to receive each batch.

4 Likes

Thanks for the overview & lovely to see you here! Also, congratulations on the open-sourcing of Malachite and the research progress in DOG! It’s great to see Comet (nee Tendermint) finally getting some P2P research love (after, ah, many years of neglect, at least by AiB… :wink:).

Thanks for the link. The protocol as described in the blog post generally makes sense to me, and it definitely seems like a major improvement on “Flood”. I have one question: is there a reason you chose to target a desired redundancy level (ratio of redundant transactions received), versus simply a desired reliability level (fraction of messages which reach their target)? My understanding of what you want here is basically to “maintain a certain reliability level with the minimum redundancy necessary”, and under dynamic network conditions, the amount of redundancy necessary might change. Targeting reliability instead of redundancy seems like it might be a more direct approach to achieve that goal (if I understand it correctly), although I expect that more metadata tracking / gossip might be required.

From this I interpret that the DPN includes the validators themselves (whose promises would be required in order to guarantee that 2/3 of validators have received the batch with id id(b) within a certain latency window). Is that accurate?

Replies to whom – the original validator who published the batch to the DPN? Is cert(id(b)) intended to be a proof that 2/3 of validators have in fact received the batch with id id(b) (e.g. cert(id(b)) might contain signatures from 2/3 of validators over id(b), or some compressed form thereof)?

In principle, I guess the DPN network could provide BFT guarantees just by running consensus (where cert(id(b)) is the consensus transcript). This probably isn’t what you want (too expensive), so it would be helpful to understand what other properties are desired (e.g. a bound on message complexity), if you have a clear idea of that.

I could also imagine a very simple DPN which would simply broadcast the batch with id id(b) to all of the validators (maybe splitting it up into chunks), and respond with a cert(id(b)) consisting of signatures of 2/3 of validators as soon as 2/3 of validators have responded. Is this along the lines of what you’re thinking of, or do you have a more specialized procedure in mind (and if so, what might it look like, or what are the requirements)?

By the way, we recently published a paper which describes a logical approach to these kinds of service commitments – wanted to flag that in case it’s helpful for your investigations here.

1 Like