Part 1: Concepts & Case Studies
The challenges of centralized server-based chat applications
A naive understanding is the simple client-server model. In a client-server model, the workload between the client (service requester) and a server (service provider) is shared such that the client makes requests for services or content to the server while the server may run one or more programs that share information with the client. The relationship is asymmetric, the server does the work for the client.
Often clients and servers communicate over a computer network on separate hardware, but both client and server may reside in the same system. A server host runs one or more server programs, which share their resources with clients. A client usually does not share any of its resources, but it requests content or service from a server. Clients, therefore, initiate communication sessions with servers, which await incoming requests.
Clients and servers are run on separate hardware devices. In a centralized server-based model that is common today, many clients make request to a single cluster of servers located in the same data center. For example, every time a user refreshes their Twitter feed, a call to Twitter’s servers is made by the client on the user’s cell phone.
Unfortunately, the architecture of centralized server-based social networks often results in the network operators obtaining and storing a broad range of personal information of their members, including their online behavior, original content and social graph. As an operator, you can use this information to determine the correct ads to feature for given subsets of users and then auction off-space.
Advantages
- Low latency - there is no communication overhead between different nodes in the network, all messaging happens on the same server.
- High performance - the system is likely optimized for high performance including high throughput, meaning it can handle an increased network load.
- High uptime- network outages are infrequent and partitions are non-existent.
- Reliability - the system can perform its expected function when required.
- Simple implementation - building an application on a distributed system is hard and may require additional engineering knowledge. Conversely, centralized applications are often easy to implement with many cloud computing servers competing.
- Scalability - it is easy to rent more cloud space.
Disadvantages
- Control - the central authority of the system has complete administrative control over the system.
- Data ownership - the operator can obtain and store user information including their online behavior, original content, and social graph. The operator can use this information to profit while sharing nothing with the user.
- Censorship Resistance - another downside to server-based social networks is they may also comply with censorship requests from state level actors or ransom attackers: like opportunistic AI agents.
- Switching Costs - unfortunately, there are high switching costs for centralized applications. Indeed, under a censorship attack, other types of network architectures also struggle. If you move from Facebook to X for instance, your social graph does not travel with you.
One interesting example of a centralized server model that has managed to side-step the disadvantages listed above is Signal, a private messaging application. Signal operates the Signal protocol, formerly known as the text secure protocol. This allows Signal the application to provide encrypted messaging for text (including attachments), voice, and video chats. Users can communicate with each other individually or in group chats. Historically, Signal used cell phone numbers as global identifiers for users (exclusively), though they recently rolled out usernames as well. Signal provides client software that allows users to independently verify their contacts. Signal is available for users on iOS, Android, and desktop compatible with Windows, macOS (UNIX) and Linux distributions.
Signal Case study - The ecosystem is moving
Signal’s co-founder Moxie Marlinspike (previously Open Whisper Systems) gave a compelling talk at the 36th Chaos Communication Congress in Leipzig, Germany. In the talk, he made a few key points.
- Software evolves - evolution is incompatible with the static nature of decentralized systems. Swift and timely upgrades are not easy.
- Centralization is not as bad as it seems (you can iterate quickly) - centralizing protocols has been a recipe for success. It’s what slack did with IRC, it’s what Facebook did with email, It’s what WhatsApp did with XMPP.
- By decentralization we really want; privacy, censorship resistance, availability and control - you can get all of these properties without decentralization.
- Open-source software (OSS) plays a larger role in availability and control than decentralization.
- Anyone who is working on decentralized systems today needs to do so without forgetting that the ecosystem is moving.
In particular, Moxie walked the conference attenders through four properties people want when they claim to want decentralized systems.
Signal tackles privacy with private groups, contact discovery, and sealed senders. This means that the Signal service has no visibility into group communication state, membership information, visibility into your contacts (social graph), and no visibility into who is messaging who (Signal uses Zero-Knowledge Proofs!).
Censorship resistance assumes that many things are more difficult to block than just one thing. For example, If you have many servers, it’s more difficult for a censor to block access to those than block access to one server. However, if something is user discoverable, it is also going to be censor discoverable. In a scenario where your server gets censored, and you switch all activity to a different server, there are high switching costs for the user (blow up your entire social graph). Moxie argues that it’s actually more effective to have one centralized service with multiple ingress points. If you have such a service with many users and access to that service gets blocked, it’s easy to spin up another ingress point, a proxy, or a VPN that everyone can switch to, enabling low switching costs.
Availability is raised as a concern Every time there is an outage or liveness failure of a centralized system. While distributed systems can offer theoretical benefits, in practice they also suffer from liveness failures. There is no free lunch.
Control in centralized systems is not strictly worse than in decentralized systems. If you have a federated environment, different services can behave differently. If you are a subscriber of one service and your provider starts to behave maliciously, you could just switch providers but not lose access to the entire network. Control, however, is derived from open-source software. It’s trivial to fork the code base and redeploy the application. Decentralized systems are not strictly better in this regard, and perhaps the overhead of the social layer to coordinate hard forks is not what gives users control. Rather, it is the software being Free and Open-Source (FOSS) that provides control to users.
Perhaps the most relevant point made by Moxie in his talk (which answers why Signal is not some type of decentralized architecture) is that decentralized architectures are difficult to upgrade. If you iterate slowly, you will get outcompeted and become irrelevant. Networks evolve. Also, security threats are in a constant state of flux. Also, the user experience language often changes and evolves quickly to satisfy new user desires. As more money, time, and focus is poured into software development, the entire ecosystem accelerates its collective pace of innovation. Those who cannot keep up perish. The ecosystem is moving.
As we transition to discussing federated architectures next, it’s worth keeping this theme in mind.
Decentralized alternatives, thinking about Federated Architectures
Federated Architectures do not have the constraint of a central authority. The role of the central authority is replaced by cooperative activity. A federated database architecture is a collection of independent database systems united into a loosely coupled federation to share and exchange information. A federation consists of instances (components) and a single federal dictionary. The instances represent individual users, applications, workstations, or other instances in an office information system.
Borrowing the definition and mental model from Heimbinger and McLeod (1985) in A Federated Architecture for Information Management, Federated architectures must balance two conflicting requirements.
- The instances should maintain as much autonomy as possible.
- Achieve a reasonable degree of information sharing or interoperability.
Autonomy
- No instance is forced to perform an activity for any other instance.
- Each instance determines the data it wants to share with other instances. Partially controlled information sharing is a fundamental goal of the federated approach.
- Each instance determines how it will view and combine existing data. Each instance must build its own global scheme to suit its needs.
- Instances must be able to enter and leave the federation.
Information Sharing
- Data communication - instances can share portions of their data with other instances. Exchanging data is a primary operation for a federated network; this requires a mechanism to share data.
- Transaction sharing - a particular instance may not want to share its data directly. For example, if the data is sensitive or has constraints. Transaction sharing allows an instance to share operations or transactions to be performed on its data without directly exposing the actual data to other instances.
- Cooperative activities - in a system of autonomous instances, the only way that the federation can function correctly is by cooperation. Components must be able to initiate a potentially complex series of actions involving cooperation with other instances. Cooperation in this context refers to negotiated data sharing.
Advantages
The main advantage of the federated architecture is that it allows individual instances to remain autonomous and control what data they share, while still enabling some sharing across instances. However, the downside is that you lose the benefit of having all data fully integrated into a single, global database schema, which composite/integrated database approaches provide.
- Low switching - can provide censorship resistance in the worst case with a credible threat to exit.
- Autonomy - operators of instances can customize feature sets and policies that suit their community needs.
- Interoperability - users can discover users from different instances with shared standards.
- Data ownership - users can choose which instances to interact with, including their own.
- Information control - in a permissioned or permissionless setting, programmable disclosure can be baked in.
Disadvantages
There exists a tradeoff between preserving autonomy of each instance vs. having a unified, integrated view of all data across the instance.
- Fragmentation can lead to poor UX - each instance may have custom configurations or customizations that can lead to inconsistent user experience across the network.
- Performance degradation based on communication complexity - coordinating communication across servers can lead to high network load and increased failure rate. Scalability can be challenging.
- Security - each instance needs to be individually secured and protected against attacks.
- Network effects - user discovery can be a challenge, as users from different instances may not be easily discoverable.
- Maintenance and upgrades - each instance may have its own upgrade schedule and software versioning. Also, this can become a coordination problem without a proper control mechanism (governance).
One annoyance or drawback of using today’s Multichat applications is that they typically do not allow for interoperability between applications. What do we mean? Have you even tried to receive a message in telegram or signal and then need to communicate in Slack? There are many of these interactions every day. A tweet is created on X that gets shared on Telegram for distribution, then again downstream to Slack for further dissemination from the relevant parties. Matrix provides one attempt to solve this problem. Although not covered here, Mastodon and the Fediverse provide another interesting case study.
[matrix] Case study
Matrix is a federated network for secure decentralized real-time communication. Matrix is open-source software. It is used for, interoperable chat, Interoperable Voice over IP (VOIP), open communication for AR/VR, and real-time IoT data fabric. Matrix can be used anywhere that needs a standard HTTP API for publishing and subscribing to data while tracking the conversation history. This affords users the ability to communicate with each other across different messaging applications, including Slack, Discord, Signal, WhatsApp, etc. Ultimately, Matrix is a big network of JSON where you can replicate JSON between a decentralized, open network of servers.
Matrix’s mission is to create a global decentralized encrypted communication network that provides an open platform for real-time communication. Matrix can replace the mobile phone networks and e-mail communications.
Matrix was designed with a certain set of values. In particular, the Matrix manifesto states that
- People should have full control over their communication.
- People should not be locked into centralized Silo’s, but instead be able to express their individual preferences to host their communication without limiting who they can reach.
- Conversing securely and privately is a human right.
- Communication should be open to everyone globally as a free, open, and unencumbered standard.
Today there exist several centralized silos like Slack, Discord, Signal, Telegram, Twitter, IRC, etc. Matrix can act as a connective tissue to link these isolated chat protocols together. The servers of Matrix are decentralized. They run a client protocol where clients talk a simple an HTTP (Hypertext Transfer Protocol) API (Application Programming Interface) across the different nodes participating in the network. The aim is that no single party owns or controls your communications. The conversations between nodes are replicated in a Merkle tree on the servers over all the participating instances.
Matrix’s architecture includes; clients, home servers, application servers, and identity servers. The application servers bridge to other messaging protocols. Identity servers map your phone number or Email address to an identifier in the matrix server to discover who is on the network and how to talk to them.
The Features of Matrix include
- Conversation history - synchronize between chat rooms.
- Group messaging - one-to-one communication.
- End-to-End Encryption (E2EE) - Double Ratchet Algorithm.
- Editable messages and reactions.
- VOIP calling for web based Real-time chat.
- Server-side push notification rules.
- Server-side search functionality.
- Read receipts, typing notifications, user presence.
- Synchronized read state and unread accounts.
- Decentralized content repository.
- “Account data” for users per room.
As a source of truth, the Matrix specification is a document that defines all the APIs that the clients use to talk to the servers. Matrix has six stable clients and up to twenty-one clients, including alpha and beta products. This level of client diversity is not dissimilar to that of the Ethereum protocol. The Synapse client maintained by Element is the most mature python implementation.
Clearly, Matrix has much to offer its users. Its decentralized design leaves no single point of failure, as anyone can run their own server. The federated architecture also provides low switching costs and thwarts operator lock-in. In addition, Matrix supports several mediums of communication which include messaging, VOIP, IoT communication, and perhaps its most notable feature interoperability with other chat-based protocols.
While Matrix certainly has much to offer, it’s not without its drawbacks. While Matrix supports end-to-end encryption, metadata like message timestamps can be visible to servers involved in the routing of messages. Client diversity can certainly be seen as a feature for decentralization and resilience of the network, but it leads to a challenging user experience, especially for folks who cannot reason about the trade-offs. This approach of client diversity may also sacrifice discoverability for users based on the particular client’s distribution. Finally, scalability can also become challenging. Synchronizing and replicating state across multiple servers can lead to high latency when communicating compared to centralized servers, which, may degrade user experience.
A Multichat architecture that synthesizes centralized and federated architectures
After reviewing the advantages and trade-offs of centralized served based vs. federated networks for building chat applications, including reviewing case studies of Signal and Matrix, it should be clear that both types of architectures have desirable properties. Is there a way to get autonomy, interoperability, high uptime, censorship resistance, control, data protection and low latency?
Perhaps. But to answer that question we should consider what Mulitchat provides that existing chat applications do not and explore what makes Multichat unique. One unique feature is building on Anoma which is unique in its own right. Anoma provides an operating system for application developers to build on, which affords developers a number of key features. As such, we’ll start by reviewing Anoma’s P2P architecture.
Motivation
Anoma’s P2P architecture can potentially provide many of the features that a chat application would desire. In particular,
- We want a smart chat protocol which will use the lowest-latency physical link to the actual user, e.g. use a LAN link if they’re on the same network
- We want “offline”-capable chat
- We want censorship-resistant P2P so that the chat servers cannot be banned and users don’t need to constantly find new VPNs
In the following section, we’ll review some of our Research on P2P networks by reviewing research by Naqib Zarin (P2P researcher), and the paper he co-authored with Isaac Sheff and Stefanie Roos, Blockchain Nodes are Heterogeneous and Your P2P Overlay Should be Too: PODS. to understand what’s special about Anoma - why you would want to build any chat application. But first let’s start with some basics.
What is a P2P network?
A P2P network is a decentralized distributed system without a central party that orchestrates communication. Since there is no orchestrator, nodes need to communicate directly with each other to maintain the ledger.
- An architecture with a central orchestrator makes coordination easy. All nodes can simply send their request and respond messages to a central node and this node will do the rest.
- Decentralized architectures are more complex.
- For example, where do you send your information? Do you send it to all nodes? If so, this means that both the number of peer connections you maintain and the cost of participation increases exponentially.
- Do we only send information to some nodes? If so, which nodes and where?
- Who do we connect with, and how do we ensure that the information can be received by all nodes even if not all nodes are connected to each other?
A P2P overlay is a virtual network that is built on top of the underlying physical infrastructure (the Internet). The P2P overlay is responsible for managing connections between nodes and ensuring information is propagated efficiently over the entire network.
Two main components to the P2P overlay
- Topology - the topology maybe either unstructured or structured.
- Communication Strategy - describes data selection, types of nodes, and time selection
Let’s start with Topologies. Network topology is the arrangement of the elements (links, nodes, etc.) of a communication network In unstructured overlays nodes are connected to each other randomly. On the other hand, a structured topology is tightly regulated with very specific rules regarding where nodes are positioned in the topology. Nodes in this topology are often DHT (distributed hash table) based. Kademlia is the most popular example.
Communication Strategies have three key components. (i) data selection, which describes what information nodes send, the type of message (entire message or an erasure coding of the message). (ii) types of nodes - who nodes send their information to. For example, do we use flooding and send messages to all neighbors or gossip and send information only to some neighbors? (iii) time selection - when do we send messages? Nodes can decide to relay messages instantly for better information dissemination, or they can also wait and aggregate multiple messages to reduce costs by consuming less bandwidth.
Features
Interoperability
Traditional P2P designs did not prioritize interoperability because it was not necessary. For example, connecting a streaming message application to a file storage system was not a desired property. Using a similar P2P architecture in decentralized systems has resulted in a lack of interoperability. For example, the blockchain ecosystem is unique in that there exist several independently maintained P2P overlays. We want to allow for information to flow between overlays to prevent Silos, but also interconnected overlays can benefit from each other in different ways. For example:
- Services - two overlays can make services available to each other from a network layer perspective (resources).
- State - two interoperable overlays can benefit from sharing a certain state. Messages of a given overlay can use the routes maintained by another overlay if those routes are shorter.
- Communication - benefit from each other by communication coordination, the network traffic can be reduced by “piggybacking” messages that are meant for the same destination. Sophisticated strategies can be developed if a source and target node map to the same IP address, whereby we don’t have to send information over the p2p network.
Scalability
There are three important scalability dimensions to consider.
- Number of nodes - if the node count increases, then we want to maintain performance.
- Volumes of data - we don’t want to overburden particular nodes with large amounts of work or data.
- Communication streams - the network should be able to handle different streams of communication.
We can achieve scalability across these three dimensions by implementing a sharding like scheme such that we split high-volume communication flows and data storage into smaller ones. From a single p2p overlay perspective, this means we have multiple sub-overlays that work together to a common goal.
Proposal: P2P Overlays Domains with sovereignty (PODS)
Source: Naqib Zarin, A brief introduction to Anoma’s P2P layer
Nodes are heterogeneous, your p2p overlay should be too. Each domain has its own overlay topology, and a distinct set of P2P intra-domain protocols, tailored to the characteristics and needs of the nodes in that domain. This new approach provides:
- Performance of typical federated p2p overlays by (i) making data interoperable, (ii) collecting data and statistics about domains, and (iii) using the collected data to make routing decisions
- Security of large or monolithic p2p overlays by (i) monitoring the health levels of other domains and (ii) allowing nodes to join the attacked domain to restore the fraction of honest nodes in that domain.
- Flexibility to accommodate node heterogeneity by allowing for different protocols to be run in different domains.
While we presented a basic intuition about a novel type of P2P architecture, this is clearly not a silver bullet solution. We look forward to feedback and collaboration from the community, including collaborating on future Anoma Research Topics (ART)s relational to the networking architecture, P2P overlay, and trust (h/t @jonathan).
…
