ART Report: Resource Machine

Resource Machine Stuff
41

To be precise, there are two kinds of storage in play here:

  • resource commitments and nullifiers, which we may want to associate with (coarse-grained) timestamps so that a given controller does not need to store them forever - and these coarse-grained timestamps would need to correspond to sub-trees so that sub-trees can be pruned
  • blob storage, which the RM doesn’t need to be concerned with at all (except to allow reading and writing - but it has nothing to do with resources, commitments, or nullifiers)

The storage engine would be responsible for deleting data, yes, but the RM must be aware of the timestamps insofar as they affect Merkle tree structure. Does that make sense?

In general this makes sense to me, although I think we should note that we will also want the ability to make compact proofs-of-existence for resource commitments in the transparent case.

In the transparent case, we should be able to either:

  • look up the resource commitment in the set of commitments stored by the controller (inexpensive, but requires that the commitment be stored by the controller), or
  • use a proof of existence for the transparent commitment (in cases where the transparent commitment is not stored by the controller anymore and only the Merkle root is - perhaps because it has been deleted after some time as was discussed earlier in this thread)

Strictly speaking we can even do the first in the shielded case, too - it just leaks information (the input commitment), which is why a ZKP of a Merkle proof is used instead.

I’m not sure what @ray might have had in mind there - @ray can you clarify?

42

Thanks. Now I understand the necessity of maintaining an extra merkle tree for transparent commitments. If the large path (1KB for a transparent resource) in transactions is acceptable, then it’s fine. It’s different from the shielded case, the path would be wrapped up in the succinct zkp proof.

The storage of commitments and nullifiers didn’t worry me because they are just 32-byte hashes, not the raw resource data. Even at a deeper depth, they may not be a heavy burden for storage(I could be wrong). Anyway, it’s great to have a deletion feature in storage.

If commitments can be deleted in storage, will nullifiers also be deleted since they have the same storage scale? The simple looking up to check the non-existence of nullifiers may not work anymore.

1 Like
43

It’s sometimes acceptable, yes, but we should also have the ability to check transparent commitments directly if the controller is storing them anyways.

We have discussed some designs for garbage collecting nullifiers, which still require that the nullifiers are stored somewhere - just not necessarily all in one place - and if you spend very old resources, you will need to fetch some old nullifiers and prove that the nullifier you are revealing wasn’t revealed before.

1 Like
44

There still appears to be a great deal of confusion about why there is even a transparent resource machine at all, which I struggle to put into words differently than I already have. Here is another attempt.

The transparent resource machine is not a real thing. It isn’t a product for doing things outside privacy of one’s own computer. It should probably be disabled entirely on public-network nodes. It’s not meant to have features ripped out or put in compared to the shielded resource machine (hereinafter the “real resource machine”), and certainly I would object strongly to editing the specification for perceived transparent-RM concerns. Please do not do this!

Like many things which aren’t real, the transparent resource machine is a vital part of the ecosystem. I created it for a reason, and that reason is traceable, interactive, debuggable work with the resource machine abstraction. With one’s compiler flag flipped to “emit transparent RM”, every part of the semantics can be followed, traced, tested against also-traceable hypothetical system states, in a way comprehensible to ordinary programmers who are not circuit wizards. Applications can be refined with full transparency into what constraints they set down and must abide by, and every failure is explicable.

And then you flip the compiler flag the other way, and emit real RM, already knowing it works.

This implies certain things, like “if the real RM uses a Merkle tree, the transparent RM must use a Merkle tree in precisely the same way”, even if you could imagine ways this could be skipped if transparent RM were somehow real, which it is not. If you’re skipping pieces in the fake, you’re deceiving the user about the real thing. Similarly, it’s important that a proof be a proof, sent to a proof verifier, even if this seems like just duplicating a lot of things several times, because it must follow all the same steps.

Think of it as a RM simulator, if you like, not an RM. It is for simulating the RM semantics, and just doesn’t run on circuits because it’s fake. That’s all.

4 Likes
45

It depends on our definition of the transparent resource machine.

If the transparent resource machine is designed to efficiently execute transparent transactions/applications, it may not be necessary to reuse the complex cryptographic protocols used in shielded RM in the future, as this would make the transparent RM heavy and inefficient. Ideally, transparent RM and shielded RM should only share some abstract interfaces for a “real” and efficient transparent RM. Concrete designs and implementations can be refined to avoid using complex cryptographic protocols in transparent RM. However, currently it makes sense to reuse the shielded protocols in order to make the transparent RM functional.

If the transparent resource machine is an “unreal” engine just in order to execute shielded TXs/APPs transparently as you mentioned(IIUC), the transparent resource machine would need to fully comply with the shielded protocols so that we can have a flag to switch between models smoothly. If it’s our target, I’m a little confused if we still need the explicit transparent RM. Could it be replaced by an interface, like run_shielded_rm_transparently, in shielded RM? This is how we achieved transparent execution of transactions in Taiga, where all circuits can be executed without generating zk proofs anymore. It would also be straightforward to implement this feature in CairoVM since cairo/juvix code can also be executed transparently.

I thought Nockma VM is designed for achieving an efficient and standalone transparent RM. But it seems I was mistaken?

Can I say the transparent RM and shielded RM should behave the same way according to your opinion. If this is true, the current implementation of the transparent RM may not be fully consistent with the Specs and compatible with the shielded RM in some ways.

So what’s the relationship between transparent RM and shielded RM, or other RMs?

2 Likes
46

Here is my perspective:

  • the resource machine is not meant to be only shielded, its privacy properties are meant to be customizable. Otherwise there would be no need for the resource machine concept at all - we can just have taiga and have some kind of transparent debug mode, as @xuyang mentioned
  • the ideal transparent resource machine would have to be the one that is implemented according to the spec and is efficient. For that, from the spec perspective we would have to ensure that the interfaces are:
    • as abstract as possible (to ensure efficient implementation for the transparent rm) YET
    • explicitly specify all of the hard design requirements (required for the shielded rm to be implementable)

In the end both variants implement the same abstract interface (specified in the spec) and the shielded/transparent rms differ only in the choice of the abstractions instantiations (with guidelines for the shielded instantiations options in the spec), not the high level design decisions. That would inevitably result in the transparent vm design be suboptimal because it shares the abstractions with the shielded rm. I would say the current spec doesn’t achieve that fully yet, but it is quite high up there (or so it seems).

If what is known to most as the transparent rm implementation is supposed to be a dummy shielded rm, my question would be: what is the status of the “real” transparent rm? do you @ray @mariari have any plans to implement it or did we have a misunderstanding regarding the need for one (because we do actually need it, see the first point)?

2 Likes
47

It’s a transaction-execution VM; it can also have an RM implemented in it, being the VM that’s already present, that’s just parsimony.

That depends on the properties you want it to have!

Any resource machine should be interchangeable, right? I rely on resource machine properties, write against them, and I support any compliant resource machine. If you rip a Merkle tree out of one out of some idea that it’s “inefficient” (it’s not, really), and my application abuses this fact to do things that would use invalid Merkle roots were it recompiled for shielded-rm, like performing unconditional operations relying on branching on some outside-the-specification fact about the uncommitted resource set at “verification” time, this is no longer true.

Can you take the Merkle tree out of the spec and make it an implementation detail of Taiga/shielded-rm/whatever in such a way that its state never changes whether a transaction succeeds or fails, or what its value is on success? Leaving aside reads entirely, though they matter too.

I think you can’t do this, no matter how many resource machines you write. They’ll all have Merkle trees.

I don’t see why anyone would ever use a transparent one “live” but you aren’t prevented from using the traceable one for whatever. Though we’d want to hammer out the UI differently; I think probably a “disclosed” RM is what’s really best.

48

I think you are mixing two questions here.

One is about the status of Merkle trees: should they be a standard for all RMs or should we have a higher level abstraction which is instantiated by Merkle trees for the shielded VMs, but not necessarily for transparent? If we want to have a higher level abstraction, can we find such an abstraction?

Right now we don’t have such an abstraction and are not actively looking for it. If we change the spec, it isn’t just because it is better for the transparent RMs, it is because we found a way that doesn’t violate the privacy properties of shielded RMs and makes lives of transparent RMs easier. Because ensuring privacy is both harder to achieve and more important than easy efficiency. If it happens that we replace Merkle trees with some abstraction, I would see it as a step towards a better RM abstraction that doesn’t depend on a specific primitive

The second question is about having transparent VMs, no matter with or without Merkle trees. Do we want them? Yes, we do. I’m not sure I can articulate all of the reasons for why, but here are some:

  • some desirable features do not exist in the shielded context, either because they are impossible to achieve (e.g., it seems it is impossible to have a private decentralised CFMM) or finding a way to achieve them takes way too much effort (e.g., out hierarchical storage only exists for transparent resources). Shielded execution is cool, but it is limiting
  • Some applications don’t care about privacy, and they are not motivated to use a very secure resource machine that is slow and inconvenient. The resource machine has many nice features that are not just about privacy. For applications that don’t care about privacy but care about efficiency, it isn’t good enough to simply use a “disclosed” RM.
  • If your goal is not to force your opinion of what is right on the users (which is not gonna work anyway) but provide options (which to me seems to be one of the ideas behind anoma) so that the user could choose, you want to support transparent execution
  • For some applications that level of privacy is unnecessary. Not every application deals with sensitive data and in some cases we might actively want to have the resources being transparent because we want the users to see the data. Could be some kind of gamification application or some utility application
4 Likes
49

Yes, I agree. The main requirement is to check resource existence, not enforce a merkle tree in RMs. At present, the merkle tree is a simple and effective solution for both shielded and transparent RMs. We might have better solutions in the future or already have some candidates.

The reason I questioned the need for a merkle tree in transparent cases is that I discovered we can verify resource existence without it. However, I was convinced to retain it because we still require resource existence check even after deletion.

If the merkle tree is a mandatory design in storage engine, all RMs should comply with it.

2 Likes
50

I think the crux is this:

If the resource machine abstraction is something you write against, then backends don’t matter; you’re using general resource-machine features and you can swap backends however you want. However, this also means that backends don’t matter; you’re limited to things the resource machine abstraction can do. This is where I stand at the moment.

If the resource machine abstraction is just a minimal protocol, then you don’t write against it. You write against specific backends like “cairo-rm” or “elixir-rm” and they provide different features. However, this also means you write against specific backends and maybe the things you write are similar and share parts, but they aren’t the same. This is (maybe? I think?) where you stand at the moment.

Is “privacy” a feature? I mostly think it isn’t; neither is “traceability”, but you can have backends optimizing for one or the other, as we do.

51

Is the Cairo-RM implementation close to this? (correct me if I’m wrong, but my understanding is that zero-knowledge cannot be achieved with the Cairo backend, making it transparent).

52

(is “fast” a feature? I also think it isn’t; ray-vm can execute faster than cairo-vm, I guess, but I don’t think about it much.)

In general I get worried when my half-baked explorations get implicitly promoted to implementation designs because they exist; TRM is a Juvix debugging tool to me. I can put it on the queue for designs! The queue for designs currently looks like:

  • Node multihoming (4 September)
  • pencilled in V2 specs chaos sessions (next week sometime, probably at least 2 sessions)

You and Xuyang will probably hate whatever comes out of that a lot less. (It will probably have a Merkle tree, though. Unless just hashing over the cm set is fine.)

53

(I really don’t have enough context here) My understanding is that we wanted to use Cairo as a backend for Taiga, and the fact that Cairo doesn’t have zk (yet) doesn’t make it an efficient transparent implementation, it is more like an accidentally transparent RM

But really a transparent resource machine simply doesn’t require a proving system like Cairo for proofs because you can use witnesses as proofs directly. From the perspective of prover computations efficiency, this would be infinitely better because it requires 0 computations as opposed to some computations, but we might decide to use a non-trivial proving system for succinctness, for example, trading prover time (0 becomes some) for verification time (as long as it takes more time to read the witness - trivial proving system verification - than to verify a cairo proof)

1 Like
54

If backends don’t matter, there is no point in having different backends. Of course they matter. You can’t really instantiate something without filling in the gaps by definition, and that is what a transparent RM is - a transparent instantiation. If backends don’t matter, you are implementing the resource machine interface that can be used by others to instantiate a concrete RM (including the transparent one), but not a transparent resource machine instantiation.I do think implementing the interface would be a good idea, but I don’t think that is what we were doing so far.

If what you mean here implies that the elixir implementation is supposed to be an interface (not sure i’m interpreting this sentence correctly), then the fact that it actually makes transparent decisions about how to instantiate things (using a trivial proving system, a trivial commitment and nullifier computations, etc is instantiating) makes it an interface that doesn’t satisfy the definition of an interface

Depends on the definition of a feature and the definition of privacy. Privacy and “fast” are strictly speaking not features because they are abstract. “Zero-knowledge” would be a feature, “proof computation less than x” is a feature too. Both can be present or not

We clearly had some misunderstanding here. I think I assumed that it was an implementation because everyone was saying so and nobody was saying otherwise. Calling it a transparent resource machine literally means it is a transparent implementation of the resource machine (because transparency/traceability is a backend feature). If it isn’t, we should stop calling it so.

To make it clear, a transparent RM conceptually is as much of an RM implementation as shielded RM (say, Taiga), the difference is in the backends used.

55

The design is guided by “copy the way the shielded rm (currently cairo-vm) works in a way that lets you compile identical code to both, replacing implementation details with ‘fake’ things, and use it to write applications for the real thing”; writing different applications for it means it should look different (but also, that sounds like it makes application development painful).

2 Likes
56

Thanks for clarifying.

In practice, for many Anoma implementations (protocol adapter) we would need succinct proofs which could be verified (verifies Anoma state changes) on a public ledger, like Ethereum for example. There probably should be a term that describes this type of “accidentally transparent” implementation. wdyt?

A resource machine implementation without succinct proofs doesn’t seem viable based on where the industry is headed (I could be wrong, and woefully ignorant here).

57

I don’t think we want a term to describe “accidentally transparent” RMs, or should rely on any accidental properties. If we need succinctness, we can just talk about [non-accidentally] succinct RMs.

Accidentally transparent RMs would be succinct, but succinct RMs are not necessarily accidentally (or not) transparent.

But if we only care about succinctness and not privacy, it allows us to simplify the instantiation design in other aspects too, for example, the way we compute commitments and nullifiers. I doubt that the fully shielded instantiation (Taiga) with an accidentally transparent proving system results in a somewhat optimal succinct RM.

It sounds reasonable to me

3 Likes
58

Just a quick summary of the discussion we had in slack in the last few days:

Types of the resource components

Initially we defined most components to be finite field elements, but during the last HHH we agreed to use a generic type with elements of fixed size (we also learned that dependent types would be handy here :wink:) to better serve the transparent case without creating problems for the shielded case. This way, we can both have finite fields if we want, and do not restrict the transparent case to operate on finite field elements without the absolute need

Should resource fields be fixed?

Initially this limitation comes from the fact that some proving systems require known-size arguments to be passed into circuits, but with the advances on the proving system front it might not always be necessary even in the shielded case. However, for backward compatibility and being friendly to more proving systems, we keep this requirement. It doesn’t seem to be very restrictive and lifting it is extremely easy, if desired.

It might also be more efficient to have resources of fixed size.

How to store data larger than the component can fit?

You can put the hash of the data in the field and pass the actual data to the logic, checking that the passed data actually corresponds to the hash stored in the corresponding field

Do we expect different proving systems to interoperate?

Right now we assume that each RM decides on up to three different proving systems to use: one for each proof type (RL, compliance, delta). Different RMs (associated with different controllers) may use different proving systems for the same proof type, in that case transferring resources would require controllers being able to verify each other proofs (and maybe something else, to be defined). Right now we don’t consider having multiple proving systems per proof type within the same RM scope

3 Likes