Multiple bundles per block vs one bundle per block

Research
1

Multiple bundles per block

This is the current approach, the operator will produce/submit multiple bundles to the consensus chain, and the consensus block may contain multiple bundles of a domain and derive a domain block.

We met several issues with this approach, while we have solution for each issue, they are still not perfectly fixed:

Duplicated transaction

A domain tx may be included in multiple bundles, all domain tx will be deduplicated before being put into the domain block, so the tx is only executed once and the user is only charged for the tx fee once but the tx is duplicated already store in the consensus block thus occupy more storage.

There are also several workaround/fix of this issue:

  • Transaction range: the operator can only include tx that fall in their range into their bundle, the size of the tx-range is set to 1/3 and every operator will have their own random range, ideally when there is no overlay each operator bundle their own shard of tx pool
    • The tx-range is probabilistic thus overlay is still possible and hence duplicated tx
    • If a tx doesn’t fall into the tx-range it has to wait for the next bundle thus latency is increased
  • Memorize produced bundle: the operator memorize the produced bundle, both bundle produced by itself or other operators (through bundle gossip in the domain network), and avoid including tx that was already included in the previous bundle again in their next bundle
    • The bundle gossip also take time thus we can’t ensure the previous produced bundle can be noted by the operator before it producing it own bundle
    • If the previous bundle wasn’t included in the consensus block due to some reasons (network delay/partition, the bundle is too big, etc) then the tx have to wait until next block thus increase latency
      • Or even worse, if the tx depends on a tx included in the previous bundle (i.e. through nonce) the tx will be illegal and the bundle will be invalid without the previous bundle, thus the operator will be slashed
  • Deduplication on the consensus side: the farmer deduplicates the domain tx in all bundles when constructing the block, so even the operator will submit bundle that contains duplicated tx this will ensure there will be no duplicated tx included in the consensus block
    • While deduplication is easy for the farmer, it is hard for other farmers to verify if the deduplication is performed correctly, i.e. verifying every domain tx that committed to the bundle extrinsic root is still exist in the consensus block

Illegal transaction

If a bundle contains an illegal tx it will be consider as invalid and the producer will be slashed. While the domain block is derived from multiple bundles we can’t ensure the final domain block doesn’t contain illegal tx.

For example, the user can submit multiple tx with the same nonce but different contents, and all of them are included in different bundle. These tx is not duplicated tx but still only one of them will executed successfully and the other will be failed thus the user only pay for once while all tx will occupy storage.

Out of limit domain block size/weight

Currently, when the farmer validate bundle, it will check if the bundle’s size/weight exceeds the max domain block size/weight, and when constructing the consensus block, it will include as many bundle as possible (according to the consensus block size and the weight of the submit_bundle). The will cause the final domain block exceed its size/weight limit, and once the limit is exceeded the following domain tx will be skip from execution thus causing a waste of storage.

There 2 approaches in general to fix the issue:

  • The farmer check the bundle size/weight based on max domain block size|weight / expected bundle number, and remove the domain block size/weight limit during execution:
    • The expected bundle number is configurable through the slot probability which is probabilistic, thus the operator may produce more or less bundle but the right number of bundle on average.
  • Add a check to the farmer when it constructing a block, to stop include more bundle for a domain once the limit is exceeded

One bundle per block

With this approach, the operator will produce/submit one bundle to the consensus chain, and the consensus block only includes at most one bundle for a specific domain, thus the domain block is derived from one bundle.

Because the bundle production is probabilistic, multiple bundles may be produced for the same domain block, in this case, the farmer only includes one bundle for a specific domain in the consensus block. The priority of the bundle may depend on their arrival order or their tip if we decide to split the domain tip to the farmer.

With this approach, the domain block is derived from one bundle and the producer can already perform comprehensive checks on the bundle to ensure the domain block won’t have any of the abovementioned issues:

  • Duplicated transaction: the operator should deduplicate the tx within a bundle before submitting it to the consensus chain, otherwise the bundle will be considered invalid by the honest operator and cause the producer to be slashed
  • Illegal transaction: similar to the duplicated tx, the operator needs to validate all tx of the bundle based on the latest domain state to ensure there is no illegal tx otherwise it will be slashed
  • Out of limit domain block size/weight: since the domain block is derived from one bundle, the bundle size/weight limit represent the domain block size/weight the producer can easily ensure these limit. And exact size/weight limit can be configured by the domain creator based on their workload/usage of the domain.

One drawback of this approach found currently is that the bundle production frequency presents the frequency of picking domain tx to the consensus chain, if a domain tx is submitted after the bundle is produced and already submitted to the consensus block, this domain tx will have to wait until next block instead of the next bundle in the multiple bundles per block approach, which will be longer.

1 Like
2

A few notes from our recent offline discussion.

Can also be done by intercepting consensus node gossip as well, but latency might be higher and construction will be less effective because of it.

This will not be a valid bundle, bundle can only include transactions that are legal as of the parent state of the consensus block. Can be fixed by introducing a chain of bundles within the same block as we discussed recently, where next bundle points to the previous, making it possible to include transactions with increasing nonces across a series of bundles.

It depends on the implementation. In case we have bundle headers with hashes of all transactions the cost is similarly trivial.

This is fine, transaction only has to be legal from the point of view of the parent state, no guarantees are made about the legality during execution and operation shouldn’t be slashed in that case.

During last discussion we decided to remove this limit for now, also On bundle weight limits sum has a follow-up discussion about defining bundle weight limit.

The other one is related to network saturation: instead of sending a few smaller bundles we send one large bundle. This can be significantly mitigated by the compact bundles implementation like the one we have for blocks already.

Similarly there is an issue in case bundles are duplicated because they are large and waste a lot of bandwidth, the mitigation is similar though.

3

This is fine from the point of view of the operator but not the farmer, since the user only pays once while all transaction data go into DSN.

I found another issue with the “one bundle per block” approach. When there are multiple domains instantiated, all of them will submit a bundle to the consensus chain.

With one large bundle per domain, depending on the bundle size the consensus block may only be able to fit one bundle, thus all other bundles have to wait until the next block. This is some sense of unfairness. Even if we split some of the tx tip to the farmer to create some kind of auction, the tip is the sum of all tx tip of a bundle instead of an individual tx tip, so whether a domain tx will be included in the consensus chain sooner not only depend its tip but also other tx’s tip.

The issue is mitigated with multiple smaller bundles since the consensus block can fit more number of bundles (probably from different domains), but under the hood, it is an issue of the capacity of the consensus block, there are still bundles that need to wait until the next block. Also, splitting tip may not work well in this case since there are duplicated tx in the bundle (unless we record the tip of every tx).

4

Currently consensus blocks are produced every 6 seconds on average while domain bundles every second. If a domain is popular enough (say it has demand for bundles every 3 seconds on average) in the one-bundle-per-block approach there will be a growing queue of bundles, for this specific domain, to be processed.

In this case, it makes sense to change the configuration for one bundle every 6 seconds, however this results in losing the benefit of having very fast confirmation for transaction processing. In this case, it may be better to follow the approach of based rollups, where transactions are submitted directly to the base layer, but processed elsewhere. The latter will also help with duplication prevention.

More generally, for the duplicated and illegal transactions:

  1. Bundles that are submitted “together” by honest operators and unaware of each other, even if they will be included in different consensus blocks, may still violate these conditions. If the result is slashing, honest operators will be slashed.
  2. It seems that once conflicting bundles are produced, a malicious farmer can force slashing if they succeed in producing a short-term fork. If the farmer is also an operator then it has an incentive for doing so, and it will be able to produce a conflicting bundle.
5

With one-bundle-per-block, the domain should configure its max bundle size/weight based on its demand.

The latency for tx will be increased but won’t make much difference for confirmation IMO, since we need to at least wait for confirmation_depth_k consensus blocks, which is set to 100 currently.

The bundle needs to submit with the ER of the latest domain block to ensure it has properly verified against the latest domain state. If a bundle is included in the consensus block it will make other bundles produced at the same time to be invalid due to they didn’t include the ER for the just-included bundle. And the consensus node will simply remove them from the tx pool, if a malicious farmer includes such a bundle in their block, the block will be considered invalid and rejected by the honest farmer thus no slashing.

The operator won’t be slashed since this can not be avoided completely, but it is indeed a waste of storage.

6

The way I see it, it is actually the opposite. I thought we agreed in last discussion that farmer gets paid in full when submit_bundle extrinsic is submitted. Yes, user pays one, but they pay as much as operators charge them.

7

Yeah, I forgot, but it is still essentially a waste of storage and the cost is transferred to the domain user with a higher storage price while most of them do nothing wrong. Thus it is still something that needs to be fixed IMO.

8

My understand – though I’m not sure if it ever was confirmed – is that the wallet shows that funds are transferred immediately when a block is farmed, and does not wait for the block to be buried deep enough in history. Of course different wallet could implement different approaches. But to clarify that it’s a UX issue, not protocol design.
The above is specifically about the base layer, but it should also hold for domain transactions: a user that submits a transaction to an operator may get an immediate confirmation that her transaction got included, even if the bundle is not submitted yet. Because of shuffling it is problematic to guarantee successful execution in the general case, but at least for simple transfers this is possible.
The latter point was considered in the past, and indeed it may be that the benefit of having multiple bundles per block is low. In that case, as pointed above, I’m not sure if there’s a real benefit in even creating a bundle – a user could directly send their transaction to the base layer, while execution will happen “offchain” by the operators, who later submit an execution receipt to “mark” the state.

9

Interesting point, this approach fixes the bundle tip and “unfairness” issue of the one-bundle-per-block I mentioned above.

Some issues with directly sending transactions to the base layer that I can think of are:

  • Illegal transaction: the farmer doesn’t maintain the domain state thus they can’t verify the transaction properly
    • It can be mitigated by sending the transaction to the operator and letting the operator verify first and then submit it to the base layer, in this case, we also need a way to detect and slash the operator if it acts maliciously
  • Unbounded domain block weight: the weight info of the domain extrinsic is part of the domain runtime even the domain runtime code exists in the consensus state, it is still too expensive to check
  • Every domain user needs to have an account in the consensus chain in order to pay the storage fee or it can be front-paid by the operator like what we plan to do right now