Monthly Archives: January 2017

Removing the wallet from TRB, first thoughts

This article summarizes my first thoughts about the correct way a Bitcoin client should enable its operator to issue transactions. It is based on my reflexion as well as my experience in broadcasting dozens of hand-crafted transactions for which the bitcoin-core built-in wallet repeatedly demonstrated its ineptitude.

As it currently stands, the wallet is an embedded part of the reference implementation. This is problematic for a variety of reasons, the most fundamental one has been exposed by Mircea Popescu in “How to cut the wallet“.

The job of the node is to connect to others via the Internet, and to exchange information with them. The job of the wallet is to prevent others from connecting via the Internet, which makes the tension quite evident.

Today, a Bitcoin client consists of three main pieces:

  1. The network protocol management part, that connects to peers and exchanges data with them;
  2. The data storage and validation part;
  3. The wallet, which uses the previous pieces to create transactions, sign and broadcast them.

The point of this discussion is to show that cutting the wallet entirely off of TRB is (1) desirable, and (2) requires only a small interface for a subset of (a) and (b)’s functionality to allow much better wallets to be built on top of TRB.

There are numerous reasons for which detaching the wallet is, in my opinion, highly desirable:

  • the job of the node and the wallet are very different (there only exists a functional dependence on the former by the latter);
  • as already stated, their coexistence in the same binary creates a fundamental tension between what the node requires, and what the wallet tries to avoid;
  • there exists no valid reason to keep the code for a wallet embedded right in the code that operates a critical piece of infrastructure, except for the current lack of a sane alternative;
  • the wallet code touches a lot of parts in TRB and adds complexity and bloat to the whole thing, I want TRB generating my addresses as much as I want my e-mail client generating my PGP keys;
  • it does a very bad job at crafting transactions with a lot of shitty heuristics, assumptions and places various artificial and unwarranted constraints are enforced on the operator;
  • external wallets, talking to a small and clearly delimited interface of TRB, could be made as very simple overlays managing a set of keys, with as much control given to the operator as his particular use case mandates;

The required interface for an external wallet, at TRB-level, is quite limited:

  • Query the UTXO for unspent outputs given a set of addresses.
  • Put an arbitrary transaction in the mempool, optionally nuking any conflicting ones[1]

While the latter doesn’t present much implementation challenges, the former is a whole different story, depending on the particular way it is implemented. In my opinion, the correct way to implement it would be to index every single UTXOs with its address (or even all TXOs, soyons fous) in order to be able to efficiently return the set of UTXOs (or TXOs if history’s required) for a given set of addresses. A lighter option would be to allow the operator to specify a list of addresses he’s interested in, and index only those.

The second option is what already exists, but it forces the conservation of the “reindexing” logic,  a list of monitored addresses along with a lot of wallet-related code. The first option allows TRB to remove this functionality completely, serve an arbitrary number of wallets, and get rid of any “sensitive” information, such as the list of addresses an operator is interested in.

This TXOs-indexed-by-address is essentially the index Electrum servers are maintaining, different implementations manage to pack this data in 20 to 50gb (the latter being the index packed pretty inefficiently and keeping not only UTXOs but also partial history for all addresses, up to a certain arbitrary count). The indexing of only a specific set of address would obviously come orders of magnitude cheaper but leave the node coupled to the wallet by having to know which addresses are part of the wallet, and which aren’t, or more precisely those which are ~definitely not~ part of the wallet.

Keeping the history indexed by address doesn’t really make much sense to me, it seems to me it’s the wallet’s job to keep the various receipts until they’re binned or archived (which doesn’t prevent a blockchain scan to be performed, should it be necessary to retrieve it again).

Yet another option might simply be to naively scan the UTXO set for any “gimme the unspent outs for these addresses” request the client gets. As of today the UTXO set is ~1.6gb. Obviously this means the transaction history for addresses can’t be directly queried from the node without a full blockchain rescan.

Whatever the approach, the node simply has to output UTXOs, and receive signed transactions as input for broadcast.

To be continued, comments more than welcome !


[1] Unlike PRB, which somehow thinks whatever’s already in your mempool has precedence over what you’re explicitly telling it to swallow.