A History Of Litecoin MWEB Development
Updated: May 6, 2022
Litecoin's latest upgrade is a masterpiece in the making. Over the past several years, the lead litecoin developer on MWEB has provided monthly updates on the Litecoin MWEB upgrade and progression. The dedication to such a monumental undertaking shouldn't be taken lightly. This was a grassroots funded project from the litecoin community alongside matching funding from Charlie Lee the creator of Litecoin and Litecoin Foundation Director. We are approaching the final progress update as the Litecoin MWEB update is ready for activation.
The litecoin community and people of the world will be able to use best money on the planet after the upgrade.
Below are the monthly updates from David Burkett, the lead Litecoin MWEB developer. If you want to catch up on the technical development and inter-working of MWEB start reading:
December 2019
Announcement
Hi everyone!
I’m David Burkett, the developer of Grin++. I’ve been working with ecurrencyhodler, Charlie, and others for several months now to design a Mimblewimble extension block so that LTC users can have confidential transactions and more fungible money.
I spoke a bit about our progress at the LTC Conference last month, but for those who are unaware, we’ve published 2 new LTC improvement proposals (LIPs):
LIP-0002: lips/lip-0002.mediawiki at master · litecoin-project/lips · GitHub 178 This describes extension blocks, and how they can be implemented and activated in LTC.
LIP-0003: lips/lip-0003.mediawiki at master · litecoin-project/lips · GitHub 150 This is all about Mimblewimble on LTC.
After the conference, much has happened. The first big announcement is that the LTC Foundation has graciously decided to fund my efforts to implement the MW extension block, and to continue working on Grin++! More details about that will be announced shortly, but I want to thank you all for the opportunity.
Since that was announced, great progress has been made. On the Grin++ side, I’ve implemented payments via TOR. Mimblewimble requires interacting with the receiver to build a transaction collaboratively, so by using TOR for that interaction, we’re able to use familiar-looking addresses, a simpler set-up, and much better privacy.
Original proposal to use TOR: https://github.com/mimblewimble/grin-rfcs/pull/24 34
Grin++ Implementation: Transacting via TOR hidden services by DavidBurkett · Pull Request #89 · GrinPlusPlus/GrinPlusPlus · GitHub 34
Also, I’ve performed the first ever (AFAICT) pre-broadcast Mimblewimble CoinJoin. For those familiar with Mimblewimble, you may know that at the time transactions are included in a block, they are already joined with all other transactions in that block. The Achilles heel of mimblewimble privacy though, has always been that transactions are broadcast before they’ve had a chance to be joined with other transactions. That means nodes monitoring the network can see the original input-to-output links of most transactions. Sending a transaction directly to a CoinJoin server before broadcasting is one of many different techniques we can use to combat that.
Original coinjoin proposal: Yo Dawg, I heard you like CoinJoins - Grin 9
Block with first coinjoined transactions: GrinScan - Block 456,188 16
On the LTC side, I started by familiarizing myself with the existing code by completing the LTC Dev Training Session. This meant starting with the latest bitcoin source, and making all modifications necessary to get it to sync.
Dev Training Session: Litecoin Developer Training Session 1 91
My BTC fork which now fully syncs the LTC blockchain: GitHub - DavidBurkett/bitcoin: Bitcoin Core integration/staging tree 39
After that, I got to work re-designing the Grin++ codebase in a way that will allow LTC to reuse Grin++ with minimal modifications. The design is beyond the scope of this update, but I’ll formally write-up the design sometime over the next few weeks and link it here.
Going forward, I will be providing monthly progress updates to keep everyone in the loop on what progress has been made. They’ll likely be much more succinct than this one, but I thought it would be good to take a few extra minutes with this first update to let everyone know who I am, where we’re at, and set some expectations for the future.
Thanks for reading, and thanks again for all of your support!
December 2019
December Progress:
After months of planning, development of the mimblewimble extension block has officially started! My efforts this month focused mostly around restructuring the core logic that will be shared between Grin++ and LTC. This involves all of the logging, serialization, crypto, error handling, and common data structures (headers/blocks/txs).
Most of this was already written for Grin++, but since some parts were hurriedly coded on a tight time schedule, I’m using this opportunity to go over every line with a fine-tooth comb. As part of this, I’m standardizing patterns that were originally followed inconsistently, backfilling missing tests, and adding better documentation.
The LTC priorities for January will be:
Determine build method - Since the code will now be spread across a few repos, a slightly more complex build process is necessary. I’m still figuring out the best way to handle this, but am currently leaning toward a simplified version of libbitcoin’s build system (https://github.com/libbitcoin/libbitcoin-build 26)
Define all LTC models - Determine the exact fields and serialization format of the headers & kernels (including signatures)
Move the database implementations over to libmw-core, cleanup to match new standards, and add full tests.
I also will be working on a faster sync mechanism for Grin next month, which LTC will directly benefit from.
For those technical-minded readers who want to follow along as changes are made, feel free to “watch” the following repos:
GrinPlusPlus/libmw-core 2
Contribute to GrinPlusPlus/libmw-core development by creating an account on GitHub.
GrinPlusPlus/libmw-ltc 2
Contribute to GrinPlusPlus/libmw-ltc development by creating an account on GitHub.
January 2020
January Progress:
The biggest news from January is that I’ve found a way to support non-interactive transactions in Mimblewimble 42! The biggest difficulty with using MW is the need for sender and receiver to communicate, which requires receiver to be online when sending. My proposal, along with an updated version of the write-up that will be released soon, eliminates that need. This removes a major UX hurdle, limits long-term maintenance requirements, and supports receiving via cold-storage, making hardware wallets easier to support.
On the development side, the build process has been determined for libmw, and local builds are working for libmw-ltc (checkout libmw-core and libmw-ltc to the same parent directory, and you should be able to build libmw-ltc). I’ll setup CI/CD in the next month or so, but the important part is multi-repo local builds are working.
I’ve also built out a robust Database framework with transactional capabilities to support atomic updates across multiple tables, and have implemented the block database querying and updates, which is coin-agnostic, and has been partially tested using LTC-specific header and block models.
The security audit results are back from Grin++, so I’ve applied all fixes to Grin++ and libmw, and will be awaiting final review from the auditors. The audit turned out to be a humbling lesson in just how complicated C++ really is. I’ve learned enormously as part of the process, and the Grin++ & libmw codebases are significantly better as a result. A huge thanks again to the contributors in the Grin, Beam, and LTC communities who made that audit possible.
On the Grin++ side, we’ve had a successful planned hardfork, our pre-hardfork syncing issues have been resolved, and Grin++ 0.7.5 is available now which has turned out to be the most stable release yet. For the first time since Grin++'s creation, the support channels are finally quiet, which means more time to focus on actual development. As long as things remain quiet, and I don’t have to focus on putting out fires, I expect an even faster development pace over the coming months.
The priority for February will be to implement the consensus rules for the LTC EB, including all validation and a full suite of tests. This is the most important part of the code, so it will be time consuming to make sure all of the details are correct, and the code has full test coverage. Once that’s complete, I will work on the API for the extension block, so we can start integrating libmw into the existing LTC codebase.
I will also focus on getting the new one-sided tx proposals thoroughly reviewed, and if no major security concerns are found, I will create an LIP for community feedback.
February 2020
Happy Leap Day!
February Progress:
The non-interactive transaction proposal has gone through a few design iterations this month. I believe we’ve finally got a design that resolves the issues found with the first write-up, and I’ve turned that into a LIP (https://github.com/litecoin-project/lips/pull/13 59). I suspect it will still take some time before the LIP has had sufficient review to be considered safe enough to accept, but I’ll continue to keep one-sided transactions in mind as I develop.
Aside from the LIP, I’ve made good progress on the validation rules, but there’s still a lot more to do. I’ve made some modifications to the original kernel design to support the ability to soft-fork in new features in the future. I’ve also started to build out the merkle mountain ranges (MMRs), which are a data structure we use to commit to kernels & outputs. Once the MMR logic is finished, I should be able to get back to the block validation logic. I didn’t get to work on the Node API like I had hoped, but that’s something I also expect to get to next month.
I’ve so far been very hesitant to give exact dates on when things should be finished, because writing blockchain software is difficult, time-consuming, and unpredictable at times. I didn’t want artificial deadlines to force us to rush through parts of the code and introduce defects or security vulnerabilities. Having said that, I think it’s finally time to commit to the first major event…
End of Summer: MW Testnet Launch - This will include all block & tx validation rules, basic p2p messaging, transaction pool, syncing, and the ability to mine blocks. This will NOT include a usable GUI wallet for casual users to test it out. Transactions will likely need to be created manually at first, or via a cli or automated tool.
March 2020
Hi everyone! It’s that time again.
March Progress:
I’ve finished up the Merkle Mountain Range (MMR) logic, and wrote some good tests around that. When originally writing Grin++, this was one of the most difficult pieces to code, and ended up being one of the messiest parts of the logic. It consistently led to many frustrations and headaches for myself. Fortunately, I learned from those mistakes, and was able to design a much cleaner implementation this time.
That was the final piece I needed in place to write the validation rules. The first pass at header and block validation is now complete, but is so far only minimally tested.
The next few months will be all about integrating with the existing litecoin codebase, and lots and lots of testing. Don’t be alarmed if the updates for April and May are shorter and less detailed, as this will be one of the most tedious and least exciting parts of the project. It’s also the most crucial part though, so lots of time will be dedicated to making sure everything is integrated correctly.
Thanks again for your continued support. Stay safe, stay home, and don’t forget to wash your hands. Happy Quarantine!
April 2020
April Progress:
I’ve built out a functional testing framework that builds valid headers, blocks, and transactions. I’ve now got some (mostly) complete end-to-end block validation tests.
I also began integrating with the litecoin codebase, starting with the ConnectBlock logic (See: https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_6):_The_Blockchain#Connecting_a_block 21). This is the part where blocks are validated to ensure valid UTXOs are being spent, no double spends, etc, and is also responsible for actually adding the blocks to the chain.
I haven’t created the repo for this portion yet, but will try to get that out on github sometime in the next few weeks, once I’ve had the chance to clean the code up a bit.
There’s a few different directions I could go once I finish testing the ConnectBlock logic, but I’m still deciding what makes sense to tackle next. All I can give at this time is a high level plan for May, which involves continued integration with the LTC codebase, and lots more testing.
It’s been a few months since I gave an update on the Grin++ side, so for those interested how that’s going, there’s been some big news recently. David Tavarez (https://github.com/davidtavarez 17) has completely taken over the UI development, so I can focus solely on the C++ node & wallet logic. He actually rewrote the entire UI (https://github.com/GrinPlusPlus/GrinPlusPlusUI/tree/1.0 23) and it’s much nicer than it was before. Things are so stable now that we actually have a release candidate for v1.0.0, what we’re calling the first non-beta version of Grin++. That release has been tested with the help of several Grin community members, and will be pushed to all Grin++ users this weekend.
Happy Friday Everyone!
May 2020
May Progress:
I’ve finished implementing the AcceptBlock and ConnectBlock integration logic, meaning that when a mimblewimble block is received, it will be fully validated, added to the chain, and the UTXO set will be updated. This ended up requiring a lot more rework than I hoped, since the approach I originally took on the mimblewimble side for adding, validating, and removing blocks didn’t line up well with the existing code’s approach to handling the UTXO set using “views” and “caches” (See “Data Access Layer and Caching” here: https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_2): 18Data_Storage#The_UTXO_set.28chainstate_leveldb.29).
The logical next step was to work on DisconnectBlock, which will remove a block from the chain in the case of chain reorgs. However, I chose instead to start working on the mempool, with the hopes that we can soon start mining Mimblewimble blocks. Being able to mine valid mw blocks will allow us to test the end-to-end flow of including mw transactions in a block, validating those blocks, removing the transactions from the mempool, adding the blocks to the chain, updating the UTXO set, and flushing everything to disk. There’s a lot that could go wrong there, so the sooner we can get everything working together, the smoother the rest of development should go.
This is all subject to change, but for now, here’s the plan between now and the end of Summer testnet launch:
June: Wrap up mempool logic, and start mining valid chains with mw extension blocks, making sure all pieces work well together.
July: Initial block download. The way initial block download is handled in mimblewimble is quite a bit different than how litecoin does it. In LTC, every single block from the genesis block is downloaded and validated in order. With mimblewimble, we don’t need the whole history, but instead only parts of each transaction, and all of the unspent outputs. I’ll need to find a way to make those two very different syncing approaches work smoothly together.
August: DisconnectBlock logic (for chain reorgs).
September: Testnet activation logic and launch of the first mimblewimble testnet. Then start fixing everything I broke
June 2020
June Progress:
The existing LTC mempool logic ended up being quite a bit more complex than I predicted, so we’ll need to revisit this area after the testnet launches. For now though, a very minimal implementation supporting MW transactions has been written.
In addition to the mempool changes, code was written to support mining extension blocks, although there are a few edge cases left to handle, and much more testing is needed.
I’ve had concerns about the way we were storing mimblewimble block data in a separate database. It was originally designed this way to be a clean separation from the existing code, in order to facilitate merging future bitcoin commits. Having separate databases is generally a bad idea though, because we lose the ability to make atomic updates, which means the 2 databases could become out of sync. This can lead to a whole host of problems, potentially even ones that are exploitable by remote attackers.
Because of these concerns, I decided to spend some time modifying the code for serializing and deserializing MW blocks & transactions to disk. I was able to take advantage of the groundwork that was laid down as part of the segwit enhancement, to cleanly support serializing the additional data without making major changes to the existing block storage format. As a result, upgraded nodes can successfully save extension block data to disk the same place they’ve always saved blocks, without having to introduce an additional database. A side effect of these changes is that it was relatively straightforward to add support for sharing mimblewimble transactions over the p2p network, which is the first step toward July’s goal of handling MW data as part of the Initial Block Download.
The high-level plan for the rest of the summer remains the same:
July: Initial Block Download
August: Chain reorg logic
September: Activation logic and testnet launch
July 2020
July Progress:
The focus this month was on the Initial Block Download. Before I can detail the progress made, I need to give some background info for those not intimately familiar with mimblewimble.
The biggest innovation behind mimblewimble is that, in order to verify the chain, you just need to know all of the unspent coins/outputs, and a small part of each transaction called the “Kernel.” These 2 things together are called the “chain state.”
In bitcoin/litecoin, each block header uses a merkle tree to commit to only the transactions in that block. Since we don’t want to require everyone to download all old mimblewimble blocks, or to know about all old, spent outputs, we use a different structure to commit to the transactions. Each mimblewimble header commits to the root of 2 different Merkle Mountain Ranges(MMRs). One represents all historical kernels up to that block, and the other represents all historical outputs/coins. Merkle Mountain Ranges are a different sort of tree that supports “pruning”, which means we can verify the root of the structure without knowing all of its members (called leaves). For an in-depth look at how this works, I recommend reading https://github.com/mimblewimble/grin/blob/master/doc/mmr.md 18
Since old blocks don’t need to be downloaded, Initial State Download (ISD) is a more accurate term for the initial sync process that I worked on this month. To facilitate the ISD I ended up defining the following P2P messages:
GETMWSTATE - This simply consists of the hash of a header sometime in the recent past. We’ll call this the “horizon” header. A new node, after syncing most of the canonical (non-mimblewimble) blocks in the chain, sends this message to a single peer, requesting the mimblewimble chain state at the time of that block.
MWSTATE - The response to a GETMWSTATE message. This contains the entire chain state, which is everything you need to validate the extension chain up to the point of the horizon. The chain state consists of:
kernels - All of the kernels in the chain, in order.
utxos - All of the outputs that had not yet been spent at the time of the horizon, including their associated rangeproofs, in order.
unspent_bitmap - A compact structure that indicates the MMR leaf indices of the provided utxos.
parent_hashes - Some additional metadata that allows you to verify the MMR root without knowing all of the past, spent outputs.
Quick sidenote - This design relies on one peer providing quite a bit of data at once. After launch, it would be ideal to parallelize the sync process using something like https://github.com/jaspervdm/grin-rfcs/blob/pibd/text/0000-parallel_ibd.md 3 or https://github.com/BeamMW/beam/wiki/UTXO-set,-horizons-and-cut-through 5
All of the logic for serializing and deserializing those messages, as well as choosing when to send them has been implemented. I also implemented the logic for verifying the kernel signatures, and the kernel MMR roots.
Unfortunately, I was short on time this month due to a Grin hardfork that ended up requiring an enormous amount of effort to support in Grin++. As