[BTC-dev] Raw Transactions Back-Port Exercise

Wed Apr 25 03:19:43 UTC 2018

URL: <http://therealbitcoin.org/ml/btc-dev/attachments/20180424/attachment.txt?sha1=dd53b69da43c81e332c0b78653b6d4548bcbc85d>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

       ..::[ Raw Transactions Back-Port Exercise ]::..

           [ Date: 2018.04.16                    ]
           [ Block Height: 518487                ]
           [ Author: mod6                        ]

0x00: [ Introduction ]

  The Real Bitcoin stems from the original version v0.5.3 [R.01].  The commonly
known Raw Transaction tools were added much later in version 0.7.0 by prb
[R.02], post Satoshi Nakamoto leaving Bitcoin.

  A set of Raw Transaction tools for The Real Bitcoin is a necessity.  What is
detailed in this post are my findings and summary of the attempted out-right
back-port from the forward version 0.7.0 into The Real Bitcoin, and why The
Bitcoin Foundation will simply need to re-implement these tools.

  This process that I went through was mainly educational, to see what would
be required to do this back port, and to get a tour through this part of the
code and what has changed in forward versions.  I'm writing this up while
it is still fresh in my mind.  The course I took, was perhaps not as ideal
as it should be, but I tried to be as systematic as I could be without having
all of the code changes fit in my head (there are a /lot/ of changes).
Luckily, I didn't spend too much time doing this at all; perhaps one afternoon,
plus another morning.

0x01: [ RPC Call Function Back-Port ]

  Starting out, I went and made a diff between versions v0.5.3 and v0.7.0
(159`555 line unified diff) [R.03].  I went through this mass and found the
parts that I knew that I needed right off, such as all of the large RPC
functions: getrawtransaction, listunspent, createrawtransaction,
decoderawtransaction, signrawtransaction, and sendrawtransaction.  In version
0.7.0, these RPC functions were not in their typical home, bitcoinrpc.cpp, and
instead emplaced in a new file called rpcrawtransaction.cpp.
This actually did cause me a bit of headache at first because linking these
functions into the pCallTable array wasn't working as they weren't in scope.
This problem was resolved by emplacing them into the bitcoinrpc.cpp file before
the pCallTable.

  With regard to the simple wiring up of the RPC calls, there are a variety of
helper functions that also needed to be updated, or imported, such as:
RPCTypeCheck, and ConvertTo, the latter of which is called by various
conditions in CommandLineRPC.  For example, in 0.5.3, ConvertTo only ever
passed in a primitive boost type, whereas in 0.7.0, ConvertTo had to be altered
to allow for JSON Array and JSON Object types.

  In addition to the above functions that were obvious we wanted to back-port,
there were two others that also resided in the rpcrawtransaction.cpp: TxToJSON
and ScriptPubKeyToJSON.  TxToJSON is relatively straight forward as it takes a
CTransaction object (0.5.3 legacy object) and turns it into JSON.  Where we
start to encounter new functions and object types, on which the whole mess is
dependent.  For instance, ScriptPubKeyToJSON makes a call to
ExtractDestinations.  This new function takes as a parameter, a vector of
CTxDestination classes.  CTxDestination is a new internal type where they have
created a boost type variant that could be one of the three following objects:
CNoDestination, CKeyID, or CScriptID.

0x02: [ New Objects, A Cascade of Fail ]

  The CKeyID and CPubKey classes are new, and implemented in the key.h header
file.  CPubKey extends from class uint160, which contains two constructors,
one default (empty) that creates a uint160 from a 'zero' value, and another that
takes a uint160 parameter and creates a new uint160 from the value passed by
reference.  CPubKey is a newly introduced encapsulation class for public keys.

  The cascade of breakage comes in where many legacy source files use
CBitcoinAddress as the means to retrieve a public key, or add one, etc.  This is
all replaced in version 0.7.0 with the CKeyID and CPubKey instead.  This
introduces a cascade of breakage throughout the code.  I even have attempted to
fix all of the breakage in such files as keystore.h and keystore.cpp, this
blooms into breakage in bitcoinrpc.cpp, script.h, script.cpp, wallet.h, and
wallet.cpp; however, upon walking through the mega-jungle of required repairs,
it became clear this was not going to work as a simple back-port.

  I'd also like to note that aside from the problems encountered by switching
from CBitcoinAddress to CKeyID and CPubKey, there are a multitude of other
changes that are introduced into script.h, script.cpp, base58.h, key.h,
keystore.h, keystore.cpp, ... the list is long, and certainly does not "fit in
head".  Many of these changes are non-obvious, undocumented, making it difficult
to discern whether these changes are in relation to the Raw Transaction tools,
or just unrelated completely.

0x03: [ Taking a Different Approach ]

  I ended up abandoning the whole back-port project as the breakage cascades
seemingly everywhere throughout the code base.  One of the main tenets of
creating vpatches is that they "fit in head".  Introduction of huge swaths of
new code does not align with those principals.  The Real Bitcoin (Reference
Implementation) should not be blindly bloating the code base with new classes
that need not exist, when current classes that already exist can be used.

  A next approach should be to implement something very similar to the v0.7.0
Raw Trasaction RPC calls, while only utilizing the current framework available
in The Real Bitcoin.  This approach will leave other structures intact and
lower risk considerably.

Please write in if I've missed the mark on any of the above, or have comments
or questions.

References:

  [R.01]: http://thebitcoin.foundation/v0.5.3-0-gd05c03a.tar.gz
  [R.02]: http://btcbase.org/log/2015-12-21#1349600
  [R.03]: See section 0xFF below to reproduce the diff I created.

OxFF:
To create the diff between 0.5.3 and 0.7.0, I cloned the 'bitcoin' project
from shithub/bitcoin ala:

$ git clone https://github.com/bitcoin/bitcoin.git
$ cd bitcoin
$ git checkout tags/v0.7.0
$ git diff tags/v0.5.3 HEAD > v0.5.3_v0.7.0.patch

$ wc -l v0.5.3_v0.7.0.patch
159555 v0.5.3_v0.7.0.patch
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (FreeBSD)

iQIcBAEBCgAGBQJa4AADAAoJEHIXBai3Hq2vLSwQAMyyMgnYKkLOcUsvX3dWc/ec
z2gwUjMPsr/nvAkbkfSelu267Q67jfY8J1d/BoWmFQAXg73n6CrVFDtgu7ekkiLy
g+Tu+wBHAyuxxl5nRLL6KLm7NZWxEESBoxm43b37JXM1cJqQ+ceDJ3YS2H81TLWt
ShB8ohaDDhei79Nj1I8/qm3HvxTICLyJCfMVBFLgakdUtPiB6ecEtl61zqBFATkf
Yw04+46zTJI0qRa5iFLRw06aC9lAgq64bgR2oG/Xvh8p2Efyv9PLYMffMRcAstCM
51Iy/Ho5VNa+QlHrt9JMAy6xNISS8Iyu+3Y7kPLC+HAxsNPL7nVVxejiPSkAPy/v
Ed4veCnmWwk+CekKgRWfvjrlqvXIppm39yy9uvIPumm3LAUodDuOPjzo9gONGw48
zbEf4A5T4E3Y5QZ08mjPxPIWiLU+IqwOLYYrbdStfxQ7xw2DWY+2kKqEiFwrdYvN
wikFvGqjg1n736LFb1c60UC3q58Vpk987pDXrwaAmgT543QfgW2gWYq7jw4hT1l5
+jquxG32DY8c73q6efZEQkwGZpwRaWMsLUcXuskMz6bKY+O2Ta+YcH81pPIlHUcr
npF5aoOhQPrLbCdjw70LNyWFDIEKVWRb0KSwHH/VelbKXZrjooL9F0evp7CwMAXW
ZVCsBlCN9FmKEIbFuQ3n
=7T4G
-----END PGP SIGNATURE-----