Newly minted Bitcoin are rewards for a mathematical race that maintains the ledger of transactions. Rules for the race and rewards are fixed and inherent in the protocol. It is the decentralised incentive structure to keep the ledger honest and the value scarce that sets Bitcoin apart from other currencies and assets. Herein I provide a detailed explanation and code of how Bitcoins are mined.
Blocks
Bitcoin transactions are grouped by timeframe (roughly 10 minutes) and ordered in blocks with meta data that meet protocol requirements. Each block is sequential and agreed by consensus. The first miner to reach meta data requirements for the current block and reach approval through network consensus is awarded the block reward in addition to all transaction fees in that block. See the two figures below to understand the incentive structure for mining.Presently, the block reward is 12.5 bitcoin. At the current price ( ~ $ 10,000 / btc ), that's more than $ 125,000 every 10 minutes! I hope that sparks your interest. It did for me. The first step in mining is to maintain a bitcoin node, which stores and validates the full ledger. This can be installed from Bitcoin.org. I will use KDB+ to interface with the node and mine Bitcoin as it is the most efficient way to handle data. Full code can be seen at: our Github page.
Block templates
A template for the current block can be attained from our local node. (for details, see: Bitcoin.org)
template: .bitcoind.getblocktemplate[template_request][`result]
Here are the details we need:
template[`height] : 605355 // block height to be mined
template[`curtime] : 1574689408 // unix epoch time
template[`coinbasevalue] : 1274375042 // block reward in satoshis
template[`version] : 536870912 // indicates which validation rules to use
template[`bits] : "1715b23e" // encoded target
template[`target] : "00000000000000000015b23e0000000000000000000000000000000000000000"
template[`previousblockhash] : "000000000000000000042e1de089b9ee56a5aeab5f49dc534f090bfe022cbec8"
first template[`transactions] :
data | "0200000002446b124c95e37465f0d8a229450f6f77cb031b188a7d2f7074b..."
txid | "9995322b1aa0c4518e031efadc9a51da4831217631010662c36516e776594e17"
Block headers
Our job as the miner is to construct the required header with this information that hashes to a lower value than the target. The header itself is a concatenated hexadecimal string consisting of the following (from Bitcoin.org😞
Bytes
Name
Data Type
Description
4
version
int32_t
The block version number indicates which set of block validation rules to follow.
32
previous block header hash
char[32]
A SHA256(SHA256()) hash in internal byte order of the previous block’s header. This ensures no previous block can be changed without also changing this block’s header.
32
merkle root hash
char[32]
A SHA256(SHA256()) hash in internal byte order. The merkle root is derived from the hashes of all transactions included in this block, ensuring that none of those transactions can be modified without modifying the header.
4
time
uint32_t
The block time is a Unix epoch time when the miner started hashing the header (according to the miner). Must be strictly greater than the median time of the previous 11 blocks. Full nodes will not accept blocks with headers more than two hours in the future according to their clock.
4
nBits
uint32_t
An encoded version of the target threshold this block’s header hash must be less than or equal to.
4
nonce
uint32_t
An arbitrary number miners change to modify the header hash in order to produce a hash less than or equal to the target threshold. If all 32-bit values are tested, the time can be updated or the coinbase transaction can be changed and the merkle root updated.
Merkle root
"Merkle roots allow for efficient and secure verification of large data structures, and in the case of blockchains, potentially boundless data sets." (coinomi) Our Merkle root function:
buildmerkleroot:{[leafs]
leafcount: count leafs;
root: $[1 = leafcount; first leafs;
[if [0 < leafcount mod 2; leafs: leafs,enlist last leafs; leafcount: 1+leafcount;]; // if odd number, duplicate last record
oddindicies: 1+2*til "i"$(leafcount%2);
evenindicies: 2*til "i"$(leafcount%2);
hashpairs: leafs[evenindicies] ,' leafs[oddindicies];
branches:{ doubleSha256Byte x } peach hashpairs;
buildmerkleroot[branches] ] ];
root };
Order transaction hashes to build the merkle root.
hashlist: {x[`hash]} each transactions;
hashlist: (enlist coinbasetranshash), hashlist;
bytehashlist: hexstring_to_hex peach hashlist;
bytehashlistBigEndian:reverse peach bytehashlist;
Merkleroot: buildmerkleroot[bytehashlistBigEndian];
If you are astute, you may have noticed the coinbase transaction hash creep in there.
Coinbase transaction
The coinbase transaction is the record of the reward we pay ourselves for mining the block. As miners, we must place the coinbase transaction at the beginning of the transaction list. Format from Bitcoin.org
Bytes
Name
Data Type
Description
4
version
int32_t
Transaction version number (note, this is signed); currently version 1 or 2. Programs creating transactions using newer consensus rules may use higher version numbers. Version 2 means that BIP 68 applies.
Varies
tx_in count
compactSize uint
Number of inputs in this transaction.
32
hash (null)
char[32]
A 32-byte null, as a coinbase has no previous outpoint.
4
index (UINT32_MAX)
uint32_t
0xffffffff, as a coinbase has no previous outpoint.
Varies
script bytes
compactSize uint
The number of bytes in the coinbase script, up to a maximum of 100 bytes.
Varies (4)
height
script
The block height of this block as required by BIP34. Uses script language: starts with a data-pushing opcode that indicates how many bytes to push to the stack followed by the block height as a little-endian unsigned integer. This script must be as short as possible, otherwise it may be rejected.
The data-pushing opcode will be 0x03 and the total size four bytes until block 16,777,216 about 300 years from now.
Varies
coinbase script
None
The coinbase field: Arbitrary data not exceeding 100 bytes minus the (4) height bytes. Miners commonly place an extra nonce in this field to update the block header merkle root during hashing.
4
sequence
uint32_t
Sequence number.
Varies
tx_out count
compactSize uint
Number of outputs in this transaction.
8
value
int64_t
Number of satoshis to spend. Coinbase transactions spend the block subsidy and collected transaction fees.
1+
pk_script bytes
compactSize uint
Number of bytes in the pubkey script. Maximum is 10,000 bytes.
Varies
pk_script
char[]
Defines the conditions which must be satisfied to spend this output.
4
lock_time
uint32_t
A time (Unix epoch time) or block number.
coinbaseversion : "01000000"
coinbasein : "01"
coinbaseprevtx : "0000000000000000000000000000000000000000000000000000000000000000"
coinbaseprevout : "ffffffff";
coinbaseseq :"00000000";
coinbaseout : "01";
cbpubkey : "866b6c77fd236f9185bb11b4e176328d7f912a4b";
scriptPubKey : "76","A9","14" , cbpubkey, "88", "AC";
coinbasesl : "19";
coinbaselocktime : "00000000";
coinbasesats: littleEndian [ coinbasevalue; 16 ];
coinbasescript : "03" , ( littleEndian [ height; 6 ] ), texttohexstr [ "Reap what you sow." ];
coinbasescriptlen : littleEndian[ "i"$( count coinbasescript ) % 2;2];
coinbasetrans: coinbaseversion, coinbasein, coinbaseprevtx, coinbaseprevout, coinbasescriptlen, coinbasescript, coinbaseseq, coinbaseout, coinbasesats, coinbasesl, scriptPubKey, coinbaselocktime;
coinbasetransbytes : hexstring_to_hex coinbasetrans;
coinbasetranshash : raze string reverse doubleSha256Byte [ coinbasetransbytes ];
Again, for the astute there are a couple of points to note here outside of the format to pay yourself. First is that there are hidden messages encoded in each mined block (see cryptograffiti for examples ). Second is that there is a pubkey script language, using op codes to spend the outputs. This is bitcoin's smart contracts (for details see pubkey scripts). We use a standard Pay-to-Public-Key-Hash opcodes:
OP_DUP ( 0x76 ) - Duplicates the top stack item.
OP_HASH160 ( 0xA9 ) - The input is hashed twice: first with SHA-256 and then with RIPEMD-160.
OP_EQUALVERIFY ( 0x88 ) - Returns 1 if the inputs are exactly equal, 0 otherwise, runs OP_VERIFY afterward.
OP_CHECKSIG ( 0xAC ) - The entire transaction's outputs, inputs, and script (from the most recently-executed OP_CODESEPARATOR to the end) are hashed. The signature used by OP_CHECKSIG must be a valid signature for this hash and public key. If it is, 1 is returned, 0 otherwise.
Hash race
Now we have the constructs to find a live solution and submit it to the blockchain for reward. The block is now simple to put together:
Block : Version , hashPrevBlock , Merkleroot , Time , Bits, 4#reverse 0x0 vs Nonce, transcounter , raze trans
Except that the header hash value must be less than the extremely improbable target value. To achieve this, miners can vary the Nonce from 0 to 4,294,967,295. They can also update the transaction pool or message text to attain a different merkle root. Miner software and hardware optimisations are geared towards looping through different Nonce value hashes at faster rates, often at great expense ( with setup, electricity and security ). Our script is a portable, energy efficient parallel CPU solution; competitve with pooled resources or a high performance cpu cluster. One run through the entire Nonce range looks like this:
curBlock: 0N! computeData`height;
latBlock: 0N! blockcount[];
i :4200000000;
while[11b ~ (curBlock > latBlock) , i > 0; // check if block is still current
.Q.gc[]; // clean memory
computehash peach i + til 100000000; // process in batches, typically takes 1-2 minutes on a normal modern computer
i : 0N! i - 100000000;
latBlock: 0N! blockcount[]; ];
One run through of 4.3 billion Nonces would meet the base rate difficulty of the genesis block a decade ago. The difficulty is self adjusting to average 10 minutes a block and now requires more iterations. In our code, we update the transaction list to create a new merkle root after exhausting the Nonce range. In cases of reward conflict, the longest transaction list wins. Here is the code and functions we use for the continous mining effort:
submit:{[headerHex]headstr:raze string headerHex; blk: headstr ,raze string computeData`trans; h (`.bitcoind.submitblock;blk);0N! "SUCCESS: ", headstr;};
hashcheck:{[finalHash;headerHex]
if [finalHash[til 7] ~ 0x00000000000000;
targ: computeData`target;
if [finalHash[7] <= targ[7];
finalHashstr: raze string finalHash;
lz:first where finalHashstr <> "0";
$[leadingzeros < lz;
[ submit[headerHex]; ];
[ simpletarget: hex_to_int targ[8 9 10 11]
simplehash: hex_to_int finalHash[8 9 10 11]
if[ simplehash < simpletarget; submit[headerHex];]
if [simplehash = simpletarget;
simpletarget: hex_to_int targ[12 13 14 15]
simplehash: hex_to_int finalHash[12 13 14 15]
if[ simplehash < simpletarget; submit[headerHex];]
if [simplehash = simpletarget;
simpletarget: hex_to_int targ[16 17 18 19]
simplehash: hex_to_int finalHash[16 17 18 19]
if[simplehash < simpletarget; submit[headerHex];]
if [simplehash = simpletarget;
simpletarget: hex_to_int targ[20 21 22 23]
simplehash: hex_to_int finalHash[20 21 22 23]
if[ simplehash < simpletarget; submit[ headerHex ] ] ] ] ] ] ] ] ] }
computehash:{[Nonce]
hashcheck[reverse sha2561[ endMerkle , 4#reverse 0x0 vs Nonce; H0; target32 ]; computeData`partialHeader , 4#reverse 0x0 vs Nonce];
};
cycleAttempts: { [ ]
computeData:: h"loadUpdates[];computeData";
H0:: sha2560 64#computeData`partialHeader;
target32:: -4#8#computeData`target;
leadingzeros:: first where (raze string computeData`target) <> "0";
endMerkle:: -12#computeData`partialHeader;
//process 100,000,000 at a time ~ 1 min on a normal modern computer
// check if block has already been mined between efforts
curBlock: 0N! computeData`height;
latBlock: 0N! blockcount[];
i :4200000000;
while[11b ~ (curBlock > latBlock) , i > 0;
.Q.gc[];
computehash peach i + til 100000000;
i : 0N! i - 100000000;
latBlock: 0N! blockcount[];
];
cycleAttempts[];
};
cycleAttempts[];
Perhaps you can see the rabbit hole. I refer the interested to the bitcoin white paper. Please visit our github page for full code. Feedback is welcome at the email below.
