One of the issues with the P2P Bitcoin network is that while it provides general connectivity for all node types, its latency that is too large for the specialized needs of most miners. Miners can spend significant amounts of time mining on top of blocks that are old, even though the Bitcoin network has already agreed on what the latest block is. That latest block just might not have propagated to all the miners yet. Relay networks are built to solve this issue.

All full-nodes that participate in the Bitcoin network include the relaying function, they validate and propagate transactions and blocks. This code in Bitcoin-Core that does peer-to-peer networking is somewhat complex, and not very efficient. From the miners perspective, the network latency resulting from this complexity over time has become a significant factor. It can sometimes take 10-15s before they receive newly discovered blocks propagated within the network. Block mining being a race between miners, this long of a delay has a direct influence on the profitability of their operation, and therefore via reduced incentives for them – on the security of the network.

Some mining pools have gone the route of forming private networks between each other, for ultra fast block propagation. On these networks they send each other blocks as soon as they discover them, so that they can all start mining on new blocks, and not waste hashing power, as soon as one is announced/discovered by one of them. Miners also have internal block relaying networks, internal to their system, to relay blocks between their own hosts in the fastest way possible.

The creation of the Bitcoin Relay Network, and subsequently FIBER (and a few others), has democratized this fast block routing technology for all miners in the ecosystem.

Bitcoin Relay Architecture


Bitcoin Relay Network

The first widely used version of a Bitcoin relay network is the “Bitcoin Relay Network” (www.bitcoinrelaynetwork.org). It was created by Matt Corallo in 2015, member of the Bitcoin-Core dev team.

alt text

There were a few main motivations for the development and introduction of the Bitcoin Relay Network – significant decrease in the number of listening nodes at the time, and selfish-mining attacks:
- the decrease in the number of listening nodes was considered drastic at the time, and was a serious concern for the core developers. There were only a few 100s of nodes on the network, that were full nodes. It was thought that people just didnt want to be full nodes anymore. This reduction in network size made the network more sensitive to denial-of-service attacks, and in general was not considered to be good for the community.
- Selfish-mining attack – this was first outlined in a Cornell University paper released in late 2013. It roughly involves holding newly discovered block back, by a miner, in private and away from the public network. When the public network mines a new block, the attacking miner immediately releases hidden blocks in an attempt to have its chain be the longest one, and therefore become the primary.

In its first implementation, the Bitcoin Relay Network started as a set of slow SPV relay nodes, that would peer with all the other participants of the P2P network. Reminder on SPV nodes – it stands for “simplified payment verification”, and those nodes maintain only a subset of the full blockchain dataset, and they verify new transactions using this modified SPV method. These nodes are also called light-weight nodes (commonly run on resource-constrained devices).

There is some evidence that when the BTC relay network was first introduced, it noticeably reduced the number of uncle blocks that were produced (uncle blocks are orphan blocks that were mined in the chain that was not the longest, they are blocks that were mined just after the correct block got added to the main chain). This increased the overall security of the network, since there was a reduction in unused (uncle) blocks that were being mined due to miners finding out about new blocks with a significant delay (due to block propagation latencies in the P2P network).

The first deployment of this relay network included only 4 regions – New York, Seattle, Amsterdam, and Singapore. These relay nodes were just relaying blocks, before doing any validation on their contents. This was clearly done for performance reasons. One of the primary reasons for performance issues in block propagation times in the BTC p2p network is the fact that full nodes do full block verification before they send new blocks to other nodes, which took some time.

In the relay network model, miners connect to relays directly, and propagate their blocks that way.
The relay network was also based on TCP, like the BTC p2p network. This was later identified as a performance bottlneck, mainly due to the TCP protocols packet resend semantics, as a measure to combat pocket loss. In subsequent relay networks (FIBRE,etc.) UDP was used, in order to avoid automatic packet resends, and instead to control that directly.

Matt did a great talk at SanFrancisco Bitcoin Devs Seminar (sep1.2015), I highly recommend watching it, it discusses relaying concepts and tech in significant detail.


FIBRE - Fast Internet Bitcoin Relay Engine

FIBRE, Fast Internet Bitcoin Relay Engine (bitcoinfibre.org), released in 2016, was also developed by Matt Corallo. It was based on several years of design and operation of the Bitcoin Relay Network. It is currently running as a public network, and is the most used open-source relaying system for Bitcoin. It is very likely that it is also widely run in private networks as well, as a part of larger infrastructures built for block mining operations.

Its codebase was designed as an extension to the Bitcoin-Core, a fork of its code. These extensions have not been merged into the Bitcoin-Core repository itself, nor are there any immediate plans to do so. Building FIBRE on top of Bitcoin-Core itself was partially motivated in order to simplify the operation of relay networks. This is important, in order to stimulate more decentralization of Bitcoins relay networks. It is based on v.0.13.1 of Bitcoin-Core, and has been updated to support SegWit. Its GitHub repository (github.com/bitcoinfibre/bitcoinfibre), and its a fork of the bitcoin/bitcoin main Bitcoin Core repositorygithub.com/bitcoin/bitcoin.

You run the FIBRE bitcoin-core binary the same way you would normally, except there are additional UDP related CLI options. Relaying peers are also specified explicitly in bitcoin.conf, as well as explicit bandwidth bucketing/classifying of peer groups, groups that peers are assigned to. There is also a distinction between “trusted” and “untrusted” nodes, where the relaying logic (its semantics) differs between the two. For example, packets from “trusted” nodes are relayed to all the other trusted peers immediately upon receipt. The codebase is also tested to work with IPv6 and for relays to be potentially behind a NAT.

Primary improvement in FIBRE is the switch from previous Bitcoin Relay Networks use of TCP, to the usage for UDP along with Forward Error Correction. TCP handles long-distance links pretty badly, where packet loss is somewhat frequent. The re-transmits of dropped packets were found to be the primary performance delay in block relaying in the previous Bitcoin Relay Network. With UDP, direct control can be taken over the protocols (used by the relay network) packet retransmit logic, which results in significant drops in overall network latency. Unlike TCP, the UDP transmission rates are pre-coded (pre-determined), so there is no slow ramp up time, or waiting for lost packets. Network optimizations include the packet sender sending different block chunks to different FIBRE peers, and then those peers in turn relay these packets between each other as soon as they get them - this gets additional bandwidth out of the network, without saturating individual network interfaces on hosts (with a host sending a single full block multiple times, redundantly, to all the peers of that host). FIBRE relays network packets as soon as they come in, it doesnt wait to assemble a block from all of its network packets, before it relays that block on to the next node.

Throughout its design FIBRE sacrifices bandwidth for latency. This was very deliberate, and a result of years of experience in operating relays over long-haul global lines. Blokcs that are sent through the network are several times their original size, because they include Error Correction data.

Forward Error Correction (FEC) also helps with packet loss by sending additional data along the relayed blocks so that the receiving peer can reconstruct the full blocks even if packet loss lost some of the block data. All this without contacting the sending peer (a lot of this technology was developed for use in Video conferencing systems).

The BIP152 Compact Block Relay work in Bitcoin-Core was happening at the same time as FIBRE was being developed, and FIBRE depends on it for compression. Due to joint-work, the “cmpctblock” message (BIP152) was designed to integrate well with FIBRE’s UDP/FEC relay model. BIP152 and FIBRE were really meant to work together from the start.

Some good thoughts on FIBRE can further be read in this post, by Matt Corallo - http://bluematt.bitcoin.ninja/2016/07/07/relay-networks/
If you’re interested in actually operating a FIBRE relay node, you can get more details in a post by Matt – http://bitcoinfibre.org/fibre-howto.html.


Compact Block Relay (BIP152)

The main premise of BIP152 is that its possible to reduce the amount of bandwidth that is necessary to propagate new blocks to full nodes in a BTC network, efficiently, when those nodes share much of the same content/transactions in the mempool. As a reminder, a mempool (memory pool, transaction pool) is a pool of yet unconfirmed (not added to any block) transactions, that is maintained by most of the nodes in the BTC network.

As the BIP152 Bitcoin Spec (github.com/bitcoin/bips/blob/master/bip-0152.mediawiki) brings up, the BTC p2p Network is not very efficient when relaying blocks. When it sends blocks over the network, each block includes all the transactions, even though most of the full nodes that are receiving those blocks already have most of the transactions that are in that block – thus making that data redundant.

BIP152 solves many of the issues with some nodes experiencing very large spikes in bandwidth consumption, if they received blocks before most of their peers – they would be flooded with requests from other peers to send out full/redundant block data across their single internet connection. This is not an optimal solution for most users that would run full BTC nodes.

New protocol messages that were introduced are: sendcmpct, cmpctblock, getblocktxn, and blocktxn. Of those, “SENDCMCT” is main protocol message responsible for switching between new “high bandwidth relaying” and “low bandwidth relaying” modes.


Ethereum

Interestingly Ethereum doesnt have nearly as big of a need for fast relay networks as Bitcoin does. Ethereums protocol rewards uncle blocks, and includes them in the longest chain, unlike Bitcoin (which is the incentive for miners to avoid submiting blocks late as much as possible, since its wasted effort).

This rewarding of uncle blocks has to do with Ethereums much shorter block times, and therefore a higher probability of multiple block solutions being found by miners around the same time. Only one will be accepted into the main chain, by the network, and the remaining blocks that did not get accepted are rewarded at a portion of the full block-discovery reward. This partial reward is rationalized by the fact that the inclusion of these uncle blocks in Ethereums main chain increases the total secirity of the network.

The details of the uncle blocks model are quite interesting, and you can read more in this Ethereum uncles blog post.


Conclusion

Bitcoin relays have evolved over the years, from being a technology exclusively used by advanced miners internaly within their networks, to being developed in a more open-source manner and adopted by a wider Bitcoin development community. They were initially motivated by concerns of centralization in block mining, and potential innability of smaller mining groups/individuals to compete in the mining process. Even though open relay networks have modest hardware footprints, their efficiency and good design give them a significant influence over the global Bitcoin network, its operation and its organization.