[Worker Proposal] WARP Network - Part 1
en·@eoseoul·
0.000 HBD[Worker Proposal] WARP Network - Part 1
Hi, we are EOSeoul.
In this article, we would like to share our initial test results from WARP Network..
# Summary
* The reason behind this test is to explore to find other viable options to reduce the latency of the mainnet network in EOS.IO.
* We have looked at the WARP network and the effect it will have on the latency and the Jitter. We have found that both the latency and the Jitter are reduced under the WARP Network, i.e. it is faster than the standard public internet.
* The tutorial will also be released after the next post on the failover function test results.
* We intend to invite BP candidates considering using the WARP network, and trustworthy candidates to build the WARP network together.
## The contents not covered in this document are as follows;
* DDoS Protection
* IP address concealment
## Focused contents are as follows;
* Background to suggest WARP network
* Experiment to reduce latency / jitter through WARP network
## Contents to be published later in Part 2 are as follows;
* Route Failover over WARP network
* Node setup tutorial for WARP Network Participation
* WARP participating node and WARP endpoint peer recruitment
# What is a WARP network?
Simply put, it is a global blockchain network architecture which aims to reduce the latency and the jitter between blockchain nodes communication.
EOSeoul is experimenting with various tests to implement the blockchain network architecture for a secure and reliable service around the world. This document mainly covers the implementation of overlay networks using cloud platforms, VPNs, and BGP routing.
In the future, WARP networks will be improved by experimenting to find the best performing combination of using failover, network acceleration and other technologies. We will continue experimenting with various security technologies.
# Role of EOSIO block generation network
<center>[Figure 1]</center>
Unlike Bitcoins or Ether where all nodes are participating in the generation of blocks on a P2P network, in EOS, blocks can only be generated by 21 block producers voted by token holders. When the block producers are elected, they produce 12 blocks in turn, then on to the next producer. A block is generated at every 500ms (0.5 second).
A transaction by a user is broadcasted to all 21 block producers, and they must process the transaction. The block producer who receive a transaction request, must verify and execute its contents, and then generate a block and broadcast. Also a block producer must validate blocks broadcasted by other producers.
Twenty-one block producers repeat the process mentioned above whenever they create a block, or when they are verifying previous blocks created by other block producers. In this way, blocks verified by more than 15 of the 21 block producers(⅔ +1) will become irreversible.
Figure 1 illustrates this on a full-mesh block producer network.
It is necessary to have a low latency and reliable communication between block producers geographically spread all over the world, so a sound network architecture is imperative.
# Notes for Consideration
EOSeoul team has been using BGP and operating the gaming services for more than 10 years. It is one of the routing protocols where it is a standard tool used for connecting the networks directly to Internet Service Providers (ISP). We have a lot of experience on building international network uplinks as well as domestic network uplinks using it, and have been performing various traffic engineering including DDoS incident responses.
In our view, it is highly likely that the communication path used amongst the block producers will include the communication path of the common traffic with relatively large bandwidth depending on the situation. Various unexpected variables can affect the quality of the network latency.
A review of these network latencies can be found in two of the [EOSIO Dawn 2.0 Release Notes](https://steemit.com/eos/@eosio/eos-io-dawn-2-0-released-and-development-update).
Originally the order of block producers were shuffled at random, but when moving between block producers from one side to the other side of the globe without local arrangements, the high latency would result in missing data transmission and resulting missed blocks. It has been planned to vote the sequence of the production but it is not yet clear the order of block productions.
It is understood that many block producers will be utilising the AWS cloud initially. Consequently it is the reason why the network is configured with P2P topology through the AWS peer. However for those who are aware of availability and security risk as well as the awareness of infrastructure operations will participate in block production through their own datacenters.
In the past, many people have considered solutions such as MPLS VPNs through the EOS Developer telegram channel. MPLS VPN configuration is expensive with a relatively complex configuration, and require dedicated lines and hardware. All block producers will require a system engineer to install and operate the system. In some cases, however, block producers with no network engineers may participate in the election as we found that with some network knowledge, a system engineer can implement it.
# WARP network test
Currently, most of the P2P communications between the current block producers use the Internet provided by an Internet service provider (ISP) or the cloud without further optimisation. Due to this issue, we suggest the following WARP Network.
The source technology used in this experiment is a technology already verified by standardised communication protocols.
However, as it is not yet clear it would work in the EOS environment, we tried to configure and experimented with infrastructure at several local locations around the world with the configuration that is similar to a live environment.
Conceptually, we will construct the network architecture,shown in figure 2, with the possibility of working with global cloud vendors, CDNs, and Internet service providers with the necessary network infrastructure.
<center>[Figure 2]</center>
Any technical issues and questions raised during the experiment will be actively communicated with the EOS community as well as other block producers. We will continuously share the processes and results.
# Initial test configuration
## Diagram
<center>[Figure 3]</center>
## Architecture
* Configure EP (End Point) Instance where a block producer in AWS Region will connect and also configure the private network for already configured EP through VPN and BGP peering in the region
* **4 test regions** (Seoul, Oregon, Frankfurt, São Paulo) considering geographical locations in AWS regions
* **The role of EP (End Point) instance:** It is configured by using commercial products supporting VPN and BGP, or using open source.
* **The Operating system of EP:** It uses VyOS from AWS Marketplace. It is an open source system based on Debian GNU and Linux, supporting dynamic routings such as BGP, OSPF, RIP, IPSec, OpenVPN, firewall, and NAT.
* Note: https://vyos.io/
* After connecting VPN tunnel to OpenVPN between EP Instance, BGP Neighbor is configured with full mesh between EPs through VPN Tunnel
* OpenVPN is to connect the adjacent EPs via VPN and Quagga is used to peer EPs via BGP
* BP Instance announces its IP to be used for P2P communication between BPs through BGP, and all BP learn the routing information of the communicating BP
* __Latency Measurement__: Using the routing path through BGP, we measured the latency between the BP instances and compared to the public Internet network
## EP-to-EP Connection Configuration
* EP Instance:
* Instance Type: t2.micro
* As the main purpose is to measure the RTT using ICMP, we decided that Instance Type does a little influence and for Instance, we used t2.micro
* Version: VyOS 1.1.8
## Configuring EP-to-BP connections
* EP Instance
* Instance Type: t2.micro
* Version: VyOS 1.1.8
* BP Instance
* Instance Type: t2.micro
* Version: Ubuntu Server 16.04
* As the main purpose is to measure the RTT using ICMP, we decided that Instance Type does a little influence and for Instance, we used t2.micro
## Setting
VPN, BGP configuration of nodes connected to WARP network will be released in the form of tutorial in a future post.
# Initial experiment measurement and results
* Measuring method: The RTT values when passing through the public Internet network or via the EOS WARP Network.
* Measured points
Network | Source IP | | Destination IP
- | - | - | -
Public Internet | EOSSeoul Public IP | <-> | Frankfurt BP Public IP
Public Internet | EOSSeoul Public IP | <-> | Oregon BP Public IP
Public Internet | EOSSeoul Public IP | <-> | São Paulo BP Public IP
WARP Network | EOSSeoul Private IP | <-> | Frankfurt BP Private IP
WARP Network | EOSSeoul Private IP | <-> | Oregon BP Private IP
WARP Network | EOSSeoul Private IP | <-> | São Paulo BP Private IP
Time: Every 2 seconds from 19:53 on May 10, 2018 to 12:53 on May 14, 2018 (GMT+9)
* Results
Scenario | WARP Network | Internet | Notes
- | - | - | -
Seoul - Frankfurt | | |
Average RTT (ms) | 264.711303 | 287.348705 | Internet is slower by 8.55%
RTT SD (ms) | 4.254607 | 4.910796 | Jitter is slightly volatile
Seoul - Oregan | | |
Average RTT (ms) | 135.543697 | 151.029003 | Internet is slower by 14.42%
RTT SD (ms) | 3.015122 | 3.192323 | Jitter is slightly volatile
Seoul - Sao Paulo | | |
Average RTT (ms) | 296.071500 | 305.732230 | Internet is slower by 3.26%
RTT SD (ms) | 2.298818 | 5.425839 | Jitter is much more volatile
# Review
* It can be confirmed that the latency and the jitter values of RTT are lower than those of the public Internet network via the WARP network.
* The configuration above can be implemented not only with AWS but also with any Public Cloud environment, such as GCP or Azure. However someone or an entity must to be in charge of operations and operational governance as well as the cost.
* We have also found that you need to be able to predict BP traffic amount and to select and validate the appropriate instances.
* It is also understood that the effect of WARP Network will be rather limited when the peer-to-peer communication between two candidates running on the AWS, as it will already be using the network between the AWS regions.
* The WARP Network uses a network between AWS regions and this particular network is outside the control of any BP.
* For any communication between two geographically close BPs, it may be better not to use a WARP Network as it may increase the latency.
* If the connection between BPs use the private IP prefixes, then the governance is required such as assigning an IP prefix to prevent IP conflicts by assigning individual IPs.
* We intend to test the public IP communication architecture in preparation of WARP network failure and block duplicate IPs and share the node configuration.
* In fact, P2P communication is performed through the WARP network in the public IP, and for any failure in the WARP network, we will switch to public internet effectively.
* We also plan to implement a redundancy configuration in case of an EP failure.
# Stay Tuned!
We have reviewed the advantages and disadvantages of the WARP network. The latter is the addition of the node configuration, the endpoint cost of the WARP network, and the failover should be possible in case of the network fails. The former is that no additional network equipment required, the ease of implementation, the reduced and stable latency when compared to the public Internet.
The WARP network will evolve. Failover will be implemented via BGP to prepare for failures. Experiments with a variety of tunneling and acceleration software will get better. We will review various public clouds and work with various Internet service providers to make them more usable.
We will be releasing configuration and tutorials for using the WARP network in part 2 of the test. At the same time, we will invite block producers around the world wishing to use the WARP network. We will also invite trusted block producers to build the WARP network together. We will then propose a Worker Proposal based on the material content.
Please stay tuned for Part 2 of this post soon!
# Any feedback.
Suggestions and questions are always welcome. Please do not hesitate to give a feedback to EOSeoul. Join the Telegram Group below to share the latest news from EOSeoul and technical discussions about EOS.
Thank you!
EOSeoul
Telegram (General Talk, 한국어) : https://t.me/eoseoul
Telegram (Developer Talk, 한국어) : https://t.me/eoseoul_testnet
Telegram (English) : http://t.me/eoseoul_en
Telegram (简体中文) : http://t.me/eoseoul_cn
Telegram (日本語) : http://t.me/eoseoul_jp
Steemit : https://steemit.com/@eoseoul
Github : https://github.com/eoseoul
Twitter : https://twitter.com/eoseoul_kor
Facebook : https://www.facebook.com/EOSeoul.kr
EOSeoul Documentations : https://github.com/eoseoul/docs