Skip to content

Lightning block sync and sample block sources #763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 82 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
0a5c29f
Add subcrate which impls a simple SPV client from Bitcoin Core RPC
TheBlueMatt May 5, 2020
55af3da
f - Use rust-bitcoin 0.24
jkczyz Oct 2, 2020
71b49e3
f - Changes needed for rebase on #649
jkczyz Sep 7, 2020
f9ff8f9
Reorder imports in lightning-block-sync
jkczyz Oct 7, 2020
265b8a9
f - Handle odd-length hex strings in hex_to_vec
jkczyz Oct 6, 2020
6d79c6f
Add unit tests for utils::hex_to_uint256
jkczyz Oct 6, 2020
a7ac3c3
Add unit tests for utils::hex_to_vec
jkczyz Oct 6, 2020
4d5ca87
Use bitcoin::hashes::hex::FromHex in utils
jkczyz Oct 7, 2020
d1af694
Replace hex_to_vec with Vec::<u8>::from_hex
jkczyz Oct 7, 2020
9b88bb1
Return Result from hex_to_uint256
jkczyz Oct 7, 2020
78c1896
Implement split_uri with http::uri::Uri
jkczyz Oct 8, 2020
801a4a5
Refactor split_uri into HttpEndpoint
jkczyz Oct 19, 2020
748d34e
f - Make http_endpoint pub
jkczyz Nov 18, 2020
75ae003
Refactor HTTP client into separate abstraction
jkczyz Oct 23, 2020
359ead8
f - Fix tokio support in HttpClient
jkczyz Dec 5, 2020
97e486a
Use type conversion for HTTP response parsing
jkczyz Oct 23, 2020
808cec3
f - Fix previousblockhash for JSON genesis block
jkczyz Nov 18, 2020
eb3ad66
f - Check that JSON response contains an object
jkczyz Oct 28, 2020
110c07d
f - Handle invalid JSON cases separately
jkczyz Oct 30, 2020
89b72f0
Add tests for HttpClient::connect
jkczyz Oct 26, 2020
8881435
f - Fix tokio support in HttpClient tests
jkczyz Dec 5, 2020
82a4dc7
Return std::io::Result from read_http_resp
jkczyz Oct 27, 2020
927ee26
f - Use ErrorKind::InvalidData instead of Other
jkczyz Oct 30, 2020
3cceaf4
Add tests for REST client
jkczyz Oct 27, 2020
d3511b3
Add tests for RPC client
jkczyz Oct 27, 2020
4ce724e
Add tests converting from Vec<u8> into responses
jkczyz Oct 28, 2020
eb45430
Add tests converting into Block from responses
jkczyz Oct 28, 2020
2fdd593
Add tests converting into BlockHash from response
jkczyz Oct 28, 2020
6400753
Add tests converting into BlockHeaderData from response
jkczyz Oct 30, 2020
90b1e5a
f - Test missing previousblockhash from JSON block
jkczyz Nov 18, 2020
13c1775
f - Correctly calculate chainwork in tests
jkczyz Nov 2, 2020
6281b6e
Replace GetHeaderResponse::to_block_header with TryFrom
jkczyz Oct 30, 2020
86efb99
Rename BlockSourceRespErr to BlockSourceError
jkczyz Oct 30, 2020
b0f2348
Add type alias for BlockSource result
jkczyz Oct 30, 2020
a82d22d
Rename BlockSourceError variants
jkczyz Oct 30, 2020
d829fbb
Convert from std::io::Error to BlockSourceError
jkczyz Oct 30, 2020
0e6069b
Refactor block source polling behind a Poll trait
jkczyz Nov 17, 2020
816afa4
Add unit tests for ChainPoller
jkczyz Nov 17, 2020
3d30eea
Add utility for looking up previous header
jkczyz Nov 29, 2020
cf518a0
Allow different networks in Blockchain test util
jkczyz Nov 29, 2020
5f862dc
Move Blockchain to test_utils module
jkczyz Nov 29, 2020
1d79ebb
Fix detection of separate chains
jkczyz Nov 30, 2020
6ca4fae
Add tests for syncing {dis}connected blocks
jkczyz Nov 30, 2020
aab43b5
Reorganize fork detection logic
jkczyz Nov 30, 2020
81f13ca
Rewrite find_fork_step using iteration
jkczyz Dec 1, 2020
b7bc528
Remove http crate dependency
jkczyz Dec 4, 2020
d2ffbb1
Replace custom chunked transfer decoding
jkczyz Dec 8, 2020
4e4bd20
Use read_exact for unchunked responses
jkczyz Dec 8, 2020
64a064a
Refactor HTTP response header parsing
jkczyz Dec 9, 2020
1b0fd6f
Move read_http_resp inside HttpClient
jkczyz Dec 9, 2020
62550df
Add tests for HttpClient::read_response
jkczyz Dec 10, 2020
18b9f44
Limit bytes for HTTP headers and body separately
jkczyz Dec 10, 2020
f717530
Replace custom header cache with HashMap
jkczyz Dec 11, 2020
4715265
Remove common tip detection across block sources
jkczyz Dec 11, 2020
7f0c61c
Prevent panics in read_too_large_message_body test
jkczyz Dec 12, 2020
73bb708
Do not consume HttpClient when calling get or post
jkczyz Dec 12, 2020
a15fbe4
Flush TcpStream after writing HTTP requests
jkczyz Dec 12, 2020
52b8d96
Set TcpStream to nonblocking
jkczyz Dec 12, 2020
bd13e30
Shutdown HttpServer explicitly to avoid panics
jkczyz Dec 12, 2020
6f382b4
Merge find_fork and find_fork_step
jkczyz Dec 12, 2020
d33c7c3
Rename steps_tx to steps
jkczyz Dec 12, 2020
6ab4b24
Proof of concept for BlockSource multiplexing
jkczyz Dec 14, 2020
7af1dbe
Reuse ChainPoller in ChainMultiplexer
jkczyz Dec 14, 2020
ade6ace
Remove MultipleChainPoller
jkczyz Dec 14, 2020
dc04ee2
Return ChainTip from MicroSPVClient::poll_best_tip
jkczyz Dec 15, 2020
e5616c3
Refactor Poll trait to use ValidatedBlockHeader
jkczyz Dec 17, 2020
a6beac8
Expand Poll trait to be an adapter for BlockSource
jkczyz Dec 17, 2020
b3bd783
Call check_builds_on from ChainPoller
jkczyz Dec 17, 2020
ffbdaa5
Move check for different chain to ChainPoller
jkczyz Dec 17, 2020
6d54ffb
Validate blocks directly in ChainPoller
jkczyz Dec 17, 2020
4a0a9b1
Implement ChainMultiplexer in terms of ChainPoller
jkczyz Dec 18, 2020
9d36b29
Remove unnecessary match expressions
jkczyz Dec 18, 2020
b0d4b20
Refactor sync_chain_monitor macro into a method
jkczyz Dec 18, 2020
2774200
Add ChainNotifier abstraction for functions
jkczyz Dec 21, 2020
24976bc
Add ChainNotifier tests for more error cases
jkczyz Dec 21, 2020
7546fd6
Remove Sized requirement from ChainListener params
jkczyz Dec 28, 2020
74e8d8d
Parameterize MicroSPVClient by ChainListener
jkczyz Dec 28, 2020
4b83f18
Rename best_chain_tip to best_known_chain_tip
jkczyz Jan 8, 2021
b652f9e
Add MicroSPVClient tests
jkczyz Jan 10, 2021
87a9b99
Rename MicroSPVClient::init to new
jkczyz Jan 10, 2021
eaedc0a
Clean up MicroSPVClient documentation
jkczyz Jan 11, 2021
c3fdc74
Remove dyn from ChainPoller
jkczyz Feb 9, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
1.30.0,
# 1.34.2 is Debian stable
1.34.2,
# 1.39.0 is MSRV for lightning-net-tokio and generates coverage
# 1.39.0 is MSRV for lightning-net-tokio and lightning-block-sync and generates coverage
1.39.0]
include:
- toolchain: stable
Expand Down Expand Up @@ -48,6 +48,15 @@ jobs:
- name: Build on Rust ${{ matrix.toolchain }}
if: "! matrix.build-net-tokio"
run: cargo build --verbose --color always -p lightning
- name: Build Block Sync Clients on Rust ${{ matrix.toolchain }} with features
if: matrix.build-net-tokio
run: |
cd lightning-block-sync
RUSTFLAGS="-C link-dead-code" cargo build --verbose --color always --features rest-client
RUSTFLAGS="-C link-dead-code" cargo build --verbose --color always --features rpc-client
RUSTFLAGS="-C link-dead-code" cargo build --verbose --color always --features rpc-client,rest-client
RUSTFLAGS="-C link-dead-code" cargo build --verbose --color always --features rpc-client,rest-client,tokio
cd ..
- name: Test on Rust ${{ matrix.toolchain }} with net-tokio
if: "matrix.build-net-tokio && !matrix.coverage"
run: cargo test --verbose --color always
Expand Down
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

members = [
"lightning",
"lightning-block-sync",
"lightning-net-tokio",
"lightning-persister",
]
Expand Down
27 changes: 27 additions & 0 deletions lightning-block-sync/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
[package]
name = "lightning-block-sync"
version = "0.0.1"
authors = ["Matt Corallo"]
license = "Apache-2.0"
edition = "2018"
description = """
Utilities to fetch the chain from Bitcoin Core REST/RPC Interfaces and feed them into Rust Lightning.
"""

[features]
rest-client = [ "serde", "serde_json", "serde_derive", "chunked_transfer" ]
rpc-client = [ "serde", "serde_json", "serde_derive", "base64", "chunked_transfer" ]

[dependencies]
bitcoin = "0.24"
lightning = { version = "0.0.11", path = "../lightning" }
tokio = { version = ">=0.2.12", features = [ "tcp", "io-util", "dns" ], optional = true }
serde = { version = "1", optional = true }
serde_json = { version = "1", optional = true }
serde_derive = { version = "1", optional = true }
base64 = { version = "0.9", optional = true }
chunked_transfer = { version = "1.3.0", optional = true }
futures = { version = "0.3.8" }

[dev-dependencies]
tokio = { version = ">=0.2.12", features = [ "macros", "rt-core" ] }
Loading