Skip to content

Commit f8ff12f

Browse files
ziggie1984ziggie1984
committed
docs: add migration doc
1 parent b7fe8fa commit f8ff12f

File tree

1 file changed

+252
-0
lines changed

1 file changed

+252
-0
lines changed

‎docs/data-migration.md

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# Data migration
2+
3+
This document describes the process of migrating `LND`'s database state from one
4+
type of database backend (for example the `bbolt` based database files `*.db`
5+
such as the `channel.db` or `wallet.db` files) to another (for example the new
6+
`postgres` or `sqlite` databases introduced in `lnd v0.14.0-beta`).
7+
8+
**Note:** Currently only migrations from `bolt` to either `postgres` or
9+
`sqlite` are supported. KV based databases will be phased out in the future. We
10+
are planning to add additonal support to migrate all `etcd` databases to `sql`
11+
databases. Moreover it is currently also not supported to move the database
12+
from `sqlite` to `postgres` or vice versa, because this migration only takes
13+
care of the key value data and there already exist the possiblity to have the
14+
invoices in native sql which this tool does not support.
15+
16+
17+
## Prepare the destination database
18+
19+
To be able to execute the migration successfully you have to update to [LND
20+
v0.19.0](https://github.com/lightningnetwork/lnd/releases/tag/untagged-6824d65b23dc77e94e22)
21+
which makes sure that the intial pre-migration state of the kv-db is up to date.
22+
23+
### Using postgres as the destination remote database
24+
25+
Prepare a user and database as described in the [Postgres](
26+
https://github.com/lightningnetwork/lnd/blob/master/docs/postgres.md)
27+
documentation. You'll need the Data Source Name (DSN) for both the data
28+
migration and then the `lnd` configuration, so keep that string somewhere
29+
(should be something with the format of `postgres://xx:yy@localhost:5432/zz`).
30+
31+
No additional steps are required to prepare the Postgres database for the data
32+
migration. The migration tool will create the database schema automatically, so
33+
no DDL scripts need to be run in advance. But to speed up the migration process
34+
you should take a look at the following [postgres server tuning guide](https://gist.github.com/djkazic/526fa3e032aea9578997f88b45b91fb9)
35+
36+
### Using sqlite as the destination remote database
37+
38+
No particular preparation is needed for `sqlite` compared to the `postgres`
39+
case. Similar to the `bolt` case there will be separated db files created for
40+
each individual `bolt` database.
41+
42+
43+
44+
## Prepare the source database
45+
46+
Assuming we want to migrate the database state from the pre-0.19.0 individual
47+
`bbolt` based `*.db` files to a remote database, we first need to make sure the
48+
source files are in the correct state.
49+
50+
The following steps should be performed *before* running the data migration (
51+
some of them are marked as optional and can be neglected for small databases
52+
e.g. 200MB
53+
):
54+
1. Stop `lnd`
55+
2. Upgrade the `lnd` binary to the latest version (e.g. `v0.19.0-beta` or later)
56+
3. (optional) Make sure to add config options like
57+
`gc-canceled-invoices-on-startup=true` and `db.bolt.auto-compact=true` to
58+
your `lnd.conf` to optimize the source database size by removing canceled
59+
invoices and compacting it on startup.
60+
4. (optional) Also make sure to migrate the revocation log for all channels
61+
active prior to `[email protected]` by activating the config setting `--db.prune-revocation`.
62+
This version introduced an optimized revocation log storage system that
63+
reduces the storage footprint. All channels will be automatically migrated
64+
to this new format when the setting is enabled.
65+
5. Start `lnd` normally, using the flags mentioned above but not yet changing
66+
any database backend related configuration options. Check the log that the
67+
database schema was migrated successfully, for example: `Checking for
68+
schema update: latest_version=XX, db_version=XX`. This relates to the
69+
kv-schema NOT any SQL schema, this makes sure all inital migration of the
70+
old database were performed.
71+
6. (optional) Remove any data from the source database that you can. The fewer
72+
entries are in the source database, the quicker the migration will complete.
73+
For example failed payments (or their failed HTLC attempts) can be removed
74+
with `lncli deletepayments`.
75+
7. Stop `lnd` again and make sure it isn't started again by accident during the
76+
data migration (e.g. disable any `systemd` or other scripts that start/stop
77+
`lnd`).
78+
79+
## Run the migration
80+
81+
Depending on the destination database type, run the migration with a command
82+
similar to one of the following examples:
83+
84+
**Example: Migrate from `bbolt` to `sqlite`:**
85+
86+
```shell
87+
lndinit --debuglevel info migrate-db \
88+
--source.bolt.data-dir /home/myuser/.lnd/data \
89+
--dest.backend sqlite \
90+
--dest.sqlite.data-dir /home/myuser/.lnd/data --network mainnet
91+
```
92+
If you were running a watchtower server, and it had a different directory set
93+
compared to the default `LND` directory make sure you also add the tower dir
94+
setting. It has to be the directory to the `watchtower` dir, excluding the
95+
`watchtower` name e.g.:
96+
`--source.bolt.tower-dir /home/myuser/towerdir`.
97+
98+
99+
**Example: Migrate from `bbolt` to `postgres`:**
100+
101+
```shell
102+
lndinit --debuglevel info migrate-db \
103+
--source.bolt.data-dir /home/myuser/.lnd/data \
104+
--dest.backend postgres \
105+
--dest.postgres.dsn=postgres://postgres:postgres@localhost:5432/postgres
106+
```
107+
108+
Also set the watchtower directory in case you used a different path, see above.
109+
110+
This migration tool depends on the directory structure of
111+
the LND software. This means make sure you link the correct folder because the
112+
`bolt` database has several database files in serveral subfolders. This
113+
migration tool will make sure that all the required databases are migrated. It
114+
will not require a `wtclient.db` or a `watchtower.db`.
115+
116+
The migration is resumable and happens in chunks of default 20MB to make it
117+
compatible with most of the systems including low power devices like
118+
raspberry pis. However if you have better setup with way more RAM feel free to
119+
increase the `--chunk-size` to something like
120+
200MB which should speed up the migration significantly. You can also change
121+
the chunk size during the migration as well.
122+
If you want to start the migration from the beginning (can only be done if the
123+
migration did still not succeed) use the flag `--force-new-migration` which can
124+
be used in combination with a new `chunksize` limit.
125+
126+
127+
## After the migration was successful
128+
129+
Make sure the whole migration process succeeds before starting the new node with
130+
the new underlying database. As mentioned above there are several database files
131+
at play here so all of them have to succeed to garantee a successful migration.
132+
133+
If the migration succeeded successfully and you see the following log entry of
134+
the migration tool you are good to go to start-up your lnd node with the new
135+
database.
136+
137+
```shell
138+
[INF]: LNDINIT !!!Migration of all mandatory db parts completed successfully!!!
139+
```
140+
141+
The mandatory dbs are:
142+
* `channel.db`
143+
* `macaroons.db`
144+
* `sphinxreplay.db`
145+
* `wallet.db`
146+
147+
The optional dbs are:
148+
149+
* `wtclient.db`
150+
* `watchtower.db`
151+
* `neutrino.db`
152+
153+
### LND config setting for `sqlite` backend
154+
155+
```shell
156+
[db]
157+
db.backend=sqlite
158+
```
159+
160+
There are several other sqlite setttings you can tweak to make it fit your
161+
needs, take a look at the [lnd-sample-config](https://github.com/lightningnetwork/lnd/blob/b6d8ecc7479f7517368814c398b0fbe0e8c52fed/sample-lnd.conf).
162+
163+
### LND config setting for `postgres` backend
164+
165+
```shell
166+
[db]
167+
db.backend=postgres
168+
169+
[postgres]
170+
171+
db.postgres.dsn=postgres://xx:yy@localhost:5432/zz
172+
```
173+
174+
Use the same connection string you used for the migration. Also take a look at
175+
the postgres knobs in the [lnd-sample-config](https://github.com/lightningnetwork/lnd/blob/b6d8ecc7479f7517368814c398b0fbe0e8c52fed/sample-lnd.conf).
176+
177+
178+
This is the output of the migration cmd help settings:
179+
180+
```shell
181+
lndinit migrate-db -h
182+
2025-04-01 00:29:24.618 [INF]: LNDINIT Version 0.1.24-beta commit=docker/v0.1.22-beta-lnd-v0.18.3-beta.rc1-10-g658c5c8-dirty, debuglevel=
183+
Usage:
184+
lndinit [OPTIONS] migrate-db [migrate-db-OPTIONS]
185+
186+
Migrate the full database state of lnd from a source (for example the
187+
set of bolt database files such as channel.db and wallet.db) database
188+
to a SQL destination database.
189+
190+
IMPORTANT: Please read the data migration guide located in the file
191+
docs/data-migration.md of the main lnd repository before using this
192+
command!
193+
194+
NOTE: The migration can take a long time depending on the amount of data
195+
that needs to be written! The migration happens in chunks therefore it
196+
can be resumed in case of an interruption. The migration also includes
197+
a verification to assure that the migration is consistent.
198+
As long as NEITHER the source nor destination database has been started/
199+
run with lnd, the migration can be repeated/resumed in case of an error
200+
since the data will just be overwritten again in the destination.
201+
202+
Once a database was successfully and completely migrated from the source
203+
to the destination, the source will be marked with a 'tombstone' tag
204+
while the destination will get an 'already migrated' tag.
205+
A database with a tombstone cannot be started with lnd anymore to
206+
prevent from an old state being used by accident.
207+
To prevent overwriting a destination database by accident, the same
208+
database/namespace pair cannot be used as the target of a data migration
209+
twice, which is checked through the 'already migrated' tag.
210+
211+
Application Options:
212+
-e, --error-on-existing Exit with code EXIT_CODE_TARGET_EXISTS (128) instead of 0 if the result of an action is already present
213+
-d, --debuglevel= Set the log level (Off, Critical, Error, Warn, Info, Debug, Trace)
214+
215+
Help Options:
216+
-h, --help Show this help message
217+
218+
[migrate-db command options]
219+
-n, --network= Network of the db files to migrate (used to navigate into the right directory) (default: mainnet)
220+
--pprof-port= Enable pprof profiling on the specified port
221+
--force-new-migration Force a new migration from the beginning of the source DB so the resume state will be discarded
222+
--chunk-size= Chunk size for the migration in bytes
223+
224+
source:
225+
--source.backend=[bolt] The source database backend. (default: bolt)
226+
227+
bolt:
228+
--source.bolt.dbtimeout= Specify the timeout value used when opening the database. (default: 1m0s)
229+
--source.bolt.data-dir= Lnd data dir where bolt dbs are located.
230+
--source.bolt.tower-dir= Lnd watchtower dir where bolt dbs for the watchtower server are located.
231+
232+
dest:
233+
--dest.backend=[postgres|sqlite] The destination database backend. (default: postgres)
234+
235+
postgres:
236+
--dest.postgres.dsn= Database connection string.
237+
--dest.postgres.timeout= Database connection timeout. Set to zero to disable.
238+
--dest.postgres.maxconnections= The maximum number of open connections to the database. Set to zero for unlimited.
239+
240+
sqlite:
241+
--dest.sqlite.data-dir= Lnd data dir where sqlite dbs are located.
242+
--dest.sqlite.tower-dir= Lnd watchtower dir where sqlite dbs for the watchtower server are located.
243+
244+
sqlite-config:
245+
--dest.sqlite.sqlite-config.timeout= The time after which a database query should be timed out.
246+
--dest.sqlite.sqlite-config.busytimeout= The maximum amount of time to wait for a database connection to become available for a query.
247+
--dest.sqlite.sqlite-config.maxconnections= The maximum number of open connections to the database. Set to zero for unlimited.
248+
--dest.sqlite.sqlite-config.pragmaoptions= A list of pragma options to set on a database connection. For example, 'auto_vacuum=incremental'. Note that the flag must be
249+
specified multiple times if multiple options are to be set.
250+
251+
252+
```

0 commit comments

Comments
 (0)