Bookkeeper migration #8410
Open
Bookkeeper migration #8410
+10,663
−5,169
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
madelinevibes
added
Status::Ready for Review
|
Cache cleared, restarting CI. |
|
Can't restart CI since it is too old, a rebase will start it again though. |
rustyrussell
force-pushed
the
guilt/bookkeeper-migrate
branch
from
be18b01 to
f649a49
Compare
rustyrussell
changed the title
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that bookkeeper de-duplicates chain_moves: we need to too! So we add an index to make this efficient. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…> nonchannel in db when we close it. This avoids us keeping references into closed channels. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…ables. We change notify_chain_mvt to wallet_save_chain_mvt, and notify_channel_mvt to wallet_save_channel_mvt, which save to the db and call the notifier themselves. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is mainly for the coming list commands, but it's just more logical to have the common fields at the top. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…nmoves. Iterating through every peer and channel every time can be very slow for large nodes, when calling wallet_coinmoves_extract for listcoinmoves. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is where all the previous work pays off: we can access the coinmoves in the db. Changelog-Added: JSON-RPC: `listchainmoves` and `listchannelmoves` commands to access the audit log of coin movements. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that the channel account on l1 doesn't account for the onchain-fulfilled HTLC: ``` > assert sum(channel1[channel_id]) == -msats_sent_to_2 E assert -50000000000 == -50100000000 E + where -50000000000 = sum([0, -50000000000]) ``` Lisa points out that this is accounted for in the chain moves, instead. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happens if l1 doesn't get a signature, so it doesn't consider the fulfill complete, but l2 does (and thus credits l1). This is a trivial test, but will matter should we later correctly account for channelmoves when onchain: in that case it will actually hard to tell, in general, what HTLC(s) were fulfilled. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Only makes sense to wait on creation, since they neither are deleted nor updated. We also enhance the list commands to take the standard index options. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Changelog-Added: JSON-RPC: `wait`: new subsystems `chainmoves` and `channelmoves`.
And note the other commands in See Also section. Note that this means handling the "outpoint" type. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Changelog-Added: JSON-RPC: `sql` plugin now supports `chainmoves` and `channelmoves` tables.
We don't yet do the other list commands, as they are not append-only: we would need to check deletes and updates. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a unique integer, and very useful for querying changes. Unlike
our generated rowid, it's *stable* across queries.
We still need an explicit rowid column for list commands which don't
(currently) have this.
Here's the documentation diff:
@@ -85,69 +85,69 @@
TABLES
------
-Note that the first column of every table is a unique integer called `rowid`: this is used for related tables to refer to specific rows in their parent. sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key.
+Note that tables which have a `created_index` field use that as the primary key (and `rowid` is an alias to this), otherwise an explicit `rowid` integer primary key is generated, whose value changes on each refresh. This field is used for related tables to refer to specific rows in their parent. (sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key).
The following tables are currently supported:
- `bkpr_accountevents` (see lightning-bkpr-listaccountevents(7))
@@ -119,14 +119,14 @@
- `payment_id` (type `hex`, sqltype `BLOB`)
- `chainmoves` indexed by `account_id` (see lightning-listchainmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
- `timestamp` (type `u64`, sqltype `INTEGER`)
- `primary_tag` (type `string`, sqltype `TEXT`)
- related table `chainmoves_extra_tags`
- - `row` (reference to `chainmoves.rowid`, sqltype `INTEGER`)
+ - `row` (reference to `chainmoves.created_index`, sqltype `INTEGER`)
- `arrindex` (index within array, sqltype `INTEGER`)
- `extra_tags` (type `string`, sqltype `TEXT`)
- `peer_id` (type `pubkey`, sqltype `BLOB`)
@@ -139,7 +139,7 @@
- `blockheight` (type `u32`, sqltype `INTEGER`)
- `channelmoves` indexed by `account_id` (see lightning-listchannelmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
@@ -204,7 +204,7 @@
- `last_stable_connection` (type `u64`, sqltype `INTEGER`)
- `forwards` indexed by `in_channel` and `in_htlc_id` (see lightning-listforwards(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `in_channel` (type `short_channel_id`, sqltype `TEXT`)
- `in_htlc_id` (type `u64`, sqltype `INTEGER`)
- `in_msat` (type `msat`, sqltype `INTEGER`)
@@ -222,7 +222,7 @@
- `htlcs` indexed by `short_channel_id` and `id` (see lightning-listhtlcs(7))
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `id` (type `u64`, sqltype `INTEGER`)
- `expiry` (type `u32`, sqltype `INTEGER`)
@@ -242,7 +242,7 @@
- `bolt12` (type `string`, sqltype `TEXT`)
- `local_offer_id` (type `hash`, sqltype `BLOB`)
- `invreq_payer_note` (type `string`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `pay_index` (type `u64`, sqltype `INTEGER`)
- `amount_received_msat` (type `msat`, sqltype `INTEGER`)
@@ -408,7 +408,7 @@
- `features` (type `hex`, sqltype `BLOB`)
- `sendpays` indexed by `payment_hash` (see lightning-listsendpays(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `id` (type `u64`, sqltype `INTEGER`)
- `groupid` (type `u64`, sqltype `INTEGER`)
- `partid` (type `u64`, sqltype `INTEGER`)
Changelog-Changed: Plugins: `sql` tables `forwards`, `htlcs`, `invoices`, `sendpays` all use `created_index` as their primary key (and `rowid` is now an alias to this).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Remove the rebalance field from channel_event, and use the find_rebalance(bkpr, ev->db_id) to look it up instead. chain_event's also had a `rebalance` field, but it was only ever set (to false), never read. Note: list_rebalances() was only used by tests, not a public API. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to access it in stmt2chain_event, so plumb it through. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We won't be able to "UPDATE chain_events", so keep a separate record of these blockheights, and lookup that when the blockheight is 0. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Python's assert gives great analysis of what the differences are, making debugging much easier. So feed it dicts, not tuples, and simply do an assert. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…njection. For the moment, we'll continue to use bookkeeper to monitor the notifications to insert these (we don't have the internal infrastructure for that, and actually these commands are probably better than using notifications). We hoist param_outpoint() into common code, since there are already two uses. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…njected. This allows the bookkeeper plugin to know it's not actually a channel account. Remove the "ignored" tag from the schema too: we removed it previously. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
… transfer_from.
Before bkpr_listaccountevents() gave entries with origin like:
{'account': "nifty's secret stash",
'blockheight': 111,
'credit_msat': 180000000,
'currency': 'bcrt',
'debit_msat': 0,
'origin': 'null',
'outpoint': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:0',
'tag': 'deposit',
'timestamp': 1679955976,
'type': 'chain'},
Changelog-Changed: Plugins: "utxo_deposit" is allows to have missing `transfer_from`, and null is not considered an account name.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…xodeposit / injectutxospend calls. And thus we absorb them as normal when they come back as "foreign" entries. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After much thought and mis-steps, I chose a simple solution: open another fd for sync comms. It's almost impossible to know what state the async one is in. jsonrpc_request_sync() is enhanced to return a valid tal object, as the current behaviour of returning a pointer to inside an array was surprising. Changelog-Changed: libplugin: you can now call the synchronous API functions at any time (not just in the init callback). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a great test, but it's very hard to simulate now we are going to be going from the internal db. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rearrange all the JSON interfaces to call refresh_moves() (async) before doing anything. This does nothing for now, but it will be useful once we transition from notifications to using the list commands. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is reliable, meaning we should never get replayed events. We have to reference count to make sure all commands are complete, before we return. In particular, annotating with descriptions can involve several calls to list commands. We need to give them the results *after* this is all complete. test_bookkeeping_descriptions() relied on log messages from notifications, which now only happen when a command is called. This changes the test a bit. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't need it now bookkeeper uses the list commands. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to be using this instead of our internal db. I also made json_out_obj() take the str arg, as it didn't and I expected it to. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Cleaner (I'm about to hand it a sha256 on the stack). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With some help (and hinderance!) from ChatGPT: the field names differ slightly from our internal db. The particilar wrinkle is that we have to restrict all queries to limit them to entries we've seen already. Our code expects this (we used to only enter it into the db when we processed it), and it would otherwise be confusing if a sql query returned inconsistent results because an event occurred while bookkeeper was processing. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There will be no more missing events (and at initialization time, we will do that as a migration). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `bookkeeper` now uses the lightningd database, not "accounts.db". Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now handles when we remove the db. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And gracefully fail for this case. There's no such thing for Postgres, but that's because dbs need to be set up by the admin. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We take over the --bookkeeper-dir and --bookkeeper-db options, and then if we can find the bookkeeper db we extract the records to initialize our chain_moves and channel_moves tables. Of course, bookkeeper now needs to not register those options. When bookkeeper gets invoked the first time, it will reconstruct everything from listchannelmoves and listcoinmoves. It cannot preserve manually-added descriptions, so we put those in the datastore for it ready to go. Note that the order of onchain_fee changes slightly from the original. But this is fine. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't have an accountdb from bookkeeper: 1. Generate a deposit chain event for every confirmed UTXO. 2. Generate an open chain event for every open, confirmed channel. 3. Generate a push/lease event if necessary. 4. Generate a fixup "journal" entry if balance is different from initial. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
I've done some more work:
I appended the fixes needed, for later rebasing. |
rustyrussell
force-pushed
the
guilt/bookkeeper-migrate
branch
2 times, most recently
from
41e21d6 to
ef8cb22
Compare
rustyrussell
force-pushed
the
guilt/bookkeeper-migrate
branch
from
ef8cb22 to
5ab28b0
Compare
niftynei
reviewed
Aug 16, 2025
This requires us to turn "sql" calls into calls to a local db, which means pulling in a lot of infrastructure. But it's possible. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on #8445
We do all the things:
It needs some more testing, and I'm sure it dosn't pass CI, but it's ready for review!
Closes: #8393