AB on Nostr: DB So the first major refactor for v2 we needed to do is store data locally in a ...
DB
So the first major refactor for v2 we needed to do is store data locally in a structured way. In v1 conceptually we thought of the connected Core Lightning node as just an API server. The browser would make requests and then store the result in memory and the UI would react to the in memory observables. For v2 we are conceptually thinking about the connections as a data source that we need to sync our local DB with. We also need to be able to make SQL type queries on the local data to be able to link all of the different data types together, so IndexedDB seemed to be the way to go.
The IndexedDB API is a little gnarly to work with raw and we also ideally wanted to have the DB be reactive so we could map the UI state directly to the DB and have it update in real time. I did some digging and came across Dexie JS (https://dexie.org/). It is a light, performant wrapper around IndexedDB and comes out of the box with "live queries" which return an observable that updates when DB updates affect that specific query. Clams is already built on the killer combo of Svelte and RXJS which play really well together, so adding Dexie was perfect!
So our connection sync strategy is that when the app loads, or the "Sync" button is clicked for a connection, we will initiate the connection, fetch all the required data, update the DB and that will directly drive the UI. We have a `lastSync` timestamp parameter stored, so the goal would be to only fetch data that has been updated since the last timestamp to avoid being bandwidth inefficient and fetching data we already have. I am still working out how we are going to do that for CoreLN (currently the only connection implemented). I am thinking that we might be able to do it with their SQL plugin and use a query that asks for all data with a timestamp after `lastSync`. Otherwise the proposed "wait" API that Rusty has been working on might be able to work nicely as well.
So v2 now has a nice structured DB that is indexed for efficient queries and reactive so that it can drive the UI directly.
So the first major refactor for v2 we needed to do is store data locally in a structured way. In v1 conceptually we thought of the connected Core Lightning node as just an API server. The browser would make requests and then store the result in memory and the UI would react to the in memory observables. For v2 we are conceptually thinking about the connections as a data source that we need to sync our local DB with. We also need to be able to make SQL type queries on the local data to be able to link all of the different data types together, so IndexedDB seemed to be the way to go.
The IndexedDB API is a little gnarly to work with raw and we also ideally wanted to have the DB be reactive so we could map the UI state directly to the DB and have it update in real time. I did some digging and came across Dexie JS (https://dexie.org/). It is a light, performant wrapper around IndexedDB and comes out of the box with "live queries" which return an observable that updates when DB updates affect that specific query. Clams is already built on the killer combo of Svelte and RXJS which play really well together, so adding Dexie was perfect!
So our connection sync strategy is that when the app loads, or the "Sync" button is clicked for a connection, we will initiate the connection, fetch all the required data, update the DB and that will directly drive the UI. We have a `lastSync` timestamp parameter stored, so the goal would be to only fetch data that has been updated since the last timestamp to avoid being bandwidth inefficient and fetching data we already have. I am still working out how we are going to do that for CoreLN (currently the only connection implemented). I am thinking that we might be able to do it with their SQL plugin and use a query that asks for all data with a timestamp after `lastSync`. Otherwise the proposed "wait" API that Rusty has been working on might be able to work nicely as well.
So v2 now has a nice structured DB that is indexed for efficient queries and reactive so that it can drive the UI directly.