rhavar at protonmail.com [ARCHIVE]

⚠️ This account is not associated with a real person. It's an archival account for mailing list messages.
🗓️ Active from 2018-02-12 to 2022-08-25
💬 Posted a total of 28 messages
📚 Participated in the following categories: bitcoin-dev, PROPOSAL
🔇 Please do not DM this user as it's used for archive purposes only.

Public Key

npub1glk4wtcgyl40v0zr2sz4vsxg5c00zx3ffrf3amr362pgx3fcd39q8l5ays

Profile Code

nprofile1qqsy0m2h9uyz06hk83p4gp2kgry2v8h3rg5535c7a3ca9q5rg5uxcjspz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0dskvj960

Publishing to

relay.damus.io

nos.lol

Author Public Key

npub1glk4wtcgyl40v0zr2sz4vsxg5c00zx3ffrf3amr362pgx3fcd39q8l5ays

Show more details

Last Notes

2023-06-08 01:13:06

- reply

📅 Original date posted:2022-08-25
📝 Original message:> Not only is JSON limited to editing only through specific software or text editors, but (in the latter case) it is fragile enough that a single missing character can cause an entire file to fail parsing. CSV is more forgiving in this regard.
I think quite simply: A forgiving format is not appropriate for a standard.
It'd be hard to understate how much extra and pointless effort it creates for everyone, and every implementation ends up creating its own defacto standard for what it produces and accepts. Even doing something as simple as adding an extra column will not be possible in the future because it'll break comparability with previous parsers.
I've literally worked on projects where the csv parser has evolved into scan-ahead to use heuristics to understand "rules" of a csv file, and then do line-by-line heuristics to override those rules in pathological cases. Makes a bit of sense when you're trying to achieve 30 years of backwards compatibility. Doesn't make sense for much else..
If your application users really like csv, then introduce an application-specific import-from-csv and export-to-csv with your own rules.
-Ryan
------- Original Message -------
On Thursday, August 25th, 2022 at 1:59 AM, Craig Raw <craigraw at gmail.com> wrote:
> Thanks for your thoughts Ryan.
>
> Without reference to the quality feedback on this proposal, I was aware when submitting it for review that it provides an excellent opportunity for bike shedding. As developers, we have all experienced frustration with data formats. One thing that I did not perhaps make clear enough is that this format is not solely intended for developers, but general users who are probably not well represented on this list.
>
> While doing research for this proposal I spoke to several professional users of Sparrow Wallet (who are not developers). They all expressed a desire for the format to integrate with their business processes, which are driven by business tools such as Excel. Labelling provides an important function in UTXO and address management in these scenarios, and needs to be accessible and manageable outside of wallet software.
>
> If this is to be achieved, it immediately rules out JSON as a data format. Not only is JSON limited to editing only through specific software or text editors, but (in the latter case) it is fragile enough that a single missing character can cause an entire file to fail parsing. CSV is more forgiving in this regard. With respect to your comments on escaping, my expectation would be that developers will be using a mature CSV library rather than handling character escaping themselves. I would rather propose a format that is generally usable, even if occasionally a label is escaped incorrectly.
>
> Finally, I'll note that CSV files are already common and uncontroversial in Bitcoin wallet software. Bitcoin Core, Electrum, Sparrow (and no doubt many others) already export addresses and/or transactions with their labels as CSV files. This proposal simply attempts to create a standard for importing and exporting all the labels in a wallet.
>
> Craig
>
> On Wed, Aug 24, 2022 at 9:01 PM <rhavar at protonmail.com> wrote:
>
>> I'd strongly suggest not using CSV. Especially for a standard. I've worked with it as an interchange format many a times, and it's always been a clusterfuck.
>>
>> Right off the bat, you have stuff like "The fields may be quoted, but this is unnecessary as the first comma in the line will always be the delimiter" which invariably leads to some implementations doing it, some implementations not doing it, and others that are intolerant of the other way.
>>
>> And you have also made the classic mistake of not strictly defining escape rules. So everyone will pick their own (e.g. some will \, escape commas, others will not cause it's quoted and escape quotes, and others will assume no escaping is required since its the last column in a csv).
>>
>> Over time it morphs into its own mini-monster that introduces so much pain.
>>
>> On a similar note, allowing alternatives (like: txid>index vs txid:index) provides no benefit, but creates additional work for implementations (who quite likely only test formats they produce) and future incompatibilities.
>>
>> I know everyone loves to hate on it, but really (line-separated?) json is the way to go.
>>
>> { "tx": "c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎", "label": "wow, such label" }
>> { "tx: "c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b", "txout": 4, "label": "omg this is so easy to parse" }
>> { "tx: "c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b", "txin": 0, "label": "wow this is going to be extensible as well" }
>>
>> -Ryan
>>
>> ------- Original Message -------
>> On Wednesday, August 24th, 2022 at 2:18 AM, Craig Raw via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>>
>>> Hi all,
>>>
>>> I would like to propose a BIP that specifies a format for the export and import of labels from a wallet. While transferring access to funds across wallet applications has been made simple through standards such as BIP39, wallet labels remain siloed and difficult to extract despite their value, particularly in a privacy context.
>>>
>>> The proposed format is a simple two column CSV file, with the reference to a transaction, address, input or output in the first column, and the label in the second column. CSV was chosen for its wide accessibility, especially to users without specific technical expertise. Similarly, the CSV file may be compressed using the ZIP format, and optionally encrypted using AES.
>>>
>>> The full text of the BIP can be found at https://github.com/craigraw/bips/blob/master/bip-wallet-labels.mediawiki and also copied below.
>>>
>>> Feedback is appreciated.
>>>
>>> Thanks,
>>> Craig Raw
>>>
>>> ---
>>>
>>> <pre>
>>> BIP: wallet-labels
>>> Layer: Applications
>>> Title: Wallet Labels Export Format
>>> Author: Craig Raw <craig at sparrowwallet.com>
>>> Comments-Summary: No comments yet.
>>> Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-wallet-labels
>>> Status: Draft
>>> Type: Informational
>>> Created: 2022-08-23
>>> License: BSD-2-Clause
>>> </pre>
>>>
>>> ==Abstract==
>>>
>>> This document specifies a format for the export of labels that may be attached to the transactions, addresses, input and outputs in a wallet.
>>>
>>> ==Copyright==
>>>
>>> This BIP is licensed under the BSD 2-clause license.
>>>
>>> ==Motivation==
>>>
>>> The export and import of funds across different Bitcoin wallet applications is well defined through standards such as BIP39, BIP32, BIP44 etc.
>>> These standards are well supported and allow users to move easily between different wallets.
>>> There is, however, no defined standard to transfer any labels the user may have applied to the transactions, addresses, inputs or outputs in their wallet.
>>> The UTXO model that Bitcoin uses makes these labels particularly valuable as they may indicate the source of funds, whether received externally or as a result of change from a prior transaction.
>>> In both cases, care must be taken when spending to avoid undesirable leaks of private information.
>>> Labels provide valuable guidance in this regard, and have even become mandatory when spending in several Bitcoin wallets.
>>> Allowing users to export their labels in a standardized way ensures that they do not experience lock-in to a particular wallet application.
>>> In addition, by using common formats, this BIP seeks to make manual or bulk management of labels accessible to users without specific technical expertise.
>>>
>>> ==Specification==
>>>
>>> In order to make the import and export of labels as widely accessible as possible, this BIP uses the comma separated values (CSV) format, which is widely supported by consumer, business, and scientific applications.
>>> Although the technical specification of CSV in RFC4180 is not always followed, the application of the format in this BIP is simple enough that compatibility should not present a problem.
>>> Moreover, the simplicity and forgiving nature of CSV (over for example JSON) lends itself well to bulk label editing using spreadsheet and text editing tools.
>>>
>>> A CSV export of labels from a wallet must be a UTF-8 encoded text file, containing one record per line, with records containing two fields delimited by a comma.
>>> The fields may be quoted, but this is unnecessary, as the first comma in the line will always be the delimiter.
>>> The first line in the file is a header, and should be ignored on import.
>>> Thereafter, each line represents a record that refers to a label applied in the wallet.
>>> The order in which these records appear is not defined.
>>>
>>> The first field in the record contains a reference to the transaction, address, input or output in the wallet.
>>> This is specified as one of the following:
>>> * Transaction ID (<tt>txid</tt>)
>>> * Address
>>> * Input (rendered as <tt>txid<index</tt>)
>>> * Output (rendered as <tt>txid>index</tt> or <tt>txid:index</tt>)
>>>
>>> The second field contains the label applied to the reference.
>>> Exporting applications may omit records with no labels or labels of zero length.
>>> Files exported should use the <tt>.csv</tt> file extension.
>>>
>>> In order to reduce file size while retaining wide accessibility, the CSV file may be compressed using the ZIP file format, using the <tt>.zip</tt> file extension.
>>> This <tt>.zip</tt> file may optionally be encrypted using either AES-128 or AES-256 encryption, which is supported by numerous applications including Winzip and 7-zip.
>>> In order to ensure that weak encryption does not proliferate, importers following this standard must refuse to import <tt>.zip</tt> files encrypted with the weaker Zip 2.0 standard.
>>> The textual representation of the wallet's extended public key (as defined by BIP32, with an <tt>xpub</tt> header) should be used as the password.
>>>
>>> ==Importing==
>>>
>>> When importing, a naive algorithm may simply match against any reference, but it is possible to disambiguate between transactions, addresses, inputs and outputs.
>>> For example in the following pseudocode:
>>> <pre>
>>> if reference length < 64
>>> Set address label
>>> else if reference length == 64
>>> Set transaction label
>>> else if reference contains '<'
>>> Set input label
>>> else
>>> Set output label
>>> </pre>
>>>
>>> Importing applications may truncate labels if necessary.
>>>
>>> ==Test Vectors==
>>>
>>> The following fragment represents a wallet label export:
>>> <pre>
>>> Reference,Label
>>> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎,Transaction
>>> 1A69TXnEM2ms9fMaY9UuiJ7415X7xZaUSg,Address
>>> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎<0,Input
>>> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎>0,Output
>>> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎:0,Output (alternative)
>>> </pre>
>>>
>>> ==Reference Implementation==
>>>
>>> TBD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20220825/b57b0503/attachment-0001.html>

2023-06-08 01:13:03

- reply

📅 Original date posted:2022-08-24
📝 Original message:I'd strongly suggest not using CSV. Especially for a standard. I've worked with it as an interchange format many a times, and it's always been a clusterfuck.
Right off the bat, you have stuff like "The fields may be quoted, but this is unnecessary as the first comma in the line will always be the delimiter" which invariably leads to some implementations doing it, some implementations not doing it, and others that are intolerant of the other way.
And you have also made the classic mistake of not strictly defining escape rules. So everyone will pick their own (e.g. some will \, escape commas, others will not cause it's quoted and escape quotes, and others will assume no escaping is required since its the last column in a csv).
Over time it morphs into its own mini-monster that introduces so much pain.
On a similar note, allowing alternatives (like: txid>index vs txid:index) provides no benefit, but creates additional work for implementations (who quite likely only test formats they produce) and future incompatibilities.
I know everyone loves to hate on it, but really (line-separated?) json is the way to go.
{ "tx": "c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎", "label": "wow, such label" }
{ "tx: "c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b", "txout": 4, "label": "omg this is so easy to parse" }
{ "tx: "c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b", "txin": 0, "label": "wow this is going to be extensible as well" }
-Ryan
------- Original Message -------
On Wednesday, August 24th, 2022 at 2:18 AM, Craig Raw via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi all,
>
> I would like to propose a BIP that specifies a format for the export and import of labels from a wallet. While transferring access to funds across wallet applications has been made simple through standards such as BIP39, wallet labels remain siloed and difficult to extract despite their value, particularly in a privacy context.
>
> The proposed format is a simple two column CSV file, with the reference to a transaction, address, input or output in the first column, and the label in the second column. CSV was chosen for its wide accessibility, especially to users without specific technical expertise. Similarly, the CSV file may be compressed using the ZIP format, and optionally encrypted using AES.
>
> The full text of the BIP can be found at https://github.com/craigraw/bips/blob/master/bip-wallet-labels.mediawiki and also copied below.
>
> Feedback is appreciated.
>
> Thanks,
> Craig Raw
>
> ---
>
> <pre>
> BIP: wallet-labels
> Layer: Applications
> Title: Wallet Labels Export Format
> Author: Craig Raw <craig at sparrowwallet.com>
> Comments-Summary: No comments yet.
> Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-wallet-labels
> Status: Draft
> Type: Informational
> Created: 2022-08-23
> License: BSD-2-Clause
> </pre>
>
> ==Abstract==
>
> This document specifies a format for the export of labels that may be attached to the transactions, addresses, input and outputs in a wallet.
>
> ==Copyright==
>
> This BIP is licensed under the BSD 2-clause license.
>
> ==Motivation==
>
> The export and import of funds across different Bitcoin wallet applications is well defined through standards such as BIP39, BIP32, BIP44 etc.
> These standards are well supported and allow users to move easily between different wallets.
> There is, however, no defined standard to transfer any labels the user may have applied to the transactions, addresses, inputs or outputs in their wallet.
> The UTXO model that Bitcoin uses makes these labels particularly valuable as they may indicate the source of funds, whether received externally or as a result of change from a prior transaction.
> In both cases, care must be taken when spending to avoid undesirable leaks of private information.
> Labels provide valuable guidance in this regard, and have even become mandatory when spending in several Bitcoin wallets.
> Allowing users to export their labels in a standardized way ensures that they do not experience lock-in to a particular wallet application.
> In addition, by using common formats, this BIP seeks to make manual or bulk management of labels accessible to users without specific technical expertise.
>
> ==Specification==
>
> In order to make the import and export of labels as widely accessible as possible, this BIP uses the comma separated values (CSV) format, which is widely supported by consumer, business, and scientific applications.
> Although the technical specification of CSV in RFC4180 is not always followed, the application of the format in this BIP is simple enough that compatibility should not present a problem.
> Moreover, the simplicity and forgiving nature of CSV (over for example JSON) lends itself well to bulk label editing using spreadsheet and text editing tools.
>
> A CSV export of labels from a wallet must be a UTF-8 encoded text file, containing one record per line, with records containing two fields delimited by a comma.
> The fields may be quoted, but this is unnecessary, as the first comma in the line will always be the delimiter.
> The first line in the file is a header, and should be ignored on import.
> Thereafter, each line represents a record that refers to a label applied in the wallet.
> The order in which these records appear is not defined.
>
> The first field in the record contains a reference to the transaction, address, input or output in the wallet.
> This is specified as one of the following:
> * Transaction ID (<tt>txid</tt>)
> * Address
> * Input (rendered as <tt>txid<index</tt>)
> * Output (rendered as <tt>txid>index</tt> or <tt>txid:index</tt>)
>
> The second field contains the label applied to the reference.
> Exporting applications may omit records with no labels or labels of zero length.
> Files exported should use the <tt>.csv</tt> file extension.
>
> In order to reduce file size while retaining wide accessibility, the CSV file may be compressed using the ZIP file format, using the <tt>.zip</tt> file extension.
> This <tt>.zip</tt> file may optionally be encrypted using either AES-128 or AES-256 encryption, which is supported by numerous applications including Winzip and 7-zip.
> In order to ensure that weak encryption does not proliferate, importers following this standard must refuse to import <tt>.zip</tt> files encrypted with the weaker Zip 2.0 standard.
> The textual representation of the wallet's extended public key (as defined by BIP32, with an <tt>xpub</tt> header) should be used as the password.
>
> ==Importing==
>
> When importing, a naive algorithm may simply match against any reference, but it is possible to disambiguate between transactions, addresses, inputs and outputs.
> For example in the following pseudocode:
> <pre>
> if reference length < 64
> Set address label
> else if reference length == 64
> Set transaction label
> else if reference contains '<'
> Set input label
> else
> Set output label
> </pre>
>
> Importing applications may truncate labels if necessary.
>
> ==Test Vectors==
>
> The following fragment represents a wallet label export:
> <pre>
> Reference,Label
> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎,Transaction
> 1A69TXnEM2ms9fMaY9UuiJ7415X7xZaUSg,Address
> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎<0,Input
> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎>0,Output
> c3bdad6e7dcd7997e16a5b7b7cf4d8f6079820ff2eedd5fcbb2ad088f767b37b‎:0,Output (alternative)
> </pre>
>
> ==Reference Implementation==
>
> TBD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20220824/f1fcaed1/attachment-0001.html>

2023-06-08 01:12:59

- reply

📅 Original date posted:2022-08-20
📝 Original message:Of all wallets I've tried, by a huge margin, bitcoin core performs the best with large wallets. I know several massive casinos like bustabit (which has a lot more transactions than your wallet) have been using bitcoin core for large wallets, so it can work.
The reason I don't recommend normal people use core, is because of the initial-block-download and not defaulting to a pruned mode.
The thing that makes your wallet slow is the large amount of wallet transactions. So the secret to running a large bitcoin core wallet is to just call `RemovePrunedFunds` on old transactions. Its actually a little tricky though, because you need to make sure the transaction is "safe to remove". The first, most obvious, reason a transaction would not be safe to remove is if it has a wallet utxo that you have not spent or have recently spent. The second, less obvious reason is if transaction B spends from transaction A, and you remove transaction B your wallet will think transaction A is unspent (but it's not!). So you have to prune "Depth first".
-Ryan
------- Original Message -------
On Saturday, August 20th, 2022 at 7:16 AM, Felipe Micaroni Lalli via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi dear devs,
>
> 1. THE ISSUE - DAEMON+CLI
> ========================
> I had a wallet in a server production since 2017 (5 years old) and when it reached about 273 MB, 2.079.337 transactions and 446.503 generated addresses, the performance started to degrade exponentially.
>
> Most of the commands, e.g. "getbalance", "walletpassphrase" and "getreceivedbyaddress" started to timeout (more than 15 minutes delay - default timeout). The CPU was 100% used (all 32 cores - with 150 load avg) and the machine became almost unusable breaking everything else, with the default config of 16 RPC threads and 15 min timeout and some attempt calls per mi
>
> Increasing the timeout and/or the RPC threads in the config file turns things even worse.
>
> Putting the wallet.dat in a very fast SSD disk and increasing the size of the cache (I tried with 8GB) have improved but I'm not sure if it is enough.
>
> 2. TEST ON BITCOIN QT
> ====================
> If you try to load the wallet in the "bitcoin-qt" everything gets stuck, even the system (OS/UI) doesn't respond anymore. You click on a button and receive the message "window doesn't respond, wait or terminate?" - if you wait it releases after a while but it is slow and hard to use the wallet anyway.
>
> 3. WHY IS THIS SO BAD?
> =====================
> This is bad because the standard client becomes almost useless for the wallet feature:
>
> 3.1) the wallet Qt already is not so popular among end users. It doesn't look modern, slow to first sync and hard to use. That's why people prefer to use Electrum or Wasabi - I personally don't care but it's the sad truth;
>
> 3.2) it becomes useless now also for servers in production, forcing them to use third party solutions for huge wallets. Even if you split in 10 wallets it will just delay 10 times more each to degrade, postponing the problem but not eliminating it. Not to mention the slow and daily degradation.
>
> 4. SHOULD WE GIVE UP THE WALLET FEATURE?
> ========================================
> Then, Bitcoin Core becomes just a reference implementation and blocks relayers, but as an application wallet itself turns into a really bad choice. --- It leads me to the following question: if we won't invest time on improving this, shouldn't we remove the wallet feature at all? Why keep a wallet feature that is not useful for the end user nor the production server? Is it useful for what then?
>
> 5. THE CURRENT "SOLUTION" IS BAD
> ===============================
> Currently, the only "solution" for huge wallets is shameful: create a new one and send the funds there from time to time. But when is the right time exactly? The performance degrades suddenly or gets worse slowly for each new address and/or tx?. And besides not being an elegant solution and "not in the handbook", it also can break a lot of things like monitoring old addresses and also can lead to privacy concerns unifying lots of inputs in a big and expensive tx.
>
> 6. OTHER USER CASES?
> ====================
> I think this could also become an issue if we have LN nodes that use the Bitcoin Core wallet infrastructure behind to open / close many channels for a long time.
>
> 7. FINAL THOUGHTS
> =================
> If moving the wallet from a HDD to a SSD improved a lot, maybe just caching the entire wallet in memory could improve even more, but I'm afraid some code optimization is also necessary.
>
> 8. SOME QUESTIONS
> ==================
> 8.1) Can we "optimize" a huge wallet without moving the funds to a new one? Like a "fsck" or eqv?
>
> 8.2) Can we improve the cache usage somehow? Putting the entire wallet in memory, for example?
>
> 8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD wallet)?
>
> 8.4) Can we tell the CLI to ignore old addresses? What if I need to watch only the last 30 days?
>
> 8.5) How to improve the I/O treatment and/or CPU usage in the main thread on Bitcoin-Qt to avoid window freezing on big and huge wallets?
>
> 8.6) In the last case (if it was not possible to optimize the wallet or the CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too big and slow, execute the command 'archive'". And then, the command "archive" could rename the current wallet to something like "wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the funds automatically? Also, would it be nice to have an "autoarchivehugewallets=1" in the file config?
>
> 9. POSSIBLE RELATED AND TESTS
> =============================
>
> [1] https://github.com/bitcoin/bitcoin/issues/15015
> [2] https://github.com/bitcoin/bitcoin/issues/15148
> [3] https://github.com/bitcoin/bitcoin/issues/16874
> [4] https://github.com/bitcoin/bitcoin/pull/17135
> [5] https://github.com/bitcoin/bitcoin/pull/18160
> [6] https://github.com/bitpay/bitcore-node/issues/463
> [7] https://github.com/RavenProject/Ravencoin/issues/499
> [8] https://github.com/sugarchain-project/sugarchain/issues/106
> [9] https://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and-times-out
> [10] https://bitcoin.stackexchange.com/a/45713/1761
>
> ANOTHER POSSIBLE BUG
> ======================
>
> Even if my node is 100% sync:
> 2022-08-20T13:11:43Z UpdateTip: new best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489 date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo)
>
> After a "loadwallet" command I am getting an old / wrong balance. The wallet is already empty because I had moved the funds to a new one 3 or 4 days ago but it is still showing the old balance. I didn't receive any warning message saying the need for a rescan or something like that.
>
> I am trying the "rescanblockchain" command but it is running and it taking a looooooooooooooong time.
>
> Best regards,
>
> Felipe.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20220820/e45141bc/attachment-0001.html>

2023-06-07 20:18:32

- reply

📅 Original date posted:2019-06-10
📝 Original message:What about this? We store a RBU ("relay bandwidth used") with each transaction in the mempool. Where RBU is defined as the size of the transaction + RBU of all transactions it has evicted.
For a replacement to be valid: The feerate must be higher than what it's evicting, and the fee must be greater than minRelayFee*RBU.
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, June 9, 2019 7:07 AM, David A. Harding via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> On Thu, Jun 06, 2019 at 02:46:54PM +0930, Rusty Russell via bitcoin-dev wrote:
>
> > Matt Corallo lf-lists at mattcorallo.com writes:
> >
> > > 2. wrt rule 4, I'd like to see a calculation of worst-case free
> > > relay. I think we're already not in a great place, but maybe it's
> > > worth it or maybe there is some other way to reduce this cost
> > > (intuitively it looks like this proposal could make things very,
> > > very, very bad).
> > >
> >
> > I think you can currently create a tx at 1 sat/byte, have it
> > propagate, then RBF it to 2 sat/byte, 3... and do that a few thousand
> > times before your transaction gets mined.
>
> Yes, the current incremental relay fee in Bitcoin Core is 0.00001000
> BTC/KvB.
>
> > If that's true, I don't think this proposal makes it worse.
>
> Here's a scenario that I think shows it being at least 20x worse.
>
> Let's imagine that you create two transactions:
>
> tx0: A very small transaction (~100 vbytes) that's just 1-in, 1-out.
> At the minimum relay fee, this costs 0.00000100 BTC
>
> tx1: A child of that transaction that's very large (~100,000 vbytes,
> or almost 400,000 bytes using special scripts that allow witness
> stuffing). At the minimum relay fee, this costs 0.00100000 BTC
>
> Under the current rules, if an attacker wants to fee-bump tx0 by the
> minimum incremental fee (a trivial amount, ~0.00000100 BTC), the
> attacker's replacement also needs to pay for the eviction of the huge
> child tx1 by that same incremental fee (~0.00100000).
>
> Thus the replacement would cost the attacker a minimum of about
> 0.00100100 (~1 mBTC) for the original transactions and 0.00100100 for
> the replacement (about 2 mBTC total).
>
> The attacker could then spend another 1 mBTC re-attaching the child and
> yet another 1 mBTC bumping again, incuring about a 2 mBTC cost per
> replacement round. At writing, 2 mBTC is about $14.00 USD---an amount
> that's probably enough to deter most attacks at scale.
>
> Under the new proposed rule 6, Mallory's replacement cost would be the
> amount to get the small tx0 to near the top of the mempool (say
> 0.00100000 BTC/KvB, so 0.00010000 BTC total). Because this would evict
> the expensive child, it would actually reduce the original amount paid
> by the attacker by 90% compared to the previous section's example where
> using RBF increased the original costs by 100%.
>
> The 0.1 mBTC cost of this attack is about $0.70 USD today for the
> roughly the same data relay use as one round of the currently possible
> attack. In short, if I haven't misplaced a decimal point or made some
> other mistake, I think the proposed rule 6 would result in approximately
> a 95% reduction in the cost paid by an attacker for wasting 400,000
> bytes of bandwidth per node (x60,000 nodes = 24 GB across all nodes, not
> counting INV overhead).
>
> Although the attacker might only get one replacement per block per
> transaction pair out of this version of the attack, they could execute
> the attack many times in parallel using different tranaction pairs. If
> this is combined with the treadmill leapfrogging Russell O'Connor
> described elsewhere in this thread, the attack could possibly be
> repeated multiple times per block per transaction pair at only slightly
> increased cost (to pay the increasing next-block transaction fees).
>
> > > > 3. wrt rule 5, I'd like to see benchmarks, it's probably a pretty
> > > > nasty DoS attack, but it may also be the case that is (a) not worse
> > > > than other fundamental issues or (b) sufficiently expensive.
> > > >
> >
> > I thought we still meet rule 5 in practice since bitcoind will never
> > even accept a tree of unconfirmed txs which is > 100 txs? That would
> > still stand, it's just that we'd still consider a replacement.
>
> Although the BIP125 limit is 100, Bitcoin Core's current default is 25.[1]
> (When RBF was implemented in Bitcoin Core, transaction ancestry was only
> tracked for purposes of ensuring valid transaction ordering within
> blocks; later when CPFP was implemented, ancestry was additionally used
> to calculate each transaction's package fee---the value of it and all
> its unconfirmed ancestors. This requires more computation to update
> the mempool metadata when the ancestry graph changes.)
>
> Again, I'd be thinking here of something similar to O'Connor's
> treadmilling attack where replacements can push each other out of the
> top mempool and so create enough churn for a CPU exhaustion attack.
>
> > > > Obviously there is also a ton more client-side knowledge required
> > > > and complexity to RBF decisions here than other previous, more
> > > > narrowly-targeted proposals.
> > > > I'd say from the lightning side it's as simple as a normal RBF policy
> > > > until you get within a few blocks of a deadline, then you increase the
> > > > fees until it's well within reach of the next block.
>
> It's already hard for wallet software to determine whether or not its
> transactions have successfully been relayed. This proposal requires LN
> wallets not only be able to guess where the next-block feerate boundary
> is in other nodes' individual mempools (now and in the future for the
> time it takes the transaction to propagate to ~100% of miners), but it
> possibly requires that under the condition that the LN wallet can't
> guess too low because it might not get another chance for relay in the
> limited time available before contract expiration.
>
> On top of that, there's O'Connor's suggestion to increase treadmilling
> costs by only allowing bumps if they're in the top-half of the
> next-block mempool.
>
> Considered that way, I worry that these constraints produce a recipe for
> paying extremely high feerates. If that's an actual risk, is that
> actually significantly better than dealing with the existing transaction
> pinning issue where one needs to pay a high total fee in order to evict
> a bunch of junk descendents? Paying lots of fees may not be the optimal
> solution to the problem of having to pay lots of fees. :-)
>
> -Dave
>
> [1] Excerpt from bitcoind -help-debug :
>
> -limitancestorcount=<n>
>
> Do not accept transactions if number of in-mempool ancestors is <n> or
>
> more (default: 25)
>
>
> -limitdescendantcount=<n>
>
> Do not accept transactions if any ancestor would have <n> or more
>
> in-mempool descendants (default: 25)
>
>
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

2023-06-07 20:18:31

- reply

📅 Original date posted:2019-06-02
📝 Original message:+1
>From an incentive-compatible point of view, miners should be accepting transactions that increase the amount of fees that can achieved with 4M weight of transactions, so it seems like a pretty sane plan.
One common problem I've run into with RBF is since you're using RBF you probably want to low ball fees. With good coin selection (*cough* coinsayer.com *cough*), it'll use that opportunity to consolidate inputs. But now let's say fees suddenly spike (pretty common), you might want to fee bump your now stuck transaction. But now that fees are high, it doesn't make sense to be consolidating so ideally you'd just replace it with a much smaller transaction (that pays higher fee rate).
So if anything, I think your proposal doesn't go far enough. I think even in "non-emergency" cases, we could get away with removing the requirement to increase the absolute fee (as long as the fee rate is increased); which also makes it incentive compatible if you assume a reasonable fee-market.
I realize it does open potential DoS vectors, but they seem reasonably small.
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, June 1, 2019 9:41 PM, Rusty Russell via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi all,
>
> I want to propose a modification to rules 3, 4 and 5 of BIP 125:
>
> To remind you of BIP 125:
> 3. The replacement transaction pays an absolute fee of at least the sum
> paid by the original transactions.
>
> 4. The replacement transaction must also pay for its own bandwidth at
> or above the rate set by the node's minimum relay fee setting.
>
> 5. The number of original transactions to be replaced and their
> descendant transactions which will be evicted from the mempool must not
> exceed a total of 100 transactions.
>
> The new "emergency RBF" rule:
>
> 6. If the original transaction was not in the first 4,000,000 weight
> units of the fee-ordered mempool and the replacement transaction is,
> rules 3, 4 and 5 do not apply.
>
> This means:
>
> 1. RBF can be used in adversarial conditions, such as lightning
> unilateral closes where the adversary has another valid transaction
> and can use it to block yours. This is a problem when we allow
> differential fees between the two current lightning transactions
> (aka "Bring Your Own Fees").
>
> 2. RBF can be used without knowing about miner's mempools, or that the
> above problem is occurring. One simply gets close to the required
> maximum height for lightning timeout, and bids to get into the next
> block.
>
> 3. This proposal does not open any significant new ability to RBF spam,
> since it can (usually) only be used once. IIUC bitcoind won't
> accept more that 100 descendents of an unconfirmed tx anyway.
>
> 4. This proposal makes RBF miner-incentive compatible. Currently the
> protocol tells miners they shouldn't accept the highest bidding tx
> for the good of the network. This conflict is particularly sharp
> in the case where the replacement tx would be immediately minable,
> which this proposal addresses.
>
> Unfortunately I haven't found time to code this up in bitcoin, but if
> there's positive response I can try.
>
> Thanks for reading!
> Rusty.
>
>
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

2023-06-07 20:17:09

- reply

📅 Original date posted:2019-03-21
📝 Original message:I'm not really sure the problem you're describing, but it sounds like something that affects normal bitcoin transactions as well.
There's certainly some interesting about the idea of "pre-fragmenting" your wallet utxo so you can make (or in payjoin: receive) payments with better privacy aspects.However, it's pretty unlikely to be practical for normal users, as it'll generally result in pretty big and cost-ineffective transactions.
In general though, there's like a 1000 different things you can do with coin selection, utxo management (and payjoin contributed input selection) but more often than not you are just making just making 1 trade off for another and good solutions will be wildly different depending on how you use your wallet.
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, March 18, 2019 3:55 AM, Kenshiro \[\] via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi,
>
> I think Payjoin can be a very good privacy solution for Bitcoin, but I have a question about it:
>
> - If a user has 1 BTC in a single address and make a payjoin payment to other person of 0.1 BTC using that address as input, the other person can see in a blockchain explorer the change address with an amount of 0.9 BTC. That's a serious privacy leak. I would like to know what will be the standard solution to this issue. An easy fix could be that the user wallet check if any address contains a BTC amount higher than a "safe" amount like 0.01 BTC or less. If some address exceed that amount the wallet could automatically make 1 payment to itself to split the amount in several addresses. In this way nobody receiving a payment from a user will ever know that he has a bitcoin balance higher than the "safe" amount.
>
> What do you think?
>
> Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190321/a8697048/attachment-0001.html>

2023-06-07 20:16:17

- reply

📅 Original date posted:2019-02-02
📝 Original message:More of a shower-thought than a BIP, but it's something I've long wish (hardware) wallets supported:
---
Abstract: Bitcoin Wallets generally ask us to trust their seed generation is both correct and honest. Especially for hardware and air gapped wallets, this is both a big ask and more or less impossible to practically verify. So we propose a bring-your-own-entropy approach in which the wallet can function completely deterministically. Our method is based on shuffling physical deck of cards. There are 52! (2^219.88) different shuffle order, which is a big enough space to be secure against collision and brute force attacks. Conveniently a shuffled deck of cards also can serve as a physical backup which is easy to hide in plain sight with great plausible deniability.
Representation:
Each card has a suit which can be represented by one of SCHD (spades, clubs, hearts, diamonds) and a value of one of 23456789TJQKA where the numbers are obvious and (T=ten, J=jack, Q=queen, K=king, A=ace) so "7 of clubs" would be represented by "7C" and a "Ten of Hearts" would be represented with "TH".
An deck of cards looks like:
2S,3S,4S,5S,6S,7S,8S,9S,TS,JS,QS,KS,AS,2C,3C,4C,5C,6C,7C,8C,9C,TC,JC,QC,KC,AC,2H,3H,4H,5H,6H,7H,8H,9H,TH,JH,QH,KH,AH,2D,3D,4D,5D,6D,7D,8D,9D,TD,JD,QD,KD,AD
And can be verified by making sure that every one of the 52 cards appears exactly once.
Step 1. Shuffle your deck of cards
This is a lot harder than you'd imagine, so do it quite a few times, with quite a few different techniques. It is advised to do at *least* 7 good quality shuffles to achieve a true cryptographically secure shuffle. Do not look at the cards while shuffling (to avoid biasing) and don't be afraid to also shuffle them face down on the table. Err on the side over over-shuffling.
See also: https://en.wikipedia.org/wiki/Shuffling#Sufficient_number_of_shuffles
Step 2. Write out the order (comma separated)
And example shuffle is:
5C,7C,4C,AS,3C,KC,AD,QS,7S,2S,5H,4D,AC,9C,3H,6H,9D,4S,8D,TD,2H,7H,JD,QD,2D,JC,KH,9S,9H,4H,6C,7D,3D,6S,2C,AH,QC,TH,TC,JS,6D,8H,8C,JH,8S,KD,QH,5D,5S,KS,TS,3S
Step 3. Sha512 it to create a seed
In the example above you should get:
dc04e4c331b1bd347581d4361841335fe0b090d39dfe5e1c258c547255cd5cf1545e2387d8a7c4dc53e03cacca049a414a9269a2ac6954429955476c56038498
Step 4. Interpret it
e.g. For bip32 you would treat the first 32 bytes as the private key, and the second 32 bytes as as the extension code.
-Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190202/5489fb15/attachment.html>

2023-06-07 20:16:05

- reply

📅 Original date posted:2019-01-27
📝 Original message:> Why does the template transaction need to be signed in step one and passed back and forth so many times? What is wrong with:
It isn't passed "back and forth so many times". It works exactly as you proposed, with the only difference is in "Step 1" the sender uses a *signed* transaction instead of an unsigned one. This is an important anti-DoS/anti-spy tactic, as it proves the sender actually owns those inputs and if the protocol is not followed to completion, the transaction can be dumped on the network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190127/785cba15/attachment.html>

2023-06-07 20:16:05

- reply

📅 Original date posted:2019-01-30
📝 Original message:On Tuesday, January 29, 2019 6:06 PM, James MacWhyte <keatonatron at gmail.com> wrote:
> I'm not convinced this is a valid concern, at least not valid enough to add extra complications to the process.
Signing a transaction is something a wallet needs to be able to do anyway AND at the final-step. And actually a signed transaction is _simpler_ because it's more standard and way format to send and validate.
> The sender could still refuse to sign the final transaction after they see the recipient's in-/outputs; "show me yours and I'll show you mine" isn't much of a spy deterrent, and nothing here prevents a DOS attack.
If the sender refuses to sign the final transaction, the receiver just propagates the template transaction which pays the receiver! So it's a pretty weak attack.
The only real attack is that the sender could double-spend the template-transaction before it's propagated, but the cost of doing this isn't free, as at the very least you need to pay the transaction fees of creating a double spend. It's not an amazingly good defence, but it's good enough that it's unlikely to get abused (and an attacker would only learn a single utxo of the receiver) .
> As an implementor, I would suggest keeping the protocol as simple as possible. By dropping the signing in the first step, the recipient doesn't need to maintain the ability to lookup and verify unspent outputs.
Being able to verify a transaction tends to be pretty simple in practice. (e.g. `testmempoolaccept` in bitcoin core's wallet) but if it's really hard for a receiver to do, it can easily just not do it... (and assume the template transaction is valid even if it's not).
But I suspect this actually complicates the job for the receiver, because now you have to deal with transaction malleability as they can now give you an invalid transaction, you sign it and then they malleate into a valid transaction with a different txid. So if you're tracking the transaction by txid, you'll get really confused...).
> It also would enforce the increased privacy, which the sender obviously wants if they are going down this path
I guess that's a valid concern. A sender might want to make a payment, but *only* if it can be done via a bustapay, while the current spec doesn't support that.
But there's no way that justifies removing the protection for receivers. Without some _basic_ protection, every company that takes bustapayments will just get constantly attacked by a simple costless `wget` that leaks their wallet utxos...
The only viable way I can see, would be the sender pays the first part of his invoice in lightning. And then pays the rest with a bustapay. Now the anti-spy thing is the fact the first part of the invoice was already paid.
But with so many moving parts, no one is ever going to implement that :P
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190130/dbe00339/attachment.html>

2023-06-07 20:16:04

- reply

📅 Original date posted:2019-01-27
📝 Original message:> Is there a missing word. "by giving a person.."? Not actually sure what
> you're getting at here but I suspect it's again tangential to this BIP
> discussion.
Correct on both points. I meant to say "giving a (txid:vout:privkey)" to a person as means of payment.
> So I think the limiting factor is in fact creating a standard that a reasonable number
> of people could agree with (and I like operational definitions, so
> subjective as it is, I like the goal of "good/clear enough that it could
> be incorporated into something like BtcPayServer")
The problem with BtcPayServer (and a lot of similar software), is that it's not very unsuitable for any sender/receiver coinjoin due to it not having its own wallet. As I understand the basic architecture is just a fancy wrapper around bip32 address generation and monitoring the payment to those addresses. This means that adding support is not only a large code change, but it also entails a substantial change for merchants (you can't just have your payments flow into your trezor, but need to run a hot wallet)
But I strongly believe that bustapay is good enough _right now_ for BtcPayServer integration (which I'd happily contribute myself, if it wasn't for my unfamiliarity of the project and c#)
> But this relates back to my first "generic" point that you haven't
> addressed here - protocol versioning and the possibility of more than
> one option. Perhaps more realistic (debatable): have the current version
> be non-PSBT but with a plan to have a version bump with PSBT in future.
> Stuff like that. It seems crazy to actually long term reject it.
Adding backwards-compatible versioning at a later stage would be pretty trivial through either the URL or HTTP header (e.g version=2) and if breaking backwards compatible is desirable it can also easily been done (e.g. bump the bip21 or send an incompatible request/response). I don't see this as a problem at all, and I'm not rejecting it long-term, I just don't think it's particularly helpful to bikeshed now, when adoption is pretty much zero.
> I don't want to be that guy, but this was a central part of the proposal
> that came of the meetup last summer and is in Haywood's blogpost. I mean
> if you came up it with separately, then cool :) But I was there, that
> was established immediately as the right way of doing this to avoid a
> trivial attack.
Oh wow. had no idea. I saw the part about the receiver spamming the sender with a bunch of transactions, where only 1 of them are real and thought "ewww" and "came up" with the idea of a "template transaction" instead. I was always wondering why no one came up it, but now it makes sense. The transaction-spam stage was just an _additional_ layer of protection.
Ok, now I feel like an idiot =) Thanks for letting me know.
> The counterargument is Laurent's statistics which I previously linked,
> suggesting that maybe 30% of txs violate this anyway, today. I'm not
> sure about that, will need more analysis; Core's SRD algo may be one
> reason, but anyway ... better to make things look like payments.
I think it's interesting, but I don't think it particularly matters. Avoiding UIH1 I think is pretty much irrelevant, as it'll likely just confuse any analysis into thinking the payment is the reverse of what it actually is. And wallets already don't care about violating UIH2(as a way to do implicit consolidation). If 30% of tx's are violating it, you can be pretty sure it means the _vast majority_ of wallets run coin selection in such a way that can violation UIH2. Most wallets use a coin selection algorithm that you can approximate with:
while !enoughMoney {
inputs += getAnotherInput();
}
and don't run a final pass that would prune superfluous inputs. Even coinsayer (shill alert) which I believe runs the most advanced coin selection algorithms, will routinely and intentionally violate UIH2 when it's ideal (e.g. most classic case: when `consolidationFeeRate >= minFeeRate`).
I'm not trying to dismiss your analysis, as I find it interesting. I'm just against increasing the cognitive burden on implementations by mentioning all this stuff, when the truth is it (barely) matters. If wallets routinely avoided UIH2 and making a UIH2 payment would "out" the transaction as much more likely to be a bustapay, then I'd definitely reconsider and provide a basic suggestion into how to try avoid it.
And like I said, I also think there's much more important things that go into "picking a contributed input" than just this.
> A last point, you also don't see value in being more explicit about
> simple things like transaction version and locktime? Even if you think
> these things should not be controlled, e.g. the protocol should allow
> either transaction version, then it'd be better to explicitly say so.
My intention was that wallets create a transaction exactly like they normally would do, and use that as the template transaction. The only time I wanted to be prescriptive was when it would increase the implementation complexity (e.g. being non-segwit compatible is a pain in the ass. So I'd rather just be "pure segwit only" transactions). But something like locktime makes no difference as long as the transaction is mempool eligible, so I'd rather just wallets do what they do anyway.
Although I think there should be a separate discussion on improving the uniformity of bitcoin transactions in general. The current state of affairs is really atrocious.
---
P.S. I know I come across as obstinate, but it's not really so. If you can come up with an alternative to bustapay with some traction, I'd love to get behind it and deprecate bustapay in favor of it. I just am pretty happy with the state of bustapay and it's status a sort of "MVP of pay2endpoint", and unless the argument is in the form: "We'd love to support it, but in order to do so we'd need X" I'm probably going to disagree.

2023-06-07 20:16:02

- reply

📅 Original date posted:2019-01-27
📝 Original message:Thanks Adam,
I have fixed the mistakes you have pointed out: https://github.com/bitcoin/bips/pull/754
Thanks for the detailed look!
> but its virtue of steganographic hiding means only minimal uptake
> is still enormously interesting and worth pursuing; that's my current feeling.
I very much agree =) I really think anything that (silently) breaks the assumption of common ownership of transaction inputs offers outsized benefits for the whole ecosystem.
One other idea I have is (way) better support for moving utxo's between wallets. The least controversial use case is moving funds between wallets you own. Like I might want to move *specific* utxo's from/to my joinmarket wallet, but not create a (privacy losing / expensive) transaction. Both core and joinmarket fail at this at a practical point of view.
Like imho it'd be pretty cool having a standardized format for (txid:vout:privatekey) with wallets showing it as "External UTXO" and preferentially spending it (and wallet not automatically importing any other utxo from that address).
Taken a bit further (this is the part which everyone hates) you could send someone money (or withdraw it from a service) by giving a person. It's not generally useful (for obvious reasons), but there's a lot of cases I think it's super cool.
---
Getting back on topic, without trying to do a point-by-point reply, I agree with pretty much everything you said but I am reluctant to make any changes.
I don't meant to be obtuse or anything, but I strongly believe the limiting factor to adoption to all these protocols is actually just getting people to implement it. I made multiple implementations of bustapay from both the sending/receiving end, so I could try develop the easiest to implement system that is still practical.
For instance I like PSBT and it's nice in theory. I actually had an original implementation using it, which is how I found some bugs in the core and golang version of PSBT). But in practice it's hugely overkill and significantly increases the implementation complexity complexity and is poorly supported. Switching to just a raw transaction actually made everything easier. (And that's not to criticise PSBT, I would definitely want to use it in other contexts).
Anyway, a big motivation for me even writing it as a BIP was to formalize my little anti-DOS trick of privately creating a "template transaction" which can just be dumped on the network as punishment. So if nothing else, hopefully I'll have demonstrated it's a pretty practical way of doing things.
--
Also your analysis on "Unnecessary Input Heuristic" is pretty cool, but I also don't like telling people to "avoid the UIH2" without providing the actual algo they should use. But really I think it's better off in a sort of article "how to pick contributed inputs" or something, as while it's nice it's not a huge deal and there's a lot of debatable tradeoffs that can/should be used. For instance the implementation I wrote for bustabit.com currently just heavily biases tainted inputs (e.g. ones associated with address reuse).
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, January 25, 2019 6:47 AM, Adam Gibson via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Ryan and list,
> I want to add some commentary to this (BIP79) to see if we can get
> further in standardizing this idea.
>
> When I first mulled it over I thought it too impractical, but its virtue
> of steganographic hiding means only minimal uptake is still enormously
> interesting and worth pursuing; that's my current feeling. I've offered
> more detailed thoughts in my blog post[1] (def not required reading here).
>
> Both Joinmarket and Samourai have started implementing this kind of
> transaction. And while that's interesting experimentally, some kind of
> cross-wallet standard would be helpful, albeit there some differences
> between that and the merchant/centralized service use-case.
>
> We might imagine as a concrete goal for this BIP to create something
> that would be acceptable for inclusion into a project like BTCPayServer,
> so that it could be used in a realistic use case by smaller bitcoin
> accepting merchants.
>
> Comments to the BIP[2] as follows, with generic comments first, and then
> specific comments for existing points in the BIP:
>
> [1] https://joinmarket.me/blog/blog/payjoin
> [2] https://github.com/bitcoin/bips/blob/master/bip-0079.mediawiki
>
> Generic comments
>
> ==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
>
> - Protocol versioning. Since inevitably (even if only merchants), this
> must be implemented by multiple wallets to be useful, the communication
> protocol will need versioning (for example i have in my
> simple/experimental Joinmarket PayJoin that sender sends min and max
> supported version and receiver responds with a chosen protocol version
> so we can update). I do understand that as a client-server model can
> apply here, we can ditch a lot of the complexities around network/p2p
> interaction, but this much at least seems necessary.
>
> - Although it has its logic, I don't think "Bustapay" is a good name for
> this protocol. I prefer "PayJoin" which is neutral sounding and
> self-descriptive. Needless to say this is not a hill I intend to die on.
>
> - PSBT/BIP174. I realise this has already been discussed, but this is a
> good example of what this standardisation was designed for, so I'd be
> against not including it, even given the reality that, as you correctly
> observe, it is not yet implemented in the majority of wallets and
> libraries. One way round that is to make it optional (possibly combined
> with above point about versioning). Note that for example you were
> observing the necessity to check the sequence number was unchanged; that
> would be encapsulated by checking equality of PSBT Input objects/fields.
> While one can make such software architecture arguments, the really
> fundamental point is the need for standards for x-wallet compatibility.
>
> - Version, Locktime: Perhaps this is not needed; in a peer to peer
> wallet scenario I think there might be logic in trying to get cover
> traffic of (Core, Electrum, others), say, by using
> last-block-locktime-mostly, as they do. Version should be 2 and sequence
> is a function of your suggestion to use BIP125. Worth noting that BIP125
> is not currently widely used on the network, though (see
> https://p2sh.info/dashboard/db/replace-by-fee?orgId=1). For this reason
> it should perhaps be explicitly only optional.
>
> - Avoidance of non-payment "Unnecessary Input Heuristic" (1, 2). For
> reference, see the definition here
> https://gist.github.com/AdamISZ/4551b947789d3216bacfcb7af25e029e#gistcomment-2796539
> and some data here
> https://gist.github.com/AdamISZ/4551b947789d3216bacfcb7af25e029e#gistcomment-2800791
> (whole comment thread may be of interest) - note this UIH name is afaik
> Chris Belcher's invention, it seems useful as a categorisation.
> So, it seems that UIH2 is more important to avoid; while some more
> sophisticated wallet coin selection algorithms may occasionally pick
> an input set where one input is larger than any output, most won't, and
> some in particular never will. So I think the text here should indicate
> that *the receiver's contributed input(s) SHOULD be chosen to avoid
> triggering the UIH2 heuristic where possible, so that the final payjoin
> transaction is maximally plausible as an ordinary payment" or similar.
> UIH1 is a nice-to-have (meaning the plausibility extends to two
> different (both wrong) payment amounts, but it may not be necessary to
> mention it in the BIP.
>
> Specific comments
> =================
>
>
> > > ====Step 4. Receiver validates, re-signs, and propagates on the
>
> bitcoin network====
>
> I believe this should say "Sender" not Receiver. Also for the next
> sentence, s/receiver/sender/:
>
> > > The receiver MUST validate the ''partial transaction'' was changed
>
> correctly and non-maliciously (to allow using potentially untrusted
> communication channels), re-sign its original inputs and propagate the
> final transaction over the bitcoin network.
>
> Your very correct highlighting of the attack vector of "receiver gives
> sender other inputs belonging to sender to unwittingly sign (described
> below), should be highlighted here, perhaps with the phrase "re-sign its
> ORIGINAL inputs" (only!)".
>
> > > When the sender is creating a "template transaction" it is done
>
> almost identically to creating a normal send, with the exception that
> only segwit inputs may be used. The sender is also encouraged to use a
> slightly more aggressive feerate than usual as well as BIP125 (Opt-in
> Full Replace-by-Fee Signaling), but neither is strictly required.
>
> "slightly more aggressive feerate than usual" - this I understand is to
> make up for receiver contributed utxo, OK.
>
> "only segwit inputs" - it certainly makes things simpler. One can work
> with non-segwit inputs but especially considering (as mentioned below)
> we really ought to "MUST" the part about matching input types, I tend to
> agree that non-segwit should be disallowed.
>
> > > The receiver must add at least one input to the transaction (the
>
> "contributed inputs"). If the receiver has no inputs, it should use a
> 500 internal server error, so the client can send the transaction as per
> normal (or try again later).
>
> Would it not be much simpler for the server to return a different
> (non-error) response indicating that it will broadcast the template tx
> in this case?
>
> > > Its generally advised to only add a single contributed input, however
>
> they are circumstances where adding more than a single input can be useful.
>
> I don't see a good reason to advise the use of only 1 input? (but this
> will also connect with the above generic comment about "UIH"). I guess
> it's because of your approach to fees. I'd prefer not to create a
> limitation here.
>
> > > To prevent an attack where a receiver is continually sent variations
>
> of the same transaction to enumerate the receivers utxo set, it is
> essential that the receiver always returns the same contributed inputs
> when it's seen the same inputs.
>
> This is an approach to avoiding this problem which has the virtue of
> simplicity, but it seems a little problematic. (1) You must keep a
> mapping of proposed payment utxos to one's proposed contributed input
> utxos, but (2) how should this be updated if you need to spend the
> contribution mentioned in (1)? Ironically use of payjoin exacerbates
> this issue, because it results in a smaller number of utxos being held
> by the receiver at any one time :) All this considered, I still see the
> value in your approach, but it might end up with a re-attempted payment
> being rejected. Certainly the more complex suggested solutions coming
> out of the summer 2018 coinjoin workshop aren't as practical as this,
> and may be overkill for small merchants/receivers.
>
> > > It is strongly preferable that the receiver makes an effort to pick a
>
> contributed input of the same type as the other transaction inputs if
> possible.
>
> I have also thought about this and you could reasonably argue this
> should be a MUST section in the BIP, that is, if the receiver cannot use
> inputs of the same type, he should fall back to the template
> transaction. A mixed-input payjoin/coinjoin is essentially
> near-perfectly identifiable as such (there is almost zero other usage of
> multi-type-input transactions), which is a very different thing than a
> non-identifiable payjoin transaction. That may or may not be OK to the
> sender. This is debatable though, for sure.
>
> > > After adding inputs to the transaction, the receiver generally will
>
> want to adjust the output that pays himself by increasing it by the sum
> of the contributed input amounts (minus any fees he wants to
> contribute). However the only strict requirement is that the receiver
> must never add or remove inputs, and must not ever decrease any
> output amount.
>
> "must never add or remove inputs" - did you mean "must never remove
> inputs"? he surely has to add one! Or, perhaps you mean he must not
> alter the list of inputs provided by the sender (in which case it should
> be clarified).
>
> "must not decrease any output amount" - I initally disagreed with this
> but it is a better option than the one I currently chose in Joinmarket
> payjoin (sender pays all fee as long as receiver utxos are not too
> much). So this means that the receiver either consciously chooses to not
> increase the fee, meaning the fee rate may be a bit low (hence your
> earlier comment about being generous, got it), or contributes via the
> payout amount. I guess the latter might break merchant software
> expecting to have amount output fixed and fees determined by change.
>
> Regards,
> Adam Gibson/waxwing
>
> On 30. 08. 18 22:24, Ryan Havar via bitcoin-dev wrote:
>
> > I've just finished writing an implementing of this, and extremely happy
> > with how it turned out. So I'd like to go and try go down the path of
> > more formally describing it and getting some comments and ultimately
> > encourage its wide-spread use.
> > ==Abstract==
> > The way bitcoin transactions are overwhelming used is known to leak more
> > information than desirable. This has lead to fungibility concerns in bitcoin
> > and a raise of unreasonably effective blockchain analysis.
> > Bustapay proposes a simple, practical way to bust these assumptions to
> > immediate
> > benefit of the sender and recievers. Furthermore it does so in such a
> > way that
> > helps recievers avoid utxo bloat, a constant problem for bitcoin merchants.
> > ==Copyright==
> > This BIP is in the public domain.
> > ==Motivation==
> > One of the most powerful heuristic's employed by those whose goal is to
> > undermine
> > bitcoin's fungiblity has been to assume all inputs of a transaction are
> > signed by
> > a single party. In the few cases this assumption does not hold, it is
> > generally
> > readibly recognizable (e.g. traditional coinjoins have a very obvious
> > structure,
> > or multisig outputs are most frequently validated onchain).
> > Bustapay requires no changes to bitcoin and creates bitcoin transactions
> > that are
> > indistinguishable from normal ones.
> > It is worth noting that this specification has been intentionally kept
> > as simple
> > as possible to encourage adoption. There are almost an endless amount of
> > extensions
> > possible but the harder the implementation of clients/server the less
> > likely it
> > will ever be done. Should bustapay enjoy widespread adoption, a "v2"
> > specification
> > will be created with desired extensions.
> > ==Specification==
> > A bustapay payment is made from a sender to a receiver.
> > Step 1. Sender creates a bitcoin transaction paying the receiver
> > This transaction must be fully valid, signed and all inputs must use
> > segwit. This transaction is known as the "template transaction". This
> > transaction must not be propagated on the bitcoin network.
> > Step 2. Sender gives the "template transaction" to the receiver
> > This would generally be done as an HTTP POST. The exact URL to submit it
> > to could be specified with a bip21 encoded address. Such as
> > bitcoin:2NABbUr9yeRCp1oUCtVmgJF8HGRCo3ifpTT?bustapay=https://bp.bustabit.com/submit
> > and the HTTP body should be the raw transaction hex encoded as text.
> > Step 3. Receiver processes the transaction and returns a partially
> > signed coinjoin
> > The receiver validates the transaction is valid, pays himself and is
> > eligible for propation. The receiver then adds one of his own inputs
> > (known as the "contributed input") and increase the output that pays
> > himself by the contributed input amount. Doing so will invalidate the
> > "template transaction"'s original input signatures, so the sender needs
> > to return this "partial transaction" back to the receiver to sign. This
> > is returned as a hex-encoded raw transaction a response to the original
> > HTTP POST request.
> > Step 4. Receiver validates, re-signs, and propagates on the bitcoin network
> > The receiver is responsible in making sure the "partial transaction"
> > returned by the sender was changed correctly (it should assume the
> > connection has been MITM'd and act accordingly), resign its original
> > inputs and propagates this transaction over the bitcoin network. The
> > client must be aware that the server can reorder inputs and outputs.
> > Step 5. Receiver observes the finalized transaction on the bitcoin network
> > Once the receiver has seen the finalized transactions on the network
> > (and has enough confirmations) it can process it like a normal payment
> > for the sent amount (as opposed to the amount that it looks like on the
> > network). If the receiver does not see the finalized transaction after a
> > timeout will propagate the original "template transaction" to ensure the
> > payment happens and function a strong anti-DoS mechanism.
> > === Implementation Notes ===
> > For anyone wanting to implement bustapay payments, here are some notes
> > for receivers:
> >
> > - A transaction can easily be checked if it's suitable for the mempool
> > with testmempoolaccept in bitcoin core 0.17
> >
> > - Tracking transactions by txid is precarious. To keep your sanity make
> > sure all inputs are segwit. But remember segwit does not prevent txid
> > malleability unless you validate the transaction. So really make sure
> > you're using testmempoolaccept at the very least
> >
> > - Bustapay could be abused by a malicious party to query if you own a
> > deposit address or not. So never accept a bustapay transaction that pays
> > an already used deposit address
> >
> > - You will need to keep a mapping of which utxos people have showed you
> > and which you revealed. So if you see them again, you can reveal the
> > same one of your own
> >
> > - Check if the transaction was already sorted according to BIP69, if so
> > ensure the result stays that way. Otherwise probably just shuffle the
> > inputs/outpus
> >
> >
> > Notes for sending applications:
> >
> > - The HTTP response must not be trusted. It should be fully validated
> > that no unexpected changes have been made to the transaction.
> >
> > - The sender should be aware the original "template transaction" may be
> > propagated at any time, and in fact can intentionally be
> > done so for the purpose of RBF as it should have a slightly higher fee
> > rate.
> >
> >
> > == Credits ==
> > The idea is obviously based upon Dr. Maxwell's seminal CoinJoin
> > proposal, and reduced scope inspired by a simplification of the "pay 2
> > endpoint" (now offline) blog post by blockstream.
> > -Ryan
> >
> > bitcoin-dev mailing list
> > bitcoin-dev at lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

2023-06-07 20:15:04

- reply

📅 Original date posted:2018-10-24
📝 Original message:Actually, I think it can be calculated a bit smarter using maths (which unfortunately I'm not very good at...). But I assume it's something like:
```
falsePositiveChances := 0.0
foreach( transaction of transactions) {
falsePositiveChances += (1 / factorial(transaction.Inputs)) * (1 / factorial(transaction.Ouputs))
}
totalFalsePositives := falsePositiveChances / transactions.length
```
If so, I get 42.4% false positive rate. So clearly bip69 is getting used a fair bit, but not nearly as much as randomization.
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, October 24, 2018 10:52 AM, <rhavar at protonmail.com> wrote:
> That's pretty easy to quantify. I wrote a quick script to grab the last few blocks, and then shuffle the inputs/outputs before testing if each transaction is bip69 or not.
>
> The result was 42% of all transactions would accidentally be bip69 when randomized.
>
> So clearly randomization is a lot more popular than bip69 at the moment, but I'm not sure that it matters much. As soon as you have more than a few inputs/outputs, you can tell with a high confidence if the transaction is bip69 or not.
>
> And of course if you're clustering a wallet, you can figure out extremely easily how that wallet behaves wrt bip6.
>
> -Ryan
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Wednesday, October 24, 2018 9:12 AM, Gregory Maxwell via bitcoin-dev bitcoin-dev at lists.linuxfoundation.org wrote:
>
> > On Wed, Oct 24, 2018 at 3:52 PM Chris Belcher via bitcoin-dev
> > bitcoin-dev at lists.linuxfoundation.org wrote:
> >
> > > Thanks for bringing our attention to this important topic.
> > > According to (https://p2sh.info/dashboard/db/bip-69-stats) around 60% of
> > > transaction follow bip69 (possibly just by chance).
> >
> > A two input randomly ordered transaction has a 50% chance of
> > 'following' bip-69. So 60% sound like a small minority.

2023-06-07 20:15:04

- reply

📅 Original date posted:2018-10-24
📝 Original message:That's pretty easy to quantify. I wrote a quick script to grab the last few blocks, and then shuffle the inputs/outputs before testing if each transaction is bip69 or not.
The result was 42% of all transactions would accidentally be bip69 when randomized.
So clearly randomization is a lot more popular than bip69 at the moment, but I'm not sure that it matters much. As soon as you have more than a few inputs/outputs, you can tell with a high confidence if the transaction is bip69 or not.
And of course if you're clustering a wallet, you can figure out extremely easily how that wallet behaves wrt bip6.
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, October 24, 2018 9:12 AM, Gregory Maxwell via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> On Wed, Oct 24, 2018 at 3:52 PM Chris Belcher via bitcoin-dev
> bitcoin-dev at lists.linuxfoundation.org wrote:
>
> > Thanks for bringing our attention to this important topic.
> > According to (https://p2sh.info/dashboard/db/bip-69-stats) around 60% of
> > transaction follow bip69 (possibly just by chance).
>
> A two input randomly ordered transaction has a 50% chance of
> 'following' bip-69. So 60% sound like a small minority.

2023-06-07 20:15:03

- reply

📅 Original date posted:2018-10-21
📝 Original message:On Sunday, October 21, 2018 2:54 PM, Pavol Rusnak <stick at satoshilabs.com> wrote:
> Your solution in the second part of the email does not solve the problem you indicated in the first part of your email.
Sorry, I'm not quite sure what parts you are referring to. I assume you might mean my first paragraph, so I'll try explain myself a bit clearer how this makes it harder to find wallet boundaries.
Right now you can generally tell if a transaction is using bip69 or not (as long as you account for the probability that it's randomly sorted to accidentally be bip69). And generally wallets are consistent if they use bip69 or not.
This can often make it massively easier to detect what is change and not. Let's say I'm clustering a wallet and know they're using a wallet that always uses bip69, and I'm looking at a transaction in that cluster and trying to guess which is the change and which is not. There's a lot of things you can use to assign a probability. The most obvious thing is looking at the amount of significant-digits of the output amounts (if they vary a lot, change tends to be the one with more), but a much more powerful one is looking at how the outputs are spent (and if they end up spend-linking back into the original cluster).
So let's say that the transaction output is spent by a non-bip69 transaction -- I right away know that it's going to (almost certainly) be a different wallet (e.g. the destination).
My (shower-thoughty) "solution" fixes this problem, because an outside observer has no way of knowing if a transaction is using deterministic sorting or not, so can not use this information to establish wallet boundaries.
--
On somewhat of a tangent I was actually fortunate enough to have someone with access to the biggest(?) bitcoin analysis service help me with a few experiments. While I was genuinely taken aback by how accurate some of their analysis can be, I also found it pretty easy to trick -- implying it relies heavily on some fragile heuristics.
I don't like to be alarmist, but I worry a lot about the fungibility of bitcoin when we have such effective blockchain analysis and a *LOT* of the ecosystem using a centralized analytics service. And in fact, we're already starting to see some minor effects of this (e.g. people already know that if they gamble their funds, they'll probably have trouble using an exchange later). And I don't think we're too far from the point where any "unidentified" bitcoin is instantly flagged as "suspicious" (and for instance, requires more explaining for by exchanges) potentially seriously harming bitcoin fungibility and it's value determined also by it's history.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20181022/6473e67a/attachment.html>

2023-06-07 20:15:03

- reply

📅 Original date posted:2018-10-21
📝 Original message:Right now it's just *way* too easy to spot the boundaries between different wallets. There's a lot of things that contribute to that, but the one that concerns me the most is the way wallets sort transaction inputs and outputs.
Some wallets and protocols (especially HW wallets) have a strong preference for deterministic sorting (i.e. using bip69), while other wallets have a lot of objections to this.
I'm not sure I fully understand the objections, but I think they can be summarized as "during the transition period there will be a lot of privacy loss" and "if in the future someone wants to use bitcoin in a way that's not compatible with bip69 their transactions will stick out heavily".
I wonder if this impasse could be solved with deterministic sorting, but based on a semi-secret. Like `sortingSecret = hmac(walletSeed, "sortingSecret")` and then there's a standardized sort order based on the sortingSecret. e.g. sort inputs/output by the `hash(data || sortingSecret)`. Wallets could come up with their own way of computing (or storing) the "sortingSecret" but from there it's standardized.
I has the advantages of deterministic sorting (as long as you know the sortingSecret) you can verify it's done correctly and externally looks totally randomized.
Am I missing something, or could this be the way forward?
-Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20181021/0c512d27/attachment.html>

2023-06-07 20:14:41

- reply

📅 Original date posted:2018-09-10
📝 Original message:Thanks Sjors,
It's nice to get hear feedback like that, it'll help make me clear some stuff up. Right now I've been working on getting a reference client for both sending and receiving, while juggling some stuff in real life.
But here is the evolving document:
https://github.com/RHavar/bips/blob/master/bip-bustapay.mediawiki
> Breaking this heuristic at scale would be good, but do we know to what degree it’s already broken? Is there any empirical research measuring its accuracy and false positive rate?
Not that I'm aware of. However from what I've seen of chainalysis, it's eerily accurate. I've seen it cluster wallets that it really shouldn't have been able to do. (address-reuse is actually the biggest culprit here, we really need to working on reducing stopping this with stuff like bip32 based address books, wallet warnings on reused addresses, etc.). As for false positives of block-chain analysis, under normal circumstances it probably shouldn't suffer any. Although it was pretty easy to confuse it with specially crafted coinjoin transactions, although a few months later seemed to realize and decluster the wallets again.
> In addition to mixers, custodial wallets and exchanges also contribute to breaking this heuristic
Well, kind of I guess. From a block-chain analysis point of view you'd be trying to figure out the transaction was to/from coinbase, rather than "coinbase on behalf of client XYZ". I guess the former is better than the later, but I think it'd be preferable if no one knew all together.
> Using PSBT?
I've got this feedback a lot, and I think it's definitely the way to go in the future. But at the moment compatibility and libraries are just not there that I'd like to stick with raw transactions for now.
> The use of HTTPS kind of forces sender and recipient to use a 3rd party service, even though this could done bilaterally.
I don't really understand this point. I just used https as an example, as I suspect that's what every merchant will want to use. I don't really mean to specify the protocol though, if the client/server support it (e.g. onion) then it should be fine.
> I assume the transaction in step (1) is some sort of collateral to insure they’re not just trying to extract private information from you?
It's not really designed as a sort of collateral, but as a way to prevent someone costlessly learning about one of your inputs. The cost is that of spending a utxo. An attacker can always trivially double spend the "template transaction" because from the network's point of view the "template transaction" will come after an alternate (say, non-rbf) spend. But in practice I don't think this is a problem, because it still imposes a cost on the attacker.
> In what ways could it be abused?
I just want to make it clear that senders can't blindly sign a transaction without verifying it. The most important thing to verify is the output amounts/addresses -- but I will make it more explicit.
Thanks once again for your feedback
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, 10 September 2018 05:30, Sjors Provoost <sjors at sprovoost.nl> wrote:
>> Op 30 aug. 2018, om 22:24 heeft Ryan Havar via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> het volgende geschreven:
>>
>> ==Motivation==
>>
>> One of the most powerful heuristic's employed by those whose goal is to undermine
>> bitcoin's fungiblity has been to assume all inputs of a transaction are signed by
>> a single party. In the few cases this assumption does not hold, it is generally
>> readibly recognizable (e.g. traditional coinjoins have a very obvious structure,
>> or multisig outputs are most frequently validated onchain).
>
> In addition to mixers, custodial wallets and exchanges also contribute to breaking this heuristic; even though there’s a single entity signing multiple inputs, that entity doesn’t represent a single owner of the funds. As with mixers, exchanges and custodial wallets can sometimes be spotted as well, but we don’t know what percentage is missed.
>
> Breaking this heuristic at scale would be good, but do we know to what degree it’s already broken? Is there any empirical research measuring its accuracy and false positive rate?
>
>> Should bustapay enjoy widespread adoption, a "v2" specification
>> will be created with desired extensions.
>
> I would not put future promises in a BIP. Rather, explain how extension might work.
>
>> ==Specification==
>>
>> A bustapay payment is made from a sender to a receiver.
>>
>> Step 1. Sender creates a bitcoin transaction paying the receiver
>>
>> This transaction must be fully valid, signed and all inputs must use segwit. This transaction is known as the "template transaction”.
>
> Using PSBT?
>
>> This transaction must not be propagated on the bitcoin network.
>
> This can’t be guaranteed, and even after step 5 a reorg could cause it to get confirmed. It’s useful to explain why this doesn’t matter.
>
>> Step 2. Sender gives the "template transaction" to the receiver
>>
>> This would generally be done as an HTTP POST.
>
>> The exact URL to submit it to could be specified with a bip21 encoded address. Such as bitcoin:2NABbUr9yeRCp1oUCtVmgJF8HGRCo3ifpTT?bustapay=https://bp.bustabit.com/submit and the HTTP body should be the raw transaction hex encoded as text.
>
> This seems too detailed. If you want to specify the message protocol, maybe that can have it’s own section where you list each of the messages, the URL, parameters and encoding. Then you can keep this overview section shorter.
>
> The use of HTTPS kind of forces sender and recipient to use a 3rd party service, even though this could done bilaterally. What if the payment request contained a (single-use) Onion URL an expiration date? The recipient would have to keep a hidden service up until the expiration date, though the sender could try again if there’s temporary reachability issue.
>
> Adding a (onion) URL to the the payment request also makes gradual adoption easier, because recipients don’t need to worry if senders support this protocol.
>
>> Step 3. Receiver processes the transaction and returns a partially signed coinjoin
>>
>> The receiver validates the transaction is valid, pays himself and is eligible for propation. The receiver then adds one of his own inputs (known as the "contributed input") and increase the output that pays himself by the contributed input amount. Doing so will invalidate the "template transaction"'s original input signatures, so the sender needs to return this "partial transaction" back to the receiver to sign. This is returned as a hex-encoded raw transaction a response to the original HTTP POST request.
>
>> * Bustapay could be abused by a malicious party to query if you own a deposit address or not. So never accept a bustapay transaction that pays an already used deposit address
>
> Indeed, once the recipient adds funds, they reveal more about themselves to the sender then they would otherwise. I think that needs more elaboration.
>
> I assume the transaction in step (1) is some sort of collateral to insure they’re not just trying to extract private information from you? However if fees are low they could still double-spend it after the recipient revealed their address, especially because the recipient has no way of RBF’ing the original (though CPFP could help). Perhaps require that the original transaction pays a fee based on the expected size of the final transaction?
>
>> Notes for sending applications:
>>
>> * The HTTP response must *not* be trusted. It should be fully validated that no unexpected changes have been made to the transaction.
>
> Not trusting anything is obvious. :-) It’s better to explicitly state what exactly needs to be verified (amounts, destinations, inputs, etc), and maybe list a few obvious shenanigans to watch out for.
>
> A more general concern is that the sender can’t know for sure the recipient really supports this protocol, so it should assume that whatever information it pings to some API could be used maliciously. In what ways could it be abused?
>
> Sjors
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180910/1a3909de/attachment-0001.html>

2023-06-07 20:14:25

- reply

📅 Original date posted:2018-09-01
📝 Original message:I think I mentioned it before, but seems semi-relevant to this thread so I'd like to throw my vote behind pretty tiny blocks on testnet (like max 50-100k weight) to try help simulate a fee-market like situation.
(Although lately there's been a lot of testnet spam and full blocks, which has really made testing easier. But I don't know how long this situation will last)
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On August 30, 2018 7:06 PM, Gregory Maxwell via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> On Thu, Aug 30, 2018 at 11:21 PM Johnson Lau via bitcoin-dev
> bitcoin-dev at lists.linuxfoundation.org wrote:
>
> > A public testnet is still useful so in articles people could make references to these transactions.
> > Maybe we could have 2 testnets at the same time, with one having a smaller block size?
>
> I would much rather have a signed blocks testnet, with a predictable
> structured reorg pattern* (and a config flag so you can make your node
> ignore all blocks that are going to get reorged out in a reorg of nth
> or larger). There are many applications where the mined testnet just
> doesn't give you anything useful... it's too stable when you want it
> to be a bit unstable and too wildly unstable when you want a bit of
> stability-- e.g. there are very few test cases where a 20,000 block
> reorg does anything useful for you; yet they happen on testnet.
>
> We looked at doing this previously in Bitcoin core and jtimon had some
> patches, but the existing approach increased the size of the
> blockindex objects in memory while not in signed testnet mode. This
> could probably have been fixed by turning one of the fields like the
> merkel root into a union of it's normal value and a pointer a
> look-aside block index that is used only in signed block testnet mode.
>
> Obviously such a mode wouldn't be a replacement for an ordinary
> testnet, but it would be a useful middle ground between regtest (that
> never sees anything remotely surprising and can't easily be used for
> collaborative testing) and full on testnet where your attempts to test
> against ordinary noise require you cope your entirely universe being
> removed from existence and replaced by something almost but not quite
> entirely different at the whim of some cthulhuian blind idiot god.
>
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

2023-06-07 20:14:17

- reply

📅 Original date posted:2018-08-30
📝 Original message:I've just finished writing an implementing of this, and extremely happy with how it turned out. So I'd like to go and try go down the path of more formally describing it and getting some comments and ultimately encourage its wide-spread use.
==Abstract==
The way bitcoin transactions are overwhelming used is known to leak more
information than desirable. This has lead to fungibility concerns in bitcoin
and a raise of unreasonably effective blockchain analysis.
Bustapay proposes a simple, practical way to bust these assumptions to immediate
benefit of the sender and recievers. Furthermore it does so in such a way that
helps recievers avoid utxo bloat, a constant problem for bitcoin merchants.
==Copyright==
This BIP is in the public domain.
==Motivation==
One of the most powerful heuristic's employed by those whose goal is to undermine
bitcoin's fungiblity has been to assume all inputs of a transaction are signed by
a single party. In the few cases this assumption does not hold, it is generally
readibly recognizable (e.g. traditional coinjoins have a very obvious structure,
or multisig outputs are most frequently validated onchain).
Bustapay requires no changes to bitcoin and creates bitcoin transactions that are
indistinguishable from normal ones.
It is worth noting that this specification has been intentionally kept as simple
as possible to encourage adoption. There are almost an endless amount of extensions
possible but the harder the implementation of clients/server the less likely it
will ever be done. Should bustapay enjoy widespread adoption, a "v2" specification
will be created with desired extensions.
==Specification==
A bustapay payment is made from a sender to a receiver.
Step 1. Sender creates a bitcoin transaction paying the receiver
This transaction must be fully valid, signed and all inputs must use segwit. This transaction is known as the "template transaction". This transaction must not be propagated on the bitcoin network.
Step 2. Sender gives the "template transaction" to the receiver
This would generally be done as an HTTP POST. The exact URL to submit it to could be specified with a bip21 encoded address. Such as bitcoin:2NABbUr9yeRCp1oUCtVmgJF8HGRCo3ifpTT?bustapay=https://bp.bustabit.com/submit and the HTTP body should be the raw transaction hex encoded as text.
Step 3. Receiver processes the transaction and returns a partially signed coinjoin
The receiver validates the transaction is valid, pays himself and is eligible for propation. The receiver then adds one of his own inputs (known as the "contributed input") and increase the output that pays himself by the contributed input amount. Doing so will invalidate the "template transaction"'s original input signatures, so the sender needs to return this "partial transaction" back to the receiver to sign. This is returned as a hex-encoded raw transaction a response to the original HTTP POST request.
Step 4. Receiver validates, re-signs, and propagates on the bitcoin network
The receiver is responsible in making sure the "partial transaction" returned by the sender was changed correctly (it should assume the connection has been MITM'd and act accordingly), resign its original inputs and propagates this transaction over the bitcoin network. The client must be aware that the server can reorder inputs and outputs.
Step 5. Receiver observes the finalized transaction on the bitcoin network
Once the receiver has seen the finalized transactions on the network (and has enough confirmations) it can process it like a normal payment for the sent amount (as opposed to the amount that it looks like on the network). If the receiver does not see the finalized transaction after a timeout will propagate the original "template transaction" to ensure the payment happens and function a strong anti-DoS mechanism.
=== Implementation Notes ===
For anyone wanting to implement bustapay payments, here are some notes for receivers:
* A transaction can easily be checked if it's suitable for the mempool with testmempoolaccept in bitcoin core 0.17
* Tracking transactions by txid is precarious. To keep your sanity make sure all inputs are segwit. But remember segwit does not prevent txid malleability unless you validate the transaction. So really make sure you're using testmempoolaccept at the very least
* Bustapay could be abused by a malicious party to query if you own a deposit address or not. So never accept a bustapay transaction that pays an already used deposit address
* You will need to keep a mapping of which utxos people have showed you and which you revealed. So if you see them again, you can reveal the same one of your own
* Check if the transaction was already sorted according to BIP69, if so ensure the result stays that way. Otherwise probably just shuffle the inputs/outpus
Notes for sending applications:
* The HTTP response must *not* be trusted. It should be fully validated that no unexpected changes have been made to the transaction.
* The sender should be aware the original "template transaction" may be propagated at any time, and in fact can intentionally be
done so for the purpose of RBF as it should have a slightly higher fee rate.
== Credits ==
The idea is obviously based upon Dr. Maxwell's seminal CoinJoin proposal, and reduced scope inspired by a simplification of the "pay 2 endpoint" (now offline) blog post by blockstream.
-Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180830/29c49730/attachment.html>

2023-06-07 20:14:08

- reply

📅 Original date posted:2018-08-20
📝 Original message:I think you have correctly identified dealing with small amounts while working in the "bitcoin" denomination as a major pain point, and I will go as far as agreeing that a standard color coding would help people eyeball what 0.000011 BTC means if they were used to seeing the same color pallet.
However, I don't think it's a good solution. I strongly suspect very few services or wallets are going to be willing to adopt such a thing as it'll make their design look ridiculous and it's cumbersome to type.
Furthermore, my experience in using BIP76 (Bits Denomination) in several services with people of varying levels of familiarity to be entirely positive, and in my strong believe a vastly superior solution that should be advocated and used more.
And BTW i think what would help the readability of the "bitcoin" denomination more than colors if it was standard to always write it to 8 decimal places (e.g. 0.00001100 BTC) which I think is a bit more useful in allowing people to eyeball the size.
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On August 18, 2018 1:10 PM, Martin Damgaard via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi bitcoin-dev at lists.linuxfoundation.org
>
> Here is my humble attempt to make a contribution to the impressive work that you all are doing.
>
> I am unfamiliar with the normal BIP procedures. I have therefore just tried to follow the example of BIP 176 by Jimmy Song, in order make something similar. I suggest a universal bitcoin value color scale, for tackling the same decimal problem, as identified by the BIP 176 proposal.
>
> I have attached the document in three different formats (*.rtf, *.pdf and *.docx) as I do not know your preferred format. I hope you will find my suggestion useful.
>
> Thank you and all the best
>
> Martin Damgaard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180820/3650a9b1/attachment.html>

2023-06-07 20:13:47

- reply

📅 Original date posted:2018-07-08
📝 Original message:More of a shower-thought, but I am currently working on a bitcoin wallet that is designed to handle "free pressure" properly (e.g. CPFP aware, transaction merging, automated progressive fee bumping, etc) and one kind of annoying thing is that there really is no fee market on testnet.
I was thinking it would be cool if tesnet blocks were a bit smaller, so they didn't just mine the entire mempool each block. A full-blown softfork on testnet seems overkill, but just changing the default mining maxBlockWeight to something like 50% of the current mempool weight would help test things and help wallet developers fixing things like "stuck transactions" and what not. (And of course some miners will not use the default, but it's not really a big deal..)
-Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180708/b0d63a30/attachment.html>

2023-06-07 20:11:23

- reply

📅 Original date posted:2018-03-24
📝 Original message:Maybe:
https://github.com/bitcoin/bitcoin/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22
Just pick something small (even if it's not interesting), struggle with it, struggle with it some more, do a git blame on the parts you need to modify and try contact the person if there's something you need help with.
I'd say start with simple and boring changes, and you'll organically get a better understanding.
But even better, go to:
https://github.com/bitcoin/bitcoin/pulls
And find some issues, reproduce the problem, test the fix -- and look at how the person did it. Post your results/feedback on the pull requests
I think you'll find in bitcoin (and cryptocurrencies in general) there's a lot more demand for elbow grease than advanced maths stuff =)
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On March 24, 2018 8:52 AM, Daniel R via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hey guys,
>
> I want to contribute to bitcoin core. I am an intermediate programmer and want to get started contributing fast. I have already cloned the git repository. Can you maybe direct me to sources where I can learn more about the structure of bitcoin core and specifically to problems where I can get experience working with the source code?
>
> I know Python, C/C++ and a bit of Java. I have advanced knowledge of cryptographic concepts and procedures and try to teach myself some of the math, especially around elliptic curves. I am currently composing a bachelor thesis around the question of Blockchain usage in the car-industry, where I try to look at different aspects of protocol-design.
>
> Best Regards
> Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180324/50b8c8b4/attachment.html>

2023-06-07 20:11:05

- reply

📅 Original date posted:2018-03-09
📝 Original message:> Still, re-reading your initital post, I'm convinced that the weakening of the
> DoS protections is probably not a huge problem, so maybe lets try this in a
> release and see what happens.
Awesome! I very much agree. The relaxation of some of these DoS prevention rules I think will really open up a lot of use cases and adoption
> Notably, if people actually use this new replacement behavior, the institutions
> doing these sweeps of unconfirmed outputs might stop doing that!
Agree, I'm pretty sure it's unintentional. I know a lot of services struggle with coin selection, so what they do is conceptually have a receive wallet from which they can sweep to their hot wallet (or cold storage) to keep their utxo manageable.
Currently some of them are sweeping unconfirmed inputs with it, but I don't think it's a conscious design choice, just something that happens to be working well now.
(FWIW I observed this behavior like 6+ months ago, I haven't kept track of if it's still happening or how often. But at the time I had to write off the idea of low-fee rbf batch transactions as it was happening too often to be feasible)
-Ryan
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On March 9, 2018 1:28 PM, Peter Todd via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>
> On Thu, Mar 08, 2018 at 03:07:43PM -0500, Russell O'Connor wrote:
>
> > On Thu, Mar 8, 2018 at 1:34 PM, Peter Todd pete at petertodd.org wrote:
> >
> > > But that's not a good argument: whether or not normal users are trying to
> > >
> > > attack each other has nothing to do with whether or not you're opening up
> > >
> > > an
> > >
> > > attack by relaxing anti-DoS protections.
> >
> > I'm not suggesting removing the anti-DoS protections. I'm suggesting that
> >
> > replaced transaction require a fee increase of at least the min-fee-rate
> >
> > times the size of all the transactions being ejected (in addition to the
> >
> > other proposed requirements).
>
> Fair: you're not removing them entirely, but you are weakening them compared to
>
> the status quo.
>
> > > Equally, how often are normal users who aren't attacking each other
> > >
> > > creating
> > >
> > > issues anyway? You can always have your wallet code just skip use of RBF
> >
> > replacements in the event that someone does spend an unconfirmed output that
> >
> > > you sent them; how often does this actually happen in practice?
> >
> > Just ask rhavar. It happens regularly.
> >
> > Not many wallets let you spend unconfirmed outputs that you didn't create.
> >
> > >
> >
> > The problem is with institutional wallets sweeping incoming payments. It
> >
> > seems that in practice they are happy to sweep unconfirmed outputs.
>
> Pity, that does sound like a problem. :(
>
> > Setting all of the above aside for a moment. We need to understand that
> >
> > rational miners are going to prefer to transactions with higher package fee
> >
> > rates regardless of whatever your personal preferred RBF policy is. If we
> >
> > do not bring the RBF policy to alignment with what is economically
> >
> > rational, then miners are going to change their own policies anyways,
> >
> > probably all in slightly different ways. It behooves everyone to develop a
> >
> > reasonable standard RBF policy, that is still robust against possible DoS
> >
> > vectors, and aligns with miner incentives, so that all participants know
> >
> > what behaviour they can reasonably expect. It is simply a bonus that this
> >
> > change in RBF policy also partially mitigates the problem of pinned
> >
> > transactions.
>
> Miners and full nodes have slightly different priorities here; it's not clear
>
> to me why it matters that they implement slightly different policies.
>
> Still, re-reading your initital post, I'm convinced that the weakening of the
>
> DoS protections is probably not a huge problem, so maybe lets try this in a
>
> release and see what happens.
>
> Notably, if people actually use this new replacement behavior, the institutions
>
> doing these sweeps of unconfirmed outputs might stop doing that! That's
>
> probably a good thing, as respends of potentially conflicted unconfirmed
>
> outputs can be dangerous in reorgs; we're better off if outputs are buried
>
> deeply before being spent again.
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> https://petertodd.org 'peter'\[:-1\]@petertodd.org
>
> bitcoin-dev mailing list
>
> bitcoin-dev at lists.linuxfoundation.org
>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

2023-06-07 20:10:54

- reply

📅 Original date posted:2018-02-18
📝 Original message:No, you are misunderstanding. The block size limit (1MB) has been replaced in favor of a block weight limit (4M weight). Bytes which must be sent to old clients are weighted at 4 units each which is what allows it to be a soft fork.
So as such, there's not two separate limits or anything.
P.S. what's up with your signature lol
-Ryan

-------- Original Message --------
On February 18, 2018 11:26 AM, CANNON via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> I have a question in reference to the increased blockspace enabled
> by the segregated witness upgrade. Is this extra blockspace beyond
> the legacy 1 MB limit limited to just witness data?
>
>
>
>Cannon
> PGP Fingerprint: 2BB5 15CD 66E7 4E28 45DC 6494 A5A2 2879 3F06 E832
> Email: cannon at cannon-ciota.info
>
> NOTICE: ALL EMAIL CORRESPONDENCE NOT SIGNED/ENCRYPTED WITH PGP SHOULD
> BE CONSIDERED POTENTIALLY FORGED, AND NOT PRIVATE.
> -----BEGIN PGP SIGNATURE-----
>
> iQIcBAEBCgAGBQJaiajLAAoJEAYDai9lH2mwDbMQAKgJseZG9oOoP9WJlESFdAzm
> jLM69KLF4RVWDQjMWCtyVayyolXMLavW6fL4GL7/ztLYSl9Pz+FDlH3Qgo1GLDx4
> fRUkmb0ApLBSAjmdqM+kWjgq3s6/oZaIkNxGeyK5SYY4QhF80Z42PWLBPKjAc7vn
> pe7Im13NwtyJiRuCmq4e1z6GmW86fFOKOsR4rlWcftHOFmH8Gz5d9KK9LbPqU6Ca
> +rde8yb+A9uXEv3O9MqeLId7FYSJixgzFGJosidVkAVQI7YZ0TAfG4HSwKjtqDz7
> SW2Bs3nDeiCr2QN7Su9TUaoSN/yKTBOw5jnql9cNOYuf4GAG0MXWinmYP8MWCiqh
> bIrzvOnlZGapX/36Fpab67VDcFnXJJc8ofqYmn+oUoW/q7geQpu/V1oz+AR9nD/d
> n8wFxvZdRlTbq2XDXySaontxNH0rd80fSG5SJtO5Js9hK/vNG+Xa7Zc+76gEtvlF
> 5G1F6MOcsoXUDCnMteuNxaZx6TPML6RuWVmbR1wXOaX4qZ01p7AsjQlTIcrlVDsB
> LVGW70wjfETBCffn1JQvFmIK7NzggIUuYfLF0IrfM4BrTa/01RnmfBOpWGiIvyIG
> qZqKLYDbW7BDkc/HMtAonR/0t6bTUv/388USSnbMakO9bvih0xxIw4NWTyoSoef/
> I+W6ny7qXt+pegLF4cYL
> =YqEM
> -----END PGP SIGNATURE-----
>
>bitcoin-dev mailing list
>bitcoin-dev at lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

2023-06-07 20:10:54

- reply

📅 Original date posted:2018-02-18
📝 Original message:> No rule has been “replaced”.
It really has been. The code is no longer checks the size of a block, but the weight of it. For all intents and purposes the block size limits has been replaced in favor of a block weight limit. It's just been designed such that the new rule is more restrictive than the one it replaces.
Now, I'm not doubting you can argue the semantics on the difference between "replacing with a more restrictive rule" and "leaving the existing rules, but adding a new one" -- but it's uninteresting and only serves to confuse novice's like CANNON into thinking that there are two rules/spaces/limits for transactions.
-Ryan

-------- Original Message --------
On February 18, 2018 1:39 PM, Eric Voskuil <eric at voskuil.org> wrote:
>As a soft fork, all preceding rules remain in effect. No rule has been “replaced”. Blocks must validate against pre-segwit rules or are invalid. Additional rules are applied that further restrict validity, and consider additional (witness) data in the context of the block.
>
> e
>
>>On Feb 18, 2018, at 09:04, Ryan Havar via bitcoin-dev bitcoin-dev at lists.linuxfoundation.org wrote:
>>No, you are misunderstanding. The block size limit (1MB) has been replaced in favor of a block weight limit (4M weight). Bytes which must be sent to old clients are weighted at 4 units each which is what allows it to be a soft fork.
>>So as such, there's not two separate limits or anything.
>>P.S. what's up with your signature lol
>>-Ryan
>>
>>-------- Original Message --------
>>>On February 18, 2018 11:26 AM, CANNON via bitcoin-dev bitcoin-dev at lists.linuxfoundation.org wrote:
>>>-----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA512
>>>I have a question in reference to the increased blockspace enabled
>>> by the segregated witness upgrade. Is this extra blockspace beyond
>>> the legacy 1 MB limit limited to just witness data?
>>>Cannon
>>> PGP Fingerprint: 2BB5 15CD 66E7 4E28 45DC 6494 A5A2 2879 3F06 E832
>>> Email: cannon at cannon-ciota.info
>>>NOTICE: ALL EMAIL CORRESPONDENCE NOT SIGNED/ENCRYPTED WITH PGP SHOULD
>>> BE CONSIDERED POTENTIALLY FORGED, AND NOT PRIVATE.
>>> -----BEGIN PGP SIGNATURE-----
>>>iQIcBAEBCgAGBQJaiajLAAoJEAYDai9lH2mwDbMQAKgJseZG9oOoP9WJlESFdAzm
>>> jLM69KLF4RVWDQjMWCtyVayyolXMLavW6fL4GL7/ztLYSl9Pz+FDlH3Qgo1GLDx4
>>> fRUkmb0ApLBSAjmdqM+kWjgq3s6/oZaIkNxGeyK5SYY4QhF80Z42PWLBPKjAc7vn
>>> pe7Im13NwtyJiRuCmq4e1z6GmW86fFOKOsR4rlWcftHOFmH8Gz5d9KK9LbPqU6Ca
>>> +rde8yb+A9uXEv3O9MqeLId7FYSJixgzFGJosidVkAVQI7YZ0TAfG4HSwKjtqDz7
>>> SW2Bs3nDeiCr2QN7Su9TUaoSN/yKTBOw5jnql9cNOYuf4GAG0MXWinmYP8MWCiqh
>>> bIrzvOnlZGapX/36Fpab67VDcFnXJJc8ofqYmn+oUoW/q7geQpu/V1oz+AR9nD/d
>>> n8wFxvZdRlTbq2XDXySaontxNH0rd80fSG5SJtO5Js9hK/vNG+Xa7Zc+76gEtvlF
>>> 5G1F6MOcsoXUDCnMteuNxaZx6TPML6RuWVmbR1wXOaX4qZ01p7AsjQlTIcrlVDsB
>>> LVGW70wjfETBCffn1JQvFmIK7NzggIUuYfLF0IrfM4BrTa/01RnmfBOpWGiIvyIG
>>> qZqKLYDbW7BDkc/HMtAonR/0t6bTUv/388USSnbMakO9bvih0xxIw4NWTyoSoef/
>>> I+W6ny7qXt+pegLF4cYL
>>> =YqEM
>>> -----END PGP SIGNATURE-----
>>>bitcoin-dev mailing list
>>>bitcoin-dev at lists.linuxfoundation.org
>>>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>bitcoin-dev mailing list
>>bitcoin-dev at lists.linuxfoundation.org
>>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

2023-06-07 20:10:44

- reply

📅 Original date posted:2018-02-12
📝 Original message:> This lead me to ponder whether the intuitive metric of satoshi/byte is, in fact, game
>theory optimal. Possibly over the short term it is, but over a longer period, those
> wishing to increase the longevity of proof of work in general might wish to consider
> more progressive fee approaches.
The constraining factor for blocks is the max-block weight. So miners are already optimizing for the right thing (creating a block that gives the most immediate reward). If miners want to start a cartel-like behavior of charging more for more value-transfer it would be incredibly harmful and even likely promote centralization (the cartel would likely not look kindly on any miner who doesn't follow their rules, and perhaps start orphaning their blocks).
Now I guess in theory you could add consensus rules that apply restrictions on the amount of "value transfer" in a block, such that miners are motivated to charge more for high-value transactions. However there's going to be almost 0 appetite from anyone to want to do anything like this, and the amount of unintended and harmful side effects would be profound. (Personally, I'd lose any interest in bitcoin if such a change was ever instated)
-Ryan

-------- Original Message --------
On February 12, 2018 12:23 PM, Melvin Carvalho via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>
>On 21 December 2017 at 22:30, Melvin Carvalho <melvincarvalho at gmail.com> wrote:
>>I asked adam back at hcpp how the block chain would be secured in the long term, once the reward goes away. The base idea has always been that fees would replace the block reward.
>>At that time fees were approximately 10% of the block reward, but have now reached 45%, with 50% potentially being crossed soon
>>
>>https://fork.lol/reward/feepct
>>
>>While this bodes well for the long term security of the coin, I think there is some legitimate concern that the fee per tx is prohibitive for some use cases, at this point in the adoption curve.
>>
>>Observations of segwit adoption show around 10% at this point
>>
>>http://segwit.party/charts/
>>
>>Watching the mempool shows that the congestion is at a peak, though it's quite possible this will come down over the long weekend. I wonder if this is of concern to some.
>>
>>https://dedi.jochen-hoenicke.de/queue/more/#24h
>>
>>I thought these data points may be of interest and are mainly FYI. Though if further discussion is deemed appropriate, it would be interesting to hear thoughts.
>>
>Just following up on this, for no other reason than I've had my eyes glued to these stats the last few weeks. I'll share a few more stats links.
>Mempool has come down significantly, as have fees. Tho, of course, this could spike any time.
>
>https://bitinfocharts.com/bitcoin/
>Typically fees are :
>
> $2.06 on tx $543 (median) # 0.38%
> $3.47 on tx $75,000 (mean) # 0.005%
>Aside: An observation on this. High value transactors seems to be getting a much better deal, than the mean. This lead me to ponder whether the intuitive metric of satoshi/byte is, in fact, game theory optimal. Possibly over the short term it is, but over a longer period, those wishing to increase the longevity of proof of work in general might wish to consider more progressive fee approaches. Naively, it might be possible to imagine some kind of gaussian distribution that picks tx according to a blended combination of sats/byte and %transacted. Perhaps something for miners and fee estimation algorithms to develop over time.
>Segwit adoption has increased, and anecdotal evidence shows that trend to continue. The release of 0.16 will I think also have a positive effect.
>Finally, I came across this wonderful site that shows lightning network adoption on mainnet
>
>http://shabang.io/
>LN is increasing well. Some blocks are not far off 1% lightning funding, which I think bodes well. I'll go out on a limb and predict that over 1% of btc tx will be lightning based by year end.
>Since such posts are not strictly development, I'll keep them to a minimum. However, I hope these stats provide useful data points for project evolution.
>
>

2023-06-07 20:10:43

- reply

📅 Original date posted:2018-02-12
📝 Original message:On February 12, 2018 5:58 PM, Peter Todd via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
> I don't actually see where the problem is here. First of all, suppose we have a
> transaction T_a that already pays Alice with a feerate sufficiently high that
> we expect it to get mined in the near future. If we want to pay Bob, we can do
> that by simply creating a double-spend of T_a that pays both Bob and Alice,
> T_{ab}. BIP125 only requires that double-spend to have an absolute fee higher
> than the minimum relay feerate * size of the transaction.
It's a bit of a made up term, but when Russel said "pinned" it refers to a child transaction that was made (i.e. in this case by Alice) and for us to replace our transaction we need to "unpin" it.
However to unpin it, the current bip125 rules require you pay not only the relay of the transactions you're throwing away, but the absolute fee. This is general is cost prohibitive, as even a normalish transaction can be spending $20 in fees.
Also FWIW this whole idea of T_a and T_ab is good, but it's also pretty impractical at the moment due to the sheer amount complexity it introduces (i.e. monitoring, seeing which confirms, trying to rebroadcast the missing one in a way that is safe against reorgs, blah blah).
>
> I just checked one of my nodes, and the absolute minimum relay fee is about
> 1/5th that of what estimatefee returns for the longest possible estimate, 48
> blocks.
If you use estimatesmartfee you should be able to get estimates all the way to 1008 or something btw
> [...]
> I think this is very important. For example, without this condition I could do
> a DoS attack by repeatedly broadcasting a transaction, then spending the
> outputs of that transaction with a very large number of child transactions, all
> of minimum fee.
I agree.
>
> A better way to solve this class of problems may be diffed tx replacement
> propagation: basically broadcast a diff between the original tx and the
> proposed replacement, allowing you to do the minimum bandwidth accounting based
> on the size of the diff instead.
This would definitely work for some specific use-case. For instance currently if you do n replacements of a transaction, each time adding an additional output .. you need to pay something like O(n^2) relay fee. If you used a diff instead, you could probably get it to O(n)ish.
But relay fee (and n) at the moment, mean it's not a big deal at all. The big flaw (imo) in bip125 is that you need to pay the absolute fee from the transactions you are evicting. And that can be from transactions you didn't even generate yourself. We can already compactly represent the diff (the new transaction invalidates it) the debate is more "Should you have to pay the absolute fee or just relay fee for the stuff you invalidate"

2023-06-07 20:10:43

- reply

📅 Original date posted:2018-02-14
📝 Original message:On February 13, 2018 1:40 PM, Peter Todd <pete at petertodd.org> wrote:
> Yeah, sorry, I just misread what scenario you guys were talking about. IIRC the
> term "pinned" may have even been invented by myself, as IIRC I noticed the
> issue when the RBF patch was being developed years ago. I don't think I had a
> solution at the time so I just punted on it.
Yeah. I posted that before it was clarified, it's just my message got held up in the moderation queue so it came out of order at an inconvenient time ><
> I'm not sure that's actually true, as you're only creating transactions sets
> that are reorg safe. Though I don't have a detailed design in mind so I may be
> missing something.
It is. T_a and T_ab are "reorg" safe, but if T_a confirms you will still need to pay Bob in way. But you need to pay him such that in a reorg occurs and suddenly T_ab is mined, you haven't doubled paid him.
I've been working on it's implementation, but it's honestly really complex and hard to test. I outlined the procedure here: https://gist.github.com/RHavar/cff76a026ece8446c898470db4f35682 which I call "Super Withdrawals".
My point though isn't that it's impossible, it's that it's sufficiently complex that it's unreasonable to expect anyone to be doing it any time soon. By relaxing any unnecessary restrictions on bip125, just makes it _drastically_ easier to do certain things.
> So here's a question: how many wallets have actually implemented CPFP fee bumps
> for incoming transactions?
Never tried it, but I recall seeing it in the electrum gui. I originally tried supporting this myself, but it's kind of annoying. It's generally a bit cost-prohibitive to create a transaction specifically for the purpose of a CPFP fee bump, but since I made transactions pretty frequently (averaged say every 8 minutes) it doesn't add an additional input for the purpose of bumping selected incoming transactions.
The work flow is reasonably smooth: Alice has sent me 1 BTC with low fees, I owe Bob some money. I source Alice's output in the payment to Bob, giving her transaction a fee bump. Both transactions confirm, everyone is happy.
However during the whole time I need to watch Alice's transaction because if it ever is replaced/conflicted, I need to immediately pay Bob (in a reorg safe way, so I don't double-pay). It's not terribly hard to do, by making sure when I pay Bob I use an additional input that I also use for any "repayment" but it's enough complexity and hard enough to test that I gave up.
The really nice thing of most current send systems (and now especially so with segwit) is everything is pretty much fire and forget. (although I just schedule in 0.5, 1, 2, 4, .... 32 hours fee bump attempts. But that's just background that can fail/succeed blindly)
>
>1. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-July/014688.html
>
> --
>https://petertodd.org 'peter'[:-1]@petertodd.org
>
>

2023-06-07 20:10:40

- reply

📅 Original date posted:2018-02-12
📝 Original message:Thank you very much for writing this up. It's worth noting that there can be multiple roots for the transactions that are getting replaced.
So for rule 3, you probably want a feeRate >= the max "package fee rate" of all replaced roots.
I am very happy with this proposal in general, as it's clearly a step in the right direction for making transaction replacement practically usable for todays services.
However, I think your new rule 4 is a bit weak. The logical extension of your proposal would be to allow a transaction (say B) be able to replace transactions (say A) by purely paying a higher fee rate, /even if it's less absolute fee/. In this simple example of B replacing A -- B should pay at least: (a.FeeRate * b.size) + relayFeeRate*(a.size + b.size)
-Ryan

-------- Original Message --------
On February 12, 2018 10:52 AM, Russell O'Connor via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>I think it is worth revisiting BIP 125's replace-by-fee policy for when to replace transactions.
>The current policy can be problematic. As noted earlier by Rhavar, sometimes one's transaction becomes pinned making it infeasible to fee bump with RBF. This happens when one makes a normal payment to a large commercial service, and, while the transaction remains unconfirmed, the commercial service creates a low-fee-rate sweep of one's payment, among a collection of others. If one wants to RBF this original payment, for example to get confirmation of the change output for use in further transactions, the current BIP 125 rules require that you make a fee bump that exceeds the combined total fees of the original transaction and the low-fee-rate sweep of the commercial service.
>
>The problem is that, while the fee rate of the sweep is low, the absolute size of the fee can still be large, making it infeasible to RBF the original transaction. BIP 125 was defined back in 2015, when perhaps rational miners did care about absolute fee amounts. However, today we are in an era where rational miners care about fee-rates more than absolute fees. The fee-rate of the large sweep transaction is low enough that we do not expect that miners will be mining it in the same block as the original transaction. Today, a rational miner will prefer a fee-bumped version of original transaction without consideration of the low-fee sweep transaction (or at least discounting the low-fee sweep in proportion to the miner's hash-rate fraction).
>
>Let me quote the five rules that define the current BIP 125 policy:
>
>>
>>One or more transactions currently in the mempool (original
>>transactions) will be replaced by a new transaction (replacement
>>transaction) that spends one or more of the same inputs if,
>>
>>
>>1. The original transactions signal replaceability explicitly or through inheritance as described in the above Summary section.
>>
>>2. The
>> replacement transaction does not contain any new unconfirmed inputs
>>that did not previously appear in the mempool. (Unconfirmed inputs are
>>inputs spending outputs from currently unconfirmed transactions.)
>>
>>3. The replacement transaction pays an absolute fee of at least the sum paid by the original transactions.
>>
>>4. The
>> replacement transaction must also pay for its own bandwidth at or above
>> the rate set by the node's minimum relay fee setting. For example, if
>>the minimum relay fee is 1 satoshi/byte and the replacement transaction
>>is 500 bytes total, then the replacement must pay a fee at least 500
>>satoshis higher than the sum of the originals.
>>
>>5. The number of
>>original transactions to be replaced and their descendant transactions
>>which will be evicted from the mempool must not exceed a total of 100
>>transactions.
>>
>>To address the new reality of rational miners' consideration, I propose changing rules 3 and 4 to something like the following.
>3'. The replacement transaction pays a fee rate of at least the effective fee rate of any chain of transactions from the set of original transactions that begins with the root of the original transaction set.
>
>4'. The
> replacement transaction must also pay for replacing the original transactions at or above
> the rate set by the node's minimum relay fee setting. For example, if
>the minimum relay fee is 1 satoshi/byte and the replacement transaction and the original transactions are 1000 bytes total, then the replacement must pay a fee at least 1000
>satoshis higher than the fee of the root transaction of the original transactions.
>Rule 3' is a fancy way of saying that the replacement transaction must have a fee rate that is larger than the package fee rate of the root of the set of transactions it replaces, where the package fee rate is the fee rate implied by considering CPFP.
>Rule 4' is an amended anti-spam rule that is intended to avoid DOS attacks from churning the mempool. I don't know if it is really necessary to pay for the size of the original transactions being evicted, but some people I chatted with thought it could be important.
>
>Other people on the mailing list have been thinking about RBF policy for far longer than I have, so I wouldn't be surprised if my proposal above is naive. However, I think it can start a conversation about addressing the problems with the current RBF policy.
>