Nostr notes by Pieter Wuille [ARCHIVE]

📅 Original date posted:2015-09-25 📝 Original message: On ...

2023-06-09T12:44:35Z

In reply to nevent1q…79pm
_________________________

📅 Original date posted:2015-09-25
📝 Original message:
On Sep 25, 2015 3:09 AM, "Rusty Russell" wrote:
> >> You can squeze some more bytes out of you want:
> >> 1) Signature should be 64 bytes (never DER encode).
> >> 2) Pubkey can be hashed bitcoin-address style, and recovered from sig.
> >
> > You can recover the pubkey from the hash and the sig? Why are we
putting the pubkey in the scriptSig then? ;)
>
> Because crypto is hard :(
>
> TBH I only learned a few months ago that you can do this.
>
> It helps if you have the (two-bit) recovery id, but you can brute force
> it AFAICT. You then check if the pubkey matches the hash you're given.

You can indeed do public key recovery om ECDSA, and you can brute force the
recovery id. In all non-pathological cases, the recovery id will be 0 or 1;
only one in about 2^128 randomly generated signatures need 2 or 3.

I don't have much context here, but is there a need for this to be ECDSA?
If not, the EC-Schnorr scheme in libsecp256k1 produces 64-byte
non-malleable signatures that support pubkey recovery without an additional
recovery id, and are compatible with the same private/public keys. The
scheme is certainly non-standard and experimental at this point, but it's
an instance of a well researched mechanism.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20150925/02de7137/attachment.html>;

📅 Original date posted:2023-02-17 🗒️ Summary of this ...

2023-06-07T23:19:50Z

In reply to nevent1q…qj23
_________________________

📅 Original date posted:2023-02-17
🗒️ Summary of this message: A proposal to use short IDs to minimize bandwidth in Bitcoin transactions has been suggested, with short ID 0 reserved for long commands. This would reduce complexity and eliminate the need for variable-length encoding.
📝 Original message:On Friday, February 17th, 2023 at 10:51 AM, Anthony Towns via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> I think it's probably less complex to close some of the doors?

> 2) are short ids available/meaningful to send prior to VERACK being
> completed?

Ah, I hadn't considered this nuance. If we don't care about them being available before VERACK negotiation, then it may be possible to introduce a way to negotiate a different short id mapping table without needing a mechanism for *re*-negotiating.

> Here's another approach:
>
> idea: we use short ids to minimise bandwidth, and don't care about
> bandwidth for long ids
>
> implementation:
> short id 0 is reserved for long commands. when received, we
> decode the first 12 bytes of the payload and treat them
> exactly the same as a v1 p2p message (trailing 0-bytes, etc)
> (if there's not 12 bytes of payload, it's just treated as an
> invalid command and dropped)
>
> short ids 1-255 are available for use as aliases of particular
> long commands
>
> (That's exactly compatible with p2p v1, and also avoids the temptation
> to try to choose short command names rather than descriptive ones -- the
> 0-padding to 12 bytes prevents you from saving any bandwidth that way;
> but that's what we have short ids for anyway)

I like this idea. It avoids the variable-length encoding question and related complexity entirely for things where we admittedly don't care about the bandwidth impact anyway.

It may also have another (rather weak) advantage, in that it may reduce how much information a passive observe may learn about application level features (sendheaders, sendaddrv2, ...) from the packet size sent (which would otherwise depend on command lengths), even when decoys are not in use, if no short commands are included for these messages.

> > - We remove 1 byte allocations for messages that are sent at most once per connection per direction
>
> I think this leaves 32 commands that get short ids initially:
>
> misc: ADDR, ADDRV2, BLOCK, FEEFILTER, GETBLOCKS, GETDATA, GETHEADERS,
> HEADERS, INV, NOTFOUND, PING, PONG, TX
> bip 35/37: FILTERADD, FILTERCLEAR, FILTERLOAD, MEMPOOL, MERKLEBLOCK
> bip 152: BLOCKTXN, CMPCTBLOCK, GETBLOCKTXN
> bip 157: CFCHECKPT, CFHEADERS, CFILTER, GETCFCHCKPT, GETCFHEADERS,
> GETCFILTERS
> bip 330: RECONCILDIFF, REQRECON, REQSKETCHEXT, SENDCMPCT, SKETCH

Sounds right.

> which drops:
>
> VERSION, VERACK, GETADDR, SENDADDRV2, SENDHEADERS, SENDTXRCNCL,
> WTXIDRELAY

Indeed.

> compared to bip 324 currently.
>
> I think the things missing from the current list (and not currently in
> use by bitcoin core) are:
>
> bip 61: REJECT
> bip 331: GETPKGTXNS, PKGTXNS, ANCPKGINFO

Do you feel REJECT should be included?

> > - Optionally, in the implementation we can attempt to move the type id mapping to the p2p layer away from the transport layer. I suspect this could also be done after the implementation is merged but might be cleaner as the mapping is a p2p concern.
>
> I agree that's fine, though I expect that we'll probably want to do it
> not long after bip 331 is ready for merge (or some other p2p improvement
> comes along)...

I do prefer that as well; it feels like the transport layer shouldn't be aware of the different command names that exist, but this is very much just an implementation issue.

Perhaps a possibility is having the transport layer translate short-command-number-N to the 12-byte command "\x00\x00..." + byte(N), and hand that to the application layer, which could then do the mapping?

Cheers,

--
Pieter

📅 Original date posted:2022-11-12 📝 Original ...

2023-06-07T23:16:42Z

In reply to nevent1q…79un
_________________________

📅 Original date posted:2022-11-12
📝 Original message:Another idea...

On Thursday, November 10th, 2022 at 4:23 PM, Pieter Wuille via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> On Thursday, November 3rd, 2022 at 1:53 PM, Murch wrote:
>
> > From what I understand we'll have about 35 message types on the network
> > with the addition of BIP324. 256 possible IDs sounds like plenty room to
> > grow, but perhaps we can be a bit more conservative:
> >
> > We could use the first bit to signal a 2-byte message ID. That allows us
> > to express 128 IDs with 1 byte, but if we need more, we get a total of
> > 2^15 IDs across 2 bytes.
>
> Yeah, effectively treating the first 1 or 2 bytes as a simple variable
> length integer is a nice way of increasing the space at low cost.

The above would really result in having two separate variable-length encodings:
* First byte 1-12 to signify length of alphabetic command
* Otherwise first bit to signify length of short command

I think we can just merge the two and have a single variable-length command structure that can be used for both: command encodings are 1 to 12 bytes, each byte's top bit indicating whether another byte follows (the top bit of byte 11 has no special meaning).

This means:
* Every alphabetic command of L characters becomes L bytes.
* 102 non-alphabetic 1-byte commands can be assigned.
* 15708 non-alphabetic 2-byte commands can be assigned.
* etc

So this gives a uniform space which commands can be assigned from, and there is no strict need for thinking of the short-binary and long-alphabetic commands as distinct. In v2, some short ones would be treated as aliases for old long-alphabetic ones. But new commands could also just be introduced as short ones only (even in v1).

WDYT?

Cheers,

--
Pieter

📅 Original date posted:2022-11-10 📝 Original message:Hi, ...

2023-06-07T23:16:41Z

In reply to nevent1q…ju28
_________________________

📅 Original date posted:2022-11-10
📝 Original message:Hi,

Thanks for the comments so far. I think these are all reasonable ideas.
Comments inline below.

On Thursday, November 3rd, 2022 at 1:53 PM, Murch wrote:

> From what I understand we'll have about 35 message types on the network
> with the addition of BIP324. 256 possible IDs sounds like plenty room to
> grow, but perhaps we can be a bit more conservative:
>
> We could use the first bit to signal a 2-byte message ID. That allows us
> to express 128 IDs with 1 byte, but if we need more, we get a total of
> 2^15 IDs across 2 bytes.

Yeah, effectively treating the first 1 or 2 bytes as a simple variable
length integer is a nice way of increasing the space at low cost.

This also doesn't need to be decided now. The initial approach could just
be avoiding allocating bytes in the 128-255 range until the need for more
space arises. If and when that is the case, the choice could be to:
* Just continue treating the first byte as the command.
* Start treating the first upper bit as a sign that another command byte
follows.
* Switch to some form of explicit signalling (option 3 is my earlier
mail).

On Thursday, November 3rd, 2022 at 6:26 PM, Jonas Schnelli wrote:

> There would be an alternative to preserve more 1 byte IDs on the cost
> of a (much) smaller 2 byte ID space: Reserve the short ID 0xFF as an
> indication for a 2 bytes short ID (additional 256 short IDs with 2 bytes).

I don't think this is needed, because we arguably already have that! If the
first byte is 0x01, then 1 more command byte follows in the current BIP324
draft. That mechanism is designed for alphabetic 1-character commands, but
nothing prevents it from also being used for other things (by using a
non-alphabetic byte there).

> Maybe the BIP should state that only frequent sent messages should reserve
> a short ID, though, the BIP itself assigns short IDs to all(?) message
> types (including low frequent messages like SENDHEADERS).
>
> Maybe exclude message types that expected to be only sent once from
> assigning a short ID?

I think that makes sense. Especially in combination with the idea avoiding
bytes with the upper bit set there is a bit more pressure on the 1-byte
space. Rarely-sent or at-most-once-sent commands don't really provide much
benefit. I'd suggest scrapping from the list:
* Version messages: version, verack
* Negotiation messages: sendaddrv2, sendheaders, sendcmpct, wtxidrelay
* Rarely-sent messages: mempool

I'm not sure to what extent filteradd/filterload/filterclear/merkleblock
are still actually used; perhaps they could be removed too?

On Monday, November 7th, 2022 at 10:20 PM, Anthony Towns wrote:

> I guess I think it would make sense to not start using a novel 1-byte
> message unless you've done something to introduce that message first;
> whether that's via approach (3) ("I'm going to use 0xE9 to mean pkgtxns")
> or via a multibyte feature support message ("I sent sendaddrv3 as a
> 10-byte message, that implies 0xA3 means addrv3 from now on").

That's fair, but I don't think it matters too much for allocation purposes;
protocol designs should still not assign overlapping values, unless the
protocols are known to never be used simultaneously?

Unless... the assignment works like "whenever the sendaddrv3 is sent, the
next available byte in range 48..127 gets allocated for addrv3". That means
no explicit mapping is needed, as long as the total number of messages from
simultaneously-active extensions isn't too large.

> I do still think it'd be better to recommend against reserving a byte for
> one-shot messages, and not do it for existing one-shot messages though.

Agree.

FWIW, if anyone was wondering about how much is actually saved by having
1-byte commands vs 12-byte commands, I've gathered statistics from two nodes
(one with many inbound connections, one only outbound) for two weeks. This is
obviously very dependent on network topology and local implementation choices,
but it may still give an idea:
* Outbound-only node:
* Around 4.5% of sent bytes are bytes 2-12 of the command.
* Sent 979.98 MiB in total.
* Outbound and inbound node:
* Around 1.6% of sent bytes are bytes 2-12 of the command.
* Sent 124.14 GiB in total.

Cheers,

--
Pieter

📅 Original date posted:2022-10-26 📝 Original message:Hi ...

2023-06-07T23:15:02Z

In reply to nevent1q…veec
_________________________

📅 Original date posted:2022-10-26
📝 Original message:Hi all,

On Saturday, October 8th, 2022 at 8:59 AM, Dhruv M <dhruv at bip324.com> wrote:

> We have refreshed the proposal for BIP324, a new bitcoin P2P protocol
> featuring opportunistic encryption, a mild bandwidth reduction, and the
> ability
> to negotiate upgrades before exchanging application messages. We'd like
> to invite community members to review the BIP[1] and the related Bitcoin
> Core
> code[2].

One open question we have regarding BIP324's design is how to deal with the
coordination of assigning the message type IDs.

For context, the current BIP324 draft introduces a notion of 1-byte message
type IDs, which take the place of the 12-byte command strings (in a backward
compatible way; it's still possible to send full strings). This offers a
mild bandwidth reduction (3 bytes per message overall), especially since many
messages on the network are fairly small.

However, it obviously raises the question of how the mapping table between the
1-byte IDs and the commands they represent should be maintained:

1. The most straightforward solution is using the BIP process as-is: let BIP324
introduce a fixed initial table, and future BIPs which introduce new
messages can introduce new mapping entries for it. In theory, this is no
worse than the current coordination difficulty about command strings, but
in practice the risk of collisions due to competing proposals is of course
significantly larger with 1-byte IDs vs. 12-byte strings.

2. An alternative approach is not using 1-byte IDs but slightly longer ones;
for example 3-byte IDs, each consisting of a 2-byte BIP number and a 1-byte
message index introduced by that BIP, at the cost of a smaller bandwidth
improvement. This significantly reduces collision risks, but doesn't remove
the coordination process concerns entirely (e.g. revisions changing what a
BIP introduces need to be taken into account and probably still mean BIPs
need to explicitly list which assignments they introduce).

3. Yet another possibility is not having a fixed table at all, and negotiate
the mapping dynamically. E.g. either side could send a message at
connection time with an explicit table of entries "when I send byte X, I
mean command Y".

4. Lastly, the whole feature could just be dropped from BIP324 (sticking with
command strings), and left for a follow-up (or independent) protocol
improvement. Since arguably this is purely an application-layer concern and
not a transport-layer one, it could even be added as an optional feature to
the (pre-BIP324) protocol today. That would however very likely mean that
BIP324 if adopted as-is isn't actually an (albeit small) bandwidth
reduction compared to today, and forego a possibility to fix a fairly
gratuitous inefficiency in the protocol from day one.

Our idea is to start out with approach (1), with a mapping table effectively
managed by the BIP process directly, but if and when collisions become a
concern (maybe due to many parallel proposals, maybe because the number of
messages just grows too big), switch to approach (3), possibly even
differentially (the sent mappings are just additions/overwrites of the
BIP-defined table mappings, rather than a full mapping).

That said, we're not all that convinced this is the best approach, and feel
this more a community/process question than a technical one, so it would be
good to see more opinions on the topic.

Cheers,

--
Dhruv, Pieter, Tim

📅 Original date posted:2022-10-12 📝 Original message:On ...

2023-06-07T23:14:19Z

In reply to nevent1q…2vcy
_________________________

📅 Original date posted:2022-10-12
📝 Original message:On Wednesday, October 12th, 2022 at 1:42 AM, Anthony Towns <aj at erisian.com.au> wrote:

> On Tue, Oct 11, 2022 at 04:18:10PM +0000, Pieter Wuille via bitcoin-dev wrote:
>
> > On Friday, October 7th, 2022 at 5:37 PM, Dario Sneidermanis via bitcoin-dev bitcoin-dev at lists.linuxfoundation.org wrote:
> >
> > > Thanks for the fast answer! It seems I missed the link to the PR, sorry for the
> > > confusion. I'm referring to the opt-in flag for full-RBF from #25353
> > > (https://github.com/bitcoin/bitcoin/pull/25353).
> > > It is not clear to me why you believe the merging of this particular pull request poses an immediate risk to you.
>
>
> Did you see the rest of Dario's reply, bottom-posted after the quoted
> text? Namely:

Oh, my mail client for some reason chose to hide all that. Dario, I'm sorry for missing this; I see now that you were certainly aware of what the PR under consideration did.

Further comments inline.

> On Fri, Oct 07, 2022 at 06:37:38PM -0300, Dario Sneidermanis via

> > The question then is whether an opt-in flag for full-RBF will have enough
> > adoption to get us from 1 to 2. If it isn't, then #25353 won't meet its
> > objective of allowing nodes participating in multi-party funding protocols
> > to assume that they can rely on full-RBF. If it is, then zero-conf applications
> > will be at severe risk (per the logic in the initial email).

>
>
> That logic seems reasonably sound to me:
>
> - if adding the option does nothing, then there's no point adding it,
> and no harm in restricting it to test nets only
>
> - if adding the option does do something, then businesses using zero-conf
> need to react immediately, or will go from approximately zero risk of
> losing funds, to substantial risk
>
> (I guess having the option today may allow you to manually switch your
> node over to supporting fullrbf in future when the majority of the network
> supports it, without needing to do an additional upgrade in the meantime;
> but that seems like a pretty weak benefit)

I certainly recognize that adding the flag is a likely step towards, over time, the full RBF policy becoming more widely adopted on the network. That is presumably the reason why people are in favor of having the flag, even default off - including me. I believe that policy's adoption is inevitable eventually, but the speed at which that is achieved is certainly a function of availability and adopted of software which provides the option.

That said, I think it's a bit of a jump to conclude that the only two options are that either the existence of the flag either has no effect at all, or poses an immediate threat to those relying on its absence. In my view, it is just what I said: a step towards getting full RBF on the network, by allowing experimentation and socializing the notion that developers believe it is time. So I have a hard time imagining how it would change anything *immediately* on the network at large (without things like default on and/or preferential peering, ...), but I still believe it's an important step.

Cheers,

--
Pieter

📅 Original date posted:2022-10-11 📝 Original message:On ...

2023-06-07T23:14:18Z

In reply to nevent1q…ct4d
_________________________

📅 Original date posted:2022-10-11
📝 Original message:On Friday, October 7th, 2022 at 5:37 PM, Dario Sneidermanis via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Hello David,
>
> Thanks for the fast answer! It seems I missed the link to the PR, sorry for the
> confusion. I'm referring to the opt-in flag for full-RBF from #25353
> (https://github.com/bitcoin/bitcoin/pull/25353).

Hello Dario,

It is not clear to me why you believe the merging of this particular pull request poses an immediate risk to you.

As explained by others, it's only a configuration option that is default off, and the possibility of running rull-RBF policy nodes on the network have been trivial for anyone who wanted to for a long time on the network.

I don't want to sound dismissive of your concerns, but at this point I'm not convinced you're actually aware of what this PR does and doesn't do.

Cheers,

--
Pieter

📅 Original date posted:2022-07-28 📝 Original ...

2023-06-07T23:12:19Z

In reply to nevent1q…0y5k
_________________________

📅 Original date posted:2022-07-28
📝 Original message:------- Original Message -------
On Thursday, July 28th, 2022 at 3:27 AM, Ali Sherief via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Essentially, zero-knowledge proofs such as Schnorr are not compatible with address message signing - the public key cannot be retrieved from the address or the signature, so the address does not actually prove the authenticity of a Schnorr signature. That's why the public key is required as an input in the first place.

Yes, that's an intentional design choice in BIP340, see note 5: https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki#cite_ref-5-0. The choice is either batch verifiability or public key recovery.

I regret ever using public key recovery when introducing the old legacy message signing scheme. It should just have used script signatures like BIP322 proposes.

> In order to make it compatible with the address signing mechanism, the zero-knowledge part would have to be sacrificed in my BIP, or else a completely separate message signing format just for Taproot would be required

You can avoid relying on public key recovery, and include the public key + BIP340 signature in the encoded signature.

> (which, in my view, is redundant - there is already the draft BIP322 which can verify anything and everything, but nobody is implementing that

I think it would be much better if people would cooperate to get BIP322 to move forward than to keep inventing other formats. It's the obvious solution in my opinion: not restricted to single-key policies, compatible with every script type, and trivially extensible to future schemes.

> , just like BIP340).

How so? Every taproot compatible wallet has a BIP340 implementation.

Cheers,

--
Pieter

📅 Original date posted:2022-02-06 📝 Original message:> ...

2023-06-07T23:03:52Z

In reply to nevent1q…lq89
_________________________

📅 Original date posted:2022-02-06
📝 Original message:> Dear Bitcoin Developers,

> -When I contacted bitInfoCharts to divide the first interval of addresses, they kindly did divided to 3 intervals. From here:
> https://bitinfocharts.com/top-100-richest-bitcoin-addresses.html
> -You can see that there are more than 3.1m addresses holding ≤ 0.000001 BTC (1000 Sat) with total value of 14.9BTC; an average of 473 Sat per address.

> -Therefore, a simple solution would be to follow the difficulty adjustment idea and just delete all those

That would be a soft-fork, and arguably could be considered theft. While commonly (but non universally) implemented standardness rules may prevent spending them currently, there is no requirement that such a rule remain in place. Depending on how feerate economics work out in the future, such outputs may not even remain uneconomical to spend. Therefore, dropping them entirely from the UTXO set is potentially destroying potentially useful funds people own.

> or at least remove them to secondary storage

Commonly adopted Bitcoin full nodes already have two levels of storage effectively (disk and in-RAM cache). It may be useful to investigate using amount as a heuristic about what to keep and how long. IIRC, not even every full node implementation even uses a UTXO model.

> for Archiving with extra cost to get them back, along with non-standard UTXOs and Burned ones (at least for publicly known, published, burn addresses).

Do you mean this as a standardness rule, or a consensus rule?

* As a standardness rule it's feasible, but it makes policy (further) deviate from economically rational behavior. There is no reason for miners to require a higher price for spending such outputs.
* As a consensus rule, I expect something like this to be very controversial. There are currently no rules that demand any minimal fee for anything, and given uncertainly over how fee levels could evolve in the future, it's unclear what those rules, if any, should be.

Cheers,

--
Pieter

📅 Original date posted:2021-12-12 📝 Original message:On ...

2023-06-07T23:01:12Z

In reply to nevent1q…x7fz
_________________________

📅 Original date posted:2021-12-12
📝 Original message:On Sunday, December 12th, 2021 at 9:23 AM, Aymeric Vitte via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Using the Tor network to bypass censorship for bitcoin can work but is a very poor solution, the Tor network is very centralized, very small, watched and controlled, with plenty of features that do not apply to other protocols than those made to be used with the Tor Browser, Pieter gave a simple example, that you can solve easily changing the circuits, the problem remains that you really need to be a super expert to escape all the dangers of the Tor network, not even sure it's possible unless you use something else than the Tor project code

FWIW, I wasn't talking about anything related to Tor's protocol or organization at all. What I meant is that because creating a hidden service has ~0 cost, it is trivial for anyone to spin up an arbitrary number of Bitcoin hidden services. Thus, if one runs a node that only connects to hidden services, it is fairly easily eclipsable.

It's just one example of a downside of (a particular way of) using Tor. That doesn't mean I recommend against using Tor for Bitcoin traffic at all; my point was simply that there are trade-offs, and aspects of privacy of the P2P protocol that Tor does not address, and thus one shouldn't assume that all problems are solved by "just use Tor".

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20211212/f92b2ade/attachment.html>;

📅 Original date posted:2021-12-11 📝 Original message:> ...

2023-06-07T23:01:10Z

In reply to nevent1q…qghz
_________________________

📅 Original date posted:2021-12-11
📝 Original message:> It is that the solution to privacy is to use privacy-enhancing network
> communications, such as TOR. I am not against a mechanism to rebroadcast
> transactions more robustly if the mempool of adjoining nodes has
> forgotten about them, but the truth is, all transactions originate from
> some node, and there are methods that allow an individual node to be
> identified as the likely source of a transaction unless privacy-enabled
> networks are utilised. Having a different method to cause rebroadcast
> does not obfuscate the origin.

You're talking about distinct aspects of transaction privacy.

The rebroadcasting approach as it exists on the network, where wallets are responsible for their own rebroadcasting, directly reveals to your peers a relation between nodes and transactions: whenever any node relays the same transaction twice, it almost certainly implies they are the origin.

This is just a node-transaction relation, and not necessarily IP-transaction relation. The latter can indeed be avoided by only connecting over Tor, or using other privacy networks, but just hiding the relation with IP addresses isn't sufficient (and has its own downsides; e.g. Tor-only connectivity is far more susceptible to partition/Eclipse/DoS attacks). For example seeing the same node (even without knowing its IP) rebroadcast two transaction lets an observe infer a relation between those transactions, and that too is a privacy leak.

I believe moving to a model where mempools/nodes themselves are responsible for rebroadcasting is a great solution to improving this specific problem, simply because if everyone rebroadcasts, the original author doing it too does not stand out anymore. It isn't "fixing privacy", it's fixing a specific leak, one of many, but this isn't a black and white property.

Cheers,

--
Pieter

📅 Original date posted:2021-08-29 📝 Original message:On ...

2023-06-07T22:58:11Z

In reply to nevent1q…zlul
_________________________

📅 Original date posted:2021-08-29
📝 Original message:On Saturday, August 28th, 2021 at 5:17 PM, ts via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Following up on my original proposal, I would like to get some more feedback of the community
>
> to see if this could be realized at some point. Also, any recommendations as to who to contact
>
> to get things rolling?

I honestly don't understand the point of what you're suggesting.

* If you're concerned about random typos, this is something already automatically protected against through the checksum (both base58check or bech32/bech32m).

* If you're concerned about accidentally entering the wrong - but honestly created - address, comparing any few characters of the address is just as good as any other. It doesn't even require the presence of a checksum. Looking at the last N characters, or the middle N, or anything except the first few, will do, and is just as good as an "external" checksum added at the end. For randomly-generated addresses (as honest ones are), each of those has exactly as much entropy.

* If you're concerned about maliciously constructed addresses, which are designed to look similar in specific places, an attacker can just as easily make the external checksum collide (and having one might even worsen this, as now the attacker can focus on exactly that, rather than needing to focus on every other character).

Things would be different if you'd suggest a checksum in another medium than text (e.g. a visual/drawing/colorcoding one). But I don't see any added value for an additional text-based checksum when addresses are already text themselves. This is even disregarding the difficulty of getting the ecosystem to adopt such changes.

Cheers,

--
Pieter

📅 Original date posted:2021-02-11 📝 Original ...

2023-06-07T18:28:29Z

In reply to nevent1q…cyac
_________________________

📅 Original date posted:2021-02-11
📝 Original message:‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, February 11, 2021 6:38 AM, Dr Maxim Orlovsky <orlovsky at protonmail.com> wrote:

> Thank you very much for all the clarifications; it’s good to have them sorted out and clearly structured. From what you wrote it follows that we still need to reserve a dedicated purpose (with new BIP) for BIP340 signatures to avoid key reuse, am I right?

Maybe, but it would be for a particular way of using keys (presumably: single-key pay-to-taproot), not just the signature scheme itself. If you go down this path you'll also want dedicated branches for multisig participation, and presumably several interesting new policies that become possible with Taproot.

The only thing ECDSA/Schnorr specific about this is that - if you want to maintain provable security - the keys used for ECDSA and BIP340 should be separated by a hardened step. It seems however that all approaches people actually use to prevent reuse do that already.

And as I said, dedicated branches only help for the simple case. For example, it doesn't address the more general problem of preventing reuse of keys in multiple distinct groups of multisig sets you participate in. If you want to solve that you need to keep track of index is for participating in what - and once you have something like that you don't need dedicated purpose based derivation at all anymore.

So I'm not sure I'd state it as us *needing* a dedicated purpose/branch for single-key P2TR (and probably many other useful ways of using taproot based spending policies...). But perhaps it's useful to have.

Greg Maxwell pointed out to me that there may be another reason to want non-reuse across ECDSA and BIP340 keys: if someone were to do all of these wrong:
* not follow BIP340 and re-use RFC6979 for BIP340 nonce generation
* reuse the same keys for both
* sign the same message with both
... you would actually leak your private key. This isn't a concern for Bitcoin transaction signing however, as the sighash (message) indirectly commits to BIP341 or not, and thus it'd be impossible to construct colliding messages. Still, it's a consideration to factor in.

Cheers,

--
Pieter

📅 Original date posted:2021-02-05 📝 Original message:On ...

2023-06-07T18:28:28Z

In reply to nevent1q…xmc4
_________________________

📅 Original date posted:2021-02-05
📝 Original message:On Friday, February 5, 2021 9:51 AM, Dr Maxim Orlovsky via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Hi,
>
> Background
>
> ====================
>
> Had a discussion last night in Bitcoin Core IRC with Peter Wuille on different topics regarding key derivations, security, key tweaks in context of Schnorr signatures & Taproot. Would like to share some action points and plans that emerged from there:
>
> 1. There is a real need for BIP-43 based new BIP with a new purpose field for keys used in Schnorr signatures. Peter will not do it (he is not very much interested in spending his time on wallet-level stuff), so someone else should.
> 2. Keys used in Schnorr signatures MUST NEVER BE used in ECDSA signatures, otherwise there is a risk of private key leak via correlation attack. This is rationale behind N1.

Hi Maxim,

thanks for bringing up this discussion here. I'd like to clarify a few things though, as I think the above is formulated far too strongly.

There are two issues here: (1) reasons to avoid reusing the same key for privacy reasons, and (2) reasons to avoid using related keys for cryptographic reasons. And in practice, solutions to the first (which we already need, unrelated to Taproot/Schnorr) mean the second is largely moot.

# Don't reuse keys for privacy/organizational reasons.

Reusing the same key in Bitcoin scripts - for use in distinct signature schemes or not - should always be avoided. It has obvious privacy implications; I don't think this is controversial, and it's a problem that exists today, unrelated to Taproot. E.g. you don't want to reuse the same keys as both single-sig and participation in multisig.

My personal view here is that distinct standard derivation paths help with this in the simple cases, but they're not a full solution in the most general case. E.g. if you want to use one seed/root to participate in multiple sets of multisig policies, you'll need some kind of index to assign to each anyway. For this reason in general I prefer solutions that explicitly track what path is used for what.

# Don't use related keys for cryptographic reasons

There are some concerns to address here, but I want to make it clear there are no known attacks against usage of related keys across ECDSA and Schnorr, and I don't expect there will be. However, there is probably no proof for this, and creating one may be tricky. On the other hand, the ecosystem (Bitcoin, but also many other applications) has accepted ECDSA long ago, while it had no security proofs whatsoever (and even the ones that exist now rely on fairly unusual assumption; a proof for security of mixed Schnorr/ECDSA usage would inherently need equally unusual assumptions too).

Now, as soon as a hardened derivation separates different uses there is no concern at all. So any solution for avoiding key reuse (section above) that works by assigning (implicitly or explicitly) different hardened derivation paths (as BIP43 etc. do) to different uses implies there is never any concern about related keys across Schnorr/ECDSA.

If the keys are not separated by a hardened step, it's more complicated. Looking at the different cases:

(1) Signing with related ECDSA keys (e.g. two unhardened child keys from the same xpub)

- This is very common on BIP32 usage today, so hopefully it is secure! Tim Ruffing pointed me today to https://link.springer.com/chapter/10.1007/978-3-030-36938-5_8 whose abstract seems to indicate they prove this is actually secure, but I don't know under what assumptions. Note that the comment there about Schnorr not having this property does not apply to BIP340, because it uses key-prefixed Schnorr.

(2) Signing with related Schnorr keys.

- I believe this is secure simply because BIP340 challenges commit to the pubkey (key prefixing), and thus a signature on one shouldn't teach you anything about others. A formal proof is probably a bit longer, but still trivial to construct.

(3) The real question: signing with a Schnorr key that is related to an ECDSA key.

- I don't expect that this is easy to prove, but I have a very hard time imagining how it could be exploitable, given the use of domain-separated hashes. Aspects such as BIP340's key prefixing and the fact that Bitcoin sighashes indirectly commit to the (set of) signing pubkeys make it even harder.

# TL;DR

* For privacy reasons, you should use separate keys/derivation branches for different uses in all circumstances (duh).
* To stay within the realm of provably security it's advisable to make sure ECDSA key and Schnorr keys use distinct hardened derivation steps.
* Even if you don't, you're really just back to the level of assurance we had about unhardened BIP32 ECDSA keys before a proof was known (which I think most people aren't even aware of).

Cheers,

--
Pieter

📅 Original date posted:2020-12-21 📝 Original message:On ...

2023-06-07T18:27:52Z

In reply to nevent1q…wjhw
_________________________

📅 Original date posted:2020-12-21
📝 Original message:On Sunday, December 20, 2020 9:37 PM, Karl-Johan Alm via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Thanks a lot for taking the time to brush up the BIP. For what it's
> worth, I am all for these changes, and I see them as clear
> improvements all around.
>
> IIRC Pieter was the one who originally suggested the two-checks
> approach (invalid, inconclusive, valid) which is being modified here,
> so would be good if you chimed in (or not -- which I'll assume means
> you don't mind).

I agree with the idea of permitting incomplete validators to return inconclusive as well. That doesn't really reduce the functionality (given that "inconclusive" was already a potential result), and it obviously makes it much more accessible to a variety of software.

This suggestion breaks the original use of inconclusive though: the ability to detect that future features are used in the signature. The idea was to use divergence between "consensus valid" and "standardness valid" as a proxy for future extensions to be detected (e.g. OP_NOPn, future witness versions, ...). I think it's undesirable that these things now become unconditionally invalid (until the BIP is updated, but once that happens old validators will give a different result than new ones).

Since the BIP no longer relies on a nebulous concept of standardness, and instead specifically defines which standardness features are to be considered, this seems easy to fix: whenever validation fails due to any of those, require reporting inconclusive instead of invalid (unless of course something actually invalid also happens). In practice I guess you'd implement that (in capable validators) by still doing validation twice, once with all features enabled that distinguish between valid/invalid, and if valid, again but now with the features enabled that distinguish between valid and (invalid or inconclusive).

Cheers,

--
Pieter

📅 Original date posted:2020-12-21 📝 Original message:On ...

2023-06-07T18:27:52Z

In reply to nevent1q…fc3d
_________________________

📅 Original date posted:2020-12-21
📝 Original message:On Monday, December 21, 2020 2:57 PM, Pieter Wuille via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> On Sunday, December 20, 2020 9:37 PM, Karl-Johan Alm via bitcoin-dev bitcoin-dev at lists.linuxfoundation.org wrote:
>
> > Thanks a lot for taking the time to brush up the BIP. For what it's
> > worth, I am all for these changes, and I see them as clear
> > improvements all around.
> > IIRC Pieter was the one who originally suggested the two-checks
> > approach (invalid, inconclusive, valid) which is being modified here,
> > so would be good if you chimed in (or not -- which I'll assume means
> > you don't mind).
>
> I agree with the idea of permitting incomplete validators to return inconclusive as well. That doesn't really reduce the functionality (given that "inconclusive" was already a potential result), and it obviously makes it much more accessible to a variety of software.
>
> This suggestion breaks the original use of inconclusive though: the ability to detect that future features are used in the signature. The idea was to use divergence between "consensus valid" and "standardness valid" as a proxy for future extensions to be detected (e.g. OP_NOPn, future witness versions, ...). I think it's undesirable that these things now become unconditionally invalid (until the BIP is updated, but once that happens old validators will give a different result than new ones).
>
> Since the BIP no longer relies on a nebulous concept of standardness, and instead specifically defines which standardness features are to be considered, this seems easy to fix: whenever validation fails due to any of those, require reporting inconclusive instead of invalid (unless of course something actually invalid also happens). In practice I guess you'd implement that (in capable validators) by still doing validation twice, once with all features enabled that distinguish between valid/invalid, and if valid, again but now with the features enabled that distinguish between valid and (invalid or inconclusive).

Re-reading your proposed text, I'm wondering if the "consensus-only validation" extension is intended to replace the inconclusive-due-to-consensus-and-standardness-differ state. If so, I don't think it does, and regardless it doesn't seem very useful.

What I'm suggestion could be specified this way:
* If validator understands the script:
* If signature is consensus valid (as far as the validator knows):
* If signature is not known to trigger standardness rules intended for future extension (well-defined set of rules listed in BIP, and revisable): return valid
* Otherwise: return inconclusive
* Otherwise: return invalid
* Otherwise: return inconclusive

Or in other words: every signature has a well-defined result (valid, invalid, inconclusive) + validators may choose to report inconclusive for anything they don't understand.

This has the property that as long as new consensus rules only change things that were covered under for-future-extension standardness rules, no two validators will ever claim valid and invalid for the same signature. Only valid+inconclusive or invalid+inconclusive.

Cheers,

--
Pieter

📅 Original date posted:2020-12-06 📝 Original ...

2023-06-07T18:27:41Z

In reply to nevent1q…0tcr
_________________________

📅 Original date posted:2020-12-06
📝 Original message:‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, December 6, 2020 5:04 AM, David A. Harding <dave at dtrt.org> wrote:

> On Sat, Dec 05, 2020 at 11:10:51PM +0000, Pieter Wuille via bitcoin-dev wrote:
>
> > I think these results really show there is no reason to try to
> > maintain the old-software-can-send-to-future-segwit-versions property,
> > given that more than one not just didn't support it, but actually sent
> > coins into a black hole.
>
> I don't think this is a good criteria to use for making a decision. We
> shouldn't deny users of working implementations the benefit of a feature
> because some other developers didn't implement it correctly.
>
> > Thus, I agree with Rusty that we should change the checksum for v1+
> > unconditionally.
>
> I disagreed with Rusty previously and he proposed we check to see how
> disruptive an address format change would be by seeing how many wallets
> already provide forward compatibility and how many would need to be
> updated for taproot no matter what address format is used. I think that
> instead is a good criteria for making a decision.
>
> I understand the results of that survey to be that only two wallets
> correctly handled v1+ BIP173 addresses. One of those wallets is Bitcoin
> Core, which I personally believe will unhesitatingly update to a new
> address format that's technically sound and which has widespread support
> (doubly so if it's just a tweak to an already-implemented checksum
> algorithm).

Hi Dave,

You're right to point out there is nuance I skipped over.

Let's look at the behavior of different classes of software/services that exist today when trying to send to v1+ addresses:

(A) Supports sending to v1+ today
* Old proposal: works, but subject to bech32 insertion issue
* New proposal: fails
(B) Fails to send to v1+ today
* Old proposal: fails
* New proposal: fails
(C) Incorrectly sends to v1+ today
* Old proposal: lost funds
* New proposal: fails

So the question is how the support for sending to v1+ in (a) software weighs up against protecting both (a) from the insertion issue, and (c) from lost funds. I do think (c) matters in this equation - people may choose to avoid adopting v1+ witnesses if it were to be known that some senders out there would misdirect funds. But the fact that (a) is small also means there is very little to gain from the old proposal.

So perhaps I should have formulated it as: the small number of v1+ compatible senders today (regardless of the reasons for that) shows how futile the attempt to have one address type for all witness versions was, and the fact that there are even some who misdirect(ed) funds is the final nail in the coffin. Changing the checksum unconditionally gives us a new attempt at that.

> Given that, I also now agree with changing the checksum for v1+.

Great.

Cheers,

--
Pieter

📅 Original date posted:2020-12-05 📝 Original message:On ...

2023-06-07T18:27:41Z

In reply to nevent1q…yd3c
_________________________

📅 Original date posted:2020-12-05
📝 Original message:On Friday, November 6, 2020 11:49 AM, Mike Schmidt via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> Well I sure picked a bad couple weeks to volunteer to send a bunch of Bitcoin test transactions...
>
> While I tested less than I would have liked, there are some notable results:

I think these results really show there is no reason to try to maintain the old-software-can-send-to-future-segwit-versions property, given that more than one not just didn't support it, but actually sent coins into a black hole.

Thus, I agree with Rusty that we should change the checksum for v1+ unconditionally. That also means that old senders are protected from the insertion issue (by failing, as we can guarantee that new-checksum addresses, even after a few errors, are invalid to old software).

I've sent another mail in this thread with details, but the TL;DR is that we should use the constant M=0x2bc830a3 rather than 0x3fffffff as previous suggested. More information on https://gist.github.com/sipa/14c248c288c3880a3b191f978a34508e.

Absent objections, I'll write up a BIP soon.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20201205/32d286a4/attachment.html>;

📅 Original date posted:2020-12-05 📝 Original message:> ...

2023-06-07T18:27:40Z

In reply to nevent1q…0ws7
_________________________

📅 Original date posted:2020-12-05
📝 Original message:> On Wednesday, October 7, 2020 5:21 PM, Rusty Russell via bitcoin-dev bitcoin-dev at lists.linuxfoundation.org wrote:
>
> > I propose an alternative to length restrictions suggested by
> > Russell in https://github.com/bitcoin/bips/pull/945: use the
> > https://gist.github.com/sipa/a9845b37c1b298a7301c33a04090b2eb variant,
> > unless the first byte is 0.
>
> Hi all,
>
> starting a slight side-thread here.

Another update, and a longer write-up.

Introduction
------------

Bech32's checksum algorithm was designed to be strong against substitution
errors, but it also provides some protection against more general classes of
errors. The final constant M that is XOR'ed into the checksum influences that
protection. BIP173 today uses M=1, but it is now known that this has a
weakness: if the final character is a "p", any number of "q" characters can be
inserted or erased right before it, without invalidating the checksum.

As it was recognized that other constants do not have this issue, the obvious
question is whether this is the only possible type of weakness, and if not, if
there is an optimal constant to use that minimizes the largest number of
weaknesses.

Since my last mail I've realized that it is actually possible to analyse the
behavior of these final constants under a wide variety of error classes
(substitutions, deletions, insertions, swaps, duplications) programatically.
Greg Maxwell and I have used this to perform an exhaustive analysis of certain
error patterns for all 2^30 possible M values, selected a number of criteria
to optimize for, and conclude that we should use as constant:

M = 0x2bc830a3

The code used to do this analysis, as well as the code used to verify some
expected properties of the final result, and more, can be found on
https://gist.github.com/sipa/14c248c288c3880a3b191f978a34508e

See results_final.txt to see how this constant compares with the previously
suggested constants 1, 0x3fffffff, and 0x3fefffff.

Properties
----------

If we define an error as a deletion of one position, a swap of adjacent
positions, a substitution in a specific position with a random character, an
insertion of one random character in a specific position, or a duplication of
the character in a specific position, then this M constant above gives us the
following properties:

* For generic HRPs and errors that don't affect the first 6 data characters,
or alternatively averaged over all HRPs (see details futher):
* Always detected:
* (P) Up to 4 substitution errors (true for any constant).
* (Q) Any valid code with the new constant, fed without modification to
software that uses the old constant 1 (true for any constant).
* Any error pattern has failure to detect probability <= 2^-30:
* (L) Any number of errors restricted to a single window of up to 4
characters.
* (B) Up to two errors restricted to a window of up to 68 characters.
* (D) Any one error made to a valid code with the new constant, and fed to
software that uses the old constant 1
* Most error patterns have probability <= 2^-30:
* (C) Up to two errors in general: out of 23926796 such error patterns,
0.0040% have probability 2^-25.
* (N) Up to three errors restricted to a window of up to 69 characters:
out of 284708444 such patterns, 0.033% have probability 2^-25.
* (O) Up to three errors in general: out of 295744442 such error patterns,
0.034% have probability 2^-25; 0.000065% have probability 2^-20.
* (G) Up to two errors made to a valid code with the new constant, and fed
to software that uses the old constant 1: out of 2831622 such error
patterns, 0.048% have probability 2^-25.

* Specifically for the bc1 HRP, with the BIP173 length restrictions:
* Always detected:
* (R) Up to 4 substitution errors (true for any constant).
* (A) Up to 3 substitution errors made to a valid code with the new
constant, and fed to software that uses the old constant 1.
* Any error pattern has failure to detect probability <= 2^-30:
* (E) Any one error.
* (F) Any one error made to a valid code with the new constant, and fed to
software that uses the old constant 1.
* (H) Up to two errors restricted to a window of 28 characters.
* Most error patterns have probability <= 2^-30:
* (J) Up to two errors in general: out of 455916 such error patterns,
0.039% have probability 2^-25; 0.0053% have 2^-20.
* (K) Any number of errors restricted to a window of 4 characters: out of
5813139 such error patterns, 0.0016% have probability 2^-25.
* (M) Up to three errors: out of 50713466 such error patterns, 0.078% have
probability 2^-25; 0.00063% have 2^-20.
* (I) Up to two errors made to a valid code with the new constant, and fed
to software that uses the old constant 1: out of 610683 such error
patterns, 0.092% have probability 2^-25; 0.00049% have probability
2^-20.

To give an idea of what these probabilities mean, consider the known BIP173
insertion issue. It admits an error pattern of 1 error (insertion in
penultimate position) that has a failure to detect probability of 2^-10:
it requires the final character to be 'p', and the inserted character to be
'q'. Assuming those are both random, we have a chance of 1 in 32*32 to hit it.
Note that the choice of *what* the error pattern is (whether it's insertion,
and where) isn't part of our probabilities: we try to make sure that *every*
pattern behaves well, not just randomly chosen ones, because presumably humans
make some kinds of errors more than others, and we don't know which ones.

All the analyzed patterns above are guaranteed to be detected with probability
2^-20 or better (and most are 2^-30). Of course, if we'd search for even
larger classes of errors, say any 4 independent errors of any type, we would
probably discover patterns with worse probabilities, but at that point the
probability of the pattern itself being hit should be taken into account.

The selection was made based on these same properties:
* Start with the set of all 2^30 constants.
* The generic properties (L), (B), (D), (C), (N), (O), and (G) were selected
for by rejecting all constants that left any worse error patterns (e.g.
all codes for which patterns matching (N) existed with failure probability
above 2^-25 were removed). All these restrictions are as strong as they
can be: making them over longer strings, wider windows, or more errors with
the same restrictions removes all remaining constants. This leaves us with
just 12054 acceptable constants.
* The same was then done for the bc1/BIP173 specific properties (A), (E), (J),
(F), (H), (K), (M), and (I). This reduces the set further to 79 acceptable
constants. The full analysis output for all of these can be found in
output.txt.
* Finally, the constant with the minimal number of worst-probability patterns
was chosen for the generic property (N). The single constant 0x2bc830a3
remains.
* This solution and a few of its expected properties were then validated using
a simple program that makes random errors (see the monte_carlo.py file).

Technical details
-----------------

For the purpose of this analysis, define an "error pattern" as a starting
length (of a valid string consisting of otherwise randomly chosen characters)
combined with a sequence of the following (in this order):
* 0 or more deletions of characters at specific positions (without
constraining what those characters are)
* 0 or more swaps of characters at specific positions with the character
following it
* 0 or more substitutions of characters at specific positions with a uniformly
randomly selected character
* 0 or more insertions of uniformly randomly selected characters at specific
positions
* 0 or more duplications of characters at specific positions (including
duplications of characters inserted/substituted in the previous steps)

Examples:
* "Start with a random valid 58 character string, remove the 17th character,
swap the 11th character with the 12th character, and insert a random
character in the 24th position" is an error pattern.
* "Replace the 17th through 24th characters in a 78 character string with
'aardvark'" is not an error pattern, because substituted characters have to
be random, and can't be specific values.

Given such a pattern, assign variable names to every input character, and to
every inserted/substituted character. For example, the pattern "Start with
a 6 character string, delete the 1st character, swap the 2nd and 3rd
character, and insert a random character between those" would be represented
as [v0 v1 v2 v3 v4 v5] and [v1 v3 v6 v2 v4 v5]. Treat these variables as
elements of GF(32), and write out the equations that both the first and second
list have a valid checksum. Due to the fact that BCH codes are linear, this is
just a linear set of equations over GF(32), and we can use Gaussian
elimination to find the size of the solution space. If the input and output
are the same length, we need to subtract the number of solutions for which the
input and output are exactly the same, which is easy to find with another set
of equations. Now compute the ratio of this number divided by (32^numvars /
32^6), where the 32^6 is due to the precondition that the input string is
valid. This gives us the probability of failure, assuming input and output are
random, apart from the known relation between the two, and the fact that both
are valid.

This technique has an important limitation: it can only reason about randomly-
chosen input strings, and the presence of the HRP and version numbers at the
start violates that assumption. These are not random, and we're forced to
make one of these concessions:
1) Ignore the problem, and treat the HRP as random. This lets us derive
properties that hold over all potential HRPs on average, but will thus fail
to account for the possibility that for a small numbers of potential HRPs
some error patterns may exist that behave worse. For technical reasons,
this averaging makes all constants behave identically for error patterns
that don't change the length of the string. Given that substitution/swap
only errors are already dealt with well due to the BCH design this is
perhaps not too important. One exception is frame-shifting errors (a
deletion in one place compensated with an insertion in another place).
2) Restrict analysis to error patterns that don't affect the first 6 actual
characters. Doing so "masks" the effect of the HRP completely.
3) Do analysis for specific HRPs only, allowing much more accurate statements,
but HRP-specific ones that may not hold for every HRP.

Our final selection primarily optimizes for 1) and 2) as those benefit all
potential uses of the encoding, but do optimize for 3) the "bc1" prefix
specifically (and the BIP173 length restriction) as a tiebreaker.

The code for this can be found under the link above, in const_analysis.cpp.

Cheers,

--
Pieter

📅 Original date posted:2020-08-19 📝 Original message:On ...

2023-06-07T18:26:21Z

In reply to nevent1q…5ue8
_________________________

📅 Original date posted:2020-08-19
📝 Original message:On Wednesday, August 12, 2020 11:49 AM, Pieter Wuille <bitcoin-dev at wuille.net> wrote:

> It is late in the process, but I feel I owe this explanation so that at least the possibility of changing can be discussed with all information.

As the responses have been pretty positive so far, we've gone ahead and made a patch to implement the change to the even tiebreaker: https://github.com/sipa/bips/pull/210

If there are no other arguments (against), I'll PR it to the BIPs repo in a week or so.

All comments/questions/... still welcome.

Cheers,

--
Pieter

📅 Original date posted:2020-08-12 📝 Original message:Hello ...

2023-06-07T18:26:19Z

In reply to nevent1q…9a5p
_________________________

📅 Original date posted:2020-08-12
📝 Original message:Hello all,

The current BIP340 draft[1] uses two different tiebreakers for conveying the Y coordinate of points: for the R point inside signatures squaredness is used, while for public keys evenness is used. Originally both used squaredness, but it was changed[2] for public keys after observing this results in additional complexity for compatibility with existing systems.

The reason for choosing squaredness as tiebreaker was performance: in non-batch signature validation, the recomputed R point must be verified to have the correct sign, to guarantee consistency with batch validation. Whether the Y coordinate is square can be computed directly in Jacobian coordinates, while determining evenness requires a conversion to affine coordinates first.

This argument of course relies on the assumption that determining whether the Y coordinate is square can be done more efficiently than a conversion to affine coordinates. It appears now that this assumption is incorrect, and the justification for picking the squaredness tiebreaking doesn't really exist. As it comes with other trade-offs (it slows down signing, and is a less conventional choice), it would seem that we should reconsider the option of having the R point use the evenness tiebreaker (like public keys).

It is late in the process, but I feel I owe this explanation so that at least the possibility of changing can be discussed with all information. On the upside, this was discovered in the context of looking into a cool improvement to libsecp256k1[5], which makes things faster in general, but specifically benefits the evenness variant.

# 1. What happened?

Computing squaredness is done through the Jacobi symbol (same inventor, but unrelated to Jacobian coordinates). Computing evenness requires converting points to affine coordinates first, and that needs a modular inverse. The assumption that Jacobi symbols are faster to compute than inverses was based on:

* A (possibly) mistaken belief about the theory: fast algorithms for both Jacobi symbols and inverses are internally based on variants of the same extended GCD algorithm[3]. Since an inverse needs to extract a full big integer out of the transition steps made in the extgcd algorithm, while the Jacobi symbol just extracts a single bit, it had seemed that any advances applicable to one would be applicable to the other, but inverses would always need additional work on top. It appears however that a class of extgcd algorithms exists (LSB based ones) that cannot be used for Jacobi calculations without losing efficiency. Recent developments[4] and a proposed implementation in libsecp256k1[5] by Peter Dettman show that using this, inverses in some cases can in fact be faster than Jacobi symbols.

* A broken benchmark. This belief was incorrectly confirmed by a broken benchmark[6] in libsecp256k1 for the libgmp-based Jacobi symbol calculation and modular inverse. The benchmark was repeatedly testing the same constant input, which apparently was around 2.5x faster than the average speed. It is a variable-time algorithm, so a good variation of inputs matters. This mistake had me (and probably others) convinced for years that Jacobi symbols were amazingly fast, while in reality they were always very close in performance to inverses.

# 2. What is the actual impact of picking evenness instead?

It is hard to make very generic statements here, as BIP340 will hopefully be used for a long time, and hardware advancements and algorithmic improvements may change the balance. That said, performance on current hardware with optimized algorithms is the best approximation we have.

The numbers below give the expected performance change from squareness to evenness, for single BIP340 validation, and for signing. Positive numbers mean evenness is faster. Batch validation is not impacted at all.

In the short term, for block validation in Bitcoin Core, the numbers for master-nogmp are probably the most relevant (as Bitcoin Core uses libsecp256k1 without libgmp, to reduce consensus-critical dependencies). If/when [5] gets merged, safegcd-nogmp will be what matters. On a longer time scale, the gmp numbers may be more relevant, as the Jacobi implementation there is certainly closer to the state of the art.

* i7-7820HQ: (verify) (sign)
- master-nogmp: -0.3% +16.1%
- safegcd-nogmp: +6.7% +17.1%
- master-gmp: +0.6% +7.7%
- safegcd-gmp: +1.6% +8.6%

* Cortex-A53: (verify) (sign)
- master-nogmp: -0.3% +15.7%
- safegcd-nogmp: +7.5% +16.9%
- master-gmp: +0.3% +4.1%
- safegcd-gmp: 0.0% +3.5%

* EPYC 7742: (verify) (sign)
- master-nogmp: -0.3% +16.8%
- safegcd-nogmp: +8.6% +18.4%
- master-gmp: 0.0% +7.4%
- safegcd-gmp: +2.3% +7.8%

In well optimized cryptographic code speedups as large as a couple percent are difficult to come by, so we would usually consider changes of this magnitude relevant. Note however that while the percentages for signing speed are larger, they are not what is unexpected here. The choice for the square tiebreaker was intended to improve verification speed at the cost of signing speed. As it turns out that it doesn't actually benefit verification speed, this is a bad trade-off.

# 3. How big a change is it

* In the BIP:
- Changing both invocations of `has_square_y` to `has_even_y`.
- Changing the `lift_x_square_y` invocation to `lift_x_even_y`.
- Applying the same change to the test vector generation code, and the resulting test vectors.
* In the libsecp256k1:
- An 8-line patch to the proposed BIP340 implementation[7]: see [8]
* In Bitcoin Core:
- Similarly small changes to the Python test reimplementation[9]
* Duplicating these changes in other draft implementations that may already exist.
* Review for all the above.

# 4. Conclusion

We discovered that the justification for using squaredness tiebreakers in BIP340 is based on a misunderstanding, and recent developments show that it may in fact be a somewhat worse choice than the alternative. It is a relatively simple change to address this, but that has be weighed against the impact of changing the standard at this stage.

Thoughts?

# 5. References

[1] https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki#design
[2] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017639.html
[3] https://en.wikipedia.org/wiki/Extended_Euclidean_algorithm
[4] https://gcd.cr.yp.to/safegcd-20190413.pdf
[5] https://github.com/bitcoin-core/secp256k1/pull/767
[6] https://github.com/bitcoin-core/secp256k1/pull/797
[7] https://github.com/bitcoin-core/secp256k1/pull/558
[8] https://github.com/sipa/secp256k1/commit/822311ca230a48d2c373f3e48b91b2a59e1371d6
[9] https://github.com/bitcoin/bitcoin/pull/17977

Cheers,

--
Pieter

📅 Original date posted:2020-03-03 📝 Original message:Hi ...

2023-06-07T18:23:11Z

In reply to nevent1q…2ja7
_________________________

📅 Original date posted:2020-03-03
📝 Original message:Hi all,

Given the recent activity and attention [1,2] around anti-covert channel
signing schemes, I decided to create this overview of the various techniques
that I know of, their trade-offs, and the various issues they protect against.
Most of this is based on various schemes by a number of authors, and credit
goes to them. I'm putting this together into something hopefully more
comprehensive, but mistakes and omissions in this writeup are likely mine.

I don't believe we have security proofs for any of the schemes, or for any of
the claims I make about the mitigation techniques below. I hope that having
all properties written up in one place makes it easier to eventually get those.

1) Security model
-----------------

When talking about signing with covert channels, we consider 3 parties:
* HW, the hardware wallet (or other offline signing device) with secret data
(a private key, or a seed from which the private key is derived).
* SW, the software wallet (or whatever communicates with HW and the network).
* OO, the outside observer who is trying to learn information about HW's
secret data.

We consider two distinct attack models:
* MSW, "malicious software wallet", but with honest HW. OO and SW are the
same party here, so this models the usual scenarios hardware wallets are
designed for, including side-channel attacks if those are considered to be
part of the threat model.
* MHW, "malicious hardware wallet", but with honest SW and no malicious party
being able to communicate with HW directly. OO and HW may have shared secret
information that SW (or anyone else) is unaware of. SW's job is trying to
prevent HW from using this to leak any information about its secret.

When both the HW and the SW are compromised, clearly no security is possible,
as all entities are controlled by the same party in that case.

In case HW uses a deterministic algorithm, it is possible to protect against
the MHW case by spot checking HW's behavior, by using an externally known
secret/seed. However, we'd like to have better than just spot checking
security, and for protection against side-channel attacks we may want
something that keeps working even when randomness is used by HW.

To keep the scope limited, we assume SW has no secret key of their own. This
rules out solutions like using 2-of-2 MuSig between HW and SW, which work, but
come with their own complications (like needing secure storage for that
secret).

2) Issues and solutions
-----------------------

In this section I will go over the various issues that exist in the MHW and MSW
models, the known mitigation techniques, and the resulting schemes.

I'm assuming a Schnorr-like signature protocol, though everything should apply
equally to ECDSA, and to a lesser extent probably also to multisignature
schemes. These variable names are used:
* H is a hash function.
* G is the curve generator.
* m is the message to be signed, known to and agreed upon by SW and HW.
* d is HW's secret key, with corresponding public key Q=dG.
* k is the secret nonce k, with corresponding public nonce R=kG.

The simplest protocol is naive Schnorr with deterministic nonce generation,
where SW only verifies that a signature created by HW is valid:

[Scheme 1: deterministic nonce, no tweak]
* SW requests a signature by sending (Q,m) to HW.
* HW computes k=H(d,m), R=kG, s=k+H(R,Q,m)d, and sends (R,s) to SW.
* SW verifies sG=R+H(R,Q,m)Q, and publishes sig (R,s) in case of success.

2.a) Predictable k value

There is a simple attack against Scheme 1 that will leak the entire private
key undetectably using a single signature, under MHW. Assume HW and OO both
have access to a shared secret a, unknown to anyone else. HW computes
k=H(a,Q,m) instead, which SW cannot distinguish from the honest k=H(d,m) as it
knows neither a or d. OO can compute k using the same formula, and thus
recover the private key as d=(s-k)/H(R,Q,m).

The general strategy to avoid this is by letting SW provide entropy that is
included into the nonce computation. A very naive (and ineffective) way of
doing that would be:
* SW generates random t, and requests a signature by sending (Q,m,t) to HW.
* HW computes k0=H(d,m,t), R0=k0G, k=k0+t, R=kG, s=k+H(R,Q,m)d, and sends
(R0,R,s) to SW.
* SW verifies sG=R+H(R,Q,m)Q, R=R0+tG, and publishes sig (R,s) if all is good.

This does not help as HW can still choose k directly, and retroactively
compute R0 as R-tG, satsifying SW's requirements. To address that, there are
two options:
* Turning R into a binding commitment to R0 and t (see Scheme 2).
* Only revealing t after HW has revealed their R0 (see Scheme 3).

The first approach is based on making R a commitment to R0 and t using
R=R0+H(R0,t)G. When applied to public keys this is known as pay-to-contract
(and is the basis for Taproot); when applied to the R point in signatures it's
known as sign-to-contract [3]. These are generally useful approaches to make
public keys and signatures commit to/timestamp external data, but using this
to protect against covert channels in signatures was first discussed by Greg
Maxwell [4]:

[Scheme 2: deterministic nonce, S2C tweak]
* SW generates random t, and requests a signature by sending (Q,m,t) to HW.
* HW computes k0=H(d,m,t), R0=k0G, k=k0+H(R0,t), R=kG,
s=k+H(R,Q,m)d, and sends (R0,R,s) to SW.
* SW verifies sG=R+H(R,Q,m)Q, R=R0+H(R0,t)G, and publishes sig (R,s) if all
is good.

The second approach is adding a round, and only revealing the tweak t after HW
has revealed their R0:

[Scheme 3: deterministic nonce, tweak revealed after nonce]
* SW requests a signature by sending (Q,m) to HW.
* HW computes k0=H(d,m), R0=k0G, and sends R0 to SW.
* SW generates a random t, and sends it to HW.
* HW computes k=k0+t, R=kG, s=k+H(R,Q,m)d, and sends (R,s) to SW.
* SW verifies sG=R+H(R,Q,m)Q, R=R0+tG, and publishes (R,s) if all is good.

2.b) Replay attacks

Scheme 3 introduces another problem however, this time under MSW. SW can ask
HW to sign the same message twice, but then pick distinct values for t (t and
t'). The resulting R points will be R=(k0+t)G and R'=(k0+t')G, and the s
values will be s=k0+t+H(R,Q,m)d and s'=k0+t'+H(R',Q,m)d. This allows SW to
compute d=(s'-t'-s+t)/(H(R',Q,m)-H(R,Q,m)). A similar problem would also exist
in Scheme 2 if t wasn't included in the formula for k0.

The problem is that SW is allowed to change their tweak while the nonce
only undergoes a linear function known to SW. There are again two ways to
address this problem:
* Making k0 generation non-repeating by including a counter or randomness
in it (Scheme 4).
* Making SW commit to their tweak before revealing it as well (Scheme 5).

[Scheme 4: counter/random nonce, tweak revealed after nonce]
* SW requests a signature by sending (Q,m) to HW.
* HW uses a global counter c, or fresh randomness b, and computes k0=H(d,m,c)
or k0=H(d,m,b), R0=k0G, and sends R0 to SW.
* SW generates a random t, and sends it to HW.
* HW computes k=k0+t, R=kG, s=k+H(R,Q,m)d, and sends (R,s) to SW.
* SW verifies sG=R+H(R,Q,m)Q, R=R0+tH, and publishes (R,s) if all is good.

A variant of Scheme 4, but with multiplicative tweak rather than additive,
and only revealing H(R0) rather than R0 immediately, was suggested by Sergio
Demian Lerner in [5].

[Scheme 5: deterministic nonce, precommited tweak revealed after nonce]
* SW generates a random t, computes h=H(t), and requests a signature by
sending (Q,m,h) to HW.
* HW computes k0=H(d,m,h), R0=k0G, and sends R0 to SW.
* SW sends t to HW.
* HW verifies h=H(t), and if so, computes k=k0+t, R=kG, s=k+H(R,Q,m)d, and
sends (R,s) to SW.
* SW verifies sG=R+H(R,Q,m)Q, R=R0+tG, and publishes (R,s) if all is good.

Scheme 5 is the one suggested by Stepan Snigirev in [2,6]. A variant with
S2C tweaking instead of additive tweaked was suggested by Andrew Poelstra
in his talk [7], with transcript by Bryan Bishop in [8].

2.c) k0 grinding

So far Schemes 2, 4, and 5 protect against predictable k values and replay
attacks. Predictable k values are however not the only way MWH can leak
secrets.

If we imagine HW has significant computational power, in Scheme 2 it can try
many different k0 values (by deviating from k0=H(d,m,t)), and observe what the
resulting (R,s) signature will be. For example, by iterating on average 256
times, it can choose 8 bits in (R,s) that convey information about k, d, or
whatever seed or master key they are derived from. Using forward error
correction (FEC) schemes, this channel of a few bits per signature may be
enough to leak an entire seed over enough signatures. Using a shared secret a
between HW and OO those bits can again be made undetectable to anyone else.

Schemes 4 and 5 are not vulnerable to this problem, as they force HW to commit
to its R0 before knowing the resulting R. One might think that Scheme 4 merely
shifts this problem to MSW, where SW can grind t to make R biased in a similar
way. We assumed that SW does not have access to any secrets however, so this
is harmless.

2.d) Statefulness

We're left with Schemes 4 and 5 that protect against all listed issues. Both
need two interaction rounds, with state that needs to be kept by HW between
the rounds (the k0 value). While not a problem in theory, this may be hard to
implement safely in simple APIs.

One possibility is sticking with our "best one-round" Scheme 2, and accepting
that that implies the k0 grinding vulnerability.

There is another possibility, namely splitting Scheme 5 into two independent
interactions with HW, where no memory between them is needed on the HW side:

[Scheme 6: deterministic nonce, precommitted tweak revealed separately]
First interaction:
* SW generates a random t, computes h=H(t), and requests the R0 point that HW
would use by sending (Q,m,h) to HW.
* HW computes k0=H(d,m,h), R0=k0G, and sends R0 to SW.
Second interaction:
* SW requests a signature by sending (Q,m,t) to HW
* HW computes k0=H(d,m,H(t)), k=k0+t, R0=R0k, R=kG, s=k+H(R,Q,m)d, and sends
(R0,R,s) to SW.
* SW verifies that R0 matches the earlier R0 it received, and that
sG=R+H(R,Q,m)Q, R=R0+tG, and publishes (R,s) if all is good.

A variant of Scheme 6, with S2C tweaking instead of additive tweaking, is what
is being worked on by Jonas Nick [9] and Marko Bencun [10] for the
libsecp256k1 library.

The same technique cannot be applied to Scheme 4, as HW inherently needs state
to keep the counter c, or to remember the randomness b between interactions
there.

2.e) Failure bias

There is yet another, and even weaker, leak that is available in MHW: whenever
HW learns what the eventual signature (R,s) will be, it could pretend to fail
and go offline, or return some kind of error. In theory, this is enough to
introduce a similar bias, though it would come at possibly enormous failure
rates. If HW is allowed to fail 255 times out of 256, it can introduce an
8-bit bias, and employ similar techniques (FEC and shared HW/OO secret).
The obvious solution is showing a big warning to the user whenever any kind of
failure occurs (including device going offline, or returning invalid
responses) that the device is failing, and that if this happens frequently, it
should be treated as malicious.

Interestingly, Scheme 6 can be adapted to reduce this (already very weak)
channel further. The observation is that HW cannot predict during the first
interaction what (R,s) is going to be, as t is not known. This means it can
only fail during the second interaction when the result is already committed
to. Thus, if failure occurs during the second interaction, SW can simply
retry it with the same t value. If that succeeds, either a glitch occurred and
was safely retried, or the device's attempt to bias was prevented. If the
failure persists, the user should still be warned - as restarting with a
different k would reintroduce the possibility for bias.

2.f) Side-channel attacks

As a last consideration, let's see if these schemes have an impact on
potential resilience against side-channel attacks. I say potential, because
these classes of attacks are in general hard to protect against and model,
as they depend on implementation details and hardware protections. Still,
there are some general observations possible.

A significant amount of time in HW is likely spent on the EC multiplications
to obtain R0 and R from k0 and k. As s=k+H(R,Q,m)d, an variation of the replay
attack is possible here. In schemes with deterministic nonces, SW can ask for
the same signature twice, but use a fault injection to hopefully (only) cause
an error in R, R'. This would reveal (R,s) and (R',s') to SW, where s' is
k+H(R',Q,m)d, which would let them compute d=(s'-s)/(H(R',Q,m)-H(R,q,m)).
There is an easy solution against this, namely verifying the final signature
(R,s) in HW before sending it to SW, as almost certainly the result of such
a fault would not result in a valid signature. This comes at an extra
computational cost, though.

For other side-channel attacks like different power analysis, research [11]
shows that introducing fresh randomness in the right place may be helpful.
This approach is called "synthetic nonces" [12]. Unfortunately usage of these
rules out the deterministic approach from Scheme 6. A variant of Scheme 5
with fresh randomness in the k0 computation can be used, though.

3) Summary
----------

Six different issues of various levels of severity were discussed:
(a) Predictable k: (MHW) a single signature leaks the entire private key.
(b) Replay attacks: (MSW) a single signature leaks the entire private key.
(c) k0 grinding: (MHW) the HW can leak n bits with 2^n work.
(d) Statefulness: HW has to correctly maintain state, complicating things.
(e) Failure bias: (MHW) a selective failure rate of (2^n-1)/2^n can be used
to leak n bits of secret per signature.
(f) Side-channel attacks: (MSW) physical access to HW can help extracting
secrets.

It seems any reasonable solution should at least protect against (a), (b), and
(c), but it seems no solution can be optimal for all of (d), (e), and (f) too.

If statelessness and protection against failure bias are prioritized, Scheme 6
seems best. Its additive tweaking can be replaced with S2C (or multiplicative)
tweaking too. S2C in particular may be desirable to unify with support for
S2C-based timestamping.

If resistance against side-channels is prioritized, solutions with synthetic
nonces seem best; either Scheme 4, or Scheme 5 with randomness added to the
k0 computation. Again, any tweaking approach can be chosen.

If the 2-round approaches for Schemes 4, 5, and 6 are really unacceptable,
Scheme 2 (with S2C tweaking) could be used, but in that case protection
against k0 grinding is reduced to spot checking. If randomness is additionally
added for side-channel resistance, the ability to spot check disappears
entirely.

4) References
-------------

[1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017649.html
[2] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017655.html
[3] https://blog.eternitywall.com/2018/04/13/sign-to-contract/
[4] https://bitcointalk.org/index.php?topic=893898.msg9861102#msg9861102
[5] https://bitcointalk.org/index.php?topic=893898.msg9841502#msg9841502
[6] https://medium.com/cryptoadvance/hardware-wallets-can-be-hacked-but-this-is-fine-a6156bbd199
[7] https://youtu.be/j9Wvz7zI_Ac?t=640
[8] https://diyhpl.us/wiki/transcripts/sf-bitcoin-meetup/2019-02-04-threshold-signatures-and-accountability/
[9] https://github.com/bitcoin-core/secp256k1/pull/590
[10] https://github.com/bitcoin-core/secp256k1/pull/669
[11] https://eprint.iacr.org/2017/985
[12] https://moderncrypto.org/mail-archive/curves/2017/000925.html

📅 Original date posted:2019-11-10 📝 Original message:On ...

2023-06-07T18:21:36Z

In reply to nevent1q…klep
_________________________

📅 Original date posted:2019-11-10
📝 Original message:On Thu, Nov 7, 2019, 18:16 David A. Harding <dave at dtrt.org> wrote:

> On Thu, Nov 07, 2019 at 02:35:42PM -0800, Pieter Wuille via bitcoin-dev
> wrote:
> > In the current draft, witness v1 outputs of length other
> > than 32 remain unencumbered, which means that for now such an
> > insertion or erasure would result in an output that can be spent by
> > anyone. If that is considered unacceptable, it could be prevented by
> > for example outlawing v1 witness outputs of length 31 and 33.
>
> Either a consensus rule or a standardness rule[1] would require anyone
> using a bech32 library supporting v1+ segwit to upgrade their library.
> Otherwise, users of old libraries will still attempt to pay v1 witness
> outputs of length 31 or 33, causing their transactions to get rejected
> by newer nodes or get stuck on older nodes. This is basically the
> problem #15846[2] was meant to prevent.
>
> If we're going to need everyone to upgrade their bech32 libraries
> anyway, I think it's probably best that the problem is fixed in the
> bech32 algorithm rather than at the consensus/standardness layer.
>

Admittedly, this affecting development of consensus or standardness rules
would feel unnatural. In addition, it also has the potential downside of
breaking batched transactions in some settings (ask an exchange for a
withdrawal to a invalid/nonstandard version, which they batch with other
outputs that then get stuck because the transaction does not go through).

So, Ideally this is indeed solved entirely on the bech32/address encoding
side of things. I did not initially expect the discussion here to go in
that direction, as that could come with all problems that rolling out a new
address scheme in the first place has. However, there may be a way to
mostly avoid those problems for the time being, while also not having any
impact on consensus or standardness rules.

I believe that most new witness programs we'd want to introduce anyway will
be 32 bytes in the future, if the option exists. It's enough for a 256-bit
hash (which has up to 128-bit collision security, and more than 128 bits is
hard to achieve in Bitcoin anyway), or for X coordinates directly. Either
of those, plus a small version number to indicate the commitment structure
should be enough to encode any spendability condition we'd want with any
achievable security level.

With that observation, I propose the following. We amend BIP173 to be
restricted to witness programs of length 20 or 32 (but still support
versions other than 0). This seems like it may be sufficient for several
years, until version numbers run out. I believe that some wallet
implementations already restrict sending to known versions only, which
means effectively no change for them in addition to normal deployment.

In the mean time we develop a variant of bech32 with better
insertion/erasure detecting properties, which will be used for witness
programs of length different from 20 or 32. If we make sure that there are
never two distinct valid checksum algorithms for the same output, I don't
believe there is any need for a new address scheme or a different HRP. The
latter is something I'd strongly try to avoid anyway, as it would mean
additional cognitive load on users because of another visually distinct
address style, plus more logistical overhead (coordination and keeping
track of 2 HRPs per chain).

I believe improving bech32 itself is preferable over changing the way
segwit addresses use bech32, as that can be done without making addresses
even longer. Furthermore, the root of the issue is in bech32, and it is
simplest to fix things there. The easiest solution is to simply change the
constant 1 that is xor'ed into the checksum before encoding it to a 30-bit
number. This has the advantage that a single checksum is never valid for
both algoritgms simultaneously. Another approach is to implicitly including
the length into the checksummed data.

What do people think?

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20191110/5894b93f/attachment.html>;

📅 Original date posted:2019-11-07 📝 Original message:Hello ...

2023-06-07T18:21:35Z

In reply to nevent1q…mdtw
_________________________

📅 Original date posted:2019-11-07
📝 Original message:Hello all,

A while ago it was discovered that bech32 has a mutation weakness (see
https://github.com/sipa/bech32/issues/51 for details). Specifically,
when a bech32 string ends with a "p", inserting or erasing "q"s right
before that "p" does not invalidate it. While insertion/erasure
robustness was not an explicit goal (BCH codes in general only have
guarantees about substitution errors), this is very much not by
design, and this specific issue could have been made much less
impactful with a slightly different approach. I'm sorry it wasn't
caught earlier.

This has little effect on the security of P2WPKH/P2WSH addresses, as
those are only valid (per BIP173) for specific lengths (42 and 62
characters respectively). Inserting 20 consecutive "p"s in a typo
seems highly improbable.

I'm making this post because this property may unfortunately influence
design decisions around bip-taproot, as was brought up in the review
session (https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-October/017427.html)
past tuesday. In the current draft, witness v1 outputs of length other
than 32 remain unencumbered, which means that for now such an
insertion or erasure would result in an output that can be spent by
anyone. If that is considered unacceptable, it could be prevented by
for example outlawing v1 witness outputs of length 31 and 33.

Thoughts?

Cheers,

--
Pieter

📅 Original date posted:2019-05-26 📝 Original message:On ...

2023-06-07T18:18:23Z

In reply to nevent1q…uvpe
_________________________

📅 Original date posted:2019-05-26
📝 Original message:On Sun, May 26, 2019, 07:07 Aymeric Vitte via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> I realized recently that my segwit implementation was not correct,
> basically some time ago, wrongly reading the specs (and misleaded by
> what follows), I thought that scriptsig would go into witness data as it
> was, but that's not the case, op_pushdata is replaced by varlen
>
> Now reading correctly the specs, they seem to be not totally correct,
> then the first question is: why OP_0 is 00 in witness data and not 0100?
> Does this apply to other op_codes? This does not look logical at all
>
> The second question is: why for non segwit inputs there is a 00 length
> in segwit data, what is the rational for that? It should just be nothing
> since you don't need this to reconciliate things
>

This is a question that belongs on https://bitcoin.stackexchange.com, not
this list.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190526/7df58353/attachment.html>;

📅 Original date posted:2019-05-23 📝 Original message:On ...

2023-06-07T18:18:14Z

In reply to nevent1q…j8zz
_________________________

📅 Original date posted:2019-05-23
📝 Original message:On Thu, 23 May 2019 at 11:33, Tamas Blummer via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
>
> Difficulty change has profound impact on miner’s production thereby introduce the biggest risk while considering an investment.
> Commodity markets offer futures and options to hedge risks on traditional trading venues. Some might soon list difficulty futures.
>
> I think we could do much better than them natively within Bitcoin.
>
> A better solution could be a transaction that uses nLocktime denominated in block height, such that it is valid after the difficulty adjusted block in the future.
> A new OP_DIFFICULTY opcode would put onto stack the value of difficulty for the block the transaction is included into.
> The output script may then decide comparing that value with a strike which key can spend it.
> The input of the transaction would be a multi-sig escrow of those who entered the bet.
> The winner would broadcast.

If the difficulty can be directly observed by the script language, you
would need to re-evaluate all scripts in unconfirmed transactions
whenever the difficulty changes. This complicates implementation of
mempools, but it also makes reasoning about validity of (chains of)
unconfirmed transactions harder, as an unconfirmed predecessor may
have conditions that change over time.

For things like block time/height, this is solved by not having the
script itself observe the context state directly, but instead having
an assertion on the state outside of script (nLockTime for absolute
time/height and nSequence for relative), and then having opcodes
inside script that observe the assertion (OP_CLTV and OP_CSV). By
doing so, script validity is a single context-free yes or not that can
be evaluated once, and the variable part is just transaction-level
reasoning that doesn't involve a full script interpreter.
Additionally, the supported assertions are restricted in such a way
that if they are true within a particular block, they're also true in
any descendant, removing the complexity of reasoning about validity
(apart from the inevitable reasoning about possible double-spend
before confirmation).

I feel a similar construction is needed for observing block
difficulty. This can be done by either having an opcode that as a side
effect of execution "posts" an assertion (e.g. "difficulty at block
height X is at least Y"), instead of putting the difficulty on the
stack. An alternative is having the assertion be part of the
transaction structure (for example in the annex we propose in
bip-taproot), and having an opcode that observes the difficulty
assertion inside script.

I don't have a strong opinion either way on the usefulness of having
difficulty-dependent transaction/scripts.

Cheers,

--
Pieter

📅 Original date posted:2019-05-22 📝 Original message:On ...

2023-06-07T18:18:02Z

In reply to nevent1q…r4du
_________________________

📅 Original date posted:2019-05-22
📝 Original message:On Tue, 21 May 2019 at 10:20, Russell O'Connor <roconnor at blockstream.io> wrote:
>
> Regarding Tapscript, the specification calls for the final value of the stack being a single non-false value:
>
>> The tapscript is executed according to the rules in the following section, with the initial stack as input
>> II. If the execution results in anything but exactly one element on the stack which evaluates to true with CastToBool(), fail.
>
> Perhaps it is worth taking this opportunity here to remove a minor wart of the Script language and instead require the stack to be exactly empty upon completion.
>
> In addition to removing a potential malleability vector, I expect it would simplify development of Bitcoin Script. A rule requiring an empty stack means that the conjunction (logical and) of two policies can be implemented by the simple concatenation of Bitcoin Scripts. This combined with the taproot ability to form the disjunction (logical or) of policies by having multiple Merkle branches, means that the translation of a policy written in disjunctive normal form (the logical ors of logical ands of primitive policies) can be straightforwardly translated to a taproot of tapscript.
>
> That said, I think the developers of miniscript <http://bitcoin.sipa.be/miniscript/miniscript.html>; are in a much better position to comment on whether my above intuition is correct given that they've had to implement a host of various calling conventions. I understand that at least some of this complexity is due to Bitcoin Script's one element stack rule.

IIRC I looked into this a few months ago, and found that the spending
cost (script size + expected witness size) of the optimal script for
every Miniscript policy at most changes by 1 WU (in either direction)
by requiring an empty stack rather than a true value, though in a
(admittedly very arbitrarily biased) distribution, more policies were
improved by it than degraded. This is not taking Taproot into account
(as many of those policies in a Taproot-supporting world should
optimally make use of the internal key and Merkle tree, rather than
turning everything into a monolithic script). I expect that this may
make the impact somewhat larger, but still never more than a 1 WU
gain.

I don't think the spending cost changes justify this change, so the
remaining criteria are complexity ones. In my view, the main benefit
would be to authors of hand-written scripts where the consistency
benefits matter, but this needs to be weighed against the mental cost
of learning the new semantics. For Miniscript itself, this doesn't
make much difference - the top level calling convention would become
'V' instead of 'T', but 'T' would still exist as a calling convention
that may be used internally; it's a few lines change.

So overall this feels like something with marginal costs, but also at
most marginal benefits. Perhaps other people have stronger opinions.

> Even if we choose not to implement the empty stack rule, we should at least require that the last element be 0x01 to remove a potential malleability vector and bring it in line with MINIMAL_IF semantics.

This feels like the right thing to do; as we're making MINIMALIF part
of consensus for Tapscript it would make sense to apply the same rule
to the "return" value of the script. There is a downside though,
namely that in some places where you'd use "<n>
OP_CHECKSEQUENCEVERIFY" or "<n> OP_CHECKLOCKTIMEVERIFY" you now need
to add an additional OP_0NOTEQUAL to convert the left-over element n
into an exact 0x01. I also can't come up with any practical benefits
that this would have; if the top stack element in a particular code
path comes directly from the input, it's insecure regardless; if there
isn't, it'll generally be a a boolean (or an intentional non-boolean
true value) computed by the script.

On Tue, 21 May 2019 at 13:05, John Newbery <john at johnnewbery.com> wrote:
>
> Hi,
>
> > A Taproot output is a SegWit output [...] with
> > version number 1, and a 33-byte witness program whose first byte is 0 or 1.
>
> Given a secret key k and public key P=(x,y), a signer with the knowledge of k
> can sign for -P=(x,p-y) since -k is the secret key for that point. Encoding the
> y value of the public key therefore adds no security.

That's a good point; without security benefit there's no reason to
make pay-to-taproots more expensive. Making them the same cost as
P2WSH is nice in any case.

> As an alternative to
> providing the y value of the taproot output key Q when constructing the taproot
> output, the signer can provide it when signing. We can also restrict the y value
> of the internal key P to be even (or high, or a quadratic residue). That gives
> us 4 options for how to set the y signs for P and Q.
>
> 1. Q sign is explictly set in the witness program, P sign is explicitly set in the control block
> => witness program is 33 bytes, 32 possible leaf versions (one for each pair of 0xc0..0xff)
> 2. Q sign is explictly set in the witness program, P sign is implicitly even
> => witness program is 33 bytes, 64 possible leaf versions (one for each 0xc0..0xff)
> 3. Q sign is explictly set in the control block, P sign is explicitly set in the control block
> => witness program is 32 bytes, 16 possible leaf versions (one for each 4-tuple of 0xc0..0xff)
> 4. Q sign is explictly set in the control block, P sign is implicitly even
> => witness program is 32 bytes, 32 possible leaf versions (one for pair of 0xc0..0xff)
>
> The current proposal uses (1). Using (3) or (4) would reduce the size of a
> taproot output by one byte to be the same size as a P2WSH output. That means
> that it's not more expensive for senders compared to sending to P2WSH.

I prefer (4). There is a slight complexity in needing a conditional
sign swap when signing (to make sure the corresponding key is even),
but I think it's minimal compared to the other changes needed here
already. I'll try to amend the reference code soon to see what impact
this idea has.

> > (native or P2SH-nested, see BIP141)
>
> I'd prefer to not support P2SH-nested TR. P2SH wrapping was useful for segwit
> v0 for compatibility reasons. Most wallets/exchanges/services now support sending
> to native segwit addresses (https://en.bitcoin.it/wiki/Bech32_adoption) and that
> will be even more true if Schnorr/Taproot activate in 12+ months time.

I'm not sure there is much to gain here. There is perhaps a minimal
fungibility improvement by not having another bit (P2SH or not) that
can leak some information about the software you're using. On the
other hand, until native taproot outputs are common, choosing P2SH
wrapped ones leak less information at output creation time. Apart from
that, I think it would only minimally impact implementation
complexity. Are there other advantages I'm missing?

Cheers,

--
Pieter

📅 Original date posted:2019-05-06 📝 Original message:Hello ...

2023-06-07T18:17:57Z

In reply to nevent1q…xwvk
_________________________

📅 Original date posted:2019-05-06
📝 Original message:Hello everyone,

Here are two BIP drafts that specify a proposal for a Taproot
softfork. A number of ideas are included:

* Taproot to make all outputs and cooperative spends indistinguishable
from eachother.
* Merkle branches to hide the unexecuted branches in scripts.
* Schnorr signatures enable wallet software to use key
aggregation/thresholds within one input.
* Improvements to the signature hashing algorithm (including signing
all input amounts).
* Replacing OP_CHECKMULTISIG(VERIFY) with OP_CHECKSIGADD, to support
batch validation.
* Tagged hashing for domain separation (avoiding issues like
CVE-2012-2459 in Merkle trees).
* Extensibility through leaf versions, OP_SUCCESS opcodes, and
upgradable pubkey types.

The BIP drafts can be found here:
* https://github.com/sipa/bips/blob/bip-schnorr/bip-taproot.mediawiki
specifies the transaction input spending rules.
* https://github.com/sipa/bips/blob/bip-schnorr/bip-tapscript.mediawiki
specifies the changes to Script inside such spends.
* https://github.com/sipa/bips/blob/bip-schnorr/bip-schnorr.mediawiki
is the Schnorr signature proposal that was discussed earlier on this
list (See https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-July/016203.html)

An initial reference implementation of the consensus changes, plus
preliminary construction/signing tests in the Python framework can be
found on https://github.com/sipa/bitcoin/commits/taproot. All
together, excluding the Schnorr signature module in libsecp256k1, the
consensus changes are around 520 LoC.

While many other ideas exist, not everything is incorporated. This
includes several ideas that can be implemented separately without loss
of effectiveness. One such idea is a way to integrate SIGHASH_NOINPUT,
which we're working on as an independent proposal.

The document explains basic wallet operations, such as constructing
outputs and signing. However, a wide variety of more complex
constructions exist. Standardizing these is useful, but out of scope
for now. It is likely also desirable to define extensions to PSBT
(BIP174) for interacting with Taproot. That too is not included here.

Cheers,

--
Pieter

📅 Original date posted:2018-11-28 📝 Original message:On ...

2023-06-07T18:15:20Z

In reply to nevent1q…4yvq
_________________________

📅 Original date posted:2018-11-28
📝 Original message:On Mon, 19 Nov 2018 at 14:37, Pieter Wuille <pieter.wuille at gmail.com> wrote:
> Here is a combined proposal:
> * Three new sighash flags are added: SIGHASH_NOINPUT, SIGHASH_NOFEE, and SIGHASH_SCRIPTMASK.
> * A new opcode OP_MASK is added, which acts as a NOP during execution.
> * The sighash is computed like in BIP143, but:
> * If SIGHASH_SCRIPTMASK is present, for every OP_MASK in scriptCode the subsequent opcode/push is removed.
> * The scriptPubKey being spent is added to the sighash, unless SIGHASH_SCRIPTMASK is set.
> * The transaction fee is added to the sighash, unless SIGHASH_NOFEE is set.
> * hashPrevouts, hashSequence, and outpoint are set to null when SIGHASH_NOINPUT is set (like BIP118, but not for scriptCode).

Thanks for all the input so far. Going over the suggestions and other ideas:

* OP_MASK should be required to be followed by a push, as suggested by
Anthony Towns. The alternative would permit substituting arbitrary
opcodes for masked pushes, which is at least very hard to reason
about. This would effectively turn it into a multi-byte OP_MASKEDPUSH
opcode.

* It's probably better to sign the amounts of all inputs, as suggested
by Johnson Lau. As that would cause default sighashes to sign all
input and output amounts, is there still a need to sign the tx fee
explicitly? Or in other words, are there situations where changing the
set of inputs or outputs after signing is desired, but the net
difference between them cannot change? If not, that would remove the
need for NOFEE.

* Do we need to keep the rule that sequence values of other inputs are
only signed with default sighash? It feels cleaner to always sign the
sequence values of all inputs that are included in the sighash anyway
(so all of them, unless ANYONECANPAY or NOINPUT, which would make it
sign only the current input's sequence value). If NOINPUT also blanks
the sequence values (as currently specified by BIP118), and all input
amounts are signed, that would make amounts/sequence values always be
treated identically.

* If MASK implies NOINPUT, and NOINPUT implies ANYONECANPAY, the 3 of
them can be encoded in just 2 bits using the
PARTIALSCRIPT/KNOWNSCRIPT/KNOWNTX/ALL_INPUTS encoding Anthony Towns
suggested.

* Regarding the discussion about preventing signatures from being
rebound to a different script(path)/checksig:
* With MAST there is indeed less need for this, but at least
single-tree MAST constructions cannot replace all script branches (a
script with 40 IF/THEN/ELSE constructions may have 2^40 different
execution paths, for which computing a Merkle tree is intractable).
* Just signing the opcode position of the CHECKSIG operator isn't
enough for all cases either. For example, you could have a complex
nested set of branches that puts a number of pubkeys on the stack, and
then a CHECKMULTISIG after the last ENDIF to verify all of them. In
such a situation, if the same key can occur in multiple combinations,
you still may want to prevent a signature generated for one
combination from being rebindable to the same key in another
combination. I believe that signing the opcode position plus the
true/false condition of all previous(?) IF statements is probably
sufficient to achieve that, but it would also introduce unnecessary
complexity for signers in most cases (see next point).
* Thinking about signing code, adding these sort of execution trace
commitments to the sighash means they need to know which checksig
operator etc. they are signing for. I believe that in practice for
example HW devices will just whatever position the wallet indicated,
rather than verifying it corresponds with a particular intended code
path. Preventing rebinding isn't very useful if an attacker can make
you bind to the wrong thing regardless, so I'm not convinced this is
even worth having by default.
* An alternative (not sure who suggested it) is to simply make every
CHECKSIG sign the opcode position of the last executed CODESEPARATOR
(and remove the earlier cut-of-scriptCode effect of CODESEPARATOR).
This gives a simple (but somewhat limited) way for scripts that need
to prevent certain kinds of cross-execution-trace rebinding.

A few misc ideas:
* (Taken from https://github.com/jl2012/bips/blob/sighash2/bip-sighash2.mediawiki)
For a default sign-everything sighash, the sighash byte can be
dropped.
* For the commitments to the scriptPubKey and scriptCode, an
intermediary hash should be used (so the data included in the sighash
includes a hash of those, rather than the script directly). This
prevents a blow up in hashing time for large scripts with many
different sighash types in its signatures.
* When masking the scriptCode, the push opcode immediately following
OP_MASKEDPUSH can be replaced by OP_VERIF (which will never collide
with any real script, as OP_VERIF makes a script invalid even when
occurring in an unexecuted branch).
* Sighashes (and really all new hashes that are introduced) should be
prefixed with a fixed 64-byte array as "tag", chosen to not collide
with any existing use of SHA256 in Bitcoin, to prevent signatures from
being re-interpretable as something else. Picking 64 bytes as tag size
means it can be efficiently implemented as just a modified SHA256 IV.

So a combined proposal:
* All existing sighash flags, plus NOINPUT and MASK
(ANYONECANPAY/NOINPUT/MASK are encoded in 2 bits).
* A new opcode called OP_MASKEDPUSH, whose only runtime behaviour is
failing if not immediately followed by a push, or when appearing as
last opcode in the script.
* Signatures are 64 plus an optional sighash byte. A missing sighash
byte implies ALL, and ALL cannot be specified explicitly.
* The sighash is computed from the following:
* A 64-byte constant tag
* Data about the spending transaction:
* The transaction version number
* The hash of txins' prevouts+amounts+sequences (or nothing if ANYONECANPAY)
* The hash of all txouts (or just the corresponding txout if
SINGLE; nothing if NONE)
* The transaction locktime
* Data about the output being spent:
* The prevout (or nothing if NOINPUT)
* The amount
* The sequence number
* The hash of the scriptPubKey (or nothing if MASK)
* Data about the script being executed:
* The hash of the scriptCode (after masking out, if MASK is set)
* The opcode number of the last executed OP_CODESEPARATOR (or
0xFFFFFFFF if none)
* The sighash mode

Cheers,

--
Pieter

📅 Original date posted:2018-11-19 📝 Original message:Hello ...

2023-06-07T18:15:15Z

In reply to nevent1q…kudz
_________________________

📅 Original date posted:2018-11-19
📝 Original message:Hello everyone,

For future segwit versions, I think it would be good add a few things
to the sighash by default that were overlooked in BIP143:
* Committing to the absolute transaction fee (in addition to just the
amount being spent in each input) would categorically remove concerns
about wallets lying about fees to HW devices or airgapped signers.
* Committing to the scriptPubKey (in addition to the scriptCode) would
prevent lying to devices about the type of output being spent, even
when the scriptCode is correct. As a reminder, the scriptCode is the
actually executed script (which is the redeemscript in non-segwit
P2SH, and the witnesscript in P2WSH/P2WPKH).

As this implies additional information that may not be desirable to
commit to in all circumstances, it makes sense to make these optional.
This obviously interacts with SIGHASH_NOINPUT, which really adds two
different ways of rebinding signatures to inputs:
* Changing the prevout (so that the txid doesn't need to be known when
the signature is created)
* Changing the script (so that the exact scriptPubKey/redeemScript/...
doesn't need to be known when the signature is created)

Of course, the second implies the first, but do all use cases require
both being able to change the prevout and (arbitrarily) changing the
scriptPubKey? While BIP118 correctly points out this is secure if the
same keys are only used in scripts with which binding is to be
permitted, I feel it would be preferable if signatures/scripts would
explicitly state what can change. One way to accomplish this is by
indicating exactly what in a script is subject to change.

Here is a combined proposal:
* Three new sighash flags are added: SIGHASH_NOINPUT, SIGHASH_NOFEE,
and SIGHASH_SCRIPTMASK.
* A new opcode OP_MASK is added, which acts as a NOP during execution.
* The sighash is computed like in BIP143, but:
* If SIGHASH_SCRIPTMASK is present, for every OP_MASK in scriptCode
the subsequent opcode/push is removed.
* The scriptPubKey being spent is added to the sighash, unless
SIGHASH_SCRIPTMASK is set.
* The transaction fee is added to the sighash, unless SIGHASH_NOFEE is set.
* hashPrevouts, hashSequence, and outpoint are set to null when
SIGHASH_NOINPUT is set (like BIP118, but not for scriptCode).

So my question is whether anyone can see ways in which this introduces
redundant flexibility, or misses obvious use cases?

Cheers,

--
Pieter

📅 Original date posted:2018-07-08 📝 Original message:On ...

2023-06-07T18:13:41Z

In reply to nevent1q…5ql6
_________________________

📅 Original date posted:2018-07-08
📝 Original message:On Sun, Jul 8, 2018, 07:26 Erik Aronesty via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> To save space, start with the wiki terminology on schnorr sigs.
>
> Consider changing the "e" term in the schnorr algorithm to hash of message
> (elligator style) to the power of r, rather than using concatenation.
>

This is a very vague description. Is there some paper you can reference, or
a more detailed explanation of the algorithm?

This would allow m of n devices to sign a transaction without any of them
> knowing a private key at all.
>
IE: each device can roll a random number as a share and the interpolation
> of that is the private key.
>
> The public shares can be broadcast and combines. And signature shares can
> be broadcast and combined.
>
> The net result of this is it really possible for an arbitrary set of
> devices to create a perfectly secure public-private key pair set.
>
At no point was the private key anywhere.
>

All of this sounds like a threshold signature scheme, which as Tim pointed
out is already possible with Schnorr.

What are the advantages of what you're describing?

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180708/396b25b7/attachment.html>;

📅 Original date posted:2018-07-08 📝 Original message:On ...

2023-06-07T18:13:40Z

In reply to nevent1q…jnhy
_________________________

📅 Original date posted:2018-07-08
📝 Original message:On Sun, Jul 8, 2018, 19:23 Erik Aronesty via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Pretty sure these non interactive sigs are more secure.
>

Schnorr signatures are provably secure in the random oracle model assuming
the discrete logarithm problem is hard in the used group.

What does "more secure" mean? Is your construction secure with weaker
assumptions?

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180708/4f9bcb27/attachment.html>;

📅 Original date posted:2018-07-11 📝 Original message:On ...

2023-06-07T18:13:33Z

In reply to nevent1q…shmk
_________________________

📅 Original date posted:2018-07-11
📝 Original message:On Tue, Jul 10, 2018 at 5:10 AM, matejcik <jan.matejek at satoshilabs.com> wrote:
> On 6.7.2018 00:06, Pieter Wuille wrote:> The only case where "malicious"
> conflicting values can occur is when
>> one of the Signers produces an invalid signature, or modifies any of
>> the other fields already present in the PSBT for consumption by
>> others. If this were an issue, it would be an issue regardless of the
>> Combiner's operation, as in topology A no Combiner is even present.
>> This is generally true I think - Combiners can always be replaced with
>> just a different (and possibly less parallel) topology of data flow.
>
> This is an interesting thesis, and also an unspoken assumption ISTM. It
> seems worth adding something like this to the spec:
> """
> In general, the result of Combiner combining two PSBTs from independent
> participants A and B should be functionally equivalent to a result
> obtained from processing the original PSBT by A and then B in a sequence.
> or, for participants performing fA(psbt) and fB(psbt):
> Combine(fA(psbt), fB(psbt)) == fA(fB(psbt)) == fB(fA(psbt))
> """

Adding that sounds like a good idea, indeed.

>> The bottom line is that a Combiner which picks arbitrarily in case of
>> conflicts will never end up with something worse than what you already
>> need to deal with. If you disregard the case of invalid fields
>> (because the result will just be an invalid transaction), then any
>> choice the Combiner makes is fine, because all the values it can pick
>> from are valid.
>
> This sounds reasonable and IMHO it would be good to have a summary of
> this argument in the Rationale section.

Sounds good.

>> If you're worried about attack surface, I don't believe rejecting
>> invalid fields ever matters. An attacker can always drop the fields
>> you don't understand before giving you the PSBT, making your behavior
>> identical to one where you'd have ignore those fields in the first
>> place.
>
> I'm reluctant to sign an input with unknown data, on the premise that there could be *anything* in that data

But the point is: you are not signing an input with unknown data. You
are signing your own interpretation (since you compute the sighash
yourself), which doesn't include what you don't understand. If that
interpretation doesn't match reality, the signature is at worst
useless. Who cares that someone added information about a transaction
that doesn't affect what you sign?

> We are most likely to implement the "do not sign with unknown fields"
> rule in any case (technically a whitelist of "known OK" field types),
> and resolve potential problems as they arise. I raised this point mainly
> because I think discussing this explicitly in the spec is beneficial: a
> distinction between mandatory and optional fields is one way, mentioning
> or prescribing possible signing strategies is another.

I don't think that's a particularly useful policy, but certainly
Signers are allowed to implement any policy they like about what they
accept in signing.

Cheers,

--
Pieter

📅 Original date posted:2018-07-05 📝 Original message:On ...

2023-06-07T18:13:33Z

In reply to nevent1q…h984
_________________________

📅 Original date posted:2018-07-05
📝 Original message:On Thu, Jul 5, 2018 at 4:52 AM, matejcik <jan.matejek at satoshilabs.com> wrote:
>> Allowing combiners to choose any value also allows for intelligent combiners to choose the
>> correct values in the case of conflicts. A smart combiner could, when combining redeem scripts
>> and witness scripts, check that the redeem scripts and witness scripts match the hash provided
>> in the UTXO (or in the redeem script) and choose the correct redeem script and witness script
>> accordingly if there were, for some reason, a conflict there.
>>
>> Can you explain why it would be unsafe for combiners to arbitrarily choose a value?
>
> We're worried that the "pick one of non-deterministic signatures" is a
> special case and that most fields don't have this property:
>
> * conflicts in UTXOs, sighash type, redeem/witness scripts, derivation
> paths, are at best a recoverable error, usually an unrecoverable error,
> at worst malicious activity.
>
> * conflict in finalized scripts, in case more than one valid
> finalization exists, might indicate that the Finalizers picked different
> ND signatures, or it might indicate two possible interpretations of the
> transaction (see next point). Picking arbitrarily in the latter case
> would be an error.
>
> * even for partial signatures: if two Signers with the same public key
> use different sighash types, the Combiner shouldn't pick the winning one
> arbitrarily.
>
> It seems generally safer to default to rejecting conflicts, and
> explicitly allowing the Combiner to process them intelligently if it
> understands the relevant fields.

So consider two possible topologies for a multiparty signing:

A) Creator and Updater produce a PSBT T. T is sent to signer 1 who
turns it into PSBT T1. T1 is then forwarded to Signer 2 who turns it
into T12. A Finalizer extracts the transaction.
B) Creator and Updater produce a PSBT T. T is sent to signer 1 and 2
simultaneously, who then produce T1 and T2 respectively. A Combiner
combines those into T12. A Finalizer extracts the transaction.

The only case where "malicious" conflicting values can occur is when
one of the Signers produces an invalid signature, or modifies any of
the other fields already present in the PSBT for consumption by
others. If this were an issue, it would be an issue regardless of the
Combiner's operation, as in topology A no Combiner is even present.
This is generally true I think - Combiners can always be replaced with
just a different (and possibly less parallel) topology of data flow.

So the question is independent of Combiners IMHO, and really about how
we deal with roles that intentionally or unintentionally produce
invalid values. I believe this is mostly not an issue. Let's go over
the cases:
* If a partial signature is invalid, the resulting transaction will be invalid.
* if a non-witness UTXO is incorrect, you'll fail to sign because the
txid mismatches the input's prevout (which you do have to check)
* If a witness UTXO is incorrect, the resulting signature will be invalid.
* If a derivation path is incorrect, the signer will fail to find the
key, or sign with the wrong key resulting in an invalid transaction.
* If a witnessscript or redeemscript is incorrect, the resulting
signature will be invalid (as those go into the scriptCode of the
sighash, and affect the type of sighashing)
* If a sighash type is incorrect, the resulting transaction may be
useless for its intended purpose (but still something every signer
agreed to sign).

So all of this boils down to dealing with the possibility that there
can be roles which intentionally or unintentionally create incorrect
fields in a PSBT, and the solution is (a) checking that prevout txids
match non-witness utxos (b) checking that the transaction you're
signing is one you want to sign (including sighash type) (c) worst
case accepting that the resulting transaction may be invalid.

Now, (c) can sometimes be caught early, by implementing additional
sanity checks for known fields. For example, rejecting PSBTs with
partial signatures that are invalid (feed them through a verifier).
This is something a Combiner can of course optionally do, but so can a
Signer or any other role.

The bottom line is that a Combiner which picks arbitrarily in case of
conflicts will never end up with something worse than what you already
need to deal with. If you disregard the case of invalid fields
(because the result will just be an invalid transaction), then any
choice the Combiner makes is fine, because all the values it can pick
from are valid.

> I agree with your response, and I also think that in technical sense,
> the worst that can happen is an invalid signature. Our concern is twofold:
>
> 1. the produced signature is most likely valid, _for a different
> transaction_ than the Creator intended. It is a transaction that the
> Signer must have authorized, so we could argue that they would not mind
> if that unintended transaction was published. Nevertheless, this opens
> an attack surface.

If you're worried about attack surface, I don't believe rejecting
invalid fields ever matters. An attacker can always drop the fields
you don't understand before giving you the PSBT, making your behavior
identical to one where you'd have ignore those fields in the first
place.

At best, you can make it protect against accidental mistakes that
would result in invalid transactions anyway.

If there is a way to sign a message in a way that can be
misinterpreted as a signature on a different message with a different
meaning, then that is a huge flaw in Bitcoin itself, and not going to
be solved by rejecting to sign unknown fields.

With regard to defense in depth:

>> I would not be opposed to having fields with an explicit flag bit that
>> says "Don't sign if you don't understand this", but I expect that that
>> can also be left for future extensions.
>
> It seems safer to consider this flag be on by default, and leave it to a
> future extension to allow non-mandatory fields. The worst case here is
> that legacy Signers can't natively process new PSBTs (solvable by a
> preprocessor) - as opposed to legacy Signers signing unintended values.

There could be some rule like "if the highest bit of the field type is
set, don't sign", but I don't think there is any current field where
such a flag would be necessary right now.

Cheers,

--
Pieter

📅 Original date posted:2018-07-04 📝 Original message:On ...

2023-06-07T18:13:32Z

In reply to nevent1q…57u2
_________________________

📅 Original date posted:2018-07-04
📝 Original message:On Wed, Jul 4, 2018 at 6:19 AM, matejcik <jan.matejek at satoshilabs.com> wrote:
> hello,
>
> we still have some concerns about the BIP as currently proposed - not
> about the format or data contents, but more about strictness and
> security properties. I have raised some in the previous e-mails, but
> they might have been lost in the overall talk about format.
>
> * Choosing from duplicate keys when combining.
> We believe that "choose whichever value it wishes" is not a good
> resolution strategy. We propose to either change this to "in case of
> conflicts, software MUST reject the conflicting PSBTs", or explain in
> more detail why picking at random is a safe choice.

Outlawing conflicting values would imply forcing all Signers to
implement fixed deterministic nonce generation, which I don't think it
very desirable. Otherwise PSBTs that got copied and signed and
combined again may fail. So I think we should see it the other way: we
choose the keys in such a way that picking arbitrarily is safe. If
there really is a future extension for which it would not be the case
that picking arbitrarily is acceptable, more data can be moved to the
keys, and leave the actual resolution strategy to the Finalizer. That
way Combiners can remain dumb and not need script-specific logic in
every interaction.

An alternative would be to have a fixed resolution strategy (for
example, when combining multiple PSBTs, pick the value from the first
one that has a particular key set), but I don't think this adds very
much - if picking the first is fine, picking a arbitrary one should be
fine too.

> * Signing records with unknown keys.
> There's been some talk about this at start, but there should be a clear
> strategy for Signers when unknown fields are encountered. We intend to
> implement the rule: "will not sign an input with any unknown fields
> present".
> Maybe it is worth codifying this behavior in the standard, or maybe
> there should be a way to mark a field as "optional" so that strict
> Signers know they can _safely_ ignore the unknown field.

Can you envision a situation in which this is needed? In every
scenario I can come up with, the worst that can happen is that the
resulting signature is just invalid. For example, if PSBT existed
before segwit, and then was later extended to support it, a pre-segwit
signer would not recognize that BIP143 would need to be used for
segwit inputs, and produce signatures using the old sighashing
algorithm. The result is just an invalid signature.

I believe that what you're trying to accomplish is preventing signing
something you don't understand, but that's an independent issue.
Signers generally will want to inspect the transaction they're
signing, or ask for confirmation w.r.t. fees or payment destinations
involved. The case where unknown fields are present for a reason you'd
want to withhold signing for will generally also just be the situation
where you don't understand the transaction you're signing.

Here is (perhaps far fetched) example of why it may not be desirable
to reject unknown fields when signing. Imagine an extension is defined
which adds pay-to-contract derivation for keys (Q = P + H(Q||C)G);
this would be a field similar to the current BIP32 derivation one, but
instead give a base key P and a contract C. Now say there is a 2-of-2
multisig in which you're one signer, and the other signer is (unknown
to you) using P2C. After the other party Updating, the input would
contain a P2C field which you don't understand - but it also isn't
something you care about or affects you.

I would not be opposed to having fields with an explicit flag bit that
says "Don't sign if you don't understand this", but I expect that that
can also be left for future extensions.

> * Fields with empty keys.
> This might be inferred from the definition, but is probably worth
> spelling out explicitly: If a field definition states that the key data
> is empty, an implementation MUST enforce this and reject PSBTs that
> contain non-empty data.
> We suggest adding something to the effect of:
> "If a key or value data in a field doesn't match the specified format,
> the PSBT is invalid. In particular, if key data is specified as "none"
> but the key contains data beyond the type specifier, implementation MUST
> reject the PSBT."
> (not sure about the languge, this should of course allow processing
> unknown fields)

Completely agree here. Any implementation that understands a
particular field must enforce whatever structure the field is known to
have.

> * "Combiner can detect inconsistencies"
> Added in response to this comment [1], the current wording looks like
> it's describing what the Combiner is _capable of_, as opposed to
> prescribing what the combiner is _allowed to_ do.
> We suggest changing to something like:
> "For every field type that the Combiner understands, it MAY also refuse
> to combine PSBTs that have inconsistencies in that field, or cause a
> conflict when combined."

Agree, just because Combiners are expected to work correctly on
unknown fields doesn't mean they can't enforce extra consistency
checks on known fields.

Cheers,

--
Pieter

📅 Original date posted:2018-06-27 📝 Original message:On ...

2023-06-07T18:13:14Z

In reply to nevent1q…ak3j
_________________________

📅 Original date posted:2018-06-27
📝 Original message:On Wed, Jun 27, 2018, 07:04 matejcik <jan.matejek at satoshilabs.com> wrote:

> hello,
>
> On 26.6.2018 22:30, Pieter Wuille wrote:
> >> (Moreover, as I wrote previously, the Combiner seems like a weirdly
> >> placed role. I still don't see its significance and why is it important
> >> to correctly combine PSBTs by agents that don't understand them. If you
> >> have a usecase in mind, please explain.
> >
> > Forward compatibility with new script types. A transaction may spend
> > inputs from different outputs, with different script types. Perhaps
> > some of these are highly specialized things only implemented by some
> > software (say HTLCs of a particular structure), in non-overlapping
> > ways where no piece of software can handle all scripts involved in a
> > single transaction. If Combiners cannot deal with unknown fields, they
> > won't be able to deal with unknown scripts.
>
> Record-based Combiners *can* deal with unknown fields. Either by
> including both versions, or by including one selected at random. This is
> the same in k-v model.
>

Yes, I wasn't claiming otherwise. This was just a response to your question
why it is important that Combiners can process unknown fields. It is not an
argument in favor of one model or the other.

> combining must be done independently by Combiner implementations for
> > each script type involved. As this is easily avoided by adding a
> > slight bit of structure (parts of the fields that need to be unique -
> > "keys"), this seems the preferable option.
>
> IIUC, you're proposing a "semi-smart Combiner" that understands and
> processes some fields but not others? That doesn't seem to change
> things. Either the "dumb" combiner throws data away before the "smart"
> one sees it, or it needs to include all of it anyway.
>

No, I'm exactly arguing against smartness in the Combiner. It should always
be possible to implement a Combiner without any script specific logic.

> No, a Combiner can pick any of the values in case different PSBTs have
> > different values for the same key. That's the point: by having a
> > key-value structure the choice of fields can be made such that
> > Combiners don't need to care about the contents. Finalizers do need to
> > understand the contents, but they only operate once at the end.
> > Combiners may be involved in any PSBT passing from one entity to
> > another.
>
> Yes. Combiners don't need to care about the contents.
> So why is it important that a Combiner properly de-duplicates the case
> where keys are the same but values are different? This is a job that,
> AFAICT so far, can be safely left to someone along the chain who
> understands that particular record.
>

That's because PSBTs can be copied, signed, and combined back together. A
Combiner which does not deduplicate (at all) would end up having every
original record present N times, one for each copy, a possibly large blowup.

For all fields I can think of right now, that type of deduplication can be
done through whole-record uniqueness.

The question whether you need whole-record uniqueness or specified-length
uniqueness (=what is offered by a key-value model) is a philosophical one
(as I mentioned before). I have a preference for stronger invariants on the
file format, so that it becomes illegal for a PSBT to contain multiple
signatures for the same key for example, and implementations do not need to
deal with the case where multiple are present.

It seems that you consider the latter PSBT "invalid". But it is well
> formed and doesn't contain duplicate records. A Finalizer, or a
> different Combiner that understands field F, can as well have the rule
> "throw away all but one" for this case.
>

It's not about considering. We're writing a specification. Either it is
made invalid, or not.

In a key-value model you can have dumb combiners that must pick one of the
keys in case of duplication, and remove the necessity of dealing with
duplication from all other implementations (which I consider to be a good
thing). In a record-based model you cannot guarantee deduplication of
records that permit repetition per type, because a dumb combiner cannot
understand what part is supposed to be unique. As a result, a record-based
model forces you to let all implementations deal with e.g. multiple partial
signatures for a single key. This is a minor issue, but in my view shows
how records are a less than perfect match for the problem at hand.

To repeat and restate my central question:
> Why is it important, that an agent which doesn't understand a particular
> field structure, can nevertheless make decisions about its inclusion or
> omission from the result (based on a repeated prefix)?
>

Again, because otherwise you may need a separate Combiner for each type of
script involved. That would be unfortunate, and is very easily avoided.

Actually, I can imagine the opposite: having fields with same "key"
> (identifying data), and wanting to combine their "values" intelligently
> without losing any of the data. Say, two Signers producing separate
> parts of a combined-signature under the same common public key?
>

That can always be avoided by using different identifying information as
key for these fields. In your example, assuming you're talking about some
form of threshold signature scheme, every party has their own "shard" of
the key, which still uniquely identifies the participant. If they have no
data that is unique to the participant, they are clones, and don't need to
interact regardless.

> In case of BIP32 derivation, computing the pubkeys is possibly
> > expensive. A simple signer can choose to just sign with whatever keys
> > are present, but they're not the only way to implement a signer, and
> > even less the only software interacting with this format. Others may
> > want to use a matching approach to find keys that are relevant;
> > without pubkeys in the format, they're forced to perform derivations
> > for all keys present.
>
> I'm going to search for relevant keys by comparing master fingerprint; I
> would expect HWWs generally don't have index based on leaf pubkeys.
> OTOH, Signers with lots of keys probably aren't resource-constrained and
> can do the derivations in case of collisions.
>

Perhaps you want to avoid signing with keys that are already signed with?
If you need to derive all the keys before even knowing what was already
signed with, you've already performed 80% of the work.

> If you take the records model, and then additionally drop the
> > whole-record uniqueness constraint, yes, though that seems pushing it
> > a bit by moving even more guarantees from the file format to
> > application level code.
>
> The "file format" makes no guarantees, because the parsing code and
> application code is the same anyway. You could say I'm proposing to
> separate these concerns ;)
>

Of course a file format can make guarantees. If certain combinations of
data in it do not satsify the specification, the file is illegal, and
implementations do not need to deal with it. Stricter file formats are
easier to deal with, because there are less edge cases to consider.

To your point: proto v2 afaik has no way to declare "whole record
uniqueness", so either you drop that (which I think is unacceptable - see
the copy/sign/combine argument above), or you deal with it in your
application code.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180627/fecd0346/attachment-0001.html>;

📅 Original date posted:2018-06-22 📝 Original message:On ...

2023-06-07T18:13:09Z

In reply to nevent1q…grdk
_________________________

📅 Original date posted:2018-06-22
📝 Original message:On Thu, Jun 21, 2018 at 12:56 PM, Peter D. Gray via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> I have personally implemented this spec on an embedded micro, as
> the signer and finalizer roles, and written multiple parsers for
> it as well. There is nothing wrong with it, and it perfectly meets
> my needs as a hardware wallet.

This is awesome to hear. We need to hear from people who have comments
or issues they encounter while implementing, but also cases where
things are fine as is.

> So, there is a good proposal already spec'ed and implemented by
> multiple parties. Andrew has been very patiently shepherding the PR
> for over six months already.
>
> PSBT is something we need, and has been missing from the ecosystem
> for a long time. Let's push this out and start talking about future
> versions after we learn from this one.

I understand you find the suggestions being brought up in this thread
to be bikeshedding over details, and I certainly agree that "changing
X will gratuitously cause us more work" is a good reason not to make
breaking changes to minutiae. However, at least abstractly speaking,
it would be highly unfortunate if the fact that someone implemented a
draft specification results in a vested interest against changes which
may materially improve the standard.

In practice, the process surrounding BIPs' production readiness is not
nearly as clear as it could be, and there are plenty of BIPs actually
deployed in production which are still marked as draft. So in reality,
truth is that this thread is "late", and also why I started the
discussion by asking what the state of implementations was. As a
result, the discussion should be "which changes are worth the hassle",
and not "what other ideas can we throw in" - and some of the things
brought up are certainly the latter.

So to get back to the question what changes are worth the hassle - I
believe the per-input derivation paths suggested by matejcik may be
one. As is written right now, I believe BIP174 requires Signers to
pretty much always parse or template match the scripts involved. This
means it is relatively hard to implement a Signer which is compatible
with many types of scripts - including ones that haven't been
considered yet. However, if derivation paths are per-input, a signer
can just produce partial signatures for all keys it has the master
for. As long as the Finalizer understands the script type, this would
mean that Signers will work with any script. My guess is that this
would be especially relevant to devices where the Signer
implementation is hard to change, like when it is implemented in a
hardware signer directly.

What do you think?

Cheers,

--
Pieter

📅 Original date posted:2018-06-21 📝 Original message:On ...

2023-06-07T18:13:06Z

In reply to nevent1q…uxxz
_________________________

📅 Original date posted:2018-06-21
📝 Original message:On Thu, Jun 21, 2018 at 4:29 AM, matejcik <jan.matejek at satoshilabs.com> wrote:
> In the case of everything per-input, the naive Signer can do this:
> 1. (in the global section) pre-serialize the transaction
> 2. (in each input) find and fill out scriptPubKey from the provided UTXO
> 3. (for a given BIP32 path) check if the master fingerprint matches
> mine, if yes, derive secret key, output pubkey, signature
> 4. goto 3 (more keys per input), goto 2 (next input)
>
> Note that this flow works perfectly for multisig; it’s going to be the
> job of a Finalizer to build the final scriptSig, but each input can have
> multiple partial signatures -- and, interestingly, the naive Signer
> doesn’t even need to know about multisig.

Ah, you're thinking of an even simpler signer than I was imagining. I
don't think this works in general, because the hash being signed
depends on the structure of the script. For example, if it is P2SH, it
is the redeemscript that goes into the scriptCode serialization rather
than the scriptPubKey. If it is segwit, BIP143 serialization needs to
be used, etc. It may work if your signing is restricted to just one of
those structures, though.

> A less naive Signer will want to check things, maybe derive a scriptSig
> itself and check if it matches the given hash, etc., but it can do this
> all in place. You go linearly through the signing flow and place a
> couple strategic assertions along the way.

Right - but I think anything but the simplest signer must do this,
just to be able to distinguish between different kinds of signature
hashing.

But you're right, having per-input redeemscript/witnessscript
simplifies things still - instead of needing to look a script hash in
a map, you can just compare it with *the* redeemscript/witnessscript.

> However, if the data is global, as is now, it gets more complicated:
> 1. (in the global section) pre-serialize the transaction, prefill lookup
> tables
> 2. (for a given BIP32 path) check if mine, then derive public key and
> store in a dictionary
> 3. (for each input) find _and parse_ scriptPubKey, extract (PK or)
> script hash
> 4. lookup redeem script based on script-hash; if not found, goto 2; if
> found, parse out public key
> 5. lookup public key in the BIP32 dictionary; if not found, goto 2
> 6. output pubkey, signature

I understand your point now. I hadn't considered the possibility of
just signing with all BIP32 derivation paths given for which the
master matches, instead of extracting pubkeys/pkhs from the script.
That's a major simplification for signers indeed. I do think you need
some conditions before to determine the script structure (see above),
but this is a good point in favour of making the derivation paths
per-input.

> In general, you seem to focus a lot on the role of Combiners, esp.
> simple Combiners. To me, that doesn’t look like a significant role. As I
> envision it, a Combiner really doesn’t need to do anything more
> complicated than merge and deduplicate records, simply based on the
> uniqueness of the whole record.

It's more a side-effect of focusing on forward compatibility. I expect
that we will have transactions with inputs spending different kinds of
outputs, and some signers may not be able to understand all of them.
However, as long as the design goal of having Combiners function
correctly for things they don't understand, everything should be able
to work together fine.

> It’s the Finalizer’s job to reconstruct and validate the result. Also
> ISTM if something messes up the PSBT (such as including multiple
> conflicting fields anywhere), it’s OK to leave it to Finalizer to fail.

Agree.

> An aside to this in particular, I’ve been thinking about the requirement
> to share derivation paths and public keys with the Creator. The spec
> assumes that this will happen; you’re talking about providing full
> xpub+chaincode too. At least, the Creator must prefill BIP32 paths and
> master key fingerprints. Possibly also prefill public keys in the redeem
> scripts?
>
> This might not be an improvement proposal, but a point worth being
> raised and maybe explained in the spec. Perhaps the original Creator
> doesn’t have access to this data, and delegates this to some
> “sub-Creators” - I imagine a coordinator sending a PSBT to signing
> parties, each of which acts as a sub-Creator (fills out derivation paths
> and public keys) and a Signer (forwarding to a HWW). Some of the
> discussion even suggests some sort of generic “key derivation field”
> with arbitrary contents - fingerprint + bip32 path? xpub + chain code?
> derivation points? encrypted xprv?

That makes sense - I think we've already touched this when discussing
the requirement for UTXOs to be added. Perhaps those aren't added by
the Creator, but by some index server. The same could be true for the
scripts or derivations paths.

And indeed, most of the information in the derivation paths is
effectively opaque to the Creator - it's just some data given out by
the Signer about its keys that gets passed back to it so it can
identify the key. There is benefit in keeping it in a fixed structure
(like xpub/chaincode, or fingerprint + derivation indexes), to
guarantee compatibility between multiple signer implementations with
access to the same key.

On Tue, Jun 19, 2018 at 5:39 PM, Jason Les via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
>>Hex encoding?
>
> I was hoping for some standard here was well and I agree using something
> more compact than hex is important. My understanding is Z85 is better for
> use with JSON than Base64, which is probably a good benefit to have here.

Both Base64 and Z85 can be stored in JSON strings without quoting
(neither uses quotation characters or backslashes), but Z85 is
slightly more compact (Z85 is 5 characters for 4 bytes, Base64 is 4
characters for 3 bytes). Both use non-alphanumeric characters, so I
don't think there is much difference w.r.t. copy-pastability either.
Z85 is far less common though.

On Thu, Jun 21, 2018 at 4:44 AM, Tomas Susanka via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
>> I think here it makes sense because there can actually only be (up to)
>> one redeemscript and (up to) one witnessscript. So if we made those
>> per-input and per-output, it may simplify signers as they don't need a
>> table lookup to find the correct one. That would also mean we can drop
>> their hashes, even if we keep a key-value model.
> Yes, indeed. Just to clarify: in the first sentence you mean "per
> output", right? There can actually only be (up to) one redeemscript and
> (up to) one witnessscript *per output*.

Up to one per output, and up to one per input - indeed.

On Thu, Jun 21, 2018 at 7:32 AM, Tomas Susanka via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
>>> A question to consider is,
>>> will there be more per-output data? If yes, it might make sense to have
>>> an output section.
>> I think it is unlikely that there would be anymore per-output data.
>
> Hmm, upon further reflection, maybe it's not even worth including *any*
> per-output data, aside from what the original transaction contains.
>
> The output redeem script is either:
> - unknown, because we have received only an address from the receiver
> - or it is known, because it is ours and in that case it doesn’t make
> sense to include it in PSBT
>
> We got stuck on the idea of the Creator providing future (output)
> redeem/witness scripts. But that seems to be a minority use case and can
> be solved efficiently via the same channels that coordinate the PSBT
> creation. Sorry to change opinions so quickly on this one.

Perhaps you're missing the reason for having output scripts? It is so
that signers that wish to known the amounts transferred can be told
which outputs of the to-be transaction are change, and thus shouldn't
be counted towards the balance. By providing the scripts and
derivation paths in a PSBT, the Creator can prove to the Signer that
certain outputs do not actually move funds to some other entity.

Based on the points before, my preference is having everything
per-input and per-output except the transaction (with empty
scriptSig/witness) itself, and having exactly one set/map per input
and output (which may include a "finalized scriptSig/witness field"
for finalized inputs). The overhead of having at least one separator
byte for every input and output in the transaction is at most a few
percent compared to the data in the transaction itself. If size is
really an issue (but I think we've already established that small size
gains aren't worth much extra complexity), we could also serialize the
transaction without scriptSigs/witnesses (which are at least one byte
each, and guaranteed to be empty).

I'm unsure about typed record vs. key-value model. If we'd go with a
per-input script approach, the key would just be a single byte ("the
redeemscript" and "the witnessscript"), so the advantage of being able
to drop the script hashes applies equally to both models. After that,
it seems the only difference seems to be that a well-defined prefix of
the records is enforced to be unique as opposed to the entire record
being enforced to be unique. I don't think there is much difference in
complexity, as Combiners and Signers still need to enforce some kind
of uniqueness even in a typed records model.

Cheers,

--
Pieter

📅 Original date posted:2018-06-19 📝 Original message:On ...

2023-06-07T18:13:04Z

In reply to nevent1q…5e9l
_________________________

📅 Original date posted:2018-06-19
📝 Original message:On Tue, Jun 19, 2018 at 7:20 AM, matejcik via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:

Thanks for your comments so far. I'm very happy to see people dig into
the details, and consider alternative approaches.

> 1) Why isn't the global type 0x03 (BIP-32 path) per-input? How do we
> know, which BIP-32 path goes to which input? The only idea that comes to
> my mind is that we should match the input's scriptPubKey's pubkey to
> this 0x03's key (the public key).

> If our understanding is correct, the BIP-32 path is global to save space
> in case two inputs share the same BIP-32 path? How often does that
> happen? And in case it does, doesn't it mean an address reuse which is
> discouraged?

Yes, the reason is address reuse. It may be discouraged, but it still
happens in practice (and unfortunately it's very hard to prevent
people from sending to the same address twice).

It's certainly possible to make them per-input (and even per-output as
suggested below), but I don't think it gains you much. At least when a
signer supports any kind of multisig, it needs to match up public keys
with derivation paths. If several can be provided, looking them up
from a global table or a per-input table shouldn't fundamentally
change anything.

However, perhaps it makes sense to get rid of the global section
entirely, and make the whole format a transaction plus per-input and
per-output extra fields. This would result in duplication in case of
key reuse, but perhaps that's worth the complexity reduction.

> 2) The global items 0x01 (redeem script) and 0x02 (witness script) are
> somewhat confusing. Let's consider only the redeem script (0x01) to make
> it simple. The value description says: "A redeem script that will be
> needed to sign a Pay-To-Script-Hash input or is spent to by an output.".
> Does this mean that the record includes both input's redeem script
> (because we need to sign it), but also a redeem script for the output
> (to verify we are sending to a correct P2SH)? To mix those two seems
> really confusing.
>
> Yet again, adding a new output section would make this more readable. We
> would include the input’s redeem script in the input section and the
> output’s redeem script again in the output section, because they’ll most
> likely differ anyway.

I think here it makes sense because there can actually only be (up to)
one redeemscript and (up to) one witnessscript. So if we made those
per-input and per-output, it may simplify signers as they don't need a
table lookup to find the correct one. That would also mean we can drop
their hashes, even if we keep a key-value model.

> 3) The sighash type 0x03 says the sighash is only a recommendation. That
> seems rather ambiguous. If the field is specified shouldn't it be binding?

Perhaps, yes.

> 4) Is it a good idea to skip records which types we are unaware of? We
> can't come up with a reasonable example, but intuitively this seems as a
> potential security issue. We think we should consider introducing a
> flag, which would define if the record is "optional". In case the signer
> encounters a record it doesn't recognize and such flag is not set, it
> aborts the procedure. If we assume the set model we could change the
> structure to <type><optional flag><length>{data}. We are not keen on
> this, but we wanted to include this idea to see what you think.

Originally there was at least this intuition for why it shouldn't be
necessary: the resulting signature for an input is either valid or
invalid. Adding information to a PSBT (which is what signers do)
either helps with that or not. The worst case is that they simply
don't have enough information to produce a signature together. But an
ignored unknown field being present should never result in signing the
wrong thing (they can always see the transaction being signed), or
failing to sign if signing was possible in the first place. Another
way of looking at it, the operation of a signer is driven by queries:
it looks at the scriptPubKey of the output being spent, sees it is
P2SH, looks for the redeemscript, sees it is P2WSH, looks for the
witnessscript, sees it is multisig, looks for other signers'
signatures, finds enough for the threshold, and proceeds to sign and
create a full transaction. If at any point one of those things is
missing or not comprehensible to the signer, he simply fails and
doesn't modify the PSBT.

However, if the sighash request type becomes mandatory, perhaps this
is not the case anymore, as misinterpreting something like this could
indeed result in an incorrect signature.

If we go down this route, if a field is marked as mandatory, can you
still act as a combiner for it? Future extensions should always
maintain the invariant that a simple combiner which just merges all
the fields and deduplicates should always be correct, I think. So such
a mandatory field should only apply to signers?

> In general, the standard is trying to be very space-conservative,
> however is that really necessary? We would argue for clarity and ease of
> use over space constraints. We think more straightforward approach is
> desired, although more space demanding. What are the arguments to make
> this as small as possible? If we understand correctly, this format is
> not intended for blockchain nor for persistent storage, so size doesn’t
> matter nearly as much.

I wouldn't say it's trying very hard to be space-conservative. The
design train of thought started from "what information does a signer
need", and found a signer would need information on the transaction to
sign, and on scripts to descend into, information on keys to derive,
and information on signatures provided by other participants. Given
that some of this information was global (at least the transaction),
and some of this information was per-input (at least the signatures),
separate scopes were needed for those. Once you have a global scope,
and you envision a signer which looks up scripts and keys in a map of
known ones (like the signing code in Bitcoin Core), there is basically
no downside to make the keys and scripts global - while giving space
savings for free to deduplication.

However, perhaps that's not the right way to think about things, and
the result is simpler if we only keep the transaction itself global,
and everything else per-input (and per-output).

I think there are good reasons to not be gratuitously large (I expect
that at least while experimenting, people will copy-paste these things
a lot and page-long copy pastes become unwieldy quickly), but maybe
not at the cost of structural complexity.

On Tue, Jun 19, 2018 at 7:22 AM, matejcik via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> hello,
> this is our second e-mail with replies to Pieter's suggestions.
>
> On 16.6.2018 01:34, pieter.wuille at gmail.com (Pieter Wuille) wrote:
>> * Key-value map model or set model.

> Just to note, we should probably use varint for the <type> field - this
> allows us, e.g., to create “namespaces” for future extensions by using
> one byte as namespace identifier and one as field identifier.

Agree, but this doesn't actually need to be specified right now. As
the key's (and or value's) interpretation (including the type) is
completely unspecified, an extension can just start using 2-byte keys
(as long as the first byte of those 2 isn't used by another field
already).

>> One exception is the "transaction" record, which needs to be unique.
>> That can either be done by adding an exception ("there can only be one
>> transaction record"), or by encoding it separately outside the normal
>> records (that may also be useful to make it clear that it is always
>> required).
>
> This seems to be the case for some fields already - i.e., an input field
> must have exactly one of Non-witness UTXO or Witness Output. So “adding
> an exception” is probably just a matter of language?

Hmm, I wouldn't say so. Perhaps the transaction's inputs and outputs
are chosen by one entity, and then sent to another entity which has
access to the UTXOs or previous transactions. So while the UTXOs must
be present before signing, I wouldn't say the file format itself must
enforce that the UTXOs are present.

However, perhaps we do want to enforce at-most one UTXO per input. If
there are more potential extensions like this, perhaps a key-value
model is better, as it's much easier to enforce no duplicate keys than
it is to add field-specific logic to combiners (especially for
extensions they don't know about yet).

> We’d also like to note that the “number of inputs” field should be
> mandatory - and as such, possibly also a candidate for outside-record field.

If we go with the "not put signatures/witnesses inside the transaction
until all of them are finalized" suggestion, perhaps the number of
inputs field can be dropped. There would be always one exactly for
each input (but some may have the "final script/witness" field and
others won't).

>> * Derivation from xpub or fingerprint
>>
>> For BIP32 derivation paths, the spec currently only encodes the 32-bit
>> fingerprint of the parent or master xpub. When the Signer only has a
>> single xprv from which everything is derived, this is obviously
>> sufficient. When there are many xprv, or when they're not available
>> indexed by fingerprint, this may be less convenient for the signer.
>> Furthermore, it violates the "PSBT contains all information necessary
>> for signing, excluding private keys" idea - at least if we don't treat
>> the chaincode as part of the private key.
>>
>> For that reason I would suggest that the derivation paths include the
>> full public key and chaincode of the parent or master things are
>> derived from. This does mean that the Creator needs to know the full
>> xpub which things are derived from, rather than just its fingerprint.
>
>
> We don’t understand the rationale for this idea. Do you see a scenario
> where an index on master fingerprint is not available but index by xpubs
> is? In our envisioned use cases at least, indexing private keys by xpubs
> (as opposed to deriving from a BIP32 path) makes no sense.

Let me elaborate.

Right now, the BIP32 fields are of the form <master
fingerprint><childidx><childidx><childidx>...

Instead, I suggest fields of the form <master pubkey><master
chaincode><childidx><childidx><childidx>...

The fingerprint in this case is identical to the first 32 bit of the
Hash160 of <master pubkey>, so certainly no information is lost by
making this change.

This may be advantageous for three reasons:
* It permits signers to have ~thousands of master keys (at which point
32-bit fingerprints would start having reasonable chance for
collisions, meaning multiple derivation attempts would be needed to
figure out which one to use).
* It permits signers to index their master keys by whatever they like
(for example, SHA256 rather than Hash160 or prefix thereof).
* It permits signers who don't store a chaincode at all, and just
protect a single private key.

Cheers,

--
Pieter

📅 Original date posted:2018-06-15 📝 Original message:Hello ...

2023-06-07T18:13:02Z

In reply to nevent1q…uxvg
_________________________

📅 Original date posted:2018-06-15
📝 Original message:Hello all,

given some recent work and discussions around BIP 174 (Partially
Signed Bitcoin Transaction Format) I'd like to bring up a few ideas.

First of all, it's unclear to me to what extent projects have already
worked on implementations, and thus to what extent the specification
is still subject to change. A response of "this is way too late" is
perfectly fine.

So here goes:

* Key-value map model or set model.

This was suggested in this thread:
https://twitter.com/matejcik/status/1002618633472892929

The motivation behind using a key-value model rather than a simple
list of records was that PSBTs can be duplicated (given to multiple
people for signing, for example), and merged back together after
signing. With a generic key-value model, any implementation can remove
the duplication even if they don't understand fields that may have
been added in future extensions.

However, almost the same can be accomplished by using the simpler set
model (the file consists of a set of records, with no duplication
allowed). This would mean that it would technically be legal to have
two partial signatures with the same key for the same input, if a
non-deterministic signer is used.

On the other hand, this means that certain data currently encoded
inside keys can be dropped, reducing the PSBT size. This is
particularly true for redeemscripts and witnessscripts, as they can
just be computed by the client when deserializing. The two types could
even be merged into just "scripts" records - as they don't need to be
separated based on the way they're looked up (Hash160 for P2SH, SHA256
for P2WSH). The same could be done for the BIP32 derivation paths,
though this may be expensive, as the client would need to derive all
keys before being able to figure out which one(s) it needs.

One exception is the "transaction" record, which needs to be unique.
That can either be done by adding an exception ("there can only be one
transaction record"), or by encoding it separately outside the normal
records (that may also be useful to make it clear that it is always
required).

* Ability for Combiners to verify two PSBT are for the same transaction

Clearly two PSBTs for incompatible transactions cannot be combined,
and this should not be allowed.

It may be easier to enforce this if the "transaction" record inside a
PSBT was required to be in a canonical form, meaning with empty
scriptSigs and witnesses. In order to do so, there could be per-input
records for "finalized scriptSig" and "finalized witness". Actually
placing those inside the transaction itself would only be allowed when
all inputs are finalized.

* Optional signing

I think all operation for the Signer responsibility should be
optional. This will inevitably lead to incompatibilities, but with the
intent of being forward compatible with future developments, I don't
think it is possible to require every implementation to support the
same set of scripts or contracts. For example, some signers may only
implement single-key P2PKH, or may only support signing SegWit inputs.
It's the user's responsibility to find compatible signers (which is
generally not an issue, as the different participants in a setup
necessarily have this figured out before being able to create an
address). This does mean that there can't be an unconditional test
vector that specifies the produced signature in certain circumstances,
but there could be "For implementations that choose to implement
signing for P2PKH inputs using RFC6979, the expected output given
input X and access to key Y is Z".

On the other hand, the Combiner and Finalizer roles can probably be
specified much more accurately than they are now.

* Derivation from xpub or fingerprint

For BIP32 derivation paths, the spec currently only encodes the 32-bit
fingerprint of the parent or master xpub. When the Signer only has a
single xprv from which everything is derived, this is obviously
sufficient. When there are many xprv, or when they're not available
indexed by fingerprint, this may be less convenient for the signer.
Furthermore, it violates the "PSBT contains all information necessary
for signing, excluding private keys" idea - at least if we don't treat
the chaincode as part of the private key.

For that reason I would suggest that the derivation paths include the
full public key and chaincode of the parent or master things are
derived from. This does mean that the Creator needs to know the full
xpub which things are derived from, rather than just its fingerprint.

* Generic key offset derivation

Whenever a BIP32 derivation path does not include any hardened steps,
the entirety of the derivation can be conveyed as "The private key for
P is equal to the private key for Q plus x", with P and Q points and x
a scalar. This representation is more flexible (it also supports
pay-to-contract derivations), more efficient, and more compact. The
downside is that it requires the Signer to support such derivation,
which I don't believe any current hardware devices do.

Would it make sense to add this as an additional derivation method?

* Hex encoding?

This is a very minor thing. But presumably there will be some standard
way to store PSBTs as text for copy-pasting - either prescribed by the
spec, or de facto. These structures may become pretty large, so
perhaps it's worth choosing something more compact than hexadecimal -
for example Base64 or even Z85 (https://rfc.zeromq.org/spec:32/Z85/).

Cheers,

--
Pieter

📅 Original date posted:2018-06-03 📝 Original message:On ...

2023-06-07T18:12:47Z

In reply to nevent1q…frr3
_________________________

📅 Original date posted:2018-06-03
📝 Original message:On Sun, Jun 3, 2018 at 9:51 AM, Jonas Schnelli via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi
>
> The BIP proposal is now available here:
> https://gist.github.com/jonasschnelli/68a2a5a5a5b796dc9992f432e794d719
>
> Reference C code is available here:
> https://github.com/jonasschnelli/bech32_keys
>
> Feedback, criticism, etc. welcome!

First of all, thanks for working on this.

I have some concerns about the use of Bech32. It is designed for
detecting 3 errors up to length 1023 (but is then picked specifically
to support 4 errors up to length 89). However, for error correction
this translates to just being able to efficiently correct 1 error
(3/2, rounded down) up to length 1023. You can of course always try
all combinations of up to N changes to the input (for any N), testing
the checksum, and comparing the results against the UTXO set or other
wallet information that may have been recovered. However, the checksum
at best gives you a small constant speedup here, not a fundamentally
improved way for recovery.

However, we can design other base32 BCH codes easily with different
properties. As we mostly care about efficient algorithms for recovery
(and not just error detection properties), it seems more important to
have good design strength (as opposed to picking a code from a large
set which happens to have better properties, but without efficient
algorithm, like Bech32).

This is what I find for codes designed for length 93 (the first length
for which efficient codes exist with length long enough to support 256
bits of data):
* correct 1 error = 3 checksum characters
* correct 2 errors = 6 checksum characters
* correct 3 errors = 10 checksum characters
* correct 4 errors = 13 checksum characters
* correct 5 errors = 16 checksum characters
* ...
* correct 8 errors = 26 checksum characters (~ length * 1.5)
* correct 11 errors = 36 checksum characters (~ maximum length without
pushing checksum + data over 93 characters)

For codes designed for length 341 (the first length enough to support
512 bits of data):
* correct 1 error = 3 checksum characters
* correct 2 errors = 7 checksum characters
* correct 3 errors = 11 checksum characters
* correct 4 errors = 15 checksum characters
* correct 5 errors = 19 checksum characters
* ...
* correct 7 errors = 26 checksum characters (~ length * 1.25)
* correct 13 errors = 51 checksum characters (~ length * 1.5)
* correct 28 errors = 102 checksum characters (~ length * 2)

So it really boils down to a trade-off between length of the code, and
recovery properties.

These two sets of codes are distinct (a code designed for length 93
has zero error correction properties when going above 93), so either
we can pick a separate code for the two purposes, or be limited to the
second set.

If there is interest, I can construct a code + implementation for any
of these in a few days probably, once the requirements are clear.

Cheers,

--
Pieter

📅 Original date posted:2018-06-03 📝 Original message:On ...

2023-06-07T18:12:39Z

In reply to nevent1q…0k7f
_________________________

📅 Original date posted:2018-06-03
📝 Original message:On Sat, Jun 2, 2018, 22:56 Tamas Blummer via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Lighter but SPV secure nodes (filter committed) would help the network
> (esp. Layer 2) to grow mesh like, but add more user that blindly follow POW.
>
> On longer term most users' security will be determined by either trusted
> hubs or POW.
> I do not know which is worse, but we should at least offer the choice to
> the user, therefore commit filters.
>

I don't think that's the point of discussion here. Of course, in order to
have filters that verifiably don't lie by omission, the filters need to be
committed to by blocks.

The question is what data that filter should contain.

There are two suggestions:
(a) The scriptPubKeys of the block's outputs, and prevouts of the block's
inputs.
(b) The scriptPubKeys of the block's outputs, and scriptPubKeys of outputs
being spent by the block's inputs.

The advantage of (a) is that it can be verified against a full block
without access to the outputs being spent by it. This allows light clients
to ban nodes that give them incorrect filters, but they do need to actually
see the blocks (partially defeating the purpose of having filters in the
first place).

The advantage of (b) is that it is more compact (scriot reuse, and outputs
spent within the same block as they are created). It also had the advantage
of being more easily usable for scanning of a wallet's transactions. Using
(a) for that in some cases may need to restart and refetch when an output
is discovered, to go test for its spending (whose outpoint is not known
ahead of time). Especially when fetching multiple filters at a time this
may be an issue.

I think both of these potentially good arguments. However, once a committed
filter exists, the advantage of (a) goes away completely - validation of
committed filters is trivial and can be done without needing the full
blocks in the first place.

So I think the question is do we aim for an uncommitted (a) first and a
committed (b) later, or go for (b) immediately?

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180602/ddc0b29e/attachment-0001.html>;

📅 Original date posted:2018-05-31 📝 Original message:On ...

2023-06-07T18:12:34Z

In reply to nevent1q…yrp2
_________________________

📅 Original date posted:2018-05-31
📝 Original message:On Fri, May 25, 2018 at 3:14 AM, Johnson Lau <jl2012 at xbt.hk> wrote:
> A graftroot design like this is a strict subset of existing signature checking rules. If this is dangerous, the existing signature checking rules must be dangerous.

While you may be right in this situation, I'm not sure that conclusion
follows from your argument. Whether or not a construction is safe does
not just depend on the consensus rules, but also on how it is used.
Otherwise you could as well argue that since OP_TRUE is possible right
now which is obviously insecure, nothing more dangerous can be
accomplished through any soft fork.

The best argument for why Graftroot does not need to be optional I
think was how Greg put it: "since the signer(s) could have signed an
arbitrary transaction instead, being able to delegate is strictly less
powerful.".

Cheers,

--
Pieter

📅 Original date posted:2018-05-25 📝 Original message:Hi ...

2023-06-07T18:12:29Z

In reply to nevent1q…9hpm
_________________________

📅 Original date posted:2018-05-25
📝 Original message:Hi all,

I spent some time working out the optimal parameter selection for the
Golomb Coded Sets that are proposed in BIP158:
https://gist.github.com/sipa/576d5f09c3b86c3b1b75598d799fc845

TL;DR: if we really want an FP rate of exactly 1 in 2^20, the Rice
parameter should be 19, not 20. If we don't, we should pick an FP rate
of 1 in a 1.4971*2^B. So for example M=784931 B=19 or M=1569861 B=20.

Cheers,

--
Pieter

📅 Original date posted:2018-05-23 📝 Original message:On ...

2023-06-07T18:12:26Z

In reply to nevent1q…c0h9
_________________________

📅 Original date posted:2018-05-23
📝 Original message:On Tue, May 22, 2018 at 11:17 AM, Pieter Wuille <pieter.wuille at gmail.com> wrote:
> Hello all,
>
> Given the recent discussions about Taproot [1] and Graftroot [2], I
> was wondering if a practical deployment needs a way to explicitly
> enable or disable the Graftroot spending path. I have no strong
> reasons why this would be necessary, but I'd like to hear other
> people's thoughts.

Thanks everyone who commented so far, but let me clarify the context
of this question first a bit more to avoid getting into the weeds too
much.

If it turns out to be necessary to explicitly commit to permitting
Graftroot spending, there are a number of approaches:
* Use a different witness version (or other marker directly in the
scriptPubKey) to enable Graftroot.
* Signal the permission to spend through Graftroot inside the Taproot
script as suggested by ZmnSCPxj.
* Make "Spend through Graftroot" a special script (possibly indirectly
with a Merkle tree in Taproot).
* Implement Graftroot as an opcode/feature inside the scripting
language (which may be more generically useful as a delegation
mechanism).
* Postpone Graftroot.

All of these are worse in either efficiency or privacy than always
permitting Graftroot spends directly. Because of that, I think we
should first focus on reasons why a lack of commitment to enabling
Graftroot may result in it being incompatible with certain use cases,
or other reasons why it could interfere with applications adopting
such outputs.

@Natanael: all of these concerns only apply to a new hypothetical
Taproot/Graftroot output type, which combines pay-to-pubkey and
pay-to-script in a single scriptPubKey that just contains a public
key. It doesn't apply to existing P2SH like constructions.

Also, the concern of making Graftroot optional does not apply to
Taproot, as the Taproot spending path's script is committed to (using
scriptPubKey = P + H(P,script)*G), allowing the script to be
explicitly chosen to be a non-spendable script, which the author could
prove is the case (without revealing it to the entire world).

It is also always possible to create a "script only" Taproot output
(without key that can unconditionally spend), by picking a pubkey that
is provably unspendable (hashing onto a curve point, in particular),
or through pubkey recovery.

Cheers,

--
Pieter

📅 Original date posted:2018-05-22 📝 Original message:Hello ...

2023-06-07T18:12:23Z

In reply to nevent1q…qpn3
_________________________

📅 Original date posted:2018-05-22
📝 Original message:Hello all,

Given the recent discussions about Taproot [1] and Graftroot [2], I
was wondering if a practical deployment needs a way to explicitly
enable or disable the Graftroot spending path. I have no strong
reasons why this would be necessary, but I'd like to hear other
people's thoughts.

As a refresher, the idea is that a script type could exists which
looks like a pubkey Q, but can be spent either:
* By signing the spending transaction directly using Q (key spending)
* By proving Q was derived as Q = P + H(P,S)*G, with a script S and
its inputs (Taproot script spending).
* By signing a script S using Q, and providing S's inputs (Graftroot
script spending).

Overall, these constructions let us create combined
pay-to-pubkey-or-script outputs that are indistinguishable, and don't
even reveal a script spending path existed in the first place when the
key spending path is used. The two approaches ((T)aproot and
(G)raftroot) for the script spending path have different trade-offs:
* T outputs can be derived noninteractively from key and scripts; G
outputs need an interaction phase where the key owner(s) sign off on
the potential script spending paths.
* T lets you prove what all the different spending paths are.
* T without any other technology only needs to reveal an additional
point when spending a single script; G needs half-aggregated
signatures [3] to achieve the same, which complicates design (see
[4]).
* G is more compact when dealing with many spending paths (O(1) in the
number of spending paths), while T spends need to be combined with
Merkle branches to deal with large number of spends (and even then is
still O(log n)).
* G spending paths can be added after the output is created; T
requires them be fixed at output creation time.

My question is whether it is safe to always permit both types of
script spending paths, or an explicit commitment to whether Graftroot
is permitted is necessary. In theory, it seems that this shouldn't be
needed: the key owners are always capable of spending the funds
anyway, so them choosing to delegate to others shouldn't enable
anything that isn't
possible by the key owners already.

There are a few concerns, however:

* Accidentally (participating in) signing a script may have more broad
consequences. Without Graftroot, that effect is limited to a single
transaction with specific inputs and outputs, and only as long as all
those inputs are unspent. A similar but weaker concern exists for
SIGHASH_NOINPUT.

* In a multisignature setting (where the top level key is an aggregate
of multiple participants), the above translates to the ability for a
(threshold satsisfying) subset of participants being able to (possibly
permanently) remove others from the set of signers (rather than for a
single output).

* In a situation where private keys are stored in an HSM, without
Graftroot an attacker needs access to the device and convince it to
sign for every output he wants to steal (assuming the HSM prevents
leaking private keys). With Graftroot, the HSM may be tricked into
signing a script that does not include itself. Arguably, in a
Graftroot setting such an HSM would need a degree of protection
similar to not leaking private keys applied to not signing scripts,
but this may be less obvious.

Overall, none of these are convincing points, but they do make me
uncomfortable about the effect the Graftroot spending path may have on
some use cases. Given that Taproot/Graftroot's primary advantage is
increasing fungibility by making all outputs look identical, it seems
good to discuss potential reasons such outputs couldn't or wouldn't be
adopted in certain applications.

[1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-January/015614.html
[2] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-February/015700.html
[3] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-May/014308.html
[4] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-March/015838.html

Cheers,

--
Pieter

📅 Original date posted:2018-05-19 📝 Original message:On ...

2023-06-07T18:12:19Z

In reply to nevent1q…ufz6
_________________________

📅 Original date posted:2018-05-19
📝 Original message:On Fri, May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Greg wrote:
> > What about also making input prevouts filter based on the scriptpubkey
> being
> > _spent_? Layering wise in the processing it's a bit ugly, but if you
> > validated the block you have the data needed.
>
> AFAICT, this would mean that in order for a new node to catch up the filter
> index (index all historical blocks), they'd either need to: build up a
> utxo-set in memory during indexing, or would require a txindex in order to
> look up the prev out's script. The first option increases the memory load
> during indexing, and the second requires nodes to have a transaction index
> (and would also add considerable I/O load). When proceeding from tip, this
> doesn't add any additional load assuming that your synchronously index the
> block as you validate it, otherwise the utxo set will already have been
> updated (the spent scripts removed).
>

I was wondering about that too, but it turns out that isn't necessary. At
least in Bitcoin Core, all the data needed for such a filter is in the
block + undo files (the latter contain the scriptPubKeys of the outputs
being spent).

I have a script running to compare the filter sizes assuming the regular
> filter switches to include the prev out's script rather than the prev
> outpoint itself. The script hasn't yet finished (due to the increased I/O
> load to look up the scripts when indexing), but I'll report back once it's
> finished.
>

That's very helpful, thank you.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180518/3077040c/attachment-0001.html>;

📅 Original date posted:2018-03-26 📝 Original message:Hello, ...

2023-06-07T18:11:15Z

In reply to nevent1q…56fc
_________________________

📅 Original date posted:2018-03-26
📝 Original message:Hello,

Thanks for starting a discussion about this idea.

A few comments inline:

On Wed, Mar 14, 2018 at 1:09 AM, Karl Johan Alm via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Hello,
>
> I am considering writing a replacement for the message signing tools
> that are currently broken for all but the legacy 1xx addresses. The
> approach (suggested by Pieter Wuille) is to do a script based
> approach. This does not seem to require a lot of effort for
> implementing in Bitcoin Core*. Below is my proposal for this system:
>
> A new structure SignatureProof is added, which is a simple scriptSig &
> witnessProgram container that can be serialized. This is passed out
> from/into the signer/verifier.
>

You need a bit more logic to deal with softforks and compatibility. The
question is which script validation flags you verify with:
* If you make them fixed, it means signatures can't evolve with new address
types being introduced that rely on new features.
* If you make it just consensus flags (following mainnet), it means that
people with old software will see future invalid signatures as always
valid; this is probably not acceptable.
* If you make it standardness flags, you will get future valid signatures
that fail to verify.

One solution is to include a version number in the signature, which
explicitly corresponds to a set of validation flags. When the version
number is something a verifier doesn't know, it can be reported as
inconclusive (it's relying on features you don't know about).

An solution is to verify twice; once with all consensus rules you know
about, and once with standardness rules. If they're both valid, the
signature is valid. If they're both invalid, the signature is invalid. If
they're different (consensus valid but standardness invalid), you report
the signature validation as inconclusive (it's relying on features you
don't know about). This approach works as long as new features only use
previous standardness-invalid scripts, but perhaps a version number is
still needed to indicate the standardness flags.

RPC commands:
>
> sign <address> <message> [<prehashed>=false]
>

Why not extend the existing signmessage/verifymessage RPC? For legacy
addresses it can fall back to the existing signature algorithm, while using
the script-based approach for all others.

>
> Generates a signature proof for <message> using the same method that
> would be used to spend coins sent to <address>.**
>
> verify <address> <message> <proof> [<prehashed>=false]
>
> Deserializes and executes the proof using a custom signature checker
> whose sighash is derived from <message>. Returns true if the check
> succeeds, and false otherwise. The scriptPubKey is derived directly
> from <address>.**
>
> Feedback welcome.
>
> -Kalle.
>
>
(**) If <prehashed> is true, <message> is the sighash, otherwise
> sighash=sha256d(message).
>

That's very dangerous I'm afraid. It could be used to trick someone into
signing off on an actual transaction, if you get them to sign a "random
looking" prehashed message. Even if you have a prehashed message, there is
no problem with treating it as hex input to a second hashing step, so I
think the prehashed option isn't needed. It's why the existing message
signing functionality always forcibly prefixes "Bitcoin signed message:",
to avoid signing something that unintentionally corresponds to a message
intended for another goal.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180326/8324b4f3/attachment.html>;

📅 Original date posted:2018-01-09 📝 Original message:On Jan ...

2023-06-07T18:09:38Z

In reply to nevent1q…mfy3
_________________________

📅 Original date posted:2018-01-09
📝 Original message:On Jan 9, 2018 13:41, "Mark Friedenbach via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

The use of the alt stack is a hack for segwit script version 0 which has
the clean stack rule. Anticipated future improvements here are to switch to
a witness script version, and a new segwit output version which supports
native MAST to save an additional 40 or so witness bytes. Either approach
would allow getting rid of the alt stack hack. They are not part of the
proposal now because it is better to do things incrementally, and because
we anticipate usage of MAST to better inform these less generic changes.

If the goal is to introduce a native MAST output type later, what is gained
by first having the tailcall semantics?

As far as I can see, this proposal does not have any benefits over Johnson
Lau's MAST idea [1]:
* It is more compact, already giving us the space savings a native MAST
version of the tail call semantics would bring.
* It does not need to work around limitations of push size limits or
cleanstack rules.
* The implementation (of code I've seen) is easier to reason about, as it's
just another case in VerifyScript (which you'd need for a native MAST
output later too) without introducing jumps or loops inside EvalScript.
* It can express the same, as even though the MBV opcode supports proving
multiple elements simultaneously, I don't see a way to use that in the tail
call. Every scenario that consists of some logic before deciding what the
tail call is going to be can be rewritten to have that logic inside each of
the branches, I believe.
* It does not interfere with static analysis (see further).
* Tail call semantics may be easier to extend in the future to enable
semantics that are not compactly representable in either proposal right
now, by allowing a single top-level script may invoke multiple subscripts,
or recursion. However, those sound even riskier and harder to analyse to
me, and I don't think there is sufficient evidence they're needed.

Native MAST outputs would need a new witness script version, which your
current proposal does indeed not need. However, I believe a new script
version will be desirable for other reasons regardless (returning invalid
opcodes to the pool of NOPs available for softforks, for example).

I will make a strong assertion: static analyzing the number of opcodes and
sigops gets us absolutely nothing. It is cargo cult safety engineering. No
need to perpetuate it when it is now in the way.

I'm not sure I agree here. While I'd like to see the separate execution
limits go away, removing them entirely and complicating future ability to
introduce unified costing towards weight of execution cost seems the wrong
way to go.

My reasoning is this: perhaps you can currently make an argument that the
current weight limit is sufficient in preventing overly expensive block
validation costs, due to a minimal ratio between script sizes and their
execution cost. But such an argument needs to rely on assumptions about
sighash caching and low per-opcode CPU time, which may change in the
future. In my view, tail call semantics gratuitously remove or complicate
the ability to reason about the executed code.

One suggestion to reduce the impact of this is limiting the per-script
execution to something proportional to the script size. However, I don't
think that addresses all potential concerns about static analysis (for
example, it doesn't easily let you prove all possible execution paths to a
participant in a smart contract).

Another idea that has been suggested on this list is to mark pushes of
potentially executable code on the stack/witness explicitly. This would
retain all ability to analyse, while still leaving the flexibility of
future extensions to tail call execution. If tail call semantics are
adopted, I strongly prefer an approach like that to blindly throwing out
all limits and analysis.

[1] https://github.com/jl2012/bips/blob/mast/bip-mast.mediawiki

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180109/6c862414/attachment.html>;

📅 Original date posted:2017-10-30 📝 Original message:On Oct ...

2023-06-07T18:07:14Z

In reply to nevent1q…fn8d
_________________________

📅 Original date posted:2017-10-30
📝 Original message:On Oct 30, 2017 15:21, "shiva sitamraju via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

For example bc1qeklep85ntjz4605drds6aww9u0qr46qzrv5xswd35uhjuj8ahfcqgf6hak
in 461e8a4aa0a0e75c06602c505bd7aa06e7116ba5cd98fd6e046e8cbeb00379d6 is 62
bytes !

...

While I get the error/checksum capabilities Bech32 brings, any user would
prefer a 20 byte address with a checksum over an address that would wrap
several lines !!

That's an unfair comparison. You're pasting a P2WSH address which contains
a 256-bit hash.

A P2WPKH address (which only contains a 160-bit hash, just like P2PKH and
P2SH) in Bech32 is only 42 characters, not 62.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20171030/00d5cdc5/attachment.html>;

📅 Original date posted:2017-09-22 📝 Original message:On ...

2023-06-07T18:05:40Z

In reply to nevent1q…0fym
_________________________

📅 Original date posted:2017-09-22
📝 Original message:On Fri, Sep 22, 2017 at 2:54 PM, Sergio Demian Lerner via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> If the variable size increase is only a few bytes, then three
> possibilities arise:
>
> - one should allow signatures to be zero padded (to reach the maximum
> size) and abandon strict DER encoding
>
> - one should allow spare witness stack elements (to pad the size to match
> the maximum size) and remove the cleanstack rule. But this is tricky
> because empty stack elements must be counted as 1 byte.
>
> - signers must loop the generation of signatures until the signature
> generated is of its maximum size.
>

Or (my preference);

- Get rid of DER encoding alltogether and switch to fixed size signatures.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170922/5cb68030/attachment.html>;

📅 Original date posted:2017-07-11 📝 Original message:On ...

2023-06-07T18:04:13Z

In reply to nevent1q…chg6
_________________________

📅 Original date posted:2017-07-11
📝 Original message:On Tue, Jul 11, 2017 at 1:36 PM, Paul Sztorc <truthcoin at gmail.com> wrote:
> Pieter,
>
> I think that you have misrepresented Chris' view by taking it out of
> context. His complete quote reads "If drivechains are successful they should
> be viewed as the way we scale -- not hard forking the protocol." Chris is
> comparing Drivechains/sidechains to a hard fork.

I apologize here; I didn't mean to misrepresent his viewpoint.

> You went on to "disagree", but every point of contention you introduced was
> something that would apply to both drivechain-sourced capacity and
> hardfork-sourced capacity. Neither improves scalability, and both allow
> users only the opportunity to select a different security model. If I
> understand you, the point at which a security model does not become
> "interesting" to you, would be the exact same point in the drivechain and
> hardfork worlds. Both, at any rate, have the same effect on "validation cost
> to auditors".

If you're talking about the extreme case where every full node in the
increased capacity single chain model corresponds to a node that
validates both chains and all transfers between them in the
drivechains, I agree. At that point they become nearly equivalent in
terms of ease of adoption, resource costs, and capacity.

However, I don't think that is a realistic expectation. When
considering drivechains as a capacity increase, I believe most people
think about a situation where there are many chains that give an
increased capacity combined, but not everyone verifies all of them.
This is what I meant with uninteresting security model, as it requires
increased miner trust for preventing the other chains' coins from
being illegally transferred to the chain you're operating on.

Regardless, people are free experiment and adopt such an approach. The
nice thing about it not being a hardfork is that it does not require
network-wide consensus to deploy. However, I don't think they offer a
security model that should be encouraged, and thus doesn't have a
place on a roadmap.

> Since their sidechain coins cannot appreciate in value relative
> to the mainchain coins, users would only opt-in if they felt that they were
> sufficiently compensated for any and all risks. Hence, it is difficult to
> list this item as a drawback when, to the user, it is a strict improvement
> (at least, by any epistemological standard that I can think of). If you have
> new objections to these claims, I'm sure we would all benefit from hearing
> them, myself most of all.

Am I right in summarizing your point here as "This approach cannot
hurt, because if it were insecure, people can choose to not use it."?
I'm not sure I agree with that, as network effects or misinformation
may push users beyond what is reasonable.

Cheers,

--
Pieter

📅 Original date posted:2017-07-11 📝 Original message:On Jul ...

2023-06-07T18:04:12Z

In reply to nevent1q…88me
_________________________

📅 Original date posted:2017-07-11
📝 Original message:On Jul 11, 2017 09:18, "Chris Stewart via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

Concept ACK.

If drivechains are successful they should be viewed as the way we scale

I strongly disagree with that statement.

Drivechains, and several earlier sidechains ideas, are not a scalability
improvement, but merely enabling users to opt-in for another security model.

While obviously any future with wider adoption will need different
technologies that have different trade-offs, and anyone is free to choose
their security model, I don't think this particular one is interesting. In
terms of validation cost to auditors, it is as bad as just a capacity
increase on chain, while simultaneously adding the extra risk of miners
being able to vote to steal your money.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170711/6c7b8145/attachment.html>;

📅 Original date posted:2017-05-15 📝 Original message:Hello ...

2023-06-07T18:01:11Z

In reply to nevent1q…m0kt
_________________________

📅 Original date posted:2017-05-15
📝 Original message:Hello all,

I would like to discuss a way of computing a UTXO set hash that is
very efficient to update, but does not support any compact proofs of
existence or non-existence.

Much has been written on the topic of various data structures and
derived hashes for the UTXO/TXO set before (including Alan Reiner's
trust-free lite nodes [1], Peter Todd's TXO MMR commitments [2] [3],
or Bram Cohen's TXO bitfield [4]). They all provide interesting extra
functionality or tradeoffs, but require invasive changes to the P2P
protocol or how wallets work, or force nodes to maintain their
database in a normative fashion. Instead, here I focus on an efficient
hash that supports nothing but comparing two UTXO sets. However, it is
not incompatible with any of those other approaches, so we can gain
some of the advantages of a UTXO hash without adopting something that
may be incompatible with future protocol enhancements.

1. Incremental hashing

Computing a hash of the UTXO set is easy when it does not need
efficient updates, and when we can assume a fixed serialization with a
normative ordering for the data in it - just serialize the whole thing
and hash it. As different software or releases may use different
database models for the UTXO set, a solution that is order-independent
would seem preferable.

This brings us to the problem of computing a hash of unordered data.
Several approaches that accomplish this through incremental hashing
were suggested in [5], including XHASH, AdHash, and MuHash. XHASH
consists of first hashing all the set elements independently, and
XORing all those hashes together. This is insecure, as Gaussian
elimination can easily find a subset of random hashes that XOR to a
given value. AdHash/MuHash are similar, except addition/multiplication
modulo a large prime are used instead of XOR. Wagner [6] showed that
attacking XHASH or AdHash is an instance of a generalized birthday
problem (called the k-sum problem in his paper, with unrestricted k),
and gives a O(2^(2*sqrt(n)-1)) algorithm to attack it (for n-bit
hashes). As a result, AdHash with 256-bit hashes only has 31 bits of
security.

Thankfully, [6] also shows that the k-sum problem cannot be
efficiently solved in groups in which the discrete logarithm problem
is hard, as an efficient k-sum solver can be used to compute discrete
logarithms. As a result, MuHash modulo a sufficiently large safe prime
is provably secure under the DL assumption. Common guidelines on
security parameters [7] say that 3072-bit DL has about 128 bits of
security. A final 256-bit hash can be applied to the 3072-bit result
without loss of security to reduce the final size.

An alternative to multiplication modulo a prime is using an elliptic
curve group. Due to the ECDLP assumption, which the security of
Bitcoin signatures already relies on, this also results in security
against k-sum solving. This approach is used in the Elliptic Curve
Multiset Hash (ECMH) in [8]. For this to work, we must "hash onto a
curve point" in a way that results in points without known discrete
logarithm. The paper suggests using (controversial) binary elliptic
curves to make that operation efficient. If we only consider
secp256k1, one approach is just reading potential X coordinates from a
PRNG until one is found that has a corresponding Y coordinate
according to the curve equation. On average, 2 iterations are needed.
A constant time algorithm to hash onto the curve exists as well [9],
but it is only slightly faster and is much more complicated to
implement.

AdHash-like constructions with a sufficiently large intermediate hash
can be made secure against Wagner's algorithm, as suggested in [10].
4160-bit hashes would be needed for 128 bits of security. When
repetition is allowed, [8] gives a stronger attack against AdHash,
suggesting that as much as 400000 bits are needed. While repetition is
not directly an issue for our use case, it would be nice if
verification software would not be required to check for duplicated
entries.

2. Efficient addition and deletion

Interestingly, both ECMH and MuHash not only support adding set
elements in any order but also deleting in any order. As a result, we
can simply maintain a running sum for the UTXO set as a whole, and
add/subtract when creating/spending an output in it. In the case of
MuHash it is slightly more complicated, as computing an inverse is
relatively expensive. This can be solved by representing the running
value as a fraction, and multiplying created elements into the
numerator and spent elements into the denominator. Only when the final
hash is desired, a single modular inverse and multiplication is needed
to combine the two.

As the update operations are also associative, H(a)+H(b)+H(c)+H(d) can
in fact be computed as (H(a)+H(b)) + (H(c)+H(d)). This implies that
all of this is perfectly parallellizable: each thread can process an
arbitrary subset of the update operations, allowing them to be
efficiently combined later.

3. Comparison of approaches

Numbers below are based on preliminary benchmarks on a single thread
of a i7-6820HQ CPU running at 3.4GHz.

(1) (MuHash) Multiplying 3072-bit hashes mod 2^3072 - 1103717 (the
largest 3072-bit safe prime).
* Needs a fast modular multiplication/inverse implementation.
* Using SHA512 + ChaCha20 for generating the hashes takes 1.2us per element.
* Modular multiplication using GMP takes 1.5us per element (2.5us
with a 60-line C+asm implementation).
* 768 bytes for maintaining a running sum (384 for numerator, 384
for denominator)
* Very common security assumption. Even if the DL assumption would
be broken (but no k-sum algorithm faster than Wagner's is found), this
still maintains 110 bits of security.

(2) (ECMH) Adding secp256k1 EC points
* Much more complicated than the previous approaches when
implementing from scratch, but almost no extra complexity when ECDSA
secp256k1 signature validation is already implemented.
* Using SHA512 + libsecp256k1's point decompression for generating
the points takes 11us per element on average.
* Addition/subtracting of N points takes 5.25us + 0.25us*N.
* 64 bytes for a running sum.
* Identical security assumption as Bitcoin's signatures.

Using the numbers above, we find that:
* Computing the hash from just the UTXO set takes (1) 2m15s (2) 9m20s
* Processing all creations and spends in an average block takes (1)
24ms (2) 100ms
* Processing precomputed per-transaction aggregates in an average
block takes (1) 3ms (2) 0.5ms

Note that while (2) has higher CPU usage than (1) in general, it has
lower latency when using precomputed per-transaction aggregates. Using
such aggregates is also more feasible as they're only 64 bytes rather
than 768. Because of simplicity, (1) has my preference.

Overall, these numbers are sufficiently low (note that they can be
parallellized) that it would be reasonable for full nodes and/or other
software to always maintain one of them, and effectively have a
rolling cryptographical checksum of the UTXO set at all times.

4. Use cases

* Replacement for Bitcoin Core's gettxoutsetinfo RPC's hash
computation. This currently requires minutes of I/O and CPU, as it
serializes and hashes the entire UTXO set. A rolling set hash would
make this instant, making the whole RPC much more usable for sanity
checking.
* Assisting in implementation of fast sync methods with known good
blocks/UTXO sets.
* Database consistency checking: by remembering the UTXO set hash of
the past few blocks (computed on the fly), a consistency check can be
done that recomputes it based on the database.

[1] https://bitcointalk.org/index.php?topic=88208.0
[2] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-May/012715.html
[3] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-February/013591.html
[4] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-March/013928.html
[5] https://cseweb.ucsd.edu/~mihir/papers/inchash.pdf
[6] https://people.eecs.berkeley.edu/~daw/papers/genbday.html
[7] https://www.keylength.com/
[8] https://arxiv.org/pdf/1601.06502.pdf
[9] https://www.di.ens.fr/~fouque/pub/latincrypt12.pdf
[10] http://csrc.nist.gov/groups/ST/hash/sha-3/Aug2014/documents/gligoroski_paper_sha3_2014_workshop.pdf

Cheers,

--
Pieter

📅 Original date posted:2017-05-07 📝 Original message:On ...

2023-06-07T18:00:53Z

In reply to nevent1q…a3he
_________________________

📅 Original date posted:2017-05-07
📝 Original message:On Mon, Mar 20, 2017 at 2:35 PM, Pieter Wuille <pieter.wuille at gmail.com>
wrote:

> Hello everyone,
>
> You can find the text here:
> https://github.com/sipa/bech32/blob/master/bip-witaddr.mediawiki
>

Responding to a few comments:

By Andreas Schildbach:

> I'm not convinced that transmitting addresses via voice should be a
usecase to target at

I think it should be. It's certainly not the most important way through
which addresses are communicated or verified, but I am trying to address
all places where humans interact with addresses. I have certainly tried to
verify addresses a few times through voice, when dealing with significant
amounts.

Regarding your QR code comments: it is certainly possible to find a more
compact QR code representation. That is not the goal of the BIP though -
it's trying to introduce one commonly recognizable format that has good
properties for all use cases, even if that means being suboptimal in
certain aspects for some.

> I don't understand your comment about non-english speaking users.
Obviously they cannot voice-communicate at all with only-english-speaking
users, so there is no need to communicate voice-communicate addresses
between them.

I assume that Peter Todd is talking about cases where English speakers are
interacting with non-native English speakers, who may know how to pronounce
numbers or alphabetical characters, but not all special characters.

> Speaking of URLs, actually Base 32 (as well as Base 43) makes QR codes
*bigger* because due to the characters used for URL parameters (?&=) those
QR codes are locked to binary mode.

I believe that is incorrect. Data in QR codes can switch from one mode to
another on a per-character basis (with an overhead of a few bits). I don't
know to what extent common QR encoders make intelligent decisions about
this, but it does not seem very hard.

By Lucas Ontivero:

> Here I think it could worth to mention that 58 requires mathematical
operations over big numbers. This is not very fast and most of the
programming languages don't provide support for big numbers OOB.

It's not that hard to emulate the bignum logic in languages that don't
support it. See for example this code in Bitcoin Core:
https://github.com/bitcoin/bitcoin/blob/v0.14.1/src/base58.cpp#L37L53. So I
think it's not necessary to go into all the possible ways Base58 can be
implemented in the document, and the existing language ("Base58 decoding is
complicated and relatively slow.") is sufficient.

> I understand that if a new generic encoding format is introduced that
could lead to some confusions but what if in the future there is a new type
of address that can also be encoded with bech32? Don't we need a address
type anyway?

I believe that it's likely that new types of outputs that may be introduced
in the future will most likely not be a simple constant byte sequence that
can be computed directly from addresses, but need some processing by the
sender. This is the case for example for Reusable/Stealth addresses and
Confidential Transactions addresses. Such outputs, if ever introduced on a
wide scale, should ideally not be representable as existing address types,
as that could not only lead to confusion, but also to lost privacy and
funds.

And, If there ever is a need for introducing a "constant scriptPubKey" type
address again, the encoding proposed in this document can be reused.
Currently, the header value can be at most 17. In the future new proposals
could give a meaning to values 18 through 31.

In general:

In the past weeks people have contributed two new reference implementations
(Haskell and Rust), and a C++ and Go one are underway (see
https://github.com/sipa/bech32).

I'd like to move forward and request a BIP number assignment for this
proposal.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170507/926b2de1/attachment.html>;

📅 Original date posted:2017-03-28 📝 Original message:On ...

2023-06-07T17:58:13Z

In reply to nevent1q…wne8
_________________________

📅 Original date posted:2017-03-28
📝 Original message:On Tue, Mar 28, 2017 at 12:56 PM, Paul Iverson via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> So I think Core can't decide on hard forks like this. It must be left up to
> the users. I think only choice is for Core to add a run-time option to allow
> node operators to increase block size limit, so that this very controversial
> decision is not coming from Core. It must come from the community.

Bitcoin Core's (nor any other software's) maintainers can already not
decide on a hard fork, and I keep being confused by the focus on Core
in this topic. Even if a hard forking change (or lack thereof) was
included into a new release, it is still up to the community to choose
to run the new software. Bitcoin Core has very intentionally no
auto-update feature, as the choice for what network rules to implement
must come from node operators, not developers. Ask yourself this: if a
new Bitcoin Core release would include a new rule that blacklists
<random famous person>'s coins. What do you think would happen? I hope
that people would refuse to update, and choose to run different full
node software.

Core is not special. It is one of many pieces of software that
implement today's Bitcoin consensus rules. If a hardfork is to take
place in a way that does not result in two currencies, it must be
clear that the entire ecosystem will adopt it. Bitcoin Core will not
merge any consensus changes that do not clearly satisfy that
criterion.

--
Pieter

📅 Original date posted:2017-03-23 📝 Original message:On ...

2023-06-07T17:57:43Z

In reply to nevent1q…psuj
_________________________

📅 Original date posted:2017-03-23
📝 Original message:On Thu, Mar 23, 2017 at 3:37 PM, Juan Garavaglia via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> Long story short, when nodes 0.13+ receive blocks from 0.13+ nodes all is
> ok, and those blocks propagate to older nodes with no issues. But when a
> block tries to be propagated from bitcoind 0.12.+ to newer ones those blocks
> are NOT being propagated to the peers with newer versions while these newer
> blocks are being propagated to peers with older versions with no issues.
>
> My conclusion is that we have a backward compatibility issue between 0.13.X+
> and older versions.

Hello Juan,

this is expected behaviour. Nodes with segwit active only download
blocks from other segwit peers, as old peers cannot provide the
witness data they need to verify the blocks.

--
Pieter

📅 Original date posted:2017-03-20 📝 Original message:Hello ...

2023-06-07T17:57:37Z

In reply to nevent1q…e7pd
_________________________

📅 Original date posted:2017-03-20
📝 Original message:Hello everyone,

I'd like to propose a new BIP for native segwit addresses to replace
BIP 142. These addresses are not required for segwit, but are more
efficient, flexible, and nicer to use.

The format is base 32 and uses a simple checksum algorithm with strong
error detection properties. Reference code in several languages as
well as a website demonstrating it are included.

You can find the text here:
https://github.com/sipa/bech32/blob/master/bip-witaddr.mediawiki

Cheers,

--
Pieter

📅 Original date posted:2017-02-26 📝 Original message:On Feb ...

2023-06-07T17:56:47Z

In reply to nevent1q…z9dh
_________________________

📅 Original date posted:2017-02-26
📝 Original message:On Feb 25, 2017 22:26, "Steve Davis" <steven.charles.davis at gmail.com> wrote:

Hi Pieter,

> On Feb 25, 2017, at 4:14 PM, Pieter Wuille <pieter.wuille at gmail.com>
wrote:
>
> Any alternative to move us away from RIPEMD160 would require:

> <snipped>

“Any alternative”? What about reverting to:

[<public_key>, OP_CHECKSIG]

snip

Could that be the alternative?

Ok, fair enough, that is an alternative that avoids the 160-bit hash
function, but not where it matters. The 80-bit collision attack only
applies to jointly constructed addresses like multisig P2SH, not single-key
ones. As far as I know for those we only rely preimage security, and
RIPEMD160 has 160 bit security there, which is even more than our ECDSA
signatures offer.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170225/6f7d3907/attachment.html>;

📅 Original date posted:2017-02-25 📝 Original message:On Feb ...

2023-06-07T17:56:46Z

In reply to nevent1q…nwu6
_________________________

📅 Original date posted:2017-02-25
📝 Original message:On Feb 25, 2017 14:09, "Steve Davis via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

Hi Peter,

I really, really don’t want to get into it but segwit has many aspects that
are less appealing, not least of which being the amount of time it would
take to reach the critical mass.

Surely there's a number of alternative approaches which could be explored,
even if only to make a fair assessment of a best response?

Any alternative to move us away from RIPEMD160 would require:
* A drafting of a softfork proposal, implementation, testing, review.
* A new address format
* Miners accepting the new consensus rules
* Wallets adopting the new address format, both on the sender side and
receiver side (which requires new signatures).

I.e., exactly the same as segwit, for which most of these are already done.
And it would still only apply to wallets adopting it.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170225/e3856947/attachment-0001.html>;

📅 Original date posted:2016-12-10 📝 Original message:On ...

2023-06-07T17:54:54Z

In reply to nevent1q…d3a8
_________________________

📅 Original date posted:2016-12-10
📝 Original message:On Sat, Dec 10, 2016 at 4:23 AM, Daniele Pinna via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> We have models for estimating the probability that a block is orphaned
> given average network bandwidth and block size.
>
> The question is, do we have objective measures of these two quantities?
> Couldn't we target an orphan_rate < max_rate?
>

Models can predict orphan rate given block size and network/hashrate
topology, but you can't control the topology (and things like FIBRE hide
the effect of block size on this as well). The result is that if you're
purely optimizing for minimal orphan rate, you can end up with a single
(conglomerate of) pools producing all the blocks. Such a setup has no
propagation delay at all, and as a result can always achieve 0 orphans.

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20161210/edca6d6b/attachment.html>;

📅 Original date posted:2016-11-16 📝 Original message:On ...

2023-06-07T17:54:23Z

In reply to nevent1q…gszf
_________________________

📅 Original date posted:2016-11-16
📝 Original message:On Wed, Nov 16, 2016 at 5:58 AM, Eric Voskuil via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> This sort of statement represents one consequence of the aforementioned
> bad precedent.
>
> Are checkpoints good now?
>

So far in this discussion, and in a thread that has forked off, I think 3
cases of implementation aspects have been mentioned that under certain
circumstances result in the validity of chains changing:
* Buried softforks (by simplifying the activation rules for certain rules)
* Not verifying BIP30 after BIP34 is active (since only under a SHA256^2
collision a duplicate txid can occur)
* The existence (and/or removal) of checkpoints (in one form or another).

None of these will influence the accepted main chain, however. If they ever
do, Bitcoin has far worse things to worry about (years-deep reorgs, or
SHA256 collisions).

If you were trying to point out that buried softforks are similar to
checkpoints in this regard, I agree. So are checkpoints good now? I believe
we should get rid of checkpoints because they seem to be misunderstood as a
security feature rather than as an optimization. I don't think buried
softforks have that problem.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20161116/8b6f55ed/attachment.html>;

📅 Original date posted:2016-10-16 📝 Original message:Hello ...

2023-06-07T17:54:00Z

In reply to nevent1q…wfe0
_________________________

📅 Original date posted:2016-10-16
📝 Original message:Hello all,

We're getting ready for Bitcoin Core's 0.13.1 release - the first one
to include segregated witness (BIP 141, 143, 144, 145) for Bitcoin
mainnet, after being extensively tested on testnet and in other
software. Following the BIP9 recommendation [1] to set the versionbits
start time a month in the future and discussion in the last IRC
meeting [2], I propose we set BIP 141's start time to November 15,
2016, 0:00 UTC (unix time 1479168000).

Note that this is just a lower bound on the time when the versionbits
signalling can begin. Activation on the network requires:
(1) This date to pass
(2) A full retarget window of 2016 blocks with 95% signalling the
version bit (bit 1 for BIP141)
(3) A fallow period consisting of another 2016 blocks.

[1] https://github.com/bitcoin/bips/blob/master/bip-0009.mediawiki
[2] http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-10-13-19.04.log.html

Cheers,

--
Pieter

📅 Original date posted:2016-08-16 📝 Original message:On Aug ...

2023-06-07T17:52:57Z

In reply to nevent1q…myna
_________________________

📅 Original date posted:2016-08-16
📝 Original message:On Aug 17, 2016 00:36, "Russell O'Connor" <roconnor at blockstream.io> wrote:

> Can I already do something similar with replace by fee, or are there
limits on that?

BIP125 and mempool eviction both require the replacing transaction to have
higher fee, to compensate for the cost of relaying the replaced
transaction(s).

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160817/3710f5b7/attachment.html>;

📅 Original date posted:2016-08-16 📝 Original message:On Aug ...

2023-06-07T17:52:56Z

In reply to nevent1q…9aux
_________________________

📅 Original date posted:2016-08-16
📝 Original message:On Aug 17, 2016 00:23, "Russell O'Connor via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> If one's goal is to mess with an transaction to prevent it from being
mined, it is more effective to just not relay the transaction rather than
to mess with the witness. Given two transactions with the same txid and
different witness data, miners and good nodes ought to mine/relay the
version with the lower cost (smaller?) witness data.

That implies that everyone will see both versions and be able to make that
choice. Unfortunately, those two versions will be definition be in conflict
with each other, and thus only one will end up paying a fee. We're can't
relay two transactions for the price of one, or we'd expose the p2p network
to a very cheap DDoS attack: just send increasingly small versions of the
same transaction.

Segwit's third party mallebility protection makes it not an issue for
dependent contracts if transactions are mauled (=apparently the verb
related to malleability), but there are still good reasons for senders not
to gratuitously make their transactions extensible in size or other
resources.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160817/2c5284a7/attachment.html>;

📅 Original date posted:2016-08-10 📝 Original message:On ...

2023-06-07T17:52:37Z

In reply to nevent1q…l794
_________________________

📅 Original date posted:2016-08-10
📝 Original message:On Wed, Aug 10, 2016 at 7:28 PM, Erik Aronesty via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> By sending a public seed, there's no way for someone to use the transmitted
> address and trace the total amount of payments to it.

Worse. By revealing a public seed, anyone who has seen it (= anyone
who ever pays you through it) can identity all payments to _any_
address derived from that seed.

--
Pieter

📅 Original date posted:2016-06-30 📝 Original message:On ...

2023-06-07T17:51:47Z

In reply to nevent1q…4r38
_________________________

📅 Original date posted:2016-06-30
📝 Original message:On Thu, Jun 30, 2016 at 11:57 AM, Eric Voskuil via bitcoin-dev
<bitcoin-dev at lists.linuxfoundation.org> wrote:
> The proliferation of node identity is my primary concern - this relates to privacy and the security of the network.

I think this is a reasonable concern.

However, node identity is already being used widely, and in a very
inadvisable way:
* Since forever there have been lists of 'good nodes' to pass in
addnode= configuration options.
* Various people run multiple nodes in different geographic locations,
peering with each other.
* Various pieces of infrastructure exist that relies on connecting to
well-behaving nodes (miner relay networks, large players peering
directly with each other, ...)
* Several lightweight clients support configuring a trusted host to connect to.

Perhaps you deplore that fact, but I believe it is inevitable that
different pieces of the network will make different choices here. You
can'tg prevent people from create connections along preexisting trust
lines. That does not mean that the network as a whole relies on first
establishing trust everywhere.

And I do think there are advantages.

BIP 151 on its own gives you opportunistic encryption. You're very
right to point out that this does not give you protection from active
attackers, and that active attacking is relatively easy through sybil
attacks. I still prefer my attacker to actually do that over just
listening in on my connection. And yes, we should also work on
improving the privacy nodes and wallets have orthogonal to encryption,
but nothing will make everything perfectly private.

BIP 151 plus a simple optional pre-shared-secret authentication
extension can improve upon pure IP-based authentication, as well as
simplify things like SSL tunnels, and onion addresses purely used as
identity. This will still require explicit configuration, but not more
than now.

BIP 151 plus a non-leaking public key authentication scheme (where
peers can query "are you the peer with pubkey X?" but don't learn
anything if the answer is no) with keys specific to the IP addresses
can give a TOFU-like security. Nodes already remember IP addresses
they've succesfully interacted with in the past, and ban IP addresses
that misbehave. Being able to tell whether a node you connect to is
the same as one you've connected to before is a natural extension of
this, and does not require establishing any real-world identity beyond
what we're already implicitly relying on.

Perhaps these use cases and their security assumptions should be
spelled out more clearly in the BIP. If there is a misunderstanding,
it should be clearly stated that BIP 151 is only a building block for
further improvements

> Secondarily I am concerned about users operating under a false assumption about the strength of privacy.

This is a widespread problem, but it exists far outside the scope of
this proposal. The privacy properties of Bitcoin are often
misrepresented and even used as advertizements. The solution is
education, not avoiding improvements because they may be
misunderstood.

> The complexity of the proposed construction is comparable to that of Bitcoin itself.

I really think this is an exaggeration. It's a diffie-hellman
handshake and a stream cipher (both very common constructions), that
apply to individual connections. There are no consensus risks nor a
requirement for coordinated change through the network. The
cryptographic code can be directly reused from a well-known project
(OpenSSH), and is very small in size.

--
Pieter

📅 Original date posted:2016-06-29 📝 Original message:On Jun ...

2023-06-07T17:51:35Z

In reply to nevent1q…v7wm
_________________________

📅 Original date posted:2016-06-29
📝 Original message:On Jun 29, 2016 07:05, "Ethan Heilman via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:
>
> >It's also not clear to me why the HMAC, vs just
SHA256(key|cipher-type|mesg). But that's probably just my crypto
ignorance...
>
> SHA256(key|cipher-type|mesg) is an extremely insecure MAC because of
> the length extension property of SHA256.

This property does technically not apply here, as the output of the hash is
kept secret, and the possible messages are constants (which are presumably
chosen in such a way that one is never an extension of another).

However, this is a good example of why you can't generically use a hash
function in places where you want a MAC (aka "a hash with a shared
secret"). Furthermore, if you already have a hash function anyway, HMAC is
very easy construct on top of it.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160629/72f9c103/attachment.html>;

📅 Original date posted:2016-06-23 📝 Original message:On Jun ...

2023-06-07T17:51:23Z

In reply to nevent1q…8h6g
_________________________

📅 Original date posted:2016-06-23
📝 Original message:On Jun 23, 2016 14:10, "Peter Todd" <pete at petertodd.org> wrote:

> Right, so you accept that we'll exert some degree of editorial control;
the
> question now is what editorial policies should we exert?

No, I do not. I am saying that some degree of editorial control will
inevitably exist, simply because there is some human making the choice of
assigning a BIP number and merging. My opinion is that we should try to
restrict that editorial control to only be subject to objective process,
and not be dependent on personal opinions.

> My argument is that rejecting BIP75 is something we should do on
> ethical/strategic grounds. You may disagree with that, but please don't
troll
> and call that "advocating censorship"

I think that you are free to express dislike of BIP75. Suggesting to remove
it for that reason is utterly ridiculous to me, whatever you want to call
it.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160623/ba7a229f/attachment.html>;

📅 Original date posted:2016-06-23 📝 Original message:On Jun ...

2023-06-07T17:51:22Z

In reply to nevent1q…x8ln
_________________________

📅 Original date posted:2016-06-23
📝 Original message:On Jun 23, 2016 12:56, "Peter Todd via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> In any case, I'd strongly argue that we remove BIP75 from the bips
repository,
> and boycott wallets that implement it. It's bad strategy for Bitcoin
developers
> to willingly participate in AML/KYC, just the same way as it's bad for
Tor to
> add wiretapping functionality, and W3C to support DRM tech. The minor
tactical
> wins you'll get our of this aren't worth it.

I hope you're not seriously suggesting to censor a BIP because you feel it
is a bad idea.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160623/b6c34061/attachment.html>;

📅 Original date posted:2016-05-09 📝 Original message:On ...

2023-06-07T17:50:20Z

In reply to nevent1q…dskn
_________________________

📅 Original date posted:2016-05-09
📝 Original message:On 05/03/2016 12:13 AM, lf-lists at mattcorallo.com (Matt Corallo) wrote:
> Hi all,
>
> The following is a BIP-formatted design spec for compact block relay
> designed to limit on wire bytes during block relay. You can find the
> latest version of this document at
> https://github.com/TheBlueMatt/bips/blob/master/bip-TODO.mediawiki.

Hi Matt,

thank you for working on this!

> ===New data structures===
> Several new data structures are added to the P2P network to relay
> compact blocks: PrefilledTransaction, HeaderAndShortIDs,
> BlockTransactionsRequest, and BlockTransactions. Additionally, we
> introduce a new variable-length integer encoding for use in these data
> structures.
>
> For the purposes of this section, CompactSize refers to the
> variable-length integer encoding used across the existing P2P protocol
> to encode array lengths, among other things, in 1, 3, 5 or 9 bytes.

This is a not, but I think it's a bit strange to have two separate
variable length integers in the same specification. I understand is one
is already the default for variable-length integers currently, and there
are reasons to use the other one for efficiency reasons in some places,
but perhaps we should aim to get everything using the latter?

> ====New VarInt====
> Variable-length integers: bytes are a MSB base-128 encoding of the number.
> The high bit in each byte signifies whether another digit follows. To make
> sure the encoding is one-to-one, one is subtracted from all but the last
> digit.

Maybe it's worth mentioning that it is based on ASN.1 BER's compressed
integer format (see
https://www.itu.int/ITU-T/studygroups/com17/languages/X.690-0207.pdf
section 8.1.3.5), though with a small modification to make every integer
have a single unique encoding.

> ====HeaderAndShortIDs====
> A HeaderAndShortIDs structure is used to relay a block header, the short
> transactions IDs used for matching already-available transactions, and a
> select few transactions which we expect a peer may be missing.
>
> |shortids||List of uint64_ts||8*shortids_length bytes||Little
> Endian||The short transaction IDs calculated from the transactions which
> were not provided explicitly in prefilledtxn

I tried to derive what length of short ids is actually necessary (some
write-up is on
https://gist.github.com/sipa/b2eb2e486156b5509ac711edd16153ed but it's
incomplete).

For any reasonable numbers I can come up with (in a very wide range),
the number of bits needed is very well approximated by:

log2(#receiver_mempool_txn * #block_txn_not_in_receiver_mempool /
acceptable_per_block_failure_rate)

For example, with 20000 mempool transactions, 2500 transactions in a
block, 95% hitrate, and a chance of 1 in 10000 blocks to fail to
reconstruct, needed_bits = log2(20000 * 2500 * (1 - 0.95) / 0.0001) =
34.54, or 5 byte txids would suffice.

Note that 1 in 10000 failures may sound like a lot, but this is for each
individual connection, and since every transmission uses separately
salted identifiers, occasional failures should not affect global
propagation. Given that transmission failures due to timeouts, network
connectivity, ... already occur much more frequently than once every few
gigabytes (what 10000 blocks corresponds to), that's probably already
more than enough.

In short: I believe 5 or 6 byte txids should be enough, but perhaps it
makes sense to allow the sender to choose (so he can weigh trying
multiple nonces against increasing the short txid length).

> ====Short transaction IDs====
> Short transaction IDs are used to represent a transaction without
> sending a full 256-bit hash. They are calculated by:
> # single-SHA256 hashing the block header with the nonce appended (in
> little-endian)
> # XORing each 8-byte chunk of the double-SHA256 transaction hash with
> each corresponding 8-byte chunk of the hash from the previous step
> # Adding each of the XORed 8-byte chunks together (in little-endian)
> iteratively to find the short transaction ID

An alternative would be using SipHash-1-3 (a form of SipHash with
reduced iteration counts; the default is SipHash-2-4). SipHash was
designed as a Message Authentication Code, where the security
requirements are much stronger than in our case (in particular, we don't
care about observers being able to finding the key, as the key is just
public knowledge here). One of the designers of SipHash has commented
that SipHash-1-3 for collision resistance in hash tables may be enough:
https://github.com/rust-lang/rust/issues/29754#issuecomment-156073946

Using SipHash-1-3 on modern hardware would take ~32 CPU cycles per txid.

> ===Implementation Notes===

There are a few more heuristics that MAY be used to improve performance:

* Receivers should treat short txids in blocks that match multiple
mempool transactions as non-matches, and request the transactions. This
significantly reduces the failure to reconstruct.

* When constructing a compact block to send, the sender can verify it
against its own mempool to check for collisions, and if so, choose to
either try another nonce, or increase the short txid length.

Cheers,

--
Pieter

📅 Original date posted:2016-01-07 📝 Original message:> ...

2023-06-07T17:47:42Z

In reply to nevent1q…ksqt
_________________________

📅 Original date posted:2016-01-07
📝 Original message:> "The problem case is where someone in a contract setup shows you a
script, which you accept as being a payment to yourself. An attacker could
use a collision attack to construct scripts with identical hashes, only one
of which does have the property you want, and steal coins.
>
> So you really want collision security, and I don't think 80 bits is
something we should encourage for that. Normal pubkey hashes don't have
that problem, as they can't be constructed to pay to you."
>
> ... but I'm unconvinced:
>
> "But it is trivial for contract wallets to protect against collision
attacks-- if you give me a script that is "gavin_pubkey CHECKSIG
arbitrary_data OP_DROP" with "I promise I'm not trying to rip you off, just
ignore that arbitrary data" a wallet can just refuse. Even more likely, a
contract wallet won't even recognize that as a pay-to-gavin transaction.
>
> I suppose it could be looking for some form of "gavin_pubkey
somebody_else_pubkey CHECKMULTISIG ... with the attacker using
somebody_else_pubkey to force the collision, but, again, trivial contract
protocol tweaks ("send along a proof you have the private key corresponding
to the public key" or "everybody pre-commits pubkeys they'll use at
protocol start") would protect against that.

Yes, this is what I worry about. We're constructing a 2-of-2 multisig
escrow in a contract. I reveal my public key A, you do a 80-bit search for
B and C such that H(A and B) = H(B and C). You tell me your keys B, and I
happily send to H(A and B), which you steal with H(B and C).

Sending along a proof does not help, you can't prove that you do not know
of a collision. Pre-committing does help, but is a very non-obvious
security requirement, something I strongly believe is far riskier in
practice.

Bitcoin does have parts that rely on economic arguments for security or
privacy, but can we please stick to using cryptography that is up to par
for parts where we can? It's a small constant factor of data, and it
categorically removes the worry about security levels.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160108/e2e921fb/attachment.html>;

📅 Original date posted:2015-12-16 📝 Original message:On ...

2023-06-07T17:46:40Z

In reply to nevent1q…533z
_________________________

📅 Original date posted:2015-12-16
📝 Original message:On Wed, Dec 16, 2015 at 10:27 PM, Jeff Garzik <jgarzik at gmail.com> wrote:
>> Not correct. I propose defining the virtual_block_size as base_size +
>> witness_size * 0.25, and limiting virtual_block_size to 1M. This
>> creates a single variable to optimize for. If accepted, miners are
>> incentived to maximize fee per virtual_block_size instead of per size.
>
>
> It is correct. There are two separate sets of economic actors and levels of
> contention for each set of space.
>
> That is true regardless of the proposed miner selection algorithm.

Maybe I haven't explained this properly, so consider this example:

A miner receives sets of 200 byte transactions with all identical
fees. Non-witness ones (whose virtual size is thus 200 bytes) and a
witness one (where 120 of the 200 bytes are witness data, so its
virtual size is 80 + 120*0.25 = 110 bytes).

The consensus rules would limit 1) the base size to 1000000 bytes 2)
the virtual size to 1000000 bytes. However, as the virtual size is
defined as the sum of the base size plus a non-negative number,
satisfying (2) always implies satisfying (1).

Thus, the miners' best strategy is to accept the witness transactions,
as it allows 1000000/110=9090 transactions rather than
1000000/200=5000.

In fact, the optimal fee maximizing strategy is always to maximize fee
per virtual size.

--
Pieter

📅 Original date posted:2015-12-26 📝 Original message:On Dec ...

2023-06-07T17:46:27Z

In reply to nevent1q…6psd
_________________________

📅 Original date posted:2015-12-26
📝 Original message:On Dec 26, 2015 23:55, "Jonathan Toomim" <j at toom.im> wrote:
>
> On Dec 26, 2015, at 8:44 AM, Pieter Wuille via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
>>
>> Furthermore, 75% is pretty terrible as a switchover point, as it
guarantees that old nodes will still see a 25% forked off chain temporarily.
>
> Yes, 75% plus a grace period is better. I prefer a grace period of about
4000 to 8000 blocks (1 to 2 months).

I think that's extremely short, even assuming there is no controversy about
changing the rules at all. Things like BIP65 and BIP66 already took
significantly longer than that, were uncontroversial, and only need miner
adoption. Full node adoption is even slower.

I think the shortest reasonable timeframe for an uncontroversial hardfork
is somewhere in the range between 6 and 12 months.

For a controversial one, not at all.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151227/9446849d/attachment.html>;

📅 Original date posted:2015-12-26 📝 Original message:On Dec ...

2023-06-07T17:46:27Z

In reply to nevent1q…37cl
_________________________

📅 Original date posted:2015-12-26
📝 Original message:On Dec 27, 2015 00:06, "Jonathan Toomim" <j at toom.im> wrote:

> Given that a supermajority of users and miners have been asking for a
hard fork to increase the blocksize for years, I do not think that
mobilizing people to upgrade their nodes is going to be hard.
>
> When we do the hard fork, we will need to encourage people to upgrade
their full nodes. We may want to request that miners not trigger the fork
until some percentage of visible full nodes have upgraded.

I am generally not interested in a system where we rely on miners to make
that judgement call to fork off nodes that don't pay attention and/or
disagree with the change. This is not because I don't trust them, but
because I believe one of the principle values of the system is that its
consensus system should be hard to change.

I can't tell you what code to run of course, but I can decide what system I
find interesting to build. And it seems many people have signed off on
working towards a plan that does not include a hard fork being scheduled
right now: https://bitcoin.org/en/bitcoin-core/capacity-increases

Cheers,

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151227/96ee711e/attachment.html>;

📅 Original date posted:2015-12-18 📝 Original message:On Dec ...

2023-06-07T17:46:23Z

In reply to nevent1q…ech7
_________________________

📅 Original date posted:2015-12-18
📝 Original message:On Dec 18, 2015 2:13 AM, "sickpig at gmail.com" <sickpig at gmail.com> wrote:
> 1.75 x 0.5 + 1 x 0.5 = 1.375
>
> after six month.
>
> An hard-fork on the others side would bring 1.75 since the activation, am
I right?

Yes.

However, SW immediately gives a 1.75 capacity increase for anyone who
adopts it, after the softfork, instantly. They don't need to wait for
anyone else.

A hard fork is an orthogonal improvement, which is also needed if we don't
want to be stuck with a constant maximum ultimately.

Hardforks can however only be deployed at a time when all full node
software can reasonably have agreed to upgrade, while a softfork can be
deployed much earlier.

They are independent improvements, and we need both. I am however of the
opinion that hard forks need a much clearer consensus and much longer
rollout timeframes to be safe (see my thread on the security of softforks).

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151218/5a9ace37/attachment.html>;

📅 Original date posted:2015-12-18 📝 Original message:On ...

2023-06-07T17:46:23Z

In reply to nevent1q…yn2f
_________________________

📅 Original date posted:2015-12-18
📝 Original message:On Fri, Dec 18, 2015 at 6:11 AM, Jeff Garzik <jgarzik at gmail.com> wrote:
>> You present this as if the Bitcoin Core development team is in charge
>> of deciding the network consensus rules, and is responsible for making
>> changes to it in order to satisfy economic demand. If that is the
>> case, Bitcoin has failed, in my opinion.
>
> Diverging from the satoshi block size change plan[1] and current economics
> would seem to require a high level of months-ahead communication to users.

I don't see any plan, but will you say the same thing when the subsidy
dwindles, and mining income seems to become uncertain? It will equally
be an economic change, which equally well will have been predictable,
and it will equally well be treatable with a hardfork to increase the
subsidy.

Yes, I'm aware the argument above is a straw man, because there was
clear expectation that the subsidy would go down asymptotically, and
much less an expectation that the blocksize would remain fixed
forever.

But I am not against a block size increase hard fork. My talk on
segregated witness even included proposed pursuing a hard fork at a
slightly later stage.

But what you're arguing for is that - despite being completely
expected - blocks grew fuller, and people didn't adapt to block size
pressure and a fee market, so the Core committee now needs to kick the
can down the road, because we can't accept the risk of economic
change. That sounds very much like a bailout to me.

Again. I am not against growth, but increasing in response to fear of
economic change is the wrong approach. Economic change is inevitable.

> Segregated Witness does not kick the can, it solves none of the problems #1,
> #3 - #8 explicitly defined and listed in email #1.
>
> 1) A plan of "SW + no hard fork" is gambling with ECE risk, gambling there
> will be no Fee Event, because the core block size is still heavily contended
> -- 100% contended at time out SW rollout.

That is an assumption. I expect demand for transactions at a given
feerate to stop growing at a certain contention level (and we've
reached some level of contention already, with mempools not being
cleared for significant amounts of time already).

> SW mitigates this
> - only after several months

That is assuming a hard fork consensus forming, deployment,
activation, ... goes faster than a softfork.

> - only assuming robust adoption rates by up-layer ecosystem software, and

That's not required. Everyone who individually switches to new
transactions gets to do 1.75x more transactions for the same price
(and at the same time gets safer contracts, better script
upgradability, and more security models at their disposal), completely
independent of whether anyone else in the ecosystem does the same.

> - only assuming transaction volume growth is flat or sub-linear

The only question is how many transactions for what price. Contention
always happens at a specific feerate level anyway.

> Those conditions must go as planned to fulfill "SW kicked the can" -- a lot
> of if's.
>
> As stated, SW is orthogonal to the drift-into-uncharted-waters problem
> outlined in email #1, which a short term bump does address.

Both SW and a short bump (which is apparently not what BIP102 does
anymore?) increase capacity available per price, and yes, they are
completely orthogonal.

My only disagreement is the motivation (avoiding economic change, as
opposed to aiming for safe growth) and the claim that a capacity
increase hardfork is easier and safe(r) to roll out quickly than
sortfork SW.

Apart from that, we clearly need to do both at some point.

--
Pieter

📅 Original date posted:2015-12-16 📝 Original message:On ...

2023-06-07T17:46:20Z

In reply to nevent1q…885h
_________________________

📅 Original date posted:2015-12-16
📝 Original message:On Wed, Dec 16, 2015 at 10:08 PM, Jeff Garzik <jgarzik at gmail.com> wrote:
>> You present this as if the Bitcoin Core development team is in charge
>> of deciding the network consensus rules, and is responsible for making
>> changes to it in order to satisfy economic demand. If that is the
>> case, Bitcoin has failed, in my opinion.
>
>
> This circles back to Problem #1: Avoidance of a choice is a still a choice
> - failing to ACK a MAX_BLOCK_SIZE increase still creates very real Economic
> Change Event risk.

We are not avoiding a choice. We don't have the authority to make a choice.

> And #3: If the likely predicted course is that Bitcoin Core will not accept
> a protocol change changing MAX_BLOCK_SIZE via hard fork in the short term,
> the core dev team should communicate that position clearly to users and
> media.

I indeed think we can communicate much better that deciding consensus
rules is not within our power.

--
Pieter

📅 Original date posted:2015-08-10 📝 Original message:On Aug ...

2023-06-07T17:33:59Z

In reply to nevent1q…zjk5
_________________________

📅 Original date posted:2015-08-10
📝 Original message:On Aug 11, 2015 12:52 AM, "Pieter Wuille" <pieter.wuille at gmail.com> wrote:
>
>
> On Aug 11, 2015 12:18 AM, "Thomas Zander via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:
>
> > Have you ever been to a concert that was far away from public
transport? They
> > typically set up bus shuttles, or taxis to get people back into town
> > afterwards.
> > The result there is always you end up waiting forever and it actually
may be
> > easier to just walk instead of wait.
> > The amount you pay is irrelevant if everyone is paying it. There still
is more
> > demand than there is capacity.
>
> That's an incorrect analogy. You choose the rate you pay, and get higher
priority when you pay more. Taxi drivers can't pick out higher-paying
customers in advance.

I'm sorry, I missed your "if everyone is paying it". This changes a lot. I
agree with you: if everyone wants to pay much then it becomes unreliable.

But I don't think that is something we can avoid with a small constant
factor block size increase, and we don't do the world a service by making
it look like it works for longer.

Let's grow within bounderies set by technology and centralization pressure
that we can agree on. Let the market decide whether how they will that will
low volume reliable transactions and/or high volume unreliable ones.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/59b198e8/attachment.html>;

📅 Original date posted:2015-08-10 📝 Original message:On Aug ...

2023-06-07T17:33:59Z

In reply to nevent1q…gz9u
_________________________

📅 Original date posted:2015-08-10
📝 Original message:On Aug 11, 2015 12:18 AM, "Thomas Zander via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Have you ever been to a concert that was far away from public transport?
They
> typically set up bus shuttles, or taxis to get people back into town
> afterwards.
> The result there is always you end up waiting forever and it actually may
be
> easier to just walk instead of wait.
> The amount you pay is irrelevant if everyone is paying it. There still is
more
> demand than there is capacity.

That's an incorrect analogy. You choose the rate you pay, and get higher
priority when you pay more. Taxi drivers can't pick out higher-paying
customers in advance.

A better comparison is Uber, which charges more in places with high demand,
and you can accept or refuse in advance. And yes, it remains reliable if
you're among those with the highest willingness to pay.

> So, no, its not unreliable for cheap free transactions.
> Its unreliable for all types of transactions.

If 2500 transactions fit in the block chain per day (assuming constant
size) and there are less than 2500 per hour that pay at least 0.001 BTC in
fee, then any transaction which pays more than 0.001 BTC will have a very
high chance of getting in a small multiple of one hour, since miners
prioritize by feerate.

If there are in addition to that 5000 transactions per hour which pay less,
then yes, they need to compete for the remaiming space and their
confirmation will be unreliable.

The whole point is that whether confirmation at a particular price point is
reliable depends on how much demand there is at that price point. And
increasing the block size out of fear of what might happen is failing to
recognize that it can always happen that there is a sudden change in demand
that outcompetes the rest.

The point is not that evolution towards a specific higher feerate needs to
happen, but an evolution to an ecosystem that accepts that there is never a
guarantee for reliability, unless you're willing to pay more than everyone
else - whatever that number is.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/b117bc74/attachment-0001.html>;

📅 Original date posted:2015-08-10 📝 Original message:On ...

2023-06-07T17:33:57Z

In reply to nevent1q…ge2w
_________________________

📅 Original date posted:2015-08-10
📝 Original message:On Mon, Aug 10, 2015 at 4:12 PM, Gavin Andresen <gavinandresen at gmail.com>
wrote:

>
> Executive summary: when networks get over-saturated, they become
> unreliable. Unreliable is bad.
>
> Unreliable and expensive is extra bad, and that's where we're headed
> without an increase to the max block size.
>

I think I see your point of view. You see demand for on-chain transactions
as a single number that grows with adoption. Once the transaction creation
rate grows close to the capacity, transactions will become unreliable, and
you consider this a bad thing.

And if you see Bitcoin as a payment system where guaranteed time to
confirmation is a feature, I fully agree. But I think that is an
unrealistic dream. It only seems reliable because of lack of use. It costs
1.5 BTC per day to create enough transactions to fill the block chain at
the minimum relay fee, and a small multiple of that at actual fee levels.
Assuming that rate remains similar with an increased block size, that
remains cheap.

If you want transactions to be cheap, it will also be cheap to make them
unreliable.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150810/0fff338b/attachment.html>;

📅 Original date posted:2015-08-11 📝 Original message:On ...

2023-06-07T17:33:51Z

In reply to nevent1q…vj4a
_________________________

📅 Original date posted:2015-08-11
📝 Original message:On Tue, Aug 11, 2015 at 11:35 PM, Michael Naber <mickeybob at gmail.com> wrote:

> Bitcoin would be better money than current money even if it were a bit
> more expensive to transact, simply because of its other great
> characteristics (trustlessness, limited supply, etc). However... it is not
> better than something else sharing all those same characteristics but which
> is also less expensive. The best money will win, and if Bitcoin doesn't
> increase capacity then it won't remain the best.
>

If it is less expensive, it is harder to be reliable (because it's easier
for a sudden new use case to outbid the available space), which is less
useful for a payment mechanism.

If it has better scale (with the same technology), it will have higher
centralization pressure. The higher price you potentially pay (in fees) to
get your transactions on a smaller block chain is the price of higher
security and independence. Perhaps the compromise is not at the optimal
place, but please stop saying "below what the technology can do". The
technology can "do" gigabyte blocks I'm sure, If you accept that you need a
small cluster to keep up with validation, and all blocks are produced by a
single miner cartel.

IMHO, Bitcoin (or any cryptocurrency) on-chain as a payment system is:
* Expensive: there is a (known in advance and agreed upon) inflation that
we're using to pay miners. But by holding Bitcoin you're paying for the
security of the system, even if it is not in fees.
* Unreliable: you never know when suddenly there will be more higher-fee
transactions that outbid you.
* Slow, unless you already trust the sender to not double spend (in which
case you don't actually need the security of the blockchain).

I don't know the future, and I don't know what use cases will develop and
what they'll want to pay or what reliability they need. But let's please
not throw out the one quality that Bitcoin is still good at: lack of
centralized parties to trust.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/a9e6e3a9/attachment.html>;

📅 Original date posted:2015-08-11 📝 Original message:On ...

2023-06-07T17:33:47Z

In reply to nevent1q…xmhf
_________________________

📅 Original date posted:2015-08-11
📝 Original message:On Tue, Aug 11, 2015 at 11:30 PM, Angel Leon via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> tell that to people in poor countries, or even in first world countries.
> The competitive thing here is a deal breaker for a lot of people who have
> no clue/don't care for decentralization,
>

Then they also don't need their transactions to be on the blockchain, right?

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/f3e65461/attachment-0001.html>;

📅 Original date posted:2015-08-11 📝 Original message:On ...

2023-06-07T17:33:45Z

In reply to nevent1q…zrkm
_________________________

📅 Original date posted:2015-08-11
📝 Original message:On Tue, Aug 11, 2015 at 9:37 PM, Michael Naber via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Hitting the limit in and of itself is not necessarily a bad thing. The
> question at hand is whether we should constrain that limit below what
> technology is capable of delivering. I'm arguing that not only we should
> not, but that we could not even if we wanted to, since competition will
> deliver capacity for global consensus whether it's in Bitcoin or in some
> other product / fork.
>

The question is not what the technology can deliver. The question is what
price we're willing to pay for that. It is not a boolean "at this size,
things break, and below it, they work". A small constant factor increase
will unlikely break anything in the short term, but it will come with
higher centralization pressure of various forms. There is discussion about
whether these centralization pressures are significant, but citing that
it's artificially constrained under the limit is IMHO a misrepresentation.
It is constrained to aim for a certain balance between utility and risk,
and neither extreme is interesting, while possibly still "working".

Consensus rules are what keeps the system together. You can't simply switch
to new rules on your own, because the rest of the system will end up
ignoring you. These rules are there for a reason. You and I may agree about
whether the 21M limit is necessary, and disagree about whether we need a
block size limit, but we should be extremely careful with change. My
position as Bitcoin Core developer is that we should merge consensus
changes only when they are uncontroversial. Even when you believe a more
invasive change is worth it, others may disagree, and the risk from
disagreement is likely larger than the effect of a small block size
increase by itself: the risk that suddenly every transaction can be spent
twice (once on each side of the fork), the very thing that the block chain
was designed to prevent.

My personal opinion is that we should aim to do a block size increase for
the right reasons. I don't think fear of rising fees or unreliability
should be an issue: if fees are being paid, it means someone is willing to
pay them. If people are doing transactions despite being unreliable, there
must be a use for them. That may mean that some use cases don't fit
anymore, but that is already the case.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/851e0acb/attachment.html>;

📅 Original date posted:2015-08-07 📝 Original message:On ...

2023-06-07T17:33:31Z

In reply to nevent1q…6lr4
_________________________

📅 Original date posted:2015-08-07
📝 Original message:On Fri, Aug 7, 2015 at 4:57 PM, Gavin Andresen via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Every once in a while the network will get lucky and we'll find six blocks
> in ten minutes. If you are deciding what transaction fee to put on your
> transaction, and you're willing to wait until that
> six-blocks-in-ten-minutes once-a-week event, submit your transaction with a
> low fee.
>
> All the higher-fee transactions waiting to be confirmed will get confirmed
> in the first five blocks and, if miners don't have any floor on the fee
> they'll accept (they will, but lets pretend they won't) then your
> very-low-fee transaction will get confirmed.
>
> In the limit, that logic becomes "wait an infinite amount of time, pay
> zero fee."
>

That's only the case when the actual rate of transactions with a non-zero
fee is below what fits in blocks. If the total production rate is higher,
even without configured floor by miners, a free transaction won't ever be
mined, as there will always be some backlog of non-free transaction. Not
saying that this is a likely outcome - it would inevitably mean that people
are creating transactions without any guarantee that they'll be mined,
which may not be what anyone is interested in. But perhaps there is some
"use" for ultra-low-priority unreliable transactions (... despite DoS
attacks).

>
> So... I have no idea what the 'market minimum fee' will be, because I have
> no idea how long people will be willing to wait, how many times they'll be
> willing to retransmit a low-fee transaction that gets evicted from
> memory-limited memory pools, or how much memory miners will be willing to
> dedicate to storing transactions that won't confirm for a long time because
> they're waiting for a flurry of blocks to be found.
>

Fair enough, I don't think anyone knows.

I guess my question (and perhaps that's what Jorge is after): do you feel
that blocks should be increased in response to (or for fear of) such a
scenario. And if so, if that is a reason for increase now, won't it be a
reason for an increase later as well? It is my impression that your answer
is yes, that this is why you want to increase the block size quickly and
significantly, but correct me if I'm wrong.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150807/a3e40340/attachment.html>;

📅 Original date posted:2015-08-07 📝 Original message:On Aug ...

2023-06-07T17:33:27Z

In reply to nevent1q…k6rt
_________________________

📅 Original date posted:2015-08-07
📝 Original message:On Aug 7, 2015 7:50 PM, "Gavin Andresen" <gavinandresen at gmail.com> wrote:
>
>
> On Fri, Aug 7, 2015 at 12:30 PM, Pieter Wuille via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
>>
>> If the incentives for running a node don't weight up against the
cost/difficulty using a full node yourself for a majority of people in the
ecosystem, I would argue that there is a problem. As Bitcoin's fundamental
improvement over other systems is the lack of need for trust, I believe
that with increased adoption should also come an increased (in absolute
terms) incentive for people to use a full node. I'm seeing the opposite
trend, and that is worrying IMHO.
>
>
> Are you saying that unless the majority of people in the ecosystem decide
to trust nothing but the genesis block hash (decide to run a full node)
there is a problem?

I shouldn't have said majority, sorry. But I do believe that as the odds at
stake in the system go up, so should those who take an interest in
verifying. That doesn't seem to be the case, and is a problem, where that
is a result of the block chain size or not.

> If so, then we do have a fundamental difference of opinion, but I've
misunderstood how you think about trust/centralization/convenience
tradeoffs in the past.
>
> I believe people in the Bitcoin ecosystem will choose different
tradeoffs, and I believe that is OK-- people should be free to make those
tradeoffs.

I agree. Though I believe that the blockchain itself cannot offer many
tradeoffs, and by trying to make it scale we hurt the whole system. The
place to introduce tradeoffs is in layers on top - there you can build
systems with various levels of trust without hurting others.

> And given that the majority of people in the ecosystem were deciding that
using a centralized service or an SPV-level-security wallet was better even
two or three years ago when blocks were tiny (I'd have to go back and dig
up number-of-full-nodes and number-of-active-wallets at the big web-wallet
providers, but I bet there were an order of magnitude more people using
centralized services than running full nodes even back then).

That's inevitable, I'm sure.

> I firmly believe that block size has very little to do with the decision
to run a full node or not.

Within certain limits, maybe not. Within certain limits, maybe it also does
not incentivize trusted miner setups like SPV mining (which hurt the
security of SPV nodes tremendously).

But if the reason for increasing is because you fear a change of economics,
then it means you prefer not dealing with that change. I believe you prefer
not dealing with it ever, and would rather have a system where as much as
possible happens on-chain, even when we unknowingly go beyond those limits.
I think we don't do the ecosystem a service by promising that such a future
is possible without compromises.

So, I think the block size should follow technological evolution, and not
reenforce the belief that the block size should follow demand. There will
always be demand, and we should learn to deal with it.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150807/e442538b/attachment.html>;

📅 Original date posted:2015-08-06 📝 Original message:On ...

2023-06-07T17:32:59Z

In reply to nevent1q…7zqw
_________________________

📅 Original date posted:2015-08-06
📝 Original message:On Fri, Aug 7, 2015 at 1:26 AM, Dave Scotese via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

"Miners can do this unilaterally" maybe, if they are a closed group, based
> on the 51% rule. But aren't they using full nodes for propagation? In this
> sense, anyone can vote by coding.
>

They don't need to use full nodes for propagation. Miners don't care when
other full nodes hear about their blocks, only whether they (eventually)
accept them.

And yes, full nodes can change what blocks they accept. That's called a
hard fork :)

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150807/2899162c/attachment.html>;

📅 Original date posted:2015-08-04 📝 Original message:I ...

2023-06-07T17:32:55Z

In reply to nevent1q…h6ek
_________________________

📅 Original date posted:2015-08-04
📝 Original message:I would like to withdraw my proposal from your self-appointed vote.

If you want to let a majority decide about economic policy of a currency, I
suggest fiat currencies. They have been using this approach for quite a
while, I hear.

Bitcoin's consensus rules are a consensus system, not a democracy. Find a
solution that everyone agrees on, or don't.
On Aug 4, 2015 9:51 AM, "jl2012 via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> As now we have some concrete proposals (
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009808.html),
> I think we should wrap up the endless debate with voting by different
> stakeholder groups.
>
> ---------------------------------
> Candidate proposals
>
> Candidate proposals must be complete BIPs with reference implementation
> which are ready to merge immediately. They must first go through the usual
> peer review process and get approved by the developers in a technical
> standpoint, without political or philosophical considerations. Any fine
> tune of a candidate proposal may not become an independent candidate,
> unless it introduces some “real” difference. “No change” is also one of the
> voting options.
> ---------------------------------
> Voter groups
>
> There will be several voter groups and their votes will be counted
> independently. (The time frames mentioned below are just for example.)
>
> Miners: miners of blocks with timestamp between 1 to 30 Sept 2015 are
> eligible to vote. One block one vote. Miners will cast their votes by
> signing with the bitcoin address in coinbase. If there are multiple
> coinbase outputs, the vote is discounted by output value / total coinbase
> output value.
> Many well-known pools are reusing addresses and they may not need to
> digitally sign their votes. In case there is any dispute, the digitally
> signed vote will be counted.
>
> Bitcoin holders: People with bitcoin in the UTXO at block 372500 (around
> early September) are eligible to vote. The total “balance” of each
> scriptPubKey is calculated and this is the weight of the vote. People will
> cast their votes by digital signature.
> Special output types:
> Multi-sig: vote must be signed according to the setting of the multi-sig.
> P2SH: the serialized script must be provided
> Publicly known private key: not eligible to vote
> Non-standard script according to latest Bitcoin Core rules: not eligible
> to vote in general. May be judged case-by-case
>
> Developers: People with certain amount of contribution in the past year in
> Bitcoin Core or other open sources wallet / alternative implementations.
> One person one vote.
>
> Exchanges: Centralized exchanges listed on Coindesk Bitcoin Index,
> Winkdex, or NYSE Bitcoin index, with 30 days volume >100,000BTC are
> invited. This includes Bitfinex, BTC China, BitStamp, BTC-E, itBit, OKCoin,
> Huobi, Coinbase. Exchanges operated for at least 1 year with 100,000BTC
> 30-day volume may also apply to be a voter in this category. One exchange
> one vote.
>
> Merchants and service providers: This category includes all bitcoin
> accepting business that is not centralized fiat-currency exchange, e.g.
> virtual or physical stores, gambling sites, online wallet service, payment
> processors like Bitpay, decentralized exchange like Localbitcoin, ETF
> operators like Secondmarket Bitcoin Investment Trust. They must directly
> process bitcoin without relying on third party. They should process at
> least 100BTC in the last 30-days. One merchant one vote.
>
> Full nodes operators: People operating full nodes for at least 168 hours
> (1 week) in July 2015 are eligible to vote, determined by the log of
> Bitnodes. Time is set in the past to avoid manipulation. One IP address one
> vote. Vote must be sent from the node’s IP address.
>
> --------------------
> Voting system
>
> Single transferable vote is applied. (
> https://en.wikipedia.org/wiki/Single_transferable_vote). Voters are
> required to rank their preference with “1”, “2”, “3”, etc, or use “N” to
> indicate rejection of a candidate.
> Vote counting starts with every voter’s first choice. The candidate with
> fewest votes is eliminated and those votes are transferred according to
> their second choice. This process repeats until only one candidate is left,
> which is the most popular candidate. The result is presented as the
> approval rate: final votes for the most popular candidate / all valid votes
>
> After the most popular candidate is determined, the whole counting process
> is repeated by eliminating this candidate, which will find the approval
> rate for the second most popular candidate. The process repeats until all
> proposals are ranked with the approval rate calculated.
>
> --------------------
> Interpretation of results:
>
> It is possible that a candidate with lower ranking to have higher approval
> rate. However, ranking is more important than the approval rate, unless the
> difference in approval rate is really huge. 90% support would be excellent;
> 70% is good; 50% is marginal; <50% is failed.
>
> --------------------
> Technical issues:
>
> Voting by the miners, developers, exchanges, and merchants are probably
> the easiest. We need a trusted person to verify the voters’ identity by
> email, website, or digital signature. The trusted person will collect votes
> and publish the named votes so anyone could verify the results.
>
> For full nodes, we need a trusted person to setup a website as an
> interface to vote. The votes with IP address will be published.
>
> For bitcoin holders, the workload could be very high and we may need some
> automatic system to collect and count the votes. If people are worrying
> about reduced security due to exposed raw public key, they should move
> their bitcoin to a new address before voting.
>
> Double voting: people are generally not allowed to change their mind after
> voting, especially for anonymous voters like bitcoin holders and solo
> miners. A double voting attempt from these classes will invalidate all
> related votes.
>
> Multiple identity: People may have multiple roles in the Bitcoin ecology.
> I believe they should be allowed to vote in all applicable categories since
> they are contributing more than other people.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150804/db57dc3a/attachment-0001.html>;

📅 Original date posted:2015-08-06 📝 Original message:On Aug ...

2023-06-07T17:32:36Z

In reply to nevent1q…9a7r
_________________________

📅 Original date posted:2015-08-06
📝 Original message:On Aug 6, 2015 9:42 PM, "Gavin Andresen via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:
2. The "market minimum fee" should be determined by the market. It should
not be up to us to decide "when is a good time."

I partially agree. The community should decide what risks it is willing to
take, and set limits accordingly. Let the market decide how that space is
best used.

>
>>
>> Would you agree that blocksize increase proposals should have such a
>> criterion/test?
>
>
> Although I've been very clear with my criterion, no, I don't think all
blocksize increase proposals should have to justify "why this size" or "why
this rate of increase." Part of my frustration with this whole debate is
we're talking about a sanity-check upper-limit; as long as it doesn't open
up some terrible new DoS possibility I don't think it really matters much
what the exact number is.

It is only a DoS protection limit if you want to rely on trusting miners. I
prefer a system where I don't have to do that.

But I agree the numbers don't matter much, for a different reason: the
market will fill up whatever space is available, and we'll have the same
discussion when the new limit doesn't seem enough anymore.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150806/7e4c45e8/attachment.html>;

📅 Original date posted:2015-08-06 📝 Original message:On ...

2023-06-07T17:32:33Z

In reply to nevent1q…hqca
_________________________

📅 Original date posted:2015-08-06
📝 Original message:On Thu, Aug 6, 2015 at 3:40 PM, Gavin Andresen via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> On Wed, Aug 5, 2015 at 9:26 PM, Jorge Timón <
> bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>> This is a much more reasonable position. I wish this had been starting
>> point of this discussion instead of "the block size limit must be
>> increased as soon as possible or bitcoin will fail".
>>
>
> It REALLY doesn't help the debate when you say patently false statements
> like that.
>
> My first blog post on this issue is here:
> http://gavinandresen.ninja/why-increasing-the-max-block-size-is-urgent
>
> ... and I NEVER say "Bitcoin will fail". I say:
>
> "If the number of transactions waiting gets large enough, the end result
> will be an over-saturated network, busy doing nothing productive. I don’t
> think that is likely– it is more likely people just stop using Bitcoin
> because transaction confirmation becomes increasingly unreliable."
>

But you seem to consider that a bad thing. Maybe saying that you're
claiming that this equals Bitcoin failing is an exaggeration, but you do
believe that evolving towards an ecosystem where there is competition for
block space is a bad thing, right?

I don't agree that "Not everyone is able to use the block chain for every
use case" is the same thing as "People stop using Bitcoin". People are
already not using it for every use case.

Here is what my proposed BIP says: "No hard forking change that relaxes the
block size limit can be guaranteed to provide enough space for every
possible demand - or even any particular demand - unless strong
centralization of the mining ecosystem is expected. Because of that, the
development of a fee market and the evolution towards an ecosystem that is
able to cope with block space competition should be considered healthy."

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150806/aff56c38/attachment.html>;

📅 Original date posted:2015-08-06 📝 Original message:On ...

2023-06-07T17:32:33Z

In reply to nevent1q…nna2
_________________________

📅 Original date posted:2015-08-06
📝 Original message:On Thu, Aug 6, 2015 at 4:21 PM, Gavin Andresen <gavinandresen at gmail.com>
wrote:

> On Thu, Aug 6, 2015 at 10:06 AM, Pieter Wuille <pieter.wuille at gmail.com>
> wrote:
>
>> But you seem to consider that a bad thing. Maybe saying that you're
>> claiming that this equals Bitcoin failing is an exaggeration, but you do
>> believe that evolving towards an ecosystem where there is competition for
>> block space is a bad thing, right?
>>
>
> No, competition for block space is good.
>

So if we would have 8 MB blocks, and there is a sudden influx of users (or
settlement systems, who serve much more users) who want to pay high fees
(let's say 20 transactions per second) making the block chain inaccessible
for low fee transactions, and unreliable for medium fee transactions (for
any value of low, medium, and high), would you be ok with that? If so, why
is 8 MB good but 1 MB not? To me, they're a small constant factor that does
not fundamentally improve the scale of the system. I dislike the outlook of
"being forever locked at the same scale" while technology evolves, so my
proposal tries to address that part. It intentionally does not try to
improve a small factor, because I don't think it is valuable.

What is bad is artificially limiting or centrally controlling the supply of
> that space.
>

It's exactly as centrally limited as the finite supply of BTC - by
consensus. You and I may agree that a finite supply is a good thing, and
may disagree about whether a consensus rule about the block size is a good
idea (and if so, at what level), but it's a choice we make as a community
about the rules of the system we want to use.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150806/d69b8f33/attachment-0001.html>;

📅 Original date posted:2015-08-04 📝 Original message:On ...

2023-06-07T17:32:27Z

In reply to nevent1q…0e4l
_________________________

📅 Original date posted:2015-08-04
📝 Original message:On Tue, Aug 4, 2015 at 3:12 PM, Gavin Andresen <gavinandresen at gmail.com>
wrote:

> On Tue, Aug 4, 2015 at 7:27 AM, Pieter Wuille via bitcoin-dev <
> bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>> I would say that things already demonstrately got terrible. The mining
>> landscape is very centralized, with apparently a majority depending on
>> agreements to trust each other's announced blocks without validation.
>>
> And that is a problem... why?
>

If miners need to form alliances of trusting each other's blocks without
validation to overcome the inefficiencies of slow block propagation, I
think we have a system that is in direct conflict with the word
"permissionless" that you use later.

> As Bitcoin grows, pieces of the ecosystem will specialize. Satoshi's
> original code did everything: hashing, block assembly, wallet, consensus,
> network. That is changing, and that is OK.
>

Specialization is perfectly fine.
>
> I believe that if the above would have happened overnight, people would
> have cried wolf. But somehow it happened slow enough, and "things kept
> working".
>
> I don't think that this is a good criterion. Bitcoin can "work" with
> gigabyte blocks today, if everyone uses the same few blockchain validation
> services, the same few online wallets, and mining is done by a cartel that
> only allows joining after signing a contract so they can sue you if you
> create an invalid block. Do you think people will then agree that "things
> got demonstratebly worse"?
>
> Don't turn Bitcoin into something uninteresting, please.
>
Why is what you, personally, find interesting relevant?
>

I find it interesting to build a system that has potential to bring about
innovation.

I understand you want to build an extremely decentralized system, where
> everybody participating trusts nothing except the genesis block hash.
>

That is not true, I'm sorry if that is the impression I gave.

I see centralization and scalability as a trade-off, and for better or for
worse, the block chain only offers one trade-off. I want to see technology
built on top that introduces lower levels of trust than typical fully
centralized systems, while offering increased convenience, speed,
reliability, and scale. I just don't think that all of that can happen on
the lowest layer without hurting everything built on top. We need different
trade-offs, and the blockchain is just one, but a very fundamental one.

I think it is more interesting to build a system that works for hundreds of
> millions of people, with no central point of control and the opportunity
> for ANYBODY to participate at any level. Permission-less innovation is what
> I find interesting.
>

That sounds amazing, but do you think that Bitcoin, as it exists today, can
scale to hundreds of millions of users, while retaining any glimpse of
permission-lessness and decentralization? I think we need low-trust
off-chain systems and other innovations to make that happen.

> And I think the current "demonstrably terrible" Bitcoin system is still
> INCREDIBLY interesting.
>

I'm happy for you, then.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150804/1128ad92/attachment.html>;

📅 Original date posted:2015-08-04 📝 Original message:I ...

2023-06-07T17:32:23Z

In reply to nevent1q…5lvz
_________________________

📅 Original date posted:2015-08-04
📝 Original message:I would say that things already demonstrately got terrible. The mining
landscape is very centralized, with apparently a majority depending on
agreements to trust each other's announced blocks without validation. Full
node count is at its historically lowest value in years, and outsourcing of
full validation keeps growing.

I believe that if the above would have happened overnight, people would
have cried wolf. But somehow it happened slow enough, and "things kept
working".

I don't think that this is a good criterion. Bitcoin can "work" with
gigabyte blocks today, if everyone uses the same few blockchain validation
services, the same few online wallets, and mining is done by a cartel that
only allows joining after signing a contract so they can sue you if you
create an invalid block. Do you think people will then agree that "things
got demonstratebly worse"?

Don't turn Bitcoin into something uninteresting, please.

--
Pieter
On Aug 4, 2015 1:04 PM, "Hector Chu via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> Mike's position is that he wants the block size limit
> to eventually be removed. That is of course an extreme view. Meanwhile,
> your view that the block size should be artificially constrained below the
> organic growth curve (in a way that will penalize a majority of existing
> and future users) lies at the other extreme. The majority position lies
> somewhere in between (i.e. a one-time increase to 8MB). This is the
> position that ultimately matters.
>
> If the block size is increased to 8MB and things get demonstrably a whole
> lot worse, then you will have a solid leg to stand on. In that case we can
> always do another hard fork later to reduce the block size back to
> something smaller, and henceforth the block size will never be touched
> again.
>
> On 4 August 2015 at 11:35, Jorge Timón <
> bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>> On Fri, Jul 31, 2015 at 4:58 PM, Mike Hearn <hearn at vinumeris.com> wrote:
>> >> How more users or more nodes can bring more miners, or more
>> importantly,
>> >> improve mining decentralization?
>> >
>> >
>> > Because the bigger the ecosystem is the more interest there is in taking
>> > part?
>>
>> As explained by Venzen, this is a non-sequitur.
>>
>> > I mean, I guess I don't know how to answer your question.
>>
>> I don't know the answer either, that's fine. It's the opposite
>> question that I've been insistently repeating and you've been
>> (consciously or not) consistently evading.
>> But that's also fine because I believe you finally answer it a few lines
>> below.
>>
>> > When Bitcoin was
>> > new it had almost no users and almost no miners. Now there are millions
>> of
>> > users and factories producing ASICs just for Bitcoin.
>>
>> The emergence of a btc price enabled the emergence of professional
>> miners, which in turn enabled the emergence of sha256d-specialized
>> hardware production companies.
>> Nothing surprising there.
>> By no means it consitutes an example of how a bigger consensus sizes
>> can cause less mining centralization.
>>
>> > Surely the correlation is obvious?
>>
>> Correlation does not imply causation. I will better leave it at that...
>>
>> >> I'm sorry, but until there's a simulation that I can run with different
>> >> sizes' testchains (for example using #6382) to somehow compare them, I
>> will
>> >> consider any value arbitrary.
>> >
>> >
>> > Gavin did run simulations. 20mb isn't arbitrary, the process behind it
>> was
>> > well documented here:
>> >
>> >
>> http://gavinandresen.ninja/does-more-transactions-necessarily-mean-more-centralized
>> >
>> > I chose 20MB as a reasonable block size to target because 170 gigabytes
>> per
>> > month comfortably fits into the typical 250-300 gigabytes per month data
>> > cap– so you can run a full node from home on a “pretty good” broadband
>> plan.
>> >
>> > Did you think 20mb was picked randomly?
>>
>> No, I think 20 MB was chosen very optimistically, considering 3rd
>> party services rates (not the same service as self-hosting) in the
>> so-called "first world". And then 20 MB goes to 20 GB, again with
>> optimistic and by no means scientific expectations.
>>
>> But where the number comes from it's not really what I'm demaning,
>> what I want is some criterion that can tell you that a given size
>> would be "too centralized" but another one isn't.
>> I haven't read any analysis on why 8GB is a better option than 7GB and
>> 9GB for a given criterion (nor one declaring 20 GB a winner over 19 GB
>> or 21 GB).
>> A simulation test passing 20 GB but not 21 GB would make it far less
>> arbitrary.
>>
>> >> Agreed on the first sentence, I'm just saying that the influence of
>> >> the blocksize in that function is monotonic: with bigger sizes, equal
>> >> or worse mining centralization.
>> >
>> >
>> > I have a hard time agreeing with this because I've seen Bitcoin go from
>> > blocks that were often empty to blocks that are often full, and in this
>> time
>> > the number of miners and hash power on the network has gone up a huge
>> amount
>> > too.
>>
>> I'm of course talking about consensus maximum blocksize, not about
>> actual blocksize.
>> Yes, again, when mining becomes profitable, economic actors tend to
>> appear and get those profits.
>> But don't confuse total hashrate improvements with an "increase in the
>> number of miners" or with mining decentralization.
>>
>> > You can argue that a miner doesn't count if they pool mine. But if a
>> miner
>> > mines on a pool that uses exactly the same software and settings as the
>> > miner would have done anyway, then it makes no difference. Miners can
>> switch
>> > between pools to find one that works the way they like, so whilst less
>> > pooling or more decentralised pools would be nice (e.g.
>> getblocktemplate),
>> > and I've written about how to push it forward before, I still say there
>> are
>> > many more miners than in the past.
>> >
>> > If I had to pick between two changes to improve mining decentralisation:
>> >
>> > 1) Lower block size
>>
>> Finally, I think you finally answered my repetitive question here.
>> If I say "Mike Hearn understands that the consensus block size maximum
>> rule is a tool for limitting mining centralization" I'm not putting
>> words in your mouth, right?
>> I think many users advocating for an increase in the consensus limit
>> don't understand this, which is extremely unfortunate for the debate.
>>
>> > 2) Finishing, documenting, and making the UX really slick for a
>> > getblocktemplate based decentralised mining pool
>> >
>> > then I'd pick (2) in a heartbeat. I think it'd be a lot more effective.
>>
>> Great! Maybe after 2 mining centralization improves so much that we're
>> confortable not only not lowering it but rather increasing it.
>>
>> >> you should be consequently advocating for full removal of the limit
>> rather
>> >> than changes towards bigger arbitrary values.
>> >
>> >
>> > I did toy with that idea a while ago. Of course there can not really be
>> no
>> > limit at all because the code assumes blocks fit into RAM/swap, and
>> nodes
>> > would just end up ignoring blocks they couldn't download in time anyway.
>> > There is obviously a physical limit somewhere.
>>
>> Did the fact that you "understand that the consensus block size
>> maximum rule is a tool for limitting mining centralization" influenced
>> your rejection of that idea at all?
>>
>> > But it is easier to find common ground with others by compromising. Is
>> 8mb
>> > better than no limit? I don't know and I don't care much: I think
>> Bitcoin
>> > adoption is a slow, hard process and we'll be lucky to increase average
>> > usage 8x over the next couple of years. So if 8mb+ is better for others,
>> > that's OK by me.
>>
>> The only way that "not caring much whther we have a consensus limit or
>> not" and "understand that the consensus block size maximum rule is a
>> tool for limitting mining centralization" at the same time is by not
>> caring about mining centralization at all.
>> Is that your position?
>>
>> If you don't care about having a limit but you don't want to limit
>> transaction volume, then ++current_size will ALWAYs be your
>> "compromise position" and no blocksize increase will ever be enough
>> until the limit is completely removed.
>> Is that your position?
>>
>> > Re: exchange profit. You can pick some other useful service provider if
>> you
>> > like. Payment processors or cold storage providers or the TREZOR
>> > manufacturers or whoever.
>>
>> Yes, and I believe the same points stand.
>>
>> > My point is you can't have a tiny high-value-transactions only currency
>> AND
>> > all the useful infrastructure that the Bitcoin community is making.
>> It's a
>> > contradiction. And without the infrastructure bitcoin ceases to be
>> > interesting even to people who are willing to pay huge sums to use it.
>>
>> You keep talking about "high-value-transactions-only" like if
>> non-urgent transaction fees rising from zero to, say, 1 satoshi, would
>> automatically result in that "high-value-transactions-only" Bitcoin.
>> Please, stop talking as if someone was proposing a
>> "high-value-transactions-only" Bitcoin. That may happen but nobody
>> really knows. If it happens it may not be bad thing necessarily (ie
>> bitcoin microtransactions can still happen using trustless payment
>> channels and x is still cheaper than x% for any transacted value
>> higher than 100) but that's really not what we're talking about here
>> so it seems distraction that can only help further polirizing this
>> discussion.
>>
>> What we're talking about here is that hitting the limit would
>> (hopefully) make miners start caring about fees. Enough that they stop
>> being irrational about free transactions. If both things happen,
>> non-urgent transaction fees will likely rise (as said, above zero).
>>
>> You think that would be a catastrophe for adoption and I disagree.
>> But (as Pieter has repeatedly explained) for any size there will be
>> use cases that will be eventually priced out.
>> So when rising this consensus limit, not increasing centralization
>> should be the priority and the potential impact in market fees a much
>> more secondary concern.
>> Do you agree with this?
>>
>> I'm sure there are many intermediate positions between "caring more
>> about mining centralization than market fees when deciding about a
>> consensus rule that limits mining centralization" and "not caring
>> about mining centralization at all".
>> I really don't want to put words in your mouth, but I honestly don't
>> know what your position is.
>> I don't really know how else can I ask the same question: you don't
>> care the consensus maximum blocksize rule being here at all or not
>> (you just said that).
>> Is it because you don't think it limits mining centralization or
>> because you don't care about limiting mining centralization with
>> consensus rules at all?
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150804/2b6fdfa3/attachment.html>;

📅 Original date posted:2016-01-07 📝 Original message:> ...

2023-06-07T17:31:33Z

In reply to nevent1q…ssee
_________________________

📅 Original date posted:2016-01-07
📝 Original message:> "The problem case is where someone in a contract setup shows you a
script, which you accept as being a payment to yourself. An attacker could
use a collision attack to construct scripts with identical hashes, only one
of which does have the property you want, and steal coins.
>
> So you really want collision security, and I don't think 80 bits is
something we should encourage for that. Normal pubkey hashes don't have
that problem, as they can't be constructed to pay to you."
>
> ... but I'm unconvinced:
>
> "But it is trivial for contract wallets to protect against collision
attacks-- if you give me a script that is "gavin_pubkey CHECKSIG
arbitrary_data OP_DROP" with "I promise I'm not trying to rip you off, just
ignore that arbitrary data" a wallet can just refuse. Even more likely, a
contract wallet won't even recognize that as a pay-to-gavin transaction.
>
> I suppose it could be looking for some form of "gavin_pubkey
somebody_else_pubkey CHECKMULTISIG ... with the attacker using
somebody_else_pubkey to force the collision, but, again, trivial contract
protocol tweaks ("send along a proof you have the private key corresponding
to the public key" or "everybody pre-commits pubkeys they'll use at
protocol start") would protect against that.

Yes, this is what I worry about. We're constructing a 2-of-2 multisig
escrow in a contract. I reveal my public key A, you do a 80-bit search for
B and C such that H(A and B) = H(B and C). You tell me your keys B, and I
happily send to H(A and B), which you steal with H(B and C).

Sending along a proof does not help, you can't prove that you do not know
of a collision. Pre-committing does help, but is a very non-obvious
security requirement, something I strongly believe is far riskier in
practice.

Bitcoin does have parts that rely on economic arguments for security or
privacy, but can we please stick to using cryptography that is up to par
for parts where we can? It's a small constant factor of data, and it
categorically removes the worry about security levels.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160108/e2e921fb/attachment.html>;

📅 Original date posted:2016-01-08 📝 Original message:On ...

2023-06-07T17:31:32Z

In reply to nevent1q…eyn9
_________________________

📅 Original date posted:2016-01-08
📝 Original message:On Fri, Jan 8, 2016 at 2:54 AM, Gavin Andresen via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> I'm saying we can eliminate one somewhat unlikely attack (that there is a
> bug in the code or test cases, today or some future version, that has to
> decide what to do with "version 0" versus "version 1" witness programs) by
> accepting the risk of another insanely, extremely unlikely attack.

Ok, just having one witness program version now is a somewhat different
proposal. It would be simpler for sure. The reasoning was that you'd need
this to not add significant overhead to small scripts, but that may not be
the case anymore. I wouldn't mind seeing numbers.

> My proposal would be to just do a version 0 witness program now, that is
> RIPEMD160(SHA256(script)).

I don't think that is wise. Bitcoin has a 128-bit security target for
everything else. We did not know that P2SH and similar constructs were
vulnerable to a collision attack at the time, but now we do, so the obvious
choice is to pick a size that is sufficiently large to maintain the 128-bit
security target. This is a no brainer to me; we're not proposing switching
to a 160-bit EC curve either, right?

> I'm really disappointed with the "Here's the spec, take it or leave it"
> attitude. What's the point of having a BIP process if the discussion just
> comes down to "We think more is better. We don't care what you think."

It is a proposal and we are discussing it. You first brought up some
criticisms in private, and I agreed with several things you said.

But it remains the proposal of a few people including me, and I do not
agree with the specific suggestion of reducing the security target for
witness scripts to 80 bits.

We are not deciding what the system will be. We're making a proposal, and
hope that due to its technical merit, the ecosystem will adopt it. You're
free to participate in that discussion.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160108/2420393c/attachment.html>;

📅 Original date posted:2015-08-10 📝 Original message:On Aug ...

2023-06-07T15:46:18Z

In reply to nevent1q…3fv5
_________________________

📅 Original date posted:2015-08-10
📝 Original message:On Aug 7, 2015 11:19 PM, "Sergio Demian Lerner via bitcoin-dev" <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> b. Reduce the block rate to a half (average 5 minute blocks)
>
> Suppose this is a one time hard fork. There no drastic technical problems
with any of them: "SPV" mining and the relay network has shown that block
propagation is not an issue for such as small change. Mining centralization
won't radically change for a 2x adjustment.

I don't understand this. All problems that result from propagation delay
are literally doubled by doing so. Centralization pressure results from the
ratio between propagation time and interblock time. Efficient propagation
algorithms like the relay network make this presumably grow sublinear with
larger blocks, but changing the interblock time affects it exactly
proportionally.

All problems that result from propagation delay are literally doubled by
doing this. Doubling the block size has a smaller effect. You may argue
that these centralization effects are small, but reducing the interblock
time has a stronger effect on them than the block size.

Also, you seem to consider SPV mining a good thing? It requires trust
between miners that know eachother, and fundamentally breaks the security
assumption of SPV clients... and if the propagation/interblock ratio was
lower, SPV mining would have less effect. I'd say it is exactly a result of
the centralization pressure we're trying to avoid.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150810/f50a4a75/attachment-0001.html>;

📅 Original date posted:2015-08-10 📝 Original message:On ...

2023-06-07T15:46:01Z

In reply to nevent1q…n64q
_________________________

📅 Original date posted:2015-08-10
📝 Original message:On Mon, Aug 10, 2015 at 4:12 PM, Gavin Andresen <gavinandresen at gmail.com>
wrote:

>
> Executive summary: when networks get over-saturated, they become
> unreliable. Unreliable is bad.
>
> Unreliable and expensive is extra bad, and that's where we're headed
> without an increase to the max block size.
>

I think I see your point of view. You see demand for on-chain transactions
as a single number that grows with adoption. Once the transaction creation
rate grows close to the capacity, transactions will become unreliable, and
you consider this a bad thing.

And if you see Bitcoin as a payment system where guaranteed time to
confirmation is a feature, I fully agree. But I think that is an
unrealistic dream. It only seems reliable because of lack of use. It costs
1.5 BTC per day to create enough transactions to fill the block chain at
the minimum relay fee, and a small multiple of that at actual fee levels.
Assuming that rate remains similar with an increased block size, that
remains cheap.

If you want transactions to be cheap, it will also be cheap to make them
unreliable.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150810/0fff338b/attachment.html>;

📅 Original date posted:2015-08-11 📝 Original message:On ...

2023-06-07T15:45:57Z

In reply to nevent1q…9e8z
_________________________

📅 Original date posted:2015-08-11
📝 Original message:On Tue, Aug 11, 2015 at 11:35 PM, Michael Naber <mickeybob at gmail.com> wrote:

> Bitcoin would be better money than current money even if it were a bit
> more expensive to transact, simply because of its other great
> characteristics (trustlessness, limited supply, etc). However... it is not
> better than something else sharing all those same characteristics but which
> is also less expensive. The best money will win, and if Bitcoin doesn't
> increase capacity then it won't remain the best.
>

If it is less expensive, it is harder to be reliable (because it's easier
for a sudden new use case to outbid the available space), which is less
useful for a payment mechanism.

If it has better scale (with the same technology), it will have higher
centralization pressure. The higher price you potentially pay (in fees) to
get your transactions on a smaller block chain is the price of higher
security and independence. Perhaps the compromise is not at the optimal
place, but please stop saying "below what the technology can do". The
technology can "do" gigabyte blocks I'm sure, If you accept that you need a
small cluster to keep up with validation, and all blocks are produced by a
single miner cartel.

IMHO, Bitcoin (or any cryptocurrency) on-chain as a payment system is:
* Expensive: there is a (known in advance and agreed upon) inflation that
we're using to pay miners. But by holding Bitcoin you're paying for the
security of the system, even if it is not in fees.
* Unreliable: you never know when suddenly there will be more higher-fee
transactions that outbid you.
* Slow, unless you already trust the sender to not double spend (in which
case you don't actually need the security of the blockchain).

I don't know the future, and I don't know what use cases will develop and
what they'll want to pay or what reliability they need. But let's please
not throw out the one quality that Bitcoin is still good at: lack of
centralized parties to trust.

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/a9e6e3a9/attachment.html>;

📅 Original date posted:2015-08-11 📝 Original message:On ...

2023-06-07T15:45:53Z

In reply to nevent1q…np42
_________________________

📅 Original date posted:2015-08-11
📝 Original message:On Tue, Aug 11, 2015 at 11:30 PM, Angel Leon via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:

> tell that to people in poor countries, or even in first world countries.
> The competitive thing here is a deal breaker for a lot of people who have
> no clue/don't care for decentralization,
>

Then they also don't need their transactions to be on the blockchain, right?

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150811/f3e65461/attachment-0001.html>;

📅 Original date posted:2015-08-07 📝 Original message:On ...

2023-06-07T15:45:40Z

In reply to nevent1q…h06c
_________________________

📅 Original date posted:2015-08-07
📝 Original message:On Fri, Aug 7, 2015 at 5:55 PM, Gavin Andresen <gavinandresen at gmail.com>
wrote:

> On Fri, Aug 7, 2015 at 11:16 AM, Pieter Wuille <pieter.wuille at gmail.com>
> wrote:
>
>> I guess my question (and perhaps that's what Jorge is after): do you feel
>> that blocks should be increased in response to (or for fear of) such a
>> scenario.
>>
>
> I think there are multiple reasons to raise the maximum block size, and
> yes, fear of Bad Things Happening as we run up against the 1MB limit is one
> of the reasons.
>
> I take the opinion of smart engineers who actually do resource planning
> and have seen what happens when networks run out of capacity very seriously.
>

This is a fundamental disagreement then. I believe that the demand is
infinite if you don't set a fee minimum (and I don't think we should), and
it just takes time for the market to find a way to fill whatever is
available - the rest goes into off-chain systems anyway. You will run out
of capacity at any size, and acting out of fear of that reality does not
improve the system. Whatever size blocks are actually produced, I believe
the result will either be something people consider too small to be
competitive ("you mean Bitcoin can only do 24 transactions per second?"
sounds almost the same as "you mean Bitcoin can only do 3 transactions per
second?"), or something that is very centralized in practice, and likely
both.

> And if so, if that is a reason for increase now, won't it be a reason for
>> an increase later as well? It is my impression that your answer is yes,
>> that this is why you want to increase the block size quickly and
>> significantly, but correct me if I'm wrong.
>>
>
> Sure, it might be a reason for an increase later. Here's my message to
> in-the-future Bitcoin engineers: you should consider raising the maximum
> block size if needed and you think the benefits of doing so (like increased
> adoption or lower transaction fees or increased reliability) outweigh the
> costs (like higher operating costs for full-nodes or the disruption caused
> by ANY consensus rule change).
>

In general that sounds reasonable, but it's a dangerous precedent to make
technical decisions based on a fear of change of economics...

--
Pieter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150807/8322a671/attachment.html>;