Nostr notes by Achow101 [ARCHIVE]

📅 Original date posted:2018-07-06 📝 Original message:Hi, ...

2023-06-07T18:13:32Z

In reply to nevent1q…xyyt
_________________________

📅 Original date posted:2018-07-06
📝 Original message:Hi,

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On July 5, 2018 12:20 PM, William Casarin <jb55 at jb55.com> wrote:

>
>
> I have another concern with the format. (my original bip comment for some context: [1])
>
> It looks like the one of the reasons I was confused is because you can
>
> only parse the format properly by first deserializing the transaction.
>
> Since there is no "length" field for the key-value map arrays, you must
>
> count the number of transaction input/outputs, and use that as the
>
> number of kv maps to parse.

I don't think this is really a problem.

Almost all roles have to deserialize the unsigned tx anyways before they can do anything.
The only role that doesn't is a simple combiner (a combiner that does sanity checks would
still have to deserialize the unsigned tx), and even then it doesn't matter. It just shoves
key value pairs together and doesn't need to know whether the map is for an input or for
an output.

>
> This is pretty brittle, because now if a Combiner writes the wrong
>
> number of key-value maps that don't align with the number of inputs and
>
> outputs in the transaction, then the psbt will not be able to be
>
> deserialized properly, but is still a valid PSBT. It can't even detect
>
> these situations, because the input and output types share the same enum
>
> values.

If a combiner writes the wrong number of key-value maps, then it would simply be invalid
to the next person that receives the PSBT. It would not deserialize properly because the
key value pairs would have incorrect values for their types. Not deserializing properly means
that the PSBT is simply invalid. The same numerical types might
be shared, but their meanings are different between the input and output types.

I don't see anywhere that says the number of key value maps MUST
>
> match the number of inputs/outputs, perhaps it's implied?

I have added that to the BIP.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On July 5, 2018 10:23 AM, Jason Les <jasonles at gmail.com> wrote:

> Has there been any thought to standardizing file names used when creating .psbt files? Maybe something that gives some reliability of being collision resistant and descriptive. For example:
>
> [8 char trim of hash of unsigned tx]+[Role that created file (Ex: Signer)]+[4 char trim of hash of data unique to that role (Ex: partial sig)]
>
> It may be useful to especially the combiner to have some idea of what files they have.
>
> -Jason Les

I haven't considered this, but I'm not sure if it is really useful. I don't think it is really necessary
for any role to know who created the PSBT. If it did, this information would generally come
out-of-band anyways as someone has to give the PSBT to that person.

Andrew

📅 Original date posted:2018-07-04 📝 Original message:Hi, ...

2023-06-07T18:13:31Z

In reply to nevent1q…pzl5
_________________________

📅 Original date posted:2018-07-04
📝 Original message:Hi,

On July 4, 2018 6:19 AM, matejcik <jan.matejek at satoshilabs.com> wrote:

>
>
> hello,
>
> we still have some concerns about the BIP as currently proposed - not
>
> about the format or data contents, but more about strictness and
>
> security properties. I have raised some in the previous e-mails, but
>
> they might have been lost in the overall talk about format.
>
> - Choosing from duplicate keys when combining.
>
> We believe that "choose whichever value it wishes" is not a good
>
> resolution strategy. We propose to either change this to "in case of
>
> conflicts, software MUST reject the conflicting PSBTs", or explain in
>
> more detail why picking at random is a safe choice.

You cannot simply reject PSBTs for having conflicting values for the same key. Especially
for the Partial Signatures, you can have two signatures for the same pubkey that are both
completely valid. This situation could happen, for example, if a signer that does not use deterministic
k values can sign multiple inputs but one input is missing a UTXO so it doesn't sign it. So it receives
one PSBT and signs the first input but not the second. It receives a PSBT for the same transaction
which has the second input's UTXO but does not have its signatures for the first input. The signer
would sign both inputs. When the two PSBTs are combined (suppose the first PSBT has other
signatures too), you will have two keys that have different values. The different values are both
valid signatures, just with different k values since they were randomly generated instead of
deterministically. If we fail to merge these, then you could potentially have a situation where
nothing can be done with the PSBTs now, or now everyone has to resign and in some specific
order to avoid the conflict. That complicates things and is much more annoying to deal with.
So a simple solution is to allow the combiner to choose any value it wants as it is likely that
both values are valid.

Allowing combiners to choose any value also allows for intelligent combiners to choose the
correct values in the case of conflicts. A smart combiner could, when combining redeem scripts
and witness scripts, check that the redeem scripts and witness scripts match the hash provided
in the UTXO (or in the redeem script) and choose the correct redeem script and witness script
accordingly if there were, for some reason, a conflict there.

Can you explain why it would be unsafe for combiners to arbitrarily choose a value?

>
> - Signing records with unknown keys.
>
> There's been some talk about this at start, but there should be a clear
>
> strategy for Signers when unknown fields are encountered. We intend to
>
> implement the rule: "will not sign an input with any unknown fields
>
> present".
>
> Maybe it is worth codifying this behavior in the standard, or maybe
>
> there should be a way to mark a field as "optional" so that strict
>
> Signers know they can safely ignore the unknown field.

I think that requiring there to be no unknowns is a safe change.

>
> And two minor points:
>
> - Fields with empty keys.
>
> This might be inferred from the definition, but is probably worth
>
> spelling out explicitly: If a field definition states that the key data
>
> is empty, an implementation MUST enforce this and reject PSBTs that
>
> contain non-empty data.
>
> We suggest adding something to the effect of:
>
> "If a key or value data in a field doesn't match the specified format,
>
> the PSBT is invalid. In particular, if key data is specified as "none"
>
> but the key contains data beyond the type specifier, implementation MUST
>
> reject the PSBT."
>
> (not sure about the languge, this should of course allow processing
>
> unknown fields)

Agreed.

>
> - "Combiner can detect inconsistencies"
>
> Added in response to this comment [1], the current wording looks like
>
> it's describing what the Combiner is capable of, as opposed to
>
> prescribing what the combiner is allowed to do.
>
> We suggest changing to something like:
>
> "For every field type that the Combiner understands, it MAY also refuse
>
> to combine PSBTs that have inconsistencies in that field, or cause a
>
> conflict when combined."

Agreed.

Andrew

📅 Original date posted:2018-06-26 📝 Original message:Hi, On ...

2023-06-07T18:13:16Z

In reply to nevent1q…zwcq
_________________________

📅 Original date posted:2018-06-26
📝 Original message:Hi,

On June 26, 2018 8:33 AM, matejcik via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

>
>
> hello,
>
> in general, I agree with my colleague Tomas, the proposed changes are
>
> good and achieve the most important things that we wanted. We'll review
>
> the proposal in more detail later.
>
> For now a couple minor things I have run into:
>
> - valid test vector 2 ("one P2PKH input and one P2SH-P2WPKH input")
>
> seems broken, at least its hex version; a delimiter seems to be missing,
>
> misplaced or corrupted

Fixed

>
> - at least the first signing vector is not updated, but you probably
>
> know that

I updated all of the tests yesterday so they should be correct now. I will be adding more tests
this week.

>
> - BIP32 derivation fields don't specify size of the "master public key",
>
> which would make it un-parsable :)

Oops, that's actually supposed to be master key fingerprint, not master public key. I have updated
the BIP to reflect this.

>
> - "Transaction format is specified as follows" and its table need to be
>
> updated.

Done.

>
> I'm still going to argue against the key-value model though.
>
> It's true that this is not significant in terms of space. But I'm more
>
> concerned about human readability, i.e., confusing future implementers.
>
> At this point, the key-value model is there "for historical reasons",
>
> except these aren't valid even before finalizing the format. The
>
> original rationale for using key-values seems to be gone (no key-based
>
> lookups are necessary). As for combining and deduplication, whether key
>
> data is present or not is now purely a stand-in for a "repeatable" flag.
>
> We could just as easily say, e.g., that the high bit of "type" specifies
>
> whether this record can be repeated.
>
> (Moreover, as I wrote previously, the Combiner seems like a weirdly
>
> placed role. I still don't see its significance and why is it important
>
> to correctly combine PSBTs by agents that don't understand them. If you
>
> have a usecase in mind, please explain.

There is a diagram in the BIP that explains this. The combiner's job is to combine two PSBTs that
are for the same transaction but contain different data such as signatures. It is easier to implement
a combiner that does not need to understand the types at all, and such combiners are forwards compatible,
so new types do not require new combiner implementations.

>
> ISTM a Combiner could just as well combine based on whole-record
>
> uniqueness, and leave the duplicate detection to the Finalizer. In case
>
> the incoming PSBTs have incompatible unique fields, the Combiner would
>
> have to fail anyway, so the Finalizer might as well do it. Perhaps it
>
> would be good to leave out the Combiner role entirely?)

A transaction that contains duplicate keys would be completely invalid. Furthermore, in the set of typed
records model, having more than one redeemScript and witnessScript should be invalid, so a combiner
would still need to understand what types are in order to avoid this situation. Otherwise it would produce
an invalid PSBT.

I also dislike the idea of having type specific things like "only one redeemScript" where a more generic
thing would work.

>
> There's two remaining types where key data is used: BIP32 derivations
>
> and partial signatures. In case of BIP32 derivation, the key data is
>
> redundant ( pubkey = derive(value) ), so I'd argue we should leave that
>
> out and save space.

I think it is still necessary to include the pubkey as not all signers who can sign for a given pubkey may
know the derivation path. For example, if a privkey is imported into a wallet, that wallet does not necessarily
know the master key fingerprint for the privkey nor would it necessarily have the master key itself in
order to derive the privkey. But it still has the privkey and can still sign for it.

>
> Re breaking change, we are proposing breaking changes anyway, existing
>
> code will need to be touched, and given that this is a hand-parsed
>
> format, changing `parse_keyvalue` to `parse_record` seems like a small
>
> matter?

The point is to not make it difficult for existing implementations to change. Mostly what has been done now is just
moving things around, not an entire format change itself. Changing to a set of typed records model require more
changes and is a complete restructuring of the format, not just moving things around.

Additionally, I think that the current model is fairly easy to hand parse. I don't think a record set model would make
it easier to follow. Furthermore, moving to Protobuf would make it harder to hand parse as varints are slightly more
confusing in protobuf than with Compact size uints. And with the key-value model, you don't need to know the types
to know whether something is valid. You don't need to interpret any data.

Andrew

📅 Original date posted:2018-06-27 📝 Original message:Hi, ...

2023-06-07T18:13:16Z

In reply to nevent1q…y5e0
_________________________

📅 Original date posted:2018-06-27
📝 Original message:Hi,

On June 26, 2018 11:09 PM, William Casarin <jb55 at jb55.com> wrote:

>
>
> Hey Andrew,
>
> If I'm reading the spec right: the way it is designed right now, you
>
> could create hundreds of thousands of zero bytes in the input or output
>
> key-value arrays. As far as I can tell this would be considered valid,
>
> as it is simply a large array of empty dictionaries. Is this right? I'm
>
> worried about buffer overflows in cases where someone sends a large blob
>
> of zeros to an unsuspecting implementation.

No, that is incorrect. That whole paragraph is actually outdated, it was intended
for the possibility of adding output maps, which we have already done. I have
removed it from the BIP.

However, it is possible for a PSBT to contain very large unknown key-value pairs
which could potentially cause a problem. But I do not think that large PSBTs should
really be a problem as they are really something that the user has to enter rather
than something received remotely without user interaction.

On June 27, 2018 6:39 AM, Andrea via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

>
>
> Hi William, Andrew, list,
>
> As noted by William there are some types missing in the global-types definition, because the number of each map for I/O must be known to the parser in order to use the correct definitions for the types. At the moment a parser reading a key-value record does not know whether it should read it as per-input type or per-output, a way to address this is to declare in advance the number of maps and ensure they are ordered (inputs first). If you haven't already worked out some types for that i propose using:
>

Parsers actually do know because that information is present in the unsigned transaction
at the beginning of each PSBT. Since each input and output must be accounted for,
a parser can just parse the unsigned transaction and from there it can know how
many inputs and outputs to expect. If it sees more or less, it should throw an error
as the transaction is invalid.

Of course this implies that implementations will need to parse the unsigned transaction,
but not all actually need to. Combiners do not need to, they just need to merge the
maps together and follow the key uniqueness rule. They don't really need to know
or care about the number of inputs and outputs, just that the PSBTs being merged
share the same unsigned transaction and have the same number of maps.

Other roles need to understand the unsigned transaction anyways, so they still need
to parse it thus this isn't really a problem for those roles.

>
> On another note I think we can set a hard limit on the size of the PSBT, currently is 'legal' to produce a very large PSBT with an excessive number of Inputs and Outputs. By excessive I mean that even in the best case scenario (using the smallest possible scriptPubKey as in P2WPKH) it is possible to create a PSBT that would certainly create an invalid transaction (because of its size) when finalized. I haven't found anything related to this in the previous discussions, please ignore this if it was already proposed/discussed.
>

I don't think such a limitation is practical or useful. A transaction can currently have, at most,
~24000 inputs and ~111000 outputs (+/- a few hundred) which is well beyond any useful limit.
Additionally, such limits may not be as extensible for future work. It is hard to determine what
is a reasonable limit on transaction size, and I don't think it is useful to have a limit. At worst
we would simply create an invalid transaction if it were too large.

Andrew

📅 Original date posted:2018-06-25 📝 Original message:Hi, On ...

2023-06-07T18:13:11Z

In reply to nevent1q…7rxw
_________________________

📅 Original date posted:2018-06-25
📝 Original message:Hi,

On June 25, 2018 12:47 PM, Tomas Susanka via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> From my perspective those are exactly the points I have felt strongly
> about. I still think "typed records" would be a better choice, but it's
> something I'm willing to compromise on. As I'm looking at the draft, we
> currently have 13 records and only 3 of them have keys... Matejcik was a
> bit keener on this, so we'll try to discuss this more during the week
> and we also look at the draft more carefully to see if we can come up
> with some nit-picks.

So there are a few reasons for not using typed records. Firstly, it is less of a breaking change to retain the key-value map model.

Secondly, it is easier to enforce uniqueness for certain things. For example, in each input, we only want to have one redeemScript and one witnessScript. With a typed records set, we would have to say that only on record of each type is allowed, which means that combiners need to understand types and be able to partially parse the records. However with a key-value model, we can more generically say that every key-value pair must have a unique key which means that combiners do not need to know anything about types and just needs to enforce key uniqueness. Since the type is the only thing in the key for redeemScripts and witnessScripts, this uniqueness automatically applies to this, as well as for other key-value pairs.

Lastly, the typed records model does not save a lot of space in a transaction. Each record has at most one extra byte in the key-value model, with records that must also have keys having no space savings. The data inside each key-value pair far exceeds one byte, so on additional byte per key-value pair isn't all that big of a deal, IMO.

Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180625/a2dc6274/attachment.html>;

📅 Original date posted:2018-06-22 📝 Original message:Hi ...

2023-06-07T18:13:09Z

In reply to nevent1q…nxf6
_________________________

📅 Original date posted:2018-06-22
📝 Original message:Hi all,

After reading the comments here about BIP 174, I would like to propose the following changes:

- Moving redeemScripts, witnessScripts, and BIP 32 derivation paths to per-input and per-output data

I think that by moving these three fields into input and output specific maps, the format will be
easier to read and simpler for signers to parse. Instead of having to be able to parse entire
scripts and extract pubkeys, the signer can simply look at which pubkeys are provided in the inputs
and sign the input based upon the presence of a pubkey for which the signer has a privkey.

A neat trick that fits well with this model is that a plain pubkey (one that is not part of a BIP 32
derivation) can still be put in a BIP 32 derivation path field where the value is just the fingerprint
of the pubkey itself. This would indicate that no derivation needs to be done from the master key, and
the master key is just the specified key itself.

Additionally, by having the redeemScript and witnessScript readily available in the input, signers
do not need to construct a map to find a redeemScript or witnessScript and can instead just look
directly in the input data. There is also no need to include the hashes of these scripts, so the key
is just the type. This also allows us to enforce the requirement for only one redeemScript and one
witnessScript per input easily by continuing to follow the generic rule of unique keys.

By using input specific and output specific fields, there is no need for the input index and the input
count types as all inputs will be accounted for.

- Finalized scriptSig and scriptWitness fields

To determine whether two PSBTs are the same, we can compare the unsigned transaction. To ensure that the
unsigned transactions are the same for two PSBTs with data for the same tx, we cannot put scriptSigs or
scriptWitnesses into it. Thus for each input, two new fields have been added to store the finalized scriptSig
and finalized scriptWitness.

- Mandatory sighash

The sighash type field will be changed from a recommendation to a requirement. Signatures will need to
use the specified sighash type for that input. If a Signer cannot sign for a particular sighash type, it
must not add a partial signature.

- Encoding

I have decided that PSBTs should either be in binary or encoded as a Base64 string. For the latter, several
Bitcoin clients already support Base64 encoding of data (for signed messages) so this will not add any extra
dependencies like Z85 would.

A draft of the revised BIP can be found here: https://github.com/achow101/bips/blob/bip174-rev/bip-0174.mediawiki
If these changes are satisfactory, I will open a PR to the BIPs repo to update the BIP tomorrow. I will also
create test vectors and update the implementation PR'ed to Core.

Andrew

📅 Original date posted:2018-06-20 📝 Original message:Hi, On ...

2023-06-07T18:13:07Z

In reply to nevent1q…mkmk
_________________________

📅 Original date posted:2018-06-20
📝 Original message:Hi,

On June 15, 2018 4:34 PM, Pieter Wuille via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

>
> Hello all,
>
> given some recent work and discussions around BIP 174 (Partially
> Signed Bitcoin Transaction Format) I'd like to bring up a few ideas.
> First of all, it's unclear to me to what extent projects have already
> worked on implementations, and thus to what extent the specification
> is still subject to change. A response of "this is way too late" is
> perfectly fine.

While I agree that the BIP itself should be revised to reflect these suggestions, I fear that it may be too late. I know of a few other developers who have implemented BIP 174 already but have not yet responded to this email.

>
> - Generic key offset derivation
>
> Whenever a BIP32 derivation path does not include any hardened steps,
> the entirety of the derivation can be conveyed as "The private key for
> P is equal to the private key for Q plus x", with P and Q points and x
> a scalar. This representation is more flexible (it also supports
> pay-to-contract derivations), more efficient, and more compact. The
> downside is that it requires the Signer to support such derivation,
> which I don't believe any current hardware devices do.
> Would it make sense to add this as an additional derivation method?

While this is a good idea, I'm not sure that implementers would understand this as it requires knowing the cryptography which makes this possible. As an optional feature, not all wallets would understand it, and those that do could create PSBTs which other wallets do not understand and thus cannot sign even if they have the private keys and actually can sign.

>
> - Hex encoding?
>
> This is a very minor thing. But presumably there will be some standard
> way to store PSBTs as text for copy-pasting - either prescribed by the
> spec, or de facto. These structures may become pretty large, so
> perhaps it's worth choosing something more compact than hexadecimal -
> for example Base64 or even Z85 (https://rfc.zeromq.org/spec:32/Z85/).

Agreed. Are there any encodings that do not have double click breaking characters?

On June 19, 2018 2:38 AM, Jonas Schnelli via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

> I think it could be more flexible (generic) in BIP174.
> It could be just a single child key {32-bit int}, or just a keypath ({32-bit int}]{32-bit int}…) which is very likely sufficient for a HWW to derive the relevant key without the creation of a lookup-window or other „maps".

This ignores all of the other times that a BIP32 keypath needs to be provided. It is not only used for multisig, there may be other times that there are multiple derivation paths and master keys due to multiple inputs and such. Adding a field specific to multisig and HWW only seems pointless and redundant to me.

On June 19, 2018 2:38 AM, Jonas Schnelli via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:

>
> A question to consider is,
> will there be more per-output data? If yes, it might make sense to have
> an output section.

I think it is unlikely that there would be anymore per-output data.

> 3) The sighash type 0x03 says the sighash is only a recommendation. That
>seems rather ambiguous. If the field is specified shouldn't it be binding?

I disagree. It is up to the signer to decide what they wish to sign, not for the creator to specify what to sign. The creator can ask the signer to sign something in a particular way, but it is ultimately up to the signer to decide.

> 4) Is it a good idea to skip records which types we are unaware of? We
> can't come up with a reasonable example, but intuitively this seems as a
> potential security issue. We think we should consider introducing a
> flag, which would define if the record is "optional". In case the signer
> encounters a record it doesn't recognize and such flag is not set, it
> aborts the procedure. If we assume the set model we could change the
> structure to <type><optional flag><length>{data}. We are not keen on
> this, but we wanted to include this idea to see what you think.

The idea behind skipping unknown types is to allow forward compatibility. A combiner written now should be able to combine transactions created in the future with new types as combining is really only just merging the maps together.

> In general, the standard is trying to be very space-conservative,
> however is that really necessary? We would argue for clarity and ease of
> use over space constraints. We think more straightforward approach is
> desired, although more space demanding. What are the arguments to make
> this as small as possible? If we understand correctly, this format is
> not intended for blockchain nor for persistent storage, so size doesn’t
> matter nearly as much.

Size is not really a constraint, but we do not want to be unnecessarily large. The PSBT still has to be transmitted to other people. It will likely be used by copy and pasting the string into a text box. Copying and pasting very long strings of text can be annoying and cumbersome. So the goal is to keep the format still relatively clear while avoiding the duplication of data.

Andrew