Nostr notes by Stefan Richter [ARCHIVE]

📅 Original date posted:2022-03-20 📝 Original message: Good ...

2023-06-09T13:05:33Z

In reply to nevent1q…x95w
_________________________

📅 Original date posted:2022-03-20
📝 Original message:
Good morning everyone,

with regards to zerobasefee, I think that the argument that HTLCs are
costly doesn't quite hold up because they are always free to an
attacker as it stands. However, I fully agree with Zmn's opinion that
it's not necessary to bang our head against any opposition to this
because we can simply follow his excellent method for overweighing
base fee. I believe this is a very natural approach to let the market
decide on the relative importance of optimized routing vs base fees.

As to Martin's approximation research, I have asked myself similar
questions. Unfortunately, the paper you cite is paywalled and not
available at sci-hub, so I haven't read it. FWIW, I believe I have a
simple proof that minimum cost flow preserves approximation FACTORS:

Let O be the original problem and A the approximated problem such that
every flow in O can be mapped 1:1 to a flow in A and vice versa. Let
every edge e in O be represented by a set of edges in A whose total
cost is within a factor (1+epsilon) for every possible flow (could be
over- or underestimating). Note that this means that every flow in O
has cost within a factor (1+epsilon) for the corresponding flow in A
and vice versa.

Now let f_a be the min cost flow in A and f_o the min cost flow in O.
Assume that c(f_o)(1+epsilon)<c(f_a). Then f_o corresponds to a flow
in A that is cheaper than f_a, but that's impossible because f_a is
the min cost flow.
QED.

Problems might arise anyway because we represent probabilities only
logarithmically in the cost, so that a factor of (1+epsilon)
corresponds to an exponent (1+epsilon) for the probabilities. But René
seems optimistic that the resulting flows look good enough in
practice.

I am still optimistic that exact solvers with something like the cost
scaling approach might also be feasible (as long as they produce
integer flows), but I am happy that this simple approximation approach
seems good enough. This should save us a lot of work because there are
many linear min cost solvers available that represent years of
cumulative work in optimization research.

Cheers
Stefan

Am Sa., 19. März 2022 um 22:09 Uhr schrieb Martin via Lightning-dev
<lightning-dev at lists.linuxfoundation.org>:
>
> Dear Carsten, Rene and fellow lightning developers,
>
> Regarding the approximation quality of the minimum convex cost flow formulation for multi-part payments on the lightning network [1] and Carsten's discussion points on Twitter [2] and on the mailing list:
>
> > 8) Quality of Approximation
> >
> > There are some problems in computer science that are hard/impossible to
> > approximate, in the sense that any kind of deviation from the optimum
> > could cause the computed results to be extremely bad. Do you have some
> > idea (or proof) that your kind of approximation isn't causing a major
> > issue? I guess a piece-wise linearization with an infinite number of
> > pieces corresponds to the optimal result. Given a finite number of
> > pieces, how large is the difference to the optimum?
>
> I did some literature research and came across an insightful paper [3] by Dorit Hochbaum from 1993, that proves proximity results for integer and continuous optimal solutions of the minimum convex cost flow as well as proximity results of the optimal solutions for a piecewise linear approximation and the original problem.
>
> Admittedly theoretical results, however, it further underpins that a piecewise linear approximation is a reasonable approach to find optimal flows and even shows that searching for optimal solutions on the continuous domain (e.g. with descent methods from convex optimization) also gives near-optimal solutions on the integer domain.
>
> Cheers,
> Martin
>
> [1] https://arxiv.org/abs/2107.05322
> [2] https://twitter.com/renepickhardt/status/1502293438498234371
> [3] https://www.worldscientific.com/doi/abs/10.1142/9789812798190_0005
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev

📅 Original date posted:2021-11-22 📝 Original message: I ...

2023-06-09T13:04:26Z

In reply to nevent1q…yat4
_________________________

📅 Original date posted:2021-11-22
📝 Original message:
I also don't believe putting a choice of more or less seconds expectation
in the UI makes for a great user experience. IMHO the goal should just be:
give the user an estimate of fees necessary to succeed within a reasonable
time. Maybe give them an option to optimize for fees only if they are
really cheap and don't care at all if the payment succeeds.

Cheers
Stefan

ZmnSCPxj via Lightning-dev <lightning-dev at lists.linuxfoundation.org>
schrieb am Mo., 22. Nov. 2021, 00:53:

> Good morning Dave,
>
> > If LN software speculatively chooses a series of attempts with a similar
> > 95%, accounting for things like the probability of a stuck payment (made
> > worse by longer CLTV timeouts on some paths), it could present users
> > with the same sort of options:
> >
> > ~1 second, x fee
> > ~3 seconds, y fee
> > ~10 seconds, z fee
> >
> > This allows the software to use its reliability scoring efficiently in
> > choosing what series of payment attempts to make and presents to the
> > user the information they need to make a choice appropriate for their
> > situation. As a bonus, it makes it easier for wallet software to move
> > towards a world where there is no user-visible difference between
> > onchain and offchain payments, e.g.:
> >
> > ~1 second, w fee
> > ~15 seconds, x fee
> > ~10 minutes, y fee
> > ~60 minutes, z fee
>
> This may not match ideally, as in the worst case a forwarding might be
> struck by literal lightning and dropped off the network while your HTLC is
> on that node, only for the relevant channel to be dropped onchain days
> later when the timeout comes due.
> Providing this "seconds" estimate does not prepare users for the
> possibility of such black swan events where a high fee transaction gets
> stalled due to an accident on the network.
>
> On the other hand, humans never really handle black swan events in any
> reasonably way anyway, and 95% of the time it will probably achieve that
> number of estimated seconds or less.
> Even the best onchain estimators fail when a thundering herd of
> speculators decides to trade Bitcoin based on random crap from the
> noosphere.
>
> The processing to figure out a payment plan also becomes significant at
> the "seconds" level, especially if you switch to mincostflow rather than
> shortestpath.
> This means the CPU speed of the local node may become significant, or if
> you are delegating pathfinding to a trusted server, the load on that
> trusted server becomes significant.
> Sigh.
>
> Why not just ask for a fee budget for a payment, and avoid committing
> ourselves to paying within some number of seconds, given that the seconds
> estimate may very well vary depending on local CPU load?
> Would users really complain overmuch if the number of seconds is not
> provided, given that we cannot really estimate this well?
>
> Regards,
> ZmnSCPxj
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20211122/60a58188/attachment.html>;

📅 Original date posted:2021-11-15 📝 Original message: ...

2023-06-09T13:04:25Z

In reply to nevent1q…vt7y
_________________________

📅 Original date posted:2021-11-15
📝 Original message:
Actually, if you look into our paper, the theory tells us the following:

1) A weighted sum of different cost aspects is attractive because it
remains convex if all the aspects are convex themselves. This cannot be
said of other methods like the harmonic mean, which kind of forces our hand
if we aim to really calculate optimal flows.

2) even in the single path case, finding a route that optimizes one goal
(say, reliability) while ensuring that another cost aspect remains under
some boundary, is a (weakly) NP hard problem. My interpretation of this
fact is that while it is certainly possible to find suitable factors for
the linear combination (by, say, gradient descent methods), we cannot
expect a method that is simple and works for every conceivable graph every
time. In practice, we have observed that the factor needs to be varied over
multiple orders of magnitude to make a meaningful impact. Mapping this onto
an easy user interface (e.g. your suggestion of a linearly feeling value
between 0 and 1) will need some trial and error engineering, IMHO.

Cheers,
Stefan

René Pickhardt via Lightning-dev <lightning-dev at lists.linuxfoundation.org>
schrieb am Mo., 15. Nov. 2021, 12:50:

> Dear Joost,
>
> First I am happy that you also agree that reliability can and should be
> expressed as a probability as discussed in [0].
>
> The problem that you address is that of feature engineering[1]. Which
> consists of two (or even more) steps:
>
> 1.) Feature selection: That means in payment delivery we will compute a
> min cost flow [2] with a chosen cost function (historically people used
> dijkstra seach for single paths with the cost function representing the
> weights on the edges of the graph -which is what most folks currently still
> do). While [2] and I personally agree with you that the cost function
> should be a combination the two features fees and reliability (as in
> successprobability) Matt Corallo righfully pointed out [3] that other
> features might be chosen in the future to deliver more optimal results. For
> example implementations currently often use CLTV as a feature (which I
> honestly find horrible) and I am currently investigating if one could add
> latency of channels or - for known IP addresses - either the geo distance
> or IP distance.
>
> 2.) Combining features: This is the question that you are asking. Often
> people use a linear weighted sum to combine features. This is what often
> happens implicitly in neural networks. While this is often good enough and
> while it is often practical to either learn the weights or give users a
> choice there are many situation where the weighted linear sum does not work
> well with the selected features. An example for the weighted sum is the
> risk-factor in c-lightning that could have been used to decide if one
> wanted the dijkstra seach to either optimize for CLTV delta or for paid
> routing fees. Also in our paper [2] in which we discuss the same two
> features that you mentioned we explain how a linear sum of two features can
> be optimal due to the lagrangian bounding principle. However in practice
> (of machine learning) it has been shown that using the harmonic mean [4]
> between features often works very well without the necessity to learn a
> weight / parameter. This has for example been done when c-lightnign
> recently switched to probabilistic path finding [5]. In this thread you
> find a long discussion and evaluation how the harmonic mean outperformed
> the linear sum.
>
> I think the main issue that you address here is that there is no universal
> truth for situations like this. In practice only tests and experience will
> help us to make good decisions.
>
> with kind Regards Rene
>
> [0]:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-March/002984.html
> [1]: https://en.wikipedia.org/wiki/Feature_engineering
> [2]: https://arxiv.org/abs/2107.05322
> [3]:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-September/003219.html
> [4]: https://en.wikipedia.org/wiki/Harmonic_mean
> [5]: https://github.com/ElementsProject/lightning/pull/4771
>
>
>
>
> On Mon, Nov 15, 2021 at 4:26 PM Joost Jager <joost.jager at gmail.com> wrote:
>
>> In Lightning pathfinding the two main variables to optimize for are
>> routing fee and reliability. Routing fee is concrete. It is the sat amount
>> that is paid when a payment succeeds. Reliability is a property of a route
>> that can be expressed as a probability. The probability that a route will
>> be successful.
>>
>> During pathfinding, route options are compared against each other. So for
>> example:
>>
>> Route A: fee 10 sat, success probability 50%
>> Route B: fee 20 sat, success probability 80%
>>
>> Which one is the better route? That depends on user preference. A patient
>> user will probably go for route A in the hope of saving on fees whereas for
>> a time-sensitive payment route B looks better.
>>
>> It would be great to offer this trade-off to the user in a simple way.
>> Preferably a single [0, 1] value that controls the selection process. At 0,
>> the route is only optimized for fees and probabilities are ignored
>> completely. At 1, the route is only optimized for reliability and fees are
>> ignored completely.
>>
>> But how to choose between the routes A and B for a value somewhere in
>> between 0 and 1? For example 0.5 - perfect balance between reliability and
>> fee. But what does that mean exactly?
>>
>> Anyone got an idea on how to approach this best? I am looking for a
>> simple formula to decide between routes, preferably with a reasonably sound
>> probability-theoretical basis (whatever that means).
>>
>> Joost
>> _______________________________________________
>> Lightning-dev mailing list
>> Lightning-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>>
>
>
> --
> https://www.rene-pickhardt.de
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20211115/42fe9021/attachment-0001.html>;

📅 Original date posted:2021-11-15 📝 Original message: It ...

2023-06-09T13:04:25Z

In reply to nevent1q…xe9n
_________________________

📅 Original date posted:2021-11-15
📝 Original message:
It seems to me there can be no such law unless P=NP. Which would also imply
Bitcoin is worthless.

Joost Jager <joost.jager at gmail.com> schrieb am Mo., 15. Nov. 2021, 13:44:

> One direction that I explored is to start with a statement by the user in
> this form:
>
> "If there is a route with a success probability of 50%, then I am willing
> to pay up to 1.8x the routing fee for an alternative route that has a 80%
> success probability"
>
> I like this because it isn't an abstract weight or factor. It is actually
> clear what this means.
>
> What I didn't yet succeed in is to find a model where I can plug in 50%,
> 80% and 1.8x and generalizes it to arbitrary inputs A% and B%. But it seems
> to me that there must be some probabilistic equation / law / rule / theorem
> / ... that can support this.
>
> Joost.
>
> On Mon, Nov 15, 2021 at 4:25 PM Joost Jager <joost.jager at gmail.com> wrote:
>
>> In Lightning pathfinding the two main variables to optimize for are
>> routing fee and reliability. Routing fee is concrete. It is the sat amount
>> that is paid when a payment succeeds. Reliability is a property of a route
>> that can be expressed as a probability. The probability that a route will
>> be successful.
>>
>> During pathfinding, route options are compared against each other. So for
>> example:
>>
>> Route A: fee 10 sat, success probability 50%
>> Route B: fee 20 sat, success probability 80%
>>
>> Which one is the better route? That depends on user preference. A patient
>> user will probably go for route A in the hope of saving on fees whereas for
>> a time-sensitive payment route B looks better.
>>
>> It would be great to offer this trade-off to the user in a simple way.
>> Preferably a single [0, 1] value that controls the selection process. At 0,
>> the route is only optimized for fees and probabilities are ignored
>> completely. At 1, the route is only optimized for reliability and fees are
>> ignored completely.
>>
>> But how to choose between the routes A and B for a value somewhere in
>> between 0 and 1? For example 0.5 - perfect balance between reliability and
>> fee. But what does that mean exactly?
>>
>> Anyone got an idea on how to approach this best? I am looking for a
>> simple formula to decide between routes, preferably with a reasonably sound
>> probability-theoretical basis (whatever that means).
>>
>> Joost
>>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20211115/3533a287/attachment.html>;

📅 Original date posted:2021-08-30 📝 Original message: Hi ...

2023-06-09T13:03:38Z

In reply to nevent1q…jpaj
_________________________

📅 Original date posted:2021-08-30
📝 Original message:
Hi Zmn!

While you have some interesting thoughts about the implementation of
min-cost flow based routing, I feel that there are at least two grave
misunderstandings here:

First, the problem with the base-fee or any similar constant is that
they make the fee function concave at the point where the flow
increases from 0 to 1. That is, there is a jump from 0 to
base-fee+1*prop_fee or similar, and the function does increase slower
from there. This is the case for any function that includes a constant
per edge that is only paid IF there is non-zero flow along that edge.

Second, while there might be additional problems in the disect-phase
(which I am sure can be handled by a plethora of techniques), the fact
remains that the min-cost flow problem itself is NP-hard when the
cost-function is of this form.

I realise that this is a fairly theoretical term, so let me shortly
describe what it means: We know (there is a simple mathematical proof
for this, see reference in our paper) that if we could find an
algorithm that solves the min-cost flow problem ON ALL GRAPHS for all
cost-functions of this form (basically, a constant per edge) in
polynomial time, we could then use this algorithm to solve ANY problem
in NP in polynomial time. This means, we would have an algorithm that
can find a solution to any problem in polynomial time whose solutions
can be VERIFIED in polynomial time. Because there are thousands of
problems of this kind that the smartest people in the world have tried
to solve for decaded without success (and just one success would be
enough to solve all of them!), it is highly doubtful that such an
algorithm exists. That is why we can be fairly certain that we cannot
in general solve the min-cost flow problem for concave functions in
polynomial time.

HOWEVER, this does not mean that in practice it isn't possible to find
good solutions for the subset of problems we are likely to encounter
in our application. It might be that the LN-graph is of a certain
structure that makes finding a solution easier. It is certain that we
do not need the optimal solution at all and approximations will be
easier to find. All we know is that we do know how to solve the
problem generally for convex functions, and that we are very confident
that we cannot in general solve it optimally for concave functions.

Cheers
Stefan

Am Mo., 30. Aug. 2021 um 12:58 Uhr schrieb ZmnSCPxj via Lightning-dev
<lightning-dev at lists.linuxfoundation.org>:
>
> Good morning list,
>
> It seems to me that the cost function defined in Pickhard-Richter
> can in fact include a base fee:
>
> -log(success_probability) + fudging_factor * amount * prop_fee + fudging_factor * base_fee
>
> (Rant: why do mathists prefer single-symbol var names, in software
> engineering there is a strong injunction **against** single-char
> var names, with exceptions for simple numeric iteration loop vars,
> I mean keeping track of the meaning of each single-symbol var name
> takes up working memory space, which is fairly limited on a wetware
> CPU, which is precisely why in software engineering there is an
> injunction against single-char var names, srsly.)
>
> It seems to me that the above is "convex", as the `fudging_factor`
> for a run will be constant, and the `base_fee` for the channel
> would also be constant, and since it seems to me, naively, that
> the paper defines "convex" as "the second derivative >= 0 for all
> `amount`", and since the derivative of a constant is 0, the above
> would still remain convex.
> (I am not a mathist and this conjecture might be completely asinine.)
>
> So, it seems to me that the *real* reason for `#zerobasefee` is
> not that the **mincostflow** algorithm cannot handle non-zero `base_fee`,
> but rather, the **disect** phase afterwards cannot handle non-zero
> `base_fee`.
>
> For example, suppose the minflowcost algorithm were instructed to
> deliver 3,000msat from `S` to `D`, and returned the following flow:
>
> S -->3000--> A ->1000-> B
> | |
> | 1000
> | v
> +-->2000-> C ->3000-> D
>
> In the "disect" phase afterwards, the above flow solution would have
> to be split into two sub-payments, a 1000 sub-payment `S-A-B-C-D`
> and a 2000 sub-payment `S-A-C-D`.
>
> However, this does mean that the base cost for `C->D` and `S->A` in
> the above flow will pay *twice* the base cost than what the above
> cost function would have computed, because in reality two independent
> payments pass through those channels.
>
> Thus, the current Pickhardt-Richter scheme cannot work with non-zero
> base fees if it were modified to consider fees, but not due to the
> mincostflow algorithm, rather because due to the disect algorithm.
>
> ### Converting Non-Zero Base Fees To Proportional Fees
>
> An alternative solution would be to optimize for a *maximum* fee
> rather than optimize for the *exact* fee.
>
> We can do this by virtually splitting up the entire payment into
> smaller bunches of value, and asking mincostflow to solve individual
> bunches rather than individual msats.
>
> For example, we can decide that the smallest practical HTLC would
> be 1000 msat.
> Let us call this the "payment unit", and the mincostflow algo
> solves for paying in these payment units instead of 1msat units.
> (Actual implementations would need to have some heuristic or
> reasoned rule-of-thumb assumption on what the smallest practical
> HTLC would be.)
>
> In our example, suppose we need to send 2001msat from `S` to `D`,
> with the payment unit being 1000 msat.
> Then we would actually have the mincostflow algorithm work to
> deliver 3 payment units (3 * 1000msat) from `S` to `D`.
>
> Let us suppose that the mincostflow algorithm returns the following
> flow:
>
> S -->3-----> A ->1----> B
> | |
> | 1 |
> | v
> +-->2----> C ->3----> D
>
> Actually, what the mincostflow algorithm uses as a cost function
> would be:
>
> -log(success_probability) + fudging_factor * amount * prop_fee * payment_unit + fudging_factor * base_fee * amount
>
> == -log(success_probability) + fudging_factor * amount * (prop_fee * payment_unit + base_fee)
>
> where: amount is in units of 1000msat, our smallest practical HTLC.
> payment_unit is the unit, i.e. 1000msat.
>
> What the above means is that the mincostflow algorithm *allocates*
> `3 * base_fee` for `C->D`, since the `amount` flowing through
> `C->D` would be 3.
> However, when we pass the above flow to the disect algorithm, it
> would actually only split this into 2 sub-payments, so the
> actual payment plan would only pay `2 * base_fee` for the
> `C->D` leg on both sub-payments.
>
> In short, this effectively converts the base fee to a proportional
> fee, removing the zerobasfee requirement imposed by the disect
> algorithm.
>
> That is, the cost computed by the mincostflow algorithm is really a
> maximum cost budget that the subsequent disect algorithm could later
> spend.
>
> In effect, this converts the base fee to a proportional fee.
>
> This may be acceptable in practice.
> This approximation has a bias against non-zerobasefee --- it would
> treat those channels as being far more expensive than they actually
> would end up being in an *actual* payment attempt --- but at least
> does not *require* zerobasefee.
> It would be able to still use non-zerobasefee channels if those are
> absolutely required to reach the destination.
>
> This should at least help create a practical payment algorithm that
> handles current LN with nonzerobasefee, and provide an impetus
> towards making zerobasefee a reality.
>
> Note that this approximation would be non-optimal --- there may
> be solutions that provide lower costs than the output of this
> approximation would provide.
>
> It may be practical to have the payment unit be adjustable by a
> higher-level (but still automated) process.
> For example, it may be considered that having more than 500 splits
> would be heuristically determined to be impractical, in which
> case the payment unit could be the actual amount divided by 500
> rounded up.
> Then if that results in extreme overestimation of base costs
> (i.e. the subsequent disect stage results in far fewer than 500
> splits) the payment unit could be adjusted upwards.
> Similarly, if that results in a payment plan that has high
> base fees, the payment unit could be adjusted downwrds.
>
> --
>
> Similarly, having particular split amounts is helpful to reduce
> the ability of intermediate nodes snooping on payment amounts,
> so having a particular fixed payment unit may also be useful
> nevertheless, even if this approximation is otherwise undesirable.
>
> That is, in the above example, we just send 3 sub-payments
> instead of the minimum 2, so that every sub-payment is always
> uniformly 1000msat.
> This leaks the least amount of information to intermediate nodes,
> especially after payment decorrelation is widely deployed.
> (this is the same principle for why a uniform unit amount is
> needed in practical CoinJoin implementations; it increases the
> anonymity set of individual sub-payments.)
>
> This would be helped greatly by payment decorrelation;
> intermediate nodes would remain uncertain that multiple
> HTLCs with the same amount and the same in-channel and
> out-channel are from the same overall payment or not.
> Assuming many software uses the same payment unit consistently,
> less information can be extracted by surveillors, and the
> increased cost may be considered justifiable in paying for
> additional privacy.
>
> --
>
> ### Whole-flow Payments
>
> Suppose we were to instead modify the LN protocol such that,
> instead of MPP having individual sub-payments that split at the
> source and join at the destination, we allow the source to
> instruct arbitrary intermediate nodes to join and/or split
> payments.
>
> That is, suppose LN allowed the source to indicate to arbitrary
> forwarding nodes to split payments and join them.
> (This would require nearly all nodes to upgrade, which may be
> impractical, but let us consider its implications first without
> committing to actually doing this.)
>
> A node that is a join node would be instructed to wait for all
> incoming HTLCs to arrive before they create any outgoing
> HTLCs.
> A node that is a split node would make more than one outgoing
> HTLCs.
> Forwarding nodes can be both join and split nodes.
>
> This would allow the output of the mincostflow algorithm to be
> used directly, without the problematic disect stage.
> Nodes that have more than one in-flow would be join nodes,
> while nodes with more than one out-flow would be split nodes.
>
> Returning to the example:
>
> S -->3000--> A ->1000-> B
> | |
> | 1000
> | v
> +-->2000-> C ->3000-> D
>
> In the above, `S` sends a *single* payment out on channel
> `S->A`, and instructs `A` to split the payment.
> Then `S` also instructs `C` to join two payments, resulting
> in a single HTLC out on `C->D` if `C` receives on both
> `A->C` and `B->C`.
>
> The effect is that, for a single payment plan (presented as a
> flow, outputted by a mincostflow algorithm) for each channel
> with nonzero flow, there would only be one HTLC that gets paid
> for using a single `base_fee`.
>
> Then, the base fee computed by the cost function would be exact
> and there would be no underestimation of the base fee for
> non-zerobasefee channels.
>
> Let us then consider the practicality of this scheme.
>
> If a hop were to fail to deliver, then some join nodes would
> be left hanging, waiting for an incoming HTLC that will not
> arrive.
>
> The failure would have to be handled by a split node, when it
> receives a `update_fail_htlc`.
> That split node cannot propagate the failure backwards to the
> source, since it would still have other outgoing HTLCs in
> play, which it cannot safely ignore; it can only propagate
> the failure backwards if all the other outgoing HTLCs also
> fail.
>
> To support this, a split node could give a request-to-fail
> signal to its other outgoing HTLCs.
> Then, any other join nodes still waiting for an incoming
> HTLC (and would not yet have created any outgoing HTLCs,
> or for the destination, would not have claimed any incoming
> HTLCs) would fail all its incoming HTLCs.
> Otherwise, an intermediate node receiving a request-to-fail
> signal from any incoming HTLCs would simply propagate
> this request-to-fail outwards to all its outgoing HTLCs.
>
> This implies that for this scheme, if *any* hop fails, the
> entire payment fails and has to be restarted "from the top".
>
> This is in contrast with the current scheme of splitting
> only at the source and joining only at the destination.
> In the current LN, if *any* hop of an MPP fails, only the
> sub-payment(s) going through that hop will need to be
> retried.
> Other sub-payment(s) can keep being "in play", and the
> source, having all the information needed, only needs to
> recompute the failing outgoing payments.
>
> On the other hand, the same information (particular hops are
> failing) would be propagated back to the source, and the
> source having to recompute the entire flow rather than just
> some subset of the payments is not much worse computationally
> (work is still dominated by `O((m*m+m*n)*log n)` where `m` is
> the channels of the *entire network* and `n` is the nodes
> of the *entire network*, amount is just a piddling little
> `O(log(u))` term).
> It also removes the need to virtually reduce the capacity of
> intermediate hops while other sub-payments are in play
> (however, a practical implementation would need to consider
> the possibility of multiple *simultaneous* outgoing payments
> anyway and would still retain this reduction of capacity in
> case a new parallel outgoing payment is triggered while one
> is still ongoing).
>
> This leads us to some fairly interesting choices:
>
> * [ ] Whole-flow payment is a brilliant idea and we should
> totally do it. `#deletethezerobasefeedebate`
> * [ ] Whole-flow payment is a horrible idea and we should
> totally not do it.
> * [ ] As we can see, the LN community needs to get together
> for better payment success, and supporting whole-flow
> payments is worse than setting `basefee=0`. `#zerobasefee`
> * [ ] As we can see, the current LN provides a vital service
> (partial failure of MPP) and intermediate nodes need to be
> compensated for this service. `#nonzerobasefee`
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev

📅 Original date posted:2021-08-31 📝 Original message: Am ...

2023-06-09T13:03:36Z

In reply to nevent1q…ejf2
_________________________

📅 Original date posted:2021-08-31
📝 Original message:
Am Di., 31. Aug. 2021 um 10:29 Uhr schrieb ZmnSCPxj via Lightning-dev
<lightning-dev at lists.linuxfoundation.org>:

> For myself, I think a variant of Pickhardt-Richter payments can be created which *adapts to* the reality of the current network where `base_fee > 0` is common, but is biased against `base_fee > 0`, can be a bridge from the current network with `base_fee > 0` and a future with `#zerobasefee`.

I have been thinking about your idea (at least what I understood of
it) of using amount*prop_fee + amount*base_fee/min_flow_size, where
min_flow_size is a suitable quantization constant (say, 10k or 100k
sats, may also chosen dynamically), as a component of the cost
function, and I am pretty sure it is great at achieving exactly what
you are proposing here. This is a nicely convex (even linear in this
component) function and so it's easy to find min-cost flows for it. It
solves the problem (that I hadn't thought about before) that you have
pointed out in splitting flows into HTLCs. If you use
min_flow_size=max_htlc_size, it is even optimal (for this
min_flow_size). If you use a smaller min_flow_size, it is still
optimal for channels with base_fee=0 but overestimates the fee for
channels with base_fee>0, and is less accurate the smaller the
min_flow_size and the larger the base_fee. So it will be biased
against channels with larger base_fee. But notice that with min-cost
flows, we are rarely looking for the cheapest solution anyway, because
these solutions (if they include more than one path) will usually
fully saturate the cheapest channels and thus have very low success
probability. So all in all, I believe you found a great practical
solution for this debate. Everybody is free to use any base_fee they
chose, we get a workable cost function, and I conjecture that
economics will convince most people to choose a zero or low base_fee.

Cheers
Stefan

📅 Original date posted:2021-08-24 📝 Original message: Good ...

2023-06-09T13:03:34Z

In reply to nevent1q…h3sy
_________________________

📅 Original date posted:2021-08-24
📝 Original message:
Good Morning Zmn!

If you'd like to understand the min-cost flow problem and algorithms
better, I would really recommend the textbook we have been citing
throughout the paper.

The algorithm you have found has a few shortcomings. It'll only work for
the linear min-cost flow problem, and it is very slow. In reality, we need
to deal with convex cost functions, and the algorithm we have used so far
uses an approach called capacity scaling in order to be much faster. It is
indeed complex enough that it has taken us about two months to understand
and implement it, discovering a nice heuristic in the process of making
mistakes.

Separable in this context means that you can simply add up the costs of the
edges to get the total costs. On second thought, your definition would
probably work here , by redefining adding up.

Convex here means that for any two amounts x, y, the cost function f in the
interval x, y does not lie below the line connecting the two points (x,
f(x)) and (y, f(y)). The intuition here is that a linear approximation
never overestimates the real cost. I guess one would need a more involved
definition for your more complex coordinates.

Cheers,
Stefan

ZmnSCPxj <ZmnSCPxj at protonmail.com> schrieb am Di., 24. Aug. 2021, 01:06:

> Good morning Stefan,
>
> > Hi Zmn! That is some amazing lateral thinking you have been applying
> there. I'm quite certain I haven't understood everything fully, but it has
> been highly entertaining to read. Will have to give it a closer read when I
> get some time.
> >
> > As a first impression, here are some preliminary observations: While I
> highly like the Haskell-style datatype, and the algorithm we use does
> mostly use Dijkstra pathfinding, I think what is really important in your
> definition is the computeCost definition. This is what we would call the
> cost function IIUC, and in order to be able to solve min-cost flow problems
> it generally has to be separable and convex. I believe your datatype merely
> hides the fact that it is neither.
>
> Well, it really depends on what min flow cost algorithms actually assume
> of the "numbers" being used.
>
> For instance, it is well known that the Dijkstra-A\*-Greedy family of
> algorithms do not handle "negative costs".
> What it really means is that the algorithms assume:
>
> a + b >= a
> a + b >= b
>
> This holds if `a` and `b` are naturals (0 or positive), but not if they
> are integers.
> 1 + -1 = 0, and 0 >= 1 is not true, thus the type for costs in those
> algorithms cannot be integer types, they have to be naturals.
> However if you restrict the type to naturals, `a + b >= a` holds, and
> thus Dijkstra and its family of algorithms work.
>
> Thus, if you are going to use Dijkstra-A\*-Greedy, you "only" need to have
> the following "operations":
>
> `+` :: Cost -> Cost -> Cost
> `<` :: Cost -> Cost -> Bool
> zero :: Cost
>
> With the following derived operations:
>
> a > b = b < a
> a >= b = not (a < b)
> a <= b = not (b < a)
> a == b = (a >= b) && (a <= b)
> a /= b = (a < b) || (a > b)
>
> And following the laws:
>
> forall (a :: Cost) => a + zero == a
> forall (a :: Cost) => zero + a == a
> forall (a :: Cost, b :: Cost) => a + b == b + a
> forall (a :: Cost, b :: Cost, c :: Cost) => (a + b) + c == a + (b + c)
> forall (a :: Cost, b :: Cost) => a + b >= a
> forall (a :: Cost, b :: Cost) => a + b >= b
>
> As a non-mathist I have no idea what "separable" and "convex" actually
> mean.
> Basic search for "convex" and "concave" tends to show up information in
> geometry, which I think is not related (though it is possible there is some
> extension of the geometric concept to pure number theory?).
> And definitions on "separable" are not understandable by me, either.
>
> What exactly are the operations involved, and what are the laws those
> operations must follow, for the data type to be "separable" and "convex"
> (vs."concave")?
>
> I guess my problem as well is that I cannot find easy-to-understand
> algorithms for min cost flow --- I can find discussions on the min cost
> flow "problem", and some allusions to solutions to that problem, but once I
> try looking into algorithms it gets quite a bit more complicated.
>
> Basically: do I need these operations?
>
> `*` :: Cost -> Cost -> Cost
> `/` :: Cost -> Cost -> Cost --- or Maybe Cost
>
> If not, then why cannot `type Cost = UnifiedCost`?
>
>
> For example, this page:
> https://www.topcoder.com/thrive/articles/Minimum%20Cost%20Flow%20Part%20Two:%20Algorithms
>
> Includes this pseudocode:
>
> Transform network G by adding source and sink
> Initial flow x is zero
> while ( Gx contains a path from s to t ) do
> Find any shortest path P from s to t
> Augment current flow x along P
> update Gx
>
> If "find any shortest path" is implemented using Dijkstra-A\*-Greedy, then
> that does not require `Cost` to be an actual numeric type, they just
> require a type that provides `+`, `<`, and `zero`, all of which follow the
> laws I pointed out, *and no more than those*.
> `UnifiedCost` follows those laws (tough note that my definition of `zero`
> has a bug, `successProbability` should be `1.0` not `0`).
>
> In short --- the output of the cost function is a `UnifiedCost` structure
> and ***not*** a number (in the traditional sense).
>
> Basically, I am deconstructing numbers here and trying to figure out what
> makes them tick, and seeing if I can use a different type to provide the
> "tick".
>
>
> Regards,
> ZmnSCPxj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210824/75aad701/attachment-0001.html>;

📅 Original date posted:2021-08-21 📝 Original message: Hi ...

2023-06-09T13:03:33Z

In reply to nevent1q…nxh3
_________________________

📅 Original date posted:2021-08-21
📝 Original message:
Hi Zmn! That is some amazing lateral thinking you have been applying there.
I'm quite certain I haven't understood everything fully, but it has been
highly entertaining to read. Will have to give it a closer read when I get
some time.

As a first impression, here are some preliminary observations: While I
highly like the Haskell-style datatype, and the algorithm we use does
mostly use Dijkstra pathfinding, I think what is really important in your
definition is the computeCost definition. This is what we would call the
cost function IIUC, and in order to be able to solve min-cost flow problems
it generally has to be separable and convex. I believe your datatype merely
hides the fact that it is neither.

Intuitively, I think that any cost function that implies a fixed cost (that
is, independent of the amount, though it might be different for every edge)
per edge is concave and in theory problematic for min-cost flow algorithms
because you could reduce some kind of NP-hard selection problem to it. I
believe that applies to most if not all of your ideas in the text. Again, I
think we should think more about how much of a problem that is in practice,
because we do have tools like approximation and parameterized algorithms,
as well as heuristics, and I also believe that, say, a moderate base fee
will not change the optimal flow much, because this will always prefer
large Htlcs anyway in order to optimize probability.

I am really grateful that you have been taking the time to read and
understand our paper and have been thinking further in this fascinating
way. I am certain good things will come of it in time.

Cheers,
Stefan

ZmnSCPxj <ZmnSCPxj at protonmail.com> schrieb am Sa., 21. Aug. 2021, 03:49:

>
> > Alternative Pathfinding?
> > ========================
>
>
> Or to put this section more succinctly: Why should cost be a number?
>
> What operations do the minimum cost flow algorithms demand of this thing
> called "cost", and can we provide those operations using something which is
> not a number but is instead a different structure?
> What is the minimal interface that the mincostflow algo demands of this
> "cost" datatype?
>
> Regards,
> ZmnSCPxj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210821/02ef209c/attachment.html>;

📅 Original date posted:2021-08-17 📝 Original message: Good ...

2023-06-09T13:03:29Z

In reply to nevent1q…sclj
_________________________

📅 Original date posted:2021-08-17
📝 Original message:
Good morning Zmn!

ZmnSCPxj <ZmnSCPxj at protonmail.com> schrieb am Mo., 16. Aug. 2021, 10:27:

>
> A reason why I suggest this is that the cost function in actual
> implementation is *already* IMO overloaded.
>
> In particular, actual implementations will have some kind of conversion
> between cltv-delta and fees-at-node.
>

That's an interesting aspect. Would this lead to a constant per edge if
incorporated in the cost function? If so, this would lead to another
generally hard problem, which, again, needs to be explored more in the
concrete cases we have here to see if we can still solve/approximate it.

However, I think that in practice, most users cannot intuitively understand
> `riskfactor`.
>

I don't think they have to. Only people like you who write actual software
probably need to.

> Similarly, I think it is easier for users to think in terms of "fee
> budget" instead.
>
> Of course, algorithms should try to keep costs as low as possible, if
> there are two alternate payment plans that are both below the fee budget,
> the one with lower actual fee is still preferred.
> But perhaps we should focus more on payment success *within some fee and
> timelock budget*.
>
> Indeed, as you point out, your real-world experiments you have done have
> involved only probability as cost.
> However, by the paper you claim to have sent 40,000,000,000msat for a cost
> of 814,000msat, or 0.002035% fee percentage, far below the 0.5% default
> `maxfeepercent` we have, which I think is fairly reasonable argument for
> "let us ignore fees and timelocks unless it hits the budget".
> (on the other hand, those numbers come from a section labelled
> "Simulation", so that may not reflect the real world experiments you had
> --- what numbers did you get for those?)
>

René is going to publish those results very soon.

Regarding payment success *within some fee and timelock budget*: the
situation is a little more complex than it appears. As you have pointed
out, at the moment, most of the routes are very cheap (too cheap, IMHO), so
you have to be very unlucky to hit an expensive flow. So in the current
environment, your approach seems to work pretty well, which is also why we
first thought about it.

Unfortunately, as you know, we have to think adversarially in this domain.
And it is clear that if we simply disregarded fees in routing, people would
try to take advantage of this. If we just set a fee budget, and try again
if it is missed, then I see some problems arise: First, what edges do you
exclude in the next try? Where is that boundary? Second, I am pretty sure
an adversary could design a DOS vector in this way by forcing people to go
through exponentially many min-cost flow rounds (which are not cheap
anyway) excluding only few edges per round.

Indeed, if you read the paper closely you will have seen that this kind of
problem (optimizing for some cost while staying under a budget for a second
cost) is (weakly) np-hard even for the single path case. So there is some
intuition that this is not as simple as you might imagine it. I personally
think that the Lagrangian style of combining the costs in a linear fashion
is very promising, but you might be successful with more direct methods as
well.

Is my suggestion not reasonable in practice?
> Is the algorithm runtime too high?
>

See above. I don't know, but I believe it would be hard to make safe
against adversaries. Including the fees in the cost function appears to be
the more holistic approach to me, since min-cost flow algorithms always
give you a globally optimized answer.

While we certainly need to defer to economic requirements, we *also* need
> to defer to engineering requirements (else Lightning cannot be implemented
> in practice, so any economic benefits it might provide are not achievable
> anyway).
>

Yes. I wholeheartedly agree. However, I prefer watering down a
mathematically correct solution as needed to building increasingly complex
ad-hoc heuristics.

As I understand the argument of Matt, we may encounter an engineering
> reason to charge some base fee (or something very much like it), so
> encouraging #zerobasefee *now* might not be the wisest course of action, as
> a future engineering problem may need to be solved with non-zero basefee
> (or something very much like it).
>

If we encountered such a reason, we could still encourage something else
IMHO. I do agree that we should not shorten our options by making a
protocol change at this time.

Best regards,

Stefan

P. S. : I have been using Clboss for some time now and I am very impressed.
Thank you for your amazing work! I would love a zerobasefee flag, though ;)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210817/90a07096/attachment.html>;

📅 Original date posted:2021-08-16 📝 Original message: Hi ...

2023-06-09T13:03:28Z

In reply to nevent1q…yk5h
_________________________

📅 Original date posted:2021-08-16
📝 Original message:
Hi Zmn et al.,

>I propose that the algorithm be modified >as such, that is, it *ignore*
the fee scheme.

We actually started out thinking like this in the event we couldn't find a
proper way to handle fees, and the real world experiments we've done so far
have only involved probability costs, no fees at all.

However, I think it is non-trivial to deal with the many cases in which too
high fees could occur, and in the end the most systematic way of dealing
with them is actually including them in the cost function.

That said, I agree with Matt that more research needs to be done about the
effect of base fees on these computations. We do know they make the
problem hard in general, but we might find a way to deal with them
reasonably in practice.

I tend to agree with AJ, that I don't believe the base fee is economically
helpful, but I also think that the market will decide that rather than the
devs (though I would argue for default Zerobasefee in the implementations).

In my view, nobody is really earning any money with the base fee, so the
discussion is kind of artificial. On the other hand, I would estimate our
approach should lead to liquidity being priced correctly in the
proportional fee instead of the price being undercut by hobbyists as is the
case now. So in the long run I expect our routing method to make running a
well-stocked LN router much more profitable.

Cheers,
Stefan

ZmnSCPxj via Lightning-dev <lightning-dev at lists.linuxfoundation.org>
schrieb am Mo., 16. Aug. 2021, 05:15:

> Good morning matt and aj,
>
> Let me cut in here.
>
> From my reading of the actual paper --- which could be a massive
> misunderstanding, as I can barely understand half the notation, I am more a
> dabbler in software engineering than a mathist --- it seems to me that it
> would be possible to replace the cost function in the planning algorithm
> with *only* the negative-log-probability, which I think is the key point of
> the paper.
>
> That is, the algorithm can be run in a mode where it *ignores* whatever
> fee scheme forwarding nodes desire.
> (@rene: correct me if I am wrong?)
>
>
> However, the algorithm then gets an extra step after getting a payment
> plan (i.e. how to route multiple sub-payments).
> It looks over the payment plan and if the fees involved are beyond some
> user-defined limit (with, say, a default of 0.5% of the total amount, as
> per the C-Lightning `pay` default), to look at the highest-fee channels in
> the payment plan.
> Then, it can rerun the flow algorithm, telling it to *disallow* the
> highest-fee channels identified if the total fees exceed the fee budget.
>
> It seems to me that this modification of the algorithm may be sufficient
> to be resilient against any and all future fee scheme we may decide for
> Lightning.
>
> This still achieves "optimality" in the sense of the paper, in a way
> similar to what is suggested in the paper.
> The paper suggests to basically ignore gossiped channels with non-zero
> basefee.
> The approach I suggest allows us to *start* without ignoring non-zero
> basefee, but to slowly degrade our view of the network by disallowing
> high-fee (whether high basefee or high propfee) channels.
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210816/f60fa4a3/attachment.html>;

📅 Original date posted:2021-08-15 📝 Original message: Good ...

2023-06-09T13:03:26Z

In reply to nevent1q…jvqc
_________________________

📅 Original date posted:2021-08-15
📝 Original message:
Good morning Zmn!

That is indeed precisely what we do. We usually quantize the min-cost flow
into minimum shares of, say, 10kSat to 100kSat. This makes the algorithm
run faster and loses very little precision. It also gives a simple way of
dealing with (reasonable) min-htlc-size values.

Cheers,
Stefan

ZmnSCPxj via Lightning-dev <lightning-dev at lists.linuxfoundation.org>
schrieb am So., 15. Aug. 2021, 16:03:

> Good morning lisa, aj, et al.,
>
>
> > The result is that micropayments have a different payment regime than
> “non-micropayments”, (which may still incentive almost irrational behavior)
> but at least there’s no *loss* felt by node operators for
> handling/supporting low value payments. 10k micropayments is worth 10sats.
> >
> > It’s also simple to implement and seems rather obvious in retrospect.
>
>
> It seems simple to implement for *forwarders*, but I think complicates the
> algorithm described by Pickhardt and Richter?
>
> On the other hand, the algorithm is targeted towards "large" payments, so
> perhaps the Pickhardt-Richter payment algo can be forced to have some
> minimum split size, and payments below this minimum size are just sent as
> single payments (on the assumption that such micropayments are so small
> that the probability of failure is negligible).
> That is, just have the `pay` command branch based on the payment size, if
> it is below the minimum size, just use the old try-and-try-until-you-die
> algo, otherwise use a variant on the Pickhardt-Richter algo that respects
> this minimum payment size.
> This somewhat implies a minimum on the possible feerate, which we could
> say is 1 ppm, maybe.
>
> So for example, the minimum size could be 1,000,000msat, or 1,000sat.
> If the payment is much larger than that, use the Pickhardt-Richter
> algorithm with zerobasefee.
> If payment is lower than that threshold, just do not split and do
> try-and-try-until-you-die.
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210815/df395dd3/attachment-0001.html>;

📅 Original date posted:2021-08-21 📝 Original message: Hi ...

2023-06-09T12:41:04Z

In reply to nevent1q…j4rw
_________________________

📅 Original date posted:2021-08-21
📝 Original message:
Hi Zmn! That is some amazing lateral thinking you have been applying there.
I'm quite certain I haven't understood everything fully, but it has been
highly entertaining to read. Will have to give it a closer read when I get
some time.

As a first impression, here are some preliminary observations: While I
highly like the Haskell-style datatype, and the algorithm we use does
mostly use Dijkstra pathfinding, I think what is really important in your
definition is the computeCost definition. This is what we would call the
cost function IIUC, and in order to be able to solve min-cost flow problems
it generally has to be separable and convex. I believe your datatype merely
hides the fact that it is neither.

Intuitively, I think that any cost function that implies a fixed cost (that
is, independent of the amount, though it might be different for every edge)
per edge is concave and in theory problematic for min-cost flow algorithms
because you could reduce some kind of NP-hard selection problem to it. I
believe that applies to most if not all of your ideas in the text. Again, I
think we should think more about how much of a problem that is in practice,
because we do have tools like approximation and parameterized algorithms,
as well as heuristics, and I also believe that, say, a moderate base fee
will not change the optimal flow much, because this will always prefer
large Htlcs anyway in order to optimize probability.

I am really grateful that you have been taking the time to read and
understand our paper and have been thinking further in this fascinating
way. I am certain good things will come of it in time.

Cheers,
Stefan

ZmnSCPxj <ZmnSCPxj at protonmail.com> schrieb am Sa., 21. Aug. 2021, 03:49:

>
> > Alternative Pathfinding?
> > ========================
>
>
> Or to put this section more succinctly: Why should cost be a number?
>
> What operations do the minimum cost flow algorithms demand of this thing
> called "cost", and can we provide those operations using something which is
> not a number but is instead a different structure?
> What is the minimal interface that the mincostflow algo demands of this
> "cost" datatype?
>
> Regards,
> ZmnSCPxj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210821/02ef209c/attachment.html>;

📅 Original date posted:2021-08-17 📝 Original message: Good ...

2023-06-09T12:41:00Z

In reply to nevent1q…0mx4
_________________________

📅 Original date posted:2021-08-17
📝 Original message:
Good morning Zmn!

ZmnSCPxj <ZmnSCPxj at protonmail.com> schrieb am Mo., 16. Aug. 2021, 10:27:

>
> A reason why I suggest this is that the cost function in actual
> implementation is *already* IMO overloaded.
>
> In particular, actual implementations will have some kind of conversion
> between cltv-delta and fees-at-node.
>

That's an interesting aspect. Would this lead to a constant per edge if
incorporated in the cost function? If so, this would lead to another
generally hard problem, which, again, needs to be explored more in the
concrete cases we have here to see if we can still solve/approximate it.

However, I think that in practice, most users cannot intuitively understand
> `riskfactor`.
>

I don't think they have to. Only people like you who write actual software
probably need to.

> Similarly, I think it is easier for users to think in terms of "fee
> budget" instead.
>
> Of course, algorithms should try to keep costs as low as possible, if
> there are two alternate payment plans that are both below the fee budget,
> the one with lower actual fee is still preferred.
> But perhaps we should focus more on payment success *within some fee and
> timelock budget*.
>
> Indeed, as you point out, your real-world experiments you have done have
> involved only probability as cost.
> However, by the paper you claim to have sent 40,000,000,000msat for a cost
> of 814,000msat, or 0.002035% fee percentage, far below the 0.5% default
> `maxfeepercent` we have, which I think is fairly reasonable argument for
> "let us ignore fees and timelocks unless it hits the budget".
> (on the other hand, those numbers come from a section labelled
> "Simulation", so that may not reflect the real world experiments you had
> --- what numbers did you get for those?)
>

René is going to publish those results very soon.

Regarding payment success *within some fee and timelock budget*: the
situation is a little more complex than it appears. As you have pointed
out, at the moment, most of the routes are very cheap (too cheap, IMHO), so
you have to be very unlucky to hit an expensive flow. So in the current
environment, your approach seems to work pretty well, which is also why we
first thought about it.

Unfortunately, as you know, we have to think adversarially in this domain.
And it is clear that if we simply disregarded fees in routing, people would
try to take advantage of this. If we just set a fee budget, and try again
if it is missed, then I see some problems arise: First, what edges do you
exclude in the next try? Where is that boundary? Second, I am pretty sure
an adversary could design a DOS vector in this way by forcing people to go
through exponentially many min-cost flow rounds (which are not cheap
anyway) excluding only few edges per round.

Indeed, if you read the paper closely you will have seen that this kind of
problem (optimizing for some cost while staying under a budget for a second
cost) is (weakly) np-hard even for the single path case. So there is some
intuition that this is not as simple as you might imagine it. I personally
think that the Lagrangian style of combining the costs in a linear fashion
is very promising, but you might be successful with more direct methods as
well.

Is my suggestion not reasonable in practice?
> Is the algorithm runtime too high?
>

See above. I don't know, but I believe it would be hard to make safe
against adversaries. Including the fees in the cost function appears to be
the more holistic approach to me, since min-cost flow algorithms always
give you a globally optimized answer.

While we certainly need to defer to economic requirements, we *also* need
> to defer to engineering requirements (else Lightning cannot be implemented
> in practice, so any economic benefits it might provide are not achievable
> anyway).
>

Yes. I wholeheartedly agree. However, I prefer watering down a
mathematically correct solution as needed to building increasingly complex
ad-hoc heuristics.

As I understand the argument of Matt, we may encounter an engineering
> reason to charge some base fee (or something very much like it), so
> encouraging #zerobasefee *now* might not be the wisest course of action, as
> a future engineering problem may need to be solved with non-zero basefee
> (or something very much like it).
>

If we encountered such a reason, we could still encourage something else
IMHO. I do agree that we should not shorten our options by making a
protocol change at this time.

Best regards,

Stefan

P. S. : I have been using Clboss for some time now and I am very impressed.
Thank you for your amazing work! I would love a zerobasefee flag, though ;)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210817/90a07096/attachment.html>;

📅 Original date posted:2021-08-16 📝 Original message: Hi ...

2023-06-09T12:40:59Z

In reply to nevent1q…2fll
_________________________

📅 Original date posted:2021-08-16
📝 Original message:
Hi Zmn et al.,

>I propose that the algorithm be modified >as such, that is, it *ignore*
the fee scheme.

We actually started out thinking like this in the event we couldn't find a
proper way to handle fees, and the real world experiments we've done so far
have only involved probability costs, no fees at all.

However, I think it is non-trivial to deal with the many cases in which too
high fees could occur, and in the end the most systematic way of dealing
with them is actually including them in the cost function.

That said, I agree with Matt that more research needs to be done about the
effect of base fees on these computations. We do know they make the
problem hard in general, but we might find a way to deal with them
reasonably in practice.

I tend to agree with AJ, that I don't believe the base fee is economically
helpful, but I also think that the market will decide that rather than the
devs (though I would argue for default Zerobasefee in the implementations).

In my view, nobody is really earning any money with the base fee, so the
discussion is kind of artificial. On the other hand, I would estimate our
approach should lead to liquidity being priced correctly in the
proportional fee instead of the price being undercut by hobbyists as is the
case now. So in the long run I expect our routing method to make running a
well-stocked LN router much more profitable.

Cheers,
Stefan

ZmnSCPxj via Lightning-dev <lightning-dev at lists.linuxfoundation.org>
schrieb am Mo., 16. Aug. 2021, 05:15:

> Good morning matt and aj,
>
> Let me cut in here.
>
> From my reading of the actual paper --- which could be a massive
> misunderstanding, as I can barely understand half the notation, I am more a
> dabbler in software engineering than a mathist --- it seems to me that it
> would be possible to replace the cost function in the planning algorithm
> with *only* the negative-log-probability, which I think is the key point of
> the paper.
>
> That is, the algorithm can be run in a mode where it *ignores* whatever
> fee scheme forwarding nodes desire.
> (@rene: correct me if I am wrong?)
>
>
> However, the algorithm then gets an extra step after getting a payment
> plan (i.e. how to route multiple sub-payments).
> It looks over the payment plan and if the fees involved are beyond some
> user-defined limit (with, say, a default of 0.5% of the total amount, as
> per the C-Lightning `pay` default), to look at the highest-fee channels in
> the payment plan.
> Then, it can rerun the flow algorithm, telling it to *disallow* the
> highest-fee channels identified if the total fees exceed the fee budget.
>
> It seems to me that this modification of the algorithm may be sufficient
> to be resilient against any and all future fee scheme we may decide for
> Lightning.
>
> This still achieves "optimality" in the sense of the paper, in a way
> similar to what is suggested in the paper.
> The paper suggests to basically ignore gossiped channels with non-zero
> basefee.
> The approach I suggest allows us to *start* without ignoring non-zero
> basefee, but to slowly degrade our view of the network by disallowing
> high-fee (whether high basefee or high propfee) channels.
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210816/f60fa4a3/attachment.html>;

📅 Original date posted:2021-08-15 📝 Original message: Good ...

2023-06-09T12:40:56Z

In reply to nevent1q…crdc
_________________________

📅 Original date posted:2021-08-15
📝 Original message:
Good morning Zmn!

That is indeed precisely what we do. We usually quantize the min-cost flow
into minimum shares of, say, 10kSat to 100kSat. This makes the algorithm
run faster and loses very little precision. It also gives a simple way of
dealing with (reasonable) min-htlc-size values.

Cheers,
Stefan

ZmnSCPxj via Lightning-dev <lightning-dev at lists.linuxfoundation.org>
schrieb am So., 15. Aug. 2021, 16:03:

> Good morning lisa, aj, et al.,
>
>
> > The result is that micropayments have a different payment regime than
> “non-micropayments”, (which may still incentive almost irrational behavior)
> but at least there’s no *loss* felt by node operators for
> handling/supporting low value payments. 10k micropayments is worth 10sats.
> >
> > It’s also simple to implement and seems rather obvious in retrospect.
>
>
> It seems simple to implement for *forwarders*, but I think complicates the
> algorithm described by Pickhardt and Richter?
>
> On the other hand, the algorithm is targeted towards "large" payments, so
> perhaps the Pickhardt-Richter payment algo can be forced to have some
> minimum split size, and payments below this minimum size are just sent as
> single payments (on the assumption that such micropayments are so small
> that the probability of failure is negligible).
> That is, just have the `pay` command branch based on the payment size, if
> it is below the minimum size, just use the old try-and-try-until-you-die
> algo, otherwise use a variant on the Pickhardt-Richter algo that respects
> this minimum payment size.
> This somewhat implies a minimum on the possible feerate, which we could
> say is 1 ppm, maybe.
>
> So for example, the minimum size could be 1,000,000msat, or 1,000sat.
> If the payment is much larger than that, use the Pickhardt-Richter
> algorithm with zerobasefee.
> If payment is lower than that threshold, just do not split and do
> try-and-try-until-you-die.
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20210815/df395dd3/attachment-0001.html>;

📅 Original date posted:2022-07-11 📝 Original message:I very ...

2023-06-07T23:11:34Z

In reply to nevent1q…0gsc
_________________________

📅 Original date posted:2022-07-11
📝 Original message:I very much agree with AJ here. This is something I remember discussing on
Bitcointalk back in 2011: I find it highly intuitive that the amount of
lost coins is not a constant fraction of the supply, because people get
better at keeping their coins with increasing value, distribution and
technology/best practices. I also think that we have observed this effect
in practice since then. The bulk of coins that are supposed to be lost (via
onchain analysis) haven't been moved since at least 2010. Of course, in
most cases, we'll never know, but the assumption of constant loss rate
seems unreasonable.

Cheers
Stefan

Anthony Towns via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org>
schrieb am Mo., 11. Juli 2022, 04:32:

> On Sat, Jul 09, 2022 at 08:46:47AM -0400, Peter Todd via bitcoin-dev wrote:
> > title: "Surprisingly, Tail Emission Is Not Inflationary"
>
> > Of course, this isn't realistic as coins are constantly being lost due to
> > deaths, forgotten passphrases, boating accidents, etc. These losses are
> > independent:
>
> This isn't necessarily true: if the losses are due to a common cause,
> then they'll be heavily correlated rather than independent; for example
> losses could be caused by a bug in a popular wallet/exchange software
> that sends funds to invalid addresses, or by a war or natural disaster
> that damages key storage hardware. They're also not independent over
> time -- people improve their key storage habits over time; eg switching
> to less buggy wallets/exchanges, validating addresses before using them,
> using distributed multisig to prevent a localised disaster from being
> catastrophic.
>
> > the *rate* of coin loss at time $$t$$ is
> > proportional to the total supply *at that moment* in time.
>
> This is the key assumption that produces the claimed result.
>
> If you're losing a constant fraction, x (Peter's \lambda), of Bitcoins
> each year, then as soon as the supply increases enough that the constant
> reward, k, corresponds to the constant fraction, ie k = x*N(t), then
> you've hit an equilibrium. (Likewise if you're losing more than you're
> increasing -- you just need to wait until N(t) decreases enough that you
> reach the same equilibrium point) You don't really need any fancy maths.
>
> But that assumption doesn't need to be true; coins could primarily be
> lost in "black swan" events (due to bugs, wars or disasters) rather
> than at a predictable rate -- with actions taken thereafter such that
> the same event repeating is no longer the same level of catastrophe,
> but instead another new black swan event is required to maintain the same
> loss rate. If that's the case, then the rate at which funds are lost will
> vary chaotically, leading to "inflationary" periods in between events,
> and comparatively strong deflationary shocks when these events occur.
>
> Alternatively, losses could be at a predictable rate that's entirely
> different to the one Peter assumes.
>
> One alternative predictable rate that seems plausible to me is if funds
> are lost due to people not be careful about losing small amounts; even
> though they are careful when amounts are larger. So when 10k BTC was
> worth $40, maybe it doesn't matter if you misplace a hard drive with
> 7500 BTC on it since that's only worth $30; but by the time 7500 BTC
> is worth $150M, maybe you take a bit more care with that, but are still
> not too worried if you lose 1.5mBTC, since that's also only worth $30.
>
> To mathematise that, perhaps there are K people holding Bitcoin, and with
> probability p, each loses $100 (in constant 2009 dollars say, so that we
> can ignore inflation) of that Bitcoin a year through carelessness. For
> an equilibrium to occur in that case, you need:
>
> N(t) + k - (100/P * Kp) = N(t)
>
> where P is the price of Bitcoin (again in constant 2009 dollars) and k
> is Peter's fixed tail subsidy. Simplifying gives:
>
> P = K * 100p/k
>
> But k and p are constant by assumption in this scenario, so equilibrium
> is reached only if price (P) is exactly proportional to number of
> users (K). That requires you to have a non-inflationary currency
> (supply is constant) with constant adoption (assume K doesn't change)
> that maintains a constant price (P=K*100p/k) in real terms even if the
> economy is otherwise expanding or contracting.
>
> More importantly, just from a goals point of view, x is something we
> should be finding ways to minimise it over time, not leave constant.
> In fact, you could argue for an even stronger goal: "the real value held
> in BTC lost each year should decrease", that is, x should be decreasing
> faster than 1/(N(t)*P).
>
> Cheers,
> aj
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20220711/2289f5fd/attachment-0001.html>;

📅 Original date posted:2019-08-21 📝 Original message:Please ...

2023-06-07T18:20:21Z

In reply to nevent1q…8rtj
_________________________

📅 Original date posted:2019-08-21
📝 Original message:Please see the github issues and the twitter discussion (e.g. here:
https://twitter.com/stefanwouldgo/status/1163801056423403520) for similar
points other people including me have made. At this point I feel there are
quite a few unclear points in the presentation and it is not clear to me if
they can be salvaged.

Am Mi., 21. Aug. 2019 um 09:32 Uhr schrieb ZmnSCPxj via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org>:

> Good morning Maxim,
>
> The Deaf Bob Attack
> ===================
>
> It seems to me that Bob can promote the N3 problem to the N2 problem.
>
> Suppose Alice contacts Bob to get the data.
> However, Bob happens to have lost the data in a tragic boating accident.
>
> Now, supposedly what Alice does in this case would be to broadcast the
> HTLC settlement transaction, whose signature was provided by Bob during
> protocol setup.
>
> But this seems unworkable.
>
> * If Bob managed to sign the HTLC settlement transaction, what `SIGHASH`
> flags did Bob sign with?
> * If it was `SIGHASH_ALL` or `SIGHASH_SINGLE`, then Bob already selected
> the decryption key at setup time.
> * If it was `SIGHASH_NONE`, then Alice could put any SCRIPT, including
> `<Alice> OP_CHECKSIG`.
>
> If Bob already selected the decryption key at setup time, then Bob can
> ignore Alice.
>
> * If Alice does not publish the HTLC settlement transaction, then Bob will
> eventually enter the N2 state and get the stake+reward.
> * If Alice *does* publish the HTLC settlement transaction, without Bob
> giving the encrypted data, then Bob can just use the hashlock and reveal
> the decryption key.
> * The decryption key is useless without the encrypted data!
>
> It seems this part is not workable?
> As the decryption key is embedded in the HTLC, Alice cannot get a
> signature from Bob without the decryption key already being selected by Bob
> (and thus already claimable even without any data being returned by Bob).
>
>
> Regards,
> ZmnSCPxj
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20190821/25a4c8a0/attachment-0001.html>;

📅 Original date posted:2015-07-23 📝 Original message:This ...

2023-06-07T15:42:46Z

In reply to nevent1q…fvl9
_________________________

📅 Original date posted:2015-07-23
📝 Original message:This looks like a prime application for this PIR library:
http://percy.sourceforge.net/apidocs/index.html

Eric Voskuil via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org>
schrieb am Do., 23. Juli 2015 um 02:07 Uhr:

> This is a good point. I didn't delve into the specifics of
> implementation due to the larger issues that I raised. Libbitcoin Server
> uses CurveZMQ, an implementation of CurveCP.
>
> http://curvecp.org
> http://curvezmq.org
> https://en.bitcoin.it/wiki/Libbitcoin_Server
>
> e
>
> On 07/22/2015 04:11 PM, gb via bitcoin-dev wrote:
> > Why RSA?
> >
> >>
> >> Here is an idea, inspired by TOR, on which I would like to have some
> >> feedback: We create an anonymous routing layer between Electrum servers
> >> and clients.
> >>
> >> * Each server S publishes a RSA public key, KS
> >> * Each client receives a list of available servers and their pubkeys
> >> * For each wallet address, addr_i, a client chooses a server S_i, and a
> >> RSA keypair (K_addr_i, k_addr_i)
> >> * The client creates a list of encrypted requests. Each request contains
> >> addr_i and K_addr_i, and is encrypted with the pubkey KS_i of S_i
> >> * The client chooses a main server M, and sends the list of encrypted
> >> requests to M
> >> * M dispatches the client's requests to the corresponding servers S_i
> >> (without the client's IP address.)
> >> * Each server decrypts the requests it receives, performs the request,
> >> and encrypts the result with K_addr_i
> >> * M receives encrypted responses, and forwards them to the client.
> >> * The client decrypts the encrypted response with k_addr_i
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150723/63b8773b/attachment.html>;