Nostr notes by Peter Bloem

Social impact statement: This technology allows population-level ...

2026-10-26T21:37:21Z

In reply to nevent1q…y0ju
_________________________

Social impact statement: This technology allows population-level massacre at a cost of a hundred dollars per person. Produced at scale it could cause more suffering and death per dollar than nuclear weapons.

However, there are also positive uses: it could be used to help runners improve their form.

Autonomous drones following running people through the woods, ...

2026-10-26T21:32:55Z

Autonomous drones following running people through the woods, without GPS. Developed, with military funds, at a public university.

This is not dual use anymore, this is single use. There is only one use here.

Sometimes I worry that I'm becoming a negative Nancy with all my ethics talk, and then I see colleagues proudly post this kind of thing without a single acknowledgment of the implications.

Arguably, something like the Iran war already qualifies, since ...

2026-10-25T11:15:25Z

In reply to nevent1q…clgg
_________________________

Arguably, something like the Iran war already qualifies, since Claude seems to have played a role in the decision to go to war, but we're looking for a more direct involvement here. Something that will make the general public say that "AI was responsible for this."

How long until AI unintentionally*, directly causes a single ...

2026-10-25T11:13:57Z

How long until AI unintentionally*, directly causes a single incident in which (a) at least hundreds of people die or (b) at least tens of thousands of people suffer something of comparable magnitude to losing your life's savings?

* Not intended by any human being. It may be that the AI itself has some vague notion of intention or that it just craps out in some random way.

I'll try to keep it brief. :) It's true that offline ...

2026-08-26T17:06:34Z

In reply to nevent1q…zdc9
_________________________

I'll try to keep it brief. :)

It's true that offline statistical analysis can only uncover correlation not causation, so that's a limit to what a purely autoregressive LLM can do.

There are two important counters though:
- LLMs aren't trained just autoregressively. They are also trained by RL, which does admit causal analysis to emerge. Admittedly, most of the power comes from the pre-training, but this removes the impossibility argument.
- A huge amount of behaviors don't require understanding of the causality of the training data. Including, and this is I admit is counter-intuitive, running causal analysis in-context. An LLM could learn to run programs from a suffiently large dataset, and those programs could include existing causal analysis algorithms, which it could then apply in context to data.
- Modern models also have a kind of internal monologue. This pushes its behavior quite far from "sampling from the data distribution", especially coupled with RL.

I see, that makes sense. I agree that that in that sense there ...

2026-08-26T13:03:18Z

In reply to nevent1q…y0xl
_________________________

I see, that makes sense. I agree that that in that sense there are fundamental parts missing.

I don't think they require deep conceptual breakthroughs to achieve, but that's then not relevant to the discussion (and other AI people do believe there are big breakthroughs required, so I'm not claiming to have the final answer here).

At this point I'm more weary of old AI luminaries claiming ...

2026-08-26T12:39:24Z

In reply to nevent1q…h2a9
_________________________

At this point I'm more weary of old AI luminaries claiming there is a big piece of the puzzle missing (and they have the answer) than I am of big AI labs. LeCun, Sutskever and all the others have a huge incentive to make blind claims like these to raise funds and reclaim relevance. I'm not seeing much evidence in these discussions.

> But for AI to be better at math than humans are, it would ...

2026-08-26T09:24:01Z

In reply to nevent1q…exkj
_________________________

> But for AI to be better at math than humans are, it would need to be fundamentally different than current LLMs.

What makes you say that? I agree that there are some gaps to general intelligence (persistence, agency, identity, long-term memory, a stake in the world) but most of those seem engineering problems rather than anything fundamental.

The hard problems (intuition, common sense, natural language use, tool use, reflection) all seem to have been solved.

What do you see as the fundamental parts that are still missing?

I know. It's not what I'm saying or thinking though.

2026-08-21T14:26:28Z

In reply to nevent1q…7fxl
_________________________

I know. It's not what I'm saying or thinking though.

Let me go point by point here. - Curation doesn't mean manual ...

2026-08-21T12:41:29Z

In reply to nevent1q…f9vg
_________________________

Let me go point by point here.
- Curation doesn't mean manual selection. It just means that you don't blindly ingest everything from the internet. This has been done since GPT2, when they used social media upvotes as a proxy for article quality. In GPT3, they included sources proportional to the quality of the source which gives you the best of both worlds.
- All sorts of junk sneaks into the data, but so long as the majority is high quality, it won't lead to model collapse.
- "Glue on Pizza" came from the Google AI summary which is a poor proxy for current AI capability.
- Watermarking can and does 100% work as advertised. For a simple insight to the idea, consider the bitstream used to feed the PRNG. If you have the complete model output of one session and the model, you can reconstruct that. Save it, use the same stream each session and you have your watermark with no changes to the model (some extra tricks are necessary to watermark subsets of the conversation).
- It is indeed relatively easy to circumvent, although the precise method they use seems to be somewhat robust to basic rephrasing. Still, it's going to catch out a lot of lazy people.

My understanding is that it's mostly happening because the EU ...

2026-08-21T11:59:05Z

In reply to nevent1q…094v
_________________________

My understanding is that it's mostly happening because the EU requires it.

Pre-training datasets are pretty carefully curated and they have strong quality filters, so I don't think there is much of a risk of model collapse in practice.

It's also not so much about special characters or word choice as about the choices between equally good options (by the models' own probabilities). Every time you choose between two equally likely options you are essentially "encoding" one bit (i.e. a coinflip) of information. You can use these bits to create a watermark without losing quality.

🥖 A new blog post! 🥖 (Yes another one.) The question of AI ...

2026-08-16T10:05:53Z

🥖 A new blog post! 🥖 (Yes another one.)

The question of AI coding is usually seen as a binary choice. You buy in or you don't. It's not, and it's important that we move the discussion on from that.

Vibe-coding is terrible. I creates sloppy code in drifting, unmaintained code bases, it rapidly deskills you and it wastes resources. But vibe-coding isn't the only fruit.

https://peterbloem.nl/blog/craft-coding

https://op.tngl.io/every-fucking-website/

2026-08-14T20:45:37Z

https://op.tngl.io/every-fucking-website/

Nostr event nevent1qqsf8vj5q0xfsy5kcnzkmeaqufa5jl5guy0jmrfac99au5ahahgjm4qzyqhc73f3afz0kfvjsf6jcttpk0pme8zdy96hzfqkt65ld0w4cel356jlusa

2026-08-13T11:44:05Z

In reply to nevent1q…296v
_________________________

From https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898.html

(This is the source of the famous "Can Submarines Swim" quote.)

Just found this quite chilling line in a Dijkstra lecture from ...

2026-08-13T11:37:58Z

Just found this quite chilling line in a Dijkstra lecture from 1984.

I would say on a higher level ML models are good for modeling ...

2026-07-25T11:04:43Z

In reply to nevent1q…qpd5
_________________________

I would say on a higher level ML models are good for modeling "an intuition" about interesting directions. This is helpful in automated proof search (or game playing) as a fast heuristic to guide the search.

For human/AI interaction, the question then becomes whether "an intuition" is "a human-like intuition". There, I think the models have a tendency to diverge from human instincts, which makes it difficult to collaborate. They "experience" the world very differently to us and what is simple to them can be very complex to us.

I'm somewhat hopeful that mathematics will become more about ...

2026-07-24T09:57:39Z

In reply to nevent1q…338u
_________________________

I'm somewhat hopeful that mathematics will become more about understanding than about who can prove the most theorems.

Once the frontier of what can be proven is entirely the domain of the machines, the question will be: what part of it can we still understand?

That kind of mathematics would require a whole new set of skills, and tools, but I think it's a worthwhile pursuit.