“Read, but not fully understood” column.

“Read, but not fully understood” column. Alignment Edition (c)

1/ I made it to LessWrong & the Alignment forum and drowned in it — not for nothing I’d been putting it off for quite a while. So, in order, what clicked, so to speak.
The whole post (https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like)
is once again about the “end of the world”, but the idea that ML is what we can measure and learn to measure, and accordingly set goals, is pretty clear.

2/ As a continuation I got carried away with generally trying to figure things out, and I liked
the explanation here of what the alignment problem even is ((https://www.lesswrong.com/posts/dHNKtQ3vTBxTfTPxu/what-is-the-alignment-problem)
I’m explicitly not translating it)

3/ When people talk about search in AI it usually all goes
here (https://www.lesswrong.com/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see)

4/ In my tabs there also snuck in
a Buffett article (https://www.berkshirehathaway.com/letters/growing.pdf)
from 2003 about trade certificates as a solution to the US trade imbalance (doesn’t that remind you of anything?)

5/ Of course right after that I started reading Scott Alexander (the guy knows how to hype) and I recommend
this one (https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/)
about how trusting studies blindly is a mistake

6/ Canonically about who we don’t like and
why it’s like that (https://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/)
— or observations about so-called outgroups

7/
I read this (https://www.lesswrong.com/posts/A2Qam9Bd9xpbb2wLQ/game-theory-as-a-dark-art)
back in uni because I was into game theory, and arguably some of the best examples are collected in that post, just for learning. For the especially interested,
this paper (https://core.ac.uk/download/pdf/6388215.pdf)
is a must-read.

I can fit a lot of info in my head, but yeah, while I’ve been flying around for years, I think I overdid it with reading this stuff
🙂
Recommend something truly Must Have on Alignment — thanks! Just not about the end of the world :) Because almost every post there one way or another touches “scaling laws and how when we have clusters of a gazillion flops we’ll get models a hundred times cooler than the current ones and then AGI/ASI and all discoveries will arrive at once or we’ll get killed”

P.S. core concepts:
Corrigibility (Корригируемость)
Corrigibility is a property of an AI where it’s willing to accept corrections or a stop signal from a human, even if that contradicts its original goals.
Example:
If the AI wants to build a bridge, and a human says: “Stop, it’s dangerous,” — a corrigible AI will comply and stop, even if it itself thinks building the bridge is better.
Why is it needed?
So the AI stays under control, even if its goals aren’t formulated perfectly.

Mesa Optimizers
Mesa-optimizers are subsystems inside an AI that themselves begin optimizing their own objectives, possibly different from the original objectives set by external training.

Inner Alignment Problem (Проблема внутреннего согласования)
This is the task of making sure the mesa-optimizer’s goals (internal) match the base optimizer’s goals (set from the outside).
In plain words:
When training an AI, we may implicitly create a “mind” inside it, and we need to guarantee that this “mind” will act in accordance with our original intentions.

And spoiler: one of the founders of the alignment forum and a major LessWrong contributor is a mathematician and scientist from Moscow — Vladimir Nesov.