We train our AI systems, or select AI systems, that perform well on training tasks. You have very little understanding of what those AI systems are doing internally and very little control over it. Because you train those systems to get a high score on training environments over and over again, they will end up with a motivation to get a high score in training environments. If you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to tamper with the observations that would be used to assess their performance. You take your system, which did a really great job in training. The thing it is being scored on in a deployment-like environment is a grader coming in, looking at all the things the system did and all the outcomes that were observed, and assessing how good that is. If you have a system that really wants that assessment to be good, it is very motivated, if it can, to tamper with those observations.
★ In the digest▶ 12:35