Build a machine that gets better at a game. To do that, a human designs how it will learn: what it should predict, how heavily to weigh different mistakes, and how to carry them into its next action. Usually, someone writes the etiquette of trial and error before the machine begins.
Could that etiquette itself be found through trial and error?
The question shifts the ground a little. We thought we were training a runner; now we seem to be training the coach. And what the coach learns is not how to offer encouragement, but how to turn experience into capability.
Outside the learning machine
Junhyuk Oh and colleagues at Google DeepMind learned rules for updating predictions and policies from the experiences of agents in diverse environments. Outside the process by which each agent improves is another process improving the rule that supports its progress. That is the basic structure of meta-learning in this study. [1]
The discovered rules are called DiscoRL. Disco57, learned from 57 Atari games, outperformed comparison methods on the ProcGen games, which had not been used for discovery. Expanding discovery to 103 environments produced Disco103, with improvements also appearing in unseen evaluation environments.
The important question is what crossed into the unfamiliar environment. A player that had memorized one game’s strategy did not simply conquer another. What transferred was the learning rule; new agents trained in their new environments. This is not a story of succeeding on first sight without practice.
It is still quite interesting enough. What proved useful elsewhere was not an answer to one problem, but a way of changing when faced with a problem.
There is also room left open in what the agent predicts. Alongside established value predictions, the system develops predictions whose meanings humans have not specified in advance, and searches for ways to update and use them. Later analysis found that the discovered predictions contained information about important events to come.
This is more than automatic adjustment of settings. Part of the novelty is allowing the search to include what might be worth anticipating in order to learn. Yet the framework defining that search is still made by researchers. The machine has not invented the whole of learning theory out of nothing.
Keeping a way to change, not just a right answer
Reusing experience usually brings to mind a store of successful examples. Keep a procedure that worked and retrieve it when a similar situation appears. That matters. But if the similar situation never comes, the store remains quiet.
From here, let us use the research as a starting point for reflection.
What if the thing worth preserving were not just a procedure, but a way to revise procedures? When a customer responds unexpectedly, what do we question? When a plan and a result diverge, which premise do we revisit? Different people can take the same experience into different next actions. There are differences not only in how much experience we have, but in how we transform it.
Of course, organizational learning cannot be equated with a neural network’s update rule. Humans have conflicting goals, positions from which they are evaluated, and failures that are hard to admit. Signals of progress do not necessarily arrive as neatly as rewards in a game.
That is why it seems more interesting to read this study as asking what we preserve as the result of learning, rather than as saying organizations should automate their learning rules too. Have we gained one more successful answer? Or become quicker to notice when a different answer is needed?
The second improvement is hard to count in pages of a manual.
Which worlds should it encounter?
One striking result is that broadening the environments used to discover the rule improved its usefulness beyond them. It was not only cleverer design that mattered, but what the system was allowed to encounter.
Still, the results do not establish that adding environments is always better. They concern particular game collections and computational conditions, not physical robots or company work. Discovery also consumes substantial computation. Automated does not necessarily mean cheap or light. Training the coach may cost more than transporting the athletes.
Even so, the designer’s job begins to look different. If less of it consists of writing update rules directly, the work of choosing rewards, including experiences, and reserving unfamiliar testing grounds does not disappear. Those choices help shape the rule that will be discovered.
Let machines do the design, and human intent vanishes. It is not that simple. Intent sometimes moves out of the answer and into the choice of worlds to explore. The move is easy to miss. Look only at the finished rule, and it can seem as though nobody designed anything at all.
Something that works, something we can explain
There is another place to pause. Discovering a useful learning rule does not mean a person can explain it in a few clear sentences. Comparing performance and explaining a mechanism are different jobs.
The researchers also analyze what the discovered predictions capture. But analysis does not make the entire system transparent at once. As automatic discovery advances, techniques for examining its discoveries become more important too. The speed of obtaining an answer and the speed of establishing reasons to trust it need not be the same.
Humans need not write every way of learning in advance. That does not mean we can also relinquish the work of checking what has been learned. Somewhere between those two propositions is a new place for design.
What experiences should the next AI have? In what kinds of worlds might a way of learning grow that has not yet occurred to us?