A coffee-carrying robot needs to learn which rules to forget

A robot carrying coffee past your laptop should probably give your phone a wide berth, too. Having to name every device would make a household helper an exhausting responsibility. Yet simply adding more instructions creates another problem: the robot can accumulate rules that no longer help.

That tension is at the heart of FLEX, a robot-learning system described in MIT’s November 9 account. Developed by researchers from MIT and New York University, it uses ordinary language to revise what a robot considers when planning its movements. The preprint appeared on October 20; MIT says the work will be presented at the Conference on Robot Learning later in November.

The appealing idea is that a correction can carry more information than its literal wording. A warning about spilling coffee on a laptop can suggest a broader concern about liquid damaging electronics. In MIT’s simulated example, that broader rule also brings a nearby phone into consideration. The experiments used GPT-4o to help make these connections. MIT’s explanation also describes candle-carrying and bowling-ball tasks, where the relevant categories become flammable and fragile objects.

FLEX gives this process three jobs: add a missing consideration, generalize an overly narrow one, and mask something irrelevant. In the coffee illustration, those jobs mean adding laptop avoidance, extending it to electronics, and dropping an unnecessary preference for moving near a bowl. These are edits to the robot’s planning criteria, rather than a growing transcript of instructions it must obey forever. The project’s method description explains how the three operations work together.

There is a consequential difference between the public explanations. MIT’s news account describes masking as computer vision removing objects that are no longer present. The paper’s method instead asks a language model whether each feature remains relevant; the project page agrees. An object can remain visible while the rule concerning it becomes unnecessary.

Nor should the coffee story be read as every edit succeeding automatically after one sentence. In the paper’s staged illustration, a phone initially overlaps the laptop. Avoiding the laptop appears to protect both. Removing the laptop exposes the missing phone rule, prompting another correction. A reassuring movement can therefore conceal the wrong explanation for it.

Language also does only part of the teaching. In FLEX, words identify which considerations matter, while demonstrations help establish their relative importance. Imagine watching a cup take a curved route across a table: the movement alone leaves open why it curved. Was it avoiding electronics or approaching another object? A sentence can identify the concern; demonstrations help the system learn the tradeoffs governing the route. That division of labor is explicit in the project’s account.

The researchers report strong results, including the highest average task scores, sometimes tied, across seven grid-navigation scenarios. But the most human-looking evidence needs careful unpacking. Twelve participants typed corrections while watching simulated robot movements. They watched simulations for safety because some comparison systems produced erratic arm motions. Separately, the researchers combined participants’ corrections with expert demonstrations collected on a physical Franka arm, then ran learning offline. This was not a study of people repeatedly correcting a physical household robot as it worked beside them. The project page distinguishes these tests.

Its main measure, “normalized reward,” places behavior between task-defined worst and best outcomes. A score of 0.99 does not mean 99 percent of actions were safe. It measures performance against the experiment’s chosen preferences. The paper defines the calculation.

One documented limitation makes the language–demonstration distinction tangible: asking for still greater distance from a laptop does not update how heavily FLEX weights that concern. The authors acknowledge that restriction. Identifying the right concern and responding adequately to its urgency remain different problems.

Independent work offers context, rather than confirmation. Albert Yu and Raymond Mooney’s DeL-TaCo research also found benefits from combining language and demonstrations to resolve ambiguous robot tasks. It tested a different method and does not replicate FLEX.

FLEX’s useful contribution is a way to revise the robot’s working assumptions—including deleting unhelpful ones. The coffee sequence supplies an equally useful caution: a cup taking the right route once cannot tell us whether the phone will still be protected when the laptop moves.