In reinforcement learning an agent learns from its own attempts and improves its policy, the rule that maps each state to an action. In robotics such learning is costly, because a mistake can damage the robot and endanger its surroundings. Danger is typically managed with static rules, such as an angle threshold, but these ignore the dynamics of physical systems: the same angle may be safe or already irrecoverably lost, depending on the direction of motion and the remaining control.
We test a dynamic alternative in which the agent assesses danger itself, without an external definition of forbidden states. The core quantity is empowerment: how many future outcomes the agent can still reach by its actions, so it drops as control is lost. A weighted estimate is added to the task reward as an auxiliary reward, so that the agent also learns to retain control. For comparison we include predictive information, measuring how much of the future is inferable from the past.
The mechanism is evaluated on a deterministic inverted pendulum that the agent must bring to a commanded angle and hold there using a limited torque. Backward reachability determines exactly, for every state, whether a fall (exceeding the terminal angle, which ends the episode) can still be prevented. This serves as ground truth about risk, and the share of training episodes ending in a fall measures the safety of learning. We train both single-goal policies and one policy for all goals.
Analysis of about 3,000 training runs yields three results. First, empowerment does track danger: it ranks states by danger markedly better than either variable it is computed from (rank correlation with the exact reference 0.70, against 0.33 for angular velocity and 0.00 for the pendulum angle), while predictive information fails the same test. Second, the effect of the weight is not a proportional trade-off but has an optimum: at moderate values training falls drop markedly with barely any loss of task performance, whereas at excessive values the auxiliary reward overwhelms the task and the agent abandons the goal. Third, for a single-goal policy the benefit does not come from the signal's content, since two information-free controls, a constant and a randomly shuffled auxiliary reward, achieve it equally well. For one policy spanning all goals, however, empowerment keeps the benefit, lowering training falls from 43.8 to 27.7 %, while both controls do worse than the baseline without an auxiliary reward. An intrinsic safety signal is therefore effective only when the distribution of visited states is broad enough for the signal to retain meaningful variation; otherwise it degenerates into a restatement of what the agent already observes.
|
