Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning
A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the ty...
Sábháilte in:
| Príomhchruthaitheoirí: | , |
|---|---|
| Formáid: | Alt |
| Foilsithe / Cruthaithe: |
2019
|
| Ábhair: | |
| Rochtain ar líne: | https://arxiv.org/abs/1909.05004 http://arxiv.org/abs/1909.05004 |
| Clibeanna: |
Níl clibeanna ann, Bí ar an gcéad duine le clib a chur leis an taifead seo!
|
| _version_ | 1879172143469559808 |
|---|---|
| author | Kusari, Arpan How, Jonathan P. |
| author_browse | How, Jonathan P. Kusari, Arpan |
| author_facet | Kusari, Arpan How, Jonathan P. |
| author_sort | Kusari, Arpan |
| building | arXiv |
| collection | Acceso abierto |
| description | A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a new solution is required for any change in these settings. This paper investigates the relationship between the reward function and the optimal value function for MORL; specifically addressing the question of how to approximate the optimal value function well beyond the set of weights for which the optimization problem was actually solved, thereby avoiding the need to recompute for any particular choice. We prove that the value function transforms smoothly given a transformation of weights of the reward function (and thus a smooth interpolation in the policy space). A Gaussian process is used to obtain a smooth interpolation over the reward function weights of the optimal value function for three well-known examples: GridWorld, Objectworld and Pendulum. The results show that the interpolation can provide very robust values for sample states and action space in discrete and continuous domain problems. Significant advantages arise from utilizing this interpolation technique in the domain of autonomous vehicles: easy, instant adaptation of user preferences while driving and true randomization of obstacle vehicle behavior preferences during training. |
| first_indexed | 2026-10-15T23:36:40Z |
| format | Article |
| fullrecord | <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ https://www.openarchives.org/OAI/2.0/oai_dc.xsd"><identifier>arxiv-1909.05004</identifier><datestamp>2020-03-04</datestamp>
<dc:title>Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning</dc:title>
<dc:creator>Kusari, Arpan</dc:creator>
<dc:creator>How, Jonathan P.</dc:creator>
<dc:subject>Machine Learning</dc:subject>
<dc:subject>Robotics</dc:subject>
<dc:description>A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a new solution is required for any change in these settings. This paper investigates the relationship between the reward function and the optimal value function for MORL; specifically addressing the question of how to approximate the optimal value function well beyond the set of weights for which the optimization problem was actually solved, thereby avoiding the need to recompute for any particular choice. We prove that the value function transforms smoothly given a transformation of weights of the reward function (and thus a smooth interpolation in the policy space). A Gaussian process is used to obtain a smooth interpolation over the reward function weights of the optimal value function for three well-known examples: GridWorld, Objectworld and Pendulum. The results show that the interpolation can provide very robust values for sample states and action space in discrete and continuous domain problems. Significant advantages arise from utilizing this interpolation technique in the domain of autonomous vehicles: easy, instant adaptation of user preferences while driving and true randomization of obstacle vehicle behavior preferences during training. </dc:description>
<dc:description>Accepted at ICRA 2020</dc:description>
<dc:date>2019-09-11</dc:date>
<dc:date>2020-03-03</dc:date>
<dc:type>text</dc:type>
<dc:identifier>https://arxiv.org/abs/1909.05004</dc:identifier>
<dc:identifier>http://arxiv.org/abs/1909.05004</dc:identifier>
</oai_dc:dc> |
| id | arxiv-1909.05004 |
| institution | arXiv |
| last_indexed | 2026-10-15T23:36:40Z |
| publishDate | 2019 |
| publishDateRange | 2019 |
| publishDateSort | 2019 |
| record_format | oai_dc |
| spelling | arxiv-1909.050042020-03-04 Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning Kusari, Arpan How, Jonathan P. Machine Learning Robotics A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a new solution is required for any change in these settings. This paper investigates the relationship between the reward function and the optimal value function for MORL; specifically addressing the question of how to approximate the optimal value function well beyond the set of weights for which the optimization problem was actually solved, thereby avoiding the need to recompute for any particular choice. We prove that the value function transforms smoothly given a transformation of weights of the reward function (and thus a smooth interpolation in the policy space). A Gaussian process is used to obtain a smooth interpolation over the reward function weights of the optimal value function for three well-known examples: GridWorld, Objectworld and Pendulum. The results show that the interpolation can provide very robust values for sample states and action space in discrete and continuous domain problems. Significant advantages arise from utilizing this interpolation technique in the domain of autonomous vehicles: easy, instant adaptation of user preferences while driving and true randomization of obstacle vehicle behavior preferences during training. Accepted at ICRA 2020 2019-09-11 2020-03-03 text https://arxiv.org/abs/1909.05004 http://arxiv.org/abs/1909.05004 |
| spellingShingle | Machine Learning Robotics Kusari, Arpan How, Jonathan P. Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning |
| title | Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning |
| title_full | Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning |
| title_fullStr | Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning |
| title_full_unstemmed | Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning |
| title_short | Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning |
| title_sort | predicting optimal value functions by interpolating reward functions in scalarized multi objective reinforcement learning |
| topic | Machine Learning Robotics |
| url | https://arxiv.org/abs/1909.05004 http://arxiv.org/abs/1909.05004 |
| work_keys_str_mv | AT kusariarpan predictingoptimalvaluefunctionsbyinterpolatingrewardfunctionsinscalarizedmultiobjectivereinforcementlearning AT howjonathanp predictingoptimalvaluefunctionsbyinterpolatingrewardfunctionsinscalarizedmultiobjectivereinforcementlearning |