Contextual Gaussian Process Bandit Optimization

2011-12-12article256 citationsclosed

No readable copy stored yet — this record is metadata only.

Advanced Bandit Algorithms Research Reinforcement Learning in Robotics Machine Learning and Algorithms

Abstract

How should we design experiments to maximize performance of a complex system, taking into account uncontrollable environmental conditions? How should we select relevant documents (ads) to display, given information about the user? These tasks can be formalized as contextual bandit problems, where at each round, we receive context (about the experimental conditions, the query), and have to choose an action (parameters, documents). The key challenge is to trade off exploration by gathering data for estimating the mean payoff function over the context-action space, and to exploit by choosing an action deemed optimal based on the gathered data. We model the payoff function as a sample from a Gaussian process defined over the joint context-action space, and develop CGP-UCB, an intuitive upper-confidence style algorithm. We show that by mixing and matching kernels for contexts and actions, CGP-UCB can handle a variety of practical ap-plications. We further provide generic tools for deriving regret bounds when using such composite kernel functions. Lastly, we evaluate our algorithm on two case studies, in the context of automated vaccine design and sensor management. We show that context-sensitive optimization outperforms no or naive use of context. 1

Citations in this library

Stored only when both papers are in the corpus — 256 citations exist upstream.

References

2002Finite-time Analysis of the Multiarmed Bandit Problem

2012Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting

2010A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

2003Using confidence bounds for exploitation-exploration trade-offs

2011Contextual Bandits with Linear Payoff Functions

2008Stochastic Linear Optimization Under Bandit Feedback

2008Multi-armed bandits in metric spaces

2007The Epoch-Greedy algorithm for contextual multi-armed bandits

Cited by

2017On Kernelized Multi-armed Bandits

2014Online Clustering of Bandits

2013Finite-Time Analysis of Kernelised Contextual Bandits

2014Input Warping for Bayesian Optimization of Non-stationary Functions

2015A Survey on Contextual Multi-armed Bandits

2018Scalable Hyperparameter Transfer Learning

2018Reinforcement Mechanism Design for e-commerce

2019Bandits and Experts in Metric Spaces