Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
The concept of reinforcement learning, where an agent learns to make decisions by taking actions in an environment to maximize rewards. The case of a simple maze, reward functions, and the concept of policies and values. It also introduces the idea of state and action spaces, and the importance of experiences and histories in reinforcement learning.
Typology: Study notes
1 / 16
This page cannot be seen from the preview
Don't miss anything!
Useful refs: Sutton & Barto, Reinforcement Learning: An Introduction , MIT Press 1998. http://www.cs.ualberta.ca/~sutton/book/the-book.html Kaelbling, Littman, & Moore, ``Reinforcement Learning: A Survey,'' Journal of Artificial Intelligence Research ,Volume 4,
http://people.csail.mit.edu/u/l/lpk/public_html/ papers/rl-survey.ps
Mid-class survey results (momentarily)
Reading 2 due today
New assignments:
Final project proposal
Due Nov 5 (Fri), 5:00 PM
To me or in my mailbox
Paper preferred
Reading 3: Due Nov 9
Bentivegna, D. C. and Atkeson, C. G. “Learning How to Behave from Observing Others” SAB'02-Workshop on Motor Control in Humans and Robots , Edinburgh, UK, August, 2002.
Pacing 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4. 1 2 3 4 5 6 7 Content Math Intuition Slides Access. Too little Too much
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4. 1 2 3 4 Useful? (binary) Quantity Length Graded? (binary) Too little Too much
V. simple maze:
Whenever Mack goes left, he gets cheese
Whenever he goes right, he gets shocked
After reward/punishment, he’s reset back to start of maze
Q: how can Mack learn to act well in this world?
In general, we think of a reward function:
R () tells us whether Mack thinks a particular outcome is good or bad
Mack before drugs:
R (cheese)=+
R (shock)=-
Mack after drugs:
R (cheese)=-
R (shock)=+
Behavior always depends on rewards (utilities)
In general: agent can be in a state s i at any time t
Can choose an action a j to take in that state
Rwd associated with a state:
R ( s i
Or with a state/action transition:
R ( s i ,a j
Series of actions leads to series of rewards
( s 1 ,a 1 )→ s 3 : R ( s 3 ); ( s 3 , a 7 )→ s 14 : R ( s 14
s 1 s 2 s 3 s 4 s 5 s 6 s 4 s 2 s 7 s 11 s 8 s 9 s 10
s 1 s 2 s 3 s 4 s 5 s 6 s 4 s 2 s 7 s 11 s 8 s 9 s 10 V(s 1 )=R(s 1 )+R(s 2 )+R(s 6
Definition : Complete set of all states agent could be in is called the state space : S
Could be discrete or continuous
We’ll usually work with discrete
Size of state space: | S |
Definition : Complete set of actions an agent could take is called the action space: A
Again, discrete or cont.
Again, we work w/ discrete
Again, size: | A |
In supervised learning, “fundamental unit of experience”: feature vector+label
Fundamental unit of experience in RL:
At time t in some state s i , take action a j , get reward r t , end up in state s k
Called an experience tuple or SARSA tuple
Set of all experience during a single episode up to time t is a history:
