Unit 
Teaching Format and Key
Resources 
Outline 
Relevant Literature 
Lecture Meetings 
HW/Quiz 
Home Assignments 
Quizzes 
Comments 
0 
Lecture notes, worked on board, First 3 hours by Dirk Kroese, because Yoni is away. Basic Probability and Markov Chains Notes (now it is v8) Students who did not study probability previously will each receive an hour of one on one support from Brendan Patch . From Class: SimpleDiscreteQueue.nb (Mathematica file). 
Probability
and Markov Chain Background 

Kroese: Nazarathy: 
HW1
presubmit: 
HW1 Partial
Solutions by Brendan: BrendanHW1partialSolutions.pdf Here are some solutions by students (none are perfect, but
all are very nice – there were other good solutions by other students
also): 
First
(probability) chapters of a book by Kroese and Chan. Illustrated
probability notes and exercises by Kroese Probability
Notes by Richard Weber (Cambridge) Markov
Chain Notes by Richard Weber (Cambridge) Phil Pollett’s STAT3004 (intro to stoch.
proc.) course at UQ 

1 
Lecture
slides From Class: 
Introduction
and overview to the various concepts control theory 
[AstKum14]:
All 

Assessed only through part of course summary. 


Wolfram
demo  PID InvertedPendulumControls Wolrfram demo  PID spring mass 
2 
On board, NOT following references closely. From
Class: 
MDP
Model Formulation and basic examples and computation 
[Put94]:
2.1, 2.2, 3 

HW2: 
HW2.pdf Here are some solutions by students (none are perfect, but
all are very nice – there were other good solutions by other students
also): 


3 
On
board, following [Put94] closely. 
Finite
Horizon MDP 
[Put94]:
4.1.2, 4.2, 4.3, 4.4, 4.5, 4.6.1, 4.6.4 

HW3: 6.4 
HW3.pdf Here are some solutions by students (none are perfect, but
all are very nice – there were other good solutions by other students
also): 

4 
On
board, following [Put94] closely. 
Infinite
Horizon MDP 
[Put94]:
5.1, 5.2, 5.3, 5.4.1, 5.4.3, 5.5, 




5 
On
board, following [Put94] closely. 
Discounted
Rewards 
[Put94]:
5.6, AppC, 6.1, 6.2, 6.3, 6.4, 6.9 

HW4:

HW4.pdf Here are some solutions by students (none are perfect, but
all are very nice – there were other good solutions by other students
also): HW5.pdf 


6 
On
board, based on parts of [Put94]. Also lecture notes: Advanced Regular Finite Markov Chains 
Average
Rewards 
[Put94]:
8.1, 8.2, 8.3, 8.4, 8.6 

HW6: 

<<
Solution to Quiz 4 on Average Rewards >> 

7 
NOT
following references closely. Taught by Julia Kuhn. Also
guest Lecture: Hanna
Kurniawati 
Partially
Observable MDP 
[Mon82] 
Oct 21: 
HW7: 
HW7.pdf 



Cancelled. 


Cancelled – to be merged with Unit 9 
Assessed
only through part of course summary. 



9 
Lecture slides <<
Lecture slides >> 
Summary and outlook (other
aspects of control theory). 
 
13.3, 13.4 


