## ·        The teaching format varies based on the material and the speed and depth in which it is to be covered.

 Unit Teaching Format and Key Resources Outline Relevant Literature Lecture Meetings HW/Quiz Meetings Home Assignments Quizzes Comments and Additional Resources 0 Lecture notes, worked on board, First 3 hours by Dirk Kroese, because Yoni is away.   Basic Probability and Markov Chains Notes (now it is v8)   Students who did not study probability previously will each receive an hour of one on one support from Brendan Patch .   From Class: SimpleDiscreteQueue.nb (Mathematica file).   Photos from 2.3 Probability and Markov Chain Background Kroese: 1.1, 1.2, 1.3,   Nazarathy: 2.1, 2.2, 2.3 HW1 pre-submit: 1.4, 2.4 HW1 post-submit: 3.4 Quiz1: 4.4 (Aug 19) HW1 Due: Aug 12   Partial Solutions by Brendan: BrendanHW1partialSolutions.pdf   Here are some solutions by students (none are perfect, but all are very nice – there were other good solutions by other students also): Quiz 1 with solution 1 Lecture slides Lecture slides From Class: PhotosFrom3.pdf Introduction and overview to the various concepts control theory [AstKum14]: All [PerMar10]: All [Put94]: 1 3.1, 3.2 Assessed only through part of course summary. 2 On board, NOT following references closely. PromotionMDPStateSpace.pdf From Class: Lects4.pdf (big ugly file). MDP Model Formulation and basic examples and computation [Put94]: 2.1, 2.2, 3 [Ber00]:1.1, 1.2 3.3, 4.1, 4.2 HW2: 5.4 HW2.pdf Due: Sep 2 Here are some solutions by students (none are perfect, but all are very nice – there were other good solutions by other students also): 3 On board, following [Put94] closely. Finite Horizon MDP [Put94]: 4.1.2, 4.2, 4.3, 4.4, 4.5, 4.6.1, 4.6.4 4.3, 5.1, 5.2, 5.3 HW3: 6.4 Quiz 2: 7.4 (Sep 9) HW3.pdf Due: Sep 9   Here are some solutions by students (none are perfect, but all are very nice – there were other good solutions by other students also): 4 On board, following [Put94] closely. Infinite Horizon MDP [Put94]: 5.1, 5.2, 5.3, 5.4.1, 5.4.3, 5.5, 6.1, 6.2, 6.3 5 On board, following [Put94] closely. Discounted Rewards [Put94]: 5.6, AppC, 6.1, 6.2, 6.3, 6.4, 6.9 7.1, 7.2, 7.3, 8.1, 8.2, 8.3,  9.1 HW4: 8.4 HW5: 9.4 (yoni) Quiz 3: 10.4  (Oct 7) HW4.pdf Due: Sep 26   Here are some solutions by students (none are perfect, but all are very nice – there were other good solutions by other students also):                       HW5.pdf Due: Oct 7 6 On board, based on parts of [Put94]. Also lecture notes:  Advanced Regular Finite Markov Chains Average Rewards [Put94]: 8.1, 8.2, 8.3, 8.4, 8.6 9.2, 9.3, 10.1, 10.2, 10.3, (10.1—10.3 were cancelled) Oct14: 11.1, 11.2, 11.3 Oct21: 12.1 HW6: Oct 14: during lecture (no tut). Quiz 4:  Oct 28: 13.2 (this is 13-13:50) Due: Oct 28 << Solution to Quiz 4 on Average Rewards >> 7 NOT following references closely. Taught by   Also guest Lecture: Hanna Kurniawati Partially Observable MDP [Mon82] [KaeLitCass98] [BaurRied11]: 5 Oct 21: 12.2, 12.3, Oct 28: 13.1 (Guest lecture) HW7: Oct 21: 12.4 HW7.pdf Due: Oct 31 8 Cancelled. Deterministic Continuous Time and State Optimal Control [Ber00]: 3 Cancelled – to be merged with Unit 9 Assessed only through part of course summary. Due: Nov 7 9 Lecture slides << Lecture slides >> Summary and outlook (other aspects of control theory). - 13.3, 13.4