机器人学习课程 Robot Learning Cornell University, Fall 2024 -lec16_slides.pdf

Model-based

ReinforcementLearning(Part2)

SanjibanChoudhury

1

OverallCoursePlan

Foundations(upuntillastclass)

AdvancedAlgorithmsandApplications

(tillendofcourse)

Topics:Generativeworldmodels,OfflineRL,VisualRepresentations,

RLHF,Humanmotionforecasting,…

Lecturers:Sanjiban,Tapo,KillianWeinberger,KuanFang,Tapo,Lerrel

Pinto,PulkitAgarwal

2

Today’sclass

DerivingMBRLloss

PracticalMBRL

TheDREAMERalgorithm

3

ModelLearningwithPlannerinLoop

(RossBagnell,2012)

CollectFit

Planner

ExpertDataModel

Trainon

Rollout

50%expertdata

andAggregatePolicy

Data

50%learnerdata

4

ModelLearningwithPlannerinLoop

Collectdatafromanexpertxpert={(s,a,s′)}

̂

FitamodelM.Computeapolicyπinthemodelviaplanninĝ

11

Initializeemptydatabufferearner←{}

Fori=1,…,N

Executepolicyπintherealworldandcollectdatâ

i

={(s,a,s′)}

i

Aggregatedataearner←earner∪

Trainanewmodelon50%expert+50%learnerdata

̂

Mi+1←Train(0.5*xpert+0.5*earner)