DIALDistributed Interactive Analysis of Large datasets.ppt

December 2001 Event Data History David Adams BNL DIALDistributed Interactive Analysis of Large datasets David Adams BNL October 15, 2003 Contents Goals of DIAL What is DIAL? Design Status Development plans Goals of DIAL 1. Demonstrate the feasibility of interactive analysis of large datasets How much data can we analyze interactively? 2. Set requirements for GRID services In particular those specific to interactive analysis Job definition: application, task, dataset Gathering and relaying results Real time monitoring (partial results) Resource management: discovery, allocation, sharing 3. Provide ATLAS with a useful analysis tool For current and upcoming data challenges Like to add another experiment to show generality What is DIAL? DIAL provides a connection between Interactive analysis framework Fitting, presentation graphics, … E.g. ROOT, JAS, … and Data processing application Natural to the data of interest E.g. athena for ATLAS DIAL distributes processing Among sites, farms, nodes To provide user with desired response time Look to other projects to provide most infrastructure Design DIAL has the following major components Dataset describing the data of interest Application defined by experiment/site Task is user extension to the application Job uses application and task to process a dataset Result is the output of a job Scheduler creates and manages jobs Together these define a high-level JDL (job definition language) Figure shows how these components interact → Status DIAL 0.5 released in September User interfaces ROOT Uses ACLiC to make all dataset/DIAL classes accessible Command line Command dial_submit processes one job Application, task and dataset taken from XML files Schedulers LocalScheduler: single local job MasterScheduler: distributed processing at a site Fork, LSF, lsrun or Condor Run at both BNL and CERN Status (cont) Datasets CbntDataset describes a single CBNT hbook file Inherits from EventDataset EventMergeDataset holds multiple