分布式OLAP引擎Pinot的聚合索引—LinkedIn.pdf

Pinot Real-time Analytics at LinkedIn Greg Brandt (2015-09-11) Agenda •  Pinot at LinkedIn •  Design and Architecture •  Segment Layout •  StarTree for Fast Aggregation •  ThirdEye for Business Monitoring Use Cases at LinkedIn •  Who’s Viewed My Profile –  (member) Use Cases at LinkedIn •  XLNT –  A/B tesng –  (internal) Use Cases at LinkedIn •  Ad campaigns –  (customers) Data Analytics Ecosystem at LinkedIn High-Level Design Goals •  SQL-like interface •  Columnar storage and indexing •  Real-time data load Non-goals •  JOIN, UDF (unpredictable performance) •  Not source of truth •  Mutability –  Immutability allows trivial data distribuon Pinot Architecture Segment Assignment Apache Helix •  A framework for building distributed, stateful applications •  Developed at LinkedIn for Espresso, (distributed OLTP database) •  Leveraged in Pinot for cluster management (Controller uses Helix libraries) •  Query Execution: Distributed Query Execution: Local 1.  Parser –  SELECT COUNT(*) … = Abstract Syntax Tree 2.  Logical Planner –  Operators like COUNT, WHERE, etc. 3.  Physical Planner –  Consider data layout (sort key, diconary encoding, compression, etc.) Layout: Columnar Storage Layout: Sorted Forward Index Layout: More Options •  Indexes –  Bitmap, –  Roaring Bitmap, –  Inverted •  Compression –  Diconary encoding, –  P4Delta •  Multi-value columns Layout: StarTree Index Based on “Iceberg-Cubes” (Bottom-Up) Invariant: scan = k records to answer any aggregation query StarTree Index Structure St