Puma与数据高速公路——实时数据流与分析.pptxVIP

Real-time Analytics at Facebook:Data Freeway and PumaAgenda1Analytics and Real-time2Data Freeway3Puma4Future WorksAnalytics and Real-timewhat and whyFacebook InsightsUse casesWebsites/Ads/Apps/PagesTime seriesDemographic break-downsUnique counts/heavy hittersMajor challengesScalabilityLatencyAnalytics based on Hadoop/HiveDailyHourlyseconds seconds Pipeline JobsCopier/LoaderHTTPMySQLScribeNFSHive Hadoop3000-node Hadoop clusterCopier/Loader: Map-Reduce hides machine failuresPipeline Jobs: Hive allows SQL-like syntaxGood scalability, but poor latency! 24 – 48 hours.How to Get Lower Latency?Small-batch ProcessingRun Map-reduce/Hive every hour, every15 min, every 5 min, …Stream ProcessingAggregate the data as soon as it arrivesHow to solve the reliability problem?How do we reduce per-batchoverhead?DecisionsStream Processing wins!Data FreewayScalable Data Stream FrameworkPumaReliable Stream Aggregation EngineData Freewayscalable data streamScribeBatchCopierHDFStail/fopenScribe Mid-TierScribeWritersScribe ClientsNFSLog ConsumerSimple push/RPC-based logging systemOpen-sourced in 2008. 100 log categories at that time.Routing driven by static configuration.Data FreewayContinuousCopierC2DataNodeC1HDFSPTail(in the plan)C2DataNodeC1ScribeClientsPTailCalligraphusMid-tierCalligraphusWritersHDFSLog ConsumerZookeeper9GB/sec at peak, 10 sec latency, 2500 log categoriesCalligraphusRPC ? File SystemEach log category is represented by 1 or more FS directoriesEach directory is an ordered list of filesBucketing supportApplication buckets are application-defined shards.Infrastructure buckets allows log streams from x B/s to x GB/sPerformanceLatency: Call sync every 7 secondsThroughput: Easily saturate 1Gbit NICContinuous CopierFile System ? File SystemLow latency and smooth network usageDeploymentImplemented as long-running map-only jobCan move to any simple job schedulerCoordinationUse lock files on HDFS for nowPlan to move to ZookeeperPTailcheckpointfilesdirectory directorydirectoryFile