Spark内存使用机制分析.pptx

Spark内存使用机制分析Deep Dive:How Spark Uses Memory议程Memory Usage OverviewMemory ContentionTungsten Memory FormatCache-aware ComputationFuture PlansWhere Spark Uses Memorystorage: memory used to cache data that will be used later.(controlled by memory manager)execution: memory used for computation in shuffles, joins, sorts and aggregations. (controlled by memory manager)others: user data structure, internal metadata, objects created by UDF, etc.ExecutionMemorySort4, 3, 5, 1, 6,2Iterator1, 2, 3, 4, 5,6Iteratortake(3)162345What if I want the sorted values again?Sort4, 3, 5, 1, 6,2Iterator1, 2, 3, 4, 5,6Iteratortake(3)162345Sort4, 3, 5, 1, 6,2Iterator1, 2, 3, 4, 5,6Iteratortake(4)162345..ExecutionMemoryStorageMemorySortCache4, 3, 5, 1, 6,2Iterator1, 2, 3, 4, 5,6Iterator123456162345…take(3)take(4)take(5)Memory ContentionHow to arbitrate memory between execution and storage?How to arbitrate memory across tasks running in parallel?How to arbitrate memory across operators running within thesame task?Challenge #1How to arbitrate memory between execution and storage?Easy, static assignment!executionstoragetotal available memoryEasy, static assignment!executionstorageSpill to DiskEasy, static assignment!executionstorageEvict LRU blocks to diskInefficient memory usage leads to bad performanceEasy, static assignment!executionstorageExecution can only use a fraction of thememory, even when there is no storage!Easy, static assignment!executionstorageEfficient use of memory required user tuningUnified Memory ManagementexecutionstorageWhat happens if there is alreadystorage?Unified Memory ManagementexecutionstorageEvict LRU blocks to diskUnified Memory ManagementexecutionstorageDesign ConsiderationsWhy evict storage, not execution?Spilled execution data will always be read back from disk, where as cached data may not.What if the application relies on cache?Unified Memory ManagementexecutionstorageThis is bad!Design ConsiderationsWhy evict storage, not execution?Spilled execution d