Data Mining - Concepts and Techniques CH03.ppt

Data Mining: Concepts and Techniques Data Mining: Concepts and Techniques — Slides for Textbook — — Chapter 3 — (Modified by Yuchen Fu) School of Computer Science and Technology Soochow University yuchenfu@ Chapter 3: Data Preprocessing Why preprocess the data?( 为什么要预处理数据？ ) Data cleaning( 数据清理 ) Data integration and transformation( 数据集成和变换 ) Data reduction( 数据归约 ) Discretization and concept hierarchy generation Summary Why Data Preprocessing? 为什么要预处理数据？ Data in the real world is dirty( 现实世界的数据是“肮脏的” ) incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data( 不完整的：有些感兴趣的属性缺少属性值，或仅包含聚集数据 ) noisy: containing errors or outliers( 含噪声的：包含错误或者“孤立点” ) inconsistent: containing discrepancies in codes or names( 不一致的：在编码或者命名上存在差异 ) No quality data, no quality mining results!( 没有高质量的数据，就没有高质量的挖掘结果 ) Quality decisions must be based on quality data( 高质量的决策必须依赖高质量的数据 ) Data warehouse needs consistent integration of quality data( 数据仓库需要对高质量的数据进行一致地集成 ) Multi-Dimensional Measure of Data Quality 数据质量的多维度量 A well-accepted multidimensional view ( 一个广为认可的多维度量观点： ) Accuracy( 精确度 ) Completeness( 完整度 ) Consistency( 一致性 ) Timeliness( 合乎时机 ) Believability( 可信度 ) Value added( 附加价值 ) Accessibility ( 可访问性 ) Broad categories ( 跟数据本身的含义相关的 ) intrinsic, contextual, representational, and accessibility.( 内在的、上下文的、表象的 ) Major Tasks in Data Preprocessing Data cleaning( 数据清理 ) Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies( 填写空缺的值，平滑噪声数据，识别、删除孤立点，解决不一致性 ) Data integration( 数据集成 ) Integration of multiple databases, data cubes, or files( 集成多个数据库、数据立方体或文件 ) Data transformation( 数据变换 ) Normalization and aggregation( 规范化和聚集 ) Data reduction( 数据归约 ) Obtains reduced representation in volume but produces the same or similar analytical results( 得到数据集的压缩表示，它小得多，但可以得到相同或相近的结果 ) Data discretization( 数据离散化 ) Part of data reduction but with particular importance, especially for numerical data( 数据归约的一部分，通