数据质量与元数据.pptVIP

数据质量与元数据 潘 定 教授、博士生导师 2009年5月 目 录 基本概念 数据质量连续统 数据质量模型 TDQM方法学 测量与改进技术 元数据与信息质量应用 数据质量工具 总 结 1 基本概念 当获得的数据与你认为的不一样时，就出现数据质量问题 数据不符合说明：垃圾进，垃圾出，etc. 不了解说明：复杂性，缺少元数据 多数据来源和表现现象 数据质量问题的高成本和普遍性 DQ问题：600 Billion/year DQ问题：管理决策面临的重要问题 例子 如何解释数据? 字段的含义？Key? 度量 ? 数据故障 输入、多种格式、缺失 / 缺省值 元数据与领域专家经验 Field 3 is Revenue. In dollars or cents? Field 7 is Usage. Is it censored? Field 4 is a censored flag. How to handle it? 数据故障 由生成数据过程之外因素导致的数据变化 Changes in data layout / data types Integer becomes string, fields swap positions, etc. Changes in scale / format Dollars vs. euros Temporary reversion to defaults Failure of a processing step Missing and default values Application do not handle NULL values well … Gaps in time series Especially when records represent incremental changes. 传统的数据质量定义（Redman，1992） Accuracy The data was recorded correctly Completeness All relevant data was recorded Timeliness The data is kept up to date Special problems in federated data: time consistency Consistency The data agrees with itself Uniqueness Entities are recorded once 传统定义的问题 … 难以测量 Accuracy and completeness are extremely difficult, perhaps impossible to measure. 与上下文无关 No accounting for what is important. 不完全 What about interpretability, accessibility, metadata, analysis, etc. 含糊不清 The conventional definitions provide no guidance towards practical improvements of the data. 新环境中的数据问题 新的数据范型 自动数据采集、大量数据，对收集数据或设计难控制 不适当使用已知的数据表示方式 联邦数据 异构集成数据、跨企业/部门数据，近似连接 数据采集与数据分析的分离 缺少一致或标准的数据表达，缺少数据含义和解释 新型应用 数据用于决策、预测，需要融合领域知识的挖掘 信息质量的定义 (1) L. English(1999) Consistently meeting all knowledge worker and end-customer expectations in all quality characteristics of the information products and services required to accomplish the enterprise mission or personal objectives; The degree to which information consistently meets the requirements and expectations of all knowledge workers who require it to perform their processes. (2) IAIDQ(2006):The fitness for use of information; information that me