DataminingConceptsandteniques03.ppt

* * * MK 08.11.02: This needs to be updated to reflect changes. * * * * * * * * * * * * Removed 09/09/10: V-optimal: with the least histogram variance (weighted sum of the original values that each bucket represents) * * * * * * * PCA and Wavelet are forms of data compression also. * * * * * 08.09.12 MK: Slide moved here from Ch. 3. Added examples. * * * * * * * Should we briefly introduce ontology? Ontology has been used extensively in bio-medical data warehouses. -- JP We should introduce this part using 1-2 slides -JH * * Chapter 3: Data Preprocessing Data Preprocessing: An Overview Data Quality Major Tasks in Data Preprocessing Data Cleaning Data Integration Data Reduction Data Transformation and Data Discretization Summary * Summary Data quality: accuracy, completeness, consistency, timeliness, believability, interpretability Data cleaning: e.g. missing/noisy values, outliers Data integration from multiple sources: Entity identification problem Remove redundancies Detect inconsistencies Data reduction Dimensionality reduction Numerosity reduction Data compression Data transformation and data discretization Normalization Concept hierarchy generation * References D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse environments. Comm. of ACM, 42:73-78, 1999 T. Dasu and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley, 2003 T. Dasu, T. Johnson, S. Muthukrishnan, V. Shkapenyuk.?Mining Database Structure; Or, How to Build a Data Quality Browser. SIGMOD’02 H. V. Jagadish et al., Special Issue on Data Reduction Techniques. Bulletin of the Technical Committee on Data Engineering, 20(4), Dec. 1997 D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, 1999 E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. IEEE Bulletin of the Technical Committee on Data Engineering. Vol.23, No.4 V. Raman and J. Hellerstein. Potters Wheel: An Interactive Framework for Data Cleaning and Transformation, VLDB’2001 T. Redm