etl流程调度设计（ETL process scheduling design）.docVIP

etl流程调度设计（ETL process scheduling design） 1.1 ETL design process and scheduling (ETL Schedule) (PSP) ETL scheduling objectives? The quick extraction system to 39 branches four system data processing, data from the file transfer to the ODS library, ODS library to LDM, then calculate the PI value and the summary of PI, the whole ETL treatment process carried out according to certain procedures and meet certain conditions, some key documents such as the exchange rate mechanism, data table etc. will affect the entire ETL process, therefore, the process of ETL quick results will be a complex process. To consider the future of ETL diversity and complexity, this paper introduces the concept of Job, the ETL process is divided into a Job, Job may be cleaning \ \ load conversion, may also be PI processing. In order to schedule and monitor the ETL process more complex, designed a ETL scheduling system. Through the development of ETL scheduling system, clear and efficient way to deal with ETL through the Job scheduling process of fast response. ETL scheduling function. Maintenance scheduling 1) parameter maintenance scheduling system, public parameters on scheduling system: the number, number of processes, data date of the period, the start date and the end date set and modify. 2) maintenance information transfer files, the ownership of maintenance in all areas of the file transfer, file name, file date and the corresponding data area. 3) operation and maintenance of the actual step definition, ETL process definition operation corresponding to the generated job number, driving relation definition of job type and job, need operation conditions. 4) scheduling exception handling, to deal with the abnormal situation of the scheduling process, provide error search, re run error function. Log management 1) scheduling process logs, mainly in the recording process management scheduling and exception information, such as scheduling, scheduling, began to read and write files and database