| 摘 要: ADC(Active learning based data cleaning system)运用主动学习的方法，在高效的清洗过程中，部分 利用用户交互，提升模型清洗能力，提高数据质量。ADC包含学习模块和选择模块。在学习模块中，模块维护一个概 率分类器，计算确定度(模型对修复结果的确定程度)，利用确定度为数据修复做决策。在选择模块中，模块运行数据选 择算法，选择最不确定、最有利于数据质量提升的数据交给用户清洗，再选择高分类贡献度的干净数据补充到训练集 中，逐步提升模型的修复能力。系统演示表明，ADC系统只需要很少的用户参与，就可以极大地提高数据质量，从而 提升了数据清洗的效率。 |
| 关键词: 数据清洗；主动学习；确定度 |
|
中图分类号: TP319
文献标识码: A
|
| 基金项目: 本文工作受国家自然科学基金(编号61370205)，上海市自然科学基金(编号13ZR1400800)资助. |
|
| The Active Learning Based Data Cleaning System |
|
GUO Kaiyan,WANG Hongya,CHENG Weidong
|
( School of Computer Science and Technology, Donghua University, Shanghai 201620, China)
|
| Abstract: In order to enhance cleaning capability of the model and improve data quality,ADC (Active-learning-based Data Cleaning system) uses active learning methods to partially utilize user interactions during the process of efficient cleaning.ADC contains two modules:the learning module and the selection module.The learning module maintains a probability classifier,calculates certainty (how the model determines the repair result),and uses certainty to make decisions for data repair.The selection module runs a data selection algorithm,which selects the data that is most uncertain and most conducive to the improvement of data quality,and then sends the results to the user for cleaning.Following this the selection module selects the clean data with high classification contribution to supplement the training set,and then gradually enhances the repair ability of the model. |
| Keywords: data cleaning;active learning;certainty |
