[1]魏文定,鄂海红,王 曦,等.云原生数据湖服务平台的设计与实现[J].计算机技术与发展,2024,34(02):17-22.[doi:10. 3969 / j. issn. 1673-629X. 2024. 02. 003]
　WEI Wen-ding,E Hai-hong,WANG Xi,et al.Design and Implementation of a Cloud Native Data Lake Platform[J].,2024,34(02):17-22.[doi:10. 3969 / j. issn. 1673-629X. 2024. 02. 003]

点击复制

云原生数据湖服务平台的设计与实现()

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:

34

期数:

2024年02期

页码:

17-22

栏目:

大数据与云计算

出版日期:

2024-02-10

文章信息/Info

Title:

Design and Implementation of a Cloud Native Data Lake Platform

文章编号:

1673-629X(2024)02-0017-06

作者:

魏文定¹ ; 鄂海红¹ ; 王 曦² ; 宋美娜¹ ; 宿兴辉³

1. 北京邮电大学 计算机学院(国家示范性软件学院),北京 100876;

2. 中国科学院信息工程研究所,北京 100085;

3. 联洋国融(北京)科技有限公司,北京 100088

Author(s):

WEI Wen-ding1 ; E Hai-hong1 ; WANG Xi2 ; SONG Mei-na1 ; SU Xing-hui3

1. School of Computer Science ( National Pilot Software Engineering School) ,Beijing University of Posts and Telecommunications,Beijing 100876,China;

2. Institute of Information Engineering,Chinese Academy of Sciences,Beijing 100085,China;

3. Lianyang Guorong ( Beijing) Technology Co. ,Ltd. , Beijing 100088,China

关键词:

云原生; 数据湖; 大数据; 生产线; 数据湖上云

Keywords:

cloud native; data lake; big data; production line; data lake on cloud

分类号:

TP311

DOI:

10. 3969 / j. issn. 1673-629X. 2024. 02. 003

摘要:

云原生数据湖已经成为数据管理和分析领域的研究热点,相关技术和应用也得到了广泛的关注和探索。 数据湖部署存在着成本高、组件之间兼容性差等问题,存算不分离制约着数据湖平台延展性,缺乏完备的数据入湖体系容易引起数据湖沼泽的形成,导致用户无法从中提取数据价值。 该文设计并实现了云原生数据湖服务平台,平台以 Kubernetes 为底层构建云原生环境,结合容器技术将数据湖组件镜像化,同时设计数据湖存算分离方案来提高数据湖平台的可扩展性和可移植性,并配合监控、组装生产线将镜像容器化,实现数据湖上云操作。 并建立用户入湖作业与云原生计算引擎之间的桥梁,对入湖信息进行预处理,提供多类型作业以满足多元化入湖场景,以统一 catalog 的方式将数据写入数据湖中。 实际运行结果表明,该平台既提高了数据湖平台的灵活性和可靠性,又确保了元数据和数据资产的规范性存储。

Abstract:

Cloud native data lakes have become a research hotspot in the field of data management and analytics,and related technologiesand applications have received widespread attention?

and exploration. Data lake deployment suffers from high cost and poor compatibilitybetween components,the lack of separation of storage and computation restricts the extensibility?

of the data lake platform,and the lack ofa complete data entry system easily causes the formation of a data lake swamp,resulting in users being unable to extract data value fromit. We design and implement a cloud native data lake service platform,which uses Kubernetes as the underlying layer to build a cloudnative environment,and combines container technology to mirror data lake components. Meanwhile,the storage and computing separationscheme of the data lake is designed to improve the scalability and portability of the data lake platform,and the image is containerized withthe monitoring and assembly production line to realize cloud operations on the data lake. The platform also establishes a bridge betweenthe user’ s entry operations and the cloud native computing engine, pre - processes the entry information, provides multiple types ofoperations to meet diverse entry scenarios,and writes data to the data lake in a unified catalog manner. The actual operation results showthat the platform not only improves the flexibility and reliability of the data lake platform, but also ensures the normative storage ofmetadata and data assets.

常用功能