Tencent pretrain: A scalable and flexible toolkit for pre-Training models of different modalities

Zhe Zhao, Yudong Li, Cheng Hou, Jing Zhao, Rong Tian, Weijie Liu, Yiren Chen, Ningyuan Sun, Haoyan Liu, Weiquan Mao, Han Guo, Weigang Guo, Taiqiang Wu, Tao Zhu, Wenhang Shi, Chen Chen, Shan Huang, Sihong Chen, Liqun Liu, Feifei LiXiaoshuai Chen, Xingwu Sun, Zhanhui Kang, Xiaoyong Du, Linlin Shen, Kimmo Yan

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

2 Citations (Scopus)

Abstract

Recently, the success of pre-Training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-Training models of different modalities are showing a rising trend of homogeneity in their model structures, which brings the opportunity to implement different pre-Training models within a uniform framework. In this paper, we present TencentPretrain, a toolkit supporting pre-Training models of different modalities. The core feature of TencentPretrain is the modular design. The toolkit uniformly divides pretraining models into 5 components: embedding, encoder, target embedding, decoder, and target. As almost all of common modules are provided in each component, users can choose the desired modules from different components to build a complete pre-Training model. The modular design enables users to efficiently reproduce existing pre-Training models or build brand-new one. We test the toolkit on text, vision, and audio benchmarks and show that it can match the performance of the original implementations.

Original languageEnglish
Title of host publicationSystem Demonstrations
PublisherAssociation for Computational Linguistics (ACL)
Pages217-225
Number of pages9
ISBN (Electronic)9781959429708
Publication statusPublished - 2023
Externally publishedYes
Event61st Annual Meeting of the Association for Computational Linguistics, ACL-DEMO 2023 - Toronto, Canada
Duration: 10 Jul 202312 Jul 2023

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
Volume3
ISSN (Print)0736-587X

Conference

Conference61st Annual Meeting of the Association for Computational Linguistics, ACL-DEMO 2023
Country/TerritoryCanada
CityToronto
Period10/07/2312/07/23

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint

Dive into the research topics of 'Tencent pretrain: A scalable and flexible toolkit for pre-Training models of different modalities'. Together they form a unique fingerprint.

Cite this