Universal Language Learning Paragims - UL2

1 minute read

This paper presents a novel framework for model pre-training. To date, there seems to be no consensus as to what the optimal pre-training procedure should be, with different procedures generalizing differently for downstream tasks later on.

This obviously poses a problem as practitioners need to switch between different models depending on the type of task they want, which is inconvenient as well as increased, unnecessary workload.

The framework presented by the authors is claimed to be universal, agnostic about later downstream tasks and readily modified to whatever downstream task it may encounter in the future. Further, the authors show that they achieve SOTA marks on a wide array of benchmarks, in a diverse range of task paradigms.

The framework combines three denoising strategies:

  • R-denoising: Traditional span corruption (2-5 tokens, 15% corruption rate)
  • S-denoising: Sequential denoising similar to prefix language modeling
  • X-denoising: Extreme span corruption (≥12 tokens or ≥30% corruption rate)

Key aspects of the framework include:

  • Architecture-agnostic design that works with both decoder-only and encoder-decoder architectures
  • Mode switching using sentinel tokens to dynamically adapt to different task types
  • State-of-the-art performance across 50+ NLP tasks
  • Strong scaling properties up to 20B parameters

The authors speculate that each of these denoising methods promotes strengthening different language skills. The authors further show that models trained in this way maintain performance stronger than baseline when scaling to larger models as well as when finetuning on downstream tasks.

My Opinion

This paper makes a significant contribution to the field as it appears to introduce a one-size-fits-all approach to pre-training which is both more universal and more performant than any other algorithm currently available. However, seeing how intuitive the framework is (“simply” mixing different proportions of already known methods) suggests that there is a lot of meat left on the bone when it comes to pretraining procedures.

I am interested to see if performance could be further improved by:

  • Exploring different mixing proportions of the denoising methods
  • Investigating whether there are scaling laws to mixing proportions (different model sizes = different mixture proportions)
  • Determining whether the mixing proportions should be modified throughout the training
  • Exploring different denoising methods altogether

Updated: