Skip to main content

Vocabulary of natural language processing

Search from vocabulary

  • Help
align
annotate
download

Concept information

NLP methods and tools > NLP software > sentencepiece

Preferred term

sentencepiece  

Broader concept

Definitional context(s)

  • SentencePiece is a powerful and flexible method for unsupervised tokenization and subword segmentation and provides an implementation of the BPE algorithm. (Lamar & Kaya, 2023)

Example

  • SentencePiece allows the user to specify special characters that cannot be crossed when constructing subword tokens both during training of the tokenizer and during tokenization of a sentence. (Soulos, Rao, Smith, Rosen, Celikyilmaz, McCoy, Jiang, Haley, Fernandez, Palangi, Gao & Smolensky, 2021)

In other languages

URI

http://data.loterre.fr/ark:/67375/8LP-RLF76QH0-3

Download this concept:

RDF/XML TURTLE JSON-LD