Skip to main

Vocabulary of natural language processing

Search from vocabulary

  • Ayuda
mapear
anotar
descargar

Concept information

NLP methods and tools > NLP software > sentencepiece

Término preferido

sentencepiece  

Concepto genérico

Contexto(s) definitorio(s)

  • SentencePiece is a powerful and flexible method for unsupervised tokenization and subword segmentation and provides an implementation of the BPE algorithm. (Lamar & Kaya, 2023)

Ejemplo

  • SentencePiece allows the user to specify special characters that cannot be crossed when constructing subword tokens both during training of the tokenizer and during tokenization of a sentence. (Soulos, Rao, Smith, Rosen, Celikyilmaz, McCoy, Jiang, Haley, Fernandez, Palangi, Gao & Smolensky, 2021)

En otras lenguas

URI

http://data.loterre.fr/ark:/67375/8LP-RLF76QH0-3

Descargue este concepto:

RDF/XML TURTLE JSON-LD