Concept information
Preferred term
special token
Definition
- A token that holds a specific function within a model, ensuring that it processes the input data correctly.
Broader concept
Example
- After concatenation we removed all the special tokens except for the first CLS the last SEP which made it structurally similar to the outputs of non-segmented transformers. (Qin, Feng & Van Durme, 2023)
- Further we add the grammar rules separated by a special token. (Macina, Daheim, Wang, Sinha, Kapur, Gurevych & Sachan, 2023)
- These profiles were generated for each hate speech instance in the training data and added in the same way as described in the previous strategy i.e. after the special token '<personalinformation>'. (Doganç & Markov, 2023)
- We remove URLs from the text and replace them with a special token i.e. [URL]. (Ruiter, Reiners, D'Sa, Kleinbauer, Fohr, Illina, Klakow, Schemer & Monnier, 2022)
- We select tokens from C and mask them in the original text T using the special token [MASK]. (Yoon, Yoon & Park, 2024)
In other languages
-
French
URI
http://data.loterre.fr/ark:/67375/8LP-GF7T4R7R-D
{{label}}
{{#each values }} {{! loop through ConceptPropertyValue objects }}
{{#if prefLabel }}
{{/if}}
{{/each}}
{{#if notation }}{{ notation }} {{/if}}{{ prefLabel }}
{{#ifDifferentLabelLang lang }} ({{ lang }}){{/ifDifferentLabelLang}}
{{#if vocabName }}
{{ vocabName }}
{{/if}}