Concept information
Preferred term
token
Definition
- In lexical analysis, atomic string with an assigned and thus identified meaning; can be either word, bi-gram or sentence. (Wikidata)
Broader concept
Narrower concepts
Example
- Given a sentence these methods predict whether each token is an event trigger and what type of event it triggered. (Liu, Li, Zhang, Yang & Zhou, 2019)
- Our model assigned each token with nearly equivalent attention score. (Liu, Li, Zhang, Yang & Zhou, 2019)
In other languages
-
French
URI
http://data.loterre.fr/ark:/67375/8LP-TX1JH6ZL-6
{{label}}
{{#each values }} {{! loop through ConceptPropertyValue objects }}
{{#if prefLabel }}
{{/if}}
{{/each}}
{{#if notation }}{{ notation }} {{/if}}{{ prefLabel }}
{{#ifDifferentLabelLang lang }} ({{ lang }}){{/ifDifferentLabelLang}}
{{#if vocabName }}
{{ vocabName }}
{{/if}}