Metrics
How Talio measures speaking
Every conversation in Talio is transcribed and scored the same way. This page explains what each number on your analytics means, what signal it comes from, and what it does not measure. Two rules apply everywhere: the goal is to be understood, not to sound native, and the most useful comparison is you against yourself on the same kind of scenario, because the scenario moves every measure, often more than practice does.
Word variety
What we measure
How varied your word choice is: how many different words would turn up in a small random sample of your words (a measure called HDD), so a long conversation does not score higher just for being long. Higher means a wider range of words, not more words.
From what signal
Your tokenized transcript from the speech recognizer. Each of your turns is split into words (as the tokenizer segments them, in Chinese too), and the index is computed over your side of the conversation only. A conversation too short to sample shows as a gap, never as zero.
What it is not
Word variety is not vocabulary size. A small vocabulary can score high by not repeating, and a large one can score low in a scenario that keeps calling for the same words. It is not a fluency score either. The scenario moves it a great deal, so compare conversations of the same type.
Read more
Pace
What we measure
Words you say per minute across your own speaking turns (characters per minute in Chinese, mora per minute in Japanese). Pauses within a turn count toward the time.
From what signal
Voice-activity detection on your audio. Each turn is timed from the start of your first stretch of speech to the end of your last. The time before you begin a turn, and time spent listening, are not counted.
What it is not
Pace is not a score to push as high as it will go. Listeners hear pace as fluency, so a pace that is too slow costs you even when every word is right, and speech that outruns your clarity is harder to follow. Compare yourself with your own recent weeks: rates differ across languages and speakers, so figures are not comparable between them.
Read more
Flow
What we measure
The average number of words you say between pauses (mean length of run), in words (characters in Chinese, mora in Japanese). Longer runs usually mean you are finding words and building sentences with less searching.
From what signal
Voice-activity detection on your audio. A run is a stretch of speech with no audible pause inside it; we divide the words you said by the number of runs.
What it is not
It is not a measure of how much you said overall, and it is not a grammar score. A long run can be a rehearsed phrase, and a short one can be a careful, correct answer. It rises as a scenario becomes familiar, so compare repeats of the same scenario.
Clarity
What we measure
How clearly a listener can make out what you say (intelligibility), from 0 to 100.
From what signal
Your audio, scored turn by turn. Where a word is unclear, the score for that turn drops; where every word comes through, the score is high.
What it is not
Clarity is about whether your words can be made out. It is not a comparison to any reference speaker, but like any listener it is somewhat harder on unfamiliar accents, so it is not accent-blind: accent is not what it measures, unclear words are. It is also not the same thing as pronunciation, which scores individual sounds.
Read more
Pronunciation
What we measure
How closely the sounds you produced match the expected sounds for each word, from 0 to 100, scored by the speech assessment engine. It scores individual sounds, and tones in Chinese.
From what signal
The pronunciation assessment engine's per-sound accuracy scores, adjusted for how clearly each word came through. Each word in your turn is aligned to its expected sounds, each sound gets an accuracy score, and the word and turn scores are averaged from those. Strengths and focus areas are ranked by how often a sound occurs and how far it sits below your own average; for Chinese, each sound category also carries a weight for how much it tends to matter for being understood.
What it is not
Pronunciation scores individual sounds and tones, not stress, rhythm, or accent. The goal is to be understood, not to sound native. A low score on a sound does not automatically make it the sound to work on first, because some sound contrasts matter far more to a listener than others.
Read more
Vocabulary
What we measure
Which words have come up in your conversations, and whether you have said them yourself. “Heard” means the tutor used the word in a conversation with you; “said” means you produced it. Words are grouped by their dictionary form, so run, ran, and running count once.
From what signal
The tutor's and your tokenized transcripts. Each word in each turn is attributed to the speaker who produced it, and the counts accumulate across your conversations.
What it is not
Neither heard nor said is, by itself, knowledge of the word. Hearing a word once does not mean you understood it, and saying it once does not mean you can use it again on demand; knowing a word builds up over repeated meetings and uses. The counts are a record of exposure and production in your conversations, not a vocabulary test.
Read more
Can-do statements
What we measure
Whether a conversation shows evidence of a specific speaking ability, such as ordering a meal or describing something that happened to you. The statements are Talio's own, written in the style of the CEFR's can-do descriptors and arranged by CEFR level.
From what signal
The tutor's and your transcripts, read together for each statement in scope for the scenario. Each conversation gives one verdict per statement: demonstrated, not demonstrated (a fair chance was there and was not taken), or no clear opportunity when the conversation never called for it. Across conversations those verdicts roll up into a tier: Demonstrated, Emerging (shown at least once), Attempted (tried, not yet shown), or No evidence yet. Demonstrated takes repeated evidence across conversations, not a single success.
What it is not
A statement marked demonstrated means the ability was shown in a conversation, not mastered. It is evidence from a conversation, not a placement, and a scenario's level caps what can be demonstrated in it: a scenario written at A2 cannot show a B2 ability, however well you spoke. Showing the same ability across several scenarios is stronger evidence than showing it once.
Read more
References
- Council of Europe (2020). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume. Strasbourg: Council of Europe Publishing.
- de Jong, N. H., Steinel, M. P., Florijn, A., Schoonen, R., & Hulstijn, J. H. (2013). Linguistic skills and speaking fluency in a second language. Applied Psycholinguistics, 34(5), 893–916.
- Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
- Koizumi, R., & In'nami, Y. (2012). Effects of text length on lexical diversity measures: Using short texts with less than 200 tokens. System, 40(4), 554–564.
- Levis, J. M. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39(3), 369–377.
- McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392.
- Munro, M. J., & Derwing, T. M. (2001). Modeling perceptions of the accentedness and comprehensibility of L2 speech: The role of speaking rate. Studies in Second Language Acquisition, 23(4), 451–468.
- Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34(4), 520–531.
- Webb, S. (2007). The effects of repetition on vocabulary knowledge. Applied Linguistics, 28(1), 46–65.