采用行业通用MOS（Mean Opinion Score）测评方法，评估得出：
自研TTS具备替换当前正在使用的Elevenlabs V2.5的质量条件，Leo与Kim音色与Elevenlabs V3的质量基本持平。Max下一步优化重点为消除伪影问题。
自研TTS在三个老师音色的“韵律”表现上，明显好于当前使用的V2.5版本，即：表现得更像老师。
稳定性与伪影方面，更新后的自研Kim明显好于当前V2.5的表现，自研Leo、Max与当前V2.5表现基本持平，即：自研TTS在表达强情绪类的语句时，更少出现破音、爆音或音色突变类问题，同时，更低概率出现电流音或噪音。
准确性与完整度方面自研TTS与V2.5表现也基本持平，整体处于行业内较高MOS评分水平。
Using the industry-standard MOS (Mean Opinion Score) method, we find:
In-house TTS is ready to replace the ElevenLabs V2.5 currently in use. Leo and Kim are essentially on par with ElevenLabs V3. Max’s next priority is removing artifacts.
Across all three teacher voices, in-house TTS is clearly stronger than production V2.5 on prosody — it sounds more like a teacher.
On stability and artifacts, the updated in-house Kim is clearly better than current V2.5; in-house Leo and Max are roughly even with V2.5. That is: in strong-emotion lines, in-house TTS is less likely to crack, clip, or jump in timbre, and less likely to add hiss or noise.
Accuracy and completeness are also essentially even with V2.5, at a high MOS level for the industry.
| 音色Voice | 对照Pairing | 准确性Accuracy | 完整度Completeness | 稳定性Stability | 韵律情感Prosody & emotion | 伪影Artifacts |
|---|---|---|---|---|---|---|
| Max | 自研 vs V2.5In-house vs V2.5 | −0.06 | −0.13 | −0.04 | +0.27 | −0.02 |
| Max | 自研 vs V3In-house vs V3 | −0.02 | +0.00 | −0.17 | −0.09 | −0.24 |
| Leo | 自研 vs V2.5In-house vs V2.5 | −0.05 | −0.08 | −0.03 | +0.22 | −0.03 |
| Leo | 自研 vs V3In-house vs V3 | +0.00 | −0.01 | −0.02 | −0.02 | +0.04 |
| Kim | 自研 vs V2.5In-house vs V2.5 | −0.05 | −0.07 | +0.24 | +0.14 | +0.33 |
| Kim | 自研 vs V3In-house vs V3 | +0.03 | +0.01 | −0.01 | +0.01 | −0.14 |
| 维度Dimension | 关注以下问题：Focuses on: |
|---|---|
| 准确性Accuracy | 错读、多读、乱读、重复读；严重含混导致听不出词Wrong, extra, garbled, or repeated words; severe slurring so a word cannot be heard |
| 完整度Completeness | 漏读、句中 / 句末截断、读到一半消失Missing words, mid- or end-of-sentence cutoffs, dying out halfway |
| 稳定性Stability | 破音、爆音、忽大忽小、音色突变Cracks, pops, wild loudness swings, sudden timbre jumps |
| 韵律情感Prosody & emotion | 节奏、停顿、重音、语气 / 情绪是否贴文本Rhythm, pauses, stress, and whether tone / emotion match the text |
| 伪影Artifacts | 底噪、电流音、咔哒声、叠音残响、环境怪声Hiss, electrical buzz, clicks, echo leftovers, ambient odd sounds |
| 评分Rating | 课堂可用性Classroom usability |
|---|---|
| 1 | 几乎不能听 / 语义毁了 · 不能用Almost unlistenable / meaning destroyed · unusable |
| 2 | 明显硬伤，当示范不放心 · 不建议用Clear hard defects; unsafe as a model · not recommended |
| 3 | 一般可用：小瑕疵，不影响跟读主信息 · 能用Usable: small flaws that do not hide the main line for shadowing · good enough |
| 4 | 清楚自然，仅细毛刺 · 推荐用Clear and natural, only fine nicks · recommended |
| 5 | 干净、准、稳、语气贴 · 示范级（宜少）Clean, accurate, stable, tone-fit · model-level (should be rare) |
维度描述：以「关键词 / 教学目标词是否读对、整句是否可懂」为主；1 处严重错读（改词、目标词错成别的）即可评低于2分。Dimension description: focus on whether key / target words are correct and the sentence is intelligible. One serious misread (changed word, target word as something else) can already be scored below 2.
| 分Score | 标准Rubric |
|---|---|
| 1 | 关键词大面积听不出；或乱读 / 反复重复；整句语义翻掉Key words largely unintelligible; or garbled / looping repeats; sentence meaning flipped |
| 2 | 1 处及以上明显错读 / 增读 / 重复，或将目标词（如 flamingo / come on）读到影响了学生的跟读One or more clear misreads / extras / repeats, or a target word (e.g. flamingo / come on) read in a way that hurts the student’s shadowing |
| 3 | 偶有极轻含混或非关键词轻微含糊，大意仍对；无重复、无改词Occasional very light slurring or a non-key word slightly vague; gist still correct; no repeats or word changes |
| 4 | 词准；最多 1 处极轻含混且不歧义Words accurate; at most one very light, unambiguous slur |
| 5 | 与文本一致，吐字清楚，无错读、无增读、无重复读Matches the text; clear articulation; no wrong, extra, or repeated words |
维度描述：不存在「大段漏读」的情况。同时，关注词是否被读全，句末是否有被切开的情况，例如：句末只读了半个词。Dimension description: there are no long-passage omissions. Also check whether words are fully spoken and whether the ending is cut off — for example, only half of the last word is read.
| 分Score | 标准Rubric |
|---|---|
| 1 | 缺关键信息（问句没问完、指令没说）或开头 / 结尾截到句子不成立Missing key information (question unfinished, instruction not spoken) or the start / end is cut so the sentence does not stand |
| 2 | 明显漏词或半截词 / 半截句，仍能猜一点意思但不能当干净示范Clear missing or half words / half sentence; some gist remains but it is not a clean model |
| 3 | 词都在；句末略急但最后一个词仍可辨All words present; ending a bit rushed but the last word is still identifiable |
| 4 | 完整；最多收尾略紧，词仍完整可跟读Complete; ending may be slightly tight, but words stay whole and shadowable |
| 5 | 与文本一一对应，无漏读、截断，收尾完整One-to-one with the text; no omissions or cutoffs; clean finish |
维度描述：几乎没有「中间缓冲」；测试原则：一次句尾炸裂 ≈ 整条不可用。强情绪句子（Come on! / Guess! / Wow!）是本测试的重点压。Dimension description: there is almost no “middle buffer.” Test principle: one end-of-line blowout ≈ the whole clip is unusable. Strong-emotion lines (Come on! / Guess! / Wow!) are the main stress of this test.
| 分Score | 标准Rubric |
|---|---|
| 1 | 破音 / 电音贯穿，或整句发飘失控，几乎无法听完Cracking / electric timbre throughout, or the whole line drifts out of control; almost impossible to finish listening |
| 2 | 有明显破音 / 电音 / 爆音（含单次句尾炸裂），听感刺耳，课堂不能当示范Clear crack / electric timbre / pop (including a single end-of-line blowout); harsh; cannot be a classroom model |
| 3 | 偶有轻微毛刺 / 发紧，一闪而过，不持续、不刺耳Occasional light rasp / tightness that flashes past; not sustained or harsh |
| 4 | 整体稳；最多极短瞬时不稳Mostly stable; at most a very brief instant of instability |
| 5 | 全程稳定，无破音、电音、爆音、明显忽大忽小或音色乱跳Stable throughout; no cracks, electric timbre, pops, obvious loudness swings, or wild timbre jumps |
维度描述：重点测评：(1) 停顿是否切错气口 (2) 重音是否在教学词 / 情感词上 (3) 语气是否匹配标点与场景（! / ? / 安抚句等）。Dimension description: Focus of scoring: (1) whether pauses cut the wrong breath points (2) whether stress is on teaching / emotion words (3) whether tone matches punctuation and scene (! / ? / soothing lines, etc.).
| 分Score | 标准Rubric |
|---|---|
| 1 | 语气完全反了或失控；或停顿碎成「逐词报站」，无法当老师话术Tone fully reversed or out of control; or pauses smashed into word-by-word station announcements — not teacher talk |
| 2 | 明显机械 / 机器人腔；或重音乱到目标词被淹没；或情绪与文本明显不符Clearly mechanical / robotic; or stress so wrong the target word is buried; or emotion clearly mismatches the text |
| 3 | 能听懂是老师在说；偶有略平、略冲或不自然小停顿，仍可用Still sounds like a teacher; occasional flatness, slight push, or unnatural small pauses; still usable |
| 4 | 节奏与语气较自然，贴合短句场景；仅细微不自然Rhythm and tone fairly natural and scene-fit; only tiny unnatural bits |
| 5 | 短句也清楚：气口对、重音对、情绪贴合（激动收尾有力但不炸——炸归稳定性）Clear even on short lines: breath, stress, and emotion fit (excited endings are strong but not blown — blowouts belong to Stability) |
维度描述：重点评估是否存在杂音、句首 / 句尾咔哒音、是否出现一声爆噪音、是否存在持续的电流底噪。合成前后的正常短静音、极轻房间感且不抢词的情况不算为伪影。Dimension description: Focus on whether there is noise, start/end clicks, a single burst, or a constant electrical floor. Normal short silence around synthesis, or a very light room tone that does not steal words, does not count as an artifact.
| 分Score | 标准Rubric |
|---|---|
| 1 | 强底噪 / 持续怪声 / 明显咔哒叠音，内容被盖住Strong hiss / constant odd noise / clear click-echo; content is covered |
| 2 | 明显底噪或间歇怪声，烦人，影响当示范听Clear hiss or intermittent odd noise; annoying; hurts use as a model |
| 3 | 轻微底噪或偶发轻杂音，不抢词Light hiss or occasional soft noise that does not steal words |
| 4 | 基本干净；需细听才察觉极轻底噪Mostly clean; only a very light floor if you listen hard |
| 5 | 干净，无明显底噪、电流底、咔哒、残响或其他外加怪声Clean; no obvious hiss, electrical floor, clicks, leftover echo, or other added odd sounds |
142 条真实 AI 老师课堂短句下，每轮测评共产生1278次标注。评估「能否作为课上示范音」。测试集覆盖以下：Across 142 real AI-teacher classroom short lines, each evaluation round produces 1,278 ratings. We evaluate whether each clip can serve as an in-class model. The set covers:
在0909和0920的两次测评中，评委盲选不同场景下，更倾向使用自研TTS还是ElevenlabsIn the 0909 and 0920 evaluations, raters blindly chose, by scenario, whether they preferred in-house TTS or ElevenLabs.
| 场景Scenario | 0909 · V2.5 | 0920 · V3 | ||
|---|---|---|---|---|
| 倾向使用自研Prefer in-house | 倾向使用V2.5Prefer V2.5 | 倾向使用自研Prefer in-house | 倾向使用V3Prefer V3 | |
| 课堂指令 / 互动短句Classroom cues / short interaction | 57% | 43% | 58% | 42% |
| 疑问 / 感叹 / 情绪句Questions / exclamations / emotion | 46% | 54% | 38% | 62% |
| 陈述与描述句Statements & descriptions | 63% | 37% | 48% | 52% |
| 专有名词 / 难词Proper nouns / hard words | 44% | 56% | 26% | 74% |
| 长句（20+ 词）Long lines (20+ words) | 73% | 27% | 43% | 57% |
| 自然拼读 / 音素段Phonics / phoneme spans | 58% | 42% | 50% | 50% |
| 多短句连读 / 列举Multi-clause / lists | 64% | 36% | 26% | 74% |
| 数字 / 字母 / 列表Numbers / letters / lists | 82% | 18% | 65% | 35% |
| 唱歌 / 韵律文本Songs / rhythmic text | 80% | 20% | 44% | 56% |
| 场景Scenario | 0909 · V2.5 | 0920 · V3 | ||
|---|---|---|---|---|
| 倾向使用自研Prefer in-house | 倾向使用V2.5Prefer V2.5 | 倾向使用自研Prefer in-house | 倾向使用V3Prefer V3 | |
| 课堂指令 / 互动短句Classroom cues / short interaction | 47% | 53% | 13% | 87% |
| 疑问 / 感叹 / 情绪句Questions / exclamations / emotion | 36% | 64% | 35% | 65% |
| 陈述与描述句Statements & descriptions | 42% | 58% | 43% | 57% |
| 专有名词 / 难词Proper nouns / hard words | 33% | 67% | 48% | 52% |
| 长句（20+ 词）Long lines (20+ words) | 33% | 67% | 37% | 63% |
| 自然拼读 / 音素段Phonics / phoneme spans | 46% | 54% | 50% | 50% |
| 多短句连读 / 列举Multi-clause / lists | 39% | 61% | 46% | 54% |
| 数字 / 字母 / 列表Numbers / letters / lists | 35% | 65% | 23% | 77% |
| 唱歌 / 韵律文本Songs / rhythmic text | 62% | 38% | 38% | 62% |
| 场景Scenario | 0909 · V2.5 | 0920 · V3 | ||
|---|---|---|---|---|
| 倾向使用自研Prefer in-house | 倾向使用V2.5Prefer V2.5 | 倾向使用自研Prefer in-house | 倾向使用V3Prefer V3 | |
| 课堂指令 / 互动短句Classroom cues / short interaction | 14% | 86% | 58% | 42% |
| 疑问 / 感叹 / 情绪句Questions / exclamations / emotion | 25% | 75% | 36% | 64% |
| 陈述与描述句Statements & descriptions | 19% | 81% | 53% | 47% |
| 专有名词 / 难词Proper nouns / hard words | 19% | 81% | 47% | 53% |
| 长句（20+ 词）Long lines (20+ words) | 28% | 72% | 55% | 45% |
| 自然拼读 / 音素段Phonics / phoneme spans | 15% | 85% | 50% | 50% |
| 多短句连读 / 列举Multi-clause / lists | 20% | 80% | 41% | 59% |
| 数字 / 字母 / 列表Numbers / letters / lists | 8% | 92% | 53% | 47% |
| 唱歌 / 韵律文本Songs / rhythmic text | 29% | 71% | 37% | 63% |
两轮都是同 142 条冻结集、盲评 A/B、每音色 3 人全量。跨轮均分绝对值会受评委池影响。Both rounds use the same 142-item frozen set, blind A/B, and 3 full-set raters per voice. Absolute means can shift with the rater pool across rounds.
| 文件File | 音色Voices | 评委Raters |
|---|---|---|
tts_eval_v2_0909 |
Max / Leo / Kim | jiayufei / lige / wuxianxin |
tts_eval_v3_0920 |
Max / Leo / Kim | lige / pengqianya / shijiaoyang |