ehabnegm/s2pro-egy-v6

Egyptian Arabic TTS on Fish S2 Pro: full report of the September 2026 campaign

Report date: 2026-09-09. Author: Claude, for Ehab Negm (EgyVoice). Covers 2026-09-07 to 2026-09-09 (runs v2 to v7, the phoneme pilots A/B/D, the aborted full run) and the July 2026 attempt (ehabnegm/s2pro-egy). Written so that the next attempt can start from what was learned rather than from zero.

---

1. Summary in ten lines

1. Goal: make Fish Audio S2 Pro pronounce Egyptian Arabic correctly for call-centre voice agents (first customer Paymob, Dograh agent 1755).
2. The training path of fish-speech for S2 Pro had four real bugs; all were found and fixed, and training became mechanically correct.
3. 258 hours of Egyptian data were assembled and tagged, plus 7.6 hours of synthetic call-centre speech (Gemini TTS, four voices) and 946 real Paymob call transcripts.
4. Six LoRA runs on Arabic script (v2, v3, v5, v6, v7) changed word error rate by a few points and pronunciation by ear only marginally; the best of them, v6 step 1500, was rated "very bad" on a long paragraph, as was everything else including the base.
5. The two structural reasons: undiacritized Arabic text carries no vowels for the model to learn, and Soniox word error rate cannot hear accent, so runs were steered blind.
6. The last day fixed both: a pronunciation compiler that turns Egyptian text into phonemes (dictionary + Gemini G2P, 102,000 word forms), an atomic phoneme-token input adapter inside Fish (the Hebrew S2-Pro recipe), and a phoneme-recogniser metric that hears the vowels the user hears.
7. A 10-hour pilot then showed, for the first time, a measurable pronunciation gain: known-word vowel accuracy 58 percent (base) → 60 (text only) → 65 (text + phonemes) → 69 (phonemes only), with the text+phoneme variant also the best on word error (9.1 percent).
8. A full run on 53,845 phonemised clips started at 16:30 UTC and was stopped at step 305 when the cost decision was taken; no checkpoint of it exists.
9. Everything reproducible is archived (S3 s2egy/archive_0909_1709, the volume, Hugging Face ehabnegm/s2pro-egy-v6); the raw corpora live on Hugging Face.
10. Recommendation for the continuation: keep the phoneme front-end and the metric, decide the base model by a zero-shot test, run the phoneme recipe at scale for 3,000+ steps, and never again judge a checkpoint without the vowel metric and a fixed listening sheet.

---

2. Timeline and cost

When (UTC)	What
Sep 7 evening	Research, plan, RunPod A100 80 GB set up, fish-speech patched, corpora prepared, VQ codes extracted
Sep 8 00:00–09:00	v2 (228 h) and v3 (176 h) trained and gated overnight; first listening packs
Sep 8 09:00–11:00	Findings: reference clip dominates accent; corpus references bad; diacritics at inference hurt; v3 step 500 best
Sep 8 11:16–22:47	v6 (258 h, dialect tags, 4,000 steps) trained; pronunciation research; zero-shot probes; dictionary; v7 data
Sep 8 13:00–21:00	Paymob transcripts pulled, customer vocabulary and overlay, Paymob test sets, fix tests, half-strength variant
Sep 8 21:00–23:30	Gemini synthetic call-centre set generated (4,276 clips, 4 voices) and quality-controlled to 3,572
Sep 8 23:33–Sep 9 05:35	v7 (v6 step 1500 + phoneme spans + Gemini) 1,500 steps, three gate sets
Sep 9 06:00–09:00	Full evaluations, latency benchmark, final long-paragraph check ("all very bad"), report v1, HF push
Sep 9 09:00–14:40	Phoneme direction: G2P pass, pronunciation compiler, phoneme-token adapter, recogniser metric, pilots A and B
Sep 9 14:40–16:15	Pilot D (phonemes only), targeted vowel metric, product-voice test
Sep 9 16:30–17:10	Full mixed run launched, stopped at step 305 on the cost decision; archive and sync; pod closed

Approximate spend: about 46 pod hours on an A100 SXM at 1.59 USD per hour, roughly 73 USD, plus Gemini TTS for 4,276 clips and Gemini Flash for the G2P (order of 20 USD), plus Soniox transcription for several thousand evaluation clips.

---

3. What was built and is reusable

• Corrected fish-speech training path for S2 Pro: (1) embed() now applies the codebook-embedding scaling that inference applies (the July collapse); (2) gradient checkpointing switched off the reentrant mode that zeroed LoRA gradients with frozen embeddings; (3) the training conversation template made identical to inference, with the voice modality token in the assistant header; (4) LoRA-only checkpoints load with strict_loading off; (5) the dataset builder no longer strips special tokens or newlines (found on the last day; see 6.4).
• Phoneme-token input adapter (apply_phon_patch.py): with EGY_PHON=1 the tokenizer maps the content of <|phoneme_start|>…<|phoneme_end|> to 42 atomic ids in a separate input-only embedding table initialised from the mean of each symbol's BPE pieces, trained alongside the LoRA, saved with it. Verified end to end.
• Egyptian pronunciation compiler (scripts/egy_compile.py): normalisation (tashkeel stripping with the gender mark on حضرتك kept, Egyptian number words, digit verbalisation), dictionary (970 hand entries incl. place names and the Paymob overlay), Gemini G2P cache (102,654 forms), letter-map fallback, fixed 42-symbol inventory, word boundary symbol. Blind agreement of the Gemini G2P with the hand dictionary: 78 percent exact, vowel skeletons almost always agree.
• Two pronunciation metrics on a universal phoneme recogniser (facebook/wav2vec2-xlsr-53-espeak-cv-ft): sentence phone error rate (noisy, floor around 0.55) and known-word vowel accuracy anchored on consonant skeletons (scripts/word_vowel_check.py, the useful one). The recogniser distinguishes el-far3 from el-for3 exactly as the user's ear did.
• Evaluation and delivery tooling: Soniox gates per checkpoint, listening-pack builders, per-sentence comparison, rescoring, customer-vocabulary extraction from Dograh transcripts, production text front-end (scripts/apply_dict.py), Gemini TTS generator and QC, latency benchmark, S3 presigned links for the phone.
• Data: v6 tree (masri 23,852 clips, podcast-filtered 32,519, Audar 26,421, Gulf 7,932), Gemini call-centre set (3,572 clips, 7.6 h), phonemised full tree (53,845 clips) and protos, pilot sets, Paymob test sets, 946 Paymob transcripts (index with caller numbers kept off-server).

---

4. Experiments and results

All word error rates are Soniox stt-async-v5 on the youssef reference unless stated; "tag" is the [egyptian accent] prefix.

4.1 Runs on Arabic script

Run	Data / setup	Score	Ear
July (s2pro-egy)	46.9 h, fast-AR-only LoRA, 1,200 steps	WER 7.1 → 3.6–6 on one paragraph	pronunciation unchanged; slow-AR attempt collapsed by step 200 (the scaling bug)
v2	A + B 228 h, slow LoRA r32/a16, 3,000 steps	youssef 11.2 → 10.5; podcast refs collapsed after 750	base often best; v2-250 slightly worse
v3	A + filtered B 176 h	best 500 (12.2); 750/1000 collapsed	"not worse, improves very slowly"; 500 > 250
v6	258 h + Gulf replay, tags, slow+fast r32/a32, 4,000 steps	quick set: 500 11.6, 1000 12.3, 1500 10.4, 2000 14.9, 2500 14.2, 3000 12.5, 3500 13.6, 4000 11.2 (0 collapses); full 116-sentence set: 9.7 with 3 collapses (base + tag 6.6, 0)	1500 better than base+tag on food and address, worse on one narration line; on Paymob sentences 6 faults vs base's 4; unintelligible on a 60-word paragraph
v6 half strength	same adapter at alpha 16	Paymob 7.7 (0 collapses) vs full 10.9 (1)	not separately rated
v7	v6-1500 + phoneme spans in 12 % of clips + Gemini ×3, 1,500 steps	quick: 750 10.5, 1500 12.5 (0/0); Paymob: 1500 7.7; full set: 1500 10.1 (0 collapses), 750 9.6	"very bad" on the long paragraph; spans had been stripped by the builder (6.4), so the phoneme part never trained

4.2 Zero-shot probes on the base model (no training)

Probe	Result
[egyptian accent] instruction	best by ear with youssef, "good" even with a bad reference; fixes الفرع without training
Arabizi as text	broken: digits read as numbers
Arabizi inside phoneme tags	broken: ayn dropped
IPA inside phoneme tags	fixed the vowel of الفرع with all references, but "j" read the English way, "x" as "ks", glottal and emphatic symbols spelled out
Diacritics (full tashkeel)	rated "all very bad"
Long paragraph	base clean 15 percent; v6-1500 unintelligible; v7 clean 16–18 percent

4.3 Customer test (Paymob, 29 real agent sentences, youssef)

Model	WER	Ear
base + tag	6.9 %	4 faults: feminine حضرتك, الخدمة, بتقدملك, the number 48
v6-1500	10.9 %, 1 collapse	6 faults: ممثلي, الثنائي, العملاء, بتقدملك, 48, one empty file
v6-1500 + IPA spans (j scheme)	19.4 %	worst: PazhMob, ماكنة…
v6-1500 half strength	7.7 %, 0 collapses	not rated
v7-1500	7.7 %, 1 collapse	not rated separately

4.4 Phoneme pilot (10 h narration, 600 steps each, slow LoRA r32/a16, 40 sentences)

Variant	Input	WER	Sentence vowel error	Known-word vowel accuracy	Long paragraph WER
base + tag	Arabic text	10.6 %	0.568	58.3 %	15 %
A	Arabic text	11.4 %	0.570	60.0 %	16 %
B	Arabic text + phoneme tokens	9.1 %	0.557	64.8 %	15 %
D	phoneme tokens only	16.5 %	0.519	68.9 %	22 %
control	base reading phonemes as text	93 %	1.97

User's ear: B best; remaining faults الدقي (weak glottal stop) and فرعنا (a G2P error, fixed in the dictionary). With the user's own 24 kHz synthetic product voice as reference, both base and B were rated very bad: the reference clip drives quality.

4.5 Other measurements

• bf16 merge keeps about 80 percent of an adapter's effect (KL 0.34 vs 0.43); merged v6-1500 12.5 vs 10.4 unmerged on the quick set; merged v7-1500 12.8 vs 12.5.
• Latency, Fish reference implementation on A100, merged v6-1500: 10 semantic tokens/s, real-time factor 2.1, estimated first audio 1.3 s. Fish's own engine reports RTF 0.195 and about 100 ms on an H200; production must run on SGLang-Omni or vLLM-Omni.
• Gemini TTS QC: 3,572 of 4,268 clips kept (WER ≤ 0.2 or CER ≤ 0.06 after spelling normalisation).
• Full-set evaluation with a second corpus reference made v6 run away (7,900-token generations); youssef-only sets were used from then on.

---

5. What was done right

• Finding and fixing the training-path bugs before spending GPU time; the July collapse is explained.
• Gating every checkpoint automatically, packaging listening packs, and keeping the user's ear as the final judge.
• Discovering the accent instruction and the reference-dominance effect early; they explain most of the by-ear results.
• Pulling the real customer transcripts and building the test from the agent's actual sentences.
• Turning around from "which model" to "which input representation and which metric" once the evidence was clear, and validating the metric against the user's ear before trusting it.
• Pilot-before-scale on the phoneme path, with an automatic launch rule instead of optimism.
• Archiving everything reproducible before closing the pod.

6. What went wrong

1. Two days of runs on a representation that cannot carry the target. Arabic script without vowels leaves el-far3 versus el-for3 undefined; every run before the pilot learned recording style, pauses and speaker identity instead of vowels.
2. Steering by word error rate. Soniox normalises away accent; v6-1500 "won" its gates while the user heard faults.
3. In-context cloning ties accent to the reference. Corpus references in training and evaluation masked the effect; the user's synthetic product voice made everything sound bad.
4. The dataset builder silently deleted the phoneme tags (re.sub(r"<.*?>", " ") plus newline collapsing). v7 trained on bare IPA text, and pilot B first crashed on unused parameters, until this was found on the last day. Any future run with special tokens must use the patched builder.
5. The IPA scheme collided with the BPE vocabulary ("j", "x", ʔ, ˤ) when passed as text; only atomic tokens fix this.
6. Data quality and domain: ASR transcripts, inconsistent number spelling, pause-heavy podcast clips, no call-centre speech until the Gemini set at the very end; colloquial data pulled formal words like ممثلي and العملاء off.
7. Late-step degradation: collapse with some references after 750–2,000 steps, runaway on long text in v6.
8. Engineering losses: the first v6 gate ran without the tag and with the wrong LoRA config; parallel Soniox jobs corrupted one gate score with empty transcripts; a slow full evaluation blocked the v7 gates for hours; the Mac's uploads to the phone view failed, solved with S3 links.
9. Higgs Audio v2 was explored as a replacement but has no Arabic and no training script; its zero-shot test never ran (device mismatch in the Transformers integration). It stays a side experiment at most.

---

7. Root causes, final form

• R1 Text without vowels (representation).
• R2 No pronunciation metric until the last day (measurement).
• R3 Reference-conditioned cloning couples accent to the reference clip (architecture).
• R4 Loss dominated by acoustics and prosody, not phonetics (objective).
• R5 Data quality, register and domain (data).
• R6 LoRA capacity and short schedules on a 4B model with a strong prior (capacity).
• R7 Tooling defects, above all the tag-stripping builder (engineering).

---

8. What the pilot proved, and what it did not

Proved: with atomic phoneme tokens the model can be pushed toward the specified vowels (58 → 65 → 69 percent), the phoneme path also improves intelligibility when text is kept (WER 9.1), and the recogniser metric moves in the direction the ear moves. Also proved: phonemes only (D) obey better but lose words after 600 steps, so text plus phonemes with partial text dropout is the right mix.

Not proved: production quality. Ten hours and 600 steps leave glottal stops weak and rare words wrong; the full run that would have answered this was stopped at step 305. The synthetic product-voice reference showed that reference quality can undo everything.

---

9. Recommendations for the continuation

1. Keep the phoneme front-end and the metric; they are model-independent and are the missing pieces the campaign lacked.
2. Choose the base by a zero-shot Egyptian test, scored by the vowel metric and a fixed 30-sentence sheet: Fish S2 Pro with the patched pipeline, Chatterbox Multilingual (Arabic listed, small enough for full fine-tune), CosyVoice 3 if its Arabic checks out. Higgs only if the team accepts teaching Arabic from zero.
3. Train the phoneme recipe at scale: text + phoneme tokens with a third of clips as phonemes only, 3,000+ steps, gates every 500 with vowel accuracy, WER and a pack. Budget: one A100 night.
4. Fix references: a few clean human Egyptian recordings as training and evaluation references; never a 24 kHz synthetic clip. If the product voice must be synthetic, resynthesise it at 44.1 kHz from a clean human take, or train a dedicated speaker.
5. Curate data: keep verified narration and the Gemini call-centre set, drop pause-heavy podcast clips, add formal-register speech, normalise numbers and gender marks in text.
6. Serve the adapter unmerged or accept the measured bf16 loss; measure latency on SGLang-Omni before promising numbers.
7. Keep the dictionary per customer as an overlay over a shared core; feed it from real transcripts as done for Paymob.

---

10. Artifacts and how to resume

Item	Where
Best exported model: v6 step 1500 merged (fp32 fish-native and bf16 HF layout), adapters v6-1500/4000 and v7-1500, dictionaries, scripts, this report	Hugging Face ehabnegm/s2pro-egy-v6 (private); volume /workspace/s2egy/final; S3 s3://x7tsbt3izc/s2egy/final/ (also half-strength and v7 merged copies)
Pilot adapters A, B, D (step 600) and v6/v7 adapters	volume /workspace/s2egy/results/*/checkpoints
Pipeline archive: patched Fish sources, all scripts, compiler, G2P cache, dictionaries, eval sets, customer data, logs	S3 s2egy/archive_0909_1709 and volume /workspace/s2egy/archive_0909_1709
Training protos: v6; v7/full/mixed/pilots were being copied when the pod closed (rebuildable from the corpora + G2P cache)	volume /workspace/s2egy/data, S3 s2egy/data
Gemini call-centre set, tagged 44.1 kHz	volume /workspace/s2egy/data/gemini_v7_0908
Raw corpora	Hugging Face ehabnegm/masri-100h-egyptian-tts-enhanced, ehabnegm/masri-podcast-300h-egyptian-tts
Paymob transcripts	volume/S3 archive (customers/), plus the local machine
Listening packs	S3 s2egy/packs/

To resume on a fresh pod: run the setup from setup/ on the volume, apply the archived patched sources (or apply_phon_patch.py plus the builder fix), restore egy_pron_dict.txt and the G2P cache, download the corpora, rebuild protos with the patched builder, and start from full_mix_launch.sh with the gate loop. About one hour of preparation plus one hour of VQ extraction.

---

11. Open items and risks

• Gemini TTS data was generated under the user's stated agreement with Google.
• Higgs Audio v2 licence and Arabic support unverified; Chatterbox and CosyVoice 3 Arabic quality unverified.
• The tokens shared in chat (Hugging Face, Gemini, Dograh, Soniox) should be rotated.
• The interrupted final sync: if the pod was only stopped, finish copying the container protos and the raw Gemini clips before terminating.