yap is a local, on-device push-to-talk dictation app for macOS. It transcribes speech with a 600M-parameter STT model (Parakeet) via MLX, then types the result into whatever app has focus. Everything runs on Apple Silicon—no cloud, no account.
We wanted to add a “clean” style that would post-process raw transcripts: remove filler words (um, uh, like, you know), fix punctuation, de-stutter repeated words, and—when the speaker is clearly dictating a list, table, or email—apply appropriate markdown formatting. All on-device, using a model small enough to coexist in memory alongside the STT model.
The model we chose was Qwen2.5-1.5B-Instruct-4bit, weighing about 1 GB. The question was whether a model this small could reliably follow complex cleanup instructions. The answer: not at zero-shot. But with DSPy prompt optimization, yes.
The 0.5B model
We also tried the 0.5B variant. It couldn’t even remove filler words at zero-shot—it parroted the input verbatim. DSPy lifted it from 69% to 84%, but it still produced empty outputs and kept fillers in cleaned text. The 1.5B model has a fundamentally higher capability floor.
The Task
Given a raw speech-to-text transcript, produce a cleaned version that:
- Removes filler words (um, uh, like, you know, basically, actually, literally)
- Removes false starts, stutters, and repeated words or phrases
- Fixes punctuation and capitalization
- Formats sequential items as markdown numbered lists
- Formats enumerations as markdown bullet lists
- Formats structured comparisons as markdown tables
- Formats formal messages with proper paragraph structure
- Converts spoken numbers, times, and money to digits ($15,000, 2 PM, 5:30)
- Wraps code identifiers in backticks
We built a dataset of 59 (transcript, cleaned) pairs covering these categories and split 70/30 into 41 training and 18 validation examples.
The Metric
A single scalar metric scored each prediction on five axes:
| Component | Weight | What it measures |
|---|---|---|
| Filler removal | 0.20 | Fraction of input filler words absent in output |
| Content preservation | 0.30 | SequenceMatcher ratio of content words vs. gold |
| Structural match | 0.25 | Whether the output has the right markdown structure (list, table, plain) |
| Brevity | 0.10 | Penalizes output longer than gold (hallucination signal) |
| Exact match | 0.15 | Bonus for normalized exact match |
The structural match component was critical: it checks whether the model produced a bullet list when the gold has one, a table when the gold has one, and penalizes unwanted structure insertion on plain-text examples.
Why the metric changed
The earlier version of the metric, used for the 0.5B experiments, lacked the structural component and used Jaccard similarity instead of
SequenceMatcher. This made it blind to formatting quality, which is why we redesigned it for the expanded dataset.
Experiment 1: Can the 0.5B Model Work?
Before committing to the 1.5B model, we tested whether DSPy could rescue the 0.5B variant. The dataset for this experiment was the simpler 35-pair set: filler removal only, with no markdown formatting.
| Optimizer | Score | Change |
|---|---|---|
| Zero-shot baseline | 69.1% | — |
| BootstrapFewShot | 76.1% | +7.0 |
| MIPROv2 (+ 1.5B teacher) | 82.8% | +13.7 |
| RandomSearch | 83.6% | +14.5 |
DSPy squeezed an impressive 14.5 points out of the 0.5B model—nearly as much absolute improvement as it would later achieve on the 1.5B. But examining the actual outputs revealed the model still couldn’t reliably follow instructions:
Verbatim parrot—zero cleanup
In: Um testing out whether or not the uh cleanup functionality works.
Out: Um testing out whether or not the uh cleanup functionality works.
Empty output—hallucinated nothing
In: So like I was thinking you know maybe we should like refactor the the database layer
Out: (empty)
Kept filler, added punctuation around it
In: Well uh I guess I guess we need to to fix the the authentication flow
Out: Well, uh, I guess we need to fix the authentication flow.
You cannot prompt-engineer your way past a model’s comprehension ceiling. In these experiments, the 0.5B model did not understand “remove this word” reliably enough, regardless of how many examples we showed it.
Experiment 2: The 1.5B Model with the Full Dataset
We expanded the dataset to 59 examples covering markdown lists, tables, professional emails, code identifiers, number formatting, and casual cleanup. We then ran the same three optimizers on the 1.5B model.
Baseline
At zero-shot—no few-shot examples, just the system prompt—the 1.5B scored 71.2%. It handled simple filler removal well but failed on structural formatting. It would not produce markdown lists or tables unprompted.
BootstrapFewShot
The simplest DSPy optimizer. It runs the model on training examples, collects successful traces where the metric exceeds a threshold, and prepends them as few-shot demonstrations in the prompt.
Configuration: max_bootstrapped_demos=4, max_labeled_demos=6, max_rounds=2
Result: 80.4% (+9.2)
The model now produced markdown formatting when shown examples of it. But the demo selection was not optimal—it used whatever traces it bootstrapped first, without searching for the best combination.
BootstrapFewShotWithRandomSearch
Samples multiple sets of bootstrapped demonstrations and evaluates each on the validation set, keeping the best.
Configuration: num_candidate_programs=16, max_bootstrapped_demos=4, max_labeled_demos=8
Result: 85.1% (+13.8)
Searching over 16 candidate demo sets found better combinations than the greedy single-pass approach.
MIPROv2
The most sophisticated optimizer. MIPROv2 jointly optimizes instructions and few-shot demonstrations using Bayesian optimization—a TPE sampler via Optuna—over the combinatorial space of instruction variants × demo sets.
Configuration: auto="medium" (18 trials, 6 instruction candidates, 12 demo sets)
MIPROv2 proposed 6 instruction candidates. The 1.5B model is capable enough to write reasonable instructions but also produced some degenerate ones:
Instruction 0 (our original—the winner)
“Clean up a raw speech-to-text transcript for professional use…”
Instruction 1 (infinite repetition loop)
“Convert the transcript ‘Convert the transcript ‘Convert the transcript ‘Convert the transcript ‘Convert…”
Instruction 2 (over-summarized)
“Convert the following transcript into a succinct one-sentence summary…”
The Bayesian search converged on Instruction 0 + Demo Set 6 in trial 4, confirmed it across trials 16 and 17, and returned it as the winner.
Result: 88.0% (+16.7)
Where the improvement came from
MIPROv2 kept our hand-written instruction verbatim. The entire lift came from demo selection. This suggests the instruction was already well-crafted and the bottleneck was showing the model what good outputs look like, not telling it how to produce them.
Final Results
| Optimizer | Score | Change from baseline |
|---|---|---|
| Zero-shot baseline | 71.2% | — |
| BootstrapFewShot | 80.4% | +9.2 |
| RandomSearch | 85.1% | +13.8 |
| MIPROv2 | 88.0% | +16.7 |
The Winning Prompt
MIPROv2’s best trial used our original instruction with 6 few-shot demonstrations. The demos were selected by Bayesian optimization from 12 bootstrapped candidate sets and cover the full taxonomy of cleanup tasks:
| # | Category | What it teaches |
|---|---|---|
| 1 | Pros/cons | Structured arguments → labeled bullet lists |
| 2 | Filler removal | Simple “um”/“uh” deletion |
| 3 | Casual tone | Preserve informal phrasing while cleaning |
| 4 | Table | Names + roles → markdown table |
| 5 | Heavy stuttering | Major deduplication and restructuring |
| 6 | Deduplication | Repeated words only, no fillers to remove |
These 6 demos are baked directly into the production prompt as chat-format few-shot examples. The full prompt—system message + 6 user/assistant pairs + the new user input—is approximately 650 tokens, leaving ample generation budget for the model’s response.
Production Outputs
The baked prompt running on the 1.5B model at inference time:
Filler removal
In: Um testing out whether or not the uh cleanup functionality works.
Out: Testing out whether or not the cleanup functionality works.
List detection
In: so the action items are first update the docs second fix the login bug and third deploy to staging
Out: Action items:
- Update the docs
- Fix the login bug
- Deploy to staging
Professional email
In: hi Sarah um I wanted to follow up on our conversation regarding the Q3 budget I’ve reviewed the numbers and I think we can allocate an additional fifteen thousand to the infrastructure team let me know if you’d like to discuss
Out: Hi Sarah, I wanted to follow up on our conversation regarding the Q3 budget. I’ve reviewed the numbers and I think we can allocate an additional $15,000 to the infrastructure team. Let me know if you’d like to discuss.
Code identifiers
In: the api endpoint is slash api slash v2 slash users and it runs on port eight zero eight zero
Out: The API endpoint is
api/v2/usersand it runs on port8080.
Heavy stuttering
In: I was I was gonna say that uh maybe we should just just use Redis for for caching
Out: I was gonna say that maybe we should just use Redis for caching.
Clean passthrough
In: The quick brown fox jumps over the lazy dog
Out: The quick brown fox jumps over the lazy dog.
Cross-Model Comparison
A natural question: could DSPy lift the cheaper 0.5B model to match the 1.5B? No.
| Model | Zero-shot | Best DSPy | Optimizer | Lift |
|---|---|---|---|---|
| 0.5B | 69.1% | 83.6% | RandomSearch | +14.5 |
| 1.5B | 71.2% | 88.0% | MIPROv2 | +16.7 |
DSPy extracted a substantial improvement from the 0.5B, but the 1.5B’s higher capability floor means its optimized score still exceeds the 0.5B’s best by 4.4 points—and critically, produces zero broken outputs: no empty strings, kept fillers, or hallucinations.
The capability floor
DSPy optimizes how you ask, but cannot change what the model understands. At 0.5B parameters, this model lacked the instruction-following capacity to reliably execute “remove this word” regardless of prompting strategy. The 1.5B model crossed that threshold.
Takeaways
- Few-shot demo selection matters more than instruction tuning at this model size. MIPROv2’s entire 16.7-point lift came from choosing the right 6 examples, not from rewriting the instruction.
- There is a capability floor below which prompting cannot help. The 0.5B model hit it. No amount of DSPy optimization fixed its inability to delete words from a string.
- Bayesian search over demo sets beats random search by about 3 points (85.1 → 88.0), and random search beats greedy selection by about 5 points (80.4 → 85.1). The search budget—16 candidates for RandomSearch, 18 trials for MIPROv2—is modest.
- The composite metric drives the optimization. Switching from a flat Jaccard metric to one with explicit structural matching improved the optimizer’s ability to find demos that teach markdown formatting, not just filler removal.
- Tiny models can punch above their weight with the right prompting. A 1.5B 4-bit model (about 1 GB) producing correctly formatted markdown tables and professional emails from messy dictation is a good outcome for something that runs alongside an STT model on a laptop.
Reproducing
The full optimization pipeline lives in the yap repository:
# Install dependencies
uv sync --group dev
# Start the model server
uv run python -m mlx_lm.server \
--model mlx-community/Qwen2.5-1.5B-Instruct-4bit \
--port 8321
# Run optimization (in another terminal)
uv run python scripts/optimize_prompt.py
# Results saved to scripts/optimized_cleanup.jsonThe training data is in scripts/dataset.json (59 pairs). The production prompt with baked demos is in yap/llm.py.
June 2026. Part of the yap project.