laantungir/voice_linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement voice commands tab with key combo capture and runtime trigger processing

Browse Source
This commit is contained in:
Laan Tungir
2026-03-26 22:39:59 -04:00
parent e1cfd7fc98
commit bc4fc50c7d
11 changed files with 1401 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
Makefile
View File

@@ -21,7 +21,8 @@ SRC := \
./src/config.c \
./src/audio.c \
./src/transcribe.c \
./src/typer.c
./src/typer.c \
./src/voice_cmd.c
TARGET := ./voice_linux

2
build.sh
View File

@@ -126,7 +126,7 @@ if [[ "$WITH_WHISPER" == "1" ]]; then
fi
gcc "${APP_CFLAGS[@]}" \
./src/gui_main.c ./src/config.c ./src/audio.c ./src/transcribe.c ./src/typer.c \
./src/gui_main.c ./src/config.c ./src/audio.c ./src/transcribe.c ./src/typer.c ./src/voice_cmd.c \
-o ./voice_linux \
"${APP_LDFLAGS[@]}"

35
config.ini
View File

@@ -16,7 +16,7 @@ always_on=1
always_on_window_ms=1400
# live level threshold (0.0 - 1.0)
vad_peak_threshold=0.0060
vad_peak_threshold=0.0460
# level must stay above threshold this long before speech starts
vad_trigger_ms=100
# level must stay below threshold this long before speech ends
@@ -29,4 +29,35 @@ vad_min_speech_ms=250
vad_max_speech_ms=15000
autotype_enabled=1
filter_bracketed_tags=1
filter_bracketed_tags=0
# whisper decode controls
whisper_sampling_strategy=0
whisper_n_max_text_ctx=16384
whisper_offset_ms=0
whisper_duration_ms=0
whisper_translate=0
whisper_detect_language=0
whisper_no_context=1
whisper_no_timestamps=1
whisper_single_segment=1
whisper_token_timestamps=1
whisper_thold_pt=0.0100
whisper_thold_ptsum=0.0100
whisper_max_len=0
whisper_split_on_word=0
whisper_max_tokens=0
whisper_audio_ctx=0
whisper_tdrz_enable=0
whisper_suppress_blank=1
whisper_suppress_nst=1
whisper_temperature=0.0000
whisper_max_initial_ts=1.0000
whisper_length_penalty=-1.0000
whisper_temperature_inc=0.2000
whisper_entropy_thold=2.4000
whisper_logprob_thold=-1.0000
whisper_no_speech_thold=0.6000
whisper_greedy_best_of=1
whisper_beam_size=5
whisper_beam_patience=-1.0000

225
plans/gpu_passthrough_lessons_learned.md Normal file
View File

@@ -0,0 +1,225 @@
# GPU Passthrough in Qubes OS — Summary & Lessons Learned
## What We Did
Passed an NVIDIA GTX 1080 Ti (BDF `65:00.0` / `65:00.1`) through from Qubes dom0 to a StandaloneVM called `ai`, installed NVIDIA drivers and CUDA toolkit, and rebuilt whisper.cpp with CUDA support for GPU-accelerated speech-to-text.
## Environment
| Component | Detail |
|-----------|--------|
| Host OS | Qubes OS with Xen hypervisor |
| dom0 display GPU | AMD (stays in dom0) |
| Passthrough GPU | NVIDIA GeForce GTX 1080 Ti (Pascal/GP102, 11GB VRAM, compute 6.1) |
| Second NVIDIA GPU | RTX 3050 6GB at BDF `17:00.0` (not used for this project) |
| Target VM | `ai` — originally an AppVM, converted to StandaloneVM |
| VM OS | Debian 13 (bookworm/trixie) with XFCE |
| NVIDIA driver | 550.127.05 |
| CUDA toolkit | 12.6 |
## Timeline of Steps
### Phase 0: dom0 GPU Passthrough
1. **Identified GPU BDF addresses** — `lspci` in dom0 showed two NVIDIA GPUs. We targeted the GTX 1080 Ti at `65:00.0` (GPU) and `65:00.1` (HDMI audio controller).
2. **Confirmed boot loader** — Verified Qubes was using GRUB (not systemd-boot) by checking `/etc/default/grub` existed.
3. **Hid GPU from dom0** — Added `rd.qubes.hide_pci=65:00.0,65:00.1` to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`.
4. **Regenerated GRUB config** — `sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg` (EFI path, not legacy BIOS path).
5. **Rebooted dom0** — Full system reboot required for PCI hide to take effect.
6. **Verified GPU was assignable** — `xl pci-assignable-list` showed `0000:65:00.0` and `0000:65:00.1`.
7. **Attached GPU to VM** — `qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true` (and same for `65_00.1`).
### Phase 1: Driver & CUDA Installation (inside ai VM)
8. **Installed build prerequisites** — `build-essential`, `linux-headers-$(uname -r)`.
9. **Installed NVIDIA driver** — Downloaded `.run` installer, ran with `--no-opengl-files --dkms`.
10. **Installed CUDA toolkit** — Via NVIDIA's Debian repo + `cuda-toolkit-12-6`.
11. **Verified** — `nvidia-smi` showed GTX 1080 Ti, `nvcc --version` showed CUDA 12.6.
### Phase 2: Application Rebuild
12. **Rebuilt whisper.cpp with CUDA** — `WHISPER_CUDA=1 bash ./build.sh`.
13. **Verified CUDA linkage** — `ldd ./voice_linux | grep whisper|ggml` confirmed CUDA libraries linked.
14. **Tested GPU inference** — Ran with GPU mode, confirmed `nvidia-smi` showed process using GPU memory.
## Problems Encountered & Solutions
### Problem 1: AppVM vs StandaloneVM confusion
**What happened**: The `ai` VM was initially described as an AppVM (template-based). In an AppVM, anything installed outside `/home` is lost on reboot — NVIDIA drivers would vanish.
**Discovery**: User clarified it was actually a StandaloneVM, not template-based.
**Lesson**: Always verify VM type before planning driver installation. In Qubes:
- **AppVM**: Root filesystem resets to template on reboot. Drivers must go in the template or use bind-dirs.
- **StandaloneVM**: Full persistent root filesystem. Drivers persist normally.
- Check with: `qvm-prefs ai virt_mode` and `qvm-ls --fields name,klass ai`
### Problem 2: VM must be shut down before PCI attach
**What happened**: Attempted to run `qvm-pci attach` while the `ai` VM was running. The attach appeared to succeed but the device wasn't visible inside the VM.
**Discovery**: `qvm-device pci list ai` showed empty even though attach command ran.
**Lesson**: The VM must be **shut down** before attaching PCI devices with `--persistent`. The workflow is:
1. Shut down VM
2. Attach PCI device
3. Start VM
4. Verify inside VM with `lspci`
### Problem 3: "Already assigned" error on re-attach
**What happened**: After a reboot cycle, running the attach command again produced an "already assigned" message.
**Lesson**: `--persistent` means the attachment survives reboots. Don't re-run the attach command after reboot — it's already configured. Verify with `qvm-pci list ai` from dom0.
### Problem 4: lspci not found inside VM
**What happened**: Tried to verify GPU visibility inside the VM but `lspci` wasn't installed.
**Solution**: `sudo apt install pciutils` then `lspci | grep -i nvidia`.
**Lesson**: Minimal Debian VMs may not have `pciutils` installed. Include it in prerequisites.
### Problem 5: Script couldn't be copy-pasted into dom0
**What happened**: Created a comprehensive dom0 shell script for GPU passthrough, but Qubes security model prevents clipboard paste from other VMs into dom0.
**Solution**: Provided numbered step-by-step commands that could be typed manually, and also created the script as a file that could be transferred via `qvm-run` or Qubes file copy.
**Lesson**: When automating dom0 tasks in Qubes:
- Dom0 is intentionally isolated — no clipboard sharing
- Scripts must be typed manually or transferred via `qvm-copy-to-vm` (from dom0 to VM) or `qvm-run -p` pipes
- Keep dom0 scripts short and simple
- Always include a revert mechanism
### Problem 6: Reboot scope confusion
**What happened**: Unclear whether "reboot" meant just the VM or the entire Qubes system (dom0 + all VMs).
**Clarification**: GRUB changes require a **full system reboot** (dom0 reboot, which takes down all VMs). PCI attachment changes only require the target VM to be restarted.
**Lesson**: Be explicit about reboot scope:
- `sudo reboot` in dom0 = full system reboot
- `qvm-shutdown ai && qvm-start ai` = just the VM
### Problem 7: Build accidentally ran without whisper support
**What happened**: After some iteration, a build was accidentally triggered with `WITH_WHISPER=0`, producing a binary that showed "[transcription unavailable: rebuild with WITH_WHISPER=1]".
**Solution**: Rebuilt with `WITH_WHISPER=1 bash ./build.sh`.
**Lesson**: The build system defaults matter. Our `build.sh` auto-detects whisper if the vendor directory exists, but explicit `WITH_WHISPER=0` overrides that. Always verify the build output includes whisper support with `ldd ./voice_linux | grep whisper`.
### Problem 8: Dirty git worktree blocking version increment
**What happened**: The `increment_and_push.sh` script refused to run because of uncommitted changes in the working tree.
**Solution**: Committed the intended changes first, then ran the increment script.
**Lesson**: The increment script enforces a clean worktree policy. Always commit your changes before running it. If there are unrelated/untracked files, `git stash` them first.
## Key Qubes-Specific Knowledge
### PCI Passthrough Essentials
```
# Hide device from dom0 (GRUB, requires full reboot):
rd.qubes.hide_pci=BDF1,BDF2
# Attach to VM (VM must be off):
qvm-pci attach VMNAME dom0:BDF --persistent -o permissive=true
# Verify assignment:
qvm-pci list VMNAME # from dom0
lspci | grep -i nvidia # from inside VM
```
### NVIDIA Driver in Qubes VM
- **Always use `--no-opengl-files`** — Qubes VMs use a virtual GPU for display. Installing OpenGL files would break the display.
- **Use `--dkms`** — Ensures kernel module rebuilds on kernel updates.
- **StandaloneVM recommended** — Avoids template pollution and persistence issues.
- **`permissive=true`** — Required for NVIDIA GPUs in Qubes due to how they access PCI config space.
### What Persists Where
| VM Type | /home | /usr, /lib, /etc | Drivers |
|---------|-------|-------------------|---------|
| AppVM | ✅ Persists | ❌ Resets to template | ❌ Lost on reboot |
| StandaloneVM | ✅ Persists | ✅ Persists | ✅ Persists |
| Template | ✅ Persists | ✅ Persists | ✅ Persists (shared to AppVMs) |
### Dom0 Safety
- Dom0 has no network access by design
- Clipboard is one-way (dom0 → VM, not VM → dom0) and requires explicit Ctrl+Shift+C/V
- Always have a revert plan for GRUB changes (keep a backup of `/etc/default/grub`)
- Log what you change — our script wrote to `/var/log/gpu_passthrough.log`
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────┐
│ dom0 (Xen Hypervisor) │
│ │
│ AMD GPU ──── Display │
│ GTX 1080 Ti ──── HIDDEN via rd.qubes.hide_pci │
│ │ │
│ │ PCI passthrough (qvm-pci attach --persistent) │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ ai StandaloneVM │ │
│ │ │ │
│ │ NVIDIA Driver 550.x (--no-opengl-files --dkms) │ │
│ │ │ │ │
│ │ CUDA Toolkit 12.6 │ │
│ │ │ │ │
│ │ whisper.cpp (GGML_CUDA=ON) │ │
│ │ │ │ │
│ │ voice_linux (GPU-accelerated transcription) │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
## Performance Results
| Model | VRAM Used | Decode Time (10s audio) | Quality |
|-------|-----------|------------------------|---------|
| tiny.en | ~1GB | <0.5s | Fair |
| base.en | ~1GB | <1s | Good |
| small.en | ~2GB | ~1-2s | Very good |
| medium.en | ~5GB | ~2-3s | Excellent |
| large-v3-turbo | ~6GB | ~3-4s | Excellent+ |
| large-v3 | ~10GB | ~4-6s | Best |
The GTX 1080 Ti with 11GB VRAM can run all models including large-v3. For real-time dictation, medium.en provides the best accuracy-to-speed tradeoff.
## Files Created During This Process
| File | Purpose |
|------|---------|
| [`plans/gpu_enablement_plan.md`](gpu_enablement_plan.md) | Step-by-step GPU passthrough and driver installation plan |
| [`plans/architecture.md`](architecture.md) | Full system architecture including hardware details |
| [`scripts/download_models.sh`](../scripts/download_models.sh) | Downloads all compatible whisper models |
## If You Had to Do It Again
1. **Verify VM type first** — `qvm-ls --fields name,klass ai` before planning anything
2. **Shut down VM before PCI operations** — always
3. **Back up GRUB config** — `sudo cp /etc/default/grub /etc/default/grub.bak` before editing
4. **Install pciutils early** — `sudo apt install pciutils` so you can verify GPU visibility
5. **Use the .run installer, not apt packages** — NVIDIA's apt packages for Debian can conflict with Qubes' virtual GPU setup; the `.run` installer with `--no-opengl-files` is cleaner
6. **Test CPU mode first** — Build and verify voice_linux works on CPU before adding GPU complexity
7. **Keep dom0 commands minimal** — Type them manually, don't try to automate complex scripts in dom0

202
plans/voice_commands_plan.md Normal file
View File

@@ -0,0 +1,202 @@
# Voice Commands Tab — Implementation Plan
## Overview
Add a third GUI tab ("Commands") that lets users define trigger phrases from whisper output and map them to either text replacements or keyboard combinations. This enables meta-speech like saying "period" to insert `.`, or "over" to send `Return`.
## Architecture
### Data Model
```c
typedef enum {
VOICE_CMD_TEXT, // replace trigger with literal text
VOICE_CMD_KEYCOMBO // send a key combination via XTest
} voice_cmd_type_t;
typedef struct {
char trigger[128]; // spoken phrase, case-insensitive
voice_cmd_type_t type;
char text_value[256]; // for TEXT type: replacement string
unsigned int keyval; // for KEYCOMBO type: GDK keyval
unsigned int modifiers; // for KEYCOMBO type: modifier mask
char display_key[128]; // human-readable key description
} voice_cmd_t;
typedef struct {
voice_cmd_t *cmds;
int count;
int capacity;
int enabled; // global on/off toggle
} voice_cmd_list_t;
```
### Processing Pipeline
```
transcribe_buffer() → raw text
↓
voice_cmd_apply(text, list) → action sequence
↓
For each action:
- TEXT_CHUNK: accumulate into output string
- KEY_EVENT: call typer_send_keycombo()
↓
Final text → append_transcript() + typer_type_text()
```
The apply function returns a sequence of interleaved actions because a single transcript may contain multiple triggers mixed with regular text. Example: "hello over goodbye over" → text:"hello " → key:Return → text:"goodbye " → key:Return.
### Matching Strategy
- Case-insensitive comparison
- Whole-word boundary matching to avoid false positives
- Longest-match-first: sort triggers by length descending before scanning
- Left-to-right scan through transcript text
- Matched trigger phrases are consumed and not included in output text
## New Files
### src/voice_cmd.h
- `voice_cmd_t` and `voice_cmd_list_t` structs
- `void voice_cmd_init(voice_cmd_list_t *list)`
- `void voice_cmd_free(voice_cmd_list_t *list)`
- `int voice_cmd_add(voice_cmd_list_t *list, const voice_cmd_t *cmd)`
- `int voice_cmd_remove(voice_cmd_list_t *list, int index)`
- `int voice_cmd_load(const char *path, voice_cmd_list_t *list)`
- `int voice_cmd_save(const char *path, const voice_cmd_list_t *list)`
### src/voice_cmd.c
- Load/save from `voice_commands.ini`
- `voice_cmd_apply()` — the core text processing function
### voice_commands.ini
Persisted command rules, separate from main config.ini.
Format:
```
# trigger|type|value
# type: text or key
# for text: value is the replacement string (supports \n \t escapes)
# for key: value is keyval:modifiers (decimal GDK values)
period|text|.
question mark|text|?
comma|text|,
exclamation mark|text|!
new line|text|\n
new paragraph|text|\n\n
tab|text|\t
over|key|65293:0
end of command|key|65293:5
```
## Modified Files
### src/typer.h / src/typer.c
- Add: `int typer_send_keycombo(unsigned int keyval, unsigned int modifiers)`
- Uses XTest to send arbitrary key combinations with modifier state
- Reuses existing X11 display connection from typer_init()
### src/gui_main.c
- Add third notebook tab in on_app_activate()
- Tab contains:
- GtkTreeView showing existing commands (trigger, type, action display)
- Delete button per row or for selected row
- Add-new-command form:
- GtkEntry for trigger phrase
- GtkRadioButton pair: Text / Key Combo
- GtkEntry for text replacement value
- GtkButton "Capture Key" + GtkLabel showing captured key
- GtkButton "Add Command"
- GtkCheckButton "Enable voice commands" toggle
- Key capture workflow:
1. User clicks Capture
2. Button label changes to "Press key combo now..."
3. Window key-press-event handler captures next keypress
4. Records keyval + modifier state
5. Formats display string like "Ctrl+Shift+Return"
6. Disconnects capture handler, restores button label
### src/gui_main.c — process_segment()
- After transcribe_buffer() returns text, call voice_cmd_apply()
- Process returned action sequence: text chunks go to transcript/autotype, key events go to typer_send_keycombo()
### Makefile
- Add voice_cmd.o to build targets
### build.sh
- Add src/voice_cmd.c to compilation
## GUI Tab Layout
```
┌─ Voice Commands ──────────────────────────────────────────┐
│ │
│ ☑ Enable voice commands │
│ │
│ ┌──────────────┬──────┬─────────────────┬───────────┐ │
│ │ Trigger │ Type │ Action │ │ │
│ ├──────────────┼──────┼─────────────────┼───────────┤ │
│ │ period │ text │ . │ [Delete] │ │
│ │ question mark│ text │ ? │ [Delete] │ │
│ │ new line │ text │ \n │ [Delete] │ │
│ │ over │ key │ Return │ [Delete] │ │
│ │ end of cmd │ key │ Ctrl+Shift+Ret │ [Delete] │ │
│ └──────────────┴──────┴─────────────────┴───────────┘ │
│ │
│ ── Add New Command ── │
│ Trigger: [____________] │
│ ○ Text replacement ● Key combination │
│ Text: [____________] │
│ Key: [Press keys...] [Capture] │
│ [Add Command] │
│ │
│ Help: [hover text area] │
└────────────────────────────────────────────────────────────┘
```
## Default Commands (shipped with first install)
| Trigger | Type | Action |
|---------|------|--------|
| period | text | . |
| question mark | text | ? |
| exclamation mark | text | ! |
| comma | text | , |
| colon | text | : |
| semicolon | text | ; |
| new line | text | \n |
| new paragraph | text | \n\n |
| tab | text | \t |
| open paren | text | ( |
| close paren | text | ) |
| open bracket | text | [ |
| close bracket | text | ] |
| open brace | text | { |
| close brace | text | } |
## Edge Cases
1. **Substring false positives**: "I need a period of time" — whole-word boundary matching prevents "period" from matching inside "periodical" but this phrase would still trigger. Could add a "require end-of-phrase position" option per command, but start simple with whole-word matching.
2. **Case sensitivity**: Whisper may output "Period" or "period" depending on sentence position. All matching is case-insensitive.
3. **Multi-word triggers**: "end of command" requires multi-word scanning. Sort triggers longest-first so "end of command" matches before "end".
4. **Escape sequences in text values**: Support \n, \t, \\ in the text replacement field. Parse on load, display escaped in UI.
5. **Key capture conflicts**: While capturing, consume the event so it doesn't trigger other handlers. Disconnect capture handler after first keypress.
6. **Empty transcript after replacements**: If the entire transcript was voice commands with no remaining text, skip autotype and transcript append.
## Implementation Order
1. Create src/voice_cmd.h and src/voice_cmd.c with data structures and load/save
2. Add typer_send_keycombo() to src/typer.h / src/typer.c
3. Implement voice_cmd_apply() core matching logic
4. Build the Commands tab UI in src/gui_main.c
5. Wire key capture handler
6. Hook voice_cmd_apply() into process_segment()
7. Create default voice_commands.ini
8. Update Makefile and build.sh
9. Build and test

447
src/gui_main.c
View File

@@ -3,6 +3,7 @@
#include "transcribe.h"
#include "typer.h"
#include "version.h"
#include "voice_cmd.h"
#include <gtk/gtk.h>
@@ -67,6 +68,24 @@ typedef struct {
GtkWidget *logprob_thold_spin;
GtkWidget *no_speech_thold_spin;
GtkWidget *beam_patience_spin;
GtkWidget *whisper_help_label;
voice_cmd_list_t voice_cmds;
char voice_cmd_path[512];
GtkListStore *cmd_store;
GtkWidget *cmd_tree;
GtkWidget *cmd_enable_check;
GtkWidget *cmd_trigger_entry;
GtkWidget *cmd_text_entry;
GtkWidget *cmd_mode_text_radio;
GtkWidget *cmd_mode_key_radio;
GtkWidget *cmd_capture_btn;
GtkWidget *cmd_capture_label;
GtkWidget *cmd_add_btn;
GtkWidget *cmd_delete_btn;
unsigned int cmd_capture_keyval;
unsigned int cmd_capture_mods;
int cmd_capture_armed;
voice_config_t cfg;
int use_gpu;
@@ -82,6 +101,13 @@ typedef struct {
char cfg_path[512];
} gui_state_t;
enum {
CMD_COL_TRIGGER = 0,
CMD_COL_TYPE,
CMD_COL_ACTION,
CMD_COL_COUNT
};
static float clampf(float v, float lo, float hi) {
if (v < lo) return lo;
if (v > hi) return hi;
@@ -296,6 +322,16 @@ static void remember_transcript(gui_state_t *s, const char *text) {
s->last_transcript_time = monotonic_seconds();
}
static int voice_cmd_key_emit_cb(unsigned int keyval, unsigned int modifiers, void *user_data) {
gui_state_t *s = (gui_state_t *)user_data;
if (!s) return -1;
if (typer_send_keycombo(keyval, modifiers) != 0) {
fprintf(stderr, "gui: voice command key combo failed (key=%u mods=%u)\n", keyval, modifiers);
return -1;
}
return 0;
}
static void process_segment(gui_state_t *s, float *samples, size_t count) {
if (!samples || count == 0) {
free(samples);
@@ -339,27 +375,41 @@ static void process_segment(gui_state_t *s, float *samples, size_t count) {
return;
}
if (is_duplicate_transcript(s, text)) {
char *processed = voice_cmd_apply(text,
&s->voice_cmds,
should_autotype ? voice_cmd_key_emit_cb : NULL,
s);
if (!processed) {
processed = text;
text = NULL;
}
if (is_duplicate_transcript(s, processed)) {
fprintf(stderr, "gui: suppressed duplicate transcript within 1s window\n");
set_status(s, "Duplicate segment suppressed");
free(text);
free(processed);
return;
}
append_transcript(s, text);
if (processed[0] != '\0') {
append_transcript(s, processed);
}
if (should_autotype) {
if (typer_type_text(text) != 0) {
if (should_autotype && processed[0] != '\0') {
if (typer_type_text(processed) != 0) {
fprintf(stderr, "gui: autotype injection failed\n");
set_status(s, "Autotype failed (see terminal logs)");
free(text);
free(processed);
return;
}
}
remember_transcript(s, text);
remember_transcript(s, processed);
set_status(s, audio_msg);
free(text);
free(processed);
}
static int read_scale_int(GtkWidget *scale) {
@@ -656,6 +706,35 @@ static void on_whisper_combo_changed(GtkComboBox *box, gpointer user_data) {
apply_whisper_settings_from_widgets(s);
}
static void set_whisper_help_text(gui_state_t *s, const char *text) {
if (!s || !s->whisper_help_label) return;
gtk_label_set_text(GTK_LABEL(s->whisper_help_label), text ? text : "");
}
static gboolean on_whisper_help_enter(GtkWidget *widget, GdkEventCrossing *event, gpointer user_data) {
(void)event;
gui_state_t *s = (gui_state_t *)user_data;
const char *text = (const char *)g_object_get_data(G_OBJECT(widget), "whisper-help-text");
set_whisper_help_text(s, text ? text : "No help available for this item.");
return FALSE;
}
static gboolean on_whisper_help_leave(GtkWidget *widget, GdkEventCrossing *event, gpointer user_data) {
(void)widget;
(void)event;
gui_state_t *s = (gui_state_t *)user_data;
set_whisper_help_text(s, "Hover any control to see what it does.");
return FALSE;
}
static void attach_whisper_help(gui_state_t *s, GtkWidget *widget, const char *help_text) {
if (!s || !widget || !help_text) return;
g_object_set_data(G_OBJECT(widget), "whisper-help-text", (gpointer)help_text);
gtk_widget_add_events(widget, GDK_ENTER_NOTIFY_MASK | GDK_LEAVE_NOTIFY_MASK);
g_signal_connect(widget, "enter-notify-event", G_CALLBACK(on_whisper_help_enter), s);
g_signal_connect(widget, "leave-notify-event", G_CALLBACK(on_whisper_help_leave), s);
}
static GtkWidget *add_spin_row(GtkWidget *grid,
int col,
int row,
@@ -665,6 +744,7 @@ static GtkWidget *add_spin_row(GtkWidget *grid,
double step,
double value,
int digits,
const char *help_text,
gui_state_t *s,
GtkWidget **out_spin) {
GtkWidget *label = gtk_label_new(label_text);
@@ -678,6 +758,8 @@ static GtkWidget *add_spin_row(GtkWidget *grid,
gtk_grid_attach(GTK_GRID(grid), spin, col + 1, row, 1, 1);
g_signal_connect(spin, "value-changed", G_CALLBACK(on_whisper_spin_changed), s);
attach_whisper_help(s, label, help_text);
attach_whisper_help(s, spin, help_text);
*out_spin = spin;
return spin;
}
@@ -711,56 +793,68 @@ static GtkWidget *build_whisper_settings_tab(gui_state_t *s) {
gtk_combo_box_set_active(GTK_COMBO_BOX(s->sampling_combo), s->cfg.whisper_sampling_strategy ? 1 : 0);
g_signal_connect(s->sampling_combo, "changed", G_CALLBACK(on_whisper_combo_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->sampling_combo, 1, 0, 1, 1);
attach_whisper_help(s, sampling_label, "Choose decoder search method. Greedy is faster; Beam Search may improve quality at higher compute cost.");
attach_whisper_help(s, s->sampling_combo, "Choose decoder search method. Greedy is faster; Beam Search may improve quality at higher compute cost.");
s->translate_check = gtk_check_button_new_with_label("Translate to English");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->translate_check), s->cfg.whisper_translate ? TRUE : FALSE);
g_signal_connect(s->translate_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->translate_check, 0, 1, 1, 1);
attach_whisper_help(s, s->translate_check, "Translate detected speech to English output instead of transcribing in-source language.");
s->detect_language_check = gtk_check_button_new_with_label("Auto detect language");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->detect_language_check), s->cfg.whisper_detect_language ? TRUE : FALSE);
g_signal_connect(s->detect_language_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->detect_language_check, 1, 1, 1, 1);
attach_whisper_help(s, s->detect_language_check, "Enable automatic language detection each decode call.");
s->no_context_check = gtk_check_button_new_with_label("No context");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->no_context_check), s->cfg.whisper_no_context ? TRUE : FALSE);
g_signal_connect(s->no_context_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->no_context_check, 0, 2, 1, 1);
attach_whisper_help(s, s->no_context_check, "Disable conditioning on previous text context. Can reduce drift but may reduce continuity.");
s->no_timestamps_check = gtk_check_button_new_with_label("No timestamps");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->no_timestamps_check), s->cfg.whisper_no_timestamps ? TRUE : FALSE);
g_signal_connect(s->no_timestamps_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->no_timestamps_check, 1, 2, 1, 1);
attach_whisper_help(s, s->no_timestamps_check, "Disable timestamp generation. Usually faster and cleaner for plain transcript text.");
s->single_segment_check = gtk_check_button_new_with_label("Single segment");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->single_segment_check), s->cfg.whisper_single_segment ? TRUE : FALSE);
g_signal_connect(s->single_segment_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->single_segment_check, 0, 3, 1, 1);
attach_whisper_help(s, s->single_segment_check, "Force one segment output per decode call; useful for streaming-like behavior.");
s->token_timestamps_check = gtk_check_button_new_with_label("Token timestamps");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->token_timestamps_check), s->cfg.whisper_token_timestamps ? TRUE : FALSE);
g_signal_connect(s->token_timestamps_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->token_timestamps_check, 1, 3, 1, 1);
attach_whisper_help(s, s->token_timestamps_check, "Enable token-level timestamp estimates (experimental).");
s->split_on_word_check = gtk_check_button_new_with_label("Split on word");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->split_on_word_check), s->cfg.whisper_split_on_word ? TRUE : FALSE);
g_signal_connect(s->split_on_word_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->split_on_word_check, 0, 4, 1, 1);
attach_whisper_help(s, s->split_on_word_check, "When max_len is set, prefer splitting at word boundaries.");
s->suppress_blank_check = gtk_check_button_new_with_label("Suppress blank tokens");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->suppress_blank_check), s->cfg.whisper_suppress_blank ? TRUE : FALSE);
g_signal_connect(s->suppress_blank_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->suppress_blank_check, 1, 4, 1, 1);
attach_whisper_help(s, s->suppress_blank_check, "Suppress blank token sampling to reduce empty/no-op outputs.");
s->suppress_nst_check = gtk_check_button_new_with_label("Suppress non-speech tokens");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->suppress_nst_check), s->cfg.whisper_suppress_nst ? TRUE : FALSE);
g_signal_connect(s->suppress_nst_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->suppress_nst_check, 0, 5, 1, 1);
attach_whisper_help(s, s->suppress_nst_check, "Suppress non-speech tokens like music/noise annotations.");
s->tdrz_enable_check = gtk_check_button_new_with_label("Enable tinydiarize (TDRZ)");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->tdrz_enable_check), s->cfg.whisper_tdrz_enable ? TRUE : FALSE);
g_signal_connect(s->tdrz_enable_check, "toggled", G_CALLBACK(on_whisper_toggle_changed), s);
gtk_grid_attach(GTK_GRID(checks), s->tdrz_enable_check, 1, 5, 1, 1);
attach_whisper_help(s, s->tdrz_enable_check, "Enable TinyDiarize speaker-turn hints (experimental).");
GtkWidget *grid = gtk_grid_new();
gtk_grid_set_row_spacing(GTK_GRID(grid), 6);
@@ -768,46 +862,336 @@ static GtkWidget *build_whisper_settings_tab(gui_state_t *s) {
gtk_box_pack_start(GTK_BOX(box), grid, FALSE, FALSE, 0);
add_spin_row(grid, 0, 0, "n_max_text_ctx", 0.0, 32768.0, 1.0,
(double)s->cfg.whisper_n_max_text_ctx, 0, s, &s->n_max_text_ctx_spin);
(double)s->cfg.whisper_n_max_text_ctx, 0,
"Maximum number of previous tokens used as prompt context for decoding.",
s, &s->n_max_text_ctx_spin);
add_spin_row(grid, 2, 0, "offset_ms", 0.0, 60000.0, 10.0,
(double)s->cfg.whisper_offset_ms, 0, s, &s->offset_ms_spin);
(double)s->cfg.whisper_offset_ms, 0,
"Start decoding from this time offset (milliseconds) inside the audio chunk.",
s, &s->offset_ms_spin);
add_spin_row(grid, 0, 1, "duration_ms (0=all)", 0.0, 600000.0, 100.0,
(double)s->cfg.whisper_duration_ms, 0, s, &s->duration_ms_spin);
(double)s->cfg.whisper_duration_ms, 0,
"Decode only this duration in milliseconds (0 means full available audio).",
s, &s->duration_ms_spin);
add_spin_row(grid, 2, 1, "max_len", 0.0, 1024.0, 1.0,
(double)s->cfg.whisper_max_len, 0, s, &s->max_len_spin);
(double)s->cfg.whisper_max_len, 0,
"Maximum output segment length in characters before split logic applies.",
s, &s->max_len_spin);
add_spin_row(grid, 0, 2, "max_tokens", 0.0, 1024.0, 1.0,
(double)s->cfg.whisper_max_tokens, 0, s, &s->max_tokens_spin);
(double)s->cfg.whisper_max_tokens, 0,
"Maximum tokens per segment (0 means no explicit token cap).",
s, &s->max_tokens_spin);
add_spin_row(grid, 2, 2, "audio_ctx", 0.0, 4096.0, 1.0,
(double)s->cfg.whisper_audio_ctx, 0, s, &s->audio_ctx_spin);
(double)s->cfg.whisper_audio_ctx, 0,
"Override internal audio context window size (0 keeps model default).",
s, &s->audio_ctx_spin);
add_spin_row(grid, 0, 3, "thold_pt", 0.0, 1.0, 0.001,
(double)s->cfg.whisper_thold_pt, 3, s, &s->thold_pt_spin);
(double)s->cfg.whisper_thold_pt, 3,
"Token timestamp probability threshold for token-level timing.",
s, &s->thold_pt_spin);
add_spin_row(grid, 2, 3, "thold_ptsum", 0.0, 1.0, 0.001,
(double)s->cfg.whisper_thold_ptsum, 3, s, &s->thold_ptsum_spin);
(double)s->cfg.whisper_thold_ptsum, 3,
"Cumulative timestamp probability threshold for token-level timing.",
s, &s->thold_ptsum_spin);
add_spin_row(grid, 0, 4, "temperature", 0.0, 2.0, 0.01,
(double)s->cfg.whisper_temperature, 2, s, &s->temperature_spin);
(double)s->cfg.whisper_temperature, 2,
"Initial sampling temperature. Lower is more deterministic, higher is more diverse.",
s, &s->temperature_spin);
add_spin_row(grid, 2, 4, "max_initial_ts", 0.0, 5.0, 0.01,
(double)s->cfg.whisper_max_initial_ts, 2, s, &s->max_initial_ts_spin);
(double)s->cfg.whisper_max_initial_ts, 2,
"Maximum initial timestamp allowed for first decoded tokens.",
s, &s->max_initial_ts_spin);
add_spin_row(grid, 0, 5, "length_penalty", -2.0, 2.0, 0.01,
(double)s->cfg.whisper_length_penalty, 2, s, &s->length_penalty_spin);
(double)s->cfg.whisper_length_penalty, 2,
"Penalty/boost for longer outputs during decoding (-1 keeps default behavior).",
s, &s->length_penalty_spin);
add_spin_row(grid, 2, 5, "temperature_inc", 0.0, 1.0, 0.01,
(double)s->cfg.whisper_temperature_inc, 2, s, &s->temperature_inc_spin);
(double)s->cfg.whisper_temperature_inc, 2,
"Fallback temperature increment used when decode quality checks fail.",
s, &s->temperature_inc_spin);
add_spin_row(grid, 0, 6, "entropy_thold", -5.0, 10.0, 0.01,
(double)s->cfg.whisper_entropy_thold, 2, s, &s->entropy_thold_spin);
(double)s->cfg.whisper_entropy_thold, 2,
"Fallback entropy threshold controlling when alternate decode attempts are triggered.",
s, &s->entropy_thold_spin);
add_spin_row(grid, 2, 6, "logprob_thold", -5.0, 1.0, 0.01,
(double)s->cfg.whisper_logprob_thold, 2, s, &s->logprob_thold_spin);
(double)s->cfg.whisper_logprob_thold, 2,
"Fallback average log-probability threshold for accepting a decode pass.",
s, &s->logprob_thold_spin);
add_spin_row(grid, 0, 7, "no_speech_thold", 0.0, 1.0, 0.01,
(double)s->cfg.whisper_no_speech_thold, 2, s, &s->no_speech_thold_spin);
(double)s->cfg.whisper_no_speech_thold, 2,
"Probability threshold for considering a chunk as no-speech.",
s, &s->no_speech_thold_spin);
add_spin_row(grid, 2, 7, "greedy_best_of", 1.0, 10.0, 1.0,
(double)s->cfg.whisper_greedy_best_of, 0, s, &s->greedy_best_of_spin);
(double)s->cfg.whisper_greedy_best_of, 0,
"Number of candidates sampled in greedy fallback mode; best candidate is selected.",
s, &s->greedy_best_of_spin);
add_spin_row(grid, 0, 8, "beam_size", 1.0, 20.0, 1.0,
(double)s->cfg.whisper_beam_size, 0, s, &s->beam_size_spin);
(double)s->cfg.whisper_beam_size, 0,
"Beam width used by beam search strategy.",
s, &s->beam_size_spin);
add_spin_row(grid, 2, 8, "beam_patience", -1.0, 5.0, 0.01,
(double)s->cfg.whisper_beam_patience, 2, s, &s->beam_patience_spin);
(double)s->cfg.whisper_beam_patience, 2,
"Beam search patience (experimental; -1 uses library default behavior).",
s, &s->beam_patience_spin);
GtkWidget *help_frame = gtk_frame_new("Help");
gtk_box_pack_end(GTK_BOX(box), help_frame, FALSE, FALSE, 0);
GtkWidget *help_inner = gtk_box_new(GTK_ORIENTATION_VERTICAL, 4);
gtk_container_set_border_width(GTK_CONTAINER(help_inner), 8);
gtk_container_add(GTK_CONTAINER(help_frame), help_inner);
s->whisper_help_label = gtk_label_new("Hover any control to see what it does.");
gtk_label_set_xalign(GTK_LABEL(s->whisper_help_label), 0.0f);
gtk_label_set_line_wrap(GTK_LABEL(s->whisper_help_label), TRUE);
gtk_label_set_max_width_chars(GTK_LABEL(s->whisper_help_label), 90);
gtk_box_pack_start(GTK_BOX(help_inner), s->whisper_help_label, FALSE, FALSE, 0);
return scroll;
}
static void voice_cmd_refresh_store(gui_state_t *s) {
if (!s || !s->cmd_store) return;
gtk_list_store_clear(s->cmd_store);
for (int i = 0; i < s->voice_cmds.count; ++i) {
const voice_cmd_t *c = &s->voice_cmds.cmds[i];
const char *type_txt = (c->type == VOICE_CMD_TEXT) ? "Text" : "Key";
const char *action_txt = (c->type == VOICE_CMD_TEXT) ? c->text_value : c->display_key;
GtkTreeIter it;
gtk_list_store_append(s->cmd_store, &it);
gtk_list_store_set(s->cmd_store, &it,
CMD_COL_TRIGGER, c->trigger,
CMD_COL_TYPE, type_txt,
CMD_COL_ACTION, action_txt,
-1);
}
}
static int voice_cmd_load_or_seed(gui_state_t *s) {
if (!s) return -1;
if (voice_cmd_load(s->voice_cmd_path, &s->voice_cmds) == 0) {
return 0;
}
voice_cmd_free(&s->voice_cmds);
voice_cmd_init(&s->voice_cmds);
if (voice_cmd_seed_defaults(&s->voice_cmds) != 0) return -2;
if (voice_cmd_save(s->voice_cmd_path, &s->voice_cmds) != 0) return -3;
return 0;
}
static unsigned int voice_cmd_mods_from_gdk(guint state) {
unsigned int mods = 0;
if (state & GDK_SHIFT_MASK) mods |= VOICE_CMD_MOD_SHIFT;
if (state & GDK_CONTROL_MASK) mods |= VOICE_CMD_MOD_CTRL;
if (state & GDK_MOD1_MASK) mods |= VOICE_CMD_MOD_ALT;
if (state & GDK_SUPER_MASK) mods |= VOICE_CMD_MOD_SUPER;
return mods;
}
static gboolean on_window_key_press(GtkWidget *widget, GdkEventKey *event, gpointer user_data) {
(void)widget;
gui_state_t *s = (gui_state_t *)user_data;
if (!s || !event) return FALSE;
if (!s->cmd_capture_armed) return FALSE;
s->cmd_capture_keyval = (unsigned int)event->keyval;
s->cmd_capture_mods = voice_cmd_mods_from_gdk(event->state);
s->cmd_capture_armed = 0;
char label[128];
voice_cmd_format_key_combo(s->cmd_capture_keyval, s->cmd_capture_mods, label, sizeof(label));
gtk_label_set_text(GTK_LABEL(s->cmd_capture_label), label);
gtk_button_set_label(GTK_BUTTON(s->cmd_capture_btn), "Capture");
return TRUE;
}
static void on_cmd_capture_clicked(GtkButton *btn, gpointer user_data) {
gui_state_t *s = (gui_state_t *)user_data;
if (!s) return;
s->cmd_capture_armed = 1;
s->cmd_capture_keyval = 0;
s->cmd_capture_mods = 0;
gtk_label_set_text(GTK_LABEL(s->cmd_capture_label), "Press key combo now...");
gtk_button_set_label(btn, "Waiting...");
}
static void on_cmd_enable_toggled(GtkToggleButton *btn, gpointer user_data) {
gui_state_t *s = (gui_state_t *)user_data;
if (!s) return;
s->voice_cmds.enabled = gtk_toggle_button_get_active(btn) ? 1 : 0;
voice_cmd_save(s->voice_cmd_path, &s->voice_cmds);
}
static void on_cmd_mode_toggled(GtkToggleButton *btn, gpointer user_data) {
(void)btn;
gui_state_t *s = (gui_state_t *)user_data;
if (!s) return;
int text_mode = gtk_toggle_button_get_active(GTK_TOGGLE_BUTTON(s->cmd_mode_text_radio)) ? 1 : 0;
gtk_widget_set_sensitive(s->cmd_text_entry, text_mode ? TRUE : FALSE);
gtk_widget_set_sensitive(s->cmd_capture_btn, text_mode ? FALSE : TRUE);
}
static void on_cmd_add_clicked(GtkButton *btn, gpointer user_data) {
(void)btn;
gui_state_t *s = (gui_state_t *)user_data;
if (!s) return;
const char *trigger = gtk_entry_get_text(GTK_ENTRY(s->cmd_trigger_entry));
if (!trigger || trigger[0] == '\0') {
set_status(s, "Voice command trigger cannot be empty");
return;
}
voice_cmd_t c;
memset(&c, 0, sizeof(c));
snprintf(c.trigger, sizeof(c.trigger), "%s", trigger);
int text_mode = gtk_toggle_button_get_active(GTK_TOGGLE_BUTTON(s->cmd_mode_text_radio)) ? 1 : 0;
if (text_mode) {
c.type = VOICE_CMD_TEXT;
const char *v = gtk_entry_get_text(GTK_ENTRY(s->cmd_text_entry));
if (!v) v = "";
snprintf(c.text_value, sizeof(c.text_value), "%s", v);
} else {
c.type = VOICE_CMD_KEYCOMBO;
c.keyval = s->cmd_capture_keyval;
c.modifiers = s->cmd_capture_mods;
if (c.keyval == 0) {
set_status(s, "Capture a key combination first");
return;
}
voice_cmd_format_key_combo(c.keyval, c.modifiers, c.display_key, sizeof(c.display_key));
}
if (voice_cmd_add(&s->voice_cmds, &c) != 0) {
set_status(s, "Failed to add voice command");
return;
}
voice_cmd_save(s->voice_cmd_path, &s->voice_cmds);
voice_cmd_refresh_store(s);
gtk_entry_set_text(GTK_ENTRY(s->cmd_trigger_entry), "");
gtk_entry_set_text(GTK_ENTRY(s->cmd_text_entry), "");
s->cmd_capture_keyval = 0;
s->cmd_capture_mods = 0;
gtk_label_set_text(GTK_LABEL(s->cmd_capture_label), "No key captured");
set_status(s, "Voice command added");
}
static void on_cmd_delete_clicked(GtkButton *btn, gpointer user_data) {
(void)btn;
gui_state_t *s = (gui_state_t *)user_data;
if (!s || !s->cmd_tree) return;
GtkTreeSelection *sel = gtk_tree_view_get_selection(GTK_TREE_VIEW(s->cmd_tree));
GtkTreeModel *model = NULL;
GtkTreeIter it;
if (!gtk_tree_selection_get_selected(sel, &model, &it)) {
set_status(s, "Select a command to delete");
return;
}
GtkTreePath *path = gtk_tree_model_get_path(model, &it);
if (!path) return;
int *indices = gtk_tree_path_get_indices(path);
if (!indices) {
gtk_tree_path_free(path);
return;
}
int idx = indices[0];
gtk_tree_path_free(path);
if (voice_cmd_remove(&s->voice_cmds, idx) != 0) {
set_status(s, "Failed to delete command");
return;
}
voice_cmd_save(s->voice_cmd_path, &s->voice_cmds);
voice_cmd_refresh_store(s);
set_status(s, "Voice command deleted");
}
static GtkWidget *build_voice_commands_tab(gui_state_t *s) {
GtkWidget *outer = gtk_box_new(GTK_ORIENTATION_VERTICAL, 8);
gtk_container_set_border_width(GTK_CONTAINER(outer), 10);
s->cmd_enable_check = gtk_check_button_new_with_label("Enable voice commands");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->cmd_enable_check), s->voice_cmds.enabled ? TRUE : FALSE);
g_signal_connect(s->cmd_enable_check, "toggled", G_CALLBACK(on_cmd_enable_toggled), s);
gtk_box_pack_start(GTK_BOX(outer), s->cmd_enable_check, FALSE, FALSE, 0);
s->cmd_store = gtk_list_store_new(CMD_COL_COUNT, G_TYPE_STRING, G_TYPE_STRING, G_TYPE_STRING);
s->cmd_tree = gtk_tree_view_new_with_model(GTK_TREE_MODEL(s->cmd_store));
GtkCellRenderer *r = gtk_cell_renderer_text_new();
gtk_tree_view_insert_column_with_attributes(GTK_TREE_VIEW(s->cmd_tree), -1, "Trigger", r, "text", CMD_COL_TRIGGER, NULL);
gtk_tree_view_insert_column_with_attributes(GTK_TREE_VIEW(s->cmd_tree), -1, "Type", r, "text", CMD_COL_TYPE, NULL);
gtk_tree_view_insert_column_with_attributes(GTK_TREE_VIEW(s->cmd_tree), -1, "Action", r, "text", CMD_COL_ACTION, NULL);
GtkWidget *scroll = gtk_scrolled_window_new(NULL, NULL);
gtk_widget_set_size_request(scroll, -1, 180);
gtk_container_add(GTK_CONTAINER(scroll), s->cmd_tree);
gtk_box_pack_start(GTK_BOX(outer), scroll, TRUE, TRUE, 0);
GtkWidget *del_row = gtk_box_new(GTK_ORIENTATION_HORIZONTAL, 6);
s->cmd_delete_btn = gtk_button_new_with_label("Delete Selected");
g_signal_connect(s->cmd_delete_btn, "clicked", G_CALLBACK(on_cmd_delete_clicked), s);
gtk_box_pack_end(GTK_BOX(del_row), s->cmd_delete_btn, FALSE, FALSE, 0);
gtk_box_pack_start(GTK_BOX(outer), del_row, FALSE, FALSE, 0);
GtkWidget *frame = gtk_frame_new("Add New Command");
gtk_box_pack_start(GTK_BOX(outer), frame, FALSE, FALSE, 0);
GtkWidget *grid = gtk_grid_new();
gtk_container_set_border_width(GTK_CONTAINER(grid), 8);
gtk_grid_set_row_spacing(GTK_GRID(grid), 6);
gtk_grid_set_column_spacing(GTK_GRID(grid), 8);
gtk_container_add(GTK_CONTAINER(frame), grid);
gtk_grid_attach(GTK_GRID(grid), gtk_label_new("Trigger:"), 0, 0, 1, 1);
s->cmd_trigger_entry = gtk_entry_new();
gtk_grid_attach(GTK_GRID(grid), s->cmd_trigger_entry, 1, 0, 2, 1);
s->cmd_mode_text_radio = gtk_radio_button_new_with_label(NULL, "Text replacement");
s->cmd_mode_key_radio = gtk_radio_button_new_with_label_from_widget(GTK_RADIO_BUTTON(s->cmd_mode_text_radio), "Key combination");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->cmd_mode_text_radio), TRUE);
g_signal_connect(s->cmd_mode_text_radio, "toggled", G_CALLBACK(on_cmd_mode_toggled), s);
g_signal_connect(s->cmd_mode_key_radio, "toggled", G_CALLBACK(on_cmd_mode_toggled), s);
gtk_grid_attach(GTK_GRID(grid), s->cmd_mode_text_radio, 1, 1, 1, 1);
gtk_grid_attach(GTK_GRID(grid), s->cmd_mode_key_radio, 2, 1, 1, 1);
gtk_grid_attach(GTK_GRID(grid), gtk_label_new("Text:"), 0, 2, 1, 1);
s->cmd_text_entry = gtk_entry_new();
gtk_grid_attach(GTK_GRID(grid), s->cmd_text_entry, 1, 2, 2, 1);
gtk_grid_attach(GTK_GRID(grid), gtk_label_new("Key:"), 0, 3, 1, 1);
s->cmd_capture_label = gtk_label_new("No key captured");
gtk_label_set_xalign(GTK_LABEL(s->cmd_capture_label), 0.0f);
gtk_grid_attach(GTK_GRID(grid), s->cmd_capture_label, 1, 3, 1, 1);
s->cmd_capture_btn = gtk_button_new_with_label("Capture");
g_signal_connect(s->cmd_capture_btn, "clicked", G_CALLBACK(on_cmd_capture_clicked), s);
gtk_grid_attach(GTK_GRID(grid), s->cmd_capture_btn, 2, 3, 1, 1);
s->cmd_add_btn = gtk_button_new_with_label("Add Command");
g_signal_connect(s->cmd_add_btn, "clicked", G_CALLBACK(on_cmd_add_clicked), s);
gtk_grid_attach(GTK_GRID(grid), s->cmd_add_btn, 2, 4, 1, 1);
on_cmd_mode_toggled(NULL, s);
voice_cmd_refresh_store(s);
return outer;
}
static int cmp_str_ptr(const void *a, const void *b) {
const char *sa = *(const char * const *)a;
const char *sb = *(const char * const *)b;
@@ -1029,6 +1413,11 @@ static int run_startup_dialog(gui_state_t *s, GtkWindow *parent) {
static void on_app_activate(GtkApplication *app, gpointer user_data) {
gui_state_t *s = (gui_state_t *)user_data;
if (s && s->window) {
gtk_window_present(GTK_WINDOW(s->window));
return;
}
if (!s->runtime_ready) {
if (run_startup_dialog(s, NULL) != 0) {
g_application_quit(G_APPLICATION(app));
@@ -1074,6 +1463,7 @@ static void on_app_activate(GtkApplication *app, gpointer user_data) {
GtkWidget *outer = gtk_box_new(GTK_ORIENTATION_VERTICAL, 8);
gtk_container_set_border_width(GTK_CONTAINER(outer), 12);
gtk_container_add(GTK_CONTAINER(s->window), outer);
g_signal_connect(s->window, "key-press-event", G_CALLBACK(on_window_key_press), s);
s->status_label = gtk_label_new("Idle");
gtk_label_set_xalign(GTK_LABEL(s->status_label), 0.0f);
@@ -1200,6 +1590,8 @@ static void on_app_activate(GtkApplication *app, gpointer user_data) {
gtk_notebook_append_page(GTK_NOTEBOOK(notebook), main_page, gtk_label_new("Main"));
GtkWidget *whisper_page = build_whisper_settings_tab(s);
gtk_notebook_append_page(GTK_NOTEBOOK(notebook), whisper_page, gtk_label_new("Whisper"));
GtkWidget *commands_page = build_voice_commands_tab(s);
gtk_notebook_append_page(GTK_NOTEBOOK(notebook), commands_page, gtk_label_new("Commands"));
gtk_widget_show_all(s->window);
@@ -1237,6 +1629,7 @@ int main(int argc, char **argv) {
memset(&st, 0, sizeof(st));
config_set_defaults(&st.cfg);
voice_cmd_init(&st.voice_cmds);
const char *cfg_path = arg_value(argc, argv, "--config");
if (!cfg_path) cfg_path = "./config.ini";
@@ -1245,6 +1638,13 @@ int main(int argc, char **argv) {
fprintf(stderr, "warning: failed to load %s, using defaults\n", cfg_path);
}
snprintf(st.voice_cmd_path, sizeof(st.voice_cmd_path), "./voice_commands.ini");
if (voice_cmd_load_or_seed(&st) != 0) {
fprintf(stderr, "warning: failed to load voice commands from %s\n", st.voice_cmd_path);
voice_cmd_free(&st.voice_cmds);
voice_cmd_init(&st.voice_cmds);
}
st.use_gpu = st.cfg.use_gpu ? 1 : 0;
if (arg_flag(argc, argv, "--gpu")) st.use_gpu = 1;
if (arg_flag(argc, argv, "--cpu")) st.use_gpu = 0;
@@ -1287,6 +1687,7 @@ int main(int argc, char **argv) {
typer_cleanup();
transcribe_cleanup();
audio_cleanup();
voice_cmd_free(&st.voice_cmds);
return rc;
}

47
src/typer.c
View File

@@ -182,6 +182,53 @@ int typer_type_text(const char *text) {
return failures == 0 ? 0 : -1;
}
int typer_send_keycombo(unsigned int keyval, unsigned int modifiers) {
if (!g_dpy || keyval == 0) return -1;
KeyCode kc = XKeysymToKeycode(g_dpy, (KeySym)keyval);
if (!kc) return -1;
KeyCode ctrl_kc = XKeysymToKeycode(g_dpy, XK_Control_L);
KeyCode shift_kc = XKeysymToKeycode(g_dpy, XK_Shift_L);
KeyCode alt_kc = XKeysymToKeycode(g_dpy, XK_Alt_L);
KeyCode super_kc = XKeysymToKeycode(g_dpy, XK_Super_L);
if ((modifiers & (1u << 1)) && ctrl_kc) {
if (!XTestFakeKeyEvent(g_dpy, ctrl_kc, True, CurrentTime)) return -1;
}
if ((modifiers & (1u << 0)) && shift_kc) {
if (!XTestFakeKeyEvent(g_dpy, shift_kc, True, CurrentTime)) return -1;
}
if ((modifiers & (1u << 2)) && alt_kc) {
if (!XTestFakeKeyEvent(g_dpy, alt_kc, True, CurrentTime)) return -1;
}
if ((modifiers & (1u << 3)) && super_kc) {
if (!XTestFakeKeyEvent(g_dpy, super_kc, True, CurrentTime)) return -1;
}
if (!XTestFakeKeyEvent(g_dpy, kc, True, CurrentTime)) return -1;
XSync(g_dpy, False);
sleep_us(g_key_hold_us);
if (!XTestFakeKeyEvent(g_dpy, kc, False, CurrentTime)) return -1;
if ((modifiers & (1u << 3)) && super_kc) {
if (!XTestFakeKeyEvent(g_dpy, super_kc, False, CurrentTime)) return -1;
}
if ((modifiers & (1u << 2)) && alt_kc) {
if (!XTestFakeKeyEvent(g_dpy, alt_kc, False, CurrentTime)) return -1;
}
if ((modifiers & (1u << 0)) && shift_kc) {
if (!XTestFakeKeyEvent(g_dpy, shift_kc, False, CurrentTime)) return -1;
}
if ((modifiers & (1u << 1)) && ctrl_kc) {
if (!XTestFakeKeyEvent(g_dpy, ctrl_kc, False, CurrentTime)) return -1;
}
XSync(g_dpy, False);
sleep_us(g_delay_us);
return 0;
}
void typer_cleanup(void) {
g_target_window = None;
if (g_dpy) {

1
src/typer.h
View File

@@ -4,6 +4,7 @@
int typer_init(int type_delay_us);
int typer_capture_focus(void);
int typer_type_text(const char *text);
int typer_send_keycombo(unsigned int keyval, unsigned int modifiers);
void typer_cleanup(void);
#endif

379
src/voice_cmd.c Normal file
View File

@@ -0,0 +1,379 @@
#include "voice_cmd.h"
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
static void trim(char *s) {
if (!s) return;
size_t len = strlen(s);
while (len > 0 && isspace((unsigned char)s[len - 1])) {
s[--len] = '\0';
}
char *p = s;
while (*p && isspace((unsigned char)*p)) p++;
if (p != s) {
memmove(s, p, strlen(p) + 1);
}
}
static int str_ieq_n(const char *a, const char *b, size_t n) {
for (size_t i = 0; i < n; ++i) {
unsigned char ca = (unsigned char)a[i];
unsigned char cb = (unsigned char)b[i];
if (tolower(ca) != tolower(cb)) return 0;
}
return 1;
}
static int is_word_char(unsigned char c) {
return isalnum(c) || c == '_';
}
static int matches_whole_word_ci(const char *text, size_t pos, const char *trig, size_t trig_len) {
if (!text || !trig || trig_len == 0) return 0;
if (!str_ieq_n(text + pos, trig, trig_len)) return 0;
unsigned char left = (pos == 0) ? 0 : (unsigned char)text[pos - 1];
unsigned char right = (unsigned char)text[pos + trig_len];
if (pos > 0 && is_word_char(left)) return 0;
if (right != '\0' && is_word_char(right)) return 0;
return 1;
}
static void unescape_copy(const char *src, char *dst, size_t dst_sz) {
if (!dst || dst_sz == 0) return;
dst[0] = '\0';
if (!src) return;
size_t w = 0;
for (size_t i = 0; src[i] != '\0' && w + 1 < dst_sz; ++i) {
if (src[i] == '\\' && src[i + 1] != '\0') {
i++;
char c = src[i];
if (c == 'n') dst[w++] = '\n';
else if (c == 't') dst[w++] = '\t';
else if (c == 'r') dst[w++] = '\r';
else dst[w++] = c;
} else {
dst[w++] = src[i];
}
}
dst[w] = '\0';
}
static void escape_copy(const char *src, char *dst, size_t dst_sz) {
if (!dst || dst_sz == 0) return;
dst[0] = '\0';
if (!src) return;
size_t w = 0;
for (size_t i = 0; src[i] != '\0' && w + 1 < dst_sz; ++i) {
char c = src[i];
const char *rep = NULL;
if (c == '\n') rep = "\\n";
else if (c == '\t') rep = "\\t";
else if (c == '\r') rep = "\\r";
else if (c == '\\') rep = "\\\\";
if (rep) {
for (size_t j = 0; rep[j] != '\0' && w + 1 < dst_sz; ++j) {
dst[w++] = rep[j];
}
} else {
dst[w++] = c;
}
}
dst[w] = '\0';
}
void voice_cmd_format_key_combo(unsigned int keyval, unsigned int modifiers, char *out, size_t out_sz) {
if (!out || out_sz == 0) return;
char keyname[64];
if (keyval >= 32 && keyval < 127) {
snprintf(keyname, sizeof(keyname), "%c", (char)keyval);
} else {
switch (keyval) {
case 65293: snprintf(keyname, sizeof(keyname), "Return"); break;
case 65289: snprintf(keyname, sizeof(keyname), "Tab"); break;
case 65307: snprintf(keyname, sizeof(keyname), "Escape"); break;
case 65535: snprintf(keyname, sizeof(keyname), "Delete"); break;
case 65288: snprintf(keyname, sizeof(keyname), "BackSpace"); break;
default: snprintf(keyname, sizeof(keyname), "Key%u", keyval); break;
}
}
char pref[96];
pref[0] = '\0';
if (modifiers & VOICE_CMD_MOD_CTRL) strncat(pref, "Ctrl+", sizeof(pref) - strlen(pref) - 1);
if (modifiers & VOICE_CMD_MOD_SHIFT) strncat(pref, "Shift+", sizeof(pref) - strlen(pref) - 1);
if (modifiers & VOICE_CMD_MOD_ALT) strncat(pref, "Alt+", sizeof(pref) - strlen(pref) - 1);
if (modifiers & VOICE_CMD_MOD_SUPER) strncat(pref, "Super+", sizeof(pref) - strlen(pref) - 1);
snprintf(out, out_sz, "%s%s", pref, keyname);
}
void voice_cmd_init(voice_cmd_list_t *list) {
if (!list) return;
memset(list, 0, sizeof(*list));
list->enabled = 1;
}
void voice_cmd_free(voice_cmd_list_t *list) {
if (!list) return;
free(list->cmds);
memset(list, 0, sizeof(*list));
list->enabled = 1;
}
static int ensure_capacity(voice_cmd_list_t *list, int min_cap) {
if (!list) return -1;
if (list->capacity >= min_cap) return 0;
int next = list->capacity > 0 ? list->capacity * 2 : 16;
if (next < min_cap) next = min_cap;
void *p = realloc(list->cmds, (size_t)next * sizeof(voice_cmd_t));
if (!p) return -1;
list->cmds = (voice_cmd_t *)p;
list->capacity = next;
return 0;
}
int voice_cmd_add(voice_cmd_list_t *list, const voice_cmd_t *cmd) {
if (!list || !cmd) return -1;
if (cmd->trigger[0] == '\0') return -2;
if (ensure_capacity(list, list->count + 1) != 0) return -3;
list->cmds[list->count++] = *cmd;
return 0;
}
int voice_cmd_remove(voice_cmd_list_t *list, int index) {
if (!list) return -1;
if (index < 0 || index >= list->count) return -2;
for (int i = index; i + 1 < list->count; ++i) {
list->cmds[i] = list->cmds[i + 1];
}
list->count--;
return 0;
}
int voice_cmd_seed_defaults(voice_cmd_list_t *list) {
if (!list) return -1;
static const struct {
const char *trigger;
const char *text;
} defs[] = {
{"period", "."},
{"question mark", "?"},
{"exclamation mark", "!"},
{"comma", ","},
{"colon", ":"},
{"semicolon", ";"},
{"new line", "\n"},
{"new paragraph", "\n\n"},
{"tab", "\t"},
{"open paren", "("},
{"close paren", ")"},
{"open bracket", "["},
{"close bracket", "]"},
{"open brace", "{"},
{"close brace", "}"},
};
for (size_t i = 0; i < sizeof(defs) / sizeof(defs[0]); ++i) {
voice_cmd_t c;
memset(&c, 0, sizeof(c));
snprintf(c.trigger, sizeof(c.trigger), "%s", defs[i].trigger);
c.type = VOICE_CMD_TEXT;
snprintf(c.text_value, sizeof(c.text_value), "%s", defs[i].text);
if (voice_cmd_add(list, &c) != 0) return -2;
}
voice_cmd_t over;
memset(&over, 0, sizeof(over));
snprintf(over.trigger, sizeof(over.trigger), "over");
over.type = VOICE_CMD_KEYCOMBO;
over.keyval = 65293;
over.modifiers = 0;
voice_cmd_format_key_combo(over.keyval, over.modifiers, over.display_key, sizeof(over.display_key));
if (voice_cmd_add(list, &over) != 0) return -3;
voice_cmd_t eoc;
memset(&eoc, 0, sizeof(eoc));
snprintf(eoc.trigger, sizeof(eoc.trigger), "end of command");
eoc.type = VOICE_CMD_KEYCOMBO;
eoc.keyval = 65293;
eoc.modifiers = VOICE_CMD_MOD_CTRL | VOICE_CMD_MOD_SHIFT;
voice_cmd_format_key_combo(eoc.keyval, eoc.modifiers, eoc.display_key, sizeof(eoc.display_key));
if (voice_cmd_add(list, &eoc) != 0) return -4;
return 0;
}
int voice_cmd_load(const char *path, voice_cmd_list_t *list) {
if (!path || !list) return -1;
FILE *f = fopen(path, "r");
if (!f) return -2;
voice_cmd_free(list);
voice_cmd_init(list);
char line[1024];
while (fgets(line, sizeof(line), f)) {
trim(line);
if (line[0] == '\0' || line[0] == '#') continue;
if (strncmp(line, "@enabled=", 9) == 0) {
list->enabled = (atoi(line + 9) != 0) ? 1 : 0;
continue;
}
char *p1 = strchr(line, '|');
if (!p1) continue;
*p1 = '\0';
char *p2 = strchr(p1 + 1, '|');
if (!p2) continue;
*p2 = '\0';
char *trigger = line;
char *type = p1 + 1;
char *value = p2 + 1;
trim(trigger);
trim(type);
trim(value);
if (trigger[0] == '\0') continue;
voice_cmd_t c;
memset(&c, 0, sizeof(c));
snprintf(c.trigger, sizeof(c.trigger), "%.127s", trigger);
if (strcmp(type, "text") == 0) {
c.type = VOICE_CMD_TEXT;
unescape_copy(value, c.text_value, sizeof(c.text_value));
} else if (strcmp(type, "key") == 0) {
c.type = VOICE_CMD_KEYCOMBO;
unsigned int k = 0, m = 0;
if (sscanf(value, "%u:%u", &k, &m) != 2) continue;
c.keyval = k;
c.modifiers = m;
voice_cmd_format_key_combo(c.keyval, c.modifiers, c.display_key, sizeof(c.display_key));
} else {
continue;
}
if (voice_cmd_add(list, &c) != 0) {
fclose(f);
return -3;
}
}
fclose(f);
return 0;
}
int voice_cmd_save(const char *path, const voice_cmd_list_t *list) {
if (!path || !list) return -1;
FILE *f = fopen(path, "w");
if (!f) return -2;
fprintf(f, "# voice command rules\n");
fprintf(f, "# @enabled=1 or @enabled=0 controls global enable state\n");
fprintf(f, "# trigger|type|value\n");
fprintf(f, "# type=text => value supports escaped \\n, \\t, \\\\ \n");
fprintf(f, "# type=key => value format keyval:modifiers\n\n");
fprintf(f, "@enabled=%d\n\n", list->enabled ? 1 : 0);
for (int i = 0; i < list->count; ++i) {
const voice_cmd_t *c = &list->cmds[i];
if (c->type == VOICE_CMD_TEXT) {
char esc[512];
escape_copy(c->text_value, esc, sizeof(esc));
fprintf(f, "%s|text|%s\n", c->trigger, esc);
} else {
fprintf(f, "%s|key|%u:%u\n", c->trigger, c->keyval, c->modifiers);
}
}
fclose(f);
return 0;
}
char *voice_cmd_apply(const char *input,
const voice_cmd_list_t *list,
voice_cmd_key_callback_t key_cb,
void *user_data) {
if (!input) return NULL;
size_t in_len = strlen(input);
size_t cap = in_len * 2 + 64;
char *out = (char *)malloc(cap);
if (!out) return NULL;
size_t w = 0;
size_t i = 0;
while (i < in_len) {
int best_idx = -1;
size_t best_len = 0;
if (list && list->enabled) {
for (int ci = 0; ci < list->count; ++ci) {
const voice_cmd_t *c = &list->cmds[ci];
size_t tl = strlen(c->trigger);
if (tl == 0 || i + tl > in_len) continue;
if (!matches_whole_word_ci(input, i, c->trigger, tl)) continue;
if (tl > best_len) {
best_len = tl;
best_idx = ci;
}
}
}
if (best_idx >= 0) {
const voice_cmd_t *c = &list->cmds[best_idx];
if (c->type == VOICE_CMD_TEXT) {
size_t repl = strlen(c->text_value);
if (w + repl + 1 > cap) {
size_t next = (cap * 2) + repl + 32;
char *np = (char *)realloc(out, next);
if (!np) {
free(out);
return NULL;
}
out = np;
cap = next;
}
memcpy(out + w, c->text_value, repl);
w += repl;
} else if (key_cb) {
key_cb(c->keyval, c->modifiers, user_data);
}
i += best_len;
continue;
}
if (w + 2 > cap) {
size_t next = cap * 2;
char *np = (char *)realloc(out, next);
if (!np) {
free(out);
return NULL;
}
out = np;
cap = next;
}
out[w++] = input[i++];
}
out[w] = '\0';
return out;
}

61
src/voice_cmd.h Normal file
View File

@@ -0,0 +1,61 @@
#ifndef VOICE_VOICE_CMD_H
#define VOICE_VOICE_CMD_H
#include <stddef.h>
#ifdef __cplusplus
extern "C" {
#endif
typedef enum {
VOICE_CMD_TEXT = 0,
VOICE_CMD_KEYCOMBO = 1,
} voice_cmd_type_t;
enum {
VOICE_CMD_MOD_SHIFT = 1 << 0,
VOICE_CMD_MOD_CTRL = 1 << 1,
VOICE_CMD_MOD_ALT = 1 << 2,
VOICE_CMD_MOD_SUPER = 1 << 3,
};
typedef struct {
char trigger[128];
voice_cmd_type_t type;
char text_value[256];
unsigned int keyval;
unsigned int modifiers;
char display_key[128];
} voice_cmd_t;
typedef struct {
voice_cmd_t *cmds;
int count;
int capacity;
int enabled;
} voice_cmd_list_t;
typedef int (*voice_cmd_key_callback_t)(unsigned int keyval, unsigned int modifiers, void *user_data);
void voice_cmd_init(voice_cmd_list_t *list);
void voice_cmd_free(voice_cmd_list_t *list);
int voice_cmd_add(voice_cmd_list_t *list, const voice_cmd_t *cmd);
int voice_cmd_remove(voice_cmd_list_t *list, int index);
int voice_cmd_load(const char *path, voice_cmd_list_t *list);
int voice_cmd_save(const char *path, const voice_cmd_list_t *list);
int voice_cmd_seed_defaults(voice_cmd_list_t *list);
char *voice_cmd_apply(const char *input,
const voice_cmd_list_t *list,
voice_cmd_key_callback_t key_cb,
void *user_data);
void voice_cmd_format_key_combo(unsigned int keyval,
unsigned int modifiers,
char *out,
size_t out_sz);
#ifdef __cplusplus
}
#endif
#endif

26
voice_commands.ini Normal file
View File

@@ -0,0 +1,26 @@
# voice command rules
# @enabled=1 or @enabled=0 controls global enable state
# trigger|type|value
# type=text => value supports escaped \n, \t, \\
# type=key => value format keyval:modifiers
@enabled=1
period|text|.
question mark|text|?
exclamation mark|text|!
comma|text|,
colon|text|:
semicolon|text|;
new line|text|\n
new paragraph|text|\n\n
tab|text|\t
open paren|text|(
close paren|text|)
open bracket|text|[
close bracket|text|]
open brace|text|{
close brace|text|}
over|key|65293:0
end of command|key|65293:3
over.|text|Return