SXSEditor

ONNX Models ONNX 模型

SXSEditor ships with 9 required ONNX models plus an optional SiFiGAN vocoder. All models are exported from the PyTorch SoulX-Singer checkpoint and converted to ONNX (opset 20) for DirectML / WebNN / CPU execution. Model files live in onnx_models/<precision>/ and are downloaded on first launch from ModelScope.

SXSEditor 内置 9 个必需的 ONNX 模型，外加一个 可选的 SiFiGAN 声码器。所有模型均由 PyTorch 版 SoulX-Singer 检查点导出，并转换为 ONNX（opset 20）以支持 DirectML / WebNN / CPU 执行。模型文件位于 onnx_models/<precision>/，首次启动时从 ModelScope 下载。

ℹ️

The authoritative list of required model files is ONNX_MODEL_FILES in src/inference/pipeline/constants.js. SiFiGAN files are listed separately in SIFIGAN_MODEL_FILES. The download manager (src/modelManager.js) maps each precision to a ModelScope repository.

必需模型文件的权威清单是 src/inference/pipeline/constants.js 中的 ONNX_MODEL_FILES。SiFiGAN 文件单独列在 SIFIGAN_MODEL_FILES 中。下载管理器（src/modelManager.js）将每种精度映射到一个 ModelScope 仓库。

The 9 Required Models 9 个必需模型

File文件 Group分组 Role作用 FP16 SizeFP16 大小
note_text_encoder.onnx svs_encodersvs_encoder Phoneme id → text embedding (EMBED_DIM=512)音素 id → 文本嵌入（EMBED_DIM=512） 2.93 MB
note_pitch_encoder.onnx svs_encodersvs_encoder MIDI pitch id → pitch embeddingMIDI 音高 id → 音高嵌入 0.13 MB
note_type_encoder.onnx svs_encodersvs_encoder Note type (rest/vocal/slur) → embedding音符类型（休止/人声/连音）→ 嵌入 0.13 MB
f0_encoder.onnx svs_encodersvs_encoder Quantized F0 id → F0 embedding (F0_BIN=361)量化 F0 id → F0 嵌入（F0_BIN=361） 0.13 MB
preflow.onnx svs_encodersvs_encoder ConvNeXtV2 pre-flow over summed embeddings对四嵌入之和做 ConvNeXtV2 预流 8.2 MB
cond_emb.onnx svs_encodersvs_encoder preflow output → diffusion condition (COND_DIM=1024)preflow 输出 → 扩散条件（COND_DIM=1024） 0.51 MB
diff_step_dml.onnx svs_diffusionsvs_diffusion Single diffusion denoising step (flow matching)单步扩散去噪（flow matching） 846 MB
vocoder_dml.onnx svs_diffusionsvs_diffusion Default Vocos vocoder: mel (128D) → 24kHz PCM默认 Vocos 声码器：mel（128D）→ 24kHz PCM 495 MB
mel_transform.onnx svs_auxiliarysvs_auxiliary STFT → mel for reference audio (Cooley-Tukey FFT)参考音频 STFT → mel（Cooley-Tukey FFT） 0.25 MB

Model Groups & Device Placement 模型分组与设备分配

Models are organized into groups (defined in MODEL_GROUPS). buildModelDeviceMapping in modelLoader.js assigns each group to a device based on total group size and available hardware:

模型被组织为若干 分组（定义于 MODEL_GROUPS）。modelLoader.js 中的 buildModelDeviceMapping 根据分组总大小与可用硬件为每个分组分配设备：

Group分组 Models模型 Total FP16 SizeFP16 总大小 Placement Rule分配规则
svs_diffusion diff_step, vocoderdiff_step, vocoder ~1.3 GB约 1.3 GB >100 MB → discrete GPU (DirectML)>100 MB → 独显（DirectML）
svs_encoder 6 encoders6 个编码器 ~12 MB约 12 MB 10–100 MB → GPU preferred, else NPU/CPU10–100 MB → 优先 GPU，否则 NPU/CPU
svs_auxiliary mel_transformmel_transform 0.25 MB0.25 MB <10 MB → NPU preferred (frees GPU VRAM), else CPU<10 MB → 优先 NPU（释放 GPU 显存），否则 CPU
fcpe FCPE pitch detectorFCPE 音高检测器 small较小 Auto-downloaded from ModelScope. Default MIDI extraction tool (replaces Basic Pitch).从 ModelScope 自动下载。默认 MIDI 提取工具（替代 Basic Pitch）。
rosvot ROSVOT voice onset detectorROSVOT 声音起始检测器 small较小 Used for audio segmentation and note onset detection.用于音频分段与音符起始检测。

Device selection priority (in selectBestDevice): discrete-gpu > npu > integrated-gpu > cpu, weighted by VRAM within each tier.

设备选择优先级（selectBestDevice）：独显 > NPU > 核显 > CPU，同级内按显存加权。

src/modelRegistry.js now includes FCPE and ROSVOT model group definitions alongside the existing SVS pipeline, RMVPE, and Basic Pitch groups. Each group declares its model files, ModelScope repository, and download metadata, enabling the download manager to handle all model types uniformly.

src/modelRegistry.js 现包含 FCPE 与 ROSVOT 模型分组定义，与现有的 SVS 管线、RMVPE 和 Basic Pitch 分组并列。每个分组声明其模型文件、ModelScope 仓库与下载元数据，使下载管理器能统一处理所有模型类型。

Precision Variants 精度变体

SXSEditor supports 4 model precisions, each in its own subdirectory of onnx_models/:

SXSEditor 支持 4 种模型精度，每种位于 onnx_models/ 下的独立子目录：

Precision精度 Subdirectory子目录 ModelScope RepoModelScope 仓库 Notes说明
fp32 onnx_models/ (root)onnx_models/（根目录） syxppp/SoulX-Singer-onnx-directml Highest quality, ~3 GB disk最高质量，约 3 GB 磁盘
fp16 onnx_models/fp16/ syxppp/SoulX-Singer-onnx-directml-fp16 Recommended; ~1.5 GB disk. Uses Float16 patch.推荐；约 1.5 GB 磁盘。使用 Float16 补丁。
int8 onnx_models/int8/ syxppp/SoulX-Singer-onnx-directml-int8 Quantized; ~1 GB. For low-VRAM GPUs.量化；约 1 GB。适用于低显存 GPU。
int8-npu onnx_models/int8-npu/ (NPU-optimized static-shape export)（NPU 优化静态形状导出） Static shapes padded to NPU_STATIC_SEQ_LEN=2048静态形状，填充到 NPU_STATIC_SEQ_LEN=2048

The set PRECISION_SUBDIR_PRECESIONS = {'int8', 'fp16', 'int8-npu'} determines which precisions live in a named subdirectory; fp32 lives at the onnx_models/ root. Different precisions can coexist — switching in Settings just changes which directory the pipeline reads from.

集合 PRECISION_SUBDIR_PRECESIONS = {'int8', 'fp16', 'int8-npu'} 决定哪些精度位于命名子目录；fp32 位于 onnx_models/ 根目录。不同精度可共存——在设置中切换只会改变管线读取的目录。

⚠️

W16A32 fallback: if the FP16 diff_step_dml.onnx fails to load (some GPUs have incomplete FP16 kernel support), the pipeline automatically reloads it from the FP32 base directory while keeping all other models in FP16. This combines FP16 memory savings with FP32 numerical stability for the diffusion step. Watch the console for "FP16 diff_step failed, falling back to FP32 (W16A32)".

W16A32 回退：若 FP16 的 diff_step_dml.onnx 加载失败（部分 GPU 的 FP16 算子支持不完整），管线会自动从 FP32 基础目录重新加载它，其余模型保持 FP16。这结合了 FP16 的显存节省与扩散步的 FP32 数值稳定性。注意控制台输出 "FP16 diff_step failed, falling back to FP32 (W16A32)"。

ℹ️

QDIT int8 diff_step: QDIT-quantized int8 diff_step models use a new ONNX signature with x (audio latent), diffusion_step (scalar), and x_mask (bool) inputs. Legacy int8 models (lacking the diffusion_step input) are auto-detected at load time, and a user prompt suggests switching to the QDIT version for better quality.

QDIT int8 diff_step：QDIT 量化的 int8 diff_step 模型采用新的 ONNX 签名，包含 x（音频隐变量）、diffusion_step（标量）与 x_mask（布尔）输入。加载时自动检测旧版 int8 模型（缺少 diffusion_step 输入），并提示用户切换到 QDIT 版本以获得更好质量。

SiFiGAN Vocoder (Optional) SiFiGAN 声码器（可选）

SiFiGAN (ICASSP 2023) is an alternative vocoder that can be swapped in at runtime via pipeline.swapVocoder('sifigan'). It is a dual-input model (mel + f0):

SiFiGAN（ICASSP 2023）是可在运行时通过 pipeline.swapVocoder('sifigan') 切换的替代声码器。它是双输入模型（mel + f0）：

File文件 Size大小 Notes说明
sifigan_vocoder_dml_fp16.onnx 23.1 MB FP16 quantized variant (preferred)FP16 量化变体（首选）
sifigan_vocoder_dml.onnx 611 MB FP32 DML-optimized variant (fallback)FP32 DML 优化变体（回退）
sifigan_stats.joblib small较小 Input feature normalization statistics输入特征归一化统计

The FP16 variant is generated locally from the FP32 model and is not on ModelScope — only the FP32 variant and stats file are downloaded. SiFiGAN files live at the onnx_models/ root (not in precision subdirectories).

FP16 变体由 FP32 模型在本地生成，不在 ModelScope 上——仅下载 FP32 变体与统计文件。SiFiGAN 文件位于 onnx_models/ 根目录（不在精度子目录中）。

💡

Frame-rate mismatch: SiFiGAN expects 200 Hz mel (SIFIGAN_HOP_SIZE=120), but the SVS pipeline produces 50 Hz mel (HOP_SIZE=480). runVocoderChunked performs 4× nearest-neighbor upsampling on both mel and f0 in the time dimension before feeding SiFiGAN, so that effectiveTotalFrames × 120 == totalFrames × 480 yields the correct output audio length.

帧率不匹配：SiFiGAN 期望 200Hz mel（SIFIGAN_HOP_SIZE=120），但 SVS 管线产出 50Hz mel（HOP_SIZE=480）。runVocoderChunked 在送入 SiFiGAN 前对 mel 与 f0 在时间维度做 4× 最近邻上采样，使 effectiveTotalFrames × 120 == totalFrames × 480，输出音频长度正确。

Execution Providers 执行提供者

Models run on one of three execution providers, selected at session-creation time by createSessionWithValidation in modelLoader.js:

模型运行在三种执行提供者之一上，由 modelLoader.js 的 createSessionWithValidation 在创建会话时选择：

Provider提供者 Runtime运行时 Process进程 Used For用途
DirectMLDirectML onnxruntime-node Main主进程 Large models (diff_step, vocoder) on discrete/integrated GPU独显/核显上的大模型（diff_step、vocoder）
WebNNWebNN onnxruntime-web Renderer渲染进程 Small models on NPU (via WebNNSessionProxy IPC)NPU 上的小模型（通过 WebNNSessionProxy IPC）
CPUCPU onnxruntime-node Main主进程 Fallback when no GPU/NPU, or session validation fails无 GPU/NPU 或会话校验失败时回退

createSessionWithValidation creates a DML session first (with enableMemPattern: false and executionMode: 'sequential' to avoid DML over-allocation), runs a dummy inference against the model's expected input shape to verify it actually executes, and falls back to CPU on any failure. NPU static-shape models skip DML validation and load directly on CPU (they are consumed by the WebNN proxy, not DML).

createSessionWithValidation 优先创建 DML 会话（设置 enableMemPattern: false 与 executionMode: 'sequential' 以避免 DML 过度分配），用模型期望的输入形状跑一次 dummy 推理验证其确实可执行，任何失败都回退到 CPU。NPU 静态形状模型跳过 DML 校验，直接在 CPU 上加载（它们由 WebNN 代理消费，而非 DML）。

WebNN Session Proxy WebNN 会话代理

When inferenceProvider='ortweb' and NPU/GPU is available, WebNNSessionProxy in modelLoader.js forwards session.run() calls to the renderer process via IPC. The main window renderer hosts onnxruntime-web and registers the WebNN execution provider. This lets small models (mel_transform, encoders) run on the NPU while heavy models stay on DirectML in the main process.

当 inferenceProvider='ortweb' 且 NPU/GPU 可用时，modelLoader.js 中的 WebNNSessionProxy 通过 IPC 将 session.run() 调用转发到渲染进程。主窗口渲染进程承载 onnxruntime-web 并注册 WebNN 执行提供者。这让小模型（mel_transform、编码器）在 NPU 上运行，而重量级模型仍留在主进程的 DirectML 上。

Model Download (First Launch) 模型下载（首次启动）

On first launch, SXSEditor detects missing model files and opens the Model Download window. The download is managed by src/main/modelDownload.js and src/modelManager.js:

首次启动时，SXSEditor 检测到模型文件缺失，会打开模型下载窗口。下载由 src/main/modelDownload.js 与 src/modelManager.js 管理：

ℹ️

Japanese (JP) models live in onnx_models/<precision>/JP/ and are downloaded from a separate LoRA repository (syxppp/SoulX-Singer-onnx-fp16-lora-jp). They are optional — Japanese inference is unavailable until the JP directory is populated. pipeline.swapLanguageModels('ja') hot-swaps them at runtime.

日语（JP）模型位于 onnx_models/<precision>/JP/，从独立的 LoRA 仓库（syxppp/SoulX-Singer-onnx-fp16-lora-jp）下载。它们是可选的——JP 目录未填充前日语推理不可用。pipeline.swapLanguageModels('ja') 可在运行时热切换。

ONNX Opset & DML Compatibility ONNX Opset 与 DML 兼容性

All models are exported with opset 20 (the maximum supported by the DirectML EP). Key DML compatibility constraints handled during export / post-processing:

所有模型以 opset 20 导出（DirectML EP 支持的最大版本）。导出 / 后处理阶段处理的 DML 兼容性约束：

Switching Precision at Runtime 运行时切换精度

Precision is controlled by settings.modelPrecision. Users switch it in Settings → Inference; the change triggers a full pipeline reload. Developers can switch programmatically:

精度由 settings.modelPrecision 控制。用户在设置 → 推理中切换；切换会触发管线完整重载。开发者可编程切换：

// rebuild the pipeline with a new precision directory
const pipeline = new OnnxSVSPipeline(modelDir, {
  modelPrecision: 'fp32',   // 'fp32' | 'fp16' | 'int8' | 'int8-npu'
  deviceId,                 // DML adapter index
  preferredDeviceType,      // 'npu' | 'webnn-gpu' | undefined
  inferenceProvider,        // 'ortnode' (default) | 'ortweb'
});
await pipeline.init();

// swap vocoder without reloading encoders/diffusion
await pipeline.swapVocoder('sifigan');
await pipeline.swapSifiganPrecision('fp16');

// swap language models (JP) without reloading the vocoder
await pipeline.swapLanguageModels('ja');

Switching vocoder type, SiFiGAN precision, or language invalidates the synthesis cache (32 entries / 300 MB LRU), so the next synthesize() re-runs the vocoder.

切换声码器类型、SiFiGAN 精度或语言都会使合成缓存（32 条 / 300MB LRU）失效，因此下一次 synthesize() 会重新运行声码器。

Device Enumeration 设备枚举

enumerateDMLDevices first tries systeminformation (cached via gpuCache.js); if it returns nothing, it falls back to creating a probe DML session on note_text_encoder.onnx and parsing ORT verbose stderr for Discovered OrtHardwareDevice lines. Each device is classified by classifyDevice (in utils/deviceClassifier.js) into discrete-gpu, integrated-gpu, npu, or cpu.

enumerateDMLDevices 优先使用 systeminformation（经 gpuCache.js 缓存）；若无结果，回退到在 note_text_encoder.onnx 上创建探测 DML 会话，并解析 ORT verbose stderr 中的 Discovered OrtHardwareDevice 行。每个设备由 classifyDevice（位于 utils/deviceClassifier.js）分类为 discrete-gpu、integrated-gpu、npu 或 cpu。

Packaged App Size Optimization 打包体积优化

The packaged app size was reduced by approximately 527 MB (32%) through three measures:

打包应用体积通过三项措施减少了约 527 MB（32%）：