Describe the feature
I would like to propose experimental support for running MeloTTS/VITS
on a discrete Rockchip RK1828 PCIe NPU using RKNN3.
The initial target model is:
- vits-melo-tts-zh_en
- Chinese + English
- 1 speaker
- 44.1 kHz
Our prototype splits the original dynamic VITS graph into:
- Step1, fixed T=96:
text encoder and deterministic duration predictor.
- CPU dynamic path:
duration rounding, monotonic alignment, prior expansion,
latent sampling, padding and cropping.
- Step2, fixed W=384:
inverse flow and waveform decoder.
Both Step1 and Step2 are converted to RKNN3 FP16 models with
do_quantization=False.
The proposed upstream contribution would be divided into:
- Split ONNX export and RKNN3 conversion tools.
- ONNX/RKNN numerical parity tools.
- A generic RKNN3 TTS runtime integrated with sherpa-onnx OfflineTts.
- Reproducible model packaging, benchmarks and documentation.
The contribution would not contain product-specific server protocols,
ALSA configuration, device paths, systemd files, or Rockchip SDK binaries.
Motivation / Use Case
sherpa-onnx already provides the vits-melo-tts-zh_en model and an
RKNN backend for several ASR, VAD and KWS models.
However, the existing RKNN implementation is based mainly on
RKNN Toolkit2/RKNPU2 and currently does not provide a VITS/MeloTTS
runtime for a discrete RK1828 device using RKNN3.
Supporting this configuration would make high-quality Chinese and
English TTS available on embedded systems where CPU-only VITS inference
does not provide sufficient latency or power efficiency.
It may also provide a reusable implementation for other VITS-family
models and future RKNN3 devices.
Additional context
Environment used by the current prototype:
- Host: RK3588
- Accelerator: discrete RK1828 PCIe NPU
- RKNN3 Toolkit/runtime: 1.0.4
- RKNN conversion target: rk1820
- Model: vits-melo-tts-zh_en
- Precision: FP16, without W4A16 quantization
Current experimental board results:
- PyTorch vs split ONNX validation: passed
- Two rounds of 20 synthesis requests: 40/40 completed
- Aggregate synthesis RTF: approximately 0.416–0.417
- First-audio latency: approximately 0.93–1.03 seconds
- Single-character RTF: 1.577–1.760 because Step2 always executes
the fixed W384 graph
Known numerical gaps:
- Step1
logw RMSE: 0.005756
- Predicted duration changed from 289 to 291 frames
- Step2 waveform cosine similarity: 0.991028
- Step2 waveform RMSE: 0.009526
This is currently an experimental prototype. We are not claiming
production readiness or full numerical parity yet.
Before preparing the implementation PRs, I would appreciate guidance
from the maintainers on the following questions:
- Should RKNN3 use a separate build option and backend directory from
the existing RKNN Toolkit2 implementation?
- Would you prefer the split-export and numerical-parity tools as the
first standalone PR?
- Is a converter that requires a user-supplied RKNN3 SDK acceptable?
References:
Describe the feature
I would like to propose experimental support for running MeloTTS/VITS
on a discrete Rockchip RK1828 PCIe NPU using RKNN3.
The initial target model is:
Our prototype splits the original dynamic VITS graph into:
text encoder and deterministic duration predictor.
duration rounding, monotonic alignment, prior expansion,
latent sampling, padding and cropping.
inverse flow and waveform decoder.
Both Step1 and Step2 are converted to RKNN3 FP16 models with
do_quantization=False.The proposed upstream contribution would be divided into:
The contribution would not contain product-specific server protocols,
ALSA configuration, device paths, systemd files, or Rockchip SDK binaries.
Motivation / Use Case
sherpa-onnx already provides the vits-melo-tts-zh_en model and an
RKNN backend for several ASR, VAD and KWS models.
However, the existing RKNN implementation is based mainly on
RKNN Toolkit2/RKNPU2 and currently does not provide a VITS/MeloTTS
runtime for a discrete RK1828 device using RKNN3.
Supporting this configuration would make high-quality Chinese and
English TTS available on embedded systems where CPU-only VITS inference
does not provide sufficient latency or power efficiency.
It may also provide a reusable implementation for other VITS-family
models and future RKNN3 devices.
Additional context
Environment used by the current prototype:
Current experimental board results:
the fixed W384 graph
Known numerical gaps:
logwRMSE: 0.005756This is currently an experimental prototype. We are not claiming
production readiness or full numerical parity yet.
Before preparing the implementation PRs, I would appreciate guidance
from the maintainers on the following questions:
the existing RKNN Toolkit2 implementation?
first standalone PR?
References:
https://github.com/k2-fsa/sherpa-onnx/tree/master/scripts/melo-tts
https://k2-fsa.github.io/sherpa/onnx/tts/pretrained_models/vits.html#vits-melo-tts-zh-en
https://github.com/k2-fsa/sherpa-onnx/tree/master/sherpa-onnx/csrc/rknn