Pose Estimation on Qualcomm® Hexagon™

Version: 1.0
Release Date: Aug 2025
Copyright: © 2025 Advantech Corporation. All rights reserved.

Overview

The Pose Estimation on Qualcomm® Hexagon™ container image provides a comprehensive environment for Pose Estimation model on Qualcomm’s DSP-enabled hardware. This containerized setup ensures full hardware acceleration, reliability, and support for popular runtime environments like QNN, SNPE, and LiteRT. At the core of this project, YOLOv8 and HRNet Pose Estimation is used as the pose estimation model. Two complementary workflows are supported:

Ultralytics Export Workflow
- An automated script directly leverages the Ultralytics YOLOv8 export utilities, producing model in TFlite format suitable to run on QCS6490 device utilizing Hexagon™ DSP.
- Offers flexibility for quick prototyping and validation on the device.
- Ideal for iterative development/training where models may frequently change before being finalized.
AI Hub Conversion Workflow
- HRNet Pose Estimation Model is downloaded from Qualcomm’s Hugging Face Models repository via an automated script
- This workflow ensures maximum utilization of the Hexagon™ DSP for INT8 inference acceleration.
- Best suited for quick deployments where optimized performance is critical.

By combining these two approaches, the project balances rapid prototyping & deployment with Ultralytics and Qualcomm® AI Hub, making it suitable for diverse edge AI applications such as robotics, industrial vision, and smart surveillance etc.

Key Features

Complete AI Framework Stack: QNN SDK (QNN, SNPE), LiteRT
Edge AI Capabilities: Optimized pipelines for real-time vision tasks (pose estimation)
Preconfigured Environment: Comes with all necessary tools pre-installed in a container
Full DSP/GPU Acceleration: Utilize Qualcomm® Hexagon™ DSP and Adreno™ GPU for fast and efficient inference
Dual Pose Estimation Workflows: Support for both Qualcomm® AI Hub conversion and Ultralytics export methods, enabling better flexibility

Quick Demonstration

Pose Estimation using Yolov8n Model

Architecture

Ultralytics Workflow

Hardware Specifications

Component	Specification
Target Hardware	Advantech AOM-2721
SoC	Qualcomm® QCS6490
GPU	Adreno™ 643
DSP	Hexagon™ 770
Memory	8GB LPDDR5

Operating System

This container is intended for QCOM Robotics Reference Distro with ROS, version 1.3-ver.1.1 OS running on QCS6490 device.

Environment	Operating System
Device Host	QCOM Robotics Reference Distro with ROS 1.3-ver.1.1
Container	Ubuntu 22.04 LTS

Software Components

Component	Version	Description
LiteRT	1.3.0	Provides QNN TFLite Delegate support for GPU and DSP acceleration
SNPE	2.29.0	Qualcomm’s Snapdragon Neural Processing Engine; optimized runtime for Snapdragon DSP/HTP
QNN	2.29.0	Qualcomm® Neural Network (QNN) runtime for executing quantized neural networks
GStreamer	1.20.7	Multimedia framework for building flexible audio/video pipelines
Python	3.10.12	Python runtime for building applications
OpenCV	4.11.0	Computer vision library for image and video processing
torch	1.8.0	Used for YOLOv8 model export only via Ultralytics export utilities
torchvision	0.9.0	Required alongside torch for model export (not used during inference)

Supported AI Capabilities

Vision Models

Model	Format	Note
YOLOv8 Detection	TFLite INT8	Downloaded from Ultralytics` official source and exported to TFLite using Ultralytics Python packages
YOLOv8 Segmentation	TFLite INT8	Downloaded from Ultralytics` official source and exported to TFLite using Ultralytics Python packages
YOLOv8 Pose Estimation	TFLite INT8	Downloaded from Ultralytics` official source and exported to TFLite using Ultralytics Python packages
Lightweight Face Detector	TFLite INT8	Converted using Qualcomm® AI Hub
FaceMap 3D Morphable Model	TFLite INT8	Converted using Qualcomm® AI Hub
DeepLabV3+ (MobileNet)	TFLite INT8	Converted using Qualcomm® AI Hub
DeepLabV3 (ResNet50)	SNPE DLC TFLite	Converted using Qualcomm® AI Hub
HRNet Pose Estimation (INT8)	TFLite INT8	Converted using Qualcomm® AI Hub
PoseNet (MobileNet V1)	TFLite	Converted using Qualcomm® AI Hub
MiDaS Depth Estimation	TFLite INT8	Converted using Qualcomm® AI Hub
MobileNet V2 (Quantized)	TFLite INT8	Converted using Qualcomm® AI Hub
Inception V3 (SNPE DLC)	SNPE DLC TFLite	Converted using Qualcomm® AI Hub
YAMNet (Audio Classification)	TFLite	Converted using Qualcomm® AI Hub
YOLO (Quantized)	TFLite INT8	Converted using Qualcomm® AI Hub

LLM Models

Model	Format	Note
Phi2	.so	Converted using Qualcomm's LLM Notebook for Phi-2
Tinyllama	.so	Converted using Qualcomm's LLM Notebook for Tinyllama
Meta Llama 3.2 1B	.so	Converted using Qualcomm's LLM Notebook for Meta Llama 3.2 1B

Note: The above tables highlight a subset of commonly used models validated for this environment. Other transformer-based or vision models may also be supported depending on runtime compatibility and hardware resources. For the most detailed and updated list of supported models and runtimes, please refer to the Qualcomm's official AI Hub.

Supported AI Model Formats

Runtime	Format	Compatible Versions
QNN	.so	2.29.0
SNPE	.dlc	2.29.0
LiteRT	.tflite	1.3.0

Hardware Acceleration Support

Accelerator	Support Level	Compatible Libraries
GPU	FP32	QNN, SNPE, LiteRT
DSP	INT8	QNN, SNPE, LiteRT

Precision Support

Precision	Support Level	Notes
FP32	CPU, GPU	Baseline precision, highest accuracy
INT8	CPU, DSP	Faster inference time, lower accuracy

Repository Structure

Pose-Estimation-on-Qualcomm-Hexagon/
├── .env                                      # Environment configuration
├── data                                      # ReadMe Data (images, gifs etc.)
├── windows-git-setup.md                      # Steps to fix LF/CRLF issues on windows while copying to device
├── AI_HUB_EXPORT_README.md                   # Qualcomm® AI Hub based model export guide
├── README.md                                 # Container overview and quick start steps
├── YOLO_EXPORT_README.md                     # Guide for Yolov8 Model export using ultralytics python package
├── build.sh                                  # Script to build the container
├── docker-compose.yml                        # Docker compose file
├── advantech-aihub-model-export.sh           # Automated AI Hub based Pose Estimation model conversion script
├── advantech-coe-model-export.sh             # Automated script to convert Yolov8 Pose Estimation model via ultralytics package
├── advantech-yolo.py                         # Run Yolov8 Pose Estimation inference pipeline
├── video/
│   └── pose_1080p_25fps.mp4   	              # Pose Test MP4 video with 1080p resolution & 25 FPS
└── wise-bench.sh                             # Script to verify AI software stack inside the container

Quick Start Guide

Clone the Repository (on your development machine)

Note for Windows Users:
If you are using Linux, no changes are needed — LF line endings are used by default.
If you are on Windows, please follow the steps in Windows Git Line Ending Setup before cloning to ensure scripts and configuration files work correctly on Device.

git clone https://github.com/Advantech-EdgeSync-Containers/Pose-Estimation-on-Qualcomm-Hexagon.git
cd Pose-Estimation-on-Qualcomm-Hexagon

Transfer the `Pose-Estimation-on-Qualcomm-Hexagon` Folder to QCS6490 Board

If you cloned the repo on a separate development machine, use scp to transfer only the relevant folder:

# From your development machine (Ubuntu or Windows PowerShell if SCP is installed)
scp -r .\Pose-Estimation-on-Qualcomm-Hexagon\ <username>@<qcs6490-ip>:/home/<username>/

Replace:

<username> – Login username on the QCS6490 board (e.g., root)
<qcs6490-ip> – IP address of the QCS6490 board (e.g., 192.168.1.42)

This will copy the folder to /home/<username>/Pose-Estimation-on-Qualcomm-Hexagon/.

Then SSH into the board:

ssh <username>@<qcs6490-ip>
cd ~/Pose-Estimation-on-Qualcomm-Hexagon

Installation

# Make the build script executable
chmod +x build.sh

# Launch the container
./build.sh

AI Accelerator and Software Stack Verification (Optional)

# Verify AI Accelerator and Software Stack Inside Docker Container
cd /workspace
chmod +x wise-bench.sh
./wise-bench.sh

Wise-bench logs are saved in the wise-bench.log file under /workspace

Model Optimization Workflows

For optimal performance, follow these recommended model conversion paths:

PyTorch Models

PyTorch → ONNX → TensorFlow → LiteRT

TensorFlow Models

TensorFlow → SavedModel → QNN(.cpp, .bin) → QNN Model Library(.so)

TensorFlow → SavedModel → SNPE(.dlc)

TensorFlow → LiteRT

Options for Exporting & Running Pose models on QCS6490

Option 1 - Ultralytics Export Workflow:

Export YOLOv8 Models via Local Script

Use the export script inside the container to convert and quantize YOLOv8 models to .tflite.

YOLO Export Instructions

Run YOLO‑Exported Model (CLI)

Camera Setup Note: If you want to run this container for camera feed, ensure the camera is connected to the edge device before running ./build.sh or starting the container. Verify camera detection using the following command:

v4l2-ctl --list-devices

Use the following command to run inference with your exported model:

python3 advantech-yolo.py --model=<MODEL_PATH> --source=<VIDEO_SOURCE> [--cam-width=<INT>] [--cam-height=<INT>] [--conf-thres=<FLOAT>] [--iou-thres=<FLOAT>] [--save=<OUTPUT_PATH>]

Here’s what each field represents:

--model Required. Path to the TFLite YOLOv8 model used for inference (e.g., model/yolov8n-pose_pose.tflite).
--source Optional (default: /etc/media/video.mp4). Can be:
- A video file path (e.g., video/pose_1080p_25fps.mp4)
- A device file for webcam input (e.g., /dev/video0)
- An RTSP video stream (e.g., rtsp://13.232.142.213:8554/mystream)
--cam-width Optional* (default: 1920).* Applicable only when --source is a webcam device. Sets the webcam's capture width.
--cam-height Optional* (default: 1080).* Applicable only when --source is a webcam device. Sets the webcam's capture height.
--conf-thres Optional (default: 0.5). Confidence threshold to filter out low-confidence detections (value between 0 and 1).
--iou-thres Optional (default: 0.5). Intersection-over-Union threshold governing Non-Maximum Suppression, removing overlapping detections.
--save Optional. File path to save the annotated inference output as an MP4 video. The footage will still be displayed live via GStreamer.

Example Commands

Run detection on a video file:

python3 advantech-yolo.py --model=model/yolov8n-pose_pose.tflite --source=video/pose_1080p_25fps.mp4

Run detection via discover (e.g. discover):

python3 advantech-yolo.py --model=model/yolov8n-pose_pose.tflite --source=discover

Run Pose Estimation via webcam (e.g. /dev/video0):

python3 advantech-yolo.py --model=model/yolov8n-pose_pose.tflite --source=/dev/video0 --cam-width=1920 --cam-height=1080

Display and Save to MP4 File:

python3 advantech-yolo.py --model=model/yolov8n-pose_pose.tflite --source=video/pose_1080p_25fps.mp4 --save=output.mp4

Option 2 - AI Hub Conversion Workflow:

Export HRNet Pose Model from Qualcomm® AI Hub

For pulling models from Qualcomm’s cloud-based hub using an API token.

AI Hub Export Instructions

Run AI Hub–Exported Model (GStreamer Application)

Customize the AI pose estimation application by editing the configuration file at /etc/configs/config_pose.json. This file defines the model, labels, runtime, and other essential settings.

Configure the Pose Estimation Application

You must edit the configuration file located at /etc/configs/config_pose.json. Here’s how to do it using vi, the standard command-line editor:

1. Open the file in vi:

vi /etc/configs/config_pose.json

If the file does not exist, vi will create one for you.

2. Basic vi usage:

You start in command mode. To enter insert mode, press i.
After editing, press Esc to go back to command mode.

3. Save and exit:

Save and quit: Type :wq and press Enter
Or simply ZZ (both uppercase)
Exit without saving: Type :q! and press Enter

4. Vi quick reference:

Command	Action
`i`	Enter insert mode (type text)
`Esc`	Return to command mode
`:wq`	Save and exit `vi`
`ZZ`	Save and exit (alternate)
`:q!`	Exit without saving

General Configuration Template

{
  "file-path": "<input video path>",
  "ml-framework": "<snpe, tflite, or qnn>",
  "model": "<Model Path>",
  "labels": "<Label Path>",
  "constants": "<Model Constants for LiteRT Model>",
  "threshold": <Post‑processing threshold, integer from 1–100>,
  "runtime": "<dsp, cpu, or gpu>"
}

Here’s what each field represents:

Use either file-path (for video input) or camera (for live camera stream), not both.
- "file-path": Source of input video, e.g., /etc/media/video.mp4.
- "camera": 0 → Primary camera
- "camera": 1 → Secondary camera
ml-framework: Runtime framework—choose one of snpe, tflite, or qnn.
model: Full path to the .tflite or .dlc model.
labels: Label file path mapping detection classes.
constants: Quantization metadata in the format:
```
hrnet,q-offsets=<8.0>,q-scales=<0.0040499246679246426>;
```
Derived from your model’s quantization parameters.
threshold: Confidence cutoff for pose estimation.
runtime: Preferred inference engine—dsp (recommended for edge), cpu, or gpu.

Example Configuration

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model": "/etc/models/hrnet_pose.tflite",
  "labels": "/etc/labels/hrnet_pose.labels",
  "constants": "hrnet,q-offsets=<8.0>,q-scales=<0.0040499246679246426>;",
  "threshold": 51,
  "runtime": "dsp"
}

This configuration:

Uses a TFLite-format HRNET POSE model running on the DSP.
Sets Pose Estimation threshold to 51%.
Points to local paths for media, model, and labels.

Execute the Pose Estimation Application

After saving your changes:

gst-ai-pose-detection --config-file=/etc/configs/config_pose.json

You'll see the input video annotated in real time—displayed with bounding boxes and pose skeleton around detected objects, based on your configuration.

Important Note: Format & Resolution Support

This GStreamer-based Pose Estimation workflow only supports MP4 and MOV input formats and is reliably tested at 1080p resolution (1920×1080). Using lower resolutions (e.g., 720p or below) may result in misaligned detection boxes and pose skeleton due to scaling and aspect ratio handling issues with the video sink.
In the camera mode, gst-ai-pose-detection relies on the qtiqmmfsrc GStreamer plugin, which is designed to interface with MIPI‑CSI connected cameras only. USB webcams are not supported. Qualcomm® Camera Overview

Recommendation – Pose Estimation by Content Type

Pose estimation performance can vary with video characteristics. For videos with clear full-body visibility (e.g., outdoor dance, sports etc.), YOLOv8n-pose often provides stable results with lower compute overhead. However, for more complex cases such as indoor yoga, low-light conditions, or poses with floor contact and occlusion, it is recommended to use higher-capacity models like YOLOv8s or YOLOv8m to achieve better accuracy and stability.

Contributing factors include:

Lighting & contrast: Low or uneven indoor lighting reduces keypoint confidence.
Motion blur: Fast transitions or slow shutter speeds cause jitter and limb swaps.
Viewpoint & foreshortening: Extreme camera angles or floor-contact poses hide joints.
Occlusion & self-occlusion: Crossed limbs or body positions floor/ground occlude landmarks.
Subject scale in frame: Distant subjects or cropped frames reduce joint resolution.
Background/clothing: Similar tones between clothing and background confuse detection.
Compression & frame rate: High compression or frame rates below 25 fps introduce artifacts.

Known Limitations

LiteRT Runtime: Limited DSP acceleration support for complex operators.
Model Batch Size: Fixed batch size is required for models targeting DSP acceleration.
GStreamer Debug Level: Using GST_DEBUG=2 or higher may cause the video output to appear in random positions on screen or behave unexpectedly. Use GST_DEBUG=0 or GST_DEBUG=1 for stable video playback.
Resolution-Dependent Alignment Issues: Option 2 is optimized for resolutions at or above 1080p. Below 1080p, detection boxes and pose skeleton may appear misaligned with objects due to scaling artifacts and coordinate rounding.

Not Included in This Repository

Component	Reason / Note
Pre-exported YOLO models	Must be exported manually using provided scripts
Qualcomm® AI Hub API Token	User must create account and retrieve token
Full training datasets	Outside scope of this deployment repo

Best Practices

Prefer INT8 quantized models for DSP acceleration
Ensure fixed batch sizes when converting models
Use lower GST_DEBUG levels for stable multimedia handling
Always validate exported models on-device after deployment

Possible Use Cases

1. Fitness & Rehabilitation

Real-time exercise feedback: AI observe and correct posture during workouts for form optimization and injury prevention.
Physical therapy training: Monitor patient movement and progression during rehab, enabling remote guidance.

2. Sports Analytics

Performance analysis: Track athlete movements to analyze techniques, optimize form, and reduce injury risks.

3. Animation, AR/VR & Gaming

Markerless motion capture: Drive character animations and virtual avatars using live body pose data from cameras.
Immersive gameplay: Enable gesture-based controls for a more engaging experience.

4. Healthcare & Assisted Living

Postural monitoring: Detect poor ergonomics or risky body alignment in real time—useful for workplaces or elder care.
Fall detection: Identify and respond to falls or sudden movements in assisted living scenarios.

5. Surveillance & Behavior Analysis

Abnormal activity detection: Analyze body postures and gestures to flag suspicious behavior or loose safety compliance.

6. Human–Robot Interaction

Gesture-driven control: Robots interpret user pose for intuitive interactions, like teleoperation or cooperative tasks.

7. Retail & AR Applications

Virtual try-ons: Use pose detection to enhance virtual fitting rooms—simulate apparel on users in real time.
Interaction tracking: Monitor customer postures and gestures to improve retail UX.

8. Human–Computer Interaction & Gesture Recognition

Touchless interfaces: Enable users to control systems or express intent using body pose or gestures (e.g., in sign language applications).

Acknowledgments

Ultralytics: For the YOLOv8 framework that powers this toolkit.

# Required specific versions:
python3 -m pip install ultralytics==8.3.176 --no-deps
python3 -m pip install ultralytics-thop==2.0.0 --no-deps

Name		Name	Last commit message	Last commit date
data		data
video		video
.env		.env
ACC-README.md		ACC-README.md
AI_HUB_EXPORT_README.md		AI_HUB_EXPORT_README.md
README.md		README.md
YOLO_EXPORT_README.md		YOLO_EXPORT_README.md
advantech-aihub-model-export.sh		advantech-aihub-model-export.sh
advantech-coe-model-export.sh		advantech-coe-model-export.sh
advantech-yolo.py		advantech-yolo.py
build.sh		build.sh
docker-compose.yml		docker-compose.yml
windows-git-setup.md		windows-git-setup.md
wise-bench.sh		wise-bench.sh

Uh oh!

Latest commit

History

Folders and files

Repository files navigation