Kineo Upgrades

A personal framework for optimization-based markerless motion capture, upgrading Kineo with DensePose and SMPL-X support.

Overview

This project focuses on upgrading and extending Kineo, a robust framework for markerless motion capture. The original work, Kineo: Kinetic Energy Optimization for Markerless Motion Capture, presents a compelling approach to estimating 3D human pose from multi-view video.

Visualization of the markerless pipeline on different poses.

Motivation

My primary motivation is to maintain an up-to-date, personal and free framework for optimization-based markerless motion capture. While the current market is dominated by feed-forward, parametric implementations (which predict pose directly from pixels), triangulation and optimization-based methods similar to my original work in ADL4D offer distinct advantages in accuracy and consistency for complex multi-view setups and unfamiliar data.

Kineo stands out due to its Auto-Calibration Framework, which allows for flexible camera setups without tedious manual calibration steps—a critical feature for rapid deployment and pairwise reprojection error based human pose triangulation and optimization.

Goals

The objective of this work is to familiarize myself with the Kineo codebase and extend its capabilities:

1. Codebase Familiarization & Modernization

  • Deep dive into the existing implementation to understand the core optimization constraints.
  • Refactor legacy components for compatibility with modern PyTorch/Python ecosystems.

2. DensePose Integration

  • Landmark Support: Integrate support for DensePose-based landmarks to provide richer visual evidence than standard keypoints.
  • Triangulation: Implement triangulation across neighboring landmarks to suppress model noise and error, improving the robustness of the 3D lifting process.

3. High-Fidelity Parametric Fitting (SMPLx)

  • Fine-Grained Pose: Support parametric pose and shape fitting using the SMPL-X model.
  • Expressiveness: Capture subtle details including breathing dynamics and basic facial expressions, moving beyond rigid body skeletal tracking.

4. Dynamic Camera Support

  • Moving Cameras: Attempt to add support for moving cameras, enabling capture in non-static environments.