VR UE4 Human Activity Sim

A VR-based Synthetic Data Generator for Human Activity Analysis, upgraded to UE4.26 with MetaHumans and YCBV support.

Overview

This project is an advanced fork of the UnrealROX framework, engineered to push the boundaries of synthetic data generation for robotic vision and human activity analysis.

It is designed as a complementary simulator to the Unity-based system presented by Zakour et al. (2021) “HOIsim”, providing a high-fidelity Unreal Engine alternative for synthesizing realistic 3D human-object interaction data.

The project aims to solve the “data hunger” of modern deep learning algorithms by providing a photorealistic, physically accurate, and programmatically variable environment. By simulating interactions in Unreal Engine 4.26, we can generate massive datasets with perfect ground truth for segmentation, depth, pose estimation, and grasping—tasks that are prohibitively expensive to annotate in the real world.

Key Points

1. Major Engine Upgrade (UE 4.26)

Modernized the core framework from UE 4.18 to 4.26, enabling:

  • MetaHuman Support: Integration of high-fidelity digital humans for hyper-realistic avatar simulation.
  • Chaos Physics: More accurate rigid body dynamics for object manipulation.
  • Ray Tracing: Enhanced lighting and reflections for closing the visual reality gap.

2. VR-Driven Interaction (HTC Vive Pro)

To capture naturalistic human behaviors, we added support for the HTC Vive Pro.

  • Human-in-the-Loop: A human operator can inhabit the simulation, performing complex grasping and manipulation tasks usage VR controllers.
  • Refined Grasping: The grasping logic was iteratively refined through intensive VR trials to ensure that virtual hand-object interactions mimic real-world physics (friction, damping, contact points).

3. Programmatic and Procedural Scene Improvement

Creating diverse scenarios manually is tedious. We introduced automation to scale up data generation:

  • Automatic NavMesh: Avatars can now autonomously navigate complex, clutter-filled indoor environments without manual waypoint graph plotting.
  • Smart Spawning: Objects and agents can be procedurally placed to create infinite scene variations.

4. Asset & Environment

  • YCBV Object Set: Full integration of the YCB Video Dataset objects, the gold standard for robotic manipulation benchmarking.
  • Pointcloud Environments: Testing the feasibility of the simulator with real lidar clouds and collision meshes.

Visuals

Robot Agent interacting within a Pointcloud Scene
Automatic NavMesh Generation & Agent Navigation
VR-based Grasping & Activity Simulation (Part 1)
VR-based Grasping & Activity Simulation (Part 2)

Impact

This tool aims to allows researchers to:

  1. Pre-train Vision Models: Use unlimited synthetic data to warm-start models before fine-tuning on scarce real-world data.
  2. Validate Robot Logic: Test high-level planning and navigation in “Digital Twin” environments before deploying to physical hardware.
  3. Analyze Human Activity: Generate annotated datasets of human-object interactions for action recognition tasks.