Towards a real-time photo-realistic animation of a human body using 2D tools

WISDOM Faculty Retreat
May 2012

Yosef Yomdin
Yevgeniy Kagan
Grisha Dinkin
Dima Batenkov

Our goal

Allow a non-expert user to create an interesting movie from her album photographs

  • Single-image object recognition
  • Imitate natural body movements
  • Near real-time performance

Geometric image processing

Low-level geometric image representation (vectorization)

Geometric models for:

  • visual patterns specification,
  • object recognition,
  • image animation.

Human model

Manual fitting - skeleton

Manual fitting - contours



2D animation

Problem: need accurate 3d body movements!

Solution: use motion capture data

Carnegie-Mellon mocap database

CMU database

CMU database

Process overview

  • Obtain 3d model + mocap data
  • Offline 3d-2d translation
  • 2d-2d translation from "standard 2d" model to the actual model from image
  • Hope the result is visually satisfying

3d-2d projection

Need to specify joint positions + layer scaling

  • 3d/2d skeleton matching
  • "Grow flesh" and measure projected segment lengths
  • Limbs occlusion
  • Fix unacceptable limb distortions

2d-2d translation

  • Unknown model proportions
  • Unknown initial pose
    1. Assume initial pose to be roughly enface
    2. Preserve angles between limbs during animation
    3. Some animations require "forced grounding"



(I.Sella, T.Flash, R.Malach)

Motion/Form Self-motion Other's motion
Other's picture

Future work

  • "Glue up" an interesting long animation from basic building blocks
  • Improve visual quality by introducing additional "layers"
  • Use 3D model for pose detection and "semi-automatic animation"

Thank you