Abstract
Modeling hand-object interactions is a fundamentally challenging task in 3D computer vision. Despite remarkable progress that has been achieved in this field, existing methods still fail to synthesize the hand-object interaction photo-realistically, suffering from degraded rendering quality caused by the heavy mutual occlusions between the hand and the object, and inaccurate hand-object pose estimation. To tackle these challenges, we present a novel free-viewpoint rendering framework, Neural Contact Radiance Field (NCRF), to reconstruct hand-object interactions from a sparse set of videos. In particular, the proposed NCRF framework consists of two key components: (a) A contact optimization field that predicts an accurate contact field from 3D query points for achieving desirable contact between the hand and the object. (b) A hand-object neural radiance field to learn an implicit hand-object representation in a static canonical space, in concert with the specifically designed hand-object motion field to produce observation-to-canonical correspondences. We jointly learn these key components where they mutually help and regularize each other with visual and geometric constraints, producing a high-quality hand-object reconstruction that achieves photo-realistic novel view synthesis. Extensive experiments on HO3D and DexYCB datasets show that our approach outperforms the current state-of-the-art in terms of both rendering quality and pose estimation accuracy.
Original language | English |
---|---|
Title of host publication | 2024 International Conference on 3D Vision |
Publisher | IEEE |
Publication status | Accepted/In press - 15 Oct 2023 |
Event | 11th International Conference on 3D Vision - Davos Congress Centre, Davos, Switzerland Duration: 18 Mar 2024 → 21 Mar 2024 https://3dvconf.github.io/2024/ |
Publication series
Name | International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT) |
---|---|
Publisher | IEEE |
ISSN (Print) | 2378-3826 |
Conference
Conference | 11th International Conference on 3D Vision |
---|---|
Abbreviated title | 3DV 2024 |
Country/Territory | Switzerland |
City | Davos |
Period | 18/03/24 → 21/03/24 |
Internet address |