Reconstruction of Dexterous 3D Motion Data from a Flexible Magnetic Sensor with Deep Learning and Structure-Aware Filtering

Jiawei Huang, Ryo Sugawara, Kinfung Chu, Taku Komura, Yoshifumi Kitamura

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


We propose IM3D+, a novel approach to reconstructing 3D motion data from a flexible magnetic flux sensor array using deep learning and a structure-aware temporal bilateral filter. Computing the 3D configuration of markers (inductor-capacitor (LC) coils) from flux sensor data is difficult because the existing numerical approaches suffer from system noise, dead angles, the need for initialization, and limitations in the sensor array's layout. We solve these issues with deep neural networks to learn the regression from the simulation flux values to the LC coils' 3D configuration, which can be applied to the actual LC coils at any location and orientation within the capture volume. To cope with the influence of system noise and the dead-angle limitation caused by the characteristics of the hardware and sensing principle, we propose a structure-aware temporal bilateral filter for reconstructing motion sequences. Our method can track various movements, including fingers that manipulate objects, beetles that move inside a vivarium with leaves and soil, and the flow of opaque fluid. Since no power supply is needed for the lightweight wireless markers, our method can robustly track movements for a very long time, making it suitable for various types of observations whose tracking is difficult with existing motion-tracking systems. Furthermore, the flexibility of the flux sensor layout allows users to reconfigure it based on their own applications, thus making our approach suitable for a variety of virtual reality applications.

Original languageEnglish
Pages (from-to)2400-2414
Number of pages15
JournalIEEE Transactions on Visualization and Computer Graphics
Issue number6
Publication statusPublished - 2022 Jun 1


  • 3D interaction
  • 3D user interface
  • deep learning
  • filtering
  • input devices
  • motion capture

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Reconstruction of Dexterous 3D Motion Data from a Flexible Magnetic Sensor with Deep Learning and Structure-Aware Filtering'. Together they form a unique fingerprint.

Cite this