IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2024) Abstracts and Workshops (VRW), pp.244-251, 2024, DOI 10.1109/VRW62533.2024.00048 The main objectives of this project, launched in July 2021, is to mitigate and solve issues regarding health care services such as rehabilitation and specific health guidance while alleviating spatio-temporal, economic, and cognitive constraints by establishing remote technology foundation. There are four themes in this project. In theme #1, we have been developing the MR3 devices consisting of Wear and Mannequin for supporting detailed assessments of customers’ physical functions and haptic interaction respectively. The central issue of theme #2 is to support intrinsic motivation for rehabilitation and exercise training through XR technologies as in virtual (self) co-embodiment and hand redirection. In addition, we have also been investigating how to deal with one-to-many and zero-to-many situations. Theme #3 has aimed on establishing AI technology foundation for creating, monitoring progress of, and updating tele-rehab programs mainly for the upper limb. Systems for always-on monitoring during daily life and work developed in theme #4 is expected to be served as a common foundation for various tele-healthcare services. This paper reports on the progress of the above-mentioned themes in this project.

1. Project progress on XR-AI platform
for tele-rehab and health guidance
Takeshi KURATA1, Kunihiro OGATA1, Shusuke KANAZAWA1, Yumeko IMAMURA1,
Akihiro SATO1, Satoki OGISO1, Yoshiyuki KOBAYASHI1,
Ryosuke ICHIKARI1, Satoshi NAKAE1, Mitsunori TADA1,
Tomoki AOYAMA2, Hiroki SHIMIZU2, Hideaki KUZUOKA3, Takuto NAKAMURA3,
Takeshi KOSHIHARA4, Masao KURODA4, Hidemitsu SORIMACHI4, Fumihiro NARUSE5
1 The National Institute of Advanced Industrial Science and Technology (AIST), Japan
2 Kyoto University, Japan 3 The University of Tokyo, Japan
4 Seiko Epson Corporation, Japan 5 eveRehab, Inc., Japan
The research is supported by a project, JPNP21004, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
XR-AI: XR powered by AI

2. 2
Multimodal XR-AI platform for tele-rehab and the reciprocal care
coupling with health guidance
MR3: MultiModal Mixed Reality for Remote Rehab
XR-AI: XR powered by AI Issues to be addressed and corresponding research themes

3. 3
Social Background and issues regarding rehabilitation and
health guidance
Specific health guidance: “Tokutei Hoken Shido” in Japanese: Nationwide lifestyle intervention program targeting metabolic syndrome

4. Remote haptic interaction with MR3 Wear & Mannequin
4
IROS-TV (YouTube)
MR3: MultiModal Mixed Reality for Remote Rehab

5. Motion A Motion B
Ogata, K., et al., Basic Study of Upper Limb Movement Estimation and Function Evaluation including Shoulder Girdle by Multi-Sensing Flexible Sensor Wear,
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.13328-13334, 2022.
MR3 Wear
5
MR3: MultiModal Mixed Reality for Remote Rehab
Reaching motion A
without consciously moving the shoulder
Reaching motion B
with consciously moving the shoulder MR3 Wear can differentiate the magnitude of shoulder motion due to
the complex composition of the shoulder girdle (scapula and clavicle).
Similar
appearance
in shoulder
movement

6. Service processes for tele-rehab and specific health guidance
6
Process [A]
• Initial consultation
• Creation of rehab
programs
Process [B]
• Care task execution
Process [C]
• Follow-up consultation
• Rehab program updating
Process [D]
(Not widespread in conventional rehab)
rehab)
• Always-on monitoring
Specific health guidance: “Tokutei Hoken Shido” in Japanese: Nationwide lifestyle intervention program targeting metabolic syndrome

7. Consultation: Assessment of ADL
7
[A] [C]
ADL: Activities of Daily Living

8. Kiyu Tanaka, et al., Effect of Hanger Reflex on Detection Thresholds for Hand Redirection during Forearm Rotation, ACM
Symposium on Applied Perception 2023
VR rehab: Intrinsic motivation support (1)
8
Haptic presentation by
hanger reflex Visual presentation
Actual rotation
More rotation
Hand redirection (Visuo-haptic illusion)
Hanger reflex:
Phenomenon of feeling a
rotational force on the head
when wearing a wire hanger
Rotational forces induced by the hanger reflex
can be elicited not only in the head but in the
wrists, waist, knee, ankles, and shoulders.
Self-efficacy augmentation through
illusion to the extent that a sense of
agency can be maintained
[B]

9. Kiyu Tanaka, et al., Effect of Hanger Reflex on Detection Thresholds for Hand Redirection during Forearm Rotation, ACM Symposium on
Applied Perception 2023
Hand redirection with/without Hanger reflex
9
Neutral Inward Outward
JND: Just Noticeable Difference, DT: Detection Threshold, PSE: Point of Subjective Equality
Between Neutral and Inward in JNDs:
• 0.12 in Neutral, 0.16 in Inward
• Statistically significant difference (p = 0.03)

10. VR rehab: Intrinsic motivation support (2)
10
Reciprocal care: Zero-to-many (No therapist) situation
Gamification:
• Sharpening knifes by performing table-reach
exercise
• Earning points and Enriching their metaverse
Communication:
• Playing the game with others
Table-reach exercise for frozen
shoulder rehab: Effective but
boring...
[B]
T. Ota, T. Nakamura and H. Kuzuoka, "Effects of Gamification and Communication in Virtual Reality Frozen Shoulder
Rehabilitation for Enhanced Rehabilitation Continuity," in IEEE Access, 2023.

11. Reciprocal care:
Simple, Gamification, and Communication
11
p : ∗ ∗ ∗ < 0.001 < ∗∗ < 0.01 < ∗ < 0.05
Relative
rehabilitation
duration
(SIMPLE
=
1)
Absolute rehabilitation duration: Significantly longer
• in GAMIFICATION than in SIMPLE
• in COMMUNICATION than in SIMPLE
• in COMMUNICATION than in GAMIFICATION
Relative rehabilitation duration: significantly longer
in COMMUNICATION than GAMIFICATION
(COMMUNICATION condition: Playing the game with the other and introducing
themselves each other for 30 seconds prior to the game)

12. Always-on monitoring: Predicting daily physical and mental status
12
Random
Forest
XGBoost SVM
Stacking
Ensemble
Nothing for imbalanced data 0.198 0.205 0.188 0.189
Data augmentation 0.208 0.211 0.202 0.192
Undersampling and bagging 0.277 0.198 0.188 0.191
RMSE of mental state prediction when it is normalized
between 0 and 1 based on arousal and valence
[D]
RMSE (Root Mean Squared Error)

13. Conclusions
• XR-AI platform for tele-rehab and health guidance
• Remote haptic interaction with MR3 Wear & Mannequin
• VR rehab focusing on intrinsic motivation support
• Hand redirection with Hanger reflex
• Reciprocal care with Gamification and Communication
• Always-on monitoring for estimating daily physical and mental status
• Future works
• User studies with actual customers and patients
• Heterogeneous reciprocal care
13
XR-AI: XR powered by AI
MR3: MultiModal Mixed Reality for Remote Rehab