基于双LSTM融合的类人机器人实时表情再现方法
|
摘要
为提高机器人表情再现的时空相似度和运动平滑度,结合序列到序列的深度学习模型,提出一种基于双LSTM(长短期记忆)融合的类人机器人实时表情再现方法.在离线机械建模阶段,首先构建逆向机械模型以实现面部特征序列到电机控制序列的逆向映射,并进一步提出运动趋势模型以规整电机连续运动的平滑度;然后,引入加权目标函数以实现两模型的融合和参数优化.在在线表情迁移阶段,以表演者面部特征序列作为融合模型的输入,并在最优参数下完成表演者面部特征序列到机器人控制序列的端-端翻译,从而达到机器人表情的帧-帧再现.实验结果表明:融合模型的电机控制偏差不超过8%,且表情再现的时空相似度和运动平滑度大于85%.与相关方法相比,提出的方法在控制偏差、时空相似度和运动平滑度方面均有较大提升.
To improve space-time similarity and motion smoothness in robot expression imitation, a real-time expression mimicking method for humanoid robot based on dual LSTM(long short-term memory) fusion is proposed by combining with the sequence to sequence deep learning model. In offline mechanical modeling phase, an inverse mechanical model is constructed firstly to fulfill the inverse mapping from facial feature sequence to motor control sequence, and a motion tendency model is further presented to wrap the smoothness of continuous motor motion. Secondly, a weighted objective function is addressed to implement the fusion of the above two models as well as the parameter optimization. In online expression transfer phase, the facial feature sequence of performer is regarded as the input of the fusion model, and the frame-to-frame expression mimicking of robot is achieved by means of the end-to-end translation from the performer facial feature sequence to robot control sequence under the optimal parameters. The experimental results indicate that the control deviations of the fusion model is lower than 8%, meanwhile, the space-time similarity and motion smoothness in expression mimicking is greater than 85%. Compared with related methods, the proposed method has a significant improvement in the control deviation, space-time similarity and motion smoothness.
To improve space-time similarity and motion smoothness in robot expression imitation, a real-time expression mimicking method for humanoid robot based on dual LSTM(long short-term memory) fusion is proposed by combining with the sequence to sequence deep learning model. In offline mechanical modeling phase, an inverse mechanical model is constructed firstly to fulfill the inverse mapping from facial feature sequence to motor control sequence, and a motion tendency model is further presented to wrap the smoothness of continuous motor motion. Secondly, a weighted objective function is addressed to implement the fusion of the above two models as well as the parameter optimization. In online expression transfer phase, the facial feature sequence of performer is regarded as the input of the fusion model, and the frame-to-frame expression mimicking of robot is achieved by means of the end-to-end translation from the performer facial feature sequence to robot control sequence under the optimal parameters. The experimental results indicate that the control deviations of the fusion model is lower than 8%, meanwhile, the space-time similarity and motion smoothness in expression mimicking is greater than 85%. Compared with related methods, the proposed method has a significant improvement in the control deviation, space-time similarity and motion smoothness.
引文
[1]Pour A G, Taheri A, Alemi M, et al. Human-robot facial expression reciprocal interaction platform:Case studies on children with autism[J]. International Journal of Social Robotics, 2018,10(2):179-198.
[2]Ren F J, Huang Z. Automatic facial expression learning method based on humanoid robot XIN-REN[J]. IEEE Transactions on Human-Machine Systems, 2016, 46(6):810-821.
[3]Beer J M, Liles K R, Wu X, et al. Affective human-robot interaction[M]//Emotions and Affect in Human Factors and Human-Computer Interaction. Cambridge, USA:Academic Press, 2017:359-381.
[4]Mohammed O, Bailly G, Pellier D. Acquiring human-robot interaction skills with transfer learning techniques[C]//ACM/IEEE International Conference on Human-Robot Interaction.Piscataway, USA:IEEE, 2017:359-360.
[5]Alibeigi M, Ahmadabadi M N, Araabi B N. A fast, robust, and incremental model for learning high-level concepts from human motions by imitation[J]. IEEE Transactions on Robotics, 2017,33(1):153-168.
[6]黄忠,任福继,胡敏.基于RBF神经网络的人形机器人在线面部表情模仿[J].机器人,2016,38(2):225-232.Huang Z, Ren F J, Hu M. Online facial expression imitation for humanoid robot based on RBF neural network[J]. Robot, 2016,38(2):225-232.
[7]Shayganfar M, Rich C, Sidner C L. A design methodology for expressing emotion on robot faces[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway,USA:IEEE, 2012:4577-4583.
[8]Trovato G, Zecca M, Kishi T, et al. Generation of humanoid robot’s facial expressions for context-aware communication[J].International Journal of Humanoid Robotics, 2013, 10(1):No.1350013.
[9]Tadesse Y, Priya S. Graphical facial expression analysis and design method:An approach to determine humanoid skin deformation[J]. Journal of Mechanisms and Robotics, 2012, 4(2):No.021010.
[10]马淦.仿人机器人表情与身体动作的人机友好交互研究[D].北京:北京理工大学,2015.Ma G. Human-robot friendly interaction using facial and body motion for a humanoid robot[D]. Beijing, Beijing Institute of Technology, 2015.
[11]信继忠,柯显信,杨阳,等.仿人机器人头颈协调系统的研制[J].机械科学与技术,2017,36(1):29-33.Xing J Z, Ke X X, Yang Y, et al. Development of head and neck coordination system of humanoid robot[J]. Mechanical Science and Technology for Aerospace Engineering, 2017, 36(1):29-33.
[12]Jaeckel P, Campbell N, Melhuish C. Facial behavior mapping–From video footage to a robot head[J]. Robotics and Autonomous Systems, 2008, 56(12):1042-1049.
[13]Magtanong E, Yamaguchi A, Takemura K, et al. Inverse kinematics solver for android faces with elastic skin[M]//Latest Advances in Robot Kinematics. Dordrecht, Netherlands:Springer,2012:181-188.
[14]沈旭.基于序列深度学习的视频分析:建模表达与应用[D].合肥:中国科学技术大学,2017.Shen X. Video analysis based on sequence deep learning:Modelling, representation and application[D]. Hefei:University of Science and Technology of China, 2017.
[15]Ji Z, Xiong K L, Pang Y W, et al. Video summarization with attention-based encoder-decoder networks[EB/OL].(2017-08-31)[2018-02-01]. https://arxiv. org/pdf/1708.09545.pdf.
[16]秦超龙,宋爱国,吴常铖,等.基于Unity3D与Kinect的康复训练机器人情景交互系统[J].仪器仪表学报,2017,38(3):530-536.Qin C L, Song A G, Wu C C, et al. Scenario interaction system of rehabilitation training robot based on Unity3D and Kinect[J].Chinese Journal of Scientific Instrument, 2017, 38(3):530-536.
[17]Kompatsiari K, Tikhanoff V, Ciardo F, et al. The importance of mutual gaze in human-robot interaction[M]. Cham, Switzerlands:Springer, 2017:443-452.
[18]杨文璐,郑加悦,占婵,等.基于Kinect传感器的瞳孔定位算法[J].传感器与微系统,2015,34(2):125-128.Yang W L, Zheng J Y, Zhan C, et al. Pupil localization algorithm based on Kinect sensor[J]. Transducer and Microsystem Technologies, 2015, 34(2):125-128.
[19]Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780.
[20]朱特浩,赵群飞,夏泽洋.利用Kinect的人体动作视觉感知算法[J].机器人,2014,36(6):647-653.Zhu T H, Zhao Q F, Xia Z Y. A visual perception algorithm for human motion by a Kinect[J]. Robot, 2014, 36(6):647-653.
[2]Ren F J, Huang Z. Automatic facial expression learning method based on humanoid robot XIN-REN[J]. IEEE Transactions on Human-Machine Systems, 2016, 46(6):810-821.
[3]Beer J M, Liles K R, Wu X, et al. Affective human-robot interaction[M]//Emotions and Affect in Human Factors and Human-Computer Interaction. Cambridge, USA:Academic Press, 2017:359-381.
[4]Mohammed O, Bailly G, Pellier D. Acquiring human-robot interaction skills with transfer learning techniques[C]//ACM/IEEE International Conference on Human-Robot Interaction.Piscataway, USA:IEEE, 2017:359-360.
[5]Alibeigi M, Ahmadabadi M N, Araabi B N. A fast, robust, and incremental model for learning high-level concepts from human motions by imitation[J]. IEEE Transactions on Robotics, 2017,33(1):153-168.
[6]黄忠,任福继,胡敏.基于RBF神经网络的人形机器人在线面部表情模仿[J].机器人,2016,38(2):225-232.Huang Z, Ren F J, Hu M. Online facial expression imitation for humanoid robot based on RBF neural network[J]. Robot, 2016,38(2):225-232.
[7]Shayganfar M, Rich C, Sidner C L. A design methodology for expressing emotion on robot faces[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway,USA:IEEE, 2012:4577-4583.
[8]Trovato G, Zecca M, Kishi T, et al. Generation of humanoid robot’s facial expressions for context-aware communication[J].International Journal of Humanoid Robotics, 2013, 10(1):No.1350013.
[9]Tadesse Y, Priya S. Graphical facial expression analysis and design method:An approach to determine humanoid skin deformation[J]. Journal of Mechanisms and Robotics, 2012, 4(2):No.021010.
[10]马淦.仿人机器人表情与身体动作的人机友好交互研究[D].北京:北京理工大学,2015.Ma G. Human-robot friendly interaction using facial and body motion for a humanoid robot[D]. Beijing, Beijing Institute of Technology, 2015.
[11]信继忠,柯显信,杨阳,等.仿人机器人头颈协调系统的研制[J].机械科学与技术,2017,36(1):29-33.Xing J Z, Ke X X, Yang Y, et al. Development of head and neck coordination system of humanoid robot[J]. Mechanical Science and Technology for Aerospace Engineering, 2017, 36(1):29-33.
[12]Jaeckel P, Campbell N, Melhuish C. Facial behavior mapping–From video footage to a robot head[J]. Robotics and Autonomous Systems, 2008, 56(12):1042-1049.
[13]Magtanong E, Yamaguchi A, Takemura K, et al. Inverse kinematics solver for android faces with elastic skin[M]//Latest Advances in Robot Kinematics. Dordrecht, Netherlands:Springer,2012:181-188.
[14]沈旭.基于序列深度学习的视频分析:建模表达与应用[D].合肥:中国科学技术大学,2017.Shen X. Video analysis based on sequence deep learning:Modelling, representation and application[D]. Hefei:University of Science and Technology of China, 2017.
[15]Ji Z, Xiong K L, Pang Y W, et al. Video summarization with attention-based encoder-decoder networks[EB/OL].(2017-08-31)[2018-02-01]. https://arxiv. org/pdf/1708.09545.pdf.
[16]秦超龙,宋爱国,吴常铖,等.基于Unity3D与Kinect的康复训练机器人情景交互系统[J].仪器仪表学报,2017,38(3):530-536.Qin C L, Song A G, Wu C C, et al. Scenario interaction system of rehabilitation training robot based on Unity3D and Kinect[J].Chinese Journal of Scientific Instrument, 2017, 38(3):530-536.
[17]Kompatsiari K, Tikhanoff V, Ciardo F, et al. The importance of mutual gaze in human-robot interaction[M]. Cham, Switzerlands:Springer, 2017:443-452.
[18]杨文璐,郑加悦,占婵,等.基于Kinect传感器的瞳孔定位算法[J].传感器与微系统,2015,34(2):125-128.Yang W L, Zheng J Y, Zhan C, et al. Pupil localization algorithm based on Kinect sensor[J]. Transducer and Microsystem Technologies, 2015, 34(2):125-128.
[19]Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780.
[20]朱特浩,赵群飞,夏泽洋.利用Kinect的人体动作视觉感知算法[J].机器人,2014,36(6):647-653.Zhu T H, Zhao Q F, Xia Z Y. A visual perception algorithm for human motion by a Kinect[J]. Robot, 2014, 36(6):647-653.