Read body movements in the video and translate into simple instructions

The job requires development of a software that can read specific parts of a human body in a given video and extract its movement data. These movement data will be used as "instructions" such as subject smiled, subject moved its finger etc. I will then use this data to move a virtual character in a game. So: 1- I would like to read the person's silhouette movements in the video file (live or pre-recorded) who will be on a one colored (may be green) background. A video will be uploaded to the developed software. The software needs to read certain points movable on the edges of the body like knees, feet, waist, elbow, wrist, fingers, shoulders for general posture, and of course the head itself and so on.. 2- I would like to read the person's general facial expressions only. This will be in a way the program will be able to read and understand basic expressions such as in emoticons. For example, it will tell the person smiles, the person is sad, angry, crying, etc whatever is possible to read from the muscles on the face. 3- The interval for the reading must not disturb the network transmission bandwidth / speeds. So I think below half a second is good, but this is the maximum limit. 4- The output "instructions" will be in a format that is usable by VR video games. These instructions will be able to move the VR characters in the game.