Phygical Expression

The Concept

The idea behind our final project is inspired by the game ‘Just Dance’ that Omar and I both played when we were younger and still continue to. It is also well-known and universal and almost every individual in our generation has played it or at least knows about it.

This is an image for reference of an older version of Just Dance where the players are supposed to make the same dance moves like the ones displayed in front of them on the screen.

Seeing that we currently live in a time with restrictions on the number of people in the same room at the same time and the limit of interaction, the installation we made requires having just one person in the frame at a time.

The first person that goes in has a snapshot taken of their first pose, this image will then be displayed on the screen presented for the people that come after and they would reenact/imitate such image. Once they do it correctly, the screen turns green for a second and plays a ‘success’ sound. They are then given a few seconds to pose in a way that expresses their thoughts, ideas, or a move that they find personal. When the timer gets to 0, the system takes a screenshot of their pose and displays this image for the next person. The next person that goes into the frame will do the same; stop, pose like the previous person, record their individualized pose, and so on.

The Final Interactive Installation


Here are some of the poses that we collected put together in a collage

The Implementation

When working on the implementation, we found that using PoseNet and ml5.neuralNetwork() would be the best way to record poses and check for them later. In order to implement it, we looked into machine learning and watched videos online to learn about it which was challenging but also really exciting. It took us a while to understand key concepts but once we did, it was fairly easy to apply them to our project and plan out how each part of the project would be coded. After setting things up, we took turns doing a standstill pose on camera as the code collected training data. The data was put into an array and saved as a .json file, which we later normalized (the large x & y values for) and used to train the model. At this point, the code runs normally and the users can go in.

The Code

Below are the most important sets of codes that are needed to explain how some of the complex functions are actually implemented.

The first function (keyPressed) is basically what we use to record the data, train it, and save it. The second function is our setup function that initializes the canvas, loads the prerecorded pose, and loads the sound effects.

These two functions are the ones that do most of the work. The first one (classifyPose) is the one that saves the different data points of the poses in the system to then compare with other poses to check if they are correct. The function after is the one that actually gets the result (gotResult) which checks if the confidence meaning the accuracy of the pose is close to 98% to leave space for a slight margin of error. If that’s the case most boolean variables change and are used elsewhere. The success sound is then taken and everything else occurs.