Author: awm330

The Concept

The idea behind our final project is inspired by the game ‘Just Dance’ that Omar and I both played when we were younger and still continue to. It is also well-known and universal and almost every individual in our generation has played it or at least knows about it.

Seeing that we currently live in a time with restrictions on the number of people in the same room at the same time and the limit of interaction, the installation we made requires having just one person in the frame at a time.

The first person that goes in has a snapshot taken of their first pose, this image will then be displayed on the screen presented for the people that come after and they would reenact/imitate such image. Once they do it correctly, the screen turns green for a second and plays a ‘success’ sound. They are then given a few seconds to pose in a way that expresses their thoughts, ideas, or a move that they find personal. When the timer gets to 0, the system takes a screenshot of their pose and displays this image for the next person. The next person that goes into the frame will do the same; stop, pose like the previous person, record their individualized pose, and so on.

The Final Interactive Installation

Here are some of the poses that we collected put together in a collage

The Implementation

When working on the implementation, we found that using PoseNet and ml5.neuralNetwork() would be the best way to record poses and check for them later. In order to implement it, we looked into machine learning and watched videos online to learn about it which was challenging but also really exciting. It took us a while to understand key concepts but once we did, it was fairly easy to apply them to our project and plan out how each part of the project would be coded. After setting things up, we took turns doing a standstill pose on camera as the code collected training data. The data was put into an array and saved as a .json file, which we later normalized (the large x & y values for) and used to train the model. At this point, the code runs normally and the users can go in.

The Code

Below are the most important sets of codes that are needed to explain how some of the complex functions are actually implemented.

For this week, Omar and I met up to work on the implementation of our project. We put a lot of thought into brainstorming to see what visuals and shaders we’ll use in order to optimize our output. As we went through the process of outlining our idea, we kept on getting more ideas of somewhat different applications of shaders as we wanted it to be even more fun and interactive.

The idea we decided to go ahead with and implement is inspired by the game ‘Just Dance’ that we both played when we were younger and still continue to. It is also well-known and universal and almost every individual in our generation has played it or knows about it.

Seeing that we currently live in a time with restrictions on the number of people in the same room at the same time and the limit of interaction, we are working on an installation that requires having just one person in a frame at a time.

To start off, an image that we initialized would first be presented to the very first person which they would reenact/imitate. Once they do it correctly and follow its silhouette, they will then be given a few seconds to stand in a position that expresses their thoughts, ideas, or a move that they find personal. When they do that and standstill, the system will take a screenshot of the individual’s pose and save this image for the next person. The next person that goes into the frame will do the same steps; stop, pose like the image, and record their individualized pose, and so on.

When working on the implementation, we found out that using PoseNet and ml5.neuralNetwork() would be the best way to record certain poses and check for them later. Looking into machine learning was challenging but also really exciting. It took us a while to understand key concepts but once we did, it was fairly easy to apply them to our project and plan out how each part of the project would be coded. After setting things up, we took turns doing a T-pose on camera as the code collected training data. The data was put into an array and saved as a .json file, which we later normalized (the large x & y values for) and used to train the model. Although we have a clear path laid out for how we’re going to move forward, we ran into some problems here. In the p5 editor, the Training Performance window is blank, so we cannot track the training of our model. We also get an error and we never get a model downloaded, as well as an error code that we spent hours trying to troubleshoot and that doesn’t seem to make a lot of sense.

The amount of holons in my life is probably uncountable, but here’s a list of some that are on the top of my head:

• Daughter/sister
• Major/minor
• Room
• Meals
• Workouts
• Heart
• Brain
• Body
• Ideas/thoughts
• Running
• Clothes
• Phone
• Steps/Walking
• Cake ingredients
• Classes/zoom
• Peers
• Covid-19
• This assignment
• …. and the list goes on

As for the practical assignment, I played around with the different shaders and followed through with Aaron in duplicating the rings visual separately and changing how often the rings appear in a sin wave, and changing the speed. I also looked at some of the shaders on ShaderToy but most of them were too complex and my browser would even crash.

I decided to work with visual 22 as I found it very interesting when I was watching the lecture. It reminded me a lot of how the JBL moves to a beat and splashes water when placed on it, https://www.youtube.com/watch?v=Oz2o6mfe2Fk&ab_channel=TechHive, as can be seen here.

This is my final product:

I didn’t really like the red color so I decided to change it to a color that I like that would also contrast nicely with both the black and the white so I went ahead with a blue-purple-pink transition. I also changed the circle size and switched around the mouseX and mouseYs. To continue with the JBL theme/idea, I decided to choose the techno sound and play around with the speed and the gain of it. It also took my some time to synchronize the sounds and the beat as much as I can and I believe that the result was not bad.

For some reason, I got really random errors throughout, some on the browser where the terminal just disconnected and some with setting up SuperCollider.

For the final project, Omar and I will be working together and so we came up with an idea that is personal to both of us, relating to Egypt where we were both born and brought up. 

Concept: An interactive map of Egypt that would work in one of two ways, we still haven’t decided. The first way is to use the mouse location on the screen and the second is by using the camera and your hand. It starts off by displaying a map of Egypt with 3 major locations that we’ve selected (Cairo, Alexandria/North Coast/ Luxor & Aswan). The viewer chooses an area and accordingly, the map starts playing sounds and images of that place. Also, within the chosen area, the movement of the viewer would also be tracked to change the images and sounds displayed; the four corners are different whereas the center would display all 4 simultaneously. We will try to incorporate sounds and images that we have recorded ourselves.

Execution:  After the viewer has picked their destination (one of the major locations), the computer’s camera would use color detection to track the hand position of the viewer who would be instructed to wear a bright and distinct colored glove. The x and y coordinates of the detected hand would be used to control the gain and opacity of sounds and images. For example, the x and y values for the middle of the screen would set the opacity of everything at 50%, and the sounds would all play at half-gain, but the x and y values at the top right corner would only set the gain and opacity at a set max (eg: 1.5 gain and 100% opacity) for 1 set of image and sound and then 0 for the other 3 sets. We’re still not set on how exactly we would go about the viewer picking a destination, we were thinking of having the map as a constant backdrop and the overlaying images and sounds would differ based on which location the mouse is placed on, but we are compelled to do something more demanding in terms of body motion.

As I watched the lecture and looked at the different visuals and how they change and work, I couldn’t help but think about the changes that I can do to each and every individual visual to make it my own. My curiosity couldn’t stop me from trying out all the visuals, testing them, and changing a few lines just to see what I can do with them.

After some time of debate between me and myself, I decided to play around with the “delay” visual as it brought back memories of my friends and I trying out cameras and filters on the different apps that we used to download when we were younger. Not only that but also its colorfulness intrigued me the most.

To make it easier for me to work with, the first change I made was that I added this line “uv.x = 1.0 – uv.x;” to mirror my screen. Even though this change was minor, I was proud of myself that I did it and it turned out to be so much easier than I had expected, because I have been trying to do that in my previous assignments but was never patient enough to make it work.

I wanted the transition to be smoother so I decreased the number of frames displayed per second for an easier flow in the visual, making it closer and closer to my childhood dreams. I also played around with the indices to make the color change closer to a shadow or an outline rather than a frame that follows the user.

When recording my screen, there was an awful lag, and my laptop fan was too high, so I decided to wait and record it at a later time

As I started this week’s “reading (listening)”, I was so confused as it had a very philosophical approach. I was about to take notes but I thought I should listen to the end of the chapter to see how Ken Wilbur would relate the topic to interactive media, artwork, or shaders. I finished listening and noticed that this wasn’t the case and the connection was left open-ended. This made me realize that it is now me, an individual, a part of the class, that should think through what was said and make the links for myself, or it was just for general knowledge.

The idea of the talk relies on the idea of how a whole is always a part of something bigger. My first thought about this idea was how one each individual working separately or taking action towards a cause helps in the overall effect towards their goal. Current real-life examples would include, for example, voting for the US president, where each whole individual puts their vote in and ends up being part of a change on the wider spectrum. Not only that, but there is also the move to veganism, individuals minimizing pollution, students in a class, and the list goes on and on. We could relate this to the domino effect, the butterfly effect, and an enormous number of well-known quotes.

He also elaborates on the 5 tenants with a focus on how the “cosmos?” (I couldn’t hear the word) all share similar characteristics. He breaks them down into the horizontal and vertical capacities of agency and communion. The horizontal capacities mainly focus on how the wholeness and partness are maintained, with the wholeness being autonomy, identity, and agency and partness being that it fits in as a part of something else, with communion, and that it ceases to exist if it fails at either one. I gave myself the chance to think if I know anything in this world that is not a whole or a part, but I couldn’t think of a single element, activity, action, person, object, nothing. This shows that literally, everything that exists has the same horizontal capacity, else, it wouldn’t exist.

The vertical capacities were more difficult for me to understand, however, with a focus on the higher level of self-transcendence and a lower level of self-dissolution. He speaks about how evolution, one of the broadest and most general topics that we know is a process of transcendence as it is beyond what we know and could ever comprehend and this shows that there is a unity to evolution.

Relating all such ideas to our class and assignments is very convenient. As I look at my breadboard and everything connected to it, I can see how all of the whole components are a part of a larger technological artwork or piece in general. They all have their purpose individually but when put together can create projects that intrigue hundreds and thousands of people. The same goes for the shaders, tidalcycles music, and pixels, all of them are wholes and they’re tiny bits and pieces of memories, but as we know, they can all be connected together to make huge projects.

For this assignment, I decided to combine the elements I liked from the “Mask blur threshold” visual as well as “Shader Intro”.

When I was watching the lecture, the most intriguing sample for me was the “Shader Intro” one when the background color was a gradient, so I decided to use this color and add it to the “Mask blur threshold” sample and add my own video to it.

The video I uploaded on the sketch was one of my friends and I on a ride in IMG Worlds, an indoor amusement park in Dubai. I chose it because it’s a fun video that I wouldn’t mind rewatching and the colors of it went along with the gradient. I also added a fun sound that felt colorful to me.

This is the recording of me and the combined sketches:

While recording, I tried to match the sound, movement of the mouse (what shows on the screen from the video I uploaded), and my movements altogether. The video from IMG had some lights on the ride and so they also matched the gradient colors. I felt like, overall, the song as well as the video went smoothly.

I also had to retake the video a few times because there were often objects in the background that weren’t substituted until I found the perfect spot where I set my camera screen backward.

Something I found surprising, however, was that my shoulders didn’t show, I was wearing a cold shoulder top and so the camera did not recognize that it was the rest of my arm, even though my arms were recognized and they are the same color.

This has probably been one of the most interesting assignments for me so far, with the implementation of visuals and the flow of the color changes with the emergence of sound.

With the current common use of thermal cameras all around us due to the coronavirus situation, the possibilities seem to be endless in terms of taking such temperatures and of people and introduce interactive experiences.

Initially, I tried changing up the colors to try to make the block colors different and the changes of different shades, however, the program would not run on the browser when I did that. I tried to understand a few lines of the code to see if I could figure it out but I couldn’t.

I had the chance to explore a huge number of sounds for this assignment and gathered some that would fit together and sound good.

The video was a bit difficult to make as I tried to trigger the different sound areas separately when I was physically moving but it was easier for me to just use my hand and stay away from the sensor for the individual sounds and then use my whole body.

My main focuses here were close to block 0 and block 56. The goal was to present a landscape where the area close to block 56 is that of a childcare service or kids playing with pebbles and toys whereas the area closer to block 0 was a party for adults.

The right side of the screen was not my focus and I noticed after finishing the recording that I set up block 27 with a sound instead of block 7. Block 63 was a back to reality type of space where it seems to be a timer about to go off, maybe for the part and maybe for the kids.

I also moved in a way to trigger the sounds of the different areas combined as if it was a bird view of all of what was going on at the same time, as I also showcased all of the sounds at the same time as well.

The author starts by distinguishing how a person’s body is perceived; a human body from the outside, in third-person, and a human soma from within in first-person. The idea of the soma seems to be very interesting as I never heard about it before, and it reminded me of random shows that I watched as a child. A statement that helped me understand the idea was when he clarified that “the mode of viewpoint is different: it is immediate proprioception – a sensory mode that provides unique data.”

The way he breaks down the third-person and first-person view of the body makes it simple and easy to understand. The idea of how first-person is factual and more grounded would apply specifically to the individual, the third-person view however would require following sets of principles and multiple observations to ensure the accuracy of the results. For example, he talks about how psychology data is immediately factual and unified whereas in third-person it needs to be analyzed and interpreted in order to reach a factual conclusion.

Steps to understand somatics:

1. Understand that somas are not bodies
2. Recognizing that self-awareness is a distinction of the human soma

In regards to the first step, the author discusses that the sensory-motor system leads to a unique way of learning. He states that self-regulation is reached through the unity of sense with acting and acting with sense. According to him, such self-regulation is vital for human survival as the internal soma’s process of self-regulation ensures the existence of the external body structure.

As for the second step, he talks about how self-sensing and self-moving are interlocked in a way where they make up the core of somatic self-organization and self-adaptation. This unified experience that is observed from the first-person point of view makes the soma distinct from the body.

Two of the prime somatic functions are awareness and consciousness, as seen in the previous paragraphs. Consciousness is a relative function that is voluntary, based on a person’s interests and the skills that they wish to develop and it cannot perform more than its self-imposed limits. Awareness, however, could be focused and is used as the only way for the soma to isolate perceptive events. It also works to isolate “new sensory-motor phenomena in order to learn to recognize and control them.

He finalizes the chapter by looking at the relation between somatic learning and sensory-motor amnesia. Somatic learning broadens our range of action and perception and hence this increases the voluntary consciousness and adaptation to the environment. He perceives this as, either, a response to amnesia or just a normal day-to-day experience to avoid the effects of stress.

The optimal human state would be when somatic freedom is reached, not only because internally, from the first-person perspective progress without distortion, but also from the third-person perspective it is when the body portrays maximal efficiency and minimal entropy, meaning decline or collapse.

I’ve noticed that every time I go through the built-in sounds and beats to try them out, I end up choosing sounds for like a movie scene rather than a dance scene.

I imagine a situation and I apply the sounds accordingly. For this assignment, I pretended that it is a scene from a haunted house where the sound increases if you get closer to a room and decreases when you move away. The third sound I had, for the heading, would speed up and slow down, this was my way of trying to make it more suspenseful.

The first half of the video shows all three sounds played at the same time and the different combinations that can occur. The second half displays each sound separately to know which one is doing what exactly.

I found the calibration on MotionCal very interesting for some reason so I wanted to attach an image of it below haha.

• 

I am still having trouble figuring out how to use the internal mic from the laptop rather than the whole MacBook with the external sounds. The trial I did ended up not recording sound at all so I need help in that.

I also wanted to make it more interactive by showing what I am doing but I didn’t know how we can add a video of ourselves and display it at the same time. I tried using my phone to record but still couldn’t add the recording of myself to the screen-recording.