Playing Street Fighter with Body Movements and Machine Learning

Machine learning algorithms are all designed to find patterns between inputs and outputs. As long as an algorithm has enough training data — and that data is consistent — it should be able to identify the correct output for any newly-presented input. That principle can be applied to virtually any kind of data, including sensor readings. That’s how Charlie Gerard was able to setup a system to play Street Fighter using only body movements.

While video game consoles like the Nintendo Wii do rely on movement for gameplay, they don’t utilize machine learning. Wii games are explicitly programmed to look for sensor readings that fall within a specific range. Gerard’s project doesn’t have any such programming. Instead, it utilizes Google’s TensorFlow machine learning platform to deduce the body movement based on input sensor readings.

In order for that to work, the system first had to be trained with a lot of data. Gerard gathered the necessary data with an Arduino MKR1000 development board that is equipped with an MPU6050 accelerometer and gyroscope. For each gesture, numerous sensor reading samples are recorded. Those are then stored in a file that becomes part of the machine learning training set. Gerard had to repeat that process for each body movement gesture.

That training data was then processed and formatted to work with TensorFlow. 80 percent of the data was used to train the machine learning model, while the other 20 percent was set aside to validate the model. By looking at the sensor readings that correspond to a specific gesture, the model was able to learn the inputs (sensor readings) that correlate with the possible outputs (recognized body movements).

The final step was to use the machine learning model to play Street Fighter. The same hardware is used during play, and the model makes a prediction about a body movement based on the sensor readings. That prediction is then used to trigger the appropriate control command in the game. It appears that Gerard has only setup a few commands, but, with enough training data, this system could be used to control every aspect of the game.