Control PC games with body movements using webcam andPython+ MediaPipe

I’ve been experimenting with building a universal gesture-based game controller — the idea is to let you play PC games using your body instead of a traditional controller or keyboard.

It uses:

MediaPipe for body pose detection

OpenCV for image processing

A custom Python pipeline I wrote to map gestures → keypresses

A standard webcam (placed above the screen) — no VR hardware or special equipment needed

Here’s a short demo of it running with Ryse: Son of Rome: https://www.youtube.com/watch?v=0k5jMM-aK1M

I’ve also tested it on other 3rd-person action games (Assassin’s Creed, The Witcher, Star Wars Jedi Fallen Order). The system sends configurable keypresses based on detected body movements — for example:

Stepping to move the character

Arm movements to attack or block

Leaning to dodge or strafe

It works surprisingly well for games that emphasize movement and timing over precision aiming.

I’m sharing this because:

I’d love feedback on the technical approach

I’m curious what other applications people can think of (beyond gaming?)

I’d like to find a few testers to help refine the system and test across more games

Eventually I’d like to turn this into an easy-to-configure tool — but for now it’s a fun experimental project. If anyone here is interested in the code, the design tradeoffs, or trying it out, I’d be happy to share more details.