BarBuddy
Smart exercise companion for safer weightlifting
University of Washington – MSTI
PCB Design & Assembly
Data Collection & Machine Learning
UI/UX Design
April – June 2025 (10 weeks)
Oulu Zhang, Rebecca Huang, Steven Liang, Xinyu Wang
ESP32, ICM-20948 IMU, MediaPipe, Edge Impulse, scikit-learn, KiCad
TL;DR
Problem
Gym-goers often lack affordable, accessible guidance on form. Incorrect posture leads to injury risks and ineffective training. Sensor-only solutions lacked context and didn't generalize well.
Solution
A hybrid sensing system combining IMU + webcam, powered by MediaPipe pose estimation and machine learning models. Users receive real-time feedback through a custom web interface that overlays pose landmarks and shows sensor data.
Impact
Context
As part of our TECHIN 515 course at UW MSTI, we set out to tackle a common challenge in both gyms and home workouts: many people struggle to maintain proper exercise form without professional guidance. This can lead to higher risk of injury and less effective training outcomes.
Initial Approach
Our first idea was to rely only on an IMU sensor to classify movements. But we soon realized the limitation — motion data alone lacked context, and it was hard to know if the exercise was truly correct across different people.
Hybrid Solution
To address this, we pivoted toward a hybrid solution: combining IMU data with visual input from a webcam. By using MediaPipe for pose estimation and machine learning models, we were able to design a system that provided clearer, more accurate real-time feedback for both gym-goers and people exercising at home.
"Without affordable guidance, people often lift with unsafe or ineffective form."
Goals
Correct Form Detection
Reliably distinguish correct vs. incorrect movements during weightlifting exercises.
Accessible Training Aid
Design a low-cost, portable solution that doesn't require expensive personal trainers.
Hybrid Accuracy
Combine IMU + vision data for better results than single-sensor approaches.
Real-Time Feedback
Deliver immediate, actionable insights during exercise sessions.
Approach
Iteration 1 — IMU-only Prototype
Idea: Use a single IMU sensor to determine whether exercises are performed correctly.
Our Idea Sketches
External feedback and testing showed IMU data lacked context, and the model couldn't generalize beyond a few individuals.
Iteration 2 — ESP32-CAM (ESP32-Sense)
Attempt: Add a camera on ESP32 to bring in visual information.
- • Very low resolution, unable to capture movement details.
- • Required two boards (one for IMU, one for camera), making the design bulky.
- • Conclusion: Not a practical solution.
Iteration 3 — USB Webcam + MediaPipe
Improvement: Switched to a standard USB webcam combined with MediaPipe for skeleton keypoint detection.
- • Clear posture estimation, much better exercise recognition.
- • Combined with IMU data, the system provided richer contextual feedback.
- • Tradeoff: Camera angle must stay consistent between training and inference.
Iteration 4 — Display & Computation Upgrade
Problems: The OLED screen was too small, feedback wasn't user-friendly. ESP32 lacked the compute power to run ML models.
- • Built a web interface to display real-time video, skeleton overlay, IMU data, and predictions.
- • Shifted training and inference to a local computer.
- • Collected 3,600+ data samples.
- • Achieved 91–98% accuracy in real-time exercise classification.
Live Demo
Data Collection & Performance
Final Outcome
Sensor and Circuit
Physical Implementation
3D Printed Housing
Complete Assembly
App Demo
Live application demonstration
Impact
Accuracy
Classification performance in real-time testing
Dataset
Labeled samples collected for training
Interface
Web UI with live feedback
Learning Outcome
Feasibility of affordable workout guidance
Reflection
Hardware Pivot
ESP32-Sense → USB webcam improved reliability and opened new possibilities for computer vision integration.
Software Pivot
On-device inference failed → shifted to laptop processing, teaching us about computational constraints.
Research Challenge
Data collection & labeling were time-intensive but crucial for achieving high accuracy results.
Key Takeaway
Iterative problem-solving led to a balanced hybrid system, showing both promise and limits of real-time fitness feedback tech.
What I Learned
This project taught me the importance of flexible problem-solving in technical design. Our willingness to pivot from pure hardware to hybrid solutions, and from on-device to distributed processing, ultimately led to a more robust and practical system. The experience highlighted how constraints can drive innovation, and how user feedback should guide technical decisions rather than the other way around.