Vision-Based Drawing System

Problem

Traditional drawing tools require dedicated input hardware. The goal was to test whether a webcam alone could support a natural, contactless drawing interface — useful both as an HCI experiment and as an accessibility-oriented interaction model.

Solution

A real-time hand-tracking pipeline that maps finger count and fingertip position to drawing actions: one finger draws, two-to-four fingers switch color, and an open hand triggers the eraser — all composited live over the camera feed.

Architecture

MediaPipe Hands detects 21 hand landmarks per frame; a lightweight gesture classifier derived from landmark positions determines the active tool (draw / select color / erase); OpenCV composites the drawing layer onto the live video feed via NumPy array operations, frame by frame.

Challenges

• —Keeping gesture classification stable under variable lighting conditions
• —Reducing jitter in the drawn line caused by frame-to-frame hand-tracking noise
• —Mapping continuous hand position to a responsive, low-latency drawing experience

Learnings

• —Hands-on experience with real-time computer-vision pipelines, beyond static image inference
• —How much classic CV pre/post-processing (smoothing, thresholds) still matters alongside ML models
• —HCI trade-offs in designing a gesture vocabulary that feels intuitive without an on-screen tutorial