The Objective
The objective was to liberate 3D digital sketching from the physical constraints of traditional peripherals like mice and electronic pens — bridging the gap between physical movement and digital rendering, allowing designers to “step into” their work and manipulate 3D space as intuitively as they would in the physical world.
The concept crystallised while exploring the Kinect sensor and its various capabilities — specifically how far it could be “hacked” to create a 3D canvas for quick spatial sketching.
Technical Evolution
Phase 1 — Geometry Engine (Processing)
Built the initial engine in Processing (Java). Created a system to plot points, calculate distances to form lines, and generate surfaces from three or more coordinates. Started 2D — established the core geometry logic before adding spatial depth.
Phase 2 — Depth Integration (Kinect)
Integrated Microsoft Kinect to transition from 2D to 3D. Mapped the sensor's depth data to the Z-axis, translating a user's physical reach into digital depth — creating a true 3D coordinate system for hand-based input.
Phase 3 — Gesture Logic & Custom Libraries
Explored OpenCV for finger-tracking to enable tool selection (mapping each digit to a drawing function). When existing libraries proved insufficient for the precision required in Processing, custom library functions were authored to improve detection accuracy.
Prototypes
Prototype progression — from 2D geometry engine to full 3D multi-modal canvas
Design Challenges & Pivots
"While finger-tracking was functional, it lacked the fidelity needed for professional sketching — false positives caused jitter in the 3D model."
To maintain the “hassle-free” feel while ensuring precision, the solution transitioned to a hybrid interface — Touch OSC on a mobile device for stable mode selection (switching between points, lines, and planes), while keeping the Kinect dedicated to high-motion spatial tracking and point manipulation.
Concept Video
Concept Video — 3D Drawing Canvas prototype walkthrough
Key Takeaways
UX insights
Gorilla Arm Effect Sustained mid-air gestures fatigue the arm quickly. 3D spatial UIs need to minimise held postures and allow users to rest between interactions.
Multimodal wins over pure gesture For complex creative tasks, combining tactile feedback (touch) with spatial movement (NUI) consistently outperforms pure gesture-based systems — each modality does what it does best.