05/10/2026
A real-time spatiotemporal transducer, converting optical motion into quantized acoustic frequencies and kinetic visual data.
The app analyzes motion using a discrete approximation of a partial time derivative applied to pixel intensity. For the RGB channels of any pixel at coordinates (x,y) near the “tripwire,” the system calculates the absolute temporal difference between the current frame t and the previous frame t−1.
If D(x,y,t) exceeds a defined threshold τ, it registers as kinetic activity. To prevent false positives from noise or wiggling, the app requires a directional spatial sequence: pixels in an adjacent Zone A must trigger just before Zone B, confirming a true geometric crossing vector.
When a crossing occurs, the geometric coordinate of the intersection is normalized into a ratio R∈[0,1]. This continuous spatial variable is mapped to a discrete frequency fn from a predefined musical scale array (e.g., Pentatonic, Lydian).
By quantizing the spatial axis, linear physical movement is translated into harmonic acoustic intervals.
The sound generation relies on additive synthesis and frequency modulation, constructing timbres using multiple oscillator nodes. The resulting sound wave S(t) is a summation of a fundamental frequency and its overtones (partials), shaped by an amplitude envelope A(t).
The intensity of the motion (the total mass of pixels exceeding τ) dictates the initial kinetic energy of the generated visual particles. The app assigns velocity vectors v=(vx,vy) to these particles, which undergo a linear dampening function over their lifespan, visually mirroring the exponential acoustic decay of the audio nodes.
