Portable IMU physics companion for ESP32-S3

Source code: https://github.com/intellar/ImpulsePlay

To clone the repository:

git clone https://github.com/intellar/ImpulsePlay

Updates:

• 2026-06: physics engine extracted into a dedicated module, with collision regression tests

• 2025-06: match-3 ball game, explosive balls, score and high score saved in LittleFS

• 2025-05: impulse-based physics engine with squash & stretch deformation

• 2025-04: animated eyes driven by IMU, with laser target mode

This project is a portable interactive companion built around an ESP32-S3, a color touch screen, and a BMI160 IMU. Tilt the device and balls roll with gravity. Shake it and inertia kicks in. Touch the screen to grab, throw, or delete balls. In another mode, a pair of cartoon eyes reacts to movement, blinks, and squashes against the edges of the screen.

The goal is not just a demo of sensors on a display. It is a small object you can pick up, move, and play with — a tactile companion that feels alive because physics, touch, and motion are tightly coupled.

The firmware runs on an ESP32-S3 Super Mini with a 240 × 320 ILI9341 touch display and a BMI160 6-axis IMU. It is developed in PlatformIO with the Arduino framework, using TFT_eSPI for rendering and LittleFS for persistent high-score storage.

Hardware

The current build targets an ESP32-S3 Super Mini (4 MB flash, 2 MB PSRAM) wired to a rectangular ILI9341 SPI display with an XPT2046 resistive touch controller.

Main components:

• ESP32-S3 Super Mini (N4R2)

• ILI9341 TFT display, 240 × 320 pixels, SPI + touch

• BMI160 IMU on I2C

• LiPo battery with voltage divider on ADC

Display wiring (HSPI)

ESP32-S3 GPIO

Display pin

11 MOSI

12 SCLK

13 MISO

9 DC

8 RST

5 CS

4 Backlight (BL)

1 Touch CS

The display uses the HSPI port at 80 MHz. Color order is configured as BGR with byte swapping to match common ILI9341 modules.

IMU wiring (I2C)

ESP32-S3 GPIO

BMI160

6 SDA

7 SCL

The BMI160 provides accelerometer and gyroscope data. A complementary quaternion filter separates slow tilt (gravity) from fast motion (inertia).

Battery monitoring

ESP32-S3 GPIO

Function

2 Battery voltage via resistor divider (×2)

Battery percentage is estimated between 3.20 V (empty) and 3.79 V (full) and shown in the top bar alongside the frame rate.

This wiring can be reproduced on a generic ESP32-S3 dev board with a compatible ILI9341 touch module. The pin assignments are defined in Firmware/platformio.ini via TFT_eSPI build flags.

Software

The firmware lives in the Firmware/ folder. Open it as a PlatformIO project.

Main libraries:

• TFT_eSPI — display and touch rendering

• BMI160_SensorAPI — IMU driver

• ArduinoJson — included for future extensions

Main software blocks:

The project supports two application modes:

• MODE_BALL — physics sandbox and match-3 game (default at startup)

• MODE_EYES — animated eyes reacting to IMU data

Within ball mode, two game modes are available from the menu:

• Game mode — score, color matching, explosive balls, game over timer

• Simulation mode — physics only, no scoring or game over

How the Code Works

The firmware follows a simple loop: read touch input, process IMU data, then render the active mode.

Touch handling runs every frame with priority order: menu clicks first, then the gear icon, then game gestures (grab, drag, throw, delete zone).

The IMU module uses a complementary filter (98% gyro, 2% accelerometer) to maintain orientation as a quaternion. From that, it produces:

• a gravity vector used by the ball physics

• gyro rates used by the eye deformation system

This separation is important: tilting the device slowly changes gravity direction, while shaking it adds inertia without confusing the two effects.

Physics Engine

Ball simulation uses a fixed-timestep impulse-based 2D engine running at 100 Hz with 2 substeps per frame. This keeps collisions stable even when the display frame rate varies.

Constants (defined in physics_engine.h):

• Ball radius: 35 px

• Restitution (bounce): 0.7

• Air friction: 0.99

• Gravity force: 1800

• Inertia sensitivity: 5.0

Each frame, the engine:

1. Accumulates real elapsed time (capped at 0.1 s to avoid the “spiral of death”)

2. Filters the IMU gravity vector (EMA α = 0.3)

3. Computes linear acceleration as gravity_vec − filtered_gravity

4. Applies gravity + inertia forces to each active ball

5. Integrates position and velocity

6. Resolves wall collisions and ball–ball collisions

7. Updates explosion particles

gravity_vec + filtered gravityapply forces + integrateclamp + bounceresolve_collision impulsesquash_v updated on impactBMI160physics_update_fixedBall stateScreen edges

Ball–ball collisions follow a classic impulse resolution (similar to simplified Box2D):

1. Positional correction (80% of overlap) to prevent balls sinking into each other

2. Impulse along the collision normal if objects are approaching

3. Visual squash state updated from impact strength

The physics module was extracted from ball_animation.cpp into physics_engine.cpp, with a standalone collision test in Firmware/test/physics_engine_collision_test.cpp.

Squash and Stretch Deformation

The project uses two complementary deformation systems — one for balls, one for eyes.

Balls

Collisions do not deform the physics shape — balls remain circles of fixed radius. Instead, each ball carries a visual state:

• squash_nx, squash_ny — impact direction

• squash_v — impact strength

On render, the ball is drawn as a rotated ellipse:

• compression up to 30% along the impact normal

• stretch along the perpendicular axis to preserve approximate 2D area

• fast decay (×0.9 per substep) for a short, punchy effect

Because TFT_eSPI has no native rotated filled ellipse, the firmware implements draw_rotated_ellipse() using scanline rasterization — one square root per horizontal line, O(height) per ball.

Eyes

Eye deformation is parametric and proactive, driven by gyroscope and tilt rather than collision impulses:

• fast rotation squints the leading eye and widens the trailing one

• edge overflow against a virtual wall compresses width and increases height

• idle animations add blinking, look-around, and subtle wobble

• a “dizzy” state triggers after rapid yaw rotation

All deformation values pass through SmoothedValue EMA filters to avoid IMU jitter.

Ball Game

In Game mode, the screen becomes a physics-based match-3 puzzle.

Gameplay:

• Up to 12 balls of different colors can exist on screen

• Touch an empty area to spawn a new ball; touch a ball to grab it

• Drag and release quickly to throw the ball with gesture velocity

• Drag a ball to the top-left corner (50 × 50 px) to delete it without explosion

• When 3 or more same-color balls touch (within 2.1× radius), they pop with a particle explosion

• Larger groups score more: 100 base + 100 per extra ball

• About 1 in 10 new balls is an explosive bomb (white)

• Tap a bomb to arm it; after 2 seconds it detonates, destroying nearby balls (+50 pts each)

• A 15-second patience timer resets after each match — if it expires, game over

• High score is saved to /highscore.txt on LittleFS

In Simulation mode, the same physics runs without scoring, bombs, or game over. Useful for testing motion and collisions.

The in-game menu (gear icon, top-right) lets you switch between Game and Simulation modes and reset the high score.

Eyes Mode

Eyes mode renders a pair of cartoon eyes on a dedicated sprite band. The eyes:

• follow device tilt

• squash and stretch with gyroscopic motion

• blink and look around when idle

• enter a dizzy state after fast rotation

A secondary laser target mode can be activated for a charging-and-firing animation tied to IMU orientation.

Mode switching between BALL and EYES is implemented in code (current_mode in main.cpp). The on-screen toggle button is currently commented out in the touch handler but can be re-enabled easily.

Touch Interaction

Touch gestures are handled in display_handler.cpp:

Gesture / Action

Tap empty space / Create or grab nearest ball (40 px radius)

Drag / Move ball with finger

Quick release (< 100 ms history) / Throw with computed velocity

Release in top-left 50×50 zone / Delete ball silently

Tap explosive ball  / Arm bomb fuse

Tap gear icon / Open / close menu

Tap during game over / Restart game

A 200 ms debounce prevents accidental double-taps after menu interactions.

Rendering and Performance

The ball mode uses full-screen double buffering via a TFT_eSprite canvas pushed each frame. This eliminates flicker on the SPI display.

The top bar shows:

• battery voltage and percentage

• current FPS

Typical performance on ESP32-S3 is around 20–25 FPS with multiple balls, particles, and UI. The main rendering cost comes from rotated ellipses and particle drawing. The physics engine itself is lightweight compared to rendering.

Compilation and Upload

The project is built with PlatformIO. From the repository root:

pio run -d Firmware

pio run -d Firmware -t upload

Or in one step:

pio run -d Firmware -t upload

To specify the USB port:

pio run -d Firmware -t upload --upload-port COMx

Serial monitor (115200 baud):

pio device monitor -b 115200 -d Firmware

PlatformIO environment: esp32-s3-supermini-n4r2 (see Firmware/platformio.ini).

On first upload, LittleFS is used automatically for high-score storage. No separate filesystem upload is required unless you add assets under data/.

Project Structure

Companion_portable_ESP32/

└── Firmware/

   ├── platformio.ini          # Board, pins, libraries

   ├── include/

   │   ├── physics_engine.h    # Ball/Particle structs, collision API

   │   ├── ball_animation.h

   │   ├── eyes_animation.h

   │   ├── display_handler.h

   │   ├── imu_handler.h

   │   └── ...

   ├── src/

   │   ├── main.cpp            # Main loop

   │   ├── physics_engine.cpp  # Fixed-timestep simulation

   │   ├── ball_animation.cpp  # Game + rendering

   │   ├── eyes_animation.cpp  # Eye deformation

   │   └── ...

   └── test/

       └── physics_engine_collision_test.cpp

Why This Project

Many ESP32 demos show sensor values as numbers on a screen. This project goes further: the IMU directly drives a playable physics world. Tilt becomes gravity. Shake becomes inertia. Touch becomes interaction. The result feels like a small companion object rather than a sensor dashboard.

The same hardware stack could live inside a handheld toy, a desk gadget, or a robot face prototype. The eyes mode shares DNA with other Intellar animated-eye projects such as the Animated eye OLED demo, while the ball game explores real-time physics and game logic on a microcontroller.

For a more integrated hardware platform with FPC satellite boards, see the Intellar Engine — the companion firmware could be adapted to that ecosystem in a future revision.

Current Status

Working today:

• ILI9341 touch display on ESP32-S3 Super Mini

• BMI160 gravity and inertia for ball physics

• Impulse-based collision engine with squash & stretch rendering

• Match-3 game with bombs, score, and persistent high score

• Simulation mode for physics-only play

• Animated eyes with IMU-driven deformation

• Battery monitoring and FPS overlay

• Extracted physics module with collision tests

Known limitations:

• Frame rate drops with many balls and particles (rendering bound)

• BALL / EYES mode switch is in code but the on-screen button is disabled

• No wireless control yet (unlike the Intellar Engine Bluetooth dashboard)

• High score writes to flash on every match (could be debounced)

Planned improvements:

• Re-enable mode toggle button on screen

• Pre-rendered ball sprites to reduce ellipse cost

• Optional Bluetooth control for remote play

• Port to Intellar Engine satellite display board

Links

• GitHub: https://github.com/intellar/ImpulsePlay

• Intellar store: https://intellar.square.site/

• Related project — Animated eye OLED: https://www.intellar.ca/blog/animated-eye-oled

• Related project — Intellar Engine: https://www.intellar.ca/blog/intellar-engine-satellite-boards

This project is not possible without your support:

https://intellar.square.site