Hand eye motion tracking

New project : hand-eye motion tracking

Changes:

2025-01-30 the project description is in development.  I will post updates as is takes form.

In this new project, I take the initial animated eye project further so I can actually interact with the system using my hands.  The eyes will respond to a close presence by steering toward anything that is close enough. They will then follow it until it moves away. In addition, the eyes are mounted on a 6axis robot flange that will re-orient itself to center the object in its field of view.

There are three parts in this project, (a) the animation of the eyes drawn on a color lcd, (b) the detection of an object using a time-of-flight (ToF) sensor and the computation of its distance and direction,  and finally (c) the robot movement in the flange referential to re-orient the screen toward the direction recovered by the ToF sensor. I made a python simulator of the robot to visualize of the transformation of the screen lookat would translate on the robot inverse cinematic. 

Let see how each part works.

a) Eyes

The eyes implementation is recovered from  the animated eyes project, but I had to do some adjustments to switch between waiting animations and hand tracking. The drawing portion is straightforward and you should check the code for the details. The eyes are drawn as concentric ellipses with different colors. The first challenge was related to hardware limitation. In the original project, the eyes are drawn on a low resolution 64x128 oled screen using the versatile arduino nano. In this new project, I wanted to show a higher resolution of the eyes with colored shades of blue. This was possible by switching to a 320x240 color lcd. 

This in itself is not a problem, but in order to obtain a fluid animation, the eyes need to be redrawn fast enough to create the illusion of movement. I quickly came to conclusion that the Arduino Nano is not fast enough to redraw the eyes, even using the tricks of the animated eyes project.  Lets review them quickly.  Frist I redraw only the pixels that changed between two consecutive frames. I had to focus on the eyes region, not the whole screen. Secondly, a pixel must be changed only once as we draw aa new frame, to avoid flickering. So, as the concentric ellipses are drawn, special care must be taken to draw only the ring of the ellipse, not drawing each ellipses over each other from the biggest to the smallest one.   Finally, the spi protocol can draw horizontal (or vertical) line segment in a single call, which is way faster than drawing each pixel of the line in successive call. So  to accelerate drawing, we draw line segment forming the ellipse instead of pixels. 

StiIl, this was not enough to get a fluid animation. I needed something with more punch. For this reason, I used the low-cost ESP32 microcontroller. It is a fast and powerful device that can handle fast SPI communication for the display,  it has a similar cost to arduino nano, it is as easy to program, and it has a lot more memory to store the program. Remember we need to store the background image directly as text in the code. I make a python convertion script to accomplish that . You can find it in the tft_face_display  github repo here: 

Python/compression_test.py

With the esp32, I had to change some initialisation code to correctly handle the new microprocessor and its library. The drawing functions are very similay to the arduino functions I used. The conversion was easy since the code mainly use the function drawFastHLine, to draw line segment. All higher level abstraction functions like draw ellipses are using this low level function. 

b) ToF sensor

In the project, I was looking for a convenient way to interact with the system. A simple but boring way is to use push buttons to trigger events or presence detector. However, I wanted not only to detect, I wanted to interact, detect a presence and know its distance and direction from the screen. The VL53L5CX ToF sensor does just that. Its is a compact device that project a wide infrared signal (Fig 2b) in a field of view of  55x61degree (Fig.2a), and measures the distance of the return signal by converting the delay it takes to come back to one of its collectors. On the sensor, there is a grid of 8x8 collector than acts as directional detector. 

[...]

c) robotics