Converting reins input to key presses for Unbridled: That Horse Game

In a recent post, I got to the point of using the reins to control Unbridled: That Horse Game -- but only as mouse input, which moves the rider's head, rather than the horse's head.  To fix this, I need a way to take mouse inputs and convert them to keyboard key presses.  It looks like this is possible with the Arduino Uno I am using, by loading some special firmware onto the Atmega8/16u helper microcontroller on the board.

Well, actually, it looks like it is possible for the Arduino Uno, but not the Arduino Uno Wifi Rev2 that I have been using for reading the sensors so far. Bummer!

Fortunately I have an Arduino Uno lurking about as well, so I should be able to do it after all.  The regular Uno has the extra connector required for reprogramming the bootloader on the MEGA8U microcontroller.  Information on how to install it can be found here.

Hmm, nope, that won't work either -- it used to work, years ago. Now the preferred approach is to use an Arduino Leonardo board, which is designed to act as a USB HID.  So third time lucky...

And I am able to create synthetic keyboard events. Bizarrely it doesn't work as a keyboard on Linux, but does under Windows.  For now, that actually suits me just fine, because I can develop with it on my Linux machine, without it providing crazy keyboard inputs, but then plug it into the Windows machine, and have it automagically start working.

Actually, it turns out that it does work on Linux, but ignores key presses during the first short time frame.  I found this out when I accidentally messed things up so that the virtual keyboard was typing stuff all the time.  This turned out to be a bit of a pain, because I couldn't use the Arduino GUI to fix this.  But it was possible from the command line:

1. Hold reset button down on Arduino while plugging it in

2. Then from command line from in the Arduino install directory: 

./resources/app/node_modules/arduino-ide-extension/build/arduino-cli upload -b arduino:avr:leonardo --port=/dev/ttyACM2  some_safe_and_simple_sketch.ino  

That uploads that sketch, replacing the problematic one.

So now to tie things together, starting with reading the mouse movement under Linux, and converting that into keyboard events to be fed through the USB port of the Arduino.  I'll start by making an Arduino sketch that basically generates synthetic key events from characters coming in the serial port.  I'll deliver that serial data to the Arduino using a USB serial adapter, of which I have several.

I now have the code so that it can read from the serial port, and trigger keyboard or mouse events. It's pretty simple in the end:

#include <Keyboard.h>

#include <Mouse.h>

#include <SoftwareSerial.h>

// Software 2nd serial port on pins 8 and 9

// RX pin has to be change interrupt capable. See:

// https://docs.arduino.cc/learn/built-in-libraries/software-serial

// (used to receive instructions for synthetic key/mouse events to generate)

SoftwareSerial mySerial = SoftwareSerial(8,9,false);

void setup()

{

Serial.begin(9600);

Keyboard.begin();

Mouse.begin();

mySerial.begin(9600);

delay(200);

mySerial.print("UART to USB Keyboard/Mouse Relay");

}

int hex=0;

int digits=0;

void loop()

{

while (mySerial.available()>0) {

unsigned char c;

c = mySerial.read();

if (c>='0'&&c<='9'||c>='a'&&c<='f') {

// Hex digit

hex = hex << 4;

if (c>'9') { c-='a'; c+=10; } else c-='0';

hex |= c;

digits++;

} else if (c=='\n'||c=='\r') {

if (digits==3) {

char msg[80];

snprintf(msg,80,"Hex = $%03x",hex);

mySerial.println(msg);

switch(hex>>8) {

case 1: // Keyboard

mySerial.println("Press normal key");

{

char s[2];

s[0]=hex&0xff; s[1]=0; Keyboard.print(s);

}

break;

case 2: // Mouse X

mySerial.println("Update mouse X");

Mouse.move(hex&0xff,0,0);

break;

case 3: // Mouse Y

mySerial.println("Update mouse Y");

Mouse.move(0,hex&0xff,0);

break;

case 4: // Mouse Wheel

mySerial.println("Update mouse wheel");

Mouse.move(0,0,hex&0xff);

break;

}

}

digits=0; hex=0;

}

}

}

The Arduino just needs the serial USB adapter RX and TX lines connected to digital pins 8 and 9, and communicates at 9600 bits per second.

Now to make a program that runs under Linux and can read the mouse inputs from the horse head (that's how we currently read the head rotation), and from that generate the key sequences to rotate the horse head in Unbridled: That Horse Game. There are plenty of examples of how to read mice input on Linux, such as here. Basically you can open /dev/input/mice, and read 3-byte mouse status events. The first byte has button status, which we don't care about, and then the next 2 bytes have X and Y position change deltas.  

We have the mouse on the horse head hinge arranged so that it experiences X axis motion when the head turns -- so we just need to read the 2nd byte, and from that, determine the position of the head, and generate the necessary key inputs to Unbridled.  Playing around with Unbridled, I have made the pleasant discovery that the rider's head turns automatically if you are in motion on a horse and the horse turns.  So I really just need to provide the head turn key events -- and to know how long those key presses need to be to get the rotation speed about right.  Unbridled (currently) doesn't really have much range of head turn -- maybe two different amounts in each direction. So the approximation can be quite simple to begin with.

I'll try auto-scaling the mouse X position, since it should remain more or less symmetric as it rotates left and right.  Hopefully any drift will be random and relatively minor.

What requires a bit more thought is how to modulate the rate of turn based on the degree of head turn.  When the riding machine is finished and has motors, the head can resist turning if you allow the horse to lean on one or both sides of the reins.  But for now, the head always follows, so we can just have a rate of turn that is proportional to the deflection of the head from centre.  That in turn can be translated into pulse-width modulation of the appropriate key presses for left and right turning.  I just need to work out the minimum key press duration for it to be noticed by the game, and use that as a minimum when turning. It seems to be somewhere in the 100ms to 200ms range.

So a bit more fiddling about, and I have it generally speaking, working.

Here's a screen shot of the game running on the laptop, if only because blogspot doesn't like posts that don't have at least one image, and below, a 3 minute video showing it all in action:

 

I need to improve the calibration and reduce the latency, by making the head turn immediately when you increase the deflection, and likely a bunch of other stuff, but as can be clearly seen, the game is being controlled using a set of reins!

It's actually interesting how much more "real" the game already feels, by using the reins to steer the horse, instead of using two fingers on the keyboard to do it.  Just trotting through the forest gently steering your way around (even if the steering is still quite unnatural in many ways, because of the poor calibration etc), has such a nice relaxing feeling to it. It is already possible to suspend disbelief more than when using the keyboard to control the horse.

Probably what I'll aim to do next is to tie in the pressure sensors for leg aids, and also the mouth sensors I've been working on, so that you can tell the horse to speed up with your legs, and even do half-passes to the left and right, since the game supports those, and tell the horse to break by pulling back on the reins (yes, this is not really good horsemanship, and it will be replaced with using your weight and balance once I get those sensors in place).  But nonetheless, it will still be a significant milestone to be able to get the horse moving and control its gait, steer it, and stop it again, all without touching the keyboard.