Learning to walk

The robot is apart on a shelf right now. The work is in simulation.

I train a policy with reinforcement learning: it gets a command, it moves the joints, and a set of reward terms score how well it did. Then I verify the trained policy against fixed criteria, and I film it, so I can see things the numbers do not describe.

The numbers below are all from simulation, 64 robots at once, over 25 seconds. None of them are from the real robot.

The right hand column is the one that matters, and it is the column most result tables leave out. Two different policies produced these rows. One is trained on flat ground and holds the line and the speed. The other is trained on slopes and does the climb. No single policy has done all of it.

The two pull against each other: train on hills and the straight line costs about 1.7 degrees more. Getting one policy to do both is the next decision, not a finished result, and until it is made this table has to be read a row at a time rather than down the column.

PlatePL-01

TitleSame robot in the simulator

ViewDetail

PlatePL-02

TitleRun 182 at 7,896 iterations

ViewClip

I built a whole software stack on a bad CAD export. The model the simulation trained against did not match the robot, so every policy on it was learning to walk as something else. In August 2026 I archived the lot to a git tag and started again from SolidWorks. Nothing carried forward.

Two smaller ones cost real time.

• A setting in the physics library was quietly clamping the commanded forward speed to at least 0.3 m/s in four attempts out of five, and taking its absolute value. So the policy never trained across the speed range I thought I was giving it. It trained at one speed, and every backward command got flipped to forward. It is the leading suspect for why speed tracking kept failing.
• The reward table on my training dashboard went stale once. I spent hours explaining a result against a scoring function the run had never used. Now the code refuses to start a run if a new reward term has no plain English note attached.

The policy currently trots: diagonal pairs of feet lifting together, the body falling between steps and being caught by the servos twice per stride. In simulation that is fine. On the real robot it asks hobby servos at 50 Hz, with backlash nobody has measured, to arrest 3.1 kg twice a stride. I do not think they will. So the next job is to steer it toward a crawl, where three feet stay planted and nothing has to be caught.

After that: a policy for getting up off the ground, then a rule for handing over between the two, then the sensors. Every joint gets a potentiometer so the robot can finally report where its legs actually are. The Pi has no analog inputs, so that needs an external ADC with twelve channels.

Putting a policy on the real robot is deferred on purpose, not blocked. The cost of deferring it is real: three numbers the simulation is guessing at, servo gain, backlash and loop latency, only get measured on hardware, and every policy trained before that is trained against guesses.