What we learned by delaying v1.4 for ten days
We finished building our first 50 v1.4 machines in early September, about a month ahead of schedule.
During our temperature accuracy test, we found that about 30% of the machines were not acting as they should, with water temperature about 2ºC above the goal temperature:
All 50 machines were identical, so what could be the problem?
We tried a variety of theories.
- Maybe the flow constrictors were mis-mounted?
- Maybe the temperature probes were of varying quality?
- Maybe the pumps were defective?
- None of our theories were right.
So, we proceeded to take 1 part out at a time of a “good” machine, and replace it in the “bad” machine, until the problem re-appeared.
The problem finally re-appeared when we replaced the section where “water mixing” happens. That's where 110ºC superheated water meets room temperature water, at varying flow rates, so that we can give you the exact water temperature you want.
So then we started taking the “bad” parts apart, and measuring everything. We couldn't find any difference between the “good” and “bad” part.
After 4 days of delays, I decided to put all 50 machines aside, and start building another 50. I then assigned 3 engineers onto the problem full time here in Hong Kong, while Ray (Seattle) and Ben (Australia) also worked on this issue remotely.
We eventually figured out that no component was defective. The problem was a temperature sensor we'd moved.
We had moved the temperature sensor closer to the point where the water mixing happens. The new position gave us slightly faster sensor feedback. We'd been testing this for almost a year with no problems, so why does it not work now?
We finally figured it out.
The new temperature probe location is right where the hot and cold water mix.
It turns out that teeny tiny changes in the angle of water flowing into this mixing chamber, caused big changes in how well, and how evenly, the two different water flows mixed.
With slight differences in angle, the new temperature probe location was measuring ever so slightly more cold water than would eventually be in the final mix. And therefore our mathematics were compensating for that cooler water by putting warmer water in. Which then gave us 2ºC hotter water, when finally measured at the coffee puck.
The bright yellow and green water tube in the upper right part of the photo, is carrying mixed water to the “old position” temperature sensor. That tube is only 1.2mm wide. There's not much water in there.
It turns out that squeezing the mixed water from the chamber, through the tube and back into a chamber, causes the water to be thoroughly and evenly mixed. Measuring the water temperature earlier gave use readings that still had hot and cold eddies flowing around.
Once we figured this out, we moved the temperature probe back to the old position, and all our machines now worked as expected.
The exercise, though stressful, was quite worthwhile. We learned something new about measuring water temperature, and mixing, and eddies.
In our latest firmware (about to be released), insights from these ten days have led to revisions to our mathematics, and specifically we now are able to stabilize our coffee puck temperature significantly better, with very slow “thick” shots that have very little added water through them. This has been a case where we typically ran about 1ºC too cool at the end of the shot, and we'd not been able to understand why. Our better understanding of water mixing dynamics has helped Ray to crack that problem.
We've never before achieved this level of “measured at the puck” temperature accuracy, with such slow shots (in my test, a 38 second espresso).
Note how, in the photo, the puck temperature is only 2ºC too cool at the start, thanks to our slight 2ºC over-temperature bump for the first 2 seconds. Traditional machines typically start 6ºC to 8ºC cooler than the goal, because the puck is at room temperature. And then the rest of the shot manages to stay with 0.3ºC at the coffee puck, during the entire shot. I'm happy!
#testing #factory #improving #growth
Updated 2020/10/05