Understanding your assumptions and thoroughly questioning them is crucial to building complex systems. I’ve found that the vast majority of mistakes I make in hardware are where I let my intuition develop an argument for why something is occurring. Intuition easily justifies the bad assumptions that thrive in our knowledge gaps.
A few months back a teammate and I ran into a bug where a sensor (IR photointerrupter) was counting way too many times when it was triggered. It was a particularly weird bug because we had another set of these already implemented and they didn’t have this problem. We started looking into the code to see if it was issue with how we were counting the interrupts.
After some careful examination of the software, we started looking at the physical hardware. The physical hardware seemed fine. The voltages were exactly as expected when the sensor was blocked and unblocked. I hooked up the outputs to a digital logic analyzer and blocked and unblocked the sensor. Sure enough, leading up to the actual pulse were lots of tiny pulses increasing in duty cycle. There were also tiny pulses trailing the actual pulse, decreasing in duty cycle. We went and checked our correctly functioning implementation and surprisingly the small leading and trailing pulses were there too! But for some reason they were being counted properly by the MCU. We were confused. So we attached analog leads to the output of our sensor. We blocked and unblocked it, expecting to see the false pulse train leading up to the actual pulse. Nope. Perfectly clean rising and falling edges on each of the implementations, exactly the same way. We were very confused.
This is painful to recount because the solution now appears so obvious in hindsight. We had evidence suggesting that our MCU was performing some strange counting of the rising and falling edges of our actual pulse. It was right in front of us. We had data showing that the duty cycle of the false triggers were increasing as the it came closer to the actual pulse. What were we missing?
We had made a major assumption that analog-to-digital conversion on the logic analyzer would be able to get a clean reading of the sensor output being pulled high and low. We assumed that the measurement tool we were using wouldn’t also be having false triggers. We were really thrown off the trail because we had another implementation of the same sensors functioning properly. All of this combined, we let our intuition grasp for answers rather than consider the facts.
After a bit of discussion, some forum reading, and more analysis of the signals we figured out the problem. The Schmitt Trigger implemented on both our logic analyzer and on our MCU were getting triggered by the rising and falling edges of the sensor being blocked and unblocked. This is a pretty common occurrence and it can usually be resolved by low-pass filtration in hardware right before the signal enters the MCU.
Schmitt Trigger Edge Detection
The reason the count on the other implementation appeared correct was because we were performing quadrature encoding with the sensors and we would increment in one direction and decrement in the other direction. The false pulses one on sensor offset the false pulses on the other since these were being used in conjunction. This is a little technical and very specific to our application but it basically masked the fact that the sensors had all the extra triggers because the end result always appeared correct.
Quadrature Encoder Waveforms
We were really confused by all the evidence presented to us simply because of the assumptions we were making. Multiple logical explanations could be presented but would conflict one another. Keep an eye out for these type of inconsistencies, as they’re a strong indicator of bad of assumptions.
The price we paid was to sit through a few painful keynote speakers and slideshows full of one-liners from Steve Blank and Charlie Munger.
“We want to do our part to make the world a better place.”
“We’re a public company that actually turns a profit, unlike our friends across the way at Salesforce”. I was wearing my Salesforce backpack, which got a couple of good chuckles from the people standing behind me.
“Feel free to grab some swag as you go, don’t walk out with the alcohol though.”
All laughs aside, it was a fun night of hanging out with all the other terrible tech people we Twitter-knew that were there soaking up the IPAs and cocktails. A group of six of us decided to head over to a nearby bar and continue our night. A few Moscow Mules and deceivingly small tequila shots and we were on our way. We sat down at a booth nearby the main bar, only to find a tin of Jenga blocks poured out onto the table. Looking around I noticed that all the other tables had different games and cards.
“It’s good we got literally the hardest game to play drunk.” We all had a good laugh. “Jesus, I’m shit at Jenga even when I’m sober.”
It turned out to be a blast. We ended up playing a variation of speed Jenga that resembles speed chess. The most interesting part was that people had written all over the blocks. The obvious vulgarities, so-and-so was here, and phone numbers, but also there were the open-ended questions that were clearly supposed to be used for a Truth-Or-Jenga variant.
Truth be told, it was the perfect thing to have at a table among friends. We had something to fiddle with and it got us talking and off our phones.
“This one says, ‘Who’s the biggest tool at the table?’” The table shook with our laughter as I was in the middle of extracting a snug bottom block and it all came down. “The fact that we need to debate this is bad.”
We all took turns with a pen drawing and squeezing our own messages onto the blank space available on the blocks.“What kind of message should I put on there? Go for the ‘wow, that is horrific’ or the kind of thing that makes someone soft smile?” I chimed in, “Yeah, I feel like I have so much bad karma to undo from arguing with strangers on Twitter that I should probably go with making a stranger smile.”
I had always hated Jenga growing up. It’s a game that's easy to get bored of and the setup time is usually longer than the time you spend playing. And it’s usually awful for little kids. They spent so much time and effort meticulously building up a tower of blocks only to have it all come crashing down. And if you’re playing with more than two people, there isn’t a winner, there’s just the idiot that knocked the blocks down.
Recently, I’ve used Jenga quite frequently as an analogy for the difficulties that people often face as aspects of their lives changes. The easiest changes are made near the top of the tower, the opinions we hold lightly, the people, places, and things we’ve only known a short time. Further down are the parts of us that carry more weight. They’re physically burdened by the weight of what rests above. Moving them could mean toppling everything that sits above, maybe even below. The manner is which they are removed has to be fitting. Sometimes a swift yank, sometimes a gentle tapping.
At the very bottom are the blocks fundamental to ourselves. They are not defined by our conviction or their severity but are directly tied to who we are. Our virtues and sense of purpose. Our health and our families. There is no easy way to change the position of these blocks without risking toppling the entire tower. Sometimes it is our decision, sometimes it isn’t.
What do you do when it all comes crashing down?
Well, you rebuild the tower just like the angry teary-eyed toddler. Meticulously, methodically, placing every last block back into place. Compared to the fraction of a second that it took come crashing down, how long does it take to build back up? Running your hands along the creases and gaps, straightening out the edges, well aware that if you want to play the game it will have to come tumbling back down eventually.
That is a harrowing truth that just about everyone hears but refuses to believe. Everything comes crashing down. It takes an indescribable amount of effort to create order from disorder and it is impossible to maintain. Refusing to rebuild and play the game is to live a life void of purpose and substance. To fill it with the superficial, to not care deeply enough about anything to a point where there isn’t even a tower to have crash down. The only solution is to proceed with great care and to learn to love rebuilding, because you’ll do it a lot.
“What did you put on your block?”
“It says, 'Make Love, Not Logical Fallacy.'"
“You’re an idiot.”
Today’s lecture is on finding energy and coenergy from electromechanical systems; basically the energy output of moving an iron object near some copper coils with current flowing through them.
- Coenergy as described by current i and flux linkage λ
- Most often the system in question, a motor or an electric generator at a power plant, are multiport — multiple energies and coenergies than can be described by the relationship as such
- In order to perform these calculations path integrals are computed going along lines OB -> BA or OC -> CA
Coenergy calculations are actually not too complicated. A couple of integrals and derivatives and you’re there.
“Now we will just compute the coenergy as it has a much simpler integration.” My whirring thoughts came grinding to a halt as I tried to rationalize his statement. It had been two years but path integrals were about the only thing I remembered from multi-variable calculus. My hand shot up abnormally quick, wobbling my coffee thermos on the desk. He gave a slight nod in my general direction.
“Are we able assume the path is conservative because the system is electrically linear? Or do we have to check it?”
He smiled. “We can’t assume the system is conservative. You can never assume any two paths are the same.” He wasn’t referencing electrical engineering anymore.
Picture a mountain. There are easier and harder ways up the mountain that ultimately lead to the exact same point. Life is very similar, it is full of these destinations and points with a limitless number of possible paths. The main distinction, and by consequence the reason life is so much harder than electromechanics, is that we can’t calculate anything.
You want to climb higher up this abstract mountain of life, up to the heights of your dreams and goals. You see your peers there and you know their stories — their hardships, and sacrifices. But you cannot know what your journey will be along the path they took because in real life, no path is conservative. Life isn’t something you can integrate with respect to time and space and reduce to a function that describes when and where to walk. No matter what steps you take, guessing and theorizing, you will never know if you are taking the best path forward. Even worse, when it is all said and done, and you have either failed or succeeded, you won’t be able to know what other paths would have resulted to.
Knowing the unknown path is futile, looking forward and even in retrospect. I sat through the rest of the class trying to decide how I should feel about this. Am I mad that I’ll never know what lies ahead and what could have been? Or would knowing torment me more than anything else ever could?
“It is imPERative you check that the flux linkage and current partial derivatives to make sure they match up. If they do not, your path is not conservative and you can’t solve for the energy values.”
“What can we do then?”
“Write down that the path is not conservative. But that’s it, nothing else.”