Jekyll2019-11-17T20:22:37-06:00http://www.tedinski.com/Ted KaminskiTed Kaminski's blog - a software design book writing project.Ted Kaminskitedinski@cs.umn.eduNovember book status update2019-11-17T20:16:35-06:002019-11-17T20:16:35-06:00http://www.tedinski.com/2019/11/17/book-status-update<p>Hello friends of the blog!</p>
<p>So you might have noticed the lack of updates for the last six months.
Short story: I got a new job, and then I got really busy learning all sorts of new things for about four months.
You know, typical new job stuff.
Turns out, it’s hard to write when your brain is melted and you just want to veg out.</p>
<p>I probably could have written something in the last couple of months, but all my good writing habits got wiped out.
I’ve got to invent some new ones and get back to it.
I’m still not 100% sure how I’ll do that.
The “once a week blog post” served me really well for over a year, but (a) I’m in a new phase of writing that doesn’t involve blog posts anymore, and (b) I don’t have quite that much time anymore.</p>
<p>(Even if I were still trying to write a new post each week, I’m not sure I could keep that pace up.
I experimented a bit, and while I used to get a post written in roughly 6–7 hours of writing time, it actually took up a lot more “thinking time” than that.
Each post probably took another 10 hours of poking about, searching for inspiration and mulling over ideas.
I don’t have that much “idle” time anymore!)</p>
<p>I’m going to figure out how to get back into things.
Here’s what I’m going to try first:</p>
<h2 id="re-outlining-a-small-book">Re-outlining a small book</h2>
<p>I’m going to try writing a book somewhat like I’d write software: start with a small thing that works, then iterate outwards.
To start, I’ll write a mini-book, hitting on just two main points:</p>
<ol>
<li>I’d like to start with core organizing ideas about design, and my “haven’t really thought about it in six months” brain comes up with two that are huge:
<ul>
<li><a href="/2018/02/06/system-boundaries.html">System boundaries</a></li>
<li><a href="/2018/02/27/the-expression-problem.html">Data, ADTs, and Objects</a></li>
</ul>
</li>
<li>Using these major ideas, I’d then like to critique current practices, ideas, ideologies. This will include some things like:
<ul>
<li>Critiquing <a href="/2018/01/30/the-one-ring-problem-abstraction-and-power.html">powerful abstractions</a></li>
<li>Critiquing <a href="/2018/02/13/inheritance-modularity.html">inheritance</a></li>
<li>Critiquing <a href="/2018/11/27/contradictory-tdd.html">test-driven development (TDD)</a></li>
<li>Critiquing <a href="/2018/05/08/case-study-unix-philosophy.html">the unix philosophy</a></li>
<li>Critiquing <a href="/2018/12/18/the-law-of-demeter.html">the law of Demeter</a></li>
<li>Critiquing <a href="/2019/04/02/solid-critique.html">SOLID</a></li>
</ul>
</li>
</ol>
<p>The basic theory of this approach is that either (a) I’ll learn this is the wrong organization (or is still too big of a chunk) and try a different approach, or (b) it could serve as the beginning of an overall organization that looks like:</p>
<ol>
<li>Present some new ideas</li>
<li>Break down existing practices</li>
<li>Build up from there</li>
</ol>
<h2 id="whats-next">What’s next?</h2>
<p>I’m going to try to get this mini-book in shape by the end of January, and if successful, I’ll probably share that with Patreon backers.
(Speaking of Patreon, it’s still going to stay on pause, y’all.)
The next steps look something like this:</p>
<ol>
<li>Write the mini-book.</li>
<li>Write up a draft outline of the remainder of the book.</li>
<li>Look for publishers, and get a contract.</li>
<li>Write the rest of the <a href="https://www.google.com/search?q=draw+the+rest+of+the+owl&tbm=isch"><del>owl</del></a> book.</li>
</ol>
<p>That last step might need some refinement. Eh, I’ll get to it later.</p>
<p>First things first: let’s see if I can get back to writing regularly now.</p>Ted Kaminskitedinski@cs.umn.eduHello friends of the blog!On ideological purity in programming2019-05-08T22:12:38-05:002019-05-08T22:12:38-05:00http://www.tedinski.com/2019/05/08/unexpected-moral-reasoning<p>One of the things that plagues this industry is the promotion of good rules of thumb into absolute rules.
What I’d like to do today is just muse a little bit on how this happens, and what we should do about it.</p>
<h2 id="moral-reasoning">Moral reasoning</h2>
<p>I’m no psychologist, so take this with a grain of salt.
But it seems to me there’s some part of our brains that does moral reasoning by a mechanism that’s different from how we do (for lack of a better word) “rational” reasoning.</p>
<p>In part, this is not a dysfunction, it’s an optimization.
To do moral reasoning in a “rational” way, you have to get really systematic about expanding out all the downstream consequences, intended or not, and even foreseen or not.
Which is… rather difficult to do.</p>
<p>Okay, just for the sake of clarity for those who might not like understated humor, that’s generally flat-out impossible.</p>
<p>Moral reasoning is a shortcut that says “Nope! Stop. That’s just wrong; don’t go that way.”
Don’t waste time trying to reason out the impossible infinite unknowable branching decision tree here, because we have a moral principle we trust that says “this will not end well.”</p>
<p>But it’s not <em>just</em> that, because this is the basic idea behind “thinking fast” and “thinking slow.”
Or in other words, intuition.
Intuition is fine; we’re already reasonably capable of recognizing when our intuitions have lead us astray.
More reliable intuition is part of what experience is supposed to give us.</p>
<p>Moral reasoning seems to be accompanied by some extra “unquestionability.”
We’re all able to follow our intuitions and then stop and think “ehh, maybe this isn’t working.”
But following “moral” principles can take us in a direction where we suffer the consequences and insist on sticking it out instead.</p>
<p>There’s even some kind of vague expectation that we’ll eventually be rewarded for sticking to the principles, even if there’s no reason to believe that’s actually true.</p>
<h2 id="ideology">Ideology</h2>
<p>I’m no… I don’t even know what, but take even more salt here.
What distinguishes an <em>ideology</em> from just any old set of beliefs, I think, is that an ideology specifies moral beliefs.</p>
<p>To use a nice uncontroversial example not directly related to programming, there’s “capitalism” and then there’s “Capitalism.”
First, we have a neutral mechanism for organizing an economy.
A purely mechanical thing about which we can study and make observations.
An intellectual curiosity.</p>
<p>The second tacks on <em>moral</em> judgments like “the outcomes of a free market are just” or “the shareholders <em>deserve</em> the profits.”
Understandably, when this ideology asks us to conclude “teachers and nurses <em>should</em> be paid poor wages,” people start wondering if maybe we should do away with that whole (at least big-C) Capitalism thing.</p>
<h2 id="technical-ideologies">Technical ideologies</h2>
<p>I bring all this up on a blog about software design because we suffer from this all the time in this industry.
The thing is, it’s ideologies about <em>programming</em>.
Ideologies de-facto introduce the idea that there’s a <em>“morally righteous”</em> way to write a program, as odd as that sounds.
I hope that sounds odd to you.</p>
<p>Object-orientation’s splash onto the scene came with a lot of associated ideology.
One of these was that <code class="highlighter-rouge">switch</code> was to always be avoided in favor of polymorphism and dynamic dispatch.
But this is just a blanket ban on <a href="/2018/01/23/data-objects-and-being-railroaded-into-misdesign.html">data types</a> which just cuts off a significant portion of the <a href="/2018/02/27/the-expression-problem.html">type design space</a> for no real reason.</p>
<p>I’ve written articles about the <a href="/2018/02/13/inheritance-modularity.html">flaws of inheritance</a>, but (in a way) even this proscription takes on ideological overtones.
I do think we’d be better off designing programming languages without the feature, but as long as it’s here, we can also look at <a href="/2019/04/16/composition-vs-extension.html">when it’s pretty safe to use it</a>.</p>
<p>“Goto considered harmful” has entered into the history of programming in such a big way, I don’t even feel the need to link to something as a reference.
But people sometimes get surprised when it’s used regularly in <a href="https://lwn.net/Kernel/LDD2/ch02.lwn#t4">the Linux kernel</a> and other C software, and for pretty good reason.
They’re even more surprised to discover that virtually every complaint about <code class="highlighter-rouge">goto</code> that Djikstra made actually doesn’t apply all that well to <code class="highlighter-rouge">goto</code> as it exists in C.
(Not that this makes <code class="highlighter-rouge">goto</code> good or anything, but the reasons <em>why</em> have to change!)</p>
<p>So much in testing has been turned into ideology.
TDD is <a href="/2019/03/19/testing-at-the-boundaries.html">sometimes</a> an effective practice, but it frequently get turned into a moral purity test.
Devotion to high coverage and unit-only (lest ye sin with integration testing?) tests can lead to “<a href="https://dhh.dk/2014/tdd-is-dead-long-live-testing.html">test-induced design damage</a>”.
My observation that <a href="/2018/06/12/types-can-be-pervasive.html">tests against non-system boundaries can be a kind of technical debt</a> still seems rather transgressive.</p>
<figure class="boxed">
<p>I think I’ve told this story before on this blog, but I’ve had to delete a unit test suite before.
It took me a long time to come around to accepting this because I kept thinking that this unit test suite was <em>good</em>, and that deleting it would obviously be <em>bad</em>.
But… it kept causing problems, and never really provided any protection against introducing new bugs.
It was another long time after I had gotten rid of it before I had any sort of “principled” justification for why that was the correct decision beyond, “look, it just breaks all the time for no good reason.”</p>
</figure>
<p>And the best ideologies are the ones we don’t even know we believe.
I routinely see automatic, unquestioned praise for “extensible designs,” but I happen to think <a href="/2018/08/07/what-makes-good-design.html">extensibility is a rather significant cost, to be paid only where necessary</a>.
It’s not automatically a feature of good design.</p>
<h2 id="why-do-we-do-this">Why do we do this?</h2>
<p>To some degree, I think it’s inevitable.
We’re trying to persuade humans to follow some rules in the short term in order to try to reap some longer-term reward we might not immediately see or appreciate.
That’s… a kind of persuasion that’s always going to touch a little on the “moral” side.</p>
<p>Partly, I think this kind of thing is desirable.
Like I said, this overlaps a lot with the idea of “thinking fast” versus “slow.”
We do actually need to build intuitions, and we need to be able to apply those intuitions.
The trouble is just that we so easily slip from “good rules of thumb” to “you’re bad for even questioning this.”</p>
<p>But another reason I think this happens to us is that these ideologies aren’t really invented in one go.
They evolve.
An ideology has to propagate somehow, and thinking about this in terms of evolutionary fitness is a bit enlightening.
One way to propagate pretty well in this industry (unfortunately) is to give people reasons to feel morally superior to those around them.
Adoption of the ideology becomes a “benefit” for more than just engineering reasons.</p>
<p>Of course, that’s also completely toxic.
Our especially strong susceptibility to this sort of “pecking order” crap is a bit of an indictment on programming culture.</p>
<h2 id="what-should-we-do-about-it">What should we do about it?</h2>
<p>The biggest problem here is that we’ve often arrived at ideological beliefs without really having <em>reasoned</em> our way into that position.
Ideologies are frequently taught unconsciously.
Either by mimicing the value judgments we see others make, or simply by explicitly being told to look at things a certain way before we’re really able to critically evaluate it.
Doing “this” is bad, and you should do “that” instead. Okay.
This thing is bad, and if I do it, I’m bad, right?</p>
<p>Experience and intelligence don’t necessarily save us from this, because it’s not fully conscious.
Experience and intelligence can perfectly well <em>just</em> help us construct ever more sophisticated rationalizations for the ideologies we haven’t questioned.
It’s not until we start to allow experience to point out the flaws in those ideologies that we can start to become wiser.</p>
<p>But pointing out the flaws in ideologies is fraught.
We have a moral reflex: if someone questions a moral principle, that’s a sign of their <em>immorality</em>, right?
Maybe that person questions TDD because they’re a bad programmer?</p>
<p>I can’t fix this for you, but I can make a suggestion.
If you find yourself about to say “no, that’s a bad and wrong approach” to a technical decision, instead apply the Socratic method.
Ask “why would we want to go with that approach?”
Ask if the approach has any obvious drawbacks, or obvious advantages.
Think, ask, explore, look.</p>
<p>For one thing, this often a more effective method for teaching a junior engineer.
If you’re actually quite right, you should be able to get them to see that by asking the appropriate questions.
You don’t need to hand down laws.
And who knows, maybe they’ll surprise you with a good answer.</p>
<h2 id="end-note">End note</h2>
<ul>
<li>
<p>As an aside, I’m not sure I really believe there are “idealists” or “pragmatists”, or that these are helpful ways of thinking about people.
One could misconstrue what I’ve said above as endorsement of pure pragmatism, but I wouldn’t agree with that.
The misuse of ideology in programming isn’t something an idealist would be interested in doing either.
This is a separate behavior, common to all.</p>
</li>
<li>
<p>I guess I’ve made something of an argument against “moral reasoning” about technical decisions, so I should clarify: ethics of course matter to developers!
But… the ethics of software is about the impact we have on the world around us, not <em>just</em> (e.g.) the narrow minutia of specific testing methodology.</p>
</li>
</ul>Ted Kaminskitedinski@cs.umn.eduOne of the things that plagues this industry is the promotion of good rules of thumb into absolute rules. What I’d like to do today is just muse a little bit on how this happens, and what we should do about it.Encapsulating mutable state2019-04-30T21:50:38-05:002019-04-30T21:50:38-05:00http://www.tedinski.com/2019/04/30/encapsulating-state<p><a href="/2019/04/16/composition-vs-extension.html">Last time</a> I finished with a request: whether anyone’s read anything about how “encapsulation of mutable state restores compositional reasoning.”
Someone (thx <a href="https://twitter.com/ericbb/status/1118928499417452546">@ericbb</a>) came through with <a href="https://semantic-domain.blogspot.com/2018/04/are-functional-programs-easier-to.html">a good link</a> (which also has a good <a href="https://www.reddit.com/r/tlaplus/comments/8f4j6j/are_functional_programs_easier_to_verify_than/">discussion on reddit</a>) that sort of danced around the topic, and it ended up giving me a good think.</p>
<p>The core proposition I’d like to advocate for is that compositional reasoning is gold.
It’s not the be-all and end-all of reasoning, but whenever we can get it, we’re better off for it.
And whenever we can <em>recover</em> it, after being forced to do without, we can significantly improve our designs.</p>
<p>State is the canonical antagonist to compositional reasoning: you might think you know what state a system is in <em>now</em>, but a moment later all that could be out the window.
So… state bad?
Obviously, we can’t do without it (well… completely and practically), but I’d also push back against this kind of blanket reaction.
State shouldn’t be banned, it should be controlled, and the best way to do that is to contain it and recover compositional reasoning.</p>
<p>But doing that is subtle.</p>
<figure class="boxed">
<h2 id="whats-compositional-reasoning">What’s compositional reasoning?</h2>
<p>Worth taking a moment to consider.
Naively, I’d say compositional reasoning allows us to understand a whole in terms of its parts.
But, I think this gets too simplistic, and I’m not ready to attempt a definition that captures all the nuance.</p>
<p>So let me start with this: inductive reasoning is compositional reasoning, but not necessarily vice-versa.
When we reason (“prove”) by induction, we’re showing a property is true of a larger object by making use of its truth on its smaller components.
But the properties we use about smaller parts don’t have to be arrived at by induction, necessarily.</p>
<p>That’s part of what’s cool about <a href="/2018/04/24/design-and-property-tests.html">property testing</a>.
We aren’t reasoning inductively there—we’re just walling off a black box, and trying to test some parts of its behavior space to see if they work as we expect.</p>
<p>MIT (controversially, funnily enough) switched its first CS course from Scheme to Python in part because they wanted to emphasize a more experimentally-driven approach to development.
In other words, they don’t want to pretend programming happens from “first principles” anymore.
We don’t exist in some Platonic ideal.
Real software is messy and vague, and often requires experiment to understand.
Distributed systems are the canonical example: generally anything that <em>can</em> happen <em>will</em> happen eventually.
That’s hard to reason about from scratch, and you’re generally not doing it strictly by induction.</p>
<p>But this doesn’t mean we should give up on compositional reasoning.
Rather the opposite: whenever we can <em>recover</em> it, instead of fully confronting the mess of complex systems, that’s a phenomenal design win.</p>
</figure>
<h2 id="encapsulating-state-attempt-1-getterssetters">Encapsulating state attempt 1: getters/setters</h2>
<p>The word “encapsulation” brings to mind object-orientation, so what of the most naive and obvious approach: getters and setters?
Let’s make our mutable state <em>private</em>, and see what happens.
Advocates point to a few advantages:</p>
<ul>
<li>Hiding representation allows it to change in the future without breaking compatibility.</li>
<li>Restricting mutations to a limited set of behaviors (the setters) allows those behaviors to change.
For example, we could add invariant checking to a setter in the future.</li>
</ul>
<p>But I disagree with these supposed advantages:</p>
<ol>
<li>
<p>Except on system boundaries where we have different priorities, when we change representation, we often <em>prefer</em> it to be a breaking change!
That way, we can just fix all the code with the old assumptions.
And on system boundaries, changes to representation often leak, leading to a desire for a different type anyway (i.e. handling two different versions of a data structure.)</p>
</li>
<li>
<p>Adding invariant checking to a setter is often a breaking change!
It doesn’t really matter if your code still <em>compiles</em> if your code doesn’t still work.</p>
</li>
<li>
<p>Setters aren’t good descriptions of behavior, and when you start describing behaviors, you often find you want more than one.
For example, you don’t gain much by having <code class="highlighter-rouge">setAmount(int)</code>, we want <code class="highlighter-rouge">moveAmount(other)</code> and <code class="highlighter-rouge">resetAmount()</code>.
But this is a minor nit about “setters” specifically, we could imagine doing this anyway.</p>
</li>
<li>
<p>We haven’t actually restored compositional reasoning.
<strong>Just because we made the mutable state into private fields doesn’t mean we’ve accomplished anything—who knows what code could be calling those setters.</strong>
We’re in exactly the same situation as when these mutable fields were just public.</p>
</li>
</ol>
<p>It’s this last one that’s killer.
“Encapsulating state” in the trivial OO sense has actually done nothing at all to help us reason about our code.</p>
<h2 id="encapsulating-state-attempt-2-invariant-behaviors">Encapsulating state attempt 2: invariant behaviors</h2>
<p>Okay, so let’s just do those things from #2 and #3 above.
We’ll have actual behaviors (not merely “setters”) and those behaviors preserve important invariants.
Does that actually change anything?</p>
<p>Yes! But only a little.</p>
<p>We can regain <em>partial</em> compositional reasoning.
<strong>If we don’t need to know anything about the state except what the invariants tell us, then we’ve recovered compositional reasoning.</strong>
The only trouble is… we often need to know more.</p>
<p>Red-black trees are a good example.
We’ve recovered enough information (through invariants) to be able to reason about our algorithm’s performance!
But that’s all the RB tree invariant really tells us.
We don’t know anything else about the <em>content</em> of the mutable tree, just from the RB tree’s invariants.</p>
<figure class="boxed">
<p>Incidentally, this brings to mind another example of non-compositional reasoning: Haskell performance.
Laziness throws a wrench in everything.
It’s generally difficult to know how long any given function will take, because it requires knowing how much time it will take to evaluate its arguments, which depends on what parts of those arguments have already been evaluated by other parts of the program.</p>
</figure>
<h2 id="encapsulating-state-attempt-3-use-data">Encapsulating state attempt 3: use data</h2>
<p>Let’s try a different strategy.
<a href="/2018/08/28/using-data-to-mutate-state.html">Using data to mutate state</a> is a fantastic approach that gets us even further towards regaining compositional reasoning.
The key benefit is that more of our program becomes pure, and pure functions are much easier to reason about.
The part of the code that actually directly manipulates state get confined to the “interpreter” that consumes the data.</p>
<p>The “encapsulation” here is totally different from the kind we see with OO.
We’re not talking about hiding representation or anything like that.
We’re just reducing the spread of code that does mutation.
More and more code that “affects the state” instead gets written as pure functions, while the code that actually <em>does</em> those changes stay in one module only.</p>
<p>This module might need expansion over time to accommodate richer ways of manipulating that state, and the ease of doing this might be related to its status as a system boundary.
It’s not a panacea, but it’s still much more confined.</p>
<h2 id="encapsulating-state-attempt-4-memoization">Encapsulating state attempt 4: memoization</h2>
<p>Another (limited) approach to trying to wall off state is to make the state not really matter.
This can take the form of a cache or memo table.
From the outside of an object with internal state of this type, there doesn’t need to be any state at all.
Except… usually this state exists for performance reasons.
And performance matters.
So that’ll pierce the veil of abstraction.</p>
<p>“Cache invalidation” is sometimes joked about as one of the hardest problems in computer science.
I think it gets that reputation in part because it looks like state that can be totally ignored… until it can’t be, and then you’re left trying to reconstruct where all you have have these assumptions littered about carelessly.</p>
<h2 id="successfully-encapsulating-state-lexically-scoped-state">Successfully encapsulating state: lexically scoped state</h2>
<p>How do we actually encapsulate state, and actually regain compositional reasoning, fully?
The technique that actually works is confining state <em>locally</em>.
Confining state to within an object (“encapsulation” as it’s usually meant) doesn’t get us there because the state is owned by the object and thus escapes, but fully-local to a function does successfully isolate that state.</p>
<p>This can take three general forms:</p>
<ol>
<li>
<p>You can just use local state within a function, and not expose it outside.
This is a technique (mostly) usable in any language.</p>
</li>
<li>
<p>You can make use of fancier type system features to help you confine that state.
This is what Haskell does with its <code class="highlighter-rouge">ST</code> (or <code class="highlighter-rouge">State</code>) monads.
We have a function that does something stateful internally, and we can <code class="highlighter-rouge">runST</code> to just do that imperative thing and get the return value.
This effectively transforms a stateful function into a pure function.</p>
</li>
<li>
<p>You can make use of fancier static analysis tools to control state visibility.
This is what Rust’s <em>borrow checker</em> does: you can mutate something freely if you exclusively own it, but once you start sharing it, it has to stay immutable.</p>
</li>
</ol>
<p>The canonical example here is sorting.
There’s a useful middle ground between a sort that mutates the array it’s given, and a totally purely functional sort.
Purely functional through-and-through leads to a lot of added inefficiency.
But… we can get something that effectively <em>looks</em> purely functional from the outside, but internally mutates its copy of the array before returning it.
Haskell and Rust can both do this, quite easily.</p>
<p>With Rust, we can take a mutable sort function like this one:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn sort<T: Ord>(v: &mut [T])
</code></pre></div></div>
<p>And we can wrap it in a way that makes it look purely functional:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fn sorted<T: Clone + Ord>(v: &[T]) -> Vec<T>
{
// Make a mutable copy of the array we were passed a non-mutable reference to.
let mut new_vec = v.to_vec();
// Mutate it into sorted order.
new_vec.sort();
// Return our copy, which can now be viewed as immutable.
new_vec
}
</code></pre></div></div>
<figure class="boxed">
<p>Quick aside: this function doesn’t necessarily return an immutable vector.
It’s “moving” its return value: once this function returns, its caller now <em>owns</em> the vector that <code class="highlighter-rouge">sorted</code> constructed.
And so its caller still has the right to consider that vector mutable.</p>
</figure>
<p>The resulting function makes no apparent or observable use of state, from a caller’s perspective.
It’s totally… encapsulated.</p>
<p>Of course, <em>internally</em> we have all the same problems state brings (i.e. when using the first <code class="highlighter-rouge">sort</code> and not the second <code class="highlighter-rouge">sorted</code>), the difference here is that we’ve actually contained it.
Once we fully wrap up all that state internally to a function call, it becomes invisible externally.
We have a (hopefully) small space of non-compositional reasoning, but:</p>
<ol>
<li>
<p>It’s contained and closed-world.
The code that can affect the state hidden inside <code class="highlighter-rouge">sorted</code> can be determined from the implementation of <code class="highlighter-rouge">sorted</code>.</p>
</li>
<li>
<p>We can be sure we got it right by rigorously testing <code class="highlighter-rouge">sorted</code>.
That is, there’s no funky state this code can get into except states reachable by applying <code class="highlighter-rouge">sorted</code> to some argument.
Any bug in <code class="highlighter-rouge">sorted</code> is observable by calling <code class="highlighter-rouge">sorted</code> on an input that triggers the bug, without needing to somehow set up an interesting system state beforehand.</p>
</li>
<li>
<p>We can used <code class="highlighter-rouge">sorted</code> without loss of compositional reasoning in the slightest.</p>
</li>
</ol>
<p>I think that’s neato.</p>
<p>That’s not the very first thing that came to mind when I used the words “encapsulation of mutable state.”
But it probably should have been.</p>Ted Kaminskitedinski@cs.umn.eduLast time I finished with a request: whether anyone’s read anything about how “encapsulation of mutable state restores compositional reasoning.” Someone (thx @ericbb) came through with a good link (which also has a good discussion on reddit) that sort of danced around the topic, and it ended up giving me a good think.Composition vs extension2019-04-16T09:50:27-05:002019-04-16T09:50:27-05:00http://www.tedinski.com/2019/04/16/composition-vs-extension<p>I have written about the drawbacks of <a href="/2018/02/13/inheritance-modularity.html">inheritance</a> in the past, and the notion that we should “prefer composition over inheritance” is well-known.
But I’d like to step back a bit today, and contrast composition with <em>extension</em>, not inheritance specifically.</p>
<p>This is a bit of a muse day.
I’m not entirely sure this is going anywhere fruitful, yet!</p>
<h2 id="the-similarities-and-differences">The similarities and differences</h2>
<p>If we think about things at the module-level, as I often like to do, then “extension” looks identical to mere “use.”
Module A imports module B, and that’s all there is to either one of these situations.
How should we understand the difference between “A extends B” and “A just uses B” or even “A is the composition of some new things with B.”
Or is there even a difference?</p>
<p>I think there is, but I’m also not sure how to clearly state it. (Hence, today’s muse post.)
Here are a few ideas about why I think there’s a difference:</p>
<ul>
<li>
<p>Inheritance clearly is a case of “extension” and not simple use.
The trouble here is drawing a line between the problems caused by inheritance specifically, versus the drawbacks of “extension” in general.</p>
</li>
<li>
<p>As I’ve argued in the past, <a href="/2018/08/07/what-makes-good-design.html">designing for extensibility has cost</a>, and we don’t always want to that.
This all comes back around to <a href="/2018/02/06/system-boundaries.html">system boundaries</a> and the desire to make things easier to <em>modify</em>.</p>
</li>
<li>
<p>Could “extension” be as simple as implementing interface types in general?
I’m not exactly sure.
Perhaps this is the root of the trouble I’m having, but it seems to me there’s room for some kinds of new interface implementations that are just a “use” versus some kinds that are an “extension.”
For example, function types are a kind of interface, but I’m not really sure it makes sense to think of every new function as an “extension.”
This seems over-broad to me.
So then what separates these two different cases?</p>
</li>
<li>
<p>What is the contrast of extension?
In other words, what does “mere use” even mean?
Is it just “composition?”
I’ve described in the past how <a href="/2018/05/15/familiar-forms-of-composition.html">programming languages are inherently compositional</a>.
We build bigger functions by composing together smaller functions, and ditto statements and expressions.
Is everything either extension or composition?</p>
</li>
</ul>
<h2 id="similar-dichotomies">Similar dichotomies</h2>
<p>Something else that strikes me as similar is the difference between a “framework” and a “library.”
Both are libraries, really, but somehow frameworks are more… encompassing.
Obviously you can merely use a library, but can one “merely use” a framework?
What’s the difference and why?</p>
<p>Likewise, when we’re building plug-ins for applications, we’re clearly extending that application’s behavior somehow.
Is this something that just exists in our minds, or is there a clear way we can define how this <strong>isn’t</strong> composition?</p>
<p>Or if I write in an application’s scripting language?
Maybe we can exploit the metaphor of a “runtime” to explain the difference?
Maybe when we have a more-constrained runtime it looks like extension, whereas a more liberal runtime starts to look more like use?</p>
<h2 id="a-story-about-my-past-work">A story about my past work</h2>
<p>One of the interesting things I did with my PhD research was develop what we called a “modular well-definedness analysis for attribute grammars,” which is way too much jargon for the uninitiated.
Here’s the de-jargoned version:</p>
<p>The neato thing about attribute grammars (think: kinda neato object-orientedish, but not exactly, purely functional programming language) is that you can really separate out the parts of a program.
To use OOish language, you can explode declaring an interface, declaring each method that’s a part of that interface, declaring types that implement that interface, and declaring each implementation of each method of an interface into separate files or modules if you want.
Obviously, this is a recipe for disorganized disaster if you really go hog-wild with that.</p>
<p>So part of what we wanted was a way of repairing things back to sensibility.
We reached for <em>modular reasoning</em> as the approach to accomplishing that.
The language lets us put declarations <em>!@$#$!@ing anywhere</em>, but we are going to come back around and re-impose order.
The over-arching rule is simple: you should be able to understand each module in terms of its imports, and without having to know all modules included in the whole program.</p>
<p>The end result was kinda nifty.
I found some minimal rules for where declarations could reside, and as a result, the program kinda “condensed” around some root declarations, with only a few decision points about how to structure things.</p>
<p>So we can start with module H (the “host”) and have module E (the “extension”) depending on it.
What’s the difference between “extension” and “mere use” here?
Well, we can restructure this program into three modules: leave H unchanged, but separate E into two modules, G which depends on none of these modules, and C, which depends on both H and G.
We do this by moving into G every declaration from E that’s not <em>forced</em> into C by the rules.
In other words, we try to restructure the “extension” into a composition (C) of two modules, H and G.</p>
<p>The difference between an “extension” and a “composition” here was just one of degree.
For “obvious compositions,” quite a lot of code could validly be moved from E to G, and the composition C was relatively light.
For “definite extensions,” sometimes literally 100% of the code could not be moved into a separate module.
It was all inseparable from H.</p>
<h2 id="an-alternative-hypothesis">An alternative hypothesis</h2>
<p>I’m not sure the above story gives good intuition, though.
I think it’s really cool, but there are parts of it that strikes me as potential coincidence.
We might use the word “extension” for multiple distinct things.
(Also, part of what my thesis work may have involved was a way to eliminate “extension” in favor of exclusively doing “composition,” and so the above might be misleading.)</p>
<p>So here’s my final thought on what the difference is:</p>
<p><strong>We’re composing things together when we can <em>reason</em> compositionally about the result, and we’re extending when we need non-compositional reasoning.</strong></p>
<p>Non-compositional reasoning is when you cannot understand something from its obvious parts.
Mutable state is the obvious example.
You can declare some mutable state and… have no idea what happens with it.
To really understand it, you need to know all possible mutators of that state.
But from looking at the state itself, you don’t necessarily know where those even are (unless the state is encapsulated).
Mutations can happen from anything that imports that module, so we’re in serious trouble.</p>
<p>The thing about being able to write extensions is that someone else might be writing an extension too.
You might think you can reason compositionally, until the system surprises you with its non-closed world.
Plug-ins conflict all the time.</p>
<p>But importantly, <strong>some interfaces can be wholly understood in their own right.</strong>
(Though, we might admit some “leakiness” to the abstraction.)
Function types are a reasonable example: you can call them by supplying the right arguments.
(Ah, but we might worry about side effects and such, but! That’s just the leaky part.)</p>
<p>How can we know everything we need to know?
Well, I’ve showed one example already back when I discussed <a href="/2018/04/24/design-and-property-tests.html">property testing</a>.
So next week (probably), I’ll think a bit more on (deep breath) category theory.
Or maybe more specifically, what we can learn about programming from it.</p>
<h2 id="end-notes">End notes</h2>
<p>I asked a lot of questions today.
Got any thoughts of your own?</p>
<ul>
<li>Has anyone seen this concept explicitly stated before: “encapsulation of mutable state restores compositional reasoning.”</li>
</ul>Ted Kaminskitedinski@cs.umn.eduI have written about the drawbacks of inheritance in the past, and the notion that we should “prefer composition over inheritance” is well-known. But I’d like to step back a bit today, and contrast composition with extension, not inheritance specifically.Controlling module dependencies2019-04-09T15:18:20-05:002019-04-09T15:18:20-05:00http://www.tedinski.com/2019/04/09/module-anti-dependencies<p><a href="/2019/04/02/solid-critique.html">Last week</a>, I mentioned this bit about modules:</p>
<blockquote>
<p>We often think the important things about a module are what abstractions it exposes, and what dependencies it has.
But the two most important questions about a module’s <em>design</em> are:</p>
<ol>
<li>What must this module <em>NOT</em> expose?</li>
<li>What dependencies must this module <em>NOT</em> have?</li>
<li>(Which creates an implied third:) What other modules must <em>NOT</em> depend upon this module?</li>
</ol>
</blockquote>
<p>When I <a href="/2018/08/14/modularity.html">talked about modules</a>, I spent quite a bit of time looking at what modules consist of, but I neglected this aspect.
The first of these questions has the more obvious answer, however—we’re just talking about encapsulation.
That’s relatively standard stuff when it comes to software design.
A module should hide details, and the best thing to designate a “detail” and so hide is some assumption that might change.</p>
<p>But the dependencies part is under-appreciated.
So today I want to work a few examples.</p>
<h2 id="how-do-we-make-two-modules-work-together">How do we make two modules work together?</h2>
<p>Suppose we have some perfectly fine existing code with two modules: A & B.
These modules are presently totally independent (no relationship between them).
We get some new requirement, and now we have to make these modules work together.
How do we make that happen?</p>
<p>We have 4 general options.</p>
<ol>
<li>We can modify the implementation of A, and introduce a dependency from A to B.</li>
<li>We can likewise modify B to depend on A.</li>
<li>We can <strong>compose</strong> the modules: introduce module C, which simply uses A & B, if they are amenable.</li>
<li>We can do dependency inversion: modify A & B to involve an interface I, then after that change we can “compose” the result and introduce our new concern in module C.</li>
</ol>
<figure class="boxed">
<img src="/assets/module-modification-composition.svg" />
<figcaption>
<p>Four options for getting two modules to work together: 1) modify A, 2) modify B, 3) composition: module C simply uses A and B, or 4) modify A & B to use an interface I, module C can use the lot.</p>
</figcaption>
</figure>
<p>As a general rule, options 1 & 2 are the simplest.
They keep the number of modules from proliferating.
However, <em>we may not want these modules to be aware of each other.</em>
This is what may drive us to options 3 & 4.
As you can see in the diagram above, options 3 & 4 keep these modules independent of each other.</p>
<p>Option 3 is nearly as simple, but we may not be able to accomplish it.
To make the modules “work together,” we might be forced to modify their implementations to accommodate that kind of composition.
In a sense, that’s exactly what option 4 is: one specific kind of modification to A & B, introducing a mediating interface in a separate module, that allows that composition to happen.</p>
<figure class="boxed">
<p>A neat thing about the difference between approaches 3 & 4: if you don’t need a fancy interface, option 4 starts to look like option 3.
For instance, if you’re writing in a functional language, and you can use higher-order functions (which use non-nominal, structural types) instead of having to define a new separate interface.
When you don’t need to declare a new interface, you don’t have to find a module to house it.</p>
</figure>
<p>So this is part of what I mean when I say that “a dependency that this module should <em>NOT</em> have.”
The simplest, most obvious, and most straightforward thing to do is to always stick with 1 & 2.
We have to reach for the others because we’re in a situation where introducing those dependencies is unacceptable.</p>
<h3 id="lets-be-more-concrete">Let’s be more concrete.</h3>
<p>I’m going to write some example Java code showing each of these options.
I’ll use the example of a <code class="highlighter-rouge">User</code> and <code class="highlighter-rouge">Book</code> classes, where we want to introduce a relationship (“books a user has read”).
To keep things brief, I have to over-simplify in a couple of ways:</p>
<ol>
<li>
<p>This is a relationship between two classes, but usually that’s the least interesting form of dependency.
The ones that matter most are between <code class="highlighter-rouge">jar</code> files (i.e. larger collections of code that are distributed and <strong>versioned</strong> together.)</p>
</li>
<li>
<p>This example is a bit in the vein of the relational model, and so these classes are all simple ADTs, not <a href="/2018/02/27/the-expression-problem.html">interfaces</a> (or “object types” in my preferred sense of the word).
Again as a result, things look a bit simple.</p>
</li>
</ol>
<p>So, here’s option 1:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class User {
// etc...
List<Book> books_read() { ... }
}
class Book {
// etc...
}
</code></pre></div></div>
<p>and of course, option 2:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class User {
// etc...
}
class Book {
// etc...
List<User> users_who_have_read() { ... }
}
</code></pre></div></div>
<p>Notice how the <em>most natural</em> relationship changes here.
When we pick one to depend on the other, we end up in a situation where one “view” on the data is preferred: either we can look at a user and see their books, or looks at a book and see its users.
We might even want both of these things, in which case, we might start to see a mutual dependency.
(Or just an awkward API, since to get a user’s “books read” list, we don’t <em>have</em> to put a method on <code class="highlighter-rouge">User</code>, it could also just be a static method on <code class="highlighter-rouge">Book</code>.)</p>
<p>Let’s try option 3:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class User {
// etc... unmodified
}
class Book {
// etc... unmodified
}
class BooksRead {
static BooksRead by_user(User u) { ... }
static BooksRead by_book(Book b) { ... }
}
</code></pre></div></div>
<p>So one nice thing about the “composition” approach here is that we can easily support both relationships, in a consistent way, without introducing dependency loops or the like.
(Again… for individual classes like <code class="highlighter-rouge">User</code> and <code class="highlighter-rouge">Book</code> from (presumably) the same package, there’s probably no issue with introducing a dependency loop.
But imagine versioned jar files or libraries that you’re distributing.)</p>
<figure class="boxed">
<p>One thing you might notice about this example is that it looks exactly like how you would model this relationship with tables in a database.
Not a coincidence.
That’s a compositional approach.</p>
</figure>
<p>Option 4:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>class User {
// etc...
<U> U lookup(EntityRelationship<User, U> r) { r.lookup_first(this); }
}
class Book {
// etc...
<T> T lookup(EntityRelationship<T, Book> r) { r.lookup_second(this); }
}
interface EntityRelationship<T, U> {
List<U> lookup_first(T e) { ... }
List<T> lookup_second(U e) { ... }
}
class BooksRead implements EntityRelationship<User, Book> {
// etc...
}
</code></pre></div></div>
<p>Now, this is contrived in several ways, but it’s an example of how an interface could sit in the middle.
It’s totally unnecessary in this situation (as our example of option 3 shows, which is clearly preferrable).
Our need to modify <code class="highlighter-rouge">User</code> and <code class="highlighter-rouge">Book</code> is solely to give them methods, so we don’t have to go looking for a class like <code class="highlighter-rouge">BooksRead</code> that’s somewhere else to find this data.
But that’s a pretty small advantage, compared to the added complexity we see here.</p>
<figure class="boxed">
<p>This example is also contrived in that we’re using an interface to get some abstraction, but an interface is totally unnecessary here.
If we were writing in C#, this example could naturally be implemented (and look more like option 3) using extension methods.
We’re not actually using the dynamic dispatch of the interface here, not really.</p>
<p>The trouble is <code class="highlighter-rouge">Book</code> and <code class="highlighter-rouge">User</code> are really just simple (single-class) ADTs.
To make this example “real,” we’d want an example where module A exports an interface with multiple implementations, and the interface we introduce here is more like a visitor, where dynamic dispatch is actually required.</p>
</figure>
<h2 id="when-do-we-care-about-non-dependencies">When do we care about non-dependencies?</h2>
<p>I think the above examples make it concrete what each of these options looks like.
But next we might wonder: how should we bother choosing among them?
Ok, the earlier options are simpler, so why would we ever go with the less simple options?</p>
<p>I’m not sure I have a good rule for this.
The problem with just always doing the simple thing here is that you can end up with a tangled mess.
It’s too easy to introduce a dependency without ever thinking about it.
And sometimes it’s just obvious, once you actually think about it, that such a dependency should not exist.
The biggest problem is just not being aware of what you’re about to do.</p>
<p>Let’s suppose module A is “a compiler” and module B is “an IDE plug-in.”
It should seem obvious that what we want is the plug-in to depend on the compiler, and we do not want to complier to depend on the plug-in.
This is just a natural result of wanting to use the compiler in situations where an IDE plug-in is irrelevant.</p>
<figure class="boxed">
<p>I choose this example in part because this is a mistake I made in the past.
It might seem odd that anyone would ever do this, but we were thinking in terms of having a “specification language” from which we “generated” a compiler.
So obviously, why not also generate an IDE plug-in from that specification, right?</p>
<p>But “specification” is just a fancy word for “program” and “generate” is a fancy way of saying “compiles.”
So our compiler started getting polluted with IDE concerns.</p>
<p>These mistakes are usually just a case of not thinking about it.
As soon as I realized what we were doing, it was obvious we should have done things differently.</p>
</figure>
<p>We can also start to imagine what things looks like for the other options.
The simple composition case looks like a compiler (module A), an IDE plug-in runtime library (B), and the plug-in that uses such a library together with a compiler (C).</p>
<p>And if we drill into how exactly the works, we can start do see something like the dependency inversion case.
How does that runtime library (B) get “composed” with the compiler?
Probably by defining some interfaces that the plug-in (C) implements using the compiler (A).
So this is similar, but there was no reason for the interfaces (“I”) to be separated into their own module; they could just remain together with the runtime (B).</p>
<figure class="boxed">
<p>There are some more funky things we can struggle with here.
If we want to keep IDE concerns separate, what about things that are really naturally written as part of the compiler?
“Jump to declaration” support, for example, is something that may be best implemented as part of the compiler, but is clearly a “concern” of the IDE.</p>
<p>The best approach here is probably to put it into the compiler, and just to ensure that it has no actual dependencies on IDE modules.
That is, you can compute the metadata to support a “jump to declaration” feature, but then leave it up to the IDE plug-in to interpret that metadata in a way that actually implements the feature for whatever IDE.</p>
</figure>
<h2 id="end-notes">End notes</h2>
<ul>
<li>One mechanistic way to spot inappropriate dependencies might be through code review.
For example, in Java, you could look at the changes to the <code class="highlighter-rouge">import</code> lists, instead of letting your eye just skip over it.</li>
</ul>Ted Kaminskitedinski@cs.umn.eduLast week, I mentioned this bit about modules:Deconstructing SOLID design principles2019-04-02T20:00:16-05:002019-04-02T20:00:16-05:00http://www.tedinski.com/2019/04/02/solid-critique<p>I got myself into this book writing project in part because I went looking for a specific kind of design book and… didn’t find it.
If you can’t find it, it’s now your job to create it, right?</p>
<p>Among the successful design aesthetics out there is <a href="https://web.archive.org/web/20150906155800/http://www.objectmentor.com/resources/articles/Principles_and_Patterns.pdf">SOLID</a> (and that’s a link to a pdf of an article introducing many of the ideas, btw).
What I’d like to do today is “critique” it, in a sense.
Or perhaps, deconstruct and re-evaluate it, in light of the things I’ve been talking about on this blog for awhile.
Why isn’t this the kind of stuff I was looking for?</p>
<h2 id="whats-context">What’s context??</h2>
<p>One of the big concepts I can’t shut up about is <a href="/2018/02/06/system-boundaries.html">system boundaries</a>.
The reason I can’t stop talking about them is they’re #1 on the list of considerations when it comes to that nebulous “context” and “trade-offs” thing people sometimes hand-wave about.
If we’re going to talk shop about design, they’re going to come up constantly.</p>
<p>You can’t talk about what designs are good without considering whether you’re working on a system boundary or not.
We must design system boundaries differently than just any other code.</p>
<p>So if you’ve got a list of things that constitute “good design,” and they do not give even the slightest consideration to system boundaries…
Well, you start to see where I raise my eyebrows.</p>
<p>Let’s dig in.</p>
<h2 id="s-single-responsibility">S: “single responsibility”</h2>
<blockquote>
<p>“A module should have only one reason to change.”</p>
</blockquote>
<p>This is the most confusingly stated principle in SOLID, in my opinion.
I’m not sure many people have a good idea of what it means.</p>
<p>In my post on <a href="/2018/08/14/modularity.html">modularity</a>, we look at what a module consists of.
I ended with a note about what I missed, and perhaps should return to with a full post of it’s own…
But I digress, here’s the short version:</p>
<p>We often think the important things about a module are what abstractions it exposes, and what dependencies it has.
But the two most important questions about a module’s <em>design</em> are:</p>
<ol>
<li>What must this module <em>NOT</em> expose?</li>
<li>What dependencies must this module <em>NOT</em> have?</li>
<li>(Which creates an implied third:) What other modules must <em>NOT</em> depend upon this module?</li>
</ol>
<p>More on this in a future post (probably), but this all goes back to an old design classic paper, “<a href="https://dl.acm.org/citation.cfm?id=361623">On the criteria to be used in decomposing systems into modules</a>.”
The underlying idea: find an assumption that might change, and encapsulate that assumption into a module.
Then, when it does change, only that module will be affected.
This makes changes easier.</p>
<p>The “single responsibility” principle is getting at this same idea, but I think it confuses things.
There’s sometimes not a single “responsibility” by itself, and there’s really little reason why <em>exactly</em> one should be the goal.
(Especially if we’re conflating, as OO advocates sometimes do, “module” with “class.” Good module designs often encompass multiple classes.)
I kind of suspect this phrasing was chosen so it would fit the SOLID acronym.</p>
<p>But there’s potentially another reason for this phrasing, and that’s the common “god-object” dysfunction in object-oriented design.
This is, pretty specifically, an OO dysfunction (there no functional programming equivalent that I know of).
I suspect it stems a little from the “is-a” relationship idea.
My <code class="highlighter-rouge">User</code> class <em>is</em> a user, right?
Okay, so, authentication goes there because a user authenticates.
Okay, so, sessions go there because users have sessions.
Okay, so, sending messages go there because users send messages.
And so on, to oblivion.</p>
<p>This leads people to break all of the good rules about designing modules, while doing things that seem completely natural and reasonable at the time.
But then <code class="highlighter-rouge">User</code> knows about everything, everything knows about <code class="highlighter-rouge">User</code>, and you’ve got spaghetti.</p>
<p>This is, as I see it, what “single responsibility” is all about.
It’s a principle named after a specific OO design dysfunction, but the actual “rules” are really about good module design.</p>
<h2 id="o-openclosed">O: “open–closed”</h2>
<blockquote>
<p>“A module should be open for extension but closed for modification.”</p>
</blockquote>
<p>Last year, I <a href="/2018/02/27/the-expression-problem.html">used the expression problem to show the different choices we have in type design</a>.
When we introduce a type, we have a choice between three good options: data, objects, or abstract data types (ADTs).
Each of these come with different trade-offs, different modes of extensibility and future maintainability, and are appropriate in different situations.</p>
<p>And SOLID is here to tell us pretty much “<em>DATA BAD, ALWAYS USE OBJECT</em>.”</p>
<p>Or at least, that’s definitely what I take away from reading its authors.
As you can imagine, this is a principle I don’t like at all.</p>
<p>The examples used are always “if we use data, we can’t extend it with new variants, but if we use an interface, we can!”
I hope that, having read my piece on type design, people immediately have a problem with that reasoning.
After all, now the interface has a fixed list of methods, and that type cannot be extended with more methods without modifying the original module!
Shoot! The example of being “open for extension” is actually “closed for extension!”</p>
<figure class="boxed">
<p>How was this missed?
Well, partly, I think this particular type design trade-off is not as well known as it should be.</p>
<p>I also think part of the explanation is the rapid growth in the number of software engineers that used to be happening when this field was younger.
When new programmers are constantly pouring in that fast, it’s hard to raise the average level of technical decision-making capability.
If you waved a magic wand and “everyone” knew something, two years later they were a minority again.</p>
<p>Coupled with that, I think there was a disparity of impact with this design decision.
Data is often a superior choice to objects, but usually it’s not earth-shattering.
(As evidenced by the dreadful support for data in most of our programming languages.)
But inappropriate use of data where an interface should have appeared can create a <em>lot</em> of problems.
Screaming “always use objects” at a pile of inexperienced developers really may have increased the average quality of the designs they produced.
Even if wrong 70% of the time, the 5% of the time it’s <em>really</em> the right choice might be overwhelming.</p>
<p>But I hope we can move past this old advice towards making the right decisions in the right circumstances.</p>
</figure>
<p>But there’s another serious problem here, and that’s the implicit assumption that extensibility is desirable.
One the one hand, we shouldn’t forget that ADTs are a legit design choice, and are deliberately (and desirably) not extensible by users (and as a result, become safely modifiable by maintainers).
I even speculate that ADTs are probably more commonly the right type design choice than interfaces!</p>
<p>The bigger issue, though, is the thing I can’t shut up about.
Let me quote <a href="/2018/08/07/what-makes-good-design.html">myself on design goals</a>:</p>
<blockquote>
<p>Extensibility and re-usability are potential goals <strong>for system boundaries only</strong>.</p>
</blockquote>
<p>Being “open for extension” is blanket advice that ignores all context.
The open-closed principle tells us everything should be extensible, but extensibility is for system boundaries, and <em>we really shouldn’t create system boundaries unnecessarily!</em>
Boundaries are expensive.
Code that’s not on a boundary is cheap and easily modified.
If it’s cheap to modify, it doesn’t need to be extensible, and making it extensible is usually a step backwards in design quality.</p>
<p>We’re much better off designing for easy <em>modification</em> wherever it’s cheap to do so.
You even might say we should be “open to modification.”</p>
<h2 id="l-liskov-substitution">L: Liskov substitution</h2>
<blockquote>
<p>“Subclasses should be substitutable for their base classes [without breaking behavior contracts].”</p>
</blockquote>
<p>I don’t object to this one at all.
Mostly, I object to the casualness with which this principle gets thrown around.
I don’t think advocates of SOLID always fully understand the implications, because <a href="/2018/02/13/inheritance-modularity.html">implementation inheritance routinely violates Liskov substitution</a> (especially when taking <a href="/2018/06/26/variance.html">variance</a> seriously).</p>
<p>Confronting this reality leads one down the path of “preferring composition over inheritance,” all the way to banning implementation inheritance (almost?) entirely.
That might be a good goal, but I’m not sure everyone understands it’s a consequence of this principle.</p>
<figure class="boxed">
<p>“<em>Almost</em> entirely” because implementation inheritance can (more or less) safely be used when:</p>
<ol>
<li>Never exposed on a system boundary (even a soft one). Or,</li>
<li>When used internally by an ADT, not an object. (This is sort of a restatement of the above, though.)</li>
</ol>
<p>That is, inheritance is less dangerous when you fully control its use and can change all users at once as you please.
You have a nice closed-world to play in.
When such a convenient tool is laying around, you’re probably not going to resist using it in these cases when it’s reasonably non-destructive.</p>
</figure>
<h2 id="i-interface-segregation">I: interface segregation</h2>
<blockquote>
<p>“Many client-specific interfaces are better than one general-purpose interface.”</p>
</blockquote>
<p>To be honest, I don’t really understand this one very well.
Or rather, I think it’s pretty vague and has a few interpretations, which means it’s not exactly one good principle.</p>
<p>One simple interpretation is that “god-interfaces” are just as potentially destructive as “god-objects.”
If you’ve read my bit on modules, you can imagine a little bit why: interfaces can create new <em>public dependencies</em> in a module just as well as classes can.
A better design is one that controls and limits their scope.
So multiple interfaces <em>that allow some public dependencies to be avoided when they’re not necessary</em> can be quite a bit better than a single mega-interface.
(I do think thinking in terms of public dependencies is critical here, though. Otherwise we might be left wondering <em>why</em> multiple smaller interfaces are supposedly “better” and <em>when</em> to actually break things up.)</p>
<p>Another interpretation, as best I can see one, concerns what we often see happen with REST APIs, or with plug-in systems, that need to evolve while supporting legacy code.
This is a principle that <em>exclusively applies to system boundaries</em>.
In this situation, we’re stuck with an interface we can’t change, because it’d break users.
So instead of changing it, we introduce a new, “version 2” of the interface, and modifying our code to work regardless of which is actually used.</p>
<p>This allows all the legacy users to be unaffected, while allowing the design to (partially) evolve.
It’s only partial, though.
Often, having to support the exact behaviors available through the legacy interface makes the most-desired change infeasible.
We’re often forced into some compromise.</p>
<p>So this isn’t so much a design principle as it is a tool for coping with hard system boundaries.
It’s a tool we’re forced into using, not something we should ever be applying from the start.
Except that if we’re to apply this tool, the design needs to be amenable to its use.
But this mostly just boils down to being cognizant of what we’re exposing as a system boundary at all.</p>
<figure class="boxed">
<p>Another interpretation of this principle boils down to rehashing some of the ideas we talked about with “single responsibility” above, and what we see below with “dependency inversion.”
So, I won’t bother.</p>
<p>But there is one thing I’d like to additionally point out: sometimes we’re just suffering from a lack of support for data in our languages.
Consider a method that returns an object.
On the one hand, we can ask “this method returns this class, but maybe it should be returning a more narrow interface instead?”
Or… we might also ask “this method returns a class, but really what we’re after is just some data that happens to be encapsulated by that class. If this language supported data better, I might just return the data type I’m interested in instead of referencing the whole object.”</p>
<p>Sometimes we just want more focused types.</p>
<p>And as a final aside, I almost said “more specific types” in that last sentence, but that’d be wrong, wouldn’t it?
After all, returning an interface type instead of a concrete class is “more focused” but technically “less specific” because it could be any class that implements the interface.
Subtlety in word choice, that.</p>
</figure>
<h2 id="d-dependency-inversion">D: dependency inversion</h2>
<blockquote>
<p>“Depend upon abstractions. Do not depend upon concretions.”</p>
</blockquote>
<p>This is one of those principles that I think is partly backwards, again.
Like <a href="/2018/12/18/the-law-of-demeter.html">the “law” of Demeter</a>, this rule, if followed, encourages you to think about how you’re <em>using</em> code.
The correct thing to do here is think about how that code is structured, instead.</p>
<p>Your task is not to “not use concretions.”
It’s to not expose the “concretions” that should not be used.</p>
<p>And again, <em>we’re only talking about system boundaries</em>, although in this case, even “soft” system boundaries (of other modules) are in scope, too.
As I mentioned in the Demeter post, the real law here is that “anything exposed by a system boundary equally becomes a system boundary.”
Don’t get caught unaware.</p>
<p>I also continue to object that many “concretions” are actually abstractions.
Advocates of SOLID seem to take the view that only interfaces are abstractions, but any type is, including ADTs and data.</p>
<p>The “inversion” this principle mentions comes about as a result of the old “<a href="/2018/07/31/interfaces-cutting-dependencies.html">interfaces cutting dependencies</a>” trick.
When you’re trying to hide a dependency (because design demands that a module not depend on another one), an interface can sit in the middle.</p>
<p>So this principle is a bit of a muddle of a few different ideas, and I think thinking in terms of system boundaries (and having a reasonable notion of “module” besides just “class”) is a better approach.</p>
<h2 id="end-notes">End notes</h2>
<p>Today’s post is way long enough, but I also think it might be suffering from a serious lack of examples.
For today, oh well. Sorry. :)</p>Ted Kaminskitedinski@cs.umn.eduI got myself into this book writing project in part because I went looking for a specific kind of design book and… didn’t find it. If you can’t find it, it’s now your job to create it, right?Testing at the boundaries2019-03-19T13:27:55-05:002019-03-19T13:27:55-05:00http://www.tedinski.com/2019/03/19/testing-at-the-boundaries<p>I don’t normally have that much appreciation for “classics.”
At the risk of outing myself as a heretic, Tolkien for example does great world-building and… that’s about it.
He may have defined a genre, but I really think other authors have done a better job with, like, plot. And characters.</p>
<p>I say this because I think there’s a lot to learn about TDD from <a href="https://www.oreilly.com/library/view/test-driven-development/0321146530/">Kent Beck’s original book</a>.
Last week, I was inspired to muse a bit about how <a href="/2019/03/11/fast-feedback-from-tests.html">the TDD process “as described” is really about dynamic object-oriented programming</a>, and more-typed “more-functional” languages interact differently.
This week, I want to point out that a lot of the things I most disliked about TDD when I encountered it “in the wild” <em>do not appear in this book</em>.</p>
<h2 id="system-boundaries-implicitly">System boundaries, implicitly</h2>
<p>I recently tweeted a <a href="https://www.youtube.com/watch?v=shngiiBfD80">good talk by Jessica Kerr about property testing</a>, and one of the interesting things is that later in the talk, she brings up <a href="/2018/02/06/system-boundaries.html">system boundaries</a>.
Except… not by name, because we don’t seem to have a widely accepted name for this concept, which is part of what I’m trying to cure with this book project.</p>
<p>One of her off-hand remarks is that, if you think you have to write unit tests for every method of every class, you’ve been mis-educated.
This was a point I started trying to make <a href="/2018/04/10/making-tests-a-positive-influence-on-design.html">last year</a>.
This is a pretty concrete form of dysfunction: somehow, somewhere out there, a lot of people have started to get the idea that this is what TDD was about.
And… it sorta makes sense, because to do TDD, you kinda have to <em>start</em> that way.</p>
<p>Kent Beck’s book opens by example, and right away in the <em>very first example</em> he displays the opposite behavior!
The opening example is all about representing money in different currencies.
He starts simple with <code class="highlighter-rouge">Dollar</code>, then starts to consider how to introduce <code class="highlighter-rouge">Franc</code>.
This precipitates a <code class="highlighter-rouge">Money</code> interface, <em>and then a refactoring of all the tests to purge direct references to each concrete class implementing <code class="highlighter-rouge">Money</code></em>.
(For example, even <code class="highlighter-rouge">new Dollar(5)</code> becomes <code class="highlighter-rouge">Money.dollars(5)</code> in the tests, so that no references remain at all.
Naturally, if these had started life in <code class="highlighter-rouge">DollarTest.java</code> they’d probably move, too.)</p>
<p>The purpose of this refactoring is to allow these class implementations to freely change, unconstrained by the tests.
This is accomplished by moving the tests from the internal implementations to the system boundary (or at least, to the “harder” system boundary).</p>
<p>This isn’t directly examined or highlighted in this way, but the motivating factor here is that <em>writing tests against the concrete classes was technical debt</em>.
That was where we had to start—the <code class="highlighter-rouge">Money</code> interface didn’t even exist at first—but as the design evolved, the old tests became a liability that needed to be paid down.
The tests needed to be refactored before things proceeded.</p>
<p>I find it very curious that the book that started TDD begins with an example showing it’s a bad idea to insist on “unit tests for every method of every class,” and yet that’s where some of the industry went.
I may have to reconsider my lack of appreciation for classics.</p>
<h3 id="how-did-we-get-here">How did we get here?</h3>
<p>This is pure guesswork, but I think human brains are just a bit broken.
For humans, ideological reasoning is moral reasoning.
TDD isn’t just another technique, it’s <em>what’s right</em> and deviating from it is <em>wrong</em>.</p>
<p>And if there’s one thing the human brain likes, it’s when doing the right thing leads to the right outcome.
So the idea that following TDD might <em>create technical debt</em> is repulsive.
Now, TDD can’t be pure and just and good, because it doesn’t always deliver us from evil.</p>
<p>And this is a seductive idea, since following TDD leads us to have tests!
Tests good! Right?</p>
<p>So we went from “drinking water is good for you” to “the only moral process for developing software is force-feeding our developers 150 L of water a day. Please ignore the bodies in our wake.”
Oops.</p>
<h2 id="tdd-falling-down">TDD falling down</h2>
<p>So last week, I mentioned how “data + static types” can change how TDD works, and noted how things like DB schema design (type design) aren’t really as amenable to TDD as a process.
Today, I’d also like to relate a story about ordinary coding where TDD falls down as well.</p>
<p>I’ve mentioned before that <a href="2018-04-10-making-tests-a-positive-influence-on-design">compilers rarely unit test</a>, but I’d like to expand on that idea with my own story.</p>
<p>Years ago, I redesigned the type system for a whole programming language.
This was an academic language, and its compiler wasn’t in the best state when I started.
Part of what I was also trying to accomplish was improving our testing, to ensure we didn’t accidentally break things for our users.</p>
<p>So when I wrote the new type system implementation, I wrote a test suite along with it.
This wasn’t exactly TDD (I feel compelled to note, even after describing how TDD can change with types last week), but it was pretty close.
The point was the same: write a little code, and quickly exercise it to get confidence it works, in the form of a test suite.</p>
<p>This went quite well!
Once the machinery was working, I wired it up into the rest of the compiler.
Tadaa! A replacement type system, complete with new unit test suite for it.</p>
<p>Years later, we deleted that entire test suite.</p>
<p>Now, I don’t regret writing it in the first place.
It was quite helpful when I was initially developing that code.
It’s a long road from writing a unification function to being able to finally run it once it’s used everywhere in the compiler implementation.
Starting by writing a function and immediately unit testing it was great.
It easily sped up every bit of the development process: writing the code (less time thinking “wait, is that right?”), debugging (“oh, <em>this</em> thing isn’t working yet!”), and so on.</p>
<p>But we didn’t end up with a useful test suite afterwards.</p>
<p>The problem was that every change after the type system was merged fit in one of two categories:</p>
<ol>
<li>
<p>The change never broke a test, nor really could it have.</p>
</li>
<li>
<p>The change broke lots of tests, and it was a design change that <em>should</em> have broken lots of tests, because the tests became wrong.</p>
</li>
</ol>
<p>So every change we made meant coming back to a pile of broken tests, and just… updating the “expected” values of each test.
We just copy & pasted the new “actual” values from the test failure.</p>
<p>This is why we deleted the test suite.
It never helped, and this was just a pointless hindrance.</p>
<p>The type system was better exercised with tests written against inputs to the compiler.
“This broken code should raise a type error about this.”
This style of test has two advantages:</p>
<ol>
<li>
<p>Because it’s written against a system boundary (the language the compiler accepts), it can almost never become technical debt and impede progress.</p>
</li>
<li>
<p>It not only exercises the typing machinery, but also the connection between expressions, the local environment, and the typing machinery.</p>
</li>
</ol>
<p>The downside, of course, is that these kinds of tests can’t be run right away while coding.
Like I said, I didn’t regret unit testing that code <em>at the time</em>.
It just didn’t leave me with a useful test suite afterwards.</p>
<h2 id="unit-tests-are-not-all-there-is">Unit tests are not all there is</h2>
<p>The TDD approach is powerful in part because it comes with tools to help make things happen.
You aren’t just instructed to write tests, you’re given a test running harness like JUnit to help you run them.</p>
<p>But another drawback of the TDD approach is that <em>this is limiting</em>, especially in your thinking.
Many things just aren’t done well with example-based unit testing.</p>
<p>Here are a few examples.</p>
<h3 id="games-or-simulations">Games or simulations</h3>
<p>How do you test a game?
The game industry is notoriously terrible with testing.
At a guess, there are three causes:</p>
<ol>
<li>Terrible working conditions. Yikes.</li>
<li>Many “one and done” products. Many games are not maintained long-term after release.</li>
<li>Wait, how <strong>do</strong> you test a game?</li>
</ol>
<p>The problem here is that unit testing is far less applicable.
Simulations depend on lots of highly-interacting components, written in a high-performance style.
It’s nice when you can find a thing here or there amenable to a unit test, but a lot of stuff is just not obviously testable!</p>
<p>Or at least, it seems not testable, when all you’ve got is a unit testing harness.</p>
<p>The secret is to build custom test harnesses.
Maybe even game-specific ones. Multiple.
This stuff isn’t actually untestable, it’s only that general-purpose xUnit-style testing fails us.
(This isn’t my area of expertise, so I don’t have good examples to give you here, however.)</p>
<h3 id="concurrency">Concurrency</h3>
<p>Concurrent systems are one of the original “maybe not so applicable” areas for TDD, but it turns out, you <strong>can</strong> test them.
Just… again not with the “up-front unit testing style.”</p>
<p>I’ve mentioned before, and will never get tired of talking about, how cool <a href="https://jepsen.io/">Jepsen</a> is for doing deep property-testing of distributed databases.
But it’s property testing, and it’s a non-unit testing style that doesn’t really give you design guidance as you go.</p>
<h3 id="distributed-systems-and-monitoring">Distributed systems and monitoring</h3>
<p>One of the most popular, and remarkably effective, testing strategies today is called “testing in production.”</p>
<p>That’s hopefully slightly a joke, but simultaneously very true.</p>
<p>We monitor running systems for failures.
We collect crash reports from user’s machines.
We log exceptions that occur when our Javascript runs in our user’s browsers.
We do “canary deployments” of new services, allowing us to discover any new failures quickly without impacting a large number of users.</p>
<p>Again, these are great testing strategies that don’t really fit into the TDD worldview.</p>
<h3 id="iteration-and-feedback">Iteration and feedback</h3>
<p>Design is always iterative, and TDD is one way to do iterative design when it comes to coding an implementation.
But that doesn’t mean the kinds of iteration TDD brings is the right kind of iteration.
You might just be spinning your wheels.</p>
<p>When talking about distributed systems, I’ve been trying to stick to the “happy path,” where we offload all the complexity onto databases, and otherwise just operate some simple services.
But sometimes your job is to write those databases.
What should you do then?</p>
<p>One answer is something like Jepsen, but Jepsen is also not a very TDD-ish approach.
It doesn’t give us design feedback as we’re writing the code, it just helps us find bugs in the implementation after the fact.</p>
<p>Another answer might be TLA+.
<a href="https://twitter.com/Hillelogram/status/1107459272953262090">Hillel calls this design up front</a>, but that’s only true if you think “before you start implementing code” as “up front design.”
But the processes of <em>writing the TLA+</em> is iterative, because design is always iterative, and we’re getting fast feedback from the machine while we do it.
We’re just operating on a higher level of abstraction, one well-suited to the problem we’re trying to solve.
(And we’re working with something where TDD isn’t really applicable as an approach anymore, too.)</p>
<p>In that tweet-thread, someone links to <a href="http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html">an interesting pile of links</a> where people contrast TDD with “thinking.”
I think it’s really unfortunate, and a definite dysfunction of TDD advocacy, that “thinking” became contrasted with TDD.
TDD is a great way to write code, when you already know what code you generally need to write.
But sometimes you have to think at a higher level of abstraction first.
TDD shouldn’t be contrasted with thinking, TDD should be contrasted with “okay, now that we know what we’re going to do, let’s plan out all the UML diagrams before we starting coding…”</p>
<p>Always think ahead about <em>problems</em>.
I’ve always felt like TDD is solely about bridging the gap between “I think I know how to solve this problem” and “this is what the code to solve this problem will actually look like.”</p>
<h2 id="end-notes">End notes</h2>
<figure class="boxed">
<p>One general thing that’s not lost on me is that the “bad TDD” approach in that previous link spins its wheels thinking about representation, instead of solving the problem.
Kent Beck’s book also spends its time thinking on representation (of <code class="highlighter-rouge">Money</code>), too.</p>
<p>One might be tempted to accuse TDD of basically being an approach to coping with the deficiencies of dynamic object-oriented languages when it comes to representing data.</p>
<p>But I think that’s unfair.
“Static types + data” gives us tools to reason about representation much better, but we still often need to course-correct our representations as we discover new things during the process of implementation.</p>
<p>What’s different is how we do that.
When you realize your types need changing in the middle of implementing the next task, you can pause what you’re currently doing, change the types, fix all the new type errors, and return to the task at hand.</p>
<p>The process is a lot more involved without the aid of static types, or without rich enough data types (such as with Java).
<a href="https://twitter.com/garybernhardt/status/1107738508288876544">This tweet from yesterday about TypeScript from Gary Bernhardt comes to mind.</a></p>
</figure>Ted Kaminskitedinski@cs.umn.eduI don’t normally have that much appreciation for “classics.” At the risk of outing myself as a heretic, Tolkien for example does great world-building and… that’s about it. He may have defined a genre, but I really think other authors have done a better job with, like, plot. And characters.Quick feedback loops while coding2019-03-11T23:04:57-05:002019-03-11T23:04:57-05:00http://www.tedinski.com/2019/03/11/fast-feedback-from-tests<p>I’ve been on a bit of distributed systems streak lately, which will make this post seem quite out of order.
But… that’s part of what this blog is for: I write down and expand on the thoughts I have when I get attacked by them.
The book can put things in order.
So today’s a jog back to thinking about testing.</p>
<p>Among the many goals of TDD is being quick about things, and to be quick by being small.
I’ve been re-reading Kent Beck’s book on TDD, and it’s almost <em>excruciating</em> how small the steps get.
There’s a real sense of <em>panic</em> to get back to being able to run the tests and see the green light again.</p>
<p>(Interestingly, Beck does not <em>prescribe</em> small changes later in the book.
He muses that larger changes are probably reasonable, but notes that practitioners of TDD almost always gravitate towards more rapid small changes by preference.)</p>
<p>Likewise, there’s a strong sense that tests should be small: isolated unit tests.
Test “one thing.”
Make sure, when tests start failing, that they can be quickly tracked down to their root cause.
Isolate parts of the system, so broken code in one place doesn’t create “spurious” test failures elsewhere.</p>
<p>I think both of these things are, in one sense, redundant.
That said, there’s still good reasons to want to do both.
I also think types (and data!) change things a lot.</p>
<h2 id="why-keep-changes-small">Why keep changes small?</h2>
<p>Part of forcing yourself to make small changes is that <em>this is simply how humans work</em>.
You have to break large problems down into small problems.
We just can’t cope with problems that are too big, and we seem to take <em>non-linear</em> (hyperbolic?) amounts of time to deal with larger and larger problems, until we hit that limit and our brains overflow and it’s impossible.
So forcing yourself to work small change by small change can increase the speed that you’re working overall, even if you’re doing “more work.”
(In quotes because this measure of “work” is by typing on a keyboard, and typing isn’t a good measure of the work involved, is it?)</p>
<p>Some might object, “work smarter, not harder!”
But this advice is rather useless.
If you get the work done faster and easier by doing a lot of small changes, instead of one big complicated change, then the “smarter” approach is the “harder” approach, so this aphorism is just… tautological.</p>
<p>But the other reason it’s important to keep changes small comes from the interaction with testing.
If you go from green to red with a small change, the thing you “broke” is narrowly identified: it’s something (or directly related to something) you just changed.
<strong>The debugging problem is simpler</strong>.
We know where to look.</p>
<p>Which means, with small changes, we don’t actually need narrowly-focused, isolated, unit tests.
The point of all that is making it clear <em>what</em> broke, but our small changes already do that!</p>
<h2 id="why-try-to-keep-tests-isolated">Why try to keep tests isolated?</h2>
<p>Well, as long-time readers of the blog know, I don’t necessarily like this.
“Over-mocking” is a design and testing failure that is a result of over-zealous attempts to keep tests more isolated than has any real benefit.</p>
<p>One of the reasons we try to keep tests isolated is that keeping tests small mean they run faster.
TDD is all about getting into that loop where you can make a small change and then run the test suite.
Big, long, expensive, time-consuming test suites impede that.
It’s kinda hard to get immediate feedback when it’s just not immediate.</p>
<p>However, there is another reason to want to better isolate tests.
Because <strong>we can’t always keep changes small</strong>.</p>
<p>Sure, that goes against the general TDD philosophy, but we can’t really lose sight of the fact that TDD is some weird human-specific hack that just kinda seems to work pretty well?
We can speculate on some reasons why, but really <em>this isn’t a principled methodology</em>.
We didn’t carefully arrive at TDD by following the empirical scientific evidence.
It’s absolutely inevitable that it’s not always going to work well.</p>
<p>When we’re forced to make larger changes, isolation helps us quickly debug test failures.
A failing isolated test is one that says “this particular small bit of code has something wrong in it.”
It narrows things down immensely.
When our change is small, we don’t really need help with that.
But when the change is large, we’re going to have to spend time debugging and tracking down each and every failure to their root cause.</p>
<p>This is one of those things that can make big changes take non-linear amounts of effort.
It’s not just our brains, there’s also something inherent to it.
Instead of “oh, this is broken, let me look at it and fix it,” we have to spend time trying to figure out <em>what</em> even is broken in the first place.
And then the necessary fix could have far-reaching implications, too!</p>
<p>So I don’t want to disparage isolation for tests!
It’s clearly a benefit, when you can get it cheaply enough.</p>
<p>But take note: the TDD method has us going with two redundant approaches to trying to keep debugging time down.
First, we keep the changes we make really small before getting back to a green test run.
Second, we try to keep the tests as reasonably isolated as possible, so that when we run into a large change, we don’t have to spend as much time tracking down why a failure happened.</p>
<h2 id="a-hypothesis-about-static-types">A hypothesis about static types</h2>
<p>I don’t think it’s a coincidence that TDD came out of the Smalltalk community.
I think this is a design approach that’s especially well-suited to the dynamic object-oriented style of programming, both it’s strengths and weaknesses.
I also don’t think it’s a coincidence that one of the immediate “walls of applicability” TDD ran into is database schema design.
It’s… just not something amendable to this approach!</p>
<p>I think, and of course cannot prove, that static types <a href="/2018/01/23/data-objects-and-being-railroaded-into-misdesign.html"><strong>and better support for data</strong></a> change things about development that affect both of the phenomenon I’ve discussed above.</p>
<ol>
<li>
<p>TDD encourages smaller units of change.
Static types and data allow rigorous (machine-aided) thinking about much larger chunks of code.
This is not a requirement to do things in bigger chunks, but it enables thinking holistically and systemically to a much greater degree.</p>
</li>
<li>
<p>TDD encourages isolation of tests to narrow the scope of large changes.
Static types and data enforce a natural isolation.
Modulo bugs involving state (an orthogonal issue?), it becomes a lot easier to track down the cause of a test failure, compared to a dynamic object-oriented design, because the test failure can only originate in a cause permitted by the types.
(Ignoring state, a lot of “cascading failures” of non-isolated tests simply become type errors instead.)</p>
</li>
</ol>
<p>One of the things I think happens here is that the “edit and typecheck” loop serves essentially the same role as “edit and test.”
We always want machine-checked feedback quickly!</p>
<h3 id="a-quick-example">A quick example</h3>
<p>Let’s write the <code class="highlighter-rouge">map</code> function in Haskell, using TDD.</p>
<p>The first step? That’s right, types.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map :: (a -> b) -> [a] -> [b]
</code></pre></div></div>
<p>This might come as a surprise to some who think of TDD as “test-first,” but you literally cannot write the test without first thinking about types.
Types are always first.</p>
<p>Now, let’s get to that test!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>test_map_single =
map id [1] == [1]
</code></pre></div></div>
<p>Okay, you could argue I should start with the empty list, but I want to speed this example along a <em>little</em>.
Now we have our test, let’s follow the rules and get an implementation that gets us back to green as fast as humanly possible!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map f x = [1] -- TYPE ERROR!
</code></pre></div></div>
<p>Oops, that’s doesn’t work. Uh…</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map f [1] = [1] -- TYPE ERROR!
</code></pre></div></div>
<p>Uh, well,</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map f [] = []
map f [x] = [x] -- TYPE ERROR!
</code></pre></div></div>
<p>Shoot.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map f [] = []
map f [x] = [f x]
-- Well, maybe now just a warning: incomplete match
</code></pre></div></div>
<p>Honestly, that should probably be an error rather than a warning, but hey!
We’ve arrived!
But… there’s that warning to fix… and how?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map f [] = []
map f [x] = [f x]
map f (x:xs) = undefined
</code></pre></div></div>
<p>LOL, got it.
I <em>said</em> TDD tries to encourage excrutiatingly small steps.
Okay, now let’s try another test…</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>test_map_increment =
map (+1) [1, 2] == [2, 3]
</code></pre></div></div>
<p>Great! It fails. And now to fix it…</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map f [] = []
map f (x:xs) = f x : map f xs
</code></pre></div></div>
<p>Well… we’re kinda forced to just finish it, aren’t we?
Are we really already done writing example-based tests for <code class="highlighter-rouge">map</code>?
If our implementation is done just like that, we’re left unable to write a test that fails anymore.
We can’t restart the “red-green-refactor” loop without going red.</p>
<p>And if we opt for <a href="/2018/04/24/design-and-property-tests.html">property testing</a>, we have an even shorter run.
Starting over from scratch:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map :: (a -> b) -> [a] -> [b]
prop_map x =
map id x == x
</code></pre></div></div>
<p>Oops, now we’re forced to go directly to the correct implementation, nothing else to do here!
I suppose the benefit here is that <em>while we’re developing that solution</em>, we can continue to get counter-examples of where our implementation fails by running the tests.
So… “TDD” is still good here!
But we’re emphatically <em>not</em> following the process as originally envisioned.
The tiny step by step process of writing test after test gets completely circumvented.
We stood up the type and property test and got <em>dragged</em> all the way to the final implementation.</p>
<p>What’s happening here is that types are a significant aid, and type checking gives us similar feedback as testing.
We’re practically railroaded into writing a correct implementation of <code class="highlighter-rouge">map</code> after just 2 example-based tests (ok, I skipped <code class="highlighter-rouge">[]</code>, so 3), and even just 1 property test is enough to force us all the way to the right implementation.</p>
<p>Although I’m not going to write out a comparable example here, I’d encourage you to try this with, say, Python.
How long until you start to have confidence that you’ve got a reliable implementation?
For reference, the <a href="https://github.com/python/cpython/blob/491ef53c1548c2b593d3c35d1e7bf25ccb443019/Lib/test/test_builtin.py#L796">Python standard library has 12 tests</a>.
That fits my intuition of about how many tests you’d probably want to write, TDD-style, to arrive at an implementation you have confidence in.</p>
<p>“TDD as described” is really focused on dynamic object-oriented languages.
It changes a lot when good types and data are involved.</p>
<h3 id="thinking-with-types">Thinking with types</h3>
<p>One of the first tasks I ever did “professionally” was design a database schema.
Sit down, figure out what tables we need, with what columns, their relationships, normalization, obvious initial indexes, and so on.
This is a task for which TDD is ill-suited, but this is also a task we can do without running any code!</p>
<figure class="boxed">
<p>Kent Beck’s original book notes database schema design as something not really addressed by TDD, and it sure seems like that hasn’t changed.
It hasn’t stopped people from trying, but oh <em>yikes.</em>
I’m gonna scream if I see “test that table x has column y” “oh no, it failed!” “modify table x to have column y” “mission accomplished!” ever again.</p>
</figure>
<p>Schema design is just a form of type design.
This is also what we’re doing <em>before we write tests</em>.
We write the types.
Can’t write tests without the types.</p>
<p>And the types can be arbitrarily large—a whole database schema is a nice example.
When programming, we’re not just talking about the signature of a function, but we might be designing the types of arguments to that function, too.
Or at least, we might with a language that has <a href="/2018/01/23/data-objects-and-being-railroaded-into-misdesign.html">good support for data</a>.</p>
<p>When we have to use objects instead of data, we end up pausing what we’re currently doing, and going to create that data class instead.
And that class probably has methods, and we so we might start by writing tests for <em>those</em> methods.
We end up following a very different path of development, like what we see in the books about TDD.</p>
<p>But in any case, type design is task we humans actually can do just fine.
We don’t actually encounter problems accomplishing this, the way we do writing implementations.
There isn’t much of a design methodology actually called for here.
Or at least, whatever design methodology we should be using isn’t anything like TDD at all.</p>
<h2 id="type-test-red-code-green-refactor">Type, test, red, code, green, refactor</h2>
<p>I think my point today drifted into several things:</p>
<ol>
<li>“Small changes” and “isolated tests” are redundant, but redundancy is useful because nothing’s perfect here.</li>
<li>Good (e.g. non-nullable, abstract, data-oriented) static types help us naturally isolate tests, reducing the need to alter designs (to over-mock) to pursue it. (State remains the major foil for isolation.)</li>
<li>Thinking with types is powerful, does not require anything like TDD itself, and is synergistic with TDD in practice.</li>
<li>Good support for data (together with static types of course) makes us apply TDD differently. It’s not the same as what happens with the purely object-oriented style. (Not just dynamic, even the statically typed object-oriented style, such as Java.)</li>
</ol>Ted Kaminskitedinski@cs.umn.eduI’ve been on a bit of distributed systems streak lately, which will make this post seem quite out of order. But… that’s part of what this blog is for: I write down and expand on the thoughts I have when I get attacked by them. The book can put things in order. So today’s a jog back to thinking about testing.Backpressure2019-03-05T12:35:00-06:002019-03-05T12:35:00-06:00http://www.tedinski.com/2019/03/05/backpressure<p>The running topic for the last few posts is the basics of distributed systems.
For today’s post, let’s start with a short story.</p>
<h2 id="the-scenario">The scenario</h2>
<p>The system was humming along just fine.
The occasional failure—even pretty big network splits—got handled, and services recovered reasonably quickly.
Updating the various parts of the system would go off without a hitch.</p>
<p>Then a transformer exploded, and the whole data center went down for a moment.</p>
<p>Never fear, right?
The power is back, all the machines come back up, everything should recover.
The occasional machine here and there with a new physical failure shouldn’t pose a problem.
The system has handled such problems before.</p>
<p>But it’s not coming back.</p>
<p>After a bit of investigation, it is discovered that the databases are under extreme load.
Every query arrives, waits to be processed, and times out.
Almost nothing is actually getting through.</p>
<p>After checking on the caches, the engineers start to get a sinking feeling.
The caches are empty.
No query can get far enough successfully that the cache even gets populated with a single entry.
The caches are taking no load off the databases, and the databases are so overloaded that nothing actually progresses anywhere.
The system could stay down <em>indefinitely.</em></p>
<p>The co-workers whose responsibility this <em>isn’t</em> have started playing <a href="https://www.youtube.com/watch?v=JwZwkk7q25I">Homestar Runner music</a>.
“<em>The system is down! Do-do do-doo-doo!</em>”
It’s not helping.</p>
<h2 id="taking-the-load-off-an-overloaded-system">Taking the load off an overloaded system</h2>
<p>There are two very basic strategies for taking the load off of an overloaded system.</p>
<ol>
<li>Exponential back-off.</li>
<li>Circuit-breakers.</li>
</ol>
<p>The first strategy is all you really need for a closed system.
All retry mechanisms exponentially back-off (<em>Retry in 5s. Retry in 10s. Retry in 20s.</em>) exactly to reduce load on what might be an overloaded system.</p>
<p>If some part of the system becomes overloaded, then requests will either fail with errors or timeout.
If we are in the scenario at the beginning of this post, where <code class="highlighter-rouge">N</code> requests per second were causing the system to fail to make any progress, we’ll soon be getting <code class="highlighter-rouge">N/2</code> requests per second.
And then, <code class="highlighter-rouge">N/4</code> and <code class="highlighter-rouge">N/8</code>, and so on.
At some point, load will drop to the point where queries succeed, caches get populated, more of the overall load is taken off the databases, and the system recovers.</p>
<p>But this isn’t a complete picture when you have an <strong>open system</strong>—one where your load is originating with a large number of uncontrolled outside parties.
Getting users to slow down their requests doesn’t help much when you’re just dealing with too many users.</p>
<p>That’s where <a href="https://martinfowler.com/bliki/CircuitBreaker.html">circuit-breakers</a> come in.</p>
<p>The fundamental problem we have here is that clients don’t have enough information.
There’s just too many of them.
(Or… maybe just we don’t trust the malicious little jerks.)
So we need a more centralized part of the system to “know things” on their behalf.</p>
<p>A circuit-breaker is exactly that bit of knowledge <em>and policy</em>.
If an API’s overall error rate grows high enough, a circuit-breaker is meant to cut off load.
The circuit breaker trips, and incoming client requests are failed with errors immediately, without attempting to transmit them on to the rest of the system.</p>
<p>The key thing a circuit-breaker is doing here is correlating multiple requests together.
If the last thousand clients all timed out, there’s no reason for this client to expect differently.
This client just doesn’t know, and so would be eagerly starting a request exactly as if the system were functioning.
So the circuit-breaker protects part of a distributed system from load coming from far too many sources.</p>
<p>Both of these strategies rely on the same thing, however: the retrying node, and the circuit-breaking node, require information about the state the rest of the system.
To get that information, we need backpressure.</p>
<h2 id="backpressure">Backpressure</h2>
<p>Exerting backpressure is why we don’t retry except at the <em>ends</em> of a system.
If we’re going to fail, we want to propagate that failure information to where it can be used.
The most important place that failure information needs to go is to the end (typically, the client) making the original request to the system.
But it equally needs to propagate through the nodes implementing circuit-breaking.
Any attempt to put retrying strategies in the middle of the system begins <em>hiding</em> this information from the nodes where it’s needed.</p>
<p>As long as the system propagates backpressure, the load-reduction strategies we discussed have the information they need to work.</p>
<p>Backpressure is a <em>compositional</em> strategy.
Any node of a larger system that correctly handles and exerts backpressure can be put together with other nodes that do likewise, and you end up with a whole system that handles and exerts backpressure.</p>
<p>(Compositional properties are great, when we can get them. Take a moment to consider idempotence—it’s not necessarily compositional. We have to think about each operation in a systemic way.)</p>
<p>So, besides following the end-to-end principle and not trying to hide failures in the middle of a system, how can we end up screwing up backpressure?</p>
<h2 id="its-always-queues-isnt-it">It’s always queues, isn’t it?</h2>
<p>The first law of queues is: <em>Every queue should have a maximum size.</em>
Queues must not grow unbounded.</p>
<p>This is the most common failure for correctly handling backpressure.
A unbounded queue will just continue to accept input from producers, even though the consumer side has been completely overloaded.
Eventually, the queue system will die (after all, RAM and disk are an implicit upper bound…) but in the meantime, backpressure is eaten.
The queue pretends everything is fine while the system burns behind it.</p>
<figure class="boxed">
<p>Slightly surprisingly, it’s often the case that, when the queue fills up to its implicit limit (RAM, etc) and the machine falls over, the developer’s response is to think “ah, we need a beefier queue machine, clearly it can’t handle the load!”
This is almost always wrong.
The first goal is to ensure that the queue can’t fall over, and that it properly exerts backpressure when overloaded.
The correct course of action after that can’t be correctly determined until you’re collecting performance data from a system that’s not just inherently mis-designed.</p>
<p>Homework: Read <a href="https://ferd.ca/queues-don-t-fix-overload.html">“Queues Don’t Fix Overload” by Fred Herbert</a></p>
</figure>
<p>Backpressure is compositional, which is great when it means that we can build a whole from its parts.
But it does have a flip side: for the system to correctly handle backpressure, each part must do so.
Sticking an unbounded queue in the middle messes things up for the <em>whole system</em> pretty effectively.</p>
<p>The second law of queues is: <em>Have you considered a maximum size of 0?</em>
A queue of size zero in a distributed system is pretty much a load balancer.
The “queue’s” job is just directly connecting a producer with a consumer (a client with a server).
(Of course, those servers have TCP accept queues… there’s always a queue involved. But those also have bounds!)</p>
<p>It’s surprising how often queues are used when there’s actually little reason to do so.
The primary reason queues make sense is switching from synchronous to asynchronous requests.
There are other times to reach for one, of course, but they’re often viewed as something helpful.
They should be viewed as something potentially dangerous.</p>
<p>There’s nothing better than a queue for smoothing over problems today, only to make them explode bigger in the future.</p>
<h2 id="end-notes">End notes</h2>
<ul>
<li>Excellent talk on queues, with a bit of simulation and queueing theory throw in: <a href="https://www.youtube.com/watch?v=1bNOO3xxMc0">Zach Tellman - Everything Will Flow</a>. Pay attention to those long tails!</li>
</ul>Ted Kaminskitedinski@cs.umn.eduThe running topic for the last few posts is the basics of distributed systems. For today’s post, let’s start with a short story.The end-to-end principle in distributed systems2019-02-27T22:47:19-06:002019-02-27T22:47:19-06:00http://www.tedinski.com/2019/02/27/end-to-end-principle<p>The theme for the last couple weeks has been basic design considerations for distributed systems.
The name of the game is reliability <a href="/2019/02/12/distributed-systems-and-time.html">in the face of failure</a>.
Last week the topic was <a href="/2019/02/20/idempotence.html">retrying failed requests and idempotence</a>, and I made an odd claim: usually, we don’t retry, instead we just propagate small failures into larger failures.
Isn’t that a bit odd, considering we’re trying to be resilient to failure?</p>
<p>Not at all!
Distributed systems are complicated, and our goal is simplicity.
We can’t make a system reliable unless we can wrap our heads around its behavior.</p>
<h2 id="the-end-to-end-principle-in-networking">The end-to-end principle in networking</h2>
<p><a href="https://en.wikipedia.org/wiki/End-to-end_principle">The end-to-end principle</a> is originally all about designing networks.</p>
<p>The general idea is, to be resilient to failures, the end-points of the network need to handle failure anyway.
So… what’s the point of adding resiliency features to every node in the intermediate network?
All you’re doing is complicating all the intermediate parts, to accomplish the same thing that the end points could have just handled themselves anyway.
And building more complicated intermediate parts can make them <em>more</em> failure-prone!</p>
<p>This principle not only makes packet switched networks more attractive, it drives us towards a particularly “dumb network” design space.
We can get <em>more reliable systems</em> by building simpler, dumber intermediate parts, and having the ends of the system take responsibility for creating reliability from unreliable parts.</p>
<figure class="boxed">
<p><a href="https://en.wikipedia.org/wiki/Bufferbloat">Bufferbloat</a> was a nasty illness the internet began suffering from about a decade ago.
(Don’t worry, it’s largely cured now.)
The problem had a simple cause: the intermediate nodes of the internet, the routers, had casually begun to develop very large buffers.
Technology had advanced, RAM was cheap, why not try to make them more reliable?</p>
<p>The end result was a <em>less</em> reliable, <em>higher</em> latency, <em>slower</em> internet.
The trouble?
Routers could take higher rates of load before they started to actually drop packets (because the buffer were bigger, and so could grow bigger), and dropping packets was the signal to the endpoints that they should all back off a bit.
So routers didn’t ask the ends to back off quickly enough, and you quickly got the actual equivalent to a freeway traffic jam.</p>
<p>One path back to a higher reliability, low latency, higher bandwidth internet?
Program routers to randomly and deliberately drop packets when their buffers start to grow.
More on this concept in a future post!</p>
</figure>
<h2 id="but-were-not-building-networks">But we’re not building networks…</h2>
<p>The original articulation of the end-to-end principle is clearly about networking, but what does that tell us about distributed system design?
After all, we’re usually already committed to HTTP, never mind TCP/IP, so what does this mean for us?</p>
<p>As general principles, two things:</p>
<ol>
<li>
<p>Keep system behavior simple!
The responsibility for retrying on failure should only fall to the <em>originator</em> (the ends!) of the request.
If an intermediate request fails, the system should generally just propagate failure outwards, not initiate a retry by itself.</p>
</li>
<li>
<p><strong>Don’t accidentally start implementing a database!</strong>
It sounds like absurdly easy to follow advice, but this is actually the most common violation of the end-to-end principle in distributed system design.
If a client requests a write (of any form), and success is reported back before the action is actually taken with appropriately reliable database, you’re gonna regret it.
We want simplicity: we want to <em>use</em> a database, we don’t want our system as a whole to start <em>being</em> a database.</p>
</li>
</ol>
<h2 id="only-the-client-retries">Only the client retries</h2>
<p>Here’s where we get to my suggestion last week: a sub-request failure should just propagate failure.
So <em>most of the time</em> when we encounter an error, we just error in turn.
Only the client, upon being notified of failure (or upon timing out), should actually attempt a re-try.</p>
<p>Breaking this rule just complicates the system… for no benefit.
The system as a whole is not likely to become noticeably more reliable if a retry happens in the middle than if the client is notified of failure at attempts a retry from the end.</p>
<p>But two nasty things can happen:</p>
<ol>
<li>
<p>The system and its behavior can become more complicated.
Introducing intermediate retry logic can, in the aggregate (especially if not carefully and empirically measured), cause the system to become <em>less</em> reliable.</p>
</li>
<li>
<p>The intermediate nodes of the system don’t have all the context.
The client at the end is reliably executing its own code in making this request happen.
It’s in a more informed position to take care of implementing <em>backpressure</em> (which will be the subject of a future post).</p>
</li>
</ol>
<p>This is one way in which spurious retries can reduce reliabilitiy.
Retries happening too often, because they’re uncoordinated, can create load amplification that increases the chance of the system falling over or staying fallen down.
If the database is overloaded, the last thing you want is retry logic that sends the database even more requests in response.</p>
<h2 id="no-accidental-databases">No accidental databases</h2>
<figure class="boxed">
<img src="/assets/accidental-database-scenario.svg" />
<figcaption>
<p>A sequence of events:</p>
<ol>
<li>The client sends a request to the server.</li>
<li>The server replies, claiming success.</li>
<li>The server <em>then</em> attempts to write to the database in service of that request.</li>
<li>The write fails or times out, and the server happens to fall over.</li>
</ol>
<p>The result? Claimed success, but the attempted action simply disappears into the void.</p>
</figcaption>
</figure>
<p>In the figure above, we have a typical “accidental database” scenario.
The trouble is that the <em>system</em> (consisting of a server and its backing database, together) has effectively become a database on its own.</p>
<p>The successful response by the server essentially placed database-like responsibilities on that system.
Here is a situation where we might start by thinking “oh, well, if the write to the database fails, now my server <em>has</em> to retry it because I already told the client we succeeded!”
But this retry attempt is a false sense of security: what if the server itself now fails?
To handle that, we start needing persistence and replication and… <em>wait a second.</em></p>
<p>The actual root cause of the problem here was claiming success without hitting the database first.
This system should be <em>end-to-end</em> instead: the request must go from client to database, and the successful response should come from database back to the client.
We should not have the middle of the system start taking responsibilities like this.</p>
<p>So retries, outside of the client, are potentially a “smell.”
Why do we need to retry here?
Why can’t we just fail, and let the client retry?
If there’s a reason, does that reason amount to “oops, we started implementing a database and didn’t realize it?”</p>
<h2 id="queues-its-just-always-queues-isnt-it">Queues: it’s just always queues, isn’t it?</h2>
<p>Again, we might think this seems obvious.
Telling a client the request succeeded before we’ve attempted it <em>does</em> seem like an odd thing to do, right?</p>
<p>But the most common case of this works like so:</p>
<ol>
<li>The client sends a request.</li>
<li>The server puts that request into a queue, and responds with success to the client.</li>
<li>The queue falls over, and was never, ever properly treated as a database. (“It’s just a queue!”)</li>
<li>The request disappears into the ether, and tech support reminds the customer that computers are just haunted and do that sometimes.</li>
</ol>
<p>Developers seem to routinely just stick queues in the system, and then don’t take them seriously.
Queues should be treated as databases by default.
A typical queue:</p>
<ol>
<li>Needs persistence. (The power goes out, how will this request end up serviced later?)</li>
<li>Needs replication. (This machine unrecoverably dies, how will this request end up serviced?)</li>
<li>Takes end-to-end responsibilities. (This request gets <em>started</em>, but the downstream system then fails. Who will retry it? The queue is now also effectively a client, but worse because it can’t fail! So we need to track downstream status, in a database, to potentially allow another server to initiate retries, after this one fails.)</li>
</ol>
<p>And more! (Queues are totally going to show up in another future post. Here’s a spoiler: every queue should have a maximum size.)</p>
<h2 id="are-these-hard-rules">Are these hard rules?</h2>
<p>No!
This is an introduction to distributed systems that’s all about keeping you on a happy, simple path.
The rules in this thread can easily become inapplicable, but usually only because the complexity of the system has really started to <em>demand</em> it.</p>
<ol>
<li>You might be implementing a database! It’s not an accident, then.</li>
<li>For scalability reasons, a full-on database-based queue might not be able to support the load it’s under, and/or there might be application-specific reasons why occasionally failing to honor a request is okay.</li>
</ol>
<p>But distributed systems get complicated fast, and one of the easiest ways to avoid the pain is to keep as close to the end-to-end principle as possible.
Clients on one end, real databases on the other end, and don’t let responsibilities leak into the middle.</p>
<h2 id="end-notes">End notes</h2>
<p><a href="https://github.com/aphyr/distsys-class">Kyle Kingsbury’s notes on distributed systems</a> are a hoot.
My favorite advice on queues:</p>
<blockquote>
<ul>
<li>Queues do not improve end-to-end latency</li>
<li>Queues do not improve mean throughput</li>
<li>Queues do not provide total event ordering</li>
<li>Queues can offer at-most-once or at-least-once delivery</li>
<li>Queues do improve burst throughput</li>
<li>Distributed queues also improve fault tolerance (if they don’t lose data)</li>
<li>Every queue is a place for things to go horribly, horribly wrong</li>
</ul>
</blockquote>
<p>Everybody just <em>loves</em> to accidentally start implementing a database, and it’s always just queues isn’t it?</p>
<ul>
<li>(Added 2019/3/4) <a href="https://doi.org/10.1145/357401.357402">End-to-end arguments in system design</a> (1984. <a href="http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf">PDF available here</a>) is the classic paper on this subject.</li>
</ul>Ted Kaminskitedinski@cs.umn.eduThe theme for the last couple weeks has been basic design considerations for distributed systems. The name of the game is reliability in the face of failure. Last week the topic was retrying failed requests and idempotence, and I made an odd claim: usually, we don’t retry, instead we just propagate small failures into larger failures. Isn’t that a bit odd, considering we’re trying to be resilient to failure?