Orion Quin
The Everyday Still Water  ·  12 / 12

RETEST · THE ACCOUNTING

Read Your Own Water

The closing essay: on retesting, on how badly people estimate their own change, and on what this collection is actually worth.


Closing  ·  ~7 min read

§Back to the First Page

And the only thing that might have moved

The first essay asked you to notice what was present, without adjusting anything. Do that again now.

The exercise is identical. What is not identical is everything else — the week you are in, what has happened since, and, most awkwardly, what you now understand the question to be asking. A retest tells you that two readings differ. It does not tell you the practice caused the difference, and the science section below is about why that is harder to establish than it looks.

What follows is about the thing everybody does wrong at this point, which is to look first and form an estimate afterwards.

§The Two Ways This Goes Wrong

Both about what the number means

One reads it as a verdict. The other refuses to look.

The verdict

The retest number becomes a judgment on the person. It is a measurement of one instrument at one moment, taken on a day that had its own weather, and treating it as a summary of twelve weeks is the amplitude error from essay two.

The refusal

Not retesting, because the answer might be unflattering. Which guarantees the only available information about whether any of this worked is a memory, and the next section is about how good those are.

Predict the change, then measure it, and keep the difference.the author, on the only honest retest

§The Line

The order, again

The same order as essay three, for the same reason and with more at stake: an estimate made after seeing the result is not an estimate.

The difference between what you predicted and what you found is worth more than either number — but be exact about what it measures. It measures your calibration: how well you estimate your own change. It does not measure whether the practice worked, because a prediction tests your expectations rather than the intervention. For that you would need a comparison you do not have.

§The Story

The accounting

The Sufi orders keep a practice called muḥāsaba — the accounting — performed daily and, in a longer form, at intervals.

The Mussar collection in this library describes the Jewish version of the same idea, and the two developed separately in traditions that were in contact and in conflict. I am not going to present the similarity as significant; two literatures concerned with self-examination arriving at a ledger is not surprising.

What is worth borrowing is the interval structure. Daily for noticing, periodically for direction, and the two ask different questions.

§The Science

Mostly evidence about how badly people estimate their own change

Retrospective estimates of personal change are poor, and they are poor in a specific way: people infer what they used to be like from what they are like now plus a theory about whether they have changed.4 Somebody who believes an intervention works will remember a worse starting point.

The related problem is response shift. If a practice changes what you understand a question to mean, your before and after scores are not measuring the same thing, and the change may be in the instrument rather than in you.5

Both of these make the sealed prediction non-optional. It is the only thing that turns a retest into information rather than a confirmation of whatever you already believe about whether this helped.

§Where the Traditions Are Silent

And one is sceptical of the whole exercise

The Daoist material is absent and hostile. It would regard the measurement of one’s own progress as a well-designed way of generating the self-consciousness that was the problem, and the more diligent the measuring the worse the condition.

That objection has force and it is not answered by this collection. What I would say against it is only that the alternative — not measuring — leaves you with a memory, and the section above is about what memories of your own change are worth.

The heroic material is silent for the reason it has been silent throughout: no account of development means nothing to retest.

§What This Collection Is Actually Worth

The accounting the closing essay owes

This is the collection in the library that runs on research rather than on a tradition, and it should be as hard on itself as the others were on their texts.

What is well supported: suppression is expensive and the costs land partly on the person opposite you. Reappraisal helps for uncontrollable situations and hurts for controllable ones. Acceptance beats struggle for what cannot be moved. Affect labelling reduces intensity a little. Social connection predicts health at a size worth taking seriously. Repetition in a consistent context is how anything here becomes automatic.

What is thinner than its reputation: the breathing station, whose famous version rests on one trial of 108 mostly-undergraduates that found no heart-rate-variability effect and no anxiety advantage over meditation, and was not powered to compare the breathing arms with each other. The habit number everybody quotes comes from a 1960 book about cosmetic surgery. The neurochemical story about co-regulation should be left alone. Emotion differentiation is mostly correlational.

What is not settled at all: whether any of this transfers from the situations it was practised in to the ones that matter. The transfer literature is not encouraging and this collection did not address it.

And the pattern that runs through it: five of these twelve stations — suppression, reappraisal, acceptance, recruitment, and the breathing — can all be used to get somebody to stop raising a problem. In each case the tell is the same one the Mussar collection found: the instruction travels down the status gradient and rarely up it. A regulation skill offered in place of a fix is a management technique, whatever the evidence behind it says.

The case against

The strongest objection to the whole collection.

Everything here makes a person better at tolerating their circumstances. Nothing here changes any circumstance, and a set of twelve techniques for being all right with how things are is exactly what an arrangement that does not intend to improve would want distributed.

The partial answer is the sorting question in essays seven and nine, which is designed to route controllable things to action rather than to regulation. Whether that is sufficient is a fair question and I do not think it is fully answered.

§The Practice

Predict, then look

Write the full prediction before touching anything. What has changed, in what direction, by how much? Seal it.

Then retest — the first essay’s exercise, and whatever else you have been marking.

Keep the difference between prediction and result. That is about your calibration, which is the durable finding; the retest score itself is one day’s weather, and neither number establishes that the practice did anything.

And decide about the year, once: which one of these twelve is worth keeping? One. The repetition essay is why.

§A Week, Then a Decision

And the decision is the point

Short, because the work was done in the previous eleven.

Day 1
Predict, in full, in writing. Before looking at anything.
Days 2–4
Retest. The first essay’s exercise, and any marks you kept.
Days 5–6
Compare prediction with result. Write one sentence about the direction of your error.
Day 7
Decide: which single one of these is worth carrying into the year? Discard the other eleven without guilt.

§What This Leaves You With

A deliberately small claim

Twelve stations, and the claim at the end is smaller than the length of the collection suggests. You can probably detect a state earlier than you could, and you have a few instruments with different evidence behind them and different failure modes.

What the retest can tell you is that two readings differ, and how badly you estimated the difference. What it cannot tell you on its own is that this collection caused anything: there is no control condition, the circumstances moved, and the practice may have changed what the question means to you. That is not a reason to skip it — it is a reason not to read it as a verdict on the twelve weeks.

Whether that transfers to the situations that actually matter is not established, and this collection did not find out.

And the thing worth carrying out of it is not a technique. It is the sorting question: can this be changed? Because five of these twelve stations are, in the wrong hands, ways of getting somebody to stop asking it.

§The Takeaway

Write the full prediction before you look at anything, then retest. Keep the difference between prediction and result — that is about your calibration, and it is the durable finding.

Then choose one station to carry into the year and discard the other eleven. And keep the sorting question: can this be changed?

§Sources

Research is cited by author and publication, with effect sizes and replication status given where they bear on whether to believe the claim. Where something rests on a single trial, or on a literature with known quality problems, the essay says so in the body rather than in a footnote. Traditions are quoted only where they add something, and the collection names the stations where they have nothing to say.

  1. On muḥāsaba — see Annemarie Schimmel, Mystical Dimensions of Islam (1975); the Jewish counterpart is in the Mussar collection in this library. The similarity is not presented as significant.
  2. Ross, M. — “Relation of implicit theories to the construction of personal histories,” Psychological Review 96(2) (1989). People inferring their past selves from present selves plus a theory of change.
  3. Conway, M. & Ross, M. — “Getting what you want by revising what you had,” JPSP 47(4) (1984). Why believing an intervention worked produces a worse remembered baseline.
  4. Schwartz, C. E. & Sprangers, M. A. G. — “Methodological approaches for assessing response shift,” Social Science & Medicine 48(11) (1999). When before and after are not measuring the same thing.
  5. Barnett, S. M. & Ceci, S. J. — “When and where do we apply what we learn?” Psychological Bulletin 128(4) (2002). The transfer problem this collection did not address.
  6. Zhuangzi, ch. 2 — the scepticism about self-measurement, cited as the objection this collection does not answer.