People rarely ask “why P?” in isolation; they ask “why P rather than Q?” Miller argued that such contrastive questions lie at the heart of human explanation. We tested this idea on Cuttlefish, an end-user planning application that schedules a home battery and appliances under a dynamic energy tariff.

The problem

Cuttlefish recommends a one-week, hourly schedule for a battery and four appliances (washer, dryer, dishwasher and electric vehicle) that satisfies user requirements while minimising cost under a dynamic tariff, such as Octopus Agile in the UK. We formalise this as a non-stationary finite-horizon planning problem, in which actions, transitions and costs vary with the timestep. Because the battery and appliances share one energy bill, the cost of a joint plan makes this a multi-effector problem.

Off-the-shelf PDDL planners cannot handle these features: few support timed initial fluents, and none support costs that depend on the affected state variables. We therefore built a domain-dependent A* planner in Rust, served through a web interface.

Contrastive questions as model restrictions

Following Krarup et al. (2021), a contrastive question restricts the planning model. In Cuttlefish, a user adds appliance requirements $\Phi’$ (e.g., “run the dishwasher only after 20:00”), producing a restricted model $\Pi’$ whose plans are also plans of $\Pi$. The explanation is the pair $\langle \pi, \pi’ \rangle$: the original and alternative schedules shown side by side, together with their cost difference:

“Your total bill increases by C(π′) − C(π) pence.”

The study

• 128 lay participants recruited via Prolific and randomly assigned to a treatment or a control group. Control group (CG) viewed the recommended schedule only. Treatment group (TG) could also ask contrastive questions (at least five per scenario).
• Two personas with different scheduling demands: Alice, an office worker, and Bob, a night-shift worker with tighter time windows. Imagine, for instance, that you cannot run the washer at night without disturbing the neighbours.
• Responses were collected on 7-point Likert scales and compared using one-tailed Welch’s t-tests.

Key findings

1. Explanations significantly increase satisfaction in both scenarios. Participants in the TG reported higher satisfaction with the recommended schedule for both Alice (p = .033) and Bob (p = .010).

2. Understanding and helpfulness trend upward. TG means are higher on every measure, but only satisfaction is significant in both scenarios. Scores are lower for Bob, consistent with his more complex requirements.

3. Background matters, but only with explanations. Within the TG, participants with a STEM background rated the system higher (summated score 35.5 vs 32.4, p = .016). Smart-home experience had no significant effect.

4. Comprehension shifts from doubt to detail with explanation. CG participants often questioned individual tasks or claimed that cheaper schedules existed. TG participants more readily accepted the schedule’s optimality; their remaining concerns were mostly about details such as washer–dryer dependencies, which the model deliberately omitted for simplicity.

Takeaways

Contrastive explanations formulated as model restrictions are a practical, interactive XAI mechanism for lay users of planning systems. Their benefit, however, comes from lessons of classic UX design: responsiveness, question forms that match user intent, and domain features users expect.

Based on “A User Study on Contrastive Explanations for Multi-Effector Temporal Planning with Non-Stationary Costs” (ICTAI 2024), with Kevin McAreavey and Weiru Liu, and its extended version in my MScR thesis. Written with the help of our dear friend Claude.