Revenue Management Pricing & forecasting pipelines
4 min read Architecture

Implementing circuit breakers for PMS failover

When the property management system stops responding, the worst thing a pricing pipeline can do is keep hammering it — piling up timeouts, exhausting connection pools, and stalling every request behind the dead dependency. A circuit breaker cuts that off: after enough failures it stops calling the failing PMS, serves a fallback, and probes for recovery before resuming. This guide implements one, extending Security Boundaries & Fallback Routing in the Core Architecture & Pricing Taxonomy pillar.

Prerequisites

  • Python 3.11+
  • Standard library time, enum, logging, dataclasses
  • A PMS client call that can time out or raise
  • A defined fallback (last-known-good inventory, or a safe default)
  • A monotonic clock for timing state transitions

Step 1 — Model the three breaker states

A circuit breaker is a small state machine: closed (calls flow), open (calls are short-circuited), and half-open (a trial call probes recovery). Encode the states and the thresholds that move between them.

python 
from __future__ import annotations

import logging
import time
from dataclasses import dataclass, field
from enum import Enum

logger = logging.getLogger("pms.breaker")


class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


@dataclass
class Breaker:
    failure_threshold: int = 5
    reset_timeout: float = 30.0        # seconds open before probing
    _state: State = field(default=State.CLOSED, init=False)
    _failures: int = field(default=0, init=False)
    _opened_at: float = field(default=0.0, init=False)

Step 2 — Decide whether a call may proceed

Before each call, ask the breaker for permission. When open, it stays open until the reset timeout elapses, then allows a single half-open trial.

python 
    def allow(self, now: float) -> bool:
        if self._state is State.CLOSED:
            return True
        if self._state is State.OPEN:
            if now - self._opened_at >= self.reset_timeout:
                self._state = State.HALF_OPEN
                logger.info("breaker half-open: probing PMS")
                return True
            return False
        # HALF_OPEN: allow exactly one trial
        return True

Step 3 — Record outcomes and transition

Success in half-open closes the breaker and clears the failure count; a failure re-opens it. In closed state, consecutive failures accumulate until the threshold trips the breaker open.

python 
    def on_success(self) -> None:
        if self._state is State.HALF_OPEN:
            logger.info("breaker closed: PMS recovered")
        self._state = State.CLOSED
        self._failures = 0

    def on_failure(self, now: float) -> None:
        self._failures += 1
        if self._state is State.HALF_OPEN or self._failures >= self.failure_threshold:
            self._state = State.OPEN
            self._opened_at = now
            logger.warning("breaker open after %d failures", self._failures)


def get_inventory(breaker: Breaker, pms_call, fallback):
    now = time.monotonic()
    if not breaker.allow(now):
        logger.info("breaker open; serving fallback")
        return fallback()
    try:
        result = pms_call()
        breaker.on_success()
        return result
    except Exception as exc:                 # timeout, connection error, 5xx
        breaker.on_failure(time.monotonic())
        logger.error("PMS call failed: %s; serving fallback", exc)
        return fallback()

Serving the fallback the instant the breaker is open — rather than attempting the call and waiting for it to time out — is what protects the rest of the pipeline: a fast, known-good default beats a slow failure every time.

Verification and testing

python 
def test_breaker_opens_then_recovers() -> None:
    breaker = Breaker(failure_threshold=2, reset_timeout=10.0)
    t = 1000.0
    for _ in range(2):
        assert breaker.allow(t) is True
        breaker.on_failure(t)
    assert breaker.allow(t) is False          # open, short-circuits
    assert breaker.allow(t + 11) is True      # half-open probe after timeout
    breaker.on_success()
    assert breaker.allow(t + 12) is True       # closed again

Common pitfalls and edge cases

  • No half-open probe. Reopening on a timer without a trial call flaps; use a single half-open probe to confirm recovery.
  • Counting expected errors. A 404 for an unknown room is not a PMS outage; only trip on connectivity and 5xx failures.
  • Shared breaker across dependencies. One breaker for the PMS and the channel manager conflates failures; use one per dependency.
  • Wall-clock timing. An NTP jump can prematurely close or hold a breaker; time transitions with time.monotonic().
  • Silent fallback. Serving stale inventory without emitting a metric hides an ongoing outage; alert whenever the breaker is open.