Idempotency Under Failure: Six RabbitMQ Tests

Idempotency matters most at the point where a system cannot tell whether an operation finished. A consumer can commit a database change and die before acknowledging the message that triggered it. The broker then has a legitimate reason to deliver that message again. The problem is not that redelivery is broken; it is that a second delivery must not become a second business effect.

I built a small RabbitMQ and SQLite laboratory to test that boundary rather than relying on an architectural slogan. On November 9, 2026, I ran six isolated scenarios covering normal processing, unacknowledged redelivery, process termination on either side of a database commit, duplicate work across two processes, and a new worker starting after the original exited. All six tests reported PASS. That result is useful, but its scope matters: it describes the specific state changes and failure windows I exercised, not universal exactly-once execution.

Idempotency Results in Brief: Six Tests, One Durable Effect

My central finding is that the broker can deliver work more than once while the application’s committed business effect remains singular. In this lab, that behavior came from a persistent message identifier, a database uniqueness constraint, and an idempotency record committed in the same SQLite transaction as the business effect. A manual RabbitMQ acknowledgment followed only after the database transaction succeeded.

• Observed: all six individual experiments passed their scripted assertions; each finished with exactly one effect row and one processed-message row for its unique test identifier.
• Observed: the broker marked actual redeliveries in T02, T03, and T04 with redelivered: true.
• Observed: two distinct worker processes handled two copies of the same logical operation in T05; only one created the effect.
• Not established: broker failover, machine power loss, high-throughput behavior, distributed replication, and exactly-once effects in external services.

This is a test of an idempotent consumer under selected failures. It is not a claim that RabbitMQ and SQLite participate in one global transaction.

Why Message Redelivery Is Normal, Not an Exception

Consider a payment-related job, inventory reservation, or account-credit operation. The publisher sends a message. The worker receives it and updates a database. Before the worker can send its acknowledgment, its process disappears. The database may now contain the intended update, while RabbitMQ still considers the delivery unfinished. Replaying the message is the safer broker behavior; applying the financial or inventory effect twice is the application’s bug.

RabbitMQ distinguishes publisher confirms from consumer acknowledgments. A publisher confirm says the broker has accepted responsibility for a published message under its queue semantics. Consumer acknowledgments say particular deliveries can be considered handled. These signals are separate. Neither creates a transaction spanning an external SQL database and the message broker.

The RabbitMQ reliability guidance explains why consumers should anticipate duplicate deliveries and design processing to be idempotent. That recommendation is the starting point for this experiment. In an at-least-once delivery model, the more interesting question is exactly where the idempotency guarantee begins and ends.

The Idempotency Boundary at the Commit–ACK Failure Window

The critical sequence is straightforward: receive, validate, begin the database transaction, check the idempotency key, record the business effect if it is new, commit, then acknowledge the delivery. If a worker fails before commit, its partial SQL writes must not survive. If it fails after commit but before ACK, replay must find the earlier committed operation and avoid applying the effect again.

That idempotency ordering prioritizes correctness over the illusion of a single delivery. Acknowledging before the commit would avoid some duplicate delivery attempts, but it would also allow the broker to discard a message whose database work never became durable. The order is a design decision with a specific recovery contract, not merely a code-style preference.

Test Environment and Method

I used a local, single-node broker and a file-backed SQL database. The tests were executed on an Apple Silicon macOS environment; the evidence manifest identifies the platform as darwin and the architecture as arm64. The recorded versions were Node.js v26.12.1, RabbitMQ 4.4.0, SQLite 3.54.1, and the pinned [email protected] dependency. RabbitMQ accepted AMQP traffic on 127.0.0.1:5672; no public broker service was needed.

Each test used a newly generated logical message ID and checked the relevant rows using that ID. T03 through T06 also generated independent queue names. T01 and T02 used dedicated fixed laboratory queues, and the scripts asserted that those queues were initially empty. This matters when reproducing them: an interrupted prior run may leave a message in a fixed queue, and that should be investigated rather than silently purged.

For the SQL state, I used two tables. business_effects represents the actual operation, with a unique message_id. processed_messages records the same identifier as its primary key. In the tests, both writes happened inside one transaction. The second table is not a replacement for the first table’s uniqueness constraint; together they make the intended invariant easier to inspect and defend.

Six Message Idempotency Tests and Their Observed Results

I deliberately separated normal delivery from three distinct replay mechanisms. A passing result only applies to the mechanism that was actually exercised. The table below reports outcomes from the original JSON evidence, not estimates or averages.

There was no repeated-trial sample or benchmark workload in this study. Six PASS results mean that six scripted scenarios passed once each; they do not provide a probability of failure, mean latency, performance improvement, or statistical confidence interval. In particular, the short recorded start-to-finish intervals should not be converted into throughput or latency claims: they include test orchestration and differ by scenario.

T01: Establishing the Normal Delivery Path

The baseline publish used a confirm channel, a durable classic queue, a persistent message, and manual consumer acknowledgment. The script then wrote both SQL rows, committed them, and sent ACK. The evidence showed publisher_confirmed: true, redelivered: false, and exactly one row in each table. The queue had zero ready messages at the final check. This verifies the idempotency happy path, not failure recovery.

T02: Commit Succeeds, the Channel Closes Without ACK

Next I kept the SQL commit but deliberately withheld the delivery acknowledgment. Closing the consumer channel returned the unacknowledged delivery to RabbitMQ. A second channel obtained it with redelivered: true. The first processing attempt created the business effect; the second did not. Both SQL tables still contained one matching row. That is real broker redelivery and successful idempotency enforcement, but it is a controlled channel closure rather than an abruptly terminated operating-system process.

T03: Killing the Worker Before COMMIT

For the first crash experiment, a child process received the message and wrote inside an open SQLite transaction. It signaled the parent that it had reached a checkpoint before COMMIT. The parent independently observed zero committed effects, then sent SIGKILL to that child. After the process exited, the database still had zero effect rows and zero processed-message rows for the test ID.

RabbitMQ subsequently redelivered the unacknowledged message with its redelivery flag set. The recovery path completed the transaction and left one effect and one processed-message record. The observed idempotency boundary is important: a message’s receipt, and even uncommitted SQL writes, did not count as a completed business operation.

T04: Killing the Worker After COMMIT but Before ACK

T04 is the mirror image. The child committed the SQL operation, deliberately sent no ACK, and reported a post-commit checkpoint. The parent confirmed the committed rows were visible, then terminated that specific worker with SIGKILL. The rows remained after the crash. RabbitMQ redelivered the same message, but the replacement processing path found the existing key and returned without creating another effect.

The archived T04 result recorded second_processing_created_effect: false, one final effect, one processed-message record, and an empty ready queue after the final acknowledgment and channel close. This experiment most directly supports the practical value of idempotency across the commit–ACK gap. It does not mean the original ACK and database commit were atomic with each other; they were not.

T05: Idempotency Across Two Worker Processes

T05 tested duplicate message processing by publishing two separate messages carrying the same logical business ID. Two independent Node.js child processes, with distinct PIDs, each obtained one copy. Both reported READY before the parent sent a common GO signal. One worker created the effect and the other found the duplicate. The JSON result recorded created_effect_workers: 1 and duplicate_suppressed_workers: 1; the database again contained only one effect row.

Here the distinction between concurrency and proof of a race is essential. The shared GO signal created a common start point, but the experiment did not instrument the exact time each worker reached SQLite’s write lock. I can conclude that two separate processes handled duplicate business work without producing a duplicate database effect in this run. I cannot conclude that every possible lock-interleaving was explored or that the design has been stress-tested at production concurrency.

T06: Idempotency Survives a Worker Restart

Finally, the first worker processed an original message and exited normally. The parent confirmed that one effect and one processed-message record existed. A new message containing the same business ID was then published, and a completely new worker process handled it. The recorded worker PIDs were 10095 and 10096. The second worker found the earlier record, created no new effect, and left the final counts at one and one.

Notice that the second message had redelivered: false. That is correct: T06 was a new publication of duplicate logical work, not automatic redelivery of the same broker delivery. It demonstrates that idempotency state survived the worker’s exit and a subsequent fresh process. RabbitMQ itself was not restarted, and no machine reboot or power-failure experiment was performed.

The Database Idempotency Boundary That Prevented Duplicate Effects

The essential implementation choice was to place the idempotency lookup, the business-effect insertion, and the processed-message insertion inside the same SQLite transaction. The scripts used BEGIN IMMEDIATE to request a write transaction before checking whether the key had already been processed. With WAL mode enabled, SQLite can permit readers while serializing concurrent writes; SQLite’s transaction documentation also makes clear that an immediate write transaction can encounter SQLITE_BUSY when another writer holds the necessary lock.

A shortened excerpt of the tested pattern is below. This is a conceptual extract rather than a complete standalone worker; the complete source lives in the laboratory archive.

db.exec('BEGIN IMMEDIATE');
try {
  const previous = db.prepare(
    'SELECT message_id FROM processed_messages WHERE message_id = ?'
  ).get(payload.message_id);

  if (!previous) {
    insertBusinessEffect(payload);
    insertProcessedMessage(payload.message_id);
  }

  db.exec('COMMIT');
} catch (error) {
  db.exec('ROLLBACK');
  throw error;
}

channel.ack(message); // only after successful COMMIT

In the actual scripts, inserts were executed as prepared SQLite statements rather than the abbreviated helper names shown above. The business-effects table imposed a unique constraint on message_id, while processed messages used that field as the primary key. Those constraints mattered as a second line of defense if code paths changed or duplicate processing attempts converged on the same identifier.

There is also a subtle modeling question. A message ID identifies a publication only if the publisher preserves it across retries. For domain correctness, the better idempotency key is often a stable operation identifier that represents the intended business action. Two different publications can represent the same operation, as T05 and T06 deliberately showed. If the publisher generates a fresh logical operation key each time it retries, a perfectly implemented table constraint cannot recognize those requests as duplicates.

An Idempotency Key Is a Business Contract

Before using this design in a production system, I would define who creates the key, when it expires, which tenant or account it belongs to, and what should happen when the same key arrives with a different payload. The lab held the payload amount at 100 and, in several test paths, checked that the earlier effect matched it. That is not a comprehensive idempotency payload-conflict policy. A real service may need a normalized payload hash, a namespace such as tenant plus operation type, and an explicit rejection path for conflicting reuse.

Retention also matters. A processed-message record cannot prevent duplication forever if it is deleted before a delayed retry arrives. Conversely, retaining every key indefinitely creates storage and maintenance costs. The safe window depends on producer retry contracts, maximum delivery delay, domain consequences, and audit requirements. Those decisions belong to the operational idempotency design, not something these six tests measured.

Why Idempotency Is Not Universal Exactly-Once Processing

Within the tested database boundary, duplicate delivery did not create duplicate committed effects. Outside that boundary, there are open questions. If a worker commits the database and then sends an email, makes a card-network request, or calls an external API, the SQLite transaction does not roll back the outside action. A replay can perform that external action again unless the downstream service also supports a reliable idempotency contract or the workflow uses a different coordination mechanism.

Nor does a durable classic queue prove the behavior of a replicated RabbitMQ cluster. The lab explicitly used x-queue-type: classic on one local node. RabbitMQ documents classic queues as nonreplicated and discusses quorum queues as an alternative for replicated data safety. I did not run quorum queues, simulate leader loss, partition a network, or restart the broker while messages were pending. The existence of a publisher confirm is important, but it should not be stretched into evidence for an untested failure topology.

There is a second kind of uncertainty around consumer ACKs and idempotency. The source code called channel.ack(message) after commit and checked the queue state. That shows the intended API call and subsequent observations, but AMQP consumer acknowledgments are not themselves a separate end-to-end database receipt protocol. For this study, the stronger evidence is the observed redelivery behavior and the database invariant, not an assertion that the database and RabbitMQ executed one atomic commit.

Beyond an Idempotency Table: Outbox, Inbox, and State Transitions

The appropriate idempotency solution depends on the boundary being protected. An inbox or processed-message table is a good fit when the consumer’s business effect is another write in the same transactional database. It lets the consumer detect repeat logical operations without requiring the broker to promise single delivery. That is the idempotency boundary I tested.

A transactional outbox addresses a different failure window: a service changes its own database but may crash before publishing the resulting event. Recording the event in the same database transaction and delivering it asynchronously avoids relying on a cross-system atomic commit, although the outbox publisher can still send duplicates. Outbox and inbox are complementary patterns; neither eliminates the need to define stable operation identity and downstream behavior.

Sometimes the business operation itself can be made naturally idempotent. Setting an order to a known state is different from incrementing a balance. If a message means “set reservation status to confirmed” and its preconditions are checked, replay may be harmless. If it means “add 100 units,” replay is dangerous without a unique operation contract. A cache-only deduplication window may be acceptable for low-impact short-lived work, but its eviction or crash behavior should be evaluated against the business risk before treating it as a correctness boundary.

How to Reproduce the Six Scenarios

The audit archive contains the exact t01.mjs through t06.mjs scripts, package.json, a pinned npm lockfile, six raw JSON evidence files, and evidence/manifest.json. I compared the archive contents with its SHA-256 manifest: every listed source and evidence file matched its recorded hash. This verifies internal archive consistency, not independent authorship or execution authenticity.

To reproduce the test suite, provision a local RabbitMQ instance accessible at 127.0.0.1:5672, use the archived Node.js version where possible, extract the source package, install pinned dependencies with npm ci, and run the six scripts individually. The following sequence is only the execution outline; it assumes RabbitMQ is already running and the archive has been extracted.

npm ci
node --check t01.mjs
node t01.mjs
node t02.mjs
node t03.mjs
node t04.mjs
node t05.mjs
node t06.mjs

Each script writes a JSON record under evidence/ and prints the observed assertions. New runs generate different message IDs, queue names where applicable, process IDs, and timestamps. Do not treat a change in those identifiers as a test failure. If a script fails, preserve its evidence record and broker state before rerunning; the T01 and T02 fixed queue names in particular may require manual investigation after an interrupted attempt.

The supplied scripts use RabbitMQ’s localhost-only default guest account for a deliberately isolated laboratory. Do not copy that connection policy unchanged into a network-accessible or production environment. Create scoped credentials and secure transport appropriate to the deployment instead. Likewise, the scripts do not implement production retries, dead-lettering, monitoring, migrations, bounded retention, or comprehensive payload validation.

Idempotency Evidence, Limitations, and What I Would Test Next

The evidence package is unusually useful because it includes executable source, machine-readable per-test assertions, pinned dependencies, and a manifest rather than only screenshots. Still, it is one local experimental session. I reviewed the archived implementation and compared the saved JSON records with the script assertions; I did not independently rerun the tests on the original Mac, and the zip does not contain a full SQLite database snapshot or broker data directory.

For a stronger next round, I would run repeated trials with randomized interruption timing; instrument lock acquisition around BEGIN IMMEDIATE; test a broker restart with messages pending; examine network partitions and replicated quorum queues; simulate database I/O errors and disk exhaustion; and verify how the service handles conflicting payloads under one key. I would also evaluate a real external side effect using a downstream idempotency key. None of those idempotency or resilience outcomes belong in this article as measured results until they are actually tested.

There are two defensible conclusions. First, in the six observed cases, redelivery or duplicate work did not produce a duplicate committed SQLite effect. Second, the system achieved that result by moving the business-level idempotency boundary into a transaction it could control. I would not describe that as a universal exactly-once delivery guarantee. I would describe it as a tested, narrow, and practical correctness contract that can be extended only when the next system boundary has evidence of its own.

Lab Review Record

Experiment date: November 9, 2026. Evidence scope: six individual scripted scenarios, six PASS records, one local RabbitMQ node, one SQLite database file. Independent verification: code-to-JSON consistency and SHA-256 archive checks; no independent execution replay. Decision: suitable evidence for the narrow idempotent database-effect claim, not for an end-to-end exactly-once guarantee.

Primary technical references: RabbitMQ: Consumer Acknowledgments and Publisher Confirms; RabbitMQ: Reliability Guide; RabbitMQ: Classic Queues; SQLite: Transactions; SQLite: PRAGMA Synchronization and WAL.

Related reading: Browse the Software Development Articles collection or return to the Articles hub.