"Sent" is not the same as "delivered". Delivery guarantees in NATS

Part 1 of a series. Part 2 will be about what happens when one worker stops keeping up.

Versions used in this text: nats-server 2.11.17, nats-py 2.15.0, nats CLI 0.4.0. Every console output in this article comes from running the included code, not from the documentation. Verified 2026-09-21.

Open Table of contents

1. One line we don’t understand
- The three levels, briefly
2. The lab: four containers
3. Core NATS: a message for whoever happens to be listening
4. JetStream: a message that waits
5. The cost: duplicates
6. Decision table
7. What’s next

1. One line we don’t understand

Say you are writing a web shop. A customer places an order, and your application has to send them a confirmation email. You don’t want to send that email in the same place where you handle the HTTP request, because the mail server can be slow and the customer shouldn’t have to wait. So you split it in two: the application announces the fact “order number 7777 was created”, and a separate process listens for such announcements and sends the emails.

Passing announcements like that around is what a message broker is for. NATS is one of them: a single, very fast server that processes connect to, some publishing messages, others receiving them. In Python it looks like this:

await nc.publish("orders.new", b"order 7777")

And this is where the problem this article is about begins. That line has an await, so it looks like something we are waiting for. The question is: what exactly are we waiting for? Once it returns, has the message reached the server? Has it reached the email process? Has the email been sent? And if the email process happened to be restarting at that moment, what happened to the message?

The answer in Core NATS is: we are waiting for practically nothing, and the message may have vanished without a trace. This is not a bug or a misconfiguration. It is a deliberate design decision, called a delivery guarantee, and it has a flip side: a mode where the message will definitely arrive, but may arrive more than once.

The three levels, briefly

Before we get into NATS, it is worth naming things. A delivery guarantee describes how many times a single message may be delivered to a receiver:

at-most-once — the message is delivered zero or one time. Never twice, but sometimes not at all. No acknowledgements, no retries, no memory.
at-least-once — the message is delivered at least once, meaning once or more. It will never be lost, but you may get it twice and you have to be prepared for that.
exactly-once — exactly one delivery. I put this in quotes and will come back to it in chapter 5, because in a distributed system the phrase is at best a shorthand.

Core NATS gives you at-most-once. JetStream — the NATS subsystem with persistent storage — gives you at-least-once. This whole article is an answer to the question of when you need the second one and what it costs, because it costs something real and hardly anyone writes about it.

I assume you know Python and roughly understand async/await. I assume no knowledge of NATS, message brokers or event-driven architectures.

2. The lab: four containers

Every example in this text is runnable and everything runs on a single docker compose. This is deliberate: to see how messages get lost, you need to be able to kill a process at a specific moment, and a Docker service can be stopped and resumed with one command.

services:
  nats:
    image: nats:2.11-alpine
    command: ["--jetstream", "--store_dir=/data", "--http_port=8222", "--name=lab"]
    ports: ["4222:4222", "8222:8222"]
    volumes: [nats-data:/data]

  nats-box:          # the `nats` CLI, for inspecting the server
    image: natsio/nats-box:latest
    environment: { NATS_URL: "nats://nats:4222" }
    command: ["sleep", "infinity"]

  publisher:         # a Python container we run scripts from
    build: .
    environment: { NATS_URL: "nats://nats:4222" }
    volumes: [".:/app"]

  core-sub:          # the subscriber we are going to kill
    build: .
    environment: { NATS_URL: "nats://nats:4222" }
    volumes: [".:/app"]
    command: python demo/01_core_sub.py

The --jetstream flag enables persistent storage; without it the server can only pass messages along. --http_port=8222 exposes a monitoring endpoint at http://localhost:8222/jsz, which shows the state of streams and consumers as JSON.

To start:

docker compose up -d
docker compose exec publisher python demo/01_core_pub.py "test"

3. Core NATS: a message for whoever happens to be listening

Core NATS has no storage. The server keeps a table in memory: who is connected and what they are listening for. When a message comes in, the server copies it to the sockets of the subscribers whose subject matches — and immediately forgets about it. It does not write it anywhere, does not wait for an acknowledgement, has no way to repeat it.

A subject is a plain dotted string, such as orders.new or demo.core. A subscriber gives the same string (or a pattern using * or >) and receives matching messages. That is all the addressing there is in NATS — there is no such thing as a pre-created queue; a subject exists because someone publishes on it.

The subscriber:

async def main() -> None:
    nc = await nats.connect(os.environ["NATS_URL"])
    sub = await nc.subscribe("demo.core")
    print("[sub] listening on demo.core", flush=True)

    async for msg in sub.messages:
        print(f"[sub] received: {msg.data.decode()}", flush=True)demo/01_core_sub.py

The publisher:

async def main() -> None:
    nc = await nats.connect(os.environ["NATS_URL"])
    for text in sys.argv[1:]:
        await nc.publish("demo.core", text.encode())
        print(f"[pub] sent: {text}", flush=True)
    await nc.flush()
    await nc.drain()demo/01_core_pub.py

Notice that the subscriber says nothing about acknowledging. There is nothing to acknowledge.

Experiment: kill the subscriber

docker compose up -d core-sub

# 1. the subscriber is alive
docker compose run --rm publisher python demo/01_core_pub.py "message-1" "message-2"

# 2. the subscriber dies
docker compose stop core-sub
docker compose run --rm publisher python demo/01_core_pub.py "message-3" "message-4"

# 3. the subscriber comes back
docker compose start core-sub
docker compose run --rm publisher python demo/01_core_pub.py "message-5"

The publisher reports success the whole time:

[pub] sent: message-1
[pub] sent: message-2
[pub] sent: message-3      <- no subscriber
[pub] sent: message-4      <- no subscriber
[pub] sent: message-5

And this is what the subscriber saw:

[sub] listening on demo.core
[sub] received: message-1
[sub] received: message-2
[sub] listening on demo.core      <- restart
[sub] received: message-5

Messages 3 and 4 are nowhere. They are not waiting in a queue, they won’t come back after the restart, there is no entry about them in the server log. Nobody reported an error — not publish(), not the server. From the publisher’s point of view everything worked.

This is at-most-once in a single picture, and this is the moment to pause. If those messages were orders, you have just lost two, and your monitoring has no basis on which to notice anything.

`flush()` does not save anything

It is tempting to think that since publish() doesn’t wait, adding await nc.flush() will do. It won’t. flush() does a round trip to the server (sends PING and waits for PONG), so it guarantees that the server received the bytes. It says nothing about whether anyone consumed those bytes. In the run above flush() was there, and messages 3 and 4 were lost anyway.

A guarantee is not the same as routing

Two things get confused because both sound like “delivery”.

The guarantee says how many times one receiver will get a given message (zero, once, many times). Routing says how many receivers are on the list. These are independent dimensions. Two processes that both call nc.subscribe("demo.core") will each get a copy of the same message — that does not break at-most-once, because each of them got it at most once.

If you want a message to go to exactly one of a group of processes, you add a group name:

sub = await nc.subscribe("demo.core", queue="workers")

The server then picks one member of the group. This is called a queue group and it is the simplest load-balancing mechanism in NATS — no acknowledgements and no retries, so when the chosen process dies mid-work, the task is gone.

	everyone gets a copy	one of the group
at-most-once (Core)	`subscribe(subject)`	`subscribe(subject, queue=...)`
at-least-once (JetStream)	separate consumers	shared `durable` (chapter 4)

The most common cause of message loss is not a failure

The demo above required killing a process. In production, at-most-once most often plays out very differently: it is enough for the receiver to be slower than the sender.

Both the server and the client keep a buffer per subscription. When the buffer fills up, messages are discarded — NATS calls this situation a slow consumer. On the nats-py side the default subscription buffer limit is 524288 messages (DEFAULT_SUB_PENDING_MSGS_LIMIT in nats/aio/subscription.py:37), so you won’t see this on low traffic — which is exactly why it comes as a surprise under load. Let’s set that limit to 10 and make a receiver that takes 50 ms per message:

async def handle(msg) -> None:
    global received
    received += 1
    await asyncio.sleep(0.05)      # pretend work: a database write, an HTTP call

async def on_error(err: Exception) -> None:
    dropped.append(err)

nc = await nats.connect(os.environ["NATS_URL"], error_cb=on_error)
await nc.subscribe("demo.slow", cb=handle, pending_msgs_limit=10)

for i in range(100):
    await nc.publish("demo.slow", f"msg-{i}".encode())
await nc.flush()
await asyncio.sleep(3)             # time to catch updemo/02_slow_sub.py

The result:

published:            100
seen by our code:     10
error events:         90
first event:          nats: slow consumer, messages dropped subject: demo.slow, sid: 1, ...

Out of a hundred messages, the code saw ten. Ninety disappeared — in a single process, with no network failure, no restart, no exception at the point of publishing. The only trace is the calls to error_cb, and error_cb is optional: had we not passed it to nats.connect(), the loss would have been completely silent.

Hence the takeaway of this chapter, more important than the process-killing demo: at-most-once doesn’t need a failure. A slow receiver is enough.

When at-most-once is the right choice

This is not a worse mode, just a different one. It is fast, writes nothing to disk and requires no duplicate handling from you. It fits wherever the next message invalidates the previous one:

metrics and telemetry (the next temperature reading overwrites the previous one anyway),
IoT sensor data,
live view refreshes, “someone is typing…”, presence status,
health checks and heartbeats,
cache invalidation,
request/reply, where the absence of a reply is information in itself.

There is also a borderline case that is easy to fall into: high frequency plus accumulated state. A player’s position in a game looks like perfect at-most-once — as long as you send the absolute position. If you send a delta (“moved 3 to the right”), a single lost message corrupts the state forever.

4. JetStream: a message that waits

JetStream is a subsystem built into the same server (enabled with the --jetstream flag). It adds two things Core doesn’t have: a stream and a consumer.

A stream is storage. You define it once, giving the subject patterns it should capture — from then on every matching message is written down and stays there according to the configured limits (maximum age, number of messages, size). You still publish to a subject, not “to the stream”; the stream only listens and archives.

A consumer is a bookmark in that storage. And here is the thing that surprises people most about NATS: the consumer lives on the server, not in your process. It is the server that remembers how far this consumer has got, which messages it has handed out and not yet received an acknowledgement for, and how many times it has repeated them. Your process merely attaches to that entity.

It is worth sorting out the names, because they get mixed up constantly:

term	where it lives	has state	survives a disconnect
client (connection)	your process	a TCP connection	no
subscription	process + server table	none	no
consumer	server	bookmark, acks, timers	yes
worker	your process	your code	no
producer	—	—	doesn’t exist as an entity

“Producer” and “worker” are roles we invent when describing the architecture. NATS doesn’t know them.

Who creates the stream

A stream is infrastructure, not part of the application: it has limits, a retention policy, replication. In this article I create it with the CLI, and the application only attaches to it:

docker compose exec nats-box nats stream add DEMO \
  --subjects="demo.orders.>" --storage=file --max-age=1h --dupe-window=2m --defaults

The alternative — js.add_stream() at application startup — is convenient in a demo and painful later: with ten pods, each tries to create the same stream, and when you change the configuration, add_stream() on an existing stream with different settings throws an error.

The pattern demo.orders.> means “everything starting with demo.orders.”. A distinction that matters later: subjects given on the stream say what is stored at all; a filter given on the consumer (filter_subject) says what that particular consumer sees of it. The second is a subset of the first.

A publish that acknowledges something

js = nc.jetstream()
ack = await js.publish("demo.orders.new", text.encode())
print(f"[pub] stored: {text}  stream={ack.stream} seq={ack.seq}")demo/03_js_pub.py

[pub] stored: order-1  stream=DEMO seq=1
[pub] stored: order-2  stream=DEMO seq=2

The difference from Core fits in the return type. js.publish() waits for a reply from the server (PubAck) containing the stream name and the message’s sequence number. Only that reply means “stored”. If the stream doesn’t exist or is full, you get an exception — unlike the silence of chapter 3.

The worker

sub = await js.pull_subscribe(
    "demo.orders.new",
    durable="orders",
    stream="DEMO",
    config=ConsumerConfig(ack_policy=AckPolicy.EXPLICIT, ack_wait=30),
)

while True:
    try:
        msgs = await sub.fetch(batch=10, timeout=5)
    except NatsTimeoutError:
        continue                      # no new messages is a normal state

    for msg in msgs:
        print(f"[worker] processing: {msg.data.decode()}", flush=True)
        await msg.ack()               # only after the workdemo/03_js_sub.py

Three things in this code need a comment.

durable="orders" is the name of the consumer on the server. A named consumer survives a process restart together with its bookmark. All processes that give the same name share that one bookmark and the same pool of work — that will be the heart of part 2.

stream="DEMO" is optional, but worth giving. Without it, before subscribing the client asks the server which stream handles the subject (find_stream_name_by_subject in nats/js/client.py:367, a request to $JS.API.STREAM.NAMES in nats/js/manager.py:73) — an extra round trip at every startup, and after a configuration rebuild it may quietly find a different stream.

except NatsTimeoutError is a Python-specific trap. When there are no new messages, fetch() doesn’t return an empty list — it raises an exception, so every fetch has to be in a try. Moreover, with batch=1 you get FetchTimeoutError, and with batch > 1 a plain nats.errors.TimeoutError (the first is a subclass of the second, see nats/js/errors.py:149). Catching only FetchTimeoutError looks correct, passes tests on single messages and crashes the worker on the first empty batch fetch. Catch nats.errors.TimeoutError.

It is also fair to say that nats-py 2.15.0 has no consume() method, which the NATS documentation recommends as the proper way of continuous fetching. In Python you are left with a loop over fetch() — exactly what the documentation advises against.

The same demo, a different result

docker compose stop js-worker
docker compose run --rm publisher python demo/03_js_pub.py "order-3" "order-4"
docker compose exec nats-box nats consumer info DEMO orders

The interesting fields of the consumer state then look like this:

waiting for delivery (num_pending):     2
unacknowledged (num_ack_pending):       0
being redelivered (num_redelivered):    0

The messages sit in the stream and wait. After docker compose start js-worker:

[worker] waiting for messages
[worker] processing: order-3
[worker] processing: order-4

The same sequence of events that in Core NATS meant irrecoverable loss here means a delay of a few seconds.

How the server knows something went wrong

The answer is: it doesn’t. It only has a stopwatch. When a consumer hands out a message, it starts ack_wait for it. If no acknowledgement arrives in that time, it decides the message has to be handed out again. The defaults, checked on server 2.11.17:

ack_wait = 30 s,
max_ack_pending = 1000 (how many messages can be handed out and unacknowledged at once),
max_deliver = -1, meaning no limit on retries.

The fact that the arbiter is a stopwatch, not knowledge of who is alive, is the source of the entire cost described in the next chapter.

5. The cost: duplicates

Experiment: a worker that dies halfway through

The script fetches one message, prints it (that is our “work”) and kills the process without acknowledging. The consumer has ack_wait=5, so we don’t have to wait half a minute:

msgs = await sub.fetch(batch=1, timeout=5)
msg = msgs[0]
meta = msg.metadata
print(f"[crash] processing: {msg.data.decode()}  "
      f"seq={meta.sequence.stream} delivery #{meta.num_delivered}", flush=True)
os._exit(1)      # hard death: no ackdemo/04_crash_worker.py

Run three times in a row, it produces a timeline that says more than a paragraph of explanation:

### T+0  first run
[crash] processing: order-1  seq=1 delivery #1
### T+1  right after the death (ack_wait=5 still running)
[crash] processing: order-2  seq=2 delivery #1
### T+8  after ack_wait has elapsed
[crash] processing: order-1  seq=1 delivery #2

Let’s read it line by line.

T+0. The worker gets message number 1 and dies. The server knows nothing about it — as far as it is concerned, the message is “handed out, awaiting acknowledgement”.

T+1. A new worker asks for work and gets message number 2, not number 1. Number 1 is still locked by the stopwatch. In other words: the failure didn’t stop the queue, it just set one item aside.

T+8. The stopwatch has run out, so the server hands out message number 1 again — with the counter at delivery #2. And here there are two separate consequences, both important:

Message number 1 was processed twice. If “processing” means sending an email, the customer got two emails. This is at-least-once in practice and it cannot be turned off — the repeat is precisely what makes this guarantee a guarantee.
Ordering stopped holding. Message 2 was handled before message 1. There is no configuration mistake here: every redelivery by definition moves a message later. Remember that sentence, because part 2 is about it.

Plus a detail in the counters that can mislead: after this run, nats consumer info shows num_delivered=2 for the message, but num_redelivered=1 on the consumer — that second counter says how many messages are currently in a redelivered state, not how many redeliveries there have been in total.

The server cannot tell slow from dead

The most important sentence about ack_wait: it is a stopwatch, not liveness detection. The server has no idea whether your worker died or is simply grinding through a batch longer than you assumed. The consequence is unpleasant: if processing takes longer than ack_wait, the server hands the same message to a second worker while the first is still working on it. For a while two processes are doing the same task.

Remedies, from simplest:

set ack_wait well above the real processing time, with a margin for the worst case, not the average;
in long tasks call msg.in_progress() (nats/aio/msg.py:139, sends +WPI), which resets the stopwatch — it can be called repeatedly;
treat idempotency as the safety net, because the first two points are heuristics.

You also have two explicit negative answers: msg.nak(delay=None) (msg.py:124) says “I can’t do this, give it to someone else, optionally after delay”, and msg.term() (msg.py:148) says “this message cannot be processed, don’t try again” — without waiting for max_deliver to run out.

`ack()` does not wait for the server

Something that looks like an implementation detail and turns out to be a source of duplicates. In nats-py 2.15.0:

msg.ack() (nats/aio/msg.py:107) is await self._client.publish(self.reply) — an ordinary publish, waiting for nothing;
msg.ack_sync(timeout=1.0) (msg.py:115) is await self._client.request(self.reply, timeout=timeout) — it waits for the server’s reply and raises on timeout.

The practical conclusion: returning from await msg.ack() does not mean the server has recorded the acknowledgement. A process that called ack() and died right after may never have delivered that acknowledgement — in which case the message comes back despite being “acknowledged”. ack_sync() closes that gap at the cost of one round trip per message, and that is an honest trade-off to make consciously, not an obvious choice.

Order of operations in the worker

One rule from which the rest follows: process, commit to the database, only then acknowledge.

for msg in msgs:
    await save_to_database(msg)   # work + commit
    await msg.ack()               # only now

The reverse order (ack() before the work) turns at-least-once into at-most-once, only hidden and harder to diagnose than in Core NATS.

Living with duplicates

Duplicates cannot be prevented; they can be made harmless. Starting with the best solutions:

1. Idempotent writes. If the operation can be written so that repeating it changes nothing, the problem disappears with no extra infrastructure:

INSERT INTO orders (id, amount) VALUES (7777, 100)
ON CONFLICT (id) DO NOTHING;

2. A deduplication table in the same transaction as the business effect. For operations that cannot be made idempotent — sending an email, charging a card. You record the identifier of the handled message and the effect in one transaction; a repeat sees the record and finishes without acting. The “same transaction” condition matters: two separate writes create a window in which the process can die between them.

3. Publisher-side deduplication. A stream with a deduplication window configured will reject a second message with the same Nats-Msg-Id header:

for attempt in (1, 2):
    ack = await js.publish(
        "demo.orders.new",
        b"order 7777, amount 100",
        headers={"Nats-Msg-Id": "order-7777"},
    )
    print(f"publish #{attempt}: seq={ack.seq} duplicate={ack.duplicate}")demo/05_dedup_pub.py

stream deduplication window: 120.0 s
messages in stream before publishing: 4
publish #1: seq=5 duplicate=None
publish #2: seq=5 duplicate=True
messages in stream after two publishes: 5

The second publish got the same sequence number and the flag duplicate=True, and the stream grew by one message, not two. The default window is 2 minutes (checked on 2.11.17) and it is a time window — once it passes, the same identifier will be accepted as a new message. A nats-py detail: on the first publish the duplicate field is None, not False, so compare with if ack.duplicate:, not is False.

And now the most important caveat of the whole chapter: publisher deduplication has nothing to do with redelivery. It protects against the case where the publisher itself sent the same thing twice (because it didn’t get a PubAck and retried). It does not protect against the run from the experiment above, where the stream held one message and the worker received it twice. These are two independent sources of duplicates, and one cure does not work on the other.

“Exactly-once” in quotes

After all this it is clear what exactly-once is in practice: not a separate operating mode of the broker, but at-least-once plus deduplication at both ends — a message identifier on the publisher side and an idempotent write on the receiver side. The broker cannot guarantee it on its own, because between “I saved the effect” and “I acknowledged the receipt” there is always a gap in which the process can die. You can move that gap and narrow it; you cannot remove it.

The practical conclusion: don’t look for a configuration that gives you exactly-once. Look for a way to make repeating an operation boring.

What NATS doesn’t have: a dead letter queue

If you set max_deliver to a finite value, then once the attempts are used up the server stops delivering the message — and that’s it. The message stays in the stream until the limits expire, the consumer forgets about it, nobody gets notified. The concept of a “dead letter queue” known from other brokers simply doesn’t exist in NATS; you have to build it: listen to the server’s advisory events about the delivery limit being exceeded and move such messages to a stream of your own. ⚠️ The exact subject of that event remains to be verified before publication.

6. Decision table

The question that settles the choice is: if this message disappears and nobody finds out — did something bad happen?

	Core NATS	JetStream
guarantee	at-most-once	at-least-once
message with no receiver	vanishes immediately	waits in the stream
retries	none	after `ack_wait`, up to `max_deliver`
state	none	on the server (consumer)
the cost	silent absence of a message	duplicates and mandatory idempotency
resource cost	memory, no writes	disk, limits, configuration
what you have to write	nothing extra	acknowledgements, idempotency, retry handling

Choose at-most-once when the next message invalidates the previous one: metrics, telemetry, IoT, live dashboards, presence status, cache invalidation, request/reply.

Choose at-least-once when every message carries unique information that cannot be reconstructed: orders, payments, emails, task queues, event sourcing, synchronisation between services, webhooks, multi-step processes.

If you are unsure about a specific subject, check whether the message describes a state (“the temperature is 21°C”) or an event (“the customer paid”). States can be reconstructed from the next reading. Events cannot.

7. What’s next

We leave part 1 with a working worker that doesn’t lose orders and copes with duplicates. In production the next question comes about a week later: one worker stops keeping up. So you start a second one — and discover that this is a separate problem, not a continuation of the same one.

Both threads of part 2 are already visible in the experiment from chapter 5. First, when the worker died, the second one got the next message — meaning processing order stops being guaranteed the moment there is more than one worker, or even a single redelivery. Second, at any given moment the server had several messages “handed out and unacknowledged” — and how many there can be is governed by max_ack_pending, of which few people know that it is shared by all processes attached to the same consumer, not counted per process.

Part 2 will answer three questions:

What consumer concurrency really is and how to tell it apart from broadcasting (and, along the way, how the queue group from chapter 3 differs from a shared named JetStream consumer, even though both are called “load balancing”).
The two confused dimensions of parallelism in Python: N processes with the same consumer name versus N asyncio tasks in one process — and why the latter won’t help with CPU-bound work.
What ordering really costs: throughput measured at max_ack_pending of 1, 10, 100 and 1000, and what to do when ordering is required only within one customer, not globally.

Finally, the one thing to carry from part 1 into part 2: delivery guarantees and concurrency are separate axes. You can have many parallel workers on Core NATS with no acknowledgements at all, and you can have a single worker on JetStream with the full guarantee. The fact that both topics meet in the configuration of the same consumer doesn’t make them the same problem — and most texts about JetStream blend them into one checklist.