"Sent" is not the same as "delivered". Delivery guarantees in NATS

Part 1 of a series. Part 2 is about what happens when one worker stops keeping up.

All the code, with the docker compose lab, is on GitHub: github.com/samadhi73/nats-python-examples.

Versions used in this text: nats-server 2.11.17, nats-py 2.15.0, nats CLI 0.4.0. Every console output in this article comes from running the included code, not from the documentation. Verified 2026-09-21.

Open Table of contents

1. One line we don’t understand
- The three levels, briefly
2. The lab: four containers
3. Core NATS: a message for whoever happens to be listening
4. JetStream: a message that waits
5. The cost: duplicates
6. Decision table
7. What’s next

1. One line we don’t understand

Say you are writing a web shop. A customer places an order, and your application has to send them a confirmation email. You don’t want to send that email in the same place where you handle the HTTP request, because the mail server can be slow and the customer shouldn’t have to wait. So you split it in two: the application announces the fact “order number 7777 was created”, and a separate process listens for such announcements and sends the emails.

Passing announcements like that around is what a message broker is for. NATS is one of them: a single, very fast server that processes connect to, some publishing messages, others receiving them. In Python it looks like this:

await nc.publish("orders.new", b"order 7777")

And this is where the problem this article is about begins. That line has an await, so it looks like something we are waiting for. The question is: what exactly are we waiting for? Once it returns, has the message reached the server? Has it reached the email process? Has the email been sent? And if the email process happened to be restarting at that moment, what happened to the message?

The answer in Core NATS is: we are waiting for practically nothing, and the message may have vanished without a trace. This is not a bug or a misconfiguration. It is a deliberate design decision, called a delivery guarantee, and it has a flip side: a mode where the message will definitely arrive, but may arrive more than once.

The three levels, briefly

Before we get into NATS, it is worth naming things. A delivery guarantee describes how many times a single message may be delivered to a receiver:

at-most-once — the message is delivered zero or one time. Never twice, but sometimes not at all. No acknowledgements, no retries, no memory.
at-least-once — the message is delivered at least once, meaning once or more. It will never be lost, but you may get it twice and you have to be prepared for that.
exactly-once — exactly one delivery. I put this in quotes and will come back to it in chapter 5, because in a distributed system the phrase is at best a shorthand.

Core NATS gives you at-most-once. JetStream — the NATS subsystem with persistent storage — gives you at-least-once. This whole article is an answer to the question of when you need the second one and what it costs, because it costs something real and hardly anyone writes about it.

I assume you know Python and roughly understand async/await. I assume no knowledge of NATS, message brokers or event-driven architectures.

2. The lab: four containers

Every example in this text is runnable and everything runs on a single docker compose. This is deliberate: to see how messages get lost, you need to be able to kill a process at a specific moment, and a Docker service can be stopped and resumed with one command.

services:
  nats:
    image: nats:2.11-alpine
    command: ["--jetstream", "--store_dir=/data", "--http_port=8222", "--name=lab"]
    ports: ["4222:4222", "8222:8222"]
    volumes: [nats-data:/data]

  nats-box:          # the `nats` CLI, for inspecting the server
    image: natsio/nats-box:latest
    environment: { NATS_URL: "nats://nats:4222" }
    command: ["sleep", "infinity"]

  publisher:         # a Python container we run scripts from
    build: .
    environment: { NATS_URL: "nats://nats:4222" }
    volumes: [".:/app"]

  core-sub:          # the subscriber we are going to kill
    build: .
    environment: { NATS_URL: "nats://nats:4222" }
    volumes: [".:/app"]
    command: python demo/01_core_sub.py

The --jetstream flag enables persistent storage; without it the server can only pass messages along. --http_port=8222 exposes a monitoring endpoint at http://localhost:8222/jsz, which shows the state of streams and consumers as JSON.

To start:

docker compose up -d
docker compose exec publisher python demo/01_core_pub.py "test"

3. Core NATS: a message for whoever happens to be listening

Core NATS has no storage. The server keeps a table in memory: who is connected and what they are listening for. When a message comes in, the server copies it to the sockets of the subscribers whose subject matches — and immediately forgets about it. It does not write it anywhere, does not wait for an acknowledgement, has no way to repeat it.

A subject is a plain dotted string, such as orders.new or demo.core. A subscriber gives the same string (or a pattern using * or >) and receives matching messages. That is all the addressing there is in NATS — there is no such thing as a pre-created queue; a subject exists because someone publishes on it.

The subscriber:

async def main() -> None:
    nc = await nats.connect(os.environ["NATS_URL"])
    sub = await nc.subscribe("demo.core")
    print("[sub] listening on demo.core", flush=True)

    async for msg in sub.messages:
        print(f"[sub] received: {msg.data.decode()}", flush=True)demo/01_core_sub.py

The publisher:

async def main() -> None:
    nc = await nats.connect(os.environ["NATS_URL"])
    for text in sys.argv[1:]:
        await nc.publish("demo.core", text.encode())
        print(f"[pub] sent: {text}", flush=True)
    await nc.flush()
    await nc.drain()demo/01_core_pub.py

Notice that the subscriber says nothing about acknowledging. There is nothing to acknowledge.

Experiment: kill the subscriber

docker compose up -d core-sub

# 1. the subscriber is alive
docker compose run --rm publisher python demo/01_core_pub.py "message-1" "message-2"

# 2. the subscriber dies
docker compose stop core-sub
docker compose run --rm publisher python demo/01_core_pub.py "message-3" "message-4"

# 3. the subscriber comes back
docker compose start core-sub
docker compose run --rm publisher python demo/01_core_pub.py "message-5"

The publisher reports success the whole time:

[pub] sent: message-1
[pub] sent: message-2
[pub] sent: message-3      <- no subscriber
[pub] sent: message-4      <- no subscriber
[pub] sent: message-5

And this is what the subscriber saw:

[sub] listening on demo.core
[sub] received: message-1
[sub] received: message-2
[sub] listening on demo.core      <- restart
[sub] received: message-5

Messages 3 and 4 are nowhere. They are not waiting in a queue, they won’t come back after the restart, there is no entry about them in the server log. Nobody reported an error — not publish(), not the server. From the publisher’s point of view everything worked.

This is at-most-once in a single picture, and this is the moment to pause. If those messages were orders, you have just lost two, and your monitoring has no basis on which to notice anything.

`flush()` does not save anything

It is tempting to think that since publish() doesn’t wait, adding await nc.flush() will do. It won’t. flush() does a round trip to the server (sends PING and waits for PONG), so it guarantees that the server received the bytes. It says nothing about whether anyone consumed those bytes. In the run above flush() was there, and messages 3 and 4 were lost anyway.

A guarantee is not the same as routing

Two things get confused because both sound like “delivery”.

The guarantee says how many times one receiver will get a given message (zero, once, many times). Routing says how many receivers are on the list. These are independent dimensions. Two processes that both call nc.subscribe("demo.core") will each get a copy of the same message — that does not break at-most-once, because each of them got it at most once.

If you want a message to go to exactly one of a group of processes, you add a group name:

sub = await nc.subscribe("demo.core", queue="workers")

The server then picks one member of the group. This is called a queue group and it is the simplest load-balancing mechanism in NATS — no acknowledgements and no retries, so when the chosen process dies mid-work, the task is gone.

	everyone gets a copy	one of the group
at-most-once (Core)	`subscribe(subject)`	`subscribe(subject, queue=...)`
at-least-once (JetStream)	separate consumers	shared `durable` (chapter 4)

The most common cause of message loss is not a failure

The demo above required killing a process. In production, at-most-once most often plays out very differently: it is enough for the receiver to be slower than the sender.

Both the server and the client keep a buffer for each subscription, a waiting room where messages sit until your code gets to them. When the waiting room is full, new messages are thrown away. NATS calls this situation a slow consumer. The client hands messages to your handler one at a time, and the next one waits until the handler has finished the previous one, so a slow handler is all it takes to fill the waiting room. In nats-py it holds 524,288 messages by default, so on low traffic you won’t see this at all, which is exactly why it comes as a surprise under load. The size is set with the pending_msgs_limit argument to subscribe(). Let’s set it to 10 and make a receiver that takes 50 ms per message:

async def handle(msg) -> None:
    global received
    received += 1
    await asyncio.sleep(0.05)      # pretend work: a database write, an HTTP call

async def on_error(err: Exception) -> None:
    dropped.append(err)

nc = await nats.connect(os.environ["NATS_URL"], error_cb=on_error)
await nc.subscribe("demo.slow", cb=handle, pending_msgs_limit=10)

for i in range(100):
    await nc.publish("demo.slow", f"msg-{i}".encode())
await nc.flush()
await asyncio.sleep(3)             # time to catch updemo/02_slow_sub.py

The result:

published:            100
seen by our code:     10
error events:         90
first event:          nats: slow consumer, messages dropped subject: demo.slow, sid: 1, ...

Out of a hundred messages, the code saw ten. Ninety disappeared — in a single process, with no network failure, no restart, no exception at the point of publishing. The only trace is the calls to error_cb, and error_cb is optional: had we not passed it to nats.connect(), the loss would have been completely silent.

Hence the takeaway of this chapter, more important than the process-killing demo: at-most-once doesn’t need a failure. A slow receiver is enough.

When at-most-once is the right choice

This is not a worse mode, just a different one. It is fast, writes nothing to disk and requires no duplicate handling from you. It fits wherever the next message invalidates the previous one:

metrics and telemetry (the next temperature reading overwrites the previous one anyway),
IoT sensor data,
live view refreshes, “someone is typing…”, presence status,
health checks and heartbeats,
cache invalidation,
request/reply, where the absence of a reply is information in itself.

There is also a borderline case that is easy to fall into: high frequency plus accumulated state. A player’s position in a game looks like perfect at-most-once — as long as you send the absolute position. If you send a delta (“moved 3 to the right”), a single lost message corrupts the state forever.

4. JetStream: a message that waits

JetStream is a subsystem built into the same server (enabled with the --jetstream flag). It adds two things Core doesn’t have: a stream and a consumer.

A stream is storage. You define it once, giving the subject patterns it should capture — from then on every matching message is written down and stays there according to the configured limits (maximum age, number of messages, size). You still publish to a subject, not “to the stream”; the stream only listens and archives.

A consumer is a bookmark in that storage. And here is the thing that surprises people most about NATS: the consumer lives on the server, not in your process. It is the server that remembers how far this consumer has got, which messages it has handed out and not yet received an acknowledgement for, and how many times it has repeated them. Your process merely attaches to that entity.

It is worth sorting out the names, because they get mixed up constantly:

term	where it lives	has state	survives a disconnect
client (connection)	your process	a TCP connection	no
subscription	process + server table	none	no
consumer	server	bookmark, acks, timers	yes
worker	your process	your code	no
producer	—	—	doesn’t exist as an entity

“Producer” and “worker” are roles we invent when describing the architecture. NATS doesn’t know them.

Who creates the stream

A stream is infrastructure, not part of the application: it has limits, a retention policy, replication. In this article I create it with the CLI, and the application only attaches to it:

docker compose exec nats-box nats stream add DEMO \
  --subjects="demo.orders.>" --storage=file --max-age=1h --dupe-window=2m --defaults

The alternative — js.add_stream() at application startup — is convenient in a demo and painful later: with ten pods, each tries to create the same stream, and when you change the configuration, add_stream() on an existing stream with different settings throws an error.

The pattern demo.orders.> means “everything starting with demo.orders.”. A distinction that matters later: subjects given on the stream say what is stored at all; a filter given on the consumer (filter_subject) says what that particular consumer sees of it. The second is a subset of the first.

A publish that acknowledges something

js = nc.jetstream()
ack = await js.publish("demo.orders.new", text.encode())
print(f"[pub] stored: {text}  stream={ack.stream} seq={ack.seq}")demo/03_js_pub.py

[pub] stored: order-1  stream=DEMO seq=1
[pub] stored: order-2  stream=DEMO seq=2

The difference from Core fits in the return type. js.publish() waits for a reply from the server (PubAck) containing the stream name and the message’s sequence number. Only that reply means “stored”. If the stream doesn’t exist or is full, you get an exception — unlike the silence of chapter 3.

The worker

sub = await js.pull_subscribe(
    "demo.orders.new",
    durable="orders",
    stream="DEMO",
    config=ConsumerConfig(ack_policy=AckPolicy.EXPLICIT, ack_wait=30),
)

while True:
    try:
        msgs = await sub.fetch(batch=10, timeout=5)
    except NatsTimeoutError:
        continue                      # no new messages is a normal state

    for msg in msgs:
        print(f"[worker] processing: {msg.data.decode()}", flush=True)
        await msg.ack()               # only after the workdemo/03_js_sub.py

Three things in this code need a comment.

durable="orders" is the name of the consumer on the server. A named consumer survives a process restart together with its bookmark. All processes that give the same name share that one bookmark and the same pool of work — that will be the heart of part 2.

stream="DEMO" is optional, but worth giving. It tells the client straight away which stream to read from. If you leave it out, the client first has to ask the server which stream stores messages on demo.orders.new, and only after the answer does it create the subscription. That means one extra question to the server every time the worker starts. The bigger problem is that the worker then reads from whatever stream the server names at that moment. If someone later moves the subject to a different stream, the worker quietly follows it. With the name written in the code, it fails with a clear error that the DEMO stream is missing.

except NatsTimeoutError needs a word of explanation, because this is where Python differs from what the NATS documentation leads you to expect. The documentation says an empty fetch is normal, not an error. In nats-py, when there are no new messages, fetch() doesn’t return an empty list. It raises an exception instead. That’s why every call to fetch() sits inside try, and “nothing new” ends up in the except branch. NatsTimeoutError is nats.errors.TimeoutError, imported under a different name so it isn’t confused with Python’s built-in TimeoutError.

The library also has a narrower FetchTimeoutError, and catching it is tempting because the name fits so well. But it only appears when the fetch uses a heartbeat. fetch(batch=10, timeout=5, heartbeat=1) asks the server to send a short, empty signal every second while it waits, meaning “I still have your request, there’s just nothing to give you yet”. Only after at least one such signal does the client know for sure that the stream was empty, and only then does it raise FetchTimeoutError. Without a heartbeat, which is the default and what our worker does, silence could just as well mean a lost request or a server that stopped responding. So the client raises the general nats.errors.TimeoutError, except FetchTimeoutError doesn’t catch it, and the worker crashes the first time the stream runs dry. Catch the general exception, because it covers both cases.

It is also fair to say that nats-py 2.15.0 has no consume() method, which the NATS documentation recommends as the proper way of continuous fetching. In Python you are left with a loop over fetch() — exactly what the documentation advises against.

The same demo, a different result

docker compose stop js-worker
docker compose run --rm publisher python demo/03_js_pub.py "order-3" "order-4"
docker compose exec nats-box nats consumer info DEMO orders

The interesting fields of the consumer state then look like this:

waiting for delivery (num_pending):     2
unacknowledged (num_ack_pending):       0
being redelivered (num_redelivered):    0

The messages sit in the stream and wait. After docker compose start js-worker:

[worker] waiting for messages
[worker] processing: order-3
[worker] processing: order-4

The same sequence of events that in Core NATS meant irrecoverable loss here means a delay of a few seconds.

How the server knows something went wrong

The answer is: it doesn’t. It only has a stopwatch. When a consumer hands out a message, it starts ack_wait for it. If no acknowledgement arrives in that time, it decides the message has to be handed out again. The defaults, checked on server 2.11.17:

ack_wait = 30 s,
max_ack_pending = 1000 (how many messages can be handed out and unacknowledged at once),
max_deliver = -1, meaning no limit on retries.

The fact that the arbiter is a stopwatch, not knowledge of who is alive, is the source of the entire cost described in the next chapter.

5. The cost: duplicates

Experiment: a worker that dies halfway through

The script fetches one message, prints it (that is our “work”) and kills the process without acknowledging. The consumer has ack_wait=5, so we don’t have to wait half a minute:

msgs = await sub.fetch(batch=1, timeout=5)
msg = msgs[0]
meta = msg.metadata
print(f"[crash] processing: {msg.data.decode()}  "
      f"seq={meta.sequence.stream} delivery #{meta.num_delivered}", flush=True)
os._exit(1)      # hard death: no ackdemo/04_crash_worker.py

Run three times in a row, it produces a timeline that says more than a paragraph of explanation:

### T+0  first run
[crash] processing: order-1  seq=1 delivery #1
### T+1  right after the death (ack_wait=5 still running)
[crash] processing: order-2  seq=2 delivery #1
### T+8  after ack_wait has elapsed
[crash] processing: order-1  seq=1 delivery #2

Let’s read it line by line.

T+0. The worker gets message number 1 and dies. The server knows nothing about it — as far as it is concerned, the message is “handed out, awaiting acknowledgement”.

T+1. A new worker asks for work and gets message number 2, not number 1. Number 1 is still locked by the stopwatch. In other words: the failure didn’t stop the queue, it just set one item aside.

T+8. The stopwatch has run out, so the server hands out message number 1 again — with the counter at delivery #2. And here there are two separate consequences, both important:

Message number 1 was processed twice. If “processing” means sending an email, the customer got two emails. This is at-least-once in practice and it cannot be turned off — the repeat is precisely what makes this guarantee a guarantee.
Ordering stopped holding. Message 2 was handled before message 1. There is no configuration mistake here: every redelivery by definition moves a message later. Remember that sentence, because part 2 is about it.

Plus a detail in the counters that can mislead: after this run, nats consumer info shows num_delivered=2 for the message, but num_redelivered=1 on the consumer — that second counter says how many messages are currently in a redelivered state, not how many redeliveries there have been in total.

The server cannot tell slow from dead

The most important sentence about ack_wait: it is a stopwatch, not liveness detection. The server has no idea whether your worker died or is simply grinding through a batch longer than you assumed. The consequence is unpleasant: if processing takes longer than ack_wait, the server hands the same message to a second worker while the first is still working on it. For a while two processes are doing the same task.

Remedies, from simplest:

set ack_wait well above the real processing time, with a margin for the worst case, not the average;
in long tasks, call msg.in_progress() from time to time. It tells the server “I’m still working on this one”, and the server starts counting ack_wait from zero again. You can call it as many times as you need, so a task can run far longer than ack_wait, as long as the worker reports in before each countdown runs out. The catch is that your code has to get the chance to call it. If the long part is a single await that doesn’t return for minutes, you need a separate asyncio task that calls in_progress() on a timer;
treat idempotency as the safety net, because the first two points are heuristics.

You also have two explicit negative answers. msg.nak() says “I can’t handle this right now, try again”. The server makes the message available again straight away, and it may land back on the same worker a moment later. So after a failure that won’t fix itself within a second, such as a database being down, use msg.nak(delay=10), which holds the message back for the given number of seconds. msg.term() says “this message cannot be processed, don’t try again”. The server stops redelivering it at once, without waiting for max_deliver to run out.

`ack()` does not wait for the server

Picture a worker that saves an order to the database and then calls await msg.ack(). The call returns, the code moves on, and a split second later a deployment kills the process. Did the server get the acknowledgement? You can’t tell. msg.ack() hands the acknowledgement over to be sent and returns straight away, without waiting for the server to say it got it. If the process dies or the connection drops before the acknowledgement actually leaves, the server never hears about it. Once ack_wait runs out, it gives the order to another worker, and the order is saved a second time, even though your code “acknowledged” it.

msg.ack_sync() closes that gap. It sends the same acknowledgement, but waits for the server to confirm it has recorded it, and raises an exception if no confirmation arrives in time (one second by default). When it returns without an error, the acknowledgement is safely on the server. The price is waiting for that reply on every single message, which adds up at thousands of messages per second. So this is a trade-off to make consciously, not an obvious choice.

Neither method closes the gap completely. If the process dies after saving to the database but before the acknowledgement is sent, the message comes back whichever method you use. That’s why the real safety net is making duplicates harmless, which is what “Living with duplicates” below is about.

Order of operations in the worker

One rule from which the rest follows: process, commit to the database, only then acknowledge.

for msg in msgs:
    await save_to_database(msg)   # work + commit
    await msg.ack()               # only now

The reverse order (ack() before the work) turns at-least-once into at-most-once, only hidden and harder to diagnose than in Core NATS.

Living with duplicates

Duplicates cannot be prevented; they can be made harmless. Starting with the best solutions:

1. Idempotent writes. If the operation can be written so that repeating it changes nothing, the problem disappears with no extra infrastructure:

INSERT INTO orders (id, amount) VALUES (7777, 100)
ON CONFLICT (id) DO NOTHING;

2. A deduplication table in the same transaction as the business effect. Some writes can’t be made idempotent. Adding 100 to an account balance adds it again on every repeat. For those, keep a table of messages you have already handled, and in one transaction both record the message’s identifier there and apply the effect:

BEGIN;
INSERT INTO processed_messages (msg_id) VALUES ('DEMO:4');  -- fails if already there
UPDATE accounts SET balance = balance + 100 WHERE id = 42;
COMMIT;

If the message comes back, the first insert fails because the identifier is already in the table, the transaction is rolled back, and the balance stays untouched. All that’s left is to acknowledge the message. A good identifier is the message’s position in the stream (msg.metadata.sequence.stream), which stays the same on every redelivery, combined with the stream name, because the numbers are only unique within one stream. Don’t use the consumer sequence number sitting right next to it, because that one changes with each delivery. Both writes have to be in one transaction. With two separate writes, the process can die between them, and you are back where you started.

This only works for effects that live in the same database. Sending an email or charging a card happens outside it, and rolling back a transaction won’t unsend an email. For payments, pass the message identifier to the payment provider as an idempotency key, if it supports one, so the provider can reject the repeat. For emails, you usually accept that a rare duplicate will get through.

3. Publisher-side deduplication. Duplicates can also appear before any worker is involved. The publisher sends a message and waits for the server’s PubAck. If that reply doesn’t arrive, say because of a brief network hiccup, the publisher can’t tell whether the message never reached the server, or whether it was stored and only the reply got lost. The sensible thing is to send it again, and in the second case the stream now holds two copies of the same order.

To prevent this, the publisher attaches an identifier to the message in the Nats-Msg-Id header. The server remembers the identifiers it has seen recently. When one comes in again, it doesn’t store the message a second time. Instead it replies as if the publish had succeeded, pointing to the message it already has and marking the reply as a duplicate:

for attempt in (1, 2):
    ack = await js.publish(
        "demo.orders.new",
        b"order 7777, amount 100",
        headers={"Nats-Msg-Id": "order-7777"},
    )
    print(f"publish #{attempt}: seq={ack.seq} duplicate={ack.duplicate}")demo/05_dedup_pub.py

stream deduplication window: 120.0 s
messages in stream before publishing: 4
publish #1: seq=5 duplicate=None
publish #2: seq=5 duplicate=True
messages in stream after two publishes: 5

The second publish got the same sequence number as the first, 5, and the stream grew by one message, not two. For the publisher this is not an error. It gets a normal confirmation and can move on.

Three details decide whether this works in practice. First, the identifier has to come from the content, such as the order number, and stay the same on every attempt. If you generate a random identifier, for example with uuid4(), inside the retry loop, every attempt gets a different one, the server has nothing to match, and the protection silently does nothing. Second, the server only remembers identifiers for a limited time, two minutes by default (checked on 2.11.17). A retry that comes later than that is stored as a new message. Third, in nats-py the duplicate field on the first publish is None, not False, so ack.duplicate is False never matches. Check it with if ack.duplicate: or if not ack.duplicate:.

And now the most important caveat of the whole chapter: publisher deduplication does nothing about redelivery. It makes sure the stream doesn’t hold two copies of the same message. It has no say in how many times the server hands that one copy to workers. The experiment earlier in this chapter, where the worker died halfway through, is exactly that second case: one message in the stream, delivered twice. So you need both protections, one at each end: an identifier on the publisher side, and processing on the worker side that survives a repeat, as in points 1 and 2.

“Exactly-once” in quotes

After all this it is clear what exactly-once is in practice: not a separate operating mode of the broker, but at-least-once plus deduplication at both ends — a message identifier on the publisher side and an idempotent write on the receiver side. The broker cannot guarantee it on its own, because between “I saved the effect” and “I acknowledged the receipt” there is always a gap in which the process can die. You can move that gap and narrow it; you cannot remove it.

The practical conclusion: don’t look for a configuration that gives you exactly-once. Look for a way to make repeating an operation boring.

What NATS doesn’t have: a dead letter queue

If you set max_deliver to a finite value, then once the attempts are used up the server stops delivering the message — and that’s it. The message stays in the stream until the limits expire, the consumer forgets about it, nobody gets notified. The concept of a “dead letter queue” known from other brokers simply doesn’t exist in NATS; you have to build it: listen to the server’s advisory events about the delivery limit being exceeded and move such messages to a stream of your own. ⚠️ The exact subject of that event remains to be verified before publication.

6. Decision table

The question that settles the choice is: if this message disappears and nobody finds out — did something bad happen?

	Core NATS	JetStream
guarantee	at-most-once	at-least-once
message with no receiver	vanishes immediately	waits in the stream
retries	none	after `ack_wait`, up to `max_deliver`
state	none	on the server (consumer)
the cost	silent absence of a message	duplicates and mandatory idempotency
resource cost	memory, no writes	disk, limits, configuration
what you have to write	nothing extra	acknowledgements, idempotency, retry handling

Choose at-most-once when the next message invalidates the previous one: metrics, telemetry, IoT, live dashboards, presence status, cache invalidation, request/reply.

Choose at-least-once when every message carries unique information that cannot be reconstructed: orders, payments, emails, task queues, event sourcing, synchronisation between services, webhooks, multi-step processes.

If you are unsure about a specific subject, check whether the message describes a state (“the temperature is 21°C”) or an event (“the customer paid”). States can be reconstructed from the next reading. Events cannot.

7. What’s next

We leave part 1 with a working worker that doesn’t lose orders and copes with duplicates. The next question comes once traffic grows: one worker stops keeping up. So you start a second one — and discover that this is a separate problem, not a continuation of the same one.

Both threads of part 2 are already visible in the experiment from chapter 5. First, when the worker died, the second one got the next message — meaning processing order stops being guaranteed the moment there is more than one worker, or even a single redelivery. Second, at any given moment the server had several messages “handed out and unacknowledged” — and how many there can be is governed by max_ack_pending, of which few people know that it is shared by all processes attached to the same consumer, not counted per process.

Part 2 answers three questions:

What consumer concurrency really is and how to tell it apart from broadcasting (and, along the way, how the queue group from chapter 3 differs from a shared named JetStream consumer, even though both are called “load balancing”).
The two confused dimensions of parallelism in Python: N processes with the same consumer name versus N asyncio tasks in one process — and why the latter won’t help with CPU-bound work.
What ordering really costs: throughput measured at max_ack_pending of 1, 10, 100 and 1000, and what to do when ordering is required only within one customer, not globally.

Finally, the one thing to carry from part 1 into part 2: delivery guarantees and concurrency are separate axes. You can have many parallel workers on Core NATS with no acknowledgements at all, and you can have a single worker on JetStream with the full guarantee. The fact that both topics meet in the configuration of the same consumer doesn’t make them the same problem — and most texts about JetStream blend them into one checklist.

Read on: One worker is not keeping up. Consumer concurrency in NATS.