Building Resilient Message Queues: Notes on At-Least-Once Delivery

The Guarantee Trap

Every engineer eventually encounters the three classic fallacies of message brokers:

1. "Messages will always arrive in order." 2. "Messages will be delivered exactly once without coordinated two-phase commit." 3. "The disk write is persisted as soon as the OS returns from write()."

The Reality of Fsync and Disk Controllers

When an application calls write() on a POSIX socket or file descriptor, the operating system kernel copies the bytes into page cache memory. The return code is immediate. But if power fails a millisecond later, the data evaporates into the ether.

To truly guarantee durability, an append-only log must issue an explicit fsync(fd) or fdatasync(fd). And even then, drive manufacturers frequently lie about physical platter flush completion unless ordered by a battery-backed write cache.

``c // The fundamental persistence barrier int append_entry(int log_fd, const void *record, size_t len) { ssize_t written = write(log_fd, record, len); if (written != (ssize_t)len) { return -1; // Partial write error } // Force disk controller to flush internal volatile DRAM cache if (fdatasync(log_fd) != 0) { return -2; // Persistence failure } return 0; } ``

Designing for failure from day one is what separates hobbyist scripts from mission-critical infrastructure.