Mastering Log Writing in Multi-Process Environments: Mechanisms to Prevent Data Corruption and Practical Approaches

Logging design in concurrent or multi-process environments may seem simple at first glance, but it is actually a domain that demands a precise understanding of the OS level. When scaling web applications or running background workers in parallel, the act of “directly writing to a single log file from multiple processes” harbors pitfalls of unexpected data corruption, such as interleaving or loss.

In this article, we will demystify the OS- and language-level mechanisms behind this issue and present robust logging design methodologies that do not hinder root cause analysis during system failures. A correct understanding of these concepts serves as a touchstone for building highly reliable systems that remain unshakable even under high loads.

Why Reconsider This Theme Now?

In recent years, concurrent processing utilizing container technologies (Docker/Kubernetes), serverless architectures, Node.js’s Cluster module leveraging multi-cores, and Python’s multiprocessing has become commonplace. In this landscape, are you casually sharing file descriptors by thinking, “I’ll just let child processes inherit the parent process’s log output as-is,” or opening files at the same path in each child process?

In truth, this is deeply intertwined with “buffering” and “atomicity” specifications at both the OS and programming language levels. Even if things seem to work fine on the surface, under heavy load, logs can get truncated mid-line, mixed together (interleaved), or in the worst-case scenario, lost entirely.

1. The Three Data Corruption Risks Lurking in Shared Log Files

When multiple processes write to the same log file simultaneously, the OS’s file I/O specifications and the language’s runtime behavior intertwine in complex ways, leading to the manifestation of the following three main issues.

① Application Buffer Contention (Implicit Buffering by the C Standard Library)

Many developers intuitively perceive writing logs as a line-by-line operation (an atomic action). However, while the OS’s write() system call itself inherently behaves atomically, high-level I/O functions provided by programming languages (e.g., C’s fprintf or fwrite, or loggers in high-level languages that wrap them) perform buffering in user space to improve performance.

This is akin to multiple people writing different messages in fragments onto a single bulletin board. Since each process has its own buffer and flushes (writes out to disk) at its own timing, log entries from another process can cut in and mix within a single line.

② Race Conditions for the File Pointer (Offset)

When a process operates on a file, the behavior of the “file offset (file pointer),” which indicates the write position, varies significantly depending on how the process is spawned (fork()) and how the file is opened.

• File Descriptor Sharing via fork(): When a child process inherits a file opened by the parent process, both share the exact same file descriptor and file offset. Writing in this state without mutual exclusion maintains serial writes, but due to the application buffering issue (①), it causes data to jumble up like a puzzle.
• Independent open() in Each Process: If each child process separately calls open() on the same file path, each process will maintain an independent file offset. Writing concurrently in this state leads to a fatal scenario: while Process A is writing to the end of the file, Process B overwrites data based on “its own end-of-file (stale EOF information),” resulting in log data loss (data destruction via overwriting).

③ Severe Performance Degradation Due to Mutual Exclusion (File Locking)

The approach of “if competition occurs, solve it by applying file locks using system calls like flock or fcntl” is logically correct. However, the overhead of acquiring and releasing exclusive locks with every write cannot be ignored.

Particularly in highly concurrent, high-load systems, file locking causes CPU resource contention and disk I/O bottlenecks, drastically degrading the overall throughput of the application. Sacrificing throughput as the price for secure logging is rarely a wise trade-off in practice.

2. Three Logging Approaches: Thorough Comparison of Pros and Cons

We compare and evaluate three representative approaches for balancing integrity and performance in a multi-process environment.

Python: Logger Pitfalls in the multiprocessing Module

Python’s standard logging module is designed to be thread-safe, but it is not process-safe. Reusing the same FileHandler across child processes leads to file offset contention and buffering interference, causing log loss or interleaving with high probability.

• Solution: Adopt an architecture that combines logging.handlers.QueueHandler and QueueListener. Each child process sends log records to a high-speed asynchronous Queue, while the main process (or a dedicated thread) centrally listens to that Queue and writes to a single file. This limits the process performing file I/O to “always exactly one.”

The PIPE_BUF Barrier in Linux

Although the O_APPEND flag guarantees append atomicity at the OS level, there is a physical limit on the “write size.” In Linux kernel specifications, if the write size via O_APPEND is less than or equal to PIPE_BUF (typically 4096 bytes or 4KB on standard Linux systems), the write is guaranteed to be indivisible (atomic).

However, when the write size per entry exceeds 4KB—such as with exception errors containing stack traces or bloated structured JSON logs—even the OS-level write can be split, allowing writes from other processes to interleave in between.

• Solution: Refine the schema of your output JSON logs to strictly manage the size of each event under 4KB, or adopt the “aggregation into a dedicated process approach (②)” mentioned above to guarantee sequential writing (serialization) at the application layer.

Q1: Even in containerized environments, should logs be written directly to local files?

A1: It is not recommended. In modern container platforms like Kubernetes or ECS, the best practice is to output all logs as streams to standard output (stdout) and standard error (stderr). This shifts low-level issues like file-writing contention outside the container, leaving collection and aggregation to dedicated agents like Fluent Bit or Vector.

Q2: If O_APPEND is set, can multiple processes safely append concurrently in any programming language?

A2: While OS-level writes (system calls) will become safe, it remains incomplete if the language’s “buffering feature” is still enabled. If the programming language’s standard I/O classes maintain internal write buffers, even if the atomic write itself succeeds, multiple lines held in memory might be flushed at irregular intervals. This can cause log chronologies to become out of order. Make sure to configure your logger to switch to “Unbuffered” or “Line-buffered” mode.

Q3: How do you safely perform log rotation when multiple processes have the same file open?

A3: This is one of the most difficult “landmines” to debug. If multiple processes rename (rotate) a file while holding the same file descriptor open, each process will keep writing to the old file descriptor it holds. As a result, no logs will be output to the newly created log file.

To avoid this issue, one of the following designs is required:

1. Use the copytruncate option of rotation tools (like logrotate), which copies the existing file and then truncates the original file to 0 bytes.
2. Build a mechanism that sends a signal (such as SIGHUP) to all active processes during rotation, prompting them to close and reopen the file descriptor.

Conclusion: Robust Logging is the Only Way to Maintain Your System’s “Security Camera”

Sharing logs in a multi-process environment is often overlooked as technical debt because the issues rarely surface during development or under low-traffic conditions. However, the logs corrupting at the exact moment the production environment experiences high load and the system begins to scream—that is the ultimate tragedy brought about by poor logging design.

Keep the following three priorities in mind as your core design principles:

1. First Choice: Output logs to standard output (stdout) and offload the subsequent aggregation to infrastructure or platform-side collectors.
2. Second Choice: If kept entirely within the application, use inter-process communication like queues to limit the threads/processes responsible for writing to exactly one.
3. Third Choice: If you must write directly to the same file from multiple processes, strictly apply O_APPEND and explicitly disable buffering.

Let us implement logging designs grounded in a solid understanding of system specifications to build robust, highly traceable systems that remain completely stable even in times of failure.

This article is also available in Japanese.