missing writing some concurrency in Go

09 Jul, 2026

It has been a long since I wrote Go. I still remember doing FOSS for CNCF folks and was deep in Go, until I got distracted by Rust ;)

I just saw an awesome Go concurrency visualization. So as I got some free time today, I am gonna write about some concurrency patterns in Go I saw being used in prod.

Background

When Communicating Sequential Processes came, the main thing it did was making communication the foundation of concurrent programming. So, if you want to communicate with 2 processes, you should always do it using fixed defined channel, instaed of directly sharing memory. It was one of the core ideas of the paper. There were many languages born out of the paper. Go’s concurrency model is influenced by Communicating Sequential Processes, where independently executing processes coordinate through communication. Go makes channels first-class values, although channels are not the only synchronization mechanism available. mutexes and atomics are also valid tools.

goroutine                                          goroutine
        │                                                  │
        ▼                                                  ▼
┌──────────────┐    ┌──────────────┐              ┌──────────────┐    ┌──────────────┐
│  Process P1  │    │  Channel C1  │              │  Process P2  │    │  Channel C2  │
└──────┬───────┘    └──────┬───────┘              └──────┬───────┘    └──────┬───────┘
       │                   │                             │                   │
       │─── Sends data ───>│                             │                   │
       │                   │                             │                   │
       │                   │── Passes data to Process P2 ─>│                   │
       │                   │                             │                   │
       │                   │                             │─── Sends data ───>│
       │                   │                             │                   │
       │<── Passes data back to Process P1 ──────────────│                   │
       │                   │                             │                   │
┌──────┴───────┐    ┌──────┴───────┐              ┌──────┴───────┐    ┌──────┴───────┐
│  Process P1  │    │  Channel C1  │              │  Process P2  │    │  Channel C2  │
└──────────────┘    └──────────────┘              └──────────────┘    └──────────────┘

Some basics

There is an overhead in creation/destruction of OS threads (system calls and memory allocation) which are preemptive. They are typically ~1-2MB stack size, have expensive context switches (saves/restores full CPU registers, kernel transition) handled by the OS kernel for small work. As we are not dealing with OS threads, instead goroutines. And goroutines (green threads are incredibly lightweight, starting at ~2-4KB of stack space) are multiplexed into multiple OS threads (usually equal to the number of CPU cores). Because the Go scheduler manages goroutines in user space, switching between them is significantly faster than a kernel-level OS thread context switch. Spawning a million goroutines is cheap relative to OS threads, very large numbers still consume stack and scheduler resources and can retain heap objects.

Traditional OS threads have a fixed stack size. If a thread runs out of stack space, it causes a stack overflow. To prevent this, languages like Java allocate large stacks (e.g., 1MB) upfront, which limits the total number of concurrent threads you can have (usually in the thousands). Goroutines solve this with a growable/shrinkable segmented stack:

They start very small (typically 2KB to 4KB).
If a goroutine needs more memory during execution, the Go runtime automatically allocates a larger segment and copies the old stack over, allowing a single program to run millions of concurrent goroutines simultaneously.

Go uses an M:N scheduler and its scheduler is implemented as 3 structs, G, M and P (all heap allocated, but are never freed, so their memory remains type stable, so the runtime can avoid write barriers in the depths of the scheduler), wherein a goroutines exists only in the virtual space of go runtime and not in the OS :

G (simply Goroutine): Represents the goroutine and its stack. When a goroutine exits, its G object is returned to a pool of free Gs and can later be reused for some other goroutine.
M (Machine): Represents an OS thread managed by the operating system, that can be executing user Go code, runtime code, a system call, or be idle. There can be any number of Ms at a time since any number of threads may be blocked in system calls.
P (Processor): Represents the logical resource (or context) required to execute Go code, such as scheduler and memory allocator state. The number of Ps usually equals the number of CPU cores (GOMAXPROCS).

The scheduler’s job is to match up a G (the code that we want to execute), an M (where we want to execute it), and a P (the rights and resources required to execute it). When an M stops executing user Go code, for example by entering a system call, it returns its P to the idle P pool. In order to resume executing user Go code, for example on return from a system call, it must acquire a P from the idle pool.

Instead of the OS kernel switching between thousands of heavy threads, the Go runtime efficiently schedules active goroutines across a small pool of OS threads. If a goroutine blocks (e.g., waiting for I/O or a channel), its associated OS thread can be detached or assigned to run other ready goroutines, maximizing CPU utilization.

If a goroutine blocks on system call, it blocks it’s running thread. But another thread is taken from the waiting queue of Scheduler and used for other runnable goroutines. However, if you communicate using channels in go which exists only in virtual space, the OS doesn’t block the thread. Such goroutines simply go in the waiting state and other runnable goroutine (from the M struct) is scheduled in it’s place. Don't communicate by sharing memory, share memory by communicating.

The go runtime scheduler also does cooperative scheduling, which means another goroutine will only be scheduled if the current one is blocking or done. This is so much better than pre-emptive scheduling which uses timely system interrupts to block and schedule a new thread as that may lead a task to take longer than needed to finish when number of threads increases, etc.

Go through the paper yourself.

Worker Pools

Now, worker pool allows us to reuse threads for multiple tasks.A worker pool creates a fixed set of worker goroutines. Its main purpose is to bound active work and resource consumption, it does not directly manage or reuse OS threads. to these we can push tasks. We can refer to it as thread pool as well. But, does this matter in Go? You do not really need a worker pool to save the system from the overhead of creating goroutines. So, this traditional pattern (often used in C++ or Java to avoid the heavy cost of spawning OS threads) is not strictly necessary for performance in Go in the same way it is in other languages. If we aren't saving on thread creation costs, why use a worker pool? We use them as a resource management and throttling mechanism.

Throttling

Imagine you are building a service that scrapes 10,000 URLs. If you spawn 10,000 goroutines simultaneously, you will likely:

Exhaust the file descriptors on your machine
Get banned by the target servers as you are making too many requests
Flood your database connections

A worker pool acts as a semaphore. It limits the number of tasks running in parallel to a fixed number, ensuring that even if you have 10,000 tasks, only 20 are active at any given moment.

So, clearly there is a semaphore pattern coming!

Controlling Resource Consumption

Even though goroutines are cheap, the work they might not be. If each goroutine performs a memory-intensive task (like image processing or large JSON decoding or whatever), spawning too many simultaneously will lead to an OOM. A worker pool allows you to constrain memory footprint.

Code Snippet

// Task represents the work to be done.
type Task int

func main() {
	const numWorkers = 3
	const numTasks = 10

	// 1. Create the communication channel.
	// Buffered channels allow the producer to keep working even if workers 
	// are slightly behind, up to the buffer limit.
	jobs := make(chan Task, numTasks)
	
	// 2. Initialize the WaitGroup to track active workers.
	var wg sync.WaitGroup

	// 3. Start the worker pool.
	for w := 1; w <= numWorkers; w++ {
		wg.Add(1) // Increment the counter for each worker started
		go worker(w, jobs, &wg)
	}

	// 4. Send tasks to the workers.
	for i := 1; i <= numTasks; i++ {
		jobs <- Task(i)
	}

	// 5. Close the channel to signal no more work is coming.
	// The range loop in the worker will terminate once the channel is 
	// empty AND closed.
	close(jobs)

	// 6. Block until all workers have finished their tasks.
	wg.Wait()
	fmt.Println("All tasks complete. Main exiting.")
}

// worker is the function run by each goroutine.
func worker(id int, jobs <-chan Task, wg *sync.WaitGroup) {
	// Ensure the worker signals it is done when the function exits.
	defer wg.Done()

	// The 'range' loop will continue until the channel is closed.
	for task := range jobs {
		fmt.Printf("Worker %d started task %d\n", id, task)
		
		// Simulate heavy work
		time.Sleep(time.Millisecond * 500)
		
		fmt.Printf("Worker %d finished task %d\n", id, task)
	}
	
	fmt.Printf("Worker %d shutting down\n", id)
}

close(jobs): This is the most important part of the pattern. Without it, the range loop in the workers will block forever, waiting for a new task that will never arrive (a deadlock/leak).
sync.WaitGroup: This ensures your main function does not terminate before the workers have finished processing the last items in the queue.
Worker Loop: The pattern for task := range jobs is idiomatic Go. It cleanly handles the transition from active work to termination without requiring complex conditional checks.

This pattern is ideal for throttling. By changing numWorkers, you control exactly how many concurrent operations (API calls, file writes, database queries) are happening at once, preventing your system from consuming too many system resources or being rate-limited by external services.

So, you use a worker pool in Go not to save the CPU from creating threads, but to protect your application, your database, or external APIs from being overwhelmed by your own concurrency. Thats why, its concurrency pattern.

Whats leak?

Because in worker pools we create goroutines once and reuse it, idle workers can become problematic. An idle worker is normal while its pool is expected to remain alive. It becomes a leak when the worker should have terminated but remains permanently blocked because its shutdown or cancellation path is broken. If your code leaks goroutines, meaning you have workers hanging around waiting for tasks that will never come, your application doesn't break in a loud, obvious way. Instead, it slowly, quietly begins to wither. No standard memory dump is gonna save your ass. And this leak will slowly compound and make your life hell.

Imagine you have a pool of 100 workers designed to process incoming web requests. Due to a bug in your shutdown logic, every time you deploy a new version of your service or trigger a specific event, 5 of those workers fail to exit. They don't do anything. They aren't consuming CPU cycles, so your monitoring dashboards stay green. They are just... sitting there. They are holding onto 2KB of stack memory each. To a modern server with gigabytes of RAM, 10KB of wasted memory is nothing really. But a month later? You have thousands of these zombie goroutines. Now, your memory usage has climbed by several hundred megabytes. Your garbage collector starts working harder to track these thousands of objects. Your application gets slower, more sluggish, and eventually, the OOM killer arrives and restarts your container without warning.

Dont start panicking and looking at memory dump if this happens, use tools like pprof to visualize the goroutine blocking profile. It will show you exactly where your workers are stuck, and more importantly, it will show you that they have been sitting on that channel operation for hours, which is your indicator that they are zombies.

Semaphore i.e Bounded Concurrency

In typical usecases, we want to achieve the same throttle effect as a worker pool, but without the lifecycle management overhead (you don't need to close channels or manage worker exit).

So for that, we can use a buffered channel as a semaphore. The size of the buffer acts as max concurrency limit.

There is no issue of idle workers here as we spawn goroutines for every task (this is a tradeoff), and they exit completely when done and the memory is reclaimed by GC. Its quite clean.

If your application processes a constant, high-volume stream of tasks, a Worker Pool is more efficient because it avoids the minor overhead of constant goroutine creation. If your application handles periodic, bursty, or ad-hoc tasks, the Semaphore Pattern is significantly cleaner, easier to maintain, and eliminates the risk of goroutine leaks.

In a traditional Worker Pool, if a worker deadlocks, you have a fixed number of stuck workers so your throughput drops to zero whihc is quite predictable and easy to monitor. In the Semaphore Pattern (where you spawn goroutines dynamically), a deadlock can be much more dangerous. New tasks will stop spawning and capacity, can lead to a complete system hang. So, never perform a task (especially one involving I/O, network, or DB calls) without a context.WithTimeout. If a task hangs, the context will cancel, the execute function should return, and the defer will release the semaphore. You can also monitor semaphone depth, since the channel is a physical object in your memory. If len(semaphore) stays at maxCapacity for an unusually long time, you know your workers are deadlocked. You can expose this as a metric (e.g., Prometheus) to alert you that your in-flight tasks are stuck. If you seriously want to do an x-ray :

Run go tool pprof http://localhost:PORT/debug/pprof/goroutine
Look for goroutines stuck in execute
Because you aren't using a long-lived pool, you will see a bunch of identical goroutines all stuck at the exact same line of code in execute. If you see 100 goroutines all waiting on the same network call, you have found your deadlock immediately

Things in action

In production, your service isn't just processing a fixed list of tasks, it’s likely reading from an endless stream (like a Kafka topic). You want to process these messages in parallel to maintain high throughput, but you need to ensure that when your service receives a termination signal (like a Kubernetes SIGTERM), it doesn't just kill the application and lose the message currently being processed.

// Consumer manages the ingestion and processing flow
func Consumer(ctx context.Context, workerCount int) {
	// 1. Channel to buffer messages between the reader and workers
	queueChan := make(chan string, 100) 
	var wg sync.WaitGroup

	// 2. Spawn worker pool
	for i := 1; i <= workerCount; i++ {
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			// Workers process until the channel is closed
			for msg := range queueChan {
				fmt.Printf("Worker %d processing: %s\n", id, msg)
				time.Sleep(500 * time.Millisecond) // Simulate work
			}
			fmt.Printf("Worker %d stopped.\n", id)
		}(i)
	}

	// 3. Main consumption loop
	// This loop pulls from the "stream" (Kafka/Queue)
	go func() {
		defer close(queueChan) // Closing allows workers to finish remaining tasks and exit
		for {
			select {
			case <-ctx.Done():
				// Triggered by OS signal or shutdown request
				fmt.Println("Shutdown signal received, stopping ingestion...")
				return
			default:
				// Simulate reading from Kafka
				msg := "kafka-message-" + time.Now().Format("15:04:05")
				
				// Attempt to send to queue; blocks if buffer is full (Backpressure)
				select {
				case queueChan <- msg:
				case <-ctx.Done():
					return
				}
			}
		}
	}()

	// 4. Graceful Shutdown Wait
	// Wait here until the context triggers and the channel is closed
	<-ctx.Done()
	wg.Wait() // Wait for all workers to finish processing current items
	fmt.Println("All workers finished. Clean shutdown complete.")
}

Interesting things in code :

Cool Backpressure Management! The queueChan is buffered (e.g., make(chan string, 100)). So, if your workers are slower than the Kafka reader, the channel will fill up. Once full, the queueChan <- msg operation will block the producer. This forces your Kafka consumer to wait, effectively applying backpressure to the upstream system. This prevents your service from running out of memory by hoarding too many messages in the RAM.
Graceful Degradation. In production, you never want to kill a process abruptly for sure. so when service receives a SIGTERM from k8s, ctx.Done() signals the reader stops pulling new messages from Kafka, close(queueChan) sends a stop signal to all workers and the hero wg.Wait(), the main function waits for the WaitGroup. Because the channel is closed, the range loop in each worker will naturally finish processing the remaining items in the buffer and then exit gracefully.

In a real production app, you would add an errChan to catch failures from execute(task). If a worker encounters a database error, it shouldn't just die! it should log the error and perhaps send the message to a Dead Letter Queue (DLQ) so you can replay it later without crashing the entire consumer.

Pipelines

Where Worker Pools are about parallelism (doing the same task N times at once), Pipelines are about composition (doing a sequence of different tasks in stages). Each ETL stage is a separate goroutine, and they are connected by channels. Each stage has its own in-channel and out-channel.

// Stage 1: Generates work
func generator(done <-chan struct{}, nums ...int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for _, n := range nums {
            select {
            case out <- n:
            case <-done: return
            }
        }
    }()
    return out
}

// Stage 2: Processes work
func sq(done <-chan struct{}, in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for n := range in {
            select {
            case out <- n * n:
            case <-done: return
            }
        }
    }()
    return out
}

This is quite a gold standard in prod.

Its modular. You can test Stage 1, Stage 2, and Stage 3 independently
The gift of independent Scaling. You can have 1 "Extract" goroutine, 10 "Transform" goroutines (a worker pool within a stage), and 2 "Load" goroutines. You match the resource usage to the bottleneck of each specific stage.
Pipelining gives True Parallelism. Because stages are connected by channels, while Stage 3 is saving message A to the database, Stage 2 can be transforming message B, and Stage 1 can be fetching message C. It keeps your CPU cores busy.

Fan out and in!

In production, you often need to Fan-Out (start multiple workers for a slow transformation stage i.e distribute tasks to multiple goroutines for parallel processing) and Fan-In (merge those multiple results back into one channel). This combines everything :

Pipeline - The structure of stages
Worker Pool - Used within a stage to process data in parallel
Semaphore/Channel Throttling - Used to ensure the pipeline doesn't consume more memory than allowed

Channels as first class primitives

Because channels are values, you can pass them, store them, and send them over other channels. This allows you to treat channels as futures or promises.

In standard concurrency, ordering is rarely guaranteed because of the Go scheduler is not deterministic. When you launch Goroutines, they complete in whatever order the CPU dictates. If you need order, you have to engineer it. Its a burning question! How do I process tasks in parallel to get speed, but output the results in the exact order they arrived?

So, its awesome for ordered data streaming, but its quite fragile when managing complex state reconciliation (like k8s does)

Technicalities

The core of this pattern lies in decoupling the order of dispatch from the timing of execution. We achieve this by establishing a strict synchronization barrier using a Channel of Channels (a chan chan T).

First we establish sequence. The dispatcher is responsible for two distinct operations:

Orchestration - It initializes a unique resultChannel for every incoming task. This channel acts as a private synchronization point for the specific task's lifecycle.
Sequencing - It pushes the newly created resultChannel into the procQueueChan. Because procQueueChan is a buffered or unbuffered FIFO channel, the order in which channels are inserted is preserved. This is the Serialization Phase.

Each worker goroutine is decoupled from the sequencing logic. It receives its specific task and its private resultChannel. Because the runtime scheduler handles goroutine preemption and execution across M OS threads, the workers execute in parallel. The completion order is arbitrary, dependent on task complexity, I/O wait times, and scheduler state. Crucially, the worker is not responsible for ordering, it is only responsible for executing the logic and signaling completion by writing to its assigned resultChannel.

Later we reestablish order. The Aggregator stage iterates over procQueueChan. Technically, it performs two levels of indirection: result := <-(<-procQueueChan).

Channel Acquisition - It blocks until it receives a channel from the queue. This is the Synchronization Barrier.
Result Retrieval - Once it acquires the resultChannel, it immediately blocks on <- resultChannel.

The Aggregator effectively waits for the workers in the precise order they were dispatched. If the first task (Task N) takes significantly longer to execute than Task N+1, the Aggregator will remain blocked at <- resultChannel_N, even if the data for Task $N + 1$ has already been computed and is sitting in its own resultChannel. This creates a deterministic output stream that mirrors the input stream's sequence.

There are thinfs to be concerned about, plenty of runtime constraints.

There can be memory pressure if you push things to scale. You are creating N channel objects for $N$ tasks. While Go's heap allocation for channels is efficient, in high-throughput systems (thousands of tasks per second), this increases GC pressure. Developers must tune GOGC or consider channel reuse patterns to mitigate heap fragmentation.
This pattern inherently suffers from Head-of-Line (HOL) blocking. If the first task in a sequence is computationally expensive or hangs, the entire pipeline's output is blocked, regardless of how quickly subsequent tasks are completed.
Each worker goroutine must have a defer block that either writes to the resultChannel or closes it. Failure to send a result will cause the Aggregator to block indefinitely, leading to a permanent stall of the pipeline.

Studtying k8s a bit

Ordered Parallelism is pretty foundational concurrency pattern in k8s controller (kinda implicit Fan-Out/Fan-In though). but K8s prefers Work Queues with internal locking mechanisms, where key-based locking acts as the serialization barrier to ensure that parallel workers don't corrupt the cluster state by processing events out of order. pattern manifests as :

The Queue Interface: Instead of passing channels, K8s uses an interface that keeps track of the Key (e.g., namespace/pod-name).
Deduplication: In a pipeline, if you have multiple events for the same resource (e.g., Pod "A" created, Pod "A" updated), a raw Fan-Out might send these to different workers simultaneously. Kubernetes Work Queues use deduplication based on an object's key. This ensures that only one worker processes a specific object at any given time, which is the only way to guarantee consistency in a state machine. The workqueue internally ensures that if the same key (the same Pod) is added to the queue multiple times while a worker is processing it, the queue deduplicates those events.
Serialization: By locking on the Key (the object identifier), K8s achieves the exact same goal as Channel of Channels! It ensures that only one worker is ever processing a specific object at any given time, preserving the order of operations for that object.
Failure Recovery: Raw channels cannot "retry" an event easily. Kubernetes Work Queues provide a Requeue mechanism where a controller can signal that it failed to process an object, allowing the queue to put it back for later retries.

the secret sauce seems to be rate limited queue :

Kubernetes must guarantee that if it receives Create Pod and then Delete Pod for the same object, it processes them in that order. If it processes "Delete" before "Create," the controller state would be permanently broken.
To handle shit be inserted: If a specific object causes a controller to panic or error consistently, you do not want to block the entire pipeline you knww
Backoff Logic: The RateLimitingQueue tracks how many times an object has failed. If it fails, it is not immediately re-queued, the queue forces a delay (e.g., 5ms, 10ms, 20ms...) before allowing it to be processed again. This prevents a faulty object from creating a hot loop that consumes 100% of the controller's CPU.

So, in k8s, things are implicit :

Implicit Fan-Out: The workqueue doesn't use a Channel of Channels to distribute work, instead, you, as the developer, define N worker goroutines that all call queue.Get() on the same workqueue instance. This is the Worker Pool pattern. It effectively performs the Fan-Out by allowing multiple workers to pull from the central queue in parallel.
No Fan-In Required: Because the workers update the Kubernetes API Server directly (the Source of Truth), there is no need to merge results back into a single channel. The API server acts as the final aggregator for the system's state.

ultimately, K8s is a distributed system that must achieve eventual consistency. This concurrency pattern is chosen for two specific reasons:

Isolation: If a worker processing a Pod event panics or hangs, it doesn't stop the workers processing Node or Service events. The queue ensures the controller stays responsive.
Ordered Reconciliation: The cluster state is essentially a giant fucking state machine. You cannot reconcile a Pod's status if you don't process the events that led to its current state in order obviosuly. This pattern gives K8s the ability to scale to thousands of Pods (parallelism) while keeping the API server state consistent (ordering).

But dont you like Rust?

Yea. Its philosophical. You might not have a great first impression of it, but often the second impression of Rust is nice.

You know what happens when a goroutine panics while holding a standard Go sync.Mutex? Go begins unwinding the call stack for that specific goroutine, executing any defer statements along the way. If you followed idiomatic Go and placed defer mu.Unlock() immediately after locking, the deferred function will execute during stack unwinding, releasing the mutex so other goroutines don't deadlock. Unless caught, the panic will travel up to the goroutine's root, crash the entire Go program, and print a stack trace. You like this? the recover() will only works if called directly inside a deferred function. There are isolation troubles as well. a panic in one goroutine does not automatically crash other goroutines, it only crashes the entire program if the panic happens on a goroutine that doesn't recover it (with the exception of the main goroutine). If Worker A panics and recovers using defer + recover(), Worker B and Worker C keep running uninterrupted. However, because Go doesn't poison mutexes, if Worker A left the shared data structure in a half-modified, broken state before panicking, Worker B will successfully acquire the lock and read that corrupted data without any warning from the runtime. I dont like this. Go trusts you. If a panic happens, the lock opens, but the data is unverified.

Lack of Compile-Time Thread Safety - Unlike Rust with rigorous ownership models, Go's compiler does not prevent data races at compile time. Standard types (like maps and slices) are unprotected. Concurrency errors, such as concurrent map writes or race conditions, are frequently discovered only at runtime—either via tests, the runtime race detector, or sudden 2AM production crashes.
Silent Data Corruption via Unpoisoned Mutexes - GGo has no concept of mutex poisoning. A Go mutex is completely agnostic to panics. If a goroutine panics, the mutex unlocks (assuming you used defer), but it remains completely valid and available for other goroutines to lock immediately. Go assumes that if you recover from the panic, the shared data is fine, or it's your responsibility to clean up any corrupted state
Goroutine Leaks - Because spawning a goroutine is as simple as prefixing a function call with the go keyword, it is easy to spawn tasks that block indefinitely on unbuffered channels or missing context cancellations. These leaked goroutines persist in memory for the lifetime of the application, gradually degrading performance without an immediate compiler or runtime warning.
Deadlock Vulnerabilities - While Go's runtime can detect simple deadlocks (e.g., all goroutines sleeping forever), complex deadlocks involving channels, select statements, and external resource locks often go undetected by the runtime. Debugging blocked channels or circular dependencies in large Go codebases is fucked up
Abstracted Scheduling Control - Go's M:N scheduler handles goroutine multiplexing onto OS threads automatically. While convenient, this abstraction gives developers very little direct control over OS thread pinning, CPU affinity, or low-level thread scheduling priorities, which can be a limitation for ultra-low-latency systems

Now that I started discussing about MUtex Poisoning

If a thread panics while holding a standard standard library Mutex (typically wrapped in an Arc>), Rust does not leave the mutex open and silent. When the thread panics, its stack unwinds, and the RAII MutexGuard goes out of scope and is dropped. During this drop, Rust's Mutex detects that the thread holding the lock panicked. It automatically poisons the mutex. The mutex is marked as poisoned because the data it protects may have been left in a half-modified, inconsistent state when the panic hit. Unlike Go, where another thread can immediately grab the unlocked mutex and read potentially corrupted data, Rust refuses to let you blindly trust the data. When another thread tries to call .lock() on a poisoned mutex, it receives a PoisonError instead of the normal data. Rust forces you (via the type system and Result handling) to explicitly choose how to handle the poisoned state before you can access the inner data. You have to handle the Err variant of the lock result, preventing silent data corruption.

use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    let data = Arc::new(Mutex::new(vec![1, 2, 3]));
    let data_clone = Arc::clone(&data);

    let handle = thread::spawn(move || {
        let mut guard = data_clone.lock().unwrap();
        guard.push(4);
        panic!("Something went wrong mid-modification!"); // Panic while holding lock
    });

    let _ = handle.join(); // Wait for thread to panic

    // Next thread tries to acquire the lock:
    match data.lock() {
        Ok(guard) => {
            println!("Data: {:?}", *guard);
        }
        Err(poisoned) => {
            // Rust caught the panic! The data might be corrupted.
            // We can access the inner data via the poison error if we want to recover it:
            let guard = poisoned.into_inner();
            println!("Recovered from poisoned mutex. Potentially inconsistent data: {:?}", *guard);
        }
    }
}

This shows you how languages differ philosophically. Go's Philosophy to me is Simplicity and Trust. Go gives you lightweight primitives (goroutines, channels, sync.Mutex) and assumes you, as the developer, will write the correct orchestration code. It trusts that if a panic occurs, you've handled it with recover() and cleaned up your data structures. It prioritizes runtime performance and flexibility, meaning it won't inject hidden overhead to track the health of every single lock. Rust's Philosophy could be Fearless Concurrency via the Type System. Rust assumes that any shared mutable state is a potential bug factory, and threads will panic unpredictably. Therefore, Rust builds safety checks directly into the compiler and types (Send, Sync, MutexGuard, PoisonError). It forces you to deal with failure states (like a poisoned mutex) at compile time or via explicit error handling, making silent data corruption much harder to achieve.

Rust is better in the Agentic world

I personally feel writing Rust with agents is just a better experience, its ideal for vibe coding. The compiler acts as an automated reviewer, it eliminates entire classes of bugs at compile time. AI agents frequently struggle with multi-threading synchronization because reasoning about concurrent state across multiple files is difficult for LLMs. Rust's type system prevents the AI from accidentally creating data races or unsafe shared-state concurrency.

Rust is built different!