Why Does AI Compute Favor Rectangles? Modal CEO Breaks Down Agent Cloud Economics

In a recent talk, Modal CEO Erik Bernhardsson raised a seemingly odd question that cuts straight to the essence: why are the compute units in data centers always rectangular? From GPU boards to rack servers, rectangles are almost the default shape for compute hardware. Behind this observation lies his full reasoning about elastic GPUs, sandbox growth, and agent cloud economics.

Rectangular Compute: An Overlooked Physical Constraint

Bernhardsson's starting point is not aesthetics but physics and manufacturing. Chip lithography, PCB routing, and rack space allocation all treat rectangles as the optimal solution. Circular or irregularly shaped chips cause extra waste in cutting and packaging, while rectangles maximize the use of silicon wafer area and rack depth.

This constraint propagates through every layer. GPUs are rectangles, servers are rectangles, racks are rectangles, and data center floor planning also tends toward rectangular grids. When cloud providers design elastic compute products, the rectangular nature of the underlying hardware determines the granularity of resource partitioning—it is very difficult to slice a rectangular GPU into perfect circular shares without creating fragmentation.

The shape of compute is not a design choice but a result of manufacturing and physical laws. Only by understanding this can you understand the real cost of resource scheduling in the cloud.

Elastic GPUs: How Hard Is It to Slice a Rectangle

Modal's core business is letting developers call GPUs on demand without worrying about the underlying machines. In his talk, Bernhardsson focused on the scheduling challenge of elastic GPUs: what users want is fine-grained, instantly available compute, while the hardware provides solid rectangles.

Resolving this contradiction requires multiple layers of abstraction. Containerization allows workloads to migrate quickly, but isolating GPU memory and compute remains a hard problem. When multiple tenants share the same GPU, memory fragmentation, compute contention, and fault isolation all increase scheduling complexity. Modal's approach is more aggressive packing and reclamation at the scheduling layer, so that idle compute returns to the pool as quickly as possible rather than being occupied long-term by a single tenant.

According to Bernhardsson, there is still substantial room to improve elastic GPU utilization. GPUs in traditional clouds have long had high idle rates, and the burstiness of agent workloads makes this problem even more pronounced—an agent may need a large amount of concurrent inference within seconds, then quickly drop off.

The difficulty of GPU memory isolation lies in the hardware's lack of a native partitioning mechanism. Traditionally, a GPU's memory is managed uniformly by the driver, and when multiple processes share it, memory fragmentation easily occurs—that is, the total amount of free memory is sufficient, but because it is non-contiguous it cannot be allocated to new tasks. NVIDIA's MIG (Multi-Instance GPU) technology provides near-native isolation by partitioning models such as the A100 into multiple independent instances at the hardware layer, but not all GPU models support it, and the partition granularity is fixed. A more general approach is time-slice rotation and memory pooling at the driver or runtime layer, but this introduces context-switching overhead and contention latency. Platforms such as Modal typically combine multiple methods: enabling MIG on supported hardware, and performing finer-grained packing and preemptive reclamation at the scheduling layer on unsupported hardware, thereby balancing isolation and utilization.

Sandbox Growth: The Infrastructure Foundation of Agent Cloud

Sandbox is another keyword Bernhardsson repeatedly mentioned. When an agent executes tasks, it needs isolated environments to run code, call tools, and verify results. The startup speed and density of sandboxes directly determine the agent's response experience.

Modal's investment in sandboxes is reflected in cold start optimization. Traditional virtual machines take tens of seconds to start, containers can achieve second-level startup, and lighter-weight isolation solutions can push startup down to the millisecond level. For agent workflows that need to frequently create and destroy environments, this gap is amplified into an order-of-magnitude cost difference.

Sandbox growth also brings new economic problems. Every sandbox occupies memory and storage, and idle sandboxes are pure cost. How to minimize the residency cost of sandboxes while ensuring isolation security is a key battleground in the competition among agent cloud platforms.

Consensus from Multi-Source Cross-Verification

Based on discussions across tech-focused UP creators on Bilibili and multiple technical channels on YouTube, there is relatively high industry consensus on the following judgments: the burstiness of agent workloads and the demand for sandbox density are forcing cloud platforms to redesign their scheduling layers; elastic GPU utilization is the core variable in the cost structure; cold start speed is an invisible threshold for agent experience. Bernhardsson's talk is consistent with these judgments.

Points of Disagreement

The community disagrees on how lightweight sandboxes should be. One camp believes that microVMs achieve the best balance between security and startup speed and should be the default solution; the other argues that process-level isolation combined with mechanisms such as seccomp is sufficient and pursues more extreme density. In his talk, Bernhardsson leaned toward pragmatic engineering trade-offs and did not give a single answer.

A microVM is a lightweight virtualization technology, typically represented by AWS Firecracker, Google gVisor, and Kata Containers. It retains the strong isolation boundary brought by hardware virtualization (each sandbox has an independent kernel) while compressing startup time to the millisecond level, with memory overhead also far lower than traditional virtual machines. Through a streamlined virtual device model and a Rust implementation, Firecracker can start a microVM in about 125 milliseconds. Process-level isolation relies on mechanisms such as Linux namespaces, cgroups, and seccomp. It has almost no additional startup overhead and offers higher density, but its isolation strength is weaker, and kernel vulnerabilities may be exploited for escape. The trade-off between the two is essentially a balance between security boundaries and resource efficiency, and different agent scenarios (such as executing untrusted code vs. running trusted workflows) may be suited to different solutions.

Agent Cloud Economics: Costs Beyond Compute

When Bernhardsson turned the topic to economics, his focus was not on GPU unit prices but on total cost of ownership. Agent cloud customers pay not only for compute but also for scheduling efficiency, cold start latency, and operational simplification.

One easily overlooked point is the hidden cost of networking and storage. Agents frequently read and write tool outputs, logs, and intermediate results, and storage I/O and network round trips erode profits. Modal's product design emphasizes preloading common dependencies into the image layer to reduce runtime pull overhead, which is essentially optimizing the cost structure of the entire chain.

From an industry perspective, agent cloud is shifting from "selling GPU hours" to "selling task completion." Hybrid billing models based on number of calls, sandbox seconds, and peak concurrency will become increasingly common, and the core competitiveness of platforms lies in minimizing the fragmentation loss of rectangular hardware.

Practical Implications for Developers

Bernhardsson's talk offers several actionable insights for frontline developers. When evaluating cloud platforms, you should look not only at GPU list prices but also at cold start time, sandbox density limits, and idle reclamation strategies. When designing agent workflows, try to break long tasks into cacheable, reusable steps to reduce dependence on immediate compute.

Rectangular compute is a metaphor: underlying physical constraints will not disappear, and what can be done is to build smarter abstractions on top of them. Modal's bet is that if scheduling and sandboxes are good enough, developers will no longer need to care about the existence of rectangles.

One-line Summary

Modal CEO analyzes the economics of elastic GPUs, sandboxes, and agent cloud under the constraint of rectangular compute.

Paragraph Summary

Starting from the physical constraint of why compute units in data centers are always rectangular, Modal CEO Erik Bernhardsson lays out the full logic of elastic GPU scheduling, sandbox infrastructure, and agent cloud economics. He points out that rectangles stem from the physical laws of chip manufacturing and rack space, and this constraint propagates to the granularity of cloud resource partitioning, making fragmented GPU scheduling a core challenge, while the burstiness of agent workloads further amplifies idle rates. The gap in sandbox cold start speed from tens of seconds to milliseconds is amplified into an order-of-magnitude cost difference, and sandbox residency cost is also key to competition. Agent cloud is shifting from selling GPU hours to selling task completion, hybrid billing models will become more common, and platform competitiveness lies in reducing the fragmentation loss of rectangular hardware. Developers should pay attention to cold start, sandbox density, and idle reclamation strategies.