From Petabytes to Exabytes: Object Storage for AI and HPC without Complexity

AI isn’t just changing how data is used; it’s rewriting how it’s stored and managed at unprecedented scale. Every new model, simulation, and checkpoint adds to a surge of unstructured data that’s growing faster than ever, outpacing what legacy systems can handle. Analysts estimate global data will exceed 393 zettabytes by 2028, with AI workloads driving much of that growth.

Object storage, once limited to backups and archives, has become essential for AI and HPC. The same S3 protocol that once handled cold archives now powers the entire AI pipeline from ingestion to training to inference. The rise of object storage is also being driven by exponential data growth from IoT-connected devices, cameras, sensors, self-driving cars, and mobile devices, all generating massive volumes of images, videos, speech recordings, and log data across industries such as finance, medicine, and logistics. Most of this new data is natively created and stored as objects, reinforcing object storage as the natural format for modern unstructured data growth. While S3 is the most widely adopted, other object storage platforms, such as Azure Blob and Google Cloud Storage, offer similar capabilities and a high degree of overlap, making object storage a flexible and broadly compatible foundation for modern data pipelines.

AI pipelines loop, checkpoint, and shuffle data across thousands of GPUs, each demanding billions of metadata operations per second. Legacy systems struggle, leaving GPUs idle. Object storage, with its native parallelism and scalability, solves the problem, but most platforms layer object on top of legacy services.

Pure Storage saw this shift coming. FlashBlade® pioneered “fast object,” treating object storage as a first-class, performance-critical technology rather than an archival layer.  With FlashBlade//EXA™, Pure Storage re-engineered storage for a world where performance, density, and simplicity are table stakes. Now, FlashBlade//EXA adds native object support, currently in limited preview, bringing the same efficiency trusted in file workloads to the new frontier of AI infrastructure. Unlike legacy parallel storage or siloed object systems, it scales capacity and performance linearly as you go from petabytes to exabytes without the tuning or complexity tax.

As AI models grow, storage must scale efficiently. That’s exactly what FlashBlade//EXA is built to do.

Scaling Object Storage for the AI Era

AI data pipelines grow hungrier by the day, scaling from terabytes to petabytes to exabytes. Models that once trained on terabytes now consume petabytes and soon will demand exabytes. AI workloads require high throughput, the ability to handle billions of metadata transactions per second, and the flexibility to accommodate different tradeoffs between performance and capacity. Traditional file systems weren’t built for that. Object storage was. It’s inherently distributed, built for parallel access, and free from the choke points that slow legacy architectures.

This shift is already here. Analysts expect object storage to grow over 20% annually through 2028, driven by AI workloads.¹ Hyperscalers have made object storage the backbone of their infrastructure. For example, AWS S3 manages more than 350 trillion objects and continues to add features like S3 Express One Zone and S3 Vectors to feed generative-AI workloads. Object storage is also continually evolving and improving to support a broader range of AI scenarios.

Enterprises building RAG and foundation-model environments are following suit. It is not just about scale but about flexibility and scalability. With object storage, teams can land unstructured data, checkpoint models, and serve inference workloads, without managing multiple storage tiers. As the name suggests, S3 stands for Simple Scalable Storage, a reminder that simplicity is a core part of what makes object storage so powerful for AI.

Once again, Pure Storage saw the next shift coming, not just to object but to an entire new scale of AI and HPC workloads. FlashBlade//EXA with Object Support combines extreme performance with object scalability, delivering consistent throughput and metadata performance across the AI pipeline, from ingestion to training to inference, while achieving industry-leading performance density. This vision aligns directly with NVIDIA’s own AI Factory reference architectures, which now explicitly include high-performance object storage as an essential tier for large-scale data pipelines.

Why Legacy Architectures Can’t Keep Up with the Dynamic Requirements of AI

AI workloads are unpredictable, dynamic, and complex, and legacy systems make them harder than they need to be. As models scale, the volume of data and metadata grows exponentially. Performance drops, GPUs sit idle, and costs rise as teams add more hardware and tuning to compensate.

Traditional file systems were built for large, predictable workloads, not the chaotic mix of reads, writes, and checkpoints that define AI and HPC pipelines. And while much of today’s data resides in object stores, traditional object services are often too slow to keep GPUs fed. They struggle with billions of small metadata operations and the bursty nature of GPU training. As a result, many environments rely on temporary file-based caching layers to accelerate data delivery and maintain high performance. While this helps mask performance gaps, it also adds architectural complexity and cost. Every bottleneck in storage slows progress and increases total cost of ownership.

FlashBlade//EXA eliminates the need for this extra tier entirely, delivering high-performance access to object data natively.

Common Challenges with Traditional Systems

• Limited object protocol maturity: Many storage stacks still treat object as secondary, often bolting it on top of existing file systems, which limits features and compatibility for AI and HPC.
• Inflexible scaling: Metadata and capacity often grow together, leading to overprovisioning and wasted resources.
• Slow or stalled training runs: Metadata hotspots and uneven I/O leave GPUs underutilized.
• Energy and footprint constraints: Power and cooling limits how far systems can scale.
• Operational inefficiency: Constant tuning and specialized staffing raise costs and delay innovation.
• Lack of flexibility: One-size-fits-all architectures can’t adapt to evolving data patterns or performance needs.

Many organizations still rely on parallel file systems or siloed object stores that need manual intervention to stay operational. They require specialized skills and frequent tuning just to maintain stability. The result is predictable: Environments that look fine on paper underperform in production.

Object storage can break that pattern, but only if it’s built for performance, metadata scaling, and simplicity from the start. That’s where FlashBlade//EXA takes a different approach.

Announcing FlashBlade//EXA with Object Support

AI’s data challenge pushes the limits of scale, performance, and efficiency that needs a smarter architecture, not more hardware. FlashBlade//EXA with native object support extends Pure Storage simplicity and performance across file and object workloads in one unified solution.

Built on FlashBlade//EXA disaggregated design, it scales metadata and data independently to eliminate overprovisioning and expand storage as workloads demand. Designed for AI, it delivers performance, efficiency, and density without tuning or tradeoffs.

Key FlashBlade//EXA Capabilities

• Simpler operations at scale: The Pure Storage proven metadata core and full S3 feature set simplify management and ensure consistent performance.
• Right-sized performance and cost: Choose configurations optimized for performance or capacity efficiency to reduce waste.
• Energy and space efficiency: DirectFlash® Modules deliver higher density, lower latency, and greater energy efficiency, ideal for power- or rack-constrained data centers.
• Metadata and data disaggregation: FlashBlade//EXA separation of metadata and data extends into the object domain, eliminating bottlenecks and handling highly performant critical metadata operations such as list.
• Future-ready I/O pathing: FlashBlade//EXA supports TCP today and RDMA in the future for faster GPU data paths.

FlashBlade//EXA object storage is designed to support AI and HPC workloads that demand massive scalability and high performance without adding operational complexity. It delivers predictable throughput and IOPS, keeping GPUs fully utilized even as data scales to exabytes.

FlashBlade//EXA gives customers a single platform that scales without compromise, handling any AI or HPC workload with the performance, density, and simplicity that define Pure Storage.

Our Track Record of Innovation in AI, Fast Object, and Efficiency

For more than a decade, Pure Storage has led the evolution of AI infrastructure, helping customers train deep learning models, run inference at scale, and optimize GPU utilization. That experience is reflected in the design of FlashBlade//EXA and our focus on performance, efficiency, and simplicity.

The proven fast object capabilities of FlashBlade, its S3 stack, and metadata core deliver billions of operations per second, keeping pace with AI pipelines that feed thousands of GPUs in parallel. In fact, in a recent internal test, FlashBlade stored more than 3 trillion objects, validating the strengths of the FlashBlade metadata core, including its performance and scalability.

With NVIDIA certifications including SuperPOD, BasePOD, and NCP, and validated solutions like AIRI® and FlashStack® for AI, Pure Storage ensures compatibility and eliminates data bottlenecks.

Pure Storage continues to advance high-performance object storage with innovations such as:

• S3 over RDMA, which creates direct data paths between FlashBlade and GPUs to reduce latency and boost throughput
• Key-Value Accelerator (KVA), enabling both file- and object-based inference workloads to run faster and more efficiently
• DirectFlash Modules of up to 300TB, redefining performance density and power efficiency while supporting exabyte-scale data sets without expanding the footprint

All of these innovations share one philosophy: reduce complexity while increasing performance. The result is a storage platform that delivers sustained speed, scalability, and efficiency without the operational overhead that slows competitors down.

What Sets Object Support on FlashBlade//EXA Apart

FlashBlade//EXA with Object Support eliminates bottlenecks in modern AI workloads, keeping GPUs busy and delivering predictable throughput and IOPS from petabyte to exabyte, with unmatched simplicity. It lets organizations scale capacity and performance independently, with the simplicity and reliability that define Pure Storage. Built on the long history of proven FlashBlade S3 support, this isn’t a new exercise and expands on years of maturity, real-world deployments, and continuous innovation.

AI infrastructure shouldn’t force tradeoffs between performance, density, and manageability. FlashBlade//EXA delivers all three: speed, efficiency, and ease of use, ensuring infrastructure scales as fast as innovation.

FlashBlade//EXA Difference

• Performance and efficiency: Maximize performance density while minimizing watts per terabyte with DirectFlash Modules for sustainable AI infrastructure.
• Proven object capabilities: Rely on a battle-tested and proven S3 stack with complete breadth of functionality, rack-level resiliency, and consistent uptime for critical workloads.
• Independent scalability: Grow metadata, capacity, and performance separately and linearly to expand from petabytes to exabytes without redesign or overprovisioning.
• Operational simplicity: Eliminate manual tuning through a unified file and object platform that’s effortless to manage at scale.
• Accelerated AI performance: Keep GPUs fully utilized with consistent, predictable data delivery across the entire pipeline.

The Road Ahead

As AI evolves and unstructured data continues to grow exponentially, Pure Storage will keep leading the innovation with FlashBlade//EXA, advancing performance, efficiency, and simplicity at exabyte scale. Learn more and join the future of AI storage at Pure.AI.

¹ IDC Worldwide File- and Object-Based Storage Market Update and Forecast, 2024–2028

FlashBlade//EXA

Experience the World’s Most Powerful Data Storage Platform for AI

Learn More