FlashBlade Surpasses 3 Trillion Objects, Reimagining What’s Possible for Scale Part 2

Read Part 1 of this blog series.

An Opening IT Geek Thought

It’s an odd oversight when you think about it. If you’re a fan of the science fiction genre, you know many stories have speculative hints to technology’s future, with some even proving true:

• Jules Verne conceived of an electric submarine in Twenty Thousand Leagues Under the Sea, roughly 90 years before it became a reality in the 1960s.
• Arthur C. Clarke speculated about communication satellites in a 1945 article, almost 20 years before their inception.
• Ray Kurzweil’s The Age of Intelligent Machines, published in 1990, anticipated the mainstream rise of artificial intelligence (AI) and its potential to surpass human intelligence. He even predicted that AI would be able to beat a human in chess, which happened in 1997 when IBM’s Deep Blue beat Garry Kasparov.

Yet despite all of those amazing concepts coming true through some incredible storytelling, none of them ever described the IT infrastructure needed to accomplish any of it. OK, that buildup and joke may have made you chuckle, but there is some truth to it. Consider what we are seeing with the growth and evolution of AI—almost all speculation is about what it can eventually accomplish, while leaving out the important details of how the underlying technology will evolve to support that future.

With that thought in mind, we recently put our FlashBlade//E™ to the test by having it create 3 trillion objects—we wanted to push the limits of science because we know AI’s possibilities are no longer science fiction.

Important Note: This test would produce the same results on a FlashBlade//S, but we wanted to run it on the model the customer who gave us this challenge is evaluating.

Recapping the Challenge

Pure Storage is an engineering-focused company, which means we embrace technical challenges. The origin of this test came from a customer who challenged the ability of FlashBlade® to create and manage 500 billion objects. 

Their skepticism was justified. The current landscape of legacy and emerging object solutions may struggle with that quantity. But for the customer, it wasn’t just about storing 500 billion objects—it was also about avoiding complicating the environment by implementing a second (or even more) separate system(s) to share the workload. Additionally, they just wanted to test the metadata component, in particular, to see if the metadata performance degrades with more objects, so the objects created didn’t have to have any data in them.

Armed with our natural curiosity and with 500 billion objects as the challenge, we went into the lab to see what we could do.

Challenges in Architecting the Test

Building a test for something that pushes the typical usage of any technology is almost as challenging as conducting the test itself. That’s because we have to simulate levels of usage that may only exist in extreme conditions. Five hundred billion objects is a lot of objects, and generating them for our test required some ingenuity to accomplish it in a reasonable timeframe.

Figure 1: One FlashBlade//E system. Three trillion objects.

We set up a FlashBlade//E system and multiple virtual machines across multiple physical hosts to generate the object creation I/O. In planning the test’s workflow, we attempted these different approaches:

• Wrote some tools in Go to generate billions of files to use for objects
• Considered an S3 mount with Fuse to attempt to populate with RapidFile Toolkit 
• Evaluated an S5cmd for multi-threaded uploads (with the compute resources we had in the lab)
• ObjectCopy within a bucket
• ObjectCopy to multiple buckets

Ultimately, none of these approaches were going to work because of the limited timeline we had to conclude this test. Plus, there were economies of scale with them that we didn’t have the resources for:

• Compute. We would have needed a lot more compute nodes to finish in days instead of months.
• Network. The HTTP traffic for object creation floods the network with a lot of overhead when you have tens of thousands of requests per second hammering it. 

Rethinking the Testing Approach

We knew our bosses were not going to purchase new compute nodes and a massive expansion in our lab’s network. We reached out to our QA team to find out how they test large configurations. Their answer was to change perspective—create the objects from the “inside.” Since the test was focused on the quantity of objects, we could use internal tooling to create the objects natively in Purity. This enabled our test to go 100 times faster and not have any external dependencies on external clients or networking.

Remember, our specific customer challenge was to test the metadata scale of FlashBlade, which meant object creation could happen wherever it worked best. Our localized approach streamlined the test in a way similar to testing a car’s engine by bypassing the gas pedal and using the throttle wire under the hood.

How We Ended Up Testing It

• We leveraged FlashBlade internal production calls to generate objects (which, in turn, created the metadata and stored the object in the system).
• This allowed us to max out the FlashBlade at 100% CPU across all of our control blades, which generated around 3.5 million objects/second in metadata and objects. 
• Each object consumed an estimated 58 bytes of data per object (all of this is metadata).
• Objects were generated at a steady pace; at no point did performance drop off over time. 
• We just let the test keep going, and going, and going… We did pause from time to time to run other, more functional tests to probe and see if we found any anomalies… which we didn’t.

What We Learned

We’re still learning the upper limits of the FlashBlade metadata’s ability to host objects in one chassis. When we began writing this blog, there were 3 trillion objects. At last count, we were at 3.8 trillion:

Figure 2: The final object count before shutting it down. 

This has us in awe and at the same time, not surprised, which is an admittedly strange mental state. Consider this: Amazon AWS manages 350 trillion objects globally, and our two-week test finished at creating 1% of that number! The bigger lesson we learned from it was how creative you need to be when testing something at scale levels outside what is considered “normal.” Time and technology are the two main variables in evaluating anything, and we needed to think differently about how we engaged the technology to produce a timely result.

And the results we got benefited from it all.

Get to Know FlashBlade

Learn more about FlashBlade and its power as the best scale-out data storage solution for unstructured data. Reach out to your Pure Storage account team to connect on how you can benefit from a storage solution built to manage 3 trillion objects (and counting).

Try FlashBlade

No hardware, no setup, no cost—no problem. Experience the self-service capabilities of FlashBlade.

Take a Test Drive