This iRunning modern AI workloads places immense pressure on the storage layer. This is especially true when building RAG pipelines that rely on multimodal data. Applications don’t just need vector embeddings; they need the matching text metadata and the original raw assets (like JPEGs or PDFs) simultaneously. Furthermore, search queries must return in milliseconds, even under heavy concurrent load.

Historically, supporting this meant stitching together a fragmented architecture. Your vector embeddings and structured metadata lived in a database, while the actual raw assets like your JPEGs or PDFs used for citations were completely siloed in a separate storage system.  

Unlike traditional columnar formats such as Parquet, which lack native vector indexing and impose significant latency for random point lookups, the Lance file format was specifically developed for ML workloads. LanceDB is built directly upon this Lance format, eliminating unnecessary architectural complexity. It offers zero-copy point lookups, native vector indexing, and an append-only versioned manifest.

With LanceDB, embeddings, metadata, and raw blobs can all be stored in a single table, enabling them to be versioned and queried together. Scaling LanceDB requires a storage layer that absorbs massive concurrency.
Vector search generates high volumes of random reads, while ingestion and compaction drive heavy metadata and I/O churn. General-purpose S3 implementations often buckle under this high-concurrency profile.

Let’s jump into the three main challenges and how Everpure™ FlashBlade^® addresses them. This blog isn’t just theory. We’ve included our observations, results, and most importantly, a Rust-based toolkit so you can try it yourself.

1. Handling concurrent writes with atomic commits

In a production pipeline, you aren’t just writing once. You have multiple parallel workers pushing embeddings and metadata simultaneously. If these workers hit the same S3 table, the storage layer must enforce strict concurrency control; otherwise, manifests will silently overwrite each other, resulting in state corruption.

Lance uses an append-only, versioned manifest architecture. When multiple writers race to create the same manifest version, the storage layer must guarantee that only one writer wins and the others are instantly rejected.

Solution 

FlashBlade natively supports S3 conditional writes via the If-None-Match header. When concurrent writers race with this header enabled, FlashBlade checks the namespace for an existing ETag. If an ETag is already present (meaning another writer just won the race), FlashBlade strictly enforces atomicity. It eliminates silent data loss by responding with an instant HTTP 412 Precondition Failed rejection, allowing LanceDB to safely increment the version and retry immediately.

Figure 1: Multiple writers attempt a conditional write (manifest version N). FlashBlade//S™ uses ETag to check for version N. One writer succeeds (200 OK) and commits N. Others fail (412 Precondition Failed), retry as N+1, and successfully commit (200 OK).

Let’s see it in action

We built a custom Rust benchmark tool so you can reproduce this yourself. Clone the repo and run it yourself.
Validation scenario: Launch 50 writers simultaneously, all racing to commit the same manifest version, to simulate a heavy commit race on a LanceDB table.

Test 1: Concurrent writes without conditional headers

• The result: All 50 writers received an HTTP 200 OK. From the client’s perspective, every single write succeeded.
• The reality: S3 defaulted to last-writer-wins. Only one writer’s manifest actually survived, while the other 49 were silently overwritten. Zero warnings, zero errors, zero indication that anything went wrong.

Test 2: Concurrent writes with If-None-Match: 

• The result: Exactly one writer received an HTTP 200 OK.
• The reality: The other 49 writers hit the backend ETag precondition check and received an immediate, explicit HTTP 412 Precondition Failed.

The impact: No silent overwrites. No data loss. The rejected clients can now safely catch the error and cleanly retry against the next manifest version.

2. Vector search at scale

Vector search is an extremely read-intensive I/O operation. Instead of streaming a few large objects, it drives a high rate of multiple S3 reads against many index fragments. That only works if the storage system can serve a lot of reads in parallel without letting latency drift.

Many S3 backends struggle here because they’re gateways in front of older block or file systems. Under heavy parallel GETs, the underlying engine pays a cost for extra translation work and locking, and response times start to climb.

Solution 

FlashBlade has an architectural advantage of using a distributed key-value design in the storage layer, which keeps S3 metadata operations fast and consistent even under heavy concurrency.

To test this, we scaled our OSS LanceDB deployment from one node to four nodes on top of FlashBlade over S3. LanceDB’s design cleanly separates compute from storage, so we could increase search capacity just by adding more stateless nodes while keeping the table on FlashBlade. As we added nodes, we saw near-linear QPS scaling and storage latency stayed within the millisecond to sub-millisecond range.

Figure 2: Near-linear vector search scale-out on FlashBlade//S.

Note: The benchmark used an IVF_PQ index with search parameters: k=10, nprobes=5, and refine_factor=1. Higher values for nprobes or k typically improve search recall but also increase S3 requests per query, which lowers the overall Queries per Second (QPS). Actual performance depends on the specific index configuration and adherence to FlashBlade best practices.

Results 

In practice, this makes FlashBlade a solid backend for LanceDB at production scale: You grow vector search capacity simply by adding more stateless nodes, while QPS scales up and storage latency stays low over S3. When you need more headroom than a single configuration can provide, you scale out further by spreading data across additional buckets or adding another FlashBlade chassis, without changing the application architecture.

Try it yourself

If you want to try this on your own infrastructure, we’ve published a small Rust benchmark tool. Follow the instructions in the LanceDB_VectorSearch repository to test LanceDB against your own S3 backend.

3. Storage footprint and I/O during LanceDB compaction

LanceDB’s append-only design is great for versioning and zero-copy reads, but it comes with a cost: fragment sprawl. Continuous writes leave behind hundreds of small files (fragments) that slowly hurt search performance, waste capacity, and slowly bloat the storage system.

Cleaning this up means running heavy compaction jobs that read and rewrite large amounts of data. On traditional storage, that I/O load can hammer the system, latency spikes, response times get bad, and storage bloat keeps growing.

Solution 

Compaction consolidates many small Lance fragments into a few well-organized files, keeping index builds fast and ensuring each query touches far fewer objects instead of hundreds of tiny ones. Everpure FlashBlade object storage is key here, easily absorbing the intense read/write bursts from optimize() and index creation without impacting live queries. 

With inline compression and deduplication, FlashBlade also slashes the storage footprint for blobs and metadata, lowering storage footprint without hurting performance.

We validated this on a 100.8-million-row Lance table stored on FlashBlade object storage via S3, running LanceDB compaction and index build and observing the impact on fragments and I/O.

• Fragments: 507 → 168 (67% reduction)
• Compaction time: ~5 minutes
• Write latency (I/O): ~4.3 ms 
• Read latency (I/O during compaction): ~2.3 ms

Figure 3: LanceDB compaction on FlashBlade//S.

Let’s see it in action

To validate this, we executed an end-to-end pipeline covering ingestion, compaction, index creation, and vector search using a pre-embedded data set on a single node. This performance analysis was conducted on a 10-million-row LanceDB table. The table contains both vector embeddings and metadata on FlashBlade object storage. The code below walks through each phase of that run. If you want to try the same workflow, clone the GitHub repo and follow the setup steps in the README.

Storage footprint analysis workloads

FlashBlade applies inline compression to everything in a Lance table, so you get performance and sensible storage usage on the same platform.

Final thoughts

LanceDB makes it clear where storage can derail modern AI pipelines: concurrent writers racing the same manifest, vector search fanning out into random S3 reads, and compaction and indexing pushing heavy I/O in the background. In our testing, FlashBlade held up across all three, enforcing atomic manifest commits with conditional S3 writes, sustaining high-concurrency vector search as we added LanceDB nodes, and running compaction and index build without getting in the way of live ingest or queries.

If you’re running RAG and vector search on LanceDB, what you need from storage is simple: no silent data loss, predictable latency under load, and the ability to keep tables tidy without taking the system offline. FlashBlade delivers that foundation while also keeping storage usage in check through efficient compression and layout.