Making LLM Agents Think Once, Then Act Fast

When we first began building the LLM-powered Everpure Storage Copilot agent to analyze infrastructure telemetry, the initial results were incredibly promising. In early demos, the agent behaved just like a smart operator: You could ask a question, watch it reason through the problem, observe its tool calls, and receive a clear answer. The experience felt both intuitive and powerful until we connected it to real production data. When we were no longer dealing with carefully curated samples or toy data sets, but rather over 40 petabytes of live telemetry, featuring individual tables that exceeded 250 billion rows—operating with real customer environments—the stakes completely shifted. That was the exact moment our architecture stopped being theoretical and collided with reality.

The obvious first idea (and why it failed)

The most straightforward approach is often the most tempting: simply let the large language model (LLM) generate SQL and query the database directly. In a sandbox environment, this method works frighteningly well, as the model quickly internalizes schemas, navigates complex joins, and produces perfectly reasonable queries. However, the moment this approach is introduced to an enterprise setting, it completely breaks down. Direct SQL generation introduces a host of security and operational risks that are fundamentally incompatible with risk-averse customers. Even if you restrict the model to read-only access, you still have to contend with potential injection vulnerabilities, accidental cross-tenant data access, schema leakage, and the severe performance impact of unbounded scans over massive tables. Furthermore, this approach leaves you without a defensible audit story to explain exactly why a specific query was executed. When customers trust you with their most sensitive infrastructure data, hoping “the model probably won’t do that” is never an acceptable security posture—so we knew we had to move on.

A safer compromise: Parameterized tools and agent loops

Our next iteration was far more conservative and, crucially, much more defensible. Instead of allowing free-form SQL generation, we defined a fixed set of parameterized queries and exposed them to the model as tools. The LLM could still choose which tool to call and supply the necessary arguments, but the underlying queries remained tightly controlled on our end. This approach fit naturally into the standard agentic loop that most frameworks encourage: pick a tool, wait for the output, read it, filter it, pick the next tool, and repeat. While this was a massive improvement from a security standpoint, from a systems perspective, it quietly introduced a whole new set of problems.

When agent loops meet real scale

As our usage grew and we pointed this loop-based system at actual production telemetry, three structural issues became entirely impossible to ignore:

• Serial execution and latency: Because every single step required a full round trip between the model and the system—model call, tool call, model call, tool call—latency skyrocketed. A single user question routinely took over 60 seconds to answer, and that was when everything was behaving exactly as intended.
• Context window collapse: Our tools frequently returned large dataframes, and those raw results were fed straight back into the model so it could decide on its next action. Unsurprisingly, context windows rapidly filled with raw data, forced truncation kicked in, and the model’s accuracy severely degraded.
• The model doing the wrong work: We soon realized that most of the model’s effort wasn’t actually spent on reasoning; it was being wasted on filtering, sorting, grouping, and reshaping data. We had inadvertently turned a highly sophisticated LLM into an incredibly expensive and fragile data processing engine. Acknowledging that tension led to our first real shift in thinking.

Our 2 big epiphanies

Epiphany #1: The model should plan, not process

LLMs are exceptional at understanding language, interpreting intent, and executing high-level reasoning, but they’re remarkably inefficient and error-prone when it comes to step-by-step data manipulation through APIs. Tools like ChatGPT feel so powerful precisely because the heavy lifting happens outside the model; search, filtering, ranking, and aggregation are handled by purpose-built backend systems, leaving the model to simply reason over reduced, cleanly structured results. We were asking our model to do computational grunt work that should never have entered its context window in the first place. So, we completely reframed the problem: What if the model just described what needed to happen, but never actually touched the data directly?

The tempting detour: Letting the model write Python

At this stage, allowing the model to generate Python code seemed like the next logical answer. Python is highly expressive, universally familiar, and perfectly suited for data manipulation. In practice, however, this idea collapses under strict enterprise constraints. Python is notoriously difficult to sandbox completely; it inherently has side effects, supports arbitrary imports, and exposes a massive attack surface. Worse still, validating Python code often requires actually executing it—which is exactly what you’re trying to avoid when dealing with untrusted, AI-generated code. If you restrict Python enough to make it genuinely safe, it quickly becomes unusable; if you leave it unrestricted, you simply cannot ship it to enterprise customers. We realized we needed a solution that was deliberately limited.

Epiphany #2: jq as a controlled transformation layer

The real breakthrough came when we recognized that we didn’t actually need a general-purpose programming language at all. What we needed was a pure, declarative way to reshape data. We realized that if we could safely execute parameterized SQL, convert those results into JSON, and allow the model to generate jq expressions to query that JSON, we could grant the model expressive power without handing it dangerous capabilities. jq turned out to be a surprisingly perfect fit: It has no file or network access, zero side effects, and is incredibly easy to statically analyze and restrict. Yet, it remains expressive enough to filter, group, aggregate, and reshape deeply nested data structures. Most importantly, jq introduced a vital layer of indirection. The model no longer interacted with databases or raw telemetry directly; instead, it merely described the required transformations. This meant large data sets stayed out of the model’s context window entirely. It’s rare in large systems for a single design choice to simultaneously improve security, performance, and accuracy, but this one did exactly that.

Our dataflow pattern

From reactive loops to executable plans

Once we had jq acting as our safe data-glue, a much larger architectural shift became possible. Instead of letting the model reactively stumble through a problem step by step, we asked it to generate a complete, end-to-end workflow plan up front. This plan is a declarative, schema-validated JSON document that explicitly specifies which tools to run, how the data should flow between them, and exactly which jq transformations apply at each step. The model does its heavy reasoning exactly once, and then our runtime executes the resulting plan asynchronously and efficiently.

The architecture: Plans, steps, and dataflow

A generated workflow plan looks roughly like this:

In this architecture, each step has a stable ID and explicitly declared dependencies. The $ref syntax elegantly connects outputs between steps, while the jqselectors seamlessly reshape or filter those JSON outputs before they feed into downstream tools. This structured approach keeps our workflows incredibly compact, fully deterministic, and remarkably easy to validate, even when the underlying data sets are staggeringly large.

jq as the ‘just right’ transform layer

You can essentially think of jq as a small, declarative language purpose-built for JSON. It supports streams, filters, maps, and reductions, all without the dangerous overhead of full programming power. It works incredibly well in practice for four main reasons:

• Safe: It has absolutely no I/O and is trivial to sandbox.
• Deterministic: Its pure transformations are entirely cacheable and reproducible.
• Declarative: The data flows read intuitively from left to right, much like standard Unix pipelines.
• LLM-friendly: The concise syntax generates tiny snippets, completely avoiding token bloat.

We found that a few common patterns covered the vast majority of our orchestration needs:

Crucially, through all of these transformations, the large underlying data sets never re-enter the model’s context window.

Our core learnings along the way

Plan once, execute smartly

Transitioning to this architecture also unlocked another highly impactful optimization. Because the workload is split, we now use a larger, more capable (and more expensive) reasoning model to generate the complex workflow plan up front. Then, we use a smaller, cheaper, and faster model to simply summarize the results and produce the final user-facing response. While workflow planning heavily benefits from deep reasoning capabilities, basic summarization and explanation do not require anywhere near the same compute capacity. Decoupling these two steps drastically reduced our operational costs while simultaneously improving overall response quality.

From an LLM-bound problem to a systems problem

Before we implemented this design, the burden of scale was entirely trapped inside the model itself. Feeding it too much data guaranteed broken context windows, endless hallucinations, and heavily degraded accuracy. With declarative plans and jq-based execution, the burden of scale finally moved back to the infrastructure layer, exactly where it belongs. Now, if a workflow needs to process substantially more data, we simply allocate more CPU, memory, and time to the runtime environment. Yes, scaling infrastructure costs money, but it’s a highly predictable and entirely controllable expense. That is a far healthier, more sustainable place to be when you’re managing tens of petabytes of sensitive telemetry.

Tradeoffs and intentional limitations

Of course, this approach is not a universal silver bullet. It heavily favors linear, data-oriented workflows over processes that require complex, highly dynamic branching logic. jq is explicitly not a general programming language, and that limitation is entirely intentional; it forces the model to think harder upfront to produce a robust, linear plan. If your use case requires highly dynamic, constantly shifting control flow, traditional graph-based orchestration frameworks are likely a better fit. But for secure, large-scale data access in enterprise environments, these declarative workflows win every time.

Let the model think once

Ultimately, the most important lesson we learned wasn’t about the specific syntax of jq or the structure of JSON. It was about fundamentally respecting the boundary between reasoning and execution. LLMs do not need to—and should not—carry petabytes of telemetry within their fragile context windows. Instead, they need to reason exactly once, articulate a clear plan, and securely delegate the structured execution to systems built to handle massive scale. By enforcing that boundary, we successfully transformed slow, chatty, and unreliable agent loops into lightning-fast, highly auditable workflows that actually survive contact with enterprise scale. We didn’t achieve this by making the model intrinsically smarter. We achieved it by finally letting the model do exactly what it is good at.