Orchestrating web_search, file_search, and computer_use from the Responses API within a single streaming loop

When you try to attach an AI agent to a backend service, the first wall you hit is implementing the loop yourself — tool call → collect results → call again. Anyone who has done this with the Chat Completions API knows what that means: parsing tool_calls, reassembling result messages, handling streaming and tool calls simultaneously… it gets messier than you'd expect. I once spent half a day tracking down a bug where JSON arguments were being cut at chunk boundaries mid-stream.

The Responses API, released in March 2025, is an attempt to absorb this loop as a first-class concept at the API level. It ships with three built-in tools: web_search, file_search, and computer_use. This article has one question to answer: When building a backend agent that needs to work with all three tools together, how do you actually register and coordinate them, and where do you need to intervene?

As of September 2026, OpenAI recommends the Responses API as the default starting point for new projects instead of the Assistants API; for the official Assistants API deprecation schedule, check the individual announcement. Keep an eye on the OpenAI Deprecations page for the latest status.

From Chat Completions to Responses API: What Changed

The Agent Loop Has Moved

In the old Chat Completions flow, developers had to run the loop themselves whenever a tool call occurred. The model would return tool_calls, you'd execute them, inject the results as role: "tool" messages, and call the API again. The Responses API moves much of this cycle inside the API itself.

web_search and file_search are hosted tools that run inside OpenAI's infrastructure. When the model calls them, result injection is handled automatically. computer_use, on the other hand, requires the actual environment (browser/OS) to be on the developer's side, so it still needs a manual loop: receive the action, execute it, and feed the result back.

State Management Moves Server-Side Too

Pass previous_response_id and the server picks up the context from the previous response. You no longer need to send the full history from the client on every request, which reduces the state management burden when building multi-turn agents.

The Role and Behavior of Each Tool

web_search — Real-Time Grounding

Use this when you need information beyond the training data cutoff or real-time data. You can control search depth with search_context_size, and cost is charged separately per query. The tool type string may vary by release — web_search_preview, web_search, etc. — so check the Web search docs for the currently supported type name. Here's an example registering it in low-cost mode:

tools = [
    {
        "type": "web_search_preview",
        "search_context_size": "low",
    }
]

file_search — Internal Knowledge Base RAG

Pre-upload files into a Vector Store and the API combines semantic and keyword search to automatically inject relevant chunks. Parsing, chunking, and embedding are handled as a managed service at upload time. Metrics like file count, storage limits, and pricing change over time, so refer directly to the File search docs for current values.

tools = [
    {
        "type": "file_search",
        "vector_store_ids": ["vs_abc123"],
    }
]

computer_use — Screenshot-Based UI Automation

Use this for automating legacy web apps or controlling GUI-based internal systems. This tool runs on a dedicated model (e.g., computer-use-preview) and may not work as-is with a general gpt-4o, so check the Computer use docs for currently available model IDs.

Because the API conversation state and the actual browser state are separate, the responsibility for synchronizing the two states falls entirely on the developer. If you downscale screenshots, you also have to handle coordinate transformation yourself.

Fitting All Three Tools into One Streaming Loop

This is the core of the article. Hosted tools (web_search, file_search) and client tools (computer_use) have different natures, but you can integrate them by registering all three in a single request, consuming the response stream, and handling only the client tool separately.

Registering All Three

from openai import OpenAI
 
client = OpenAI()
 
TOOLS = [
    {"type": "web_search_preview", "search_context_size": "low"},
    {"type": "file_search", "vector_store_ids": ["vs_internal_kb"]},
    {
        "type": "computer_use_preview",
        "display_width": 1280,
        "display_height": 800,
        "environment": "browser",
    },
]
 
SYSTEM = """
When answering, follow this order:
1) First use file_search to check the internal KB.
2) Only use web_search if the internal docs don't have it or you need recent information.
3) Only use computer_use for tasks that require UI interaction.
""".strip()

There's a reason to state priorities explicitly in the system prompt. When all three tools are registered together, the model tends to pick web search first — meaning paid web search queries fire even for questions that could be answered from internal documents, leaking cost. Explicit priorities reduce this bias.

The Event Consumption Loop

Responses API streaming pushes SSE events with typed schemas. Unlike the raw delta parsing of Chat Completions, the event schema is well-defined, making handling clearer. However, which fields each event type carries may change over time, so cross-reference the attributes accessed in the code below against the Responses streaming events reference. Treat event names and fields as illustrative examples.

def run_turn(user_input, previous_response_id=None, screenshot_b64=None):
    input_items = [
        {
            "role": "user",
            "content": [{"type": "input_text", "text": user_input}],
        }
    ]
    if screenshot_b64:
        input_items[0]["content"].append({
            "type": "input_image",
            "image_url": f"data:image/png;base64,{screenshot_b64}",
        })
 
    stream = client.responses.create(
        model="computer-use-preview",
        instructions=SYSTEM,
        tools=TOOLS,
        input=input_items,
        previous_response_id=previous_response_id,
        stream=True,
    )
 
    pending_computer_calls = []
    final_response = None
 
    for event in stream:
        et = event.type
 
        if et == "response.output_text.delta":
            print(event.delta, end="", flush=True)
 
        elif et.startswith("response.web_search_call."):
            print(f"\n[web_search event: {et}]")
 
        elif et.startswith("response.file_search_call."):
            print(f"\n[file_search event: {et}]")
 
        elif et == "response.output_item.done":
            item = event.item
            if getattr(item, "type", None) == "computer_call":
                pending_computer_calls.append(item)
 
        elif et == "response.completed":
            final_response = event.response
 
    return final_response, pending_computer_calls

Two key points. First, progress events for web_search and file_search are only used to display a "searching" badge in the UI. Check the reference for the actual field names in the event payload (including whether the query string is exposed) before accessing them — accessing wrong attributes causes runtime errors. Second, computer_call is queued for separate processing after the stream ends.

Chaining computer_calls in a Follow-Up Loop

After executing the collected computer_call items, attach the results using the Responses API's computer_call_output schema and request the next response. This is a different format from Chat Completions' role: "tool" + tool_call_id pattern.

def capture_screenshot() -> str:  # conceptual example
    ...
 
def execute_action(action) -> None:  # conceptual example
    ...
 
def drive_agent(user_input):
    previous_id = None
    screenshot = None
    prompt = user_input
 
    while True:
        response, computer_calls = run_turn(prompt, previous_id, screenshot)
        previous_id = response.id
 
        if not computer_calls:
            return response
 
        follow_up_items = []
        for call in computer_calls:
            execute_action(call.action)
            new_shot = capture_screenshot()
            follow_up_items.append({
                "type": "computer_call_output",
                "call_id": call.call_id,
                "output": {
                    "type": "input_image",
                    "image_url": f"data:image/png;base64,{new_shot}",
                },
            })
 
        stream = client.responses.create(
            model="computer-use-preview",
            tools=TOOLS,
            input=follow_up_items,
            previous_response_id=previous_id,
            stream=True,
        )
        prompt = ""
        screenshot = None

A few things to watch out for:

• Field names are computer_call_output / call_id. Using role: "tool" · tool_call_id from Chat Completions muscle memory will conflict with the Responses API input schema.
• call_id is the reference value exposed by the computer_call item. Don't confuse it with the item's own id. It's safest to verify the exact field against the response object schema.
• Keeping stream=True for follow-up requests lets you handle text deltas and the next action in the same consumption loop, maintaining consistency.

Decision Flow for Tool Coordination

Common Pitfalls in Streaming

Function Argument Deltas Are Cut at Chunk Boundaries

If you register custom function tools alongside the built-ins, argument JSON arrives across multiple delta events. The event names vary by release (e.g., response.function_call_arguments.delta / .done), but the principle is the same. Don't parse deltas as they arrive — accumulate them until the done event and then process. Check the streaming events reference for the exact event names and reflect them in your code.

import json
 
buffers = {}
 
for event in stream:
    if event.type.endswith(".function_call_arguments.delta"):
        buffers.setdefault(event.call_id, "")
        buffers[event.call_id] += event.delta
    elif event.type.endswith(".function_call_arguments.done"):
        args = json.loads(buffers.pop(event.call_id))

file_search Chunking Cannot Be Controlled Directly

It's not uncommon to see tables in PDFs or information spanning page boundaries get retrieved incorrectly. Because you can't fine-tune the chunking strategy, it's more reliable to pre-process structured data into plain text before putting it in the store, or to build a separate summary index.

web_search Bias and Cost

When all three tools are registered together, the model frequently defaults to web search. To enforce internal document use, state priorities explicitly in the system prompt and start with search_context_size set to low, bumping it up only when needed.

computer_use State Synchronization

When browser state and API state fall out of sync, the model can get stuck replaying coordinates based on a stale screenshot. Always capture a fresh screenshot after each action, and explicitly manage the coordinate transformation matrix when downscaling. For long-running loops where retry, timeout, and resume logic intersect, combining with a workflow orchestrator like Temporal is a pattern seen in production.

Use the Agents SDK or Roll Your Own?

OpenAI's official open-source openai-agents-python abstracts multi-agent workflows, handoffs, and guardrails on top of the Responses API. It delivers streaming events as typed objects, so you don't need to parse SSE directly.

For a single agent with a single streaming loop, the openai Python package alone is sufficient without the SDK. If multiple agents hand off work to each other, or if you need guardrails and tracing, adopting the SDK makes more sense. For long-running loops like computer_use where failures and resumes are frequent, the SDK alone isn't enough — consider pairing it with a workflow orchestrator in front.

Summary

The Responses API delivers two concrete benefits: the execution loop for hosted tools moves into the API, and streaming events are organized into a typed schema. Up to this point, the code gets cleaner. But tools like computer_use — where the environment lives on the developer's side — still require you to write the loop on the client, and that's where error-prone concerns pile up: state synchronization, field names, and re-request schema.

If you're building a backend agent that handles all three tools together, here's the approach I'd recommend: register all three tools in one request, pin usage priorities in the system prompt, and in the streaming event consumption loop, queue only computer_call items for separate handling. Event names and fields change over time, so before hardcoding anything, cross-reference the streaming events reference and the Computer use docs for the current schema.

References

• OpenAI — New tools for building agents
• OpenAI — New tools and features in the Responses API
• OpenAI Official Docs — Using tools
• OpenAI Official Docs — Web search
• OpenAI Official Docs — File search
• OpenAI Official Docs — Computer use
• OpenAI Official Docs — Streaming API responses
• OpenAI Official Docs — Responses streaming events
• OpenAI Deprecations page
• OpenAI Agents SDK — Streaming
• GitHub — openai/openai-agents-python
• Microsoft Learn — Azure OpenAI Responses API