AI Agent Framework Security: 10 Frameworks Audited

A developer runs pip install, decorates three functions with @tool, and points an agent at them. The agent now has a shell in your process. Nothing in that sequence asked a security question, and nothing answered one.

That gap is worth measuring rather than assuming. So on 16 September 2026 we installed ten widely used agent frameworks into clean, isolated Python 3.12.3 virtual environments, took the default install with no extras, and ran a probe tool through each framework’s own tool-invocation path to see what it could reach.

This article reports what we found. It is not a ranking, and no framework here is being called insecure. The finding is narrower and more useful than that: AI agent framework security, at the default layer, is mostly an empty set — deliberately so.

Key Takeaways

• No framework in this audit sandboxes tool execution by default. In every case we tested, the tool ran inside the parent Python process, with the same PID.
• Credential isolation is not a default feature anywhere. Our probe tool read a process environment variable in all ten frameworks. None of them provide a mechanism that would have stopped it out of the box.
• Network egress is not a framework-layer control. Our probe opened an outbound TCP connection from inside the tool in every framework. Where connections failed, the container’s proxy allowlist stopped them, not the framework.
• Human approval exists as a first-class API almost everywhere, and is switched off almost everywhere.
  • needs_approval=False, require_confirmation=False, requires_approval=False, approval_mode="never_require", human_input=False — four different vendors, one default.
• Tool-call logging is off by default in eight of ten. The OpenAI Agents SDK is the exception: tracing_disabled is False and trace_include_sensitive_data resolves to True. CrewAI initializes anonymous telemetry unless an opt-out variable is set.
• smolagents is the only framework that restricts model-written code by default, and that restriction does not extend to your own tools.
• CrewAI removed its code interpreter. In 1.15.21, allow_code_execution is deprecated and the field text points developers to external sandbox services.

---

Quick Navigation

---

The Configuration You Never See

Most security conversations about agents start at the model. Can it be jailbroken, will it leak the system prompt, does it resist injection. Those are real questions and they are well covered.

The question underneath them gets asked far less often. When the model does something unexpected, what is the blast radius — and who decided that radius?

The honest answer is usually: nobody decided. It was inherited. The framework shipped a default, the developer never changed it, and the boundary of the agent’s authority ended up being whatever the Python process happened to have.

That is what this audit measures.

---

How This AI Agent Framework Security Audit Was Run

Method first, because an AI agent framework security claim is only as good as the thing that produced it.

Each framework was installed with a plain pip install <package> into its own fresh virtual environment on Python 3.12.3 in a Linux container, with no extras and no configuration files. No API keys were set, so no agent loop was driven by a live model. Instead we built a probe tool using each framework’s own tool decorator or tool class and invoked it through that framework’s own invocation path.

The probe returned three things: its process ID, the value of an environment variable set in the parent process, and the result of opening a TCP connection to an external host.

That design removes the model from the experiment on purpose. We are not testing whether an LLM decides to misbehave. We are testing what the framework permits when it does.

Everything below is labelled executed (we ran it and recorded the output) or source-verified (we read the installed source or resolved a default at runtime without exercising the full agent loop). Nothing here is inferred from marketing pages.

The definitions used throughout:

• Logging — whether tool calls are recorded somewhere without the developer configuring an exporter, and what those records contain.
• Default — the behaviour of the library as installed, with no keyword arguments, no environment variables and no configuration file. If you must pass an argument to get a control, it is not a default.
• Sandbox — a boundary that prevents tool or model-generated code from reaching the host’s filesystem, environment and network. Restricting what the model may write is a weaker property than isolating where it runs; we separate the two.
• Egress — outbound network access from the process executing tools. A framework-level egress control would be a deny list or allowlist the framework itself enforces.
• Credential scope — which secrets a given tool can read. Full isolation would mean a tool receives only the credentials it was granted and cannot read the rest of the process environment.
• Human approval — a framework mechanism that pauses execution before a tool call and requires an external decision to continue.

---

Which Frameworks Made the List, and Why

Ten frameworks, selected for active maintenance, real adoption and comparable scope. Three notable names were deliberately excluded.

AutoGen and Semantic Kernel are both in maintenance mode. Microsoft positions Microsoft Agent Framework as the direct successor to both, and it reached 1.0 on 3 April 2026, so Agent Framework 1.18.0 stands in for that lineage. AG2, the community fork of AutoGen maintained outside Microsoft, is a separate codebase and needs its own audit rather than inheriting AutoGen’s.

Two included projects are not strictly equivalent to the rest. LlamaIndex and Haystack both began as retrieval frameworks and grew agent layers. Their tool-calling surface is comparable, but neither ships a code-execution path — an asymmetry that is a finding rather than a gap in the test.

---

AI Agent Framework Security Defaults: The Table

All versions installed and tested on 16 September 2026. “In-process” means the tool executed inside the parent Python interpreter with the same PID.

Framework	Version tested	Sandbox by default	Network egress	Credential scope	Human approval	Tool-call logging	Evidence
LangChain / LangGraph	langchain 1.4.1, core 1.6.3, langgraph 1.2.11	None — in-process	Not controlled by framework	No isolation; env readable	Opt-in (middleware=()) ¹	Off (tracing_is_enabled() → False)	Executed
LlamaIndex	llama-index-core 0.14.24	N/A — no code execution shipped	Not controlled by framework	No isolation; env readable	Opt-in (InputRequiredEvent)	Off (Null handlers) ²	Executed
CrewAI	crewai 1.15.21	Code interpreter removed ³	Not controlled by framework	No isolation; env readable	Opt-in (human_input=False)	Anonymous telemetry on ⁴	Executed
OpenAI Agents SDK	openai-agents 0.22.2	None — in-process	Not controlled by framework	No isolation; env readable	Opt-in (needs_approval=False)	On by default ⁵	Executed
Google ADK	google-adk 2.9.1	Optional; code_executor=None ⁶	Not controlled by framework	No isolation	Opt-in (require_confirmation=False)	Off unless OTLP endpoint set	Source-verified
Microsoft Agent Framework	agent-framework 1.18.0	None — in-process	Not controlled by framework	No isolation; env readable	Opt-in (approval_mode="never_require")	Metadata only; no exporter ⁷	Executed
PydanticAI	pydantic-ai-slim 2.43.0	None — in-process	Not controlled by framework	No isolation; env readable	Opt-in (requires_approval=False)	Off (Logfire not installed)	Executed
smolagents	smolagents 1.26.0	Restricted interpreter ⁸	Not controlled by framework	Env readable from tools ⁹	Not a framework default	Off (no OTel installed)	Executed
Strands Agents	strands-agents 1.56.0	sandbox=None ¹⁰	Not controlled by framework	No isolation; env readable	Opt-in (interventions=None)	Off unless OTLP endpoint set	Executed
Haystack	haystack-ai 3.1.1	N/A — no code execution shipped	Not controlled by framework	No isolation; env readable	Opt-in (hooks)	Off; content tracing False ¹¹	Executed

Footnotes

• is_content_tracing_enabled resolves False on the default ProxyTracer.
• create_agent() resolves middleware=() and checkpointer=None. HumanInTheLoopMiddleware exists and must be added explicitly.
• The instrumentation dispatcher loads with NullSpanHandler and NullEventHandler.
• In 1.15.21 both allow_code_execution and code_execution_mode are marked deprecated, with field text stating the code interpreter is no longer available and directing users to dedicated sandbox services.
• Telemetry initializes unless OTEL_SDK_DISABLED, CREWAI_DISABLE_TELEMETRY or CREWAI_DISABLE_TRACKING is set. The source restricts what is recorded and comments explicitly that environment values are never captured.
• A default RunConfig() resolves tracing_disabled=False and trace_include_sensitive_data=True, exporting to https://api.openai.com/v1/traces/ingest. Separately, the SDK’s local debug flags default to not writing model and tool payloads to the Python logger.
• LlmAgent.code_executor defaults to None. ADK ships container, GKE, Vertex AI and Agent Engine executors alongside an explicitly named UnsafeLocalCodeExecutor.
• OpenTelemetry instrumentation defaults on but emits metadata only; message content requires ENABLE_SENSITIVE_DATA. The framework contacts no external system itself.
• CodeAgent(executor_type="local") runs a restricted AST interpreter. Our tests:
  • import os blocked,
  • __import__('os') blocked,
  • open('/etc/hostname') blocked,
  • import math allowed.
  • Eleven modules are permitted by default.
• The restriction applies to model-generated code, not to tools. Our @tool function read the environment variable normally.
• The Agent constructor exposes a Sandbox type, defaulting to None. checkpointing=False, load_tools_from_directory=False.

---

What the AI Agent Framework Security Results Show

Five patterns are worth pulling out of that table.

The process is the boundary, in all ten

Our probe returned the parent PID in every framework where it ran. The tool read the environment variable. It opened a socket. No framework interposed anything.

This is not an accusation; it is how Python libraries work. A decorator that wraps a callable cannot meaningfully restrict that callable. But it means the honest description of default AI agent framework security is that your agent’s authority equals your process’s authority — same OS user, same environment, same network position.

Approval is built, funded and off

The most consistent finding in the audit. Four vendors with different architectures converged on the same default:

• OpenAI Agents SDK — needs_approval=False
• Google ADK — require_confirmation=False
• PydanticAI — requires_approval=False
• Microsoft Agent Framework — approval_mode="never_require"

CrewAI’s Task.human_input is False. LangChain ships HumanInTheLoopMiddleware but resolves middleware=(). LlamaIndex and Strands expose interrupt and intervention types that do nothing until wired up.

The capability is mature. The default is not to use it. A security review that asks “does this framework support approval gates?” will get a reassuring yes and learn nothing.

Logging defaults split three ways

Most frameworks record nothing until you configure an exporter. Google ADK, Strands and Microsoft Agent Framework all wait for an OTLP endpoint. LlamaIndex loads null handlers. Haystack’s tracer reports content tracing disabled. LangSmith tracing resolves to False with no environment variables set.

Two break the pattern, in opposite directions.

1. The OpenAI Agents SDK enables tracing by default and resolves trace_include_sensitive_data to True, sending spans to OpenAI’s ingest endpoint. That is documented behaviour with a documented opt-out (OPENAI_AGENTS_DISABLE_TRACING), and for many teams it is a feature — but it moves prompt and tool data off your infrastructure, which deserves a decision rather than a discovery.
2. CrewAI initializes anonymous usage telemetry unless one of three opt-out variables is set. The source is careful about scope and comments that environment values are never recorded. It is still a default outbound connection.

Egress belongs to the runtime, and our own test proved it

Every framework let the probe open an outbound connection. Not one of them offers a framework-level allowlist.

The demonstration arrived by accident. Our test container runs behind a proxy with a domain allowlist, so connections to hosts outside that list fail — from inside every framework, identically. The control that actually governed egress during this audit was the container’s, not the library’s. That is the whole distinction in one observation.

smolagents restricts the model, not the process

smolagents is the one framework whose default does real work. CodeAgent uses a local Python interpreter that evaluates the model’s code against an import allowlist of eleven modules. Our tests confirmed import os, __import__('os') and open() all raise.

That is a genuine default-on control, aimed precisely at what goes wrong with code agents. It is also not an OS sandbox, the project does not claim it is, and it does not constrain your own tools — our @tool function read the environment without difficulty. Restricting what the model may write and isolating where it runs are different guarantees, and only the first is on by default here.

---

What These Defaults Mean in Production

Translated into the shapes these systems actually take:

• An agent calling an external API. The tool holds a key from the environment. So does every other tool in the process. Compromise one through a poisoned search result, and the credentials in scope are all of them.
• A coding agent executing commands. Outside smolagents, model-generated code runs with your process’s rights — working directory, environment, and any cloud instance metadata endpoint the host can reach.
• An agent reading files. No framework-level path restriction exists anywhere in this audit. open() resolves to open().
• An agent handling customer information. The question is not whether the framework logs, but where. With the OpenAI Agents SDK’s defaults, tool arguments and outputs travel in spans to an external endpoint. If that data is regulated, the default is a data-flow decision and belongs in your records of processing.
• An agent performing a financial workflow. Nothing pauses before the transfer unless you make it.

None of these are framework bugs. They are the predictable result of orchestration libraries doing orchestration and leaving isolation to the layer that can enforce it.

---

AI Agent Framework Security vs Runtime Security

This is the distinction that makes the table usable rather than alarming.

A framework is not a runtime. The framework decides what the agent is asked to do and in what order. The runtime decides what the resulting code is permitted to touch. They are different layers with different enforcement powers, and a Python decorator sits firmly in the first.

Real isolation in an agent system comes from:

• Containers and VMs — process, filesystem and user boundaries
• Sandboxed execution services — separate machines for model-written code
• OS permissions — the user the agent runs as, and what that user owns
• Network policies — egress rules at the cluster, VPC or proxy layer
• Secret managers — short-lived, per-tool credentials injected at call time
• Identity systems — the agent as a principal with its own scoped grants
• Observability platforms — where the traces land and how long they persist

So read every “None” in the table precisely. It means the framework does not automatically provide that control. It does not mean the application lacks it. A LangGraph agent in a locked-down container with a network policy and short-lived tokens is well isolated, and the framework contributed none of that. The same agent run as your laptop user with a .env file of production keys is not isolated at all, and the framework contributed nothing there either.

The framework layer is simply not where this question gets answered. Our companion piece on sandbox isolation layers covers the runtime side in detail.

---

What Changes When You Configure the Framework

Short version, because this is a reference point rather than a tutorial.

• Approval gates — flip the flag: needs_approval=True, require_confirmation=True, requires_approval=True, approval_mode, HumanInTheLoopMiddleware, human_input=True. Highest value per line of code in this list.
• Sandboxing — Google ADK ships container, GKE and Vertex executors you can assign to an agent. smolagents supports remote executors. Everyone else expects you to bring your own.
• Scoped credentials — pass secrets into tool constructors or dependency objects rather than reading os.environ inside the tool.
• Tracing — set an OTLP endpoint, or disable the default exporter where one exists. Decide where spans land before deciding whether they carry content.

None of this is hard. It is just not free, and it does not happen on pip install.

---

The 50-Input Reality Check

Here is where the audit connects to evaluation practice.

A benchmark score tells you how often a model produced an acceptable answer on a curated set. It tells you nothing about what the system did to get there — how many tool calls, against which endpoints, carrying which credentials, with what recovery behaviour when a tool timed out.

Run fifty representative inputs through your actual agent, with your actual tools, and record the tool-call trace for each. You will learn four things a leaderboard cannot give you: the cost distribution, the latency tail, the failure shape rather than the failure rate, and the real set of external systems your agent touches when it is confused.

That last one is the security artefact. It is also the one that most teams discover they cannot produce, because tracing was off — which is exactly what the logging column of this table predicts. We made the broader case for building your own evaluation set in why benchmark scores fail, and the telemetry requirements in the four signals an agent stack must emit.

---

What Security Teams Should Ask Before Approving an Agent Framework

Ten questions. They are deliberately about the deployed system, not the library.

1. What can the agent execute? Is there a code path, and if so, where does that code run — this process, a container, or a remote sandbox?
2. What can it access? Which filesystem paths and which internal services are reachable from the process, regardless of what the tools claim to do?
3. Where can it connect? Is there an egress policy, and is it enforced at the network layer rather than by tool descriptions?
4. Which credentials can it see? List what is in the process environment, then assume every tool can read all of it, because it can.
5. Which actions require approval? Name the specific tools that pause, and show the flag or middleware that makes them pause.
6. What gets logged? Tool names only, or arguments and outputs too?
7. Where do logs go? Your collector, or a vendor endpoint — and did anyone choose that deliberately?
8. How long are traces retained, and do they contain regulated data?
9. What happens when a tool fails? Does the agent retry, escalate, or improvise a different tool?
10. What happens when the model behaves unexpectedly? What is the worst single action available to it right now, and who would see it happen?

A framework choice answers almost none of these. That is the point of the table.

---

Sources

Framework repositories (primary sources for every default reported):

• LangChain — https://github.com/langchain-ai/langchain
• LangGraph — https://github.com/langchain-ai/langgraph
• LlamaIndex — https://github.com/run-llama/llama_index
• CrewAI — https://github.com/crewAIInc/crewAI
• OpenAI Agents SDK — https://github.com/openai/openai-agents-python
• Google ADK (Python) — https://github.com/google/adk-python
• Microsoft Agent Framework — https://github.com/microsoft/agent-framework
• PydanticAI — https://github.com/pydantic/pydantic-ai
• smolagents — https://github.com/huggingface/smolagents
• Strands Agents SDK — https://github.com/strands-agents/sdk-python
• Haystack — https://github.com/deepset-ai/haystack

Standards and security research:

• OWASP Top 10 for LLM Applications — https://owasp.org/www-project-top-10-for-large-language-model-applications/
• NIST AI Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework

Referenced in the article body:

• LangChain Experimental sunset notice (surfaced as a deprecation warning during our own install) — https://github.com/langchain-ai/langchain-experimental/issues/87

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.