AI Agent Framework Security: A Review Guide for 10 Frameworks

Installing a framework and registering tools does not, by itself, document the security boundaries of the resulting deployment.

This article sets out questions for evaluating agent-framework security. It does not report substantiated UniverseBlend execution results. Assess sandboxing, network access, credential scope, approval requirements and logging for the exact framework version and deployment configuration under review.

The guide is a review framework, not a ranking or a report of installed-framework tests.

Key Takeaways

• Evaluate the deployed system and its exact configuration.
• Record execution boundaries, network access and credential scope.
• Check approval requirements and logging destinations explicitly.
• Distinguish documented behaviour from observed behaviour.
• No framework installation, container probe or execution result is reported here.

Quick Navigation

---

The Configuration You Never See

Treat configuration as part of the security review. A documented capability does not establish whether that capability is enabled in a particular deployment.

How to Assess Agent Framework Security

A reproducible assessment should record the framework version, runtime, configuration, test commands and observed outputs. Distinguish documented behaviour from directly observed behaviour, and mark anything not established by either as unverified.

The definitions used throughout:

• Logging — whether tool calls are recorded somewhere without the developer configuring an exporter, and what those records contain.
• Default — the behaviour of the library as installed, with no keyword arguments, no environment variables and no configuration file. If you must pass an argument to get a control, it is not a default.
• Sandbox — a boundary that prevents tool or model-generated code from reaching the host’s filesystem, environment and network. Restricting what the model may write is a weaker property than isolating where it runs; we separate the two.
• Egress — outbound network access from the process executing tools. A framework-level egress control would be a deny list or allowlist the framework itself enforces.
• Credential scope — which secrets a given tool can read. Full isolation would mean a tool receives only the credentials it was granted and cannot read the rest of the process environment.
• Human approval — a framework mechanism that pauses execution before a tool call and requires an external decision to continue.

---

Which Frameworks Made the List, and Why

The guide covers LangChain / LangGraph, LlamaIndex, CrewAI, OpenAI Agents SDK, Google ADK, Microsoft Agent Framework, PydanticAI, smolagents, Strands Agents and Haystack. Their repository links are retained in Sources. Inclusion is not a test result or a security endorsement.

Recording the Assessment

For each framework, record the exact version, execution boundary, network policy, credential scope, approval requirements and logging configuration. Attach documentation or observed output to each assessment. No version-tested or execution-result table is presented here.

What the Security Review Should Examine

Use these five review areas to identify what is established and what still needs testing.

Execution boundaries

Where do tools execute, and which filesystem, process and user boundaries apply?

Approval requirements

Which actions require approval in the actual configuration, and how is the pause enforced?

Logging configuration

Which events and payloads are recorded, where are they sent, and what retention applies?

Network access

Which destinations can tools reach, and which layer enforces the network policy? No socket or proxy test result is reported here.

Code-execution restrictions

Distinguish restrictions on model-generated code from isolation of the process and developer-written tools. Establish each property from documentation and reproducible tests.

Applying the Review in Production

• For an API tool, identify which credentials are available and how access is scoped.
• For a coding agent, identify the execution environment and reachable files and services.
• For customer information, establish whether traces include sensitive content and where it is sent.
• For consequential actions, verify the configured approval path before deployment.

AI Agent Framework Security vs Runtime Security

Assess framework controls and runtime controls separately.

A framework is not a runtime. The framework decides what the agent is asked to do and in what order. The runtime decides what the resulting code is permitted to touch. They are different layers with different enforcement powers, and a Python decorator sits firmly in the first.

Real isolation in an agent system comes from:

• Containers and VMs — process, filesystem and user boundaries
• Sandboxed execution services — separate machines for model-written code
• OS permissions — the user the agent runs as, and what that user owns
• Network policies — egress rules at the cluster, VPC or proxy layer
• Secret managers — short-lived, per-tool credentials injected at call time
• Identity systems — the agent as a principal with its own scoped grants
• Observability platforms — where the traces land and how long they persist

An absence of evidence about a framework control does not establish the security of the application. Document the deployment’s runtime, permissions, network policy and credential configuration separately.

The framework layer is simply not where this question gets answered. Our companion piece on sandbox isolation layers covers the runtime side in detail.

---

What Changes When You Configure the Framework

• Consult the exact version’s documentation for supported approval mechanisms.
• Choose and verify an appropriate execution boundary for tools and generated code.
• Scope credentials to the tools that need them.
• Configure logging and decide whether records may contain sensitive payloads.

The 50-Input Reality Check

The following is a proposed evaluation exercise, not a report of a completed test.

A benchmark score tells you how often a model produced an acceptable answer on a curated set. It tells you nothing about what the system did to get there — how many tool calls, against which endpoints, carrying which credentials, with what recovery behaviour when a tool timed out.

Run fifty representative inputs through your actual agent, with your actual tools, and record the tool-call trace for each. You will learn four things a leaderboard cannot give you: the cost distribution, the latency tail, the failure shape rather than the failure rate, and the real set of external systems your agent touches when it is confused.

Preserve the traces and configuration if you perform this exercise. Related reading: why benchmark scores fail · the four signals an agent stack must emit. No result is asserted here.

---

What Security Teams Should Ask Before Approving an Agent Framework

Ten questions. They are deliberately about the deployed system, not the library.

1. What can the agent execute? Is there a code path, and if so, where does that code run — this process, a container, or a remote sandbox?
2. What can it access? Which filesystem paths and which internal services are reachable from the process, regardless of what the tools claim to do?
3. Where can it connect? Is there an egress policy, and is it enforced at the network layer rather than by tool descriptions?
4. Which credentials can it see? Record the credentials exposed by the execution environment and verify which tools can access them.
5. Which actions require approval? Name the specific tools that pause, and show the flag or middleware that makes them pause.
6. What gets logged? Tool names only, or arguments and outputs too?
7. Where do logs go? Your collector, or a vendor endpoint — and did anyone choose that deliberately?
8. How long are traces retained, and do they contain regulated data?
9. What happens when a tool fails? Does the agent retry, escalate, or improvise a different tool?
10. What happens when the model behaves unexpectedly? What is the worst single action available to it right now, and who would see it happen?

Use the answers as a deployment review record; a framework name alone is not evidence that a control is effective.

---

Sources

Framework repositories for version-specific documentation and further review:

• LangChain — https://github.com/langchain-ai/langchain
• LangGraph — https://github.com/langchain-ai/langgraph
• LlamaIndex — https://github.com/run-llama/llama_index
• CrewAI — https://github.com/crewAIInc/crewAI
• OpenAI Agents SDK — https://github.com/openai/openai-agents-python
• Google ADK (Python) — https://github.com/google/adk-python
• Microsoft Agent Framework — https://github.com/microsoft/agent-framework
• PydanticAI — https://github.com/pydantic/pydantic-ai
• smolagents — https://github.com/huggingface/smolagents
• Strands Agents SDK — https://github.com/strands-agents/sdk-python
• Haystack — https://github.com/deepset-ai/haystack

Standards and security research:

• OWASP Top 10 for LLM Applications — https://owasp.org/www-project-top-10-for-large-language-model-applications/
• NIST AI Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework

Referenced in the article body:

• LangChain Experimental sunset notice — https://github.com/langchain-ai/langchain-experimental/issues/87

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.