mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-17 04:31:55 +00:00
Add sandbox pool support to HermesAgentBaseEnv
Added directly to HermesAgentBaseEnv (no subclass needed): Config fields: - tool_pool_mode: 'default' (terminal tool), 'nomad', or 'modal' - Full Nomad settings: nomad_address, sandbox_job_id, slots_per_container, etc. - Full Modal settings: modal_image, modal_gpu, modal_slots_per_sandbox, etc. - Shared: allow_network, require_sandbox, purge_job_on_start/shutdown Methods: - _start_sandbox_backend() / _stop_sandbox_backend() - lifecycle - setup_trajectory_workspace() - optional hook for workspace prep - verify_and_score_trajectory() - optional hook for in-sandbox verification - env_manager() / process_manager() - lifecycle cleanup When tool_pool_mode='default': everything works as before (terminal tool) When tool_pool_mode='nomad'/'modal': activates sandbox pool from atropos/backends/
This commit is contained in:
Shannon Sands
parent
507b77c4ac
commit
98d945f6de
2 changed files with 197 additions and 88 deletions
|
|
@ -1,99 +1,55 @@
|
|||
# Active Context
|
||||
|
||||
## Current Focus
|
||||
Consolidating the two Atropos environment systems and fixing tool calling to use proper OpenAI-spec approach instead of ICL.
|
||||
Adding sandbox pool support directly to `HermesAgentBaseEnv` so that `tool_pool_mode=modal/nomad` works alongside the default terminal-tool approach.
|
||||
|
||||
## PR Feedback from Lead Dev (Feb 10, 2026)
|
||||
## Implementation Plan (Feb 10, 2026)
|
||||
|
||||
The PR was rejected because our approach has three fundamental issues:
|
||||
|
||||
### Issue 1: ManagedServer doesn't pass `tools={}` to `apply_chat_template()`
|
||||
- When using Phase 2 (VLLM/SGLang for RL training), `ManagedServer` needs to pass tools to `tokenizer.apply_chat_template(tools=...)`
|
||||
- This makes the system prompt include tool definitions the way models were trained to expect
|
||||
- **Fix**: Atropos PR #366 adds `tool_call_parser` support to ManagedServer (branch: `tool_call_support`)
|
||||
|
||||
### Issue 2: ICL prompt vs proper tool calling
|
||||
- Our code embeds tools as XML in the system prompt (`<tools>...</tools>`)
|
||||
- Proper approach: pass `tools=` parameter in `chat_completion()` calls and let the tokenizer's chat template handle formatting
|
||||
- All Hermes datasets train on the proper format, not ICL
|
||||
|
||||
### Issue 3: Only Hermes `<tool_call>` parser, no multi-model support
|
||||
- Our code only handles Hermes-style `<tool_call>` XML parsing
|
||||
- Proper approach: parser registry supporting 11+ model families (hermes, qwen, deepseek, llama, mistral, etc.)
|
||||
|
||||
## Architecture: What Exists Now (Two Parallel Systems)
|
||||
|
||||
### `environments/` (Teknium's proper approach) ✅ CORRECT
|
||||
```
|
||||
environments/
|
||||
├── agent_loop.py ← Uses tools= in chat_completion() (OpenAI spec)
|
||||
├── hermes_base_env.py ← Phase 1 (OpenAI) + Phase 2 (ManagedServer + parser)
|
||||
├── tool_context.py ← ToolContext for reward functions
|
||||
├── tool_call_parsers/ ← 11 model parsers (hermes, qwen, deepseek, llama, etc.)
|
||||
│ ├── __init__.py ← Registry with get_parser(), register_parser()
|
||||
│ ├── hermes_parser.py
|
||||
│ ├── qwen_parser.py
|
||||
│ ├── deepseek_v3_parser.py
|
||||
│ ├── llama_parser.py
|
||||
│ ├── mistral_parser.py
|
||||
│ └── ... (11 total)
|
||||
├── terminal_test_env.py ← Working example: file creation tasks
|
||||
├── hermes_swe_env.py ← SWE environment
|
||||
└── patches.py ← Async-safe monkey patches
|
||||
### Goal
|
||||
The command should work:
|
||||
```bash
|
||||
python environments/swe_smith_oracle_env.py process \
|
||||
--env.tool_pool_mode modal \
|
||||
--env.modal_image python:3.11
|
||||
```
|
||||
|
||||
**How it works correctly:**
|
||||
1. `HermesAgentLoop.run()` passes `tools=self.tool_schemas` to `chat_completion()`
|
||||
2. ManagedServer passes tools to `tokenizer.apply_chat_template(tools=...)`
|
||||
3. Parser registry reconstructs `tool_calls` from raw model output
|
||||
4. Tool execution uses hermes-agent's `handle_function_call()` from `model_tools.py`
|
||||
### Changes to `environments/hermes_base_env.py`:
|
||||
|
||||
### `atropos/` (Our sandbox-optimized code) - PARTIALLY REDUNDANT
|
||||
```
|
||||
atropos/
|
||||
├── agent/atropos_agent.py ← ICL-based agent (REDUNDANT with agent_loop.py)
|
||||
├── envs/agent_env.py ← Environment with sandbox backends (PARTIALLY REDUNDANT)
|
||||
├── envs/swe_smith_oracle_env.py ← SWE env using sandbox (KEEP - port to new base)
|
||||
├── backends/ ← Sandbox backends (KEEP - valuable infrastructure)
|
||||
│ ├── modal_backend.py ← Modal sandbox pool
|
||||
│ ├── nomad_backend.py ← Nomad/Docker/Singularity
|
||||
│ └── base.py ← ToolBackend protocol
|
||||
├── slots/ ← Slot multiplexing (KEEP)
|
||||
├── nomad/ ← Nomad client (KEEP)
|
||||
├── tools/ ← Sandbox tool registry (PARTIALLY REDUNDANT)
|
||||
└── sandbox_server.py ← HTTP server in containers (KEEP)
|
||||
**1. Add config fields to `HermesAgentEnvConfig`:**
|
||||
- `tool_pool_mode: str = "default"` — "default" (terminal tool), "nomad", or "modal"
|
||||
- Nomad fields: `nomad_address`, `sandbox_job_id`, `sandbox_image`, `slots_per_container`, etc.
|
||||
- Modal fields: `modal_app_name`, `modal_image`, `modal_gpu`, `modal_slots_per_sandbox`, etc.
|
||||
- Shared: `allow_network`, `require_sandbox`, `purge_job_on_start`, `purge_job_on_shutdown`
|
||||
|
||||
**2. Add methods to `HermesAgentBaseEnv`:**
|
||||
- `_start_sandbox_backend()` / `_stop_sandbox_backend()` — lifecycle management
|
||||
- `setup_trajectory_workspace(item, exec_tool, trajectory_id)` → optional hook (no-op default)
|
||||
- `verify_and_score_trajectory(item, result, exec_tool)` → optional hook (calls compute_reward by default)
|
||||
|
||||
**3. Modify `collect_trajectory()`:**
|
||||
- When `tool_pool_mode == "default"`: existing behavior (terminal tool handles isolation)
|
||||
- When `tool_pool_mode in ("nomad", "modal")`: acquire slot → run agent with sandbox-backed tools → verify → release
|
||||
|
||||
**4. Port SWE env to `environments/`:**
|
||||
- Move/rewrite `swe_smith_oracle_env.py` to subclass `HermesAgentBaseEnv`
|
||||
- Override `setup_trajectory_workspace()` (git clone/worktree)
|
||||
- Override `verify_and_score_trajectory()` (pytest verification)
|
||||
|
||||
### Key Imports
|
||||
```python
|
||||
from atropos.backends import create_tool_backend # Nomad/Modal backends
|
||||
from atropos.backends.base import ToolBackend
|
||||
from atropos.slots.executor import ExecutionResult
|
||||
```
|
||||
|
||||
## Plan: Consolidate into `environments/`
|
||||
### What's Already Working
|
||||
- ✅ atroposlib with tool_call_support (ManagedServer has tool_call_parser)
|
||||
- ✅ GSM8k agent env with HermesAgentBaseEnv (Phase 1 tested, process mode)
|
||||
- ✅ mini-swe-agent installed (terminal tool available)
|
||||
- ✅ Modal backend (tested, working with sandboxes)
|
||||
- ✅ Nomad/Singularity backends (tested, working)
|
||||
- ✅ Tool call parsers (11+ models)
|
||||
|
||||
### What to KEEP from `atropos/`:
|
||||
- `backends/` - Modal, Nomad, Singularity backends (valuable infrastructure for scale)
|
||||
- `slots/` - Slot multiplexing
|
||||
- `nomad/` - Nomad client
|
||||
- `sandbox_server.py` - Container HTTP server
|
||||
- `Dockerfile` - Sandbox container image
|
||||
|
||||
### What to REMOVE/REPLACE:
|
||||
- `atropos/agent/atropos_agent.py` → replaced by `environments/agent_loop.py`
|
||||
- `atropos/envs/agent_env.py` → functionality merged into `environments/hermes_base_env.py`
|
||||
- `atropos/tools/` → replaced by `model_tools.py` + `tools/` (hermes-agent's standard tools)
|
||||
|
||||
### What to CREATE:
|
||||
- `environments/gsm8k_agent_env.py` → GSM8k with tool calling, subclasses `HermesAgentBaseEnv`
|
||||
- Update `environments/hermes_base_env.py` to optionally use sandbox backends (Nomad/Modal) for terminal isolation when needed for scale
|
||||
|
||||
### Steps:
|
||||
1. Install atropos `tool_call_support` branch (PR #366)
|
||||
2. Create `environments/gsm8k_agent_env.py` using `HermesAgentBaseEnv`
|
||||
3. Port `swe_smith_oracle_env.py` to use `HermesAgentBaseEnv`
|
||||
4. Make sandbox backends accessible from `HermesAgentBaseEnv` (terminal_backend config)
|
||||
5. Remove redundant `atropos/agent/` and `atropos/envs/agent_env.py`
|
||||
6. Clean up `atropos/tools/` (keep only sandbox-specific tools)
|
||||
7. Update tinker-atropos gsm8k env to use proper base class
|
||||
8. Test everything end-to-end
|
||||
|
||||
## Previous Completed Work
|
||||
- Modal backend integration (Feb 8) - KEEP backends, update integration point
|
||||
- Main branch merge (Feb 9) - completed
|
||||
- Singularity/Apptainer (Feb 6) - KEEP
|
||||
- Memory Bank initialized (Feb 5)
|
||||
### What Blocks
|
||||
- Tinker billing (402 error) — can't test Phase 2 training yet
|
||||
- No VLLM on this machine — can't test ManagedServer locally
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue