+-----------------------------------------------------------------------------------------+
|                  Multimodal Computer Use Agent Perception-Action Loop                   |
+-----------------------------------------------------------------------------------------+

[1. USER OBJECTIVE: "Open Salesforce, search for Acme Corp, and update ARR to $120,000"]
                                     |
                                     v
+-----------------------------------------------------------------+
| 2. SCREENSHOT CAPTURE (OS Display Layer / Virtual X11 Frame):   |
| - Captures high-res 1920x1080 display frame.                    |
| - Applies Dynamic Resolution scaling (reduces token overhead).  |
+--------------------------------+--------------------------------+
                                 |
                                 v (Passes RGB Image to Multimodal VLA Model)
+-----------------------------------------------------------------+
| 3. MULTIMODAL REASONING & VISUAL GROUNDING (Claude 3.5 / VLA):  |
| - Identifies 'Search Salesforce' input box at coordinates (450, 120).
| - Emits Batch Action: [MouseMove(450, 120), Click(), Type("Acme Corp"), Key("Enter")]
+--------------------------------+--------------------------------+
                                 |
                                 v (Executes on OS Hardware)
+-----------------------------------------------------------------+
| 4. OS ACTUATION LAYER (PyAutoGUI / XDOTool / Docker Desktop):   |
| - Moves hardware mouse pointer & fires hardware keystrokes.     |
+--------------------------------+--------------------------------+
                                 |
                                 v (Loops back: Captures new screenshot to verify update!)
[5. VERIFICATION & SUCCESS: ARR updated; Emits completion notification!]

+-----------------------------------------------------------------------------------------+
|                  Visual Grounding Evolution (2024 -> 2026)                              |
+-----------------------------------------------------------------------------------------+

2024: SET-OF-MARK (SoM) PROMPTING (Heuristic Overlay)
- Required running local object-detection models (YOLO / Grounding DINO) to draw numbered
  bounding boxes on top of the screenshot before feeding to LLM.
- High compute latency (~800ms) and brittle detection boxes.

2026: NATIVE MULTIMODAL SPATIAL ATTENTION (The Modern Standard)
- Frontier models natively calculate normalized pixel coordinates [x, y] in floating point (0.0 to 1.0).
- Visual Zoom: If a button is tiny (12x12px), the model requests a crop region:
  'crop_and_zoom(x1=0.4, y1=0.2, x2=0.5, y2=0.3)' to achieve sub-pixel precision!

# computer_use_agent.py
import anthropic
import pyautogui
import base64
import io
import time
from PIL import Image

client = anthropic.Anthropic()

# 1. Capture Active Screen as Base64 JPEG
def capture_screenshot():
  screenshot = pyautogui.screenshot()
  # Downscale to 1280x800 for optimal token efficiency
  screenshot = screenshot.resize((1280, 800), Image.Resampling.LANCZOS)
  
  buffered = io.BytesIO()
  screenshot.save(buffered, format="JPEG", quality=85)
  return base64.b64encode(buffered.getvalue()).decode("utf-8")

# 2. Execute Hardware OS Actions
def execute_computer_action(action_type, coordinate=None, text=None, key=None):
  print(f"[OS Actuator] Executing {action_type} at {coordinate}...")
  
  if action_type == "mouse_move" and coordinate:
    pyautogui.moveTo(coordinate[0], coordinate[1], duration=0.2)
  elif action_type == "left_click":
    pyautogui.click()
  elif action_type == "type" and text:
    pyautogui.write(text, interval=0.05)
  elif action_type == "key" and key:
    pyautogui.press(key)
  elif action_type == "screenshot":
    time.sleep(0.5) # Wait for UI animation to settle

# 3. Autonomous Computer-Use Agent Loop
def run_computer_agent(user_prompt: str, max_turns=10):
  messages = [
      {
          "role": "user",
          "content": [
              {"type": "text", "text": user_prompt},
              {
                  "type": "image",
                  "source": {
                      "type": "base64",
                      "media_type": "image/jpeg",
                      "data": capture_screenshot(),
                  },
              },
          ],
      }
  ]

  turn = 0
  while turn < max_turns:
    turn += 1
    print(f"\n[Agent] Turn {turn}/{max_turns}: Evaluating screen state...")

    # Call Anthropic Computer Use Tool
    response = client.beta.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        tools=[
            {
                "type": "computer_20241022",
                "name": "computer",
                "display_width_px": 1280,
                "display_height_px": 800,
            }
        ],
        messages=messages,
        betas=["computer-use-2024-10-22"],
    )

    # Check if model emitted computer actions
    tool_calls = [block for block in response.content if block.type == "tool_use"]
    
    if not tool_calls:
      print("[Agent] Task completed:", response.content[0].text)
      break

    # Execute all batch tool actions
    for tool_call in tool_calls:
      action = tool_call.input.get("action")
      coord = tool_call.input.get("coordinate")
      text = tool_call.input.get("text")
      key = tool_call.input.get("key")
      execute_computer_action(action, coordinate=coord, text=text, key=key)

    # Capture updated screen and feed back to agent
    messages.append({"role": "assistant", "content": response.content})
    messages.append({
        "role": "user",
        "content": [
            {
                "type": "tool_result",
                "tool_use_id": tool_calls[0].id,
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/jpeg",
                            "data": capture_screenshot(),
                        },
                    }
                ],
            }
        ],
    })

+-----------------------------------------------------------------------------------------+
|                  Enterprise Computer Use Security Perimeter                             |
+-----------------------------------------------------------------------------------------+

1. ISOLATED VIRTUAL DESKTOPS (VDI / Docker X11)
   - NEVER run computer use on an employee's personal physical workstation!
   - Run agents inside disposable cloud VDI desktop containers (Kasm Workspaces / Docker).

2. DOMAIN & APPLICATION WHITELISTS
   - Restrict network egress: Agent can only access approved enterprise SaaS URLs.
   - OS App Whitelist: Allow Chrome & Slack; forcefully block Terminal, Bash, and Settings.

3. HARDWARE-LEVEL APPROVAL PROMPTS
   - Any financial wire button or admin permission toggle triggers a modal pause requiring
     a human manager to authorize the click.

       +-------------------------------------------------------------+
       |             OSWorld Complex Desktop Task Success (%)        |
       +-------------------------------------------------------------+
 Traditional Scripting (Selenium/Macros) | ============= [14.0%] (Breaks on UI updates)
 Early Multimodal VLM (Zero-Shot 2024)   | ================== [22.4%]
 Modern VLA + Batch Computer Use (2026)  | ==================================== [46.8%] (3.3x Boost!)
                                         +-------------------------------------+
                                         0%     12%     24%     36%     48%