Token Calculator Tool (Robust) Documentation

Overview

The TokenCalculatorToolRobust is an enhanced token calculator with improved robustness, error handling, and accuracy. It provides accurate token counting using tiktoken, comprehensive validation, thread-safe operations, and currency precision.

File: tools/token_calculator_tool_robust.py Class: TokenCalculatorToolRobust Version: 1.0.0 Status: ✅ Active

Architecture

Design Principles

Accuracy: Uses tiktoken for accurate token counting

Robustness: Comprehensive error handling and validation

Thread Safety: Thread-safe operations with file locking

Currency Precision: Uses Decimal for precise calculations

Caching: Token counting cache for performance

Rate Limiting: Prevents API abuse

Core Components

class TokenCalculatorToolRobust(BaseTool):
    - memory_agent: MemoryAgent - For workspace management
    - pricing_data: Dict - LLM pricing configuration
    - _tokenizers: Dict - Tokenizer instances
    - _cache: Dict - Token counting cache
    - _lock: RLock - Thread safety lock

Key Features

1. Accurate Token Counting

Uses tiktoken library for:

Accurate token counting per model

Model-specific tokenizers

Fast tokenization

Reliable counts

2. Comprehensive Validation

Validates:

Currency values (Decimal precision)

Percentage values (0-1 range)

Token counts (non-negative, reasonable limits)

Model names (format validation)

3. Thread Safety

Thread-safe operations:

RLock for file operations

Safe concurrent access

Prevents race conditions

4. Currency Precision

Uses Decimal for:

Precise cost calculations

No floating-point errors

Accurate budget tracking

5. Caching

Token counting cache:

Reduces redundant calculations

Configurable TTL (default: 5 minutes)

Performance optimization

6. Rate Limiting

Prevents API abuse:

Max calls per minute (default: 60)

Automatic throttling

Request tracking

Usage

Calculate Token Cost

from tools.token_calculator_tool_robust import TokenCalculatorToolRobust
from agents.memory_agent import MemoryAgent

tool = TokenCalculatorToolRobust(memory_agent=memory_agent)
Calculate cost
success, result = await tool.execute(
    operation="calculate_cost",
    model="gpt-4",
    prompt_tokens=1000,
    completion_tokens=500
)

Track Usage

# Track token usage
success, result = await tool.execute(
    operation="track_usage",
    model="gpt-4",
    prompt_tokens=1000,
    completion_tokens=500
)

Configuration

Pricing Config

Located at:

config/llm_pricing_config.json

Usage Log

Stored at:

data/monitoring/token_usage.json

Cache

Stored at:

data/monitoring/token_cache.json

Limitations

Current Limitations

tiktoken Dependency: Requires tiktoken for accuracy

Limited Models: May not support all models

Basic Caching: Simple cache implementation

No Historical Analysis: No trend analysis

Single System: Single system only

Recommended Improvements

More Models: Support more LLM models

Advanced Caching: Better cache strategies

Historical Analysis: Trend tracking

Multi-System: Support distributed systems

Real-Time Alerts: Budget alerts

Visualization: Charts and graphs

API Integration: REST API access

Integration

With Memory Agent

Uses memory agent for:

Workspace management

Usage logging

Data persistence

Examples

Budget Tracking

# Check budget status
success, status = await tool.execute(
    operation="check_budget",
    period="daily"
)

Technical Details

Dependencies

tiktoken: Accurate token counting (optional but recommended)

decimal.Decimal: Currency precision

threading.RLock: Thread safety

agents.memory_agent.MemoryAgent: Workspace management

Tokenizer Initialization

Initializes tokenizers for:

OpenAI models (gpt-3.5, gpt-4, etc.)

Anthropic models (claude, etc.)

Other supported models

Future Enhancements

More Models: Expand model support

Advanced Analytics: Trend analysis

Real-Time Alerts: Budget alerts

Visualization: Charts and dashboards

API Integration: REST API

Multi-System: Distributed tracking

Predictive Budgeting: ML-based predictions

---