pgadmin4/web/pgadmin/llm/tests/README.md

LLM Module Tests

This directory contains comprehensive tests for the pgAdmin LLM/AI functionality.

Test Files

Python Tests

test_client.py - LLM Client Tests

Tests the core LLM client functionality including:

• Provider initialization (Anthropic, OpenAI, Ollama)
• API key loading from files and environment variables
• Graceful handling of missing API keys
• User preference overrides
• Provider selection logic
• Whitespace handling in API keys

Key Features:

• Tests pass even without API keys configured
• Mocks external API calls
• Tests all three provider types

test_reports.py - Report Generation Tests

Tests report generation functionality including:

• Security, performance, and design report types
• Server, database, and schema level reports
• Report request validation
• Progress callback functionality
• Error handling during generation
• Markdown formatting

Key Features:

• Tests data collection from PostgreSQL
• Validates report structure
• Tests streaming progress updates

test_chat.py - Chat Session Tests

Tests interactive chat functionality including:

• Chat session initialization
• Message history management
• Context passing (database, SQL queries)
• Streaming responses
• Token counting for context management
• Maximum history limits
• Error handling

Key Features:

• Tests conversation flow
• Validates context integration
• Tests memory management

test_compaction.py - Conversation Compaction Tests

Tests the conversation history compaction module including:

• Token estimation with provider-specific ratios
• SQL content token multiplier
• History compaction with token budget enforcement
• First message and recent window preservation
• Low-value message dropping by importance classification
• Tool call/result pair integrity during compaction
• History deserialization from frontend JSON format
• Conversational message filtering (stripping tool internals)

Key Features:

• Tests all five importance classification tiers
• Validates tool pair preservation (no orphaned tool results)
• Tests round-trip serialization/deserialization
• Tests edge cases (empty history, within-budget, unknown roles)

test_views.py - API Endpoint Tests

Tests Flask endpoints including:

• /llm/status - LLM availability check
• /llm/reports/security/* - Security report endpoints
• /llm/reports/performance/* - Performance report endpoints
• /llm/reports/design/* - Design review endpoints
• /llm/chat - Chat endpoint
• Streaming endpoints with SSE

Key Features:

• Tests authentication and permissions
• Tests API error responses
• Tests SSE streaming format

JavaScript Tests

AIReport.spec.js - AIReport Component Tests

Tests the React component for AI report display including:

• Component rendering in light and dark modes
• Theme detection from body styles
• Progress display during generation
• Error handling
• Markdown rendering
• Download functionality
• SSE event handling
• Support for all report categories and types

Key Features:

• Tests with React Testing Library
• Mocks EventSource for SSE
• Tests theme transitions
• Validates accessibility

Running the Tests

Python Tests

From the web directory:

# Run all LLM tests
python -m pytest pgadmin/llm/tests/

# Run specific test file
python -m pytest pgadmin/llm/tests/test_client.py

# Run specific test case
python -m pytest pgadmin/llm/tests/test_client.py::LLMClientTestCase::test_anthropic_provider_with_api_key

# Run with coverage
python -m pytest --cov=pgadmin/llm pgadmin/llm/tests/

JavaScript Tests

From the web directory:

# Run all JavaScript tests
yarn run test:karma

# Run specific test file
yarn run test:karma -- --file regression/javascript/llm/AIReport.spec.js

Test Coverage

What's Tested

✅ LLM client initialization with all providers ✅ API key loading from files and environment ✅ Graceful handling of missing API keys ✅ User preference overrides ✅ Report generation for all categories (security, performance, design) ✅ Report generation for all levels (server, database, schema) ✅ Chat session management and history ✅ Conversation history compaction and token budgets ✅ Conversational message filtering ✅ History serialization/deserialization round-trip ✅ Streaming progress updates via SSE ✅ API endpoint authentication and authorization ✅ React component rendering in both themes ✅ Dark mode text color detection ✅ Error handling throughout the stack

What's Mocked

• External LLM API calls (Anthropic, OpenAI, Ollama)
• PostgreSQL database connections
• File system access for API keys
• EventSource for SSE streaming
• Theme detection (window.getComputedStyle)

Environment Variables for Testing

These environment variables can be set for integration testing with real APIs:

# For Anthropic
export ANTHROPIC_API_KEY="your-api-key"

# For OpenAI
export OPENAI_API_KEY="your-api-key"

# For Ollama
export OLLAMA_API_URL="http://localhost:11434"

Note: Tests are designed to pass without these variables set. They will mock API responses when keys are not available.

Test Philosophy

1. Graceful Degradation: All tests pass even without API keys configured
2. Mocking by Default: External APIs are mocked to avoid dependencies
3. Comprehensive Coverage: Tests cover happy paths, error cases, and edge cases
4. Documentation: Tests serve as documentation for expected behavior
5. Integration Ready: Tests can be run with real APIs when keys are provided

Adding New Tests

When adding new functionality to the LLM module:

1. Add unit tests to the appropriate test file
2. Mock external dependencies
3. Test both success and failure cases
4. Test with and without API keys/configuration
5. Update this README with new test coverage

Troubleshooting

Common Issues

Import errors: Make sure you're running tests from the web directory

API key warnings: These are expected - tests should pass without API keys

Theme mocking errors: Ensure fake_theme.js is available in regression/javascript/

EventSource not found: This is mocked in JavaScript tests, ensure mocks are properly set up