docs/troubleshooting/issue_analysis_resolution.md

Issue Analysis and Resolution Plan

Executive Summary

This document analyzes the multiple issues observed in the application logs, identifies root causes, and provides a comprehensive resolution plan with file-level and line-level tasks.

Issues Identified

0. Web Search Implementation Issues (FIXED ✅)

Problems:

1. DuckDuckGo used by default instead of Serper (even when Serper API key available)
2. Serper used invalid source="serper" (should be source="web")
3. SearchXNG used invalid source="searchxng" (should be source="web")
4. Serper and SearchXNG missing title truncation (would cause validation errors)
5. Missing tool name mappings in SearchHandler

Root Causes:

• Default web_search_provider was "duckduckgo" instead of "auto"
• No auto-detection logic to prefer Serper when API key available
• Source type mismatches with SourceName literal
• Missing title truncation in Serper/SearchXNG implementations

Fixes Applied:

• ✅ Changed default to "auto" with auto-detection logic
• ✅ Fixed Serper to use source="web" and add title truncation
• ✅ Fixed SearchXNG to use source="web" and add title truncation
• ✅ Added tool name mappings in SearchHandler
• ✅ Improved factory to auto-detect best available provider

Status: ✅ FIXED - All web search issues resolved

---

1. Citation Title Validation Error (FIXED ✅)

Error: 1 validation error for Citation\ntitle\n String should have at most 500 characters

Root Cause: DuckDuckGo search results can return titles longer than 500 characters, but the Citation model enforces a maximum length of 500 characters.

Location: src/tools/web_search.py:61

Fix Applied: Added title truncation to 500 characters before creating Citation objects.

Status: ✅ FIXED - Code updated in src/tools/web_search.py

---

2. 403 Forbidden Errors on HuggingFace Inference API

Error: status_code: 403, model_name: Qwen/Qwen3-Next-80B-A3B-Thinking, body: Forbidden

Root Causes:

1. OAuth Scope Missing: The OAuth token may not have the inference-api scope required for accessing HuggingFace Inference API
2. Model Access Restrictions: Some models (e.g., Qwen/Qwen3-Next-80B-A3B-Thinking) may require:
   • Gated model access approval
   • Specific provider access
   • Account-level permissions
3. Provider Selection: Pydantic AI's HuggingFaceProvider doesn't support explicit provider selection (e.g., "nebius", "hyperbolic"), which may be required for certain models
4. Token Format: The OAuth token might not be correctly extracted or formatted

Evidence from Logs:

• OAuth authentication succeeds: OAuth user authenticated username=Tonic
• Token is extracted: OAuth token extracted from oauth_token.token attribute
• But API calls fail: status_code: 403, model_name: Qwen/Qwen3-Next-80B-A3B-Thinking, body: Forbidden

Impact: All LLM operations fail, causing:

• Planner agent execution failures
• Observation generation failures
• Knowledge gap evaluation failures
• Tool selection failures
• Judge assessment failures
• Report writing failures

Status: ⚠️ INVESTIGATION REQUIRED

---

3. 422 Unprocessable Entity Errors

Error: status_code: 422, model_name: meta-llama/Llama-3.1-70B-Instruct, body: Unprocessable Entity

Root Cause:

• Model/provider compatibility issues
• The model meta-llama/Llama-3.1-70B-Instruct on provider hyperbolic may be in staging mode or have specific requirements
• Request format may not match provider expectations

Evidence from Logs:

• Model meta-llama/Llama-3.1-70B-Instruct is in staging mode for provider hyperbolic. Meant for test purposes only.
• Followed by: status_code: 422, model_name: meta-llama/Llama-3.1-70B-Instruct, body: Unprocessable Entity

Impact: Judge assessment fails, causing research loops to continue indefinitely with low confidence scores.

Status: ⚠️ INVESTIGATION REQUIRED

---

4. MCP Server Warning

Warning: This MCP server includes a tool that has a gr.State input, which will not be updated between tool calls.

Root Cause: Gradio MCP integration issue with state management.

Impact: Minor - functionality may be affected but not critical.

Status: ℹ️ INFORMATIONAL

---

5. Modal TTS Function Setup Failure

Error: modal_tts_function_setup_failed error='Local state is not initialized - app is not locally available'

Root Cause: Modal TTS function requires local Modal app initialization, which isn't available in HuggingFace Spaces environment.

Impact: Text-to-speech functionality unavailable, but not critical for core functionality.

Status: ℹ️ INFORMATIONAL

---

Root Cause Analysis

OAuth Token Flow

1. Token Extraction (src/app.py:617-628):

   if hasattr(oauth_token, "token"):
       token_value = oauth_token.token
   

   ✅ Working correctly - Logs confirm token extraction

2. Token Passing (src/app.py:125, src/agent_factory/judges.py:54):

   effective_api_key = oauth_token or os.getenv("HF_TOKEN") or os.getenv("HUGGINGFACE_API_KEY")
   hf_provider = HuggingFaceProvider(api_key=effective_api_key)
   

   ✅ Working correctly - Token is passed to HuggingFaceProvider

3. API Calls (Pydantic AI internal):

   • Pydantic AI's HuggingFaceProvider uses AsyncInferenceClient internally
   • The api_key parameter should be passed to the underlying client
   • ❓ Unknown: Whether the token format or scope is correct

HuggingFaceProvider Limitations

Key Finding: The code comments indicate:

# Note: The hf_provider parameter is accepted but not used here because HuggingFaceProvider
# from pydantic-ai doesn't support provider selection. Provider selection happens at the
# InferenceClient level (used in HuggingFaceChatClient for advanced mode).

This means:

• HuggingFaceProvider doesn't support explicit provider selection (e.g., "nebius", "hyperbolic")
• Provider selection is automatic or uses default HuggingFace Inference API endpoint
• Some models may require specific providers, which can't be specified

Model Access Issues

The logs show attempts to use:

• Qwen/Qwen3-Next-80B-A3B-Thinking - May require gated access
• meta-llama/Llama-3.1-70B-Instruct - May have provider-specific restrictions
• Qwen/Qwen3-235B-A22B-Instruct-2507 - May require special permissions

---

Resolution Plan

Phase 1: Immediate Fixes (Completed)

✅ Task 1.1: Fix Citation title validation error

• File: src/tools/web_search.py
• Line: 60-61
• Change: Add title truncation to 500 characters
• Status: ✅ COMPLETED

---

Phase 2: OAuth Token Investigation and Fixes

Task 2.1: Add Token Validation and Debugging

Files to Modify:

• src/utils/llm_factory.py
• src/agent_factory/judges.py
• src/app.py

Subtasks:

1. Add token format validation (check if token is a valid string)
2. Add token length logging (without exposing actual token)
3. Add scope verification (if possible via API)
4. Add detailed error logging for 403 errors

Line-Level Tasks:

• src/utils/llm_factory.py:139: Add token validation before creating HuggingFaceProvider
• src/agent_factory/judges.py:54: Add token validation and logging
• src/app.py:125: Add token format validation

Task 2.2: Improve Error Handling for 403 Errors

Files to Modify:

• src/agent_factory/judges.py
• src/agents/*.py (all agent files)

Subtasks:

1. Catch ModelHTTPError with status_code 403 specifically
2. Provide user-friendly error messages
3. Suggest solutions (re-authenticate, check scope, use alternative model)
4. Log detailed error information for debugging

Line-Level Tasks:

• src/agent_factory/judges.py:159: Add specific 403 error handling
• src/agents/knowledge_gap.py: Add error handling in agent execution
• src/agents/tool_selector.py: Add error handling in agent execution
• src/agents/thinking.py: Add error handling in agent execution
• src/agents/writer.py: Add error handling in agent execution

Task 2.3: Add Fallback Mechanisms

Files to Modify:

• src/agent_factory/judges.py
• src/utils/llm_factory.py

Subtasks:

1. Define fallback model list (publicly available models)
2. Implement automatic fallback when primary model fails with 403
3. Log fallback model selection
4. Continue with fallback model if available

Line-Level Tasks:

• src/agent_factory/judges.py:30-66: Add fallback model logic in get_model()
• src/utils/llm_factory.py:121-153: Add fallback model logic in get_pydantic_ai_model()

Task 2.4: Document OAuth Scope Requirements

Files to Create/Modify:

• docs/troubleshooting/oauth_403_errors.md ✅ CREATED
• README.md: Add OAuth setup instructions
• src/app.py:114-120: Enhance existing comments

Subtasks:

1. Document required OAuth scopes
2. Provide troubleshooting steps
3. Add examples of correct OAuth configuration
4. Link to HuggingFace documentation

---

Phase 3: 422 Error Handling

Task 3.1: Add 422 Error Handling

Files to Modify:

• src/agent_factory/judges.py
• src/utils/llm_factory.py

Subtasks:

1. Catch 422 errors specifically
2. Detect staging mode warnings
3. Automatically switch to alternative provider or model
4. Log provider/model compatibility issues

Line-Level Tasks:

• src/agent_factory/judges.py:159: Add 422 error handling
• src/utils/llm_factory.py: Add provider fallback logic

Task 3.2: Provider Selection Enhancement

Files to Modify:

• src/utils/huggingface_chat_client.py
• src/app.py

Subtasks:

1. Investigate if HuggingFaceProvider can be configured with provider
2. If not, use HuggingFaceChatClient for provider selection
3. Add provider fallback chain
4. Log provider selection and failures

Line-Level Tasks:

• src/utils/huggingface_chat_client.py:29-64: Enhance provider selection
• src/app.py:154: Consider using HuggingFaceChatClient for provider support

---

Phase 4: Enhanced Logging and Monitoring

Task 4.1: Add Comprehensive Error Logging

Files to Modify:

• All agent files
• src/agent_factory/judges.py
• src/utils/llm_factory.py

Subtasks:

1. Log token presence (not value) at key points
2. Log model selection and provider
3. Log HTTP status codes and error bodies
4. Log fallback attempts and results

Task 4.2: Add User-Friendly Error Messages

Files to Modify:

• src/app.py
• src/orchestrator/graph_orchestrator.py

Subtasks:

1. Convert technical errors to user-friendly messages
2. Provide actionable solutions
3. Link to documentation
4. Suggest alternative models or configurations

---

Implementation Priority

High Priority (Blocking Issues)

1. ✅ Citation title validation (COMPLETED)
2. OAuth token validation and debugging
3. 403 error handling with fallback
4. User-friendly error messages

Medium Priority (Quality Improvements)

5. 422 error handling
6. Provider selection enhancement
7. Comprehensive logging

Low Priority (Nice to Have)

8. MCP server warning fix
9. Modal TTS setup (environment-specific)

---

Testing Plan

Unit Tests

• Test Citation title truncation with various lengths
• Test token validation logic
• Test fallback model selection
• Test error handling for 403, 422 errors

Integration Tests

• Test OAuth token flow end-to-end
• Test model fallback chain
• Test provider selection
• Test error recovery

Manual Testing

• Verify OAuth login with correct scope
• Test with various models
• Test error scenarios
• Verify user-friendly error messages

---

Success Criteria

1. ✅ Citation validation errors eliminated
2. 403 errors handled gracefully with fallback
3. 422 errors handled with provider/model fallback
4. Clear error messages for users
5. Comprehensive logging for debugging
6. Documentation updated with troubleshooting steps

---

References

• HuggingFace OAuth Scopes
• Pydantic AI HuggingFace Provider
• HuggingFace Inference API
• HuggingFace Inference Providers