Troubleshooting

Common issues and solutions when using Chisel CLI with step-by-step debugging guides

Resolve common issues when using Chisel CLI with comprehensive troubleshooting guides and solutions.

Quick Issue Resolution

Most issues can be resolved quickly by checking authentication, upload size, and GPU memory requirements.

Issue Type	Quick Check	Common Solution
Authentication	chisel.auth.is_authenticated()	Re-authenticate with chisel command
Upload fails	Directory size < 100MB	Add files to .gitignore
GPU memory	Reduce batch size	Use larger GPU type
Script not found	Script in upload directory	Check upload_dir parameter
Network issues	Check connectivity	Verify CHISEL_BACKEND_URL

Installation Issues

ModuleNotFoundError: No module named 'chisel'

Symptoms: Import error when trying to use Chisel CLI

Causes:

• Chisel CLI not installed
• Wrong virtual environment
• Installation failed

Solutions:

Install in Development Mode

# Recommended: Install in virtual environment
python -m venv .venv
source .venv/bin/activate  # Linux/Mac
# OR .venv\Scripts\activate  # Windows

pip install chisel-cli

After successful installation, you should be able to import chisel and run chisel --help.

externally-managed-environment Error

Symptoms: pip refuses to install packages due to externally managed environment

Cause: System Python protection on newer Linux distributions

Solutions:

Virtual Environment (Recommended)

# Create isolated environment
python -m venv .venv
source .venv/bin/activate

# Install Chisel
pip install chisel-cli

Always prefer virtual environments over system-wide installations to avoid conflicts.

Permission Errors During Installation

Symptoms: Permission denied errors during pip install

Causes:

• Insufficient permissions for system directories
• Write-protected directories
• System Python restrictions

Solutions:

Virtual Environment Solution

# Best practice: Use virtual environment
python -m venv chisel_env
source chisel_env/bin/activate
pip install chisel-cli

Authentication Issues

Browser Doesn't Open for Authentication

Symptoms: No browser window opens when running chisel for the first time

Common Causes:

• SSH/remote environment without display
• Headless system
• Browser configuration issues
• Firewall blocking browser communication

Solutions:

SSH Environment

# Use SSH port forwarding
ssh -L 8000:localhost:8000 user@remote-host

# Then run chisel command on remote host
chisel python my_script.py
# Browser will open on your local machine

For automated environments, obtain an API key through the web interface and use environment variables.

Authentication Fails After Success

Symptoms: Authentication worked before but now fails

Causes:

• Expired credentials
• Corrupted credential file
• Backend URL changed
• Network connectivity issues

Step-by-step Resolution:

1. Check current status

   from chisel.auth import is_authenticated
   print(f"Authenticated: {is_authenticated()}")
   

2. Clear existing credentials

   from chisel.auth import clear_credentials
   clear_credentials()
   print("🔓 Credentials cleared")
   

3. Test backend connectivity

   # Test if backend is reachable
   curl -v http://localhost:8000/health
   
   # Or custom backend
   curl -v $CHISEL_BACKEND_URL/health
   

4. Re-authenticate

   # Let chisel handle authentication
   chisel python -c "print('Testing authentication')"
   

After re-authentication, test with a simple command to verify it’s working.

API Key Errors

Symptoms: “Invalid API key” or “API key expired” errors

Immediate Solutions:

Clear and Re-authenticate

from chisel.auth import clear_credentials, authenticate

# Clear potentially corrupted credentials
clear_credentials()

# Re-authenticate
try:
    api_key = authenticate()
    print("✅ New authentication successful")
except RuntimeError as e:
    print(f"❌ Re-authentication failed: {e}")

Runtime Issues

CUDA Out of Memory

Symptoms: RuntimeError: CUDA out of memory during execution

Immediate Solutions:

Reduce Batch Size

# Original code causing memory issues
batch_size = 256  # Too large

# Reduced batch size
batch_size = 32  # Start smaller

# Or dynamic batch size
import torch

def find_safe_batch_size(model, input_shape, start_size=32):
    batch_size = start_size
    while batch_size >= 1:
        try:
            test_input = torch.randn(batch_size, *input_shape, device='cuda')
            with torch.no_grad():
                _ = model(test_input)
            print(f"✅ Safe batch size: {batch_size}")
            return batch_size
        except RuntimeError as e:
            if "out of memory" in str(e):
                batch_size //= 2
                torch.cuda.empty_cache()
            else:
                raise e
    return 1

If memory errors persist even with optimizations, you may need a larger GPU configuration.

Job Submission Fails

Symptoms: ❌ Work upload failed or upload errors

Common Causes:

• Upload directory too large (>100MB)
• Network connectivity issues
• Too many files in directory
• Binary files included

Solutions:

Reduce Upload Size

# Check current directory size
du -sh .
du -sh * | sort -hr

# Add large files to .gitignore
echo "__pycache__/" >> .gitignore
echo "*.pyc" >> .gitignore
echo ".venv/" >> .gitignore
echo "data/" >> .gitignore
echo "*.model" >> .gitignore
echo "*.pkl" >> .gitignore

# Use specific upload directory

# Use smaller upload directory
app = ChiselApp("my-app", upload_dir="./src")

# Instead of uploading everything
# app = ChiselApp("my-app", upload_dir=".")

Keep your upload directory under 50MB for fastest uploads and best reliability.

Script Not Found in Archive

Symptoms: ❌ Script not found in uploaded archive

Causes:

• Script outside upload directory
• Incorrect relative paths
• Script name mismatch

Solutions:

Fix Upload Directory

# Ensure script is in upload directory
app = ChiselApp("my-app", upload_dir=".")

# If script is in subdirectory
app = ChiselApp("my-app", upload_dir="./scripts")

# Check script location
import os
print(f"Current directory: {os.getcwd()}")
print(f"Script exists: {os.path.exists('my_script.py')}")

Argument Passing Issues

Symptoms: error: unrecognized arguments or arguments not passed correctly

Solutions:

Correct Argument Format

# Correct format
chisel python script.py --arg1 value1 --arg2 value2

# With quotes for complex arguments
chisel python script.py --text "hello world" --data '[1,2,3]'

# Multiple arguments
chisel python train.py --epochs 100 --lr 0.001 --batch-size 32

Performance Issues

Slow GPU Execution

Symptoms: GPU execution slower than expected

Diagnostic Steps:

1. Verify GPU usage

   import torch
   
   print(f"CUDA available: {torch.cuda.is_available()}")
   print(f"Device count: {torch.cuda.device_count()}")
   print(f"Current device: {torch.cuda.current_device()}")
   print(f"Device name: {torch.cuda.get_device_name()}")
   
   # Check if tensors are on GPU
   tensor = torch.randn(100, device='cuda')
   print(f"Tensor device: {tensor.device}")
   

2. Monitor GPU utilization

   # Add to your script
   def print_gpu_utilization():
       import torch
       allocated = torch.cuda.memory_allocated() / 1024**3
       reserved = torch.cuda.memory_reserved() / 1024**3
       print(f"GPU Memory: {allocated:.1f}GB allocated, {reserved:.1f}GB reserved")
   
   # Call periodically during processing
   print_gpu_utilization()
   

3. Optimize data placement

   # Ensure data is on GPU
   data = data.cuda()
   model = model.cuda()
   
   # Use non_blocking transfer
   data = data.cuda(non_blocking=True)
   
   # Keep data on GPU between operations
   # Don't move back and forth unnecessarily
   

Performance Optimizations:

Mixed Precision

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

for batch in dataloader:
    with autocast():
        output = model(batch)
        loss = criterion(output, target)
    
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

Memory Leaks

Symptoms: GPU memory usage increases over time

Detection and Solutions:

Monitor Memory Leaks

import torch

def track_memory(step_name):
    allocated = torch.cuda.memory_allocated() / 1024**3
    reserved = torch.cuda.memory_reserved() / 1024**3
    print(f"{step_name}: {allocated:.2f}GB allocated, {reserved:.2f}GB reserved")

# Use throughout your code
track_memory("Start")
# ... your code ...
track_memory("After processing")

Network and Connectivity

Connection Timeouts

Symptoms: Network timeouts or connection errors

Diagnostic Steps:

Test Backend Connectivity

# Test basic connectivity
ping your-backend-host

# Test HTTP connectivity
curl -v http://localhost:8000/health
curl -v $CHISEL_BACKEND_URL/health

# Test with timeout
curl --connect-timeout 10 --max-time 30 $CHISEL_BACKEND_URL/health

Streaming Connection Issues

Symptoms: Real-time output not showing or connection drops

Causes:

• Corporate firewall blocking streaming
• Network instability
• Backend overload

Solutions:

Test Streaming

# Check if streaming works
curl -N $CHISEL_BACKEND_URL/api/v1/stream-test

# Test with chisel
chisel python -c "
import time
for i in range(10):
    print(f'Output line {i}')
    time.sleep(1)
"

Debugging Tools

Enable Debug Mode

Enable comprehensive debugging for detailed error information:

Debug Logging

import logging
import os

# Enable debug logging
if os.getenv('CHISEL_DEBUG'):
    logging.basicConfig(level=logging.DEBUG)
    
    # PyTorch debugging
    import torch
    torch.autograd.set_detect_anomaly(True)
    os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

System Information Collection

Collect comprehensive system information for debugging:

System Info Script

def collect_system_info():
    import sys, torch, os, platform
    
    print("=== Chisel Debug Information ===")
    print(f"Platform: {platform.platform()}")
    print(f"Python: {sys.version}")
    print(f"PyTorch: {torch.__version__}")
    
    # CUDA info
    print(f"CUDA available: {torch.cuda.is_available()}")
    if torch.cuda.is_available():
        print(f"CUDA version: {torch.version.cuda}")
        print(f"GPU count: {torch.cuda.device_count()}")
        for i in range(torch.cuda.device_count()):
            props = torch.cuda.get_device_properties(i)
            print(f"GPU {i}: {props.name} ({props.total_memory/1e9:.1f}GB)")
    
    # Chisel environment
    chisel_vars = {k: v for k, v in os.environ.items() if k.startswith('CHISEL')}
    print(f"Chisel environment: {chisel_vars}")
    
    # Authentication status
    try:
        from chisel.auth import is_authenticated
        print(f"Authenticated: {is_authenticated()}")
    except Exception as e:
        print(f"Auth check failed: {e}")

# Run this to collect debug info
collect_system_info()

Common Error Patterns

Quick reference for the most common error patterns and their solutions:

Error Message	Cause	Quick Fix
ModuleNotFoundError: No module named 'chisel'	Not installed	pip install chisel-cli
CUDA out of memory	Insufficient GPU memory	Reduce batch size or use larger GPU
Script not found	Wrong upload directory	Check upload_dir parameter
Authentication failed	Invalid credentials	clear_credentials() and re-authenticate
Connection timeout	Network issues	Check connectivity and backend URL
Work upload failed	Directory too large	Add files to .gitignore

Getting Help

If you’re still experiencing issues after trying these solutions, gather the following information before seeking help:

Information to Collect

1. System Information (use the debug script above)
2. Complete Error Output (full traceback)
3. Minimal Reproduction Code (smallest example that fails)
4. Job ID (if applicable from Chisel output)
5. Network Environment (corporate, home, cloud, etc.)

Support Channels

Email Support

Direct support for urgent issues and detailed troubleshooting

GitHub Issues

Report bugs, request features, and community support

GitHub Discussions

Ask questions, share ideas, and community help

Documentation

Complete API reference and configuration guides

Reporting Template

When reporting issues, use this template:

## Issue Description
Brief description of the problem

## Environment
- OS: [e.g., Ubuntu 22.04, macOS 13.0, Windows 11]
- Python: [e.g., 3.9.7]
- PyTorch: [e.g., 2.0.1]
- Chisel CLI: [version or commit hash]

## Steps to Reproduce
1. Step one
2. Step two
3. Step three

## Expected Behavior
What you expected to happen

## Actual Behavior
What actually happened

## Error Output

Complete error message and traceback

## Additional Context
- Job ID (if applicable)
- Network environment
- Any workarounds tried

Related Documentation

Configuration

Optimize settings and environment variables

API Reference

Complete function and class documentation

Examples

Working examples and best practices

Getting Started

Installation and setup guide