build-eye

# Build-Eye CI Monitoring System **Build-Eye** - vLLM-Ascend CI Build Automation Monitoring and Root Cause Analysis System ## System Overview Build-Eye is a CI build monitoring system designed specifically for the vLLM-Ascend project, capable of: - **Automatically monitoring** CI builds in the `vllm-project/vllm-ascend` repository. - **Intelligently classifying** build failure root causes (Code Issues / Infrastructure Issues / Interference Issues). - **Generating suggestions** for low-cost, actionable fix plans. - **Archiving reports** by automatically storing standardized reports in the `build-eye` repository. ## Monitoring Targets - Primary monitored repository: https://github.com/vllm-project/vllm-ascend - Archive hosting repository: https://github.com/winson-00178005/build-eye.git ## Failure Root Cause Classification ### 1. PR Code Issues - Test assertion failures - Compilation errors (CMake, clang) - Python import errors - vLLM API incompatibilities - Ascend kernel compilation issues ### 2. Infrastructure Issues - K8s cache-service failures (`cache-service.nginx-pypi-cache`) - Runner unavailability - NPU hardware issues (910B/910C/310P) - CANN toolkit issues - HCCL multi-card communication failures - Docker image pull failures - Csrc cache failures - Build timeouts ### 3. Multi-PR Concurrent Interference - Multiple PRs merged within a short period - Differences in vLLM version matrices - Runner resource contention - Impact from CANN image updates ## Quick Start ### 1. Configure GitHub Token Refer to `docs/token-setup.md` to configure the required keys. ### 2. Install Dependencies ```bash pip install -r requirements.txt ``` ### 3. Manually Trigger Monitoring ```bash python scripts/monitor/fetch_runs.py --output data/workflow_runs.json python scripts/monitor/collect_metadata.py --input data/workflow_runs.json --output data/build_metadata.json python scripts/classify/classifier.py --input data/build_metadata.json --output data/classifications.json python scripts/recommend/recommender.py --input data/classifications.json --output data/recommendations.json python scripts/report/generator.py --input data/recommendations.json --output reports/ ``` ### 4. Archive Reports ```bash python scripts/archive/archiver.py --input reports/ --repo https://github.com/winson-00178005/build-eye.git ``` ## GitHub Actions Workflow The system supports two trigger modes: ### Scheduled Polling Automatically checks for recent build failures every 6 hours. Workflow file: `.github/workflows/monitor.yml` ### Manual Trigger Trigger manually via GitHub Actions' `workflow_dispatch`. Optional parameters: - `lookback_hours`: Number of past hours to check (default 24) - `dry_run`: Dry run mode (does not archive) - `target_repo`: Target repository ## Project Structure ``` build-eye/ ├── .github/workflows/ │ └── monitor.yml # GitHub Actions workflow ├── scripts/ │ ├── monitor/ # CI monitoring module │ │ ├── github_client.py # GitHub API client │ │ ├── fetch_runs.py # Fetch workflow runs │ │ ├── collect_metadata.py # Collect metadata │ │ └── config_loader.py # Configuration loader │ ├── classify/ # Failure classification module │ │ ├── classifier.py # Classification engine │ │ ├── code_detector.py # Code issue detection │ │ ├── infra_detector.py # Infrastructure detection │ │ └── interference_detector.py # Interference detection │ ├── recommend/ # Fix recommendation module │ │ ├── recommender.py # Recommendation generator │ │ └── templates.py # Recommendation templates │ ├── report/ # Report generation module │ │ ├── generator.py # Report generator │ │ ├── formatter.py # Formatting tools │ │ └── summary.py # Summary generation │ └── archive/ # Report archiving module │ ├── archiver.py # Archiver │ └── git_client.py # Git client ├── config/ │ └── config.yaml # System configuration ├── templates/ │ └ example_reports.py # Report examples ├── tests/ # Test directory ├── docs/ │ └ token-setup.md # Token configuration guide ├── reports/ # Report output directory ├ requirements.txt # Python dependencies ├ requirements-dev.txt # Development dependencies └ README.md # This document ``` ## Configuration Options ### config/config.yaml ```yaml target_repository: owner: vllm-project repo: vllm-ascend url: https://github.com/vllm-project/vllm-ascend branch: main monitored_workflows: - pr_test_full.yaml - pr_test_light.yaml archive_repository: owner: winson-00178005 repo: build-eye url: https://github.com/winson-00178005/build-eye.git monitoring: polling_interval_hours: 6 lookback_hours: 24 ``` ### Environment Variables - `GITHUB_TOKEN`: GitHub API access token - `ARCHIVE_TOKEN`: Write token for the archive repository - `TARGET_REPO_OWNER`: Target repository owner - `TARGET_REPO_NAME`: Target repository name ## Report Format Each report includes: - YAML frontmatter (metadata) - Summary (1-2 sentences) - Root cause analysis (classification, confidence, reasoning) - Evidence (matched patterns, links, log snippets) - Fix recommendations (priority suggestions, detailed steps) - Related PRs (only for interference classification) Report archive path: `reports/YYYY/MM/DD/<category>-pr-<number>.md` ## Running Tests ```bash pip install -r requirements-dev.txt pytest tests/ ``` ## Extending Nightly Pipelines The system is designed to support nightly pipeline monitoring. Simply add the following to the configuration: ```yaml target_repository: monitored_workflows: - schedule_nightly_test_a2.yaml - schedule_nightly_test_a3.yaml ``` ## Maintenance and Extension ### Adding New Classification Rules Add new detectors in `scripts/classify/`: ```python def detect_new_pattern(log_excerpt: str) -> dict: patterns = [...] # Implement detection logic ``` Then call it in `classifier.py`. ### Adding New Recommendation Templates Add new templates in `scripts/recommend/templates.py`. ## License Apache License 2.0 - See the LICENSE file for details. ## Contact Issue feedback: https://github.com/winson-00178005/build-eye/issues