About this project

## Overview Hospital-Recruit-QA-Bot is a QA-oriented automation pipeline that automatically collects recruitment notices from 42 major domestic hospitals every morning and verifies the integrity of the collected data. Rather than simple crawling, it aims for a structure that includes only reliable data in the final output by implementing a verification gateway. ## Core Features ### 1. RaiT Verification Gateway - Real-time verification of missing values and format errors in collected raw data based on the RaiT (Responsible AI Testing) framework. - Only notices that pass (PASS) the verification are included in the final output and registered in `seen_posts`, blocking unsuitable data from mixing into the results. ### 2. CI/CD Pipeline (GitHub Actions) - Automatically executes every day at 9:00 AM (KST) in an Ubuntu virtual environment. - After crawling, the bot automatically commits and pushes the updated `seen_posts.csv` to continuously prevent duplicate collection. ### 3. Duplicate Data Filtering - Local and cloud environments share the same history DB (`seen_posts.csv`), excluding previously identified notices and reporting only new ones. ## QA Verification Scenarios The test cases organized in the README are as follows: | ID | Test Item | Expected Result | Status | |---|---|---|---| | TC-01 | RaiT Data Validity Verification | Automatic determination of missing values and format errors via `verify_rait_compliance` logic | Pass | | TC-02 | Data Integrity Guard | Only new data receiving a PASS judgment is reflected in the duplicate prevention list | Pass | | TC-03 | Integrated Report Branch Storage | Excel generation with Total/New/RaiT_FAIL data separated into individual sheets | Pass | | TC-04 | 42 Hospital Target Access | Verification of normal access and data parsing for target sites, excluding security-blocked sites | Pass | ## Workflow 1. **Trigger:** GitHub Actions scheduler automatically executes every day at 9:00 AM (KST) 2. **Collection:** Dynamic data collection from hospital recruitment pages based on Selenium 3. **QA Verification:** Data integrity check through RaiT logic 4. **Deduplication:** Filtering duplicate notices by comparing with `seen_posts.csv` 5. **Auto-Update:** Automatic commit and push of verified new data to the repository 6. **Report:** Final Excel results and error logs are stored as GitHub Artifacts ## Tech Stack - **Language:** Python 3.10 - **Libraries:** Selenium, Pandas, Openpyxl, Webdriver-manager - **Automation:** GitHub Actions (CI/CD) - **Storage:** CSV (`seen_posts.csv`) and Excel outputs ## Project Structure ```text .github/workflows/ # CI/CD pipeline settings (crawler.yml) top-hospitals/ ├── qa/ # Crawling logic per hospital and RaiT verification module ├── main.py # Main execution engine (QA gateway and data handling) └── seen_posts.csv # History DB for duplicate prevention (auto-updated) ``` ## Troubleshooting Record - **Path Portability Between Environments:** Execution errors occurred due to path differences between local and GitHub Actions virtual servers; this was resolved by creating an environment-independent execution structure using dynamic relative paths based on `os.path`. - **Data Consistency and Duplicate Management:** Duplicate issues during repeated collection were solved by linking `seen_posts.csv` history management with GitHub Actions auto-commits. - **Environment Isolation:** Source code and data outputs were separated by refining `.gitignore`. ## Known Constraints - Samsung Changwon Hospital blocks access from GitHub Actions (Ubuntu/Overseas IP) environments due to strengthened security policies. If data from this hospital is required, it is recommended to run it directly in a local environment; the CI/CD pipeline is configured to automatically skip this target for stability. ## Future Plans - Automation of unit/integration tests through the introduction of Pytest - Advancement of multi-dimensional verification logic to determine contextual errors and text consistency - Automatic transmission of results and RaiT FAIL reports via Slack/Email notification integration ## License and Notices Follows the MIT License. The README states that this project is for the purpose of improving personal technical skills and as a portfolio, and prohibits the commercial use of collected information. It specifies that the copyright of recruitment information belongs to each hospital, and the repository may be turned private upon request from the original copyright holder. It also mentions the application of delay times to minimize server load and the blocking of harmful data through the RaiT gateway.