clicky

Clicky is an AI-powered teaching assistant that resides as a companion near your cursor on macOS. It can observe your screen, engage in conversation, and visually point to specific interface elements. This repository contains the open-source version of Clicky, released under an MIT license, allowing developers to tinker, customize, or build upon it. ## Features - **Screen awareness**: Uses ScreenCaptureKit to capture and analyze screen content. - **Voice interaction**: Push-to-talk voice capture with real-time transcription via AssemblyAI. - **Conversational AI**: Streams transcripts and screenshots to Claude for context-aware responses. - **Text-to-speech**: Plays responses through ElevenLabs TTS. - **Cursor pointing**: Claude can embed special tags to move the cursor to specific UI elements across multiple monitors. - **Menu bar app**: Runs as a lightweight menu bar application with a control panel and a full-screen overlay window. ## Architecture - **Swift app** (`leanring-buddy/`): Contains the main application logic, including the central state machine, panel UI, Claude streaming client, ElevenLabs TTS client, overlay window, and AssemblyAI transcription providers. - **Cloudflare Worker** (`worker/`): A small proxy that securely holds API keys and routes requests to `/chat`, `/tts`, and `/transcribe-token` endpoints. - **CLAUDE.md**: A comprehensive architecture document intended for AI agents to understand the codebase. ## Setup ### Prerequisites - macOS 14.2+ (for ScreenCaptureKit) - Xcode 15+ - Node.js 18+ (for the Cloudflare Worker) - A Cloudflare account (free tier works) - API keys for Anthropic, AssemblyAI, and ElevenLabs ### Quick Start with Claude Code 1. Install and run Claude Code. 2. Paste the provided prompt to clone the repo, read `CLAUDE.md`, and get guided through the setup. ### Manual Setup 1. **Set up the Cloudflare Worker**: Navigate to `worker/`, run `npm install`, then add your API keys as secrets using `npx wrangler secret put`. Set the ElevenLabs voice ID in `wrangler.toml`. Deploy with `npx wrangler deploy`. 2. **Run the Worker locally** (optional): Use `npx wrangler dev` and create a `.dev.vars` file with your keys. Update the proxy URLs in the Swift code to point to `http://localhost:8787`. 3. **Update proxy URLs**: Search for `clicky-proxy` in the Swift code and replace with your Worker URL. 4. **Open in Xcode**: Open `leanring-buddy.xcodeproj`, select the scheme, set your signing team, and run with Cmd+R. ### Permissions Required - Microphone (for push-to-talk) - Accessibility (for global keyboard shortcut) - Screen Recording (for screenshots) - Screen Content (for ScreenCaptureKit) ## Project Structure ``` leanring-buddy/ # Swift source CompanionManager.swift # Central state machine CompanionPanelView.swift # Menu bar panel UI ClaudeAPI.swift # Claude streaming client ElevenLabsTTSClient.swift # Text-to-speech playback OverlayWindow.swift # Blue cursor overlay AssemblyAI*.swift # Real-time transcription BuddyDictation*.swift # Push-to-talk pipeline worker/ # Cloudflare Worker proxy src/index.ts # Three routes: /chat, /tts, /transcribe-token CLAUDE.md # Full architecture doc ``` ## Contributing PRs are welcome. If using Claude Code, it can read `CLAUDE.md` to understand the codebase and assist with feature development or bug fixes. Feedback can be directed to the author on X (@farzatv).