PythonTkinterFlaskChromaDBPiper TTSWhisperOpenAI APISQLite

Jarvis AI Assistant

A learning-focused AI assistant project exploring voice synthesis, vector databases, and voice commands. Inspired by JARVIS from Iron Man.

Cooper Howling
Status: active
Source Code

A personal learning project exploring AI integration, voice synthesis, and natural language processing. Inspired by JARVIS from Iron Man, with gamification elements from Solo Leveling for health and fitness tracking.

Status: Still in active development — new features and integrations being added regularly.

Overview

Jarvis started as an experiment to learn how different AI technologies work together—voice synthesis, vector databases, natural language processing, and desktop UI frameworks. The goal was to build something like Tony Stark’s JARVIS: a voice-controlled assistant that actually understands context and can execute real commands.

Later, I added a gamification layer inspired by Solo Leveling’s progression system—turning physical and health tracking into an RPG-style experience with XP, levels, and quests. The system provides improvement recommendations based on your tracked data, making fitness tracking more engaging and motivating.

Key Features

  • Authentic JARVIS Voice — Paul Bettany-style voice using Piper TTS with enhanced British accent
  • Voice Commands — Natural language processing for hands-free control
  • Google Integration — Voice-controlled Google Calendar and Gmail access
  • Email Management — Send and read emails via voice commands
  • Desktop Application — Modern Tkinter-based UI with tabbed interface
  • Memory System — Vector storage with ChromaDB for contextual conversations
  • Fitness & Health Tracking — Workout logging, nutrition monitoring, recovery tracking
  • RPG Gamification — Solo Leveling-inspired progression system with XP, levels, and quests
  • Improvement Recommendations — AI-powered suggestions based on tracked health data
  • Web Interface — Flask-based dashboard and API
  • Task Management — Automated task execution and scheduling

Architecture

  • Backend: Python 3.8+ with modular architecture (50+ modules)
  • Desktop UI: Tkinter with minimalist neon theme and semi-transparent design
  • Web Interface: Flask with progressive web app capabilities
  • Voice System: Piper TTS for synthesis, Whisper for recognition
  • Storage: ChromaDB for vector storage, SQLite for structured data
  • AI Integration: OpenAI API for language models and natural language understanding
  • Testing: Comprehensive E2E test suite with pytest

Technical Highlights

The project demonstrates clean modular architecture with domain-organized modules (fitness, voice, calendar, memory, etc.). The desktop application features intelligent memory management with a 10GB threshold, ensuring optimal performance. The E2E testing system validates all functionality automatically, from application startup to voice interaction to GUI responsiveness.

What I Learned

This project was all about exploring how modern AI technologies integrate:

  • Voice Synthesis — Learning Piper TTS and achieving natural-sounding speech with proper pacing and British accent
  • Vector Databases — Understanding ChromaDB for semantic memory and contextual retrieval
  • API Integration — Connecting to Google Calendar and Gmail APIs for voice-controlled productivity
  • Natural Language Processing — Parsing voice commands and extracting intent from casual speech
  • Gamification Design — Translating Solo Leveling’s progression system into real-world health tracking
  • Data-Driven Recommendations — Building AI models that analyze fitness patterns and suggest improvements
  • Desktop UI Development — Building responsive Tkinter applications with memory optimization
  • Modular Architecture — Organizing 50+ modules into a maintainable codebase
  • OAuth & Authentication — Implementing secure Google OAuth flows for API access

The biggest challenge was making all these technologies work together seamlessly. Getting the voice recognition to accurately parse commands and then execute the right API calls required a lot of iteration. The Google integrations were particularly tricky—handling OAuth tokens, managing API rate limits, and parsing email/calendar data into something the voice system could read back naturally.

The gamification layer added another dimension—designing a progression system that actually motivates real-world behavior change. The improvement recommendations needed to be specific enough to be useful but general enough to adapt to different fitness levels and goals.