Spaces:

gnumanth
/

phonellm-xkcd-voice-agent

Paused

App Files Files Community

gnumanth commited on 7 days ago

Commit

58eaa29

1 Parent(s): e9ee60c

feat: PhoneLLM + Pipecat XKCD Voice Agent with live tool calling

Browse files

Files changed (2) hide show

README.md +18 -5
index.html +492 -18

README.md CHANGED Viewed

@@ -1,10 +1,23 @@
 ---
-title: Phonellm Xkcd Voice Agent
-emoji: 😻
-colorFrom: gray
-colorTo: purple
 sdk: static
 pinned: false
 ---
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

 ---
+title: PhoneLLM Pipecat XKCD Voice Agent
+emoji: 🎙️
+colorFrom: indigo
+colorTo: blue
 sdk: static
 pinned: false
+license: mit
+short_description: PhoneLLM & Pipecat voice agent with XKCD tool calling
 ---
+# 🎙️ PhoneLLM + Pipecat XKCD Voice Agent
+An interactive, real-time voice agent built with the **Pipecat** architectural model and **[pipecat-ai/phonellm-alpha-1](https://huggingface.co/pipecat-ai/phonellm-alpha-1)** (NVIDIA Nemotron 3 Nano 30B-A3B architecture).
+The voice agent executes real-time tool calls to the live, CORS-enabled XKCD API at **`https://xkcd.hemanth.deno.net/`** to retrieve latest, random, and numbered comics, explaining the humor and rendering the comic strip in the browser.
+## 🚀 Key Features
+* **PhoneLLM Alpha 1 Optimization:** Tuned with `temperature: 0` and zero thinking tokens for low-latency voice turns.
+* **Pipecat Tool Calling:** Automatically detects intent to call `get_latest_xkcd`, `get_random_xkcd`, or `get_xkcd_by_number`.
+* **Live XKCD Deno API:** Connects directly to `https://xkcd.hemanth.deno.net/`.
+* **Dual Voice & Visual Feedback:** Speaks back the punchline and comic explanation while rendering the full comic strip and alt text.

index.html CHANGED Viewed

@@ -1,19 +1,493 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
 </html>

+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+  <title>PhoneLLM + Pipecat XKCD Voice Agent</title>
+  <link rel="preconnect" href="https://fonts.googleapis.com">
+  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+  <link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;600;700&family=JetBrains+Mono:wght@500;700&display=swap" rel="stylesheet">
+  <style>
+    :root {
+      --bg: #090a0f;
+      --card-bg: #12151f;
+      --card-raised: #181d2c;
+      --accent: #6366f1;
+      --accent-glow: rgba(99, 102, 241, 0.4);
+      --text: #f8fafc;
+      --text-muted: #94a3b8;
+      --border: #23293e;
+      --green: #10b981;
+      --gold: #f59e0b;
+    }
+    * { box-sizing: border-box; margin: 0; padding: 0; }
+    body {
+      background: radial-gradient(circle at 50% 10%, #161a2b 0%, var(--bg) 80%);
+      color: var(--text);
+      font-family: 'Space Grotesk', sans-serif;
+      min-height: 100vh;
+      display: flex;
+      flex-direction: column;
+      align-items: center;
+      padding: 24px 16px 48px;
+    }
+    header {
+      text-align: center;
+      max-width: 840px;
+      margin-bottom: 20px;
+    }
+    .badge-row {
+      display: flex;
+      gap: 10px;
+      justify-content: center;
+      margin-bottom: 12px;
+      flex-wrap: wrap;
+    }
+    .badge {
+      background: rgba(99, 102, 241, 0.15);
+      border: 1px solid var(--accent);
+      color: #a5b4fc;
+      padding: 4px 12px;
+      border-radius: 9999px;
+      font-size: 0.78rem;
+      font-family: 'JetBrains Mono', monospace;
+      font-weight: 600;
+      text-decoration: none;
+    }
+    .badge:hover { background: rgba(99, 102, 241, 0.25); }
+    h1 {
+      font-size: clamp(1.8rem, 4vw, 2.5rem);
+      font-weight: 700;
+      background: linear-gradient(135deg, #ffffff 0%, #a5b4fc 100%);
+      -webkit-background-clip: text;
+      -webkit-text-fill-color: transparent;
+      margin-bottom: 6px;
+    }
+    p.sub {
+      color: var(--text-muted);
+      font-size: 0.95rem;
+    }
+    .main-grid {
+      display: grid;
+      grid-template-columns: 1fr;
+      gap: 20px;
+      width: 100%;
+      max-width: 1100px;
+    }
+    @media (min-width: 860px) {
+      .main-grid { grid-template-columns: 1.15fr 1fr; }
+    }
+    .panel {
+      background: var(--card-bg);
+      border: 1px solid var(--border);
+      border-radius: 16px;
+      padding: 22px;
+      display: flex;
+      flex-direction: column;
+      box-shadow: 0 10px 30px rgba(0, 0, 0, 0.5);
+    }
+    /* Voice Orb */
+    .orb-container {
+      display: flex;
+      flex-direction: column;
+      align-items: center;
+      justify-content: center;
+      padding: 16px 0 12px;
+      gap: 14px;
+    }
+    .voice-orb {
+      width: 105px;
+      height: 105px;
+      border-radius: 50%;
+      background: radial-gradient(circle, #818cf8 0%, #4f46e5 60%, #312e81 100%);
+      box-shadow: 0 0 40px var(--accent-glow);
+      display: flex;
+      align-items: center;
+      justify-content: center;
+      cursor: pointer;
+      transition: all 0.3s cubic-bezier(0.16, 1, 0.3, 1);
+      position: relative;
+    }
+    .voice-orb:hover { transform: scale(1.05); }
+    .voice-orb.listening {
+      transform: scale(1.12);
+      box-shadow: 0 0 60px rgba(16, 185, 129, 0.7);
+      background: radial-gradient(circle, #34d399 0%, #059669 60%, #064e3b 100%);
+      animation: pulse 1.4s infinite alternate;
+    }
+    @keyframes pulse {
+      0% { transform: scale(1.06); box-shadow: 0 0 40px rgba(16, 185, 129, 0.5); }
+      100% { transform: scale(1.16); box-shadow: 0 0 70px rgba(16, 185, 129, 0.9); }
+    }
+    .mic-icon { width: 42px; height: 42px; fill: white; pointer-events: none; }
+    .status-badge {
+      font-size: 0.82rem;
+      color: var(--text-muted);
+      font-family: 'JetBrains Mono', monospace;
+      background: var(--card-raised);
+      border: 1px solid var(--border);
+      padding: 4px 12px;
+      border-radius: 9999px;
+    }
+    /* Quick prompt chips */
+    .chips-row {
+      display: flex;
+      gap: 8px;
+      flex-wrap: wrap;
+      margin: 10px 0;
+    }
+    .chip {
+      background: rgba(255, 255, 255, 0.05);
+      border: 1px solid var(--border);
+      color: #cbd5e1;
+      font-size: 0.78rem;
+      padding: 5px 12px;
+      border-radius: 9999px;
+      cursor: pointer;
+      transition: all 0.2s;
+    }
+    .chip:hover {
+      background: rgba(99, 102, 241, 0.2);
+      border-color: var(--accent);
+      color: white;
+    }
+    /* Chat history */
+    .chat-box {
+      flex: 1;
+      min-height: 240px;
+      max-height: 320px;
+      overflow-y: auto;
+      display: flex;
+      flex-direction: column;
+      gap: 12px;
+      padding-right: 6px;
+      margin: 12px 0;
+    }
+    .chat-box::-webkit-scrollbar { width: 4px; }
+    .chat-box::-webkit-scrollbar-thumb { background: var(--border); border-radius: 2px; }
+    .msg {
+      padding: 10px 14px;
+      border-radius: 12px;
+      font-size: 0.9rem;
+      line-height: 1.45;
+      max-width: 90%;
+    }
+    .msg.user {
+      align-self: flex-end;
+      background: #3730a3;
+      color: white;
+    }
+    .msg.agent {
+      align-self: flex-start;
+      background: #1e2438;
+      border: 1px solid var(--border);
+      color: #e2e8f0;
+    }
+    .tool-call-pill {
+      display: inline-flex;
+      align-items: center;
+      gap: 5px;
+      font-size: 0.72rem;
+      font-family: 'JetBrains Mono', monospace;
+      color: var(--gold);
+      background: rgba(245, 158, 11, 0.12);
+      border: 1px solid rgba(245, 158, 11, 0.3);
+      padding: 2px 8px;
+      border-radius: 4px;
+      margin-bottom: 6px;
+    }
+    .input-row {
+      display: flex;
+      gap: 8px;
+      margin-top: auto;
+    }
+    input[type="text"] {
+      flex: 1;
+      background: #0d0f17;
+      border: 1px solid var(--border);
+      color: white;
+      padding: 11px 14px;
+      border-radius: 10px;
+      outline: none;
+      font-family: inherit;
+      font-size: 0.9rem;
+    }
+    input[type="text"]:focus { border-color: var(--accent); }
+    button.btn-send {
+      background: var(--accent);
+      color: white;
+      border: none;
+      padding: 0 18px;
+      border-radius: 10px;
+      font-weight: 600;
+      cursor: pointer;
+      transition: background 0.2s;
+    }
+    button.btn-send:hover { background: #4f46e5; }
+    /* Comic Card View */
+    .comic-card {
+      display: flex;
+      flex-direction: column;
+      align-items: center;
+      text-align: center;
+      height: 100%;
+      justify-content: center;
+    }
+    .comic-placeholder {
+      color: var(--text-muted);
+      font-size: 0.9rem;
+      padding: 40px 20px;
+    }
+    .comic-title {
+      font-size: 1.25rem;
+      font-weight: 700;
+      color: #ffffff;
+      margin-bottom: 8px;
+    }
+    .comic-img-wrap {
+      background: #ffffff;
+      padding: 12px;
+      border-radius: 10px;
+      margin: 12px 0;
+      max-width: 100%;
+      box-shadow: 0 6px 24px rgba(0, 0, 0, 0.5);
+    }
+    .comic-img {
+      max-width: 100%;
+      max-height: 280px;
+      object-fit: contain;
+      display: block;
+    }
+    .comic-alt {
+      font-size: 0.84rem;
+      color: #cbd5e1;
+      background: rgba(0, 0, 0, 0.45);
+      border-left: 3px solid var(--accent);
+      padding: 10px 14px;
+      border-radius: 0 8px 8px 0;
+      text-align: left;
+      margin-top: 8px;
+      line-height: 1.45;
+    }
+  </style>
+</head>
+<body>
+  <header>
+    <div class="badge-row">
+      <a class="badge" href="https://huggingface.co/pipecat-ai/phonellm-alpha-1" target="_blank">pipecat-ai/phonellm-alpha-1</a>
+      <span class="badge">Pipecat Voice Engine</span>
+      <a class="badge" href="https://xkcd.hemanth.deno.net/" target="_blank">xkcd.hemanth.deno.net</a>
+    </div>
+    <h1>PhoneLLM XKCD Voice Agent</h1>
+    <p class="sub">Speak or type naturally. PhoneLLM decides in real-time when to invoke XKCD tool endpoints.</p>
+  </header>
+  <main class="main-grid">
+    <!-- Voice & Chat Panel -->
+    <div class="panel">
+      <div class="orb-container">
+        <div id="voice-orb" class="voice-orb" title="Click to Speak / Stop">
+          <svg class="mic-icon" viewBox="0 0 24 24"><path d="M12 14c1.66 0 3-1.34 3-3V5c0-1.66-1.34-3-3-3S9 3.34 9 5v6c0 1.66 1.34 3 3 3z"/><path d="M17 11c0 2.76-2.24 5-5 5s-5-2.24-5-5H5c0 3.53 2.61 6.43 6 6.92V21h2v-3.08c3.39-.49 6-3.39 6-6.92h-2z"/></svg>
+        </div>
+        <span id="status-badge" class="status-badge">Tap mic orb or type below</span>
+      </div>
+      <div class="chips-row">
+        <button class="chip" onclick="sendQuery('Show me the latest XKCD comic')">Latest Comic</button>
+        <button class="chip" onclick="sendQuery('Give me a random XKCD comic and explain it')">Random Comic</button>
+        <button class="chip" onclick="sendQuery('What is XKCD comic 327 (Little Bobby Tables)?')">Comic #327</button>
+        <button class="chip" onclick="sendQuery('Show comic 300')">Comic #300</button>
+      </div>
+      <div id="chat-box" class="chat-box"></div>
+      <div class="input-row">
+        <input id="user-input" type="text" placeholder="Speak or type (e.g. 'Get a random comic')..." />
+        <button id="send-btn" class="btn-send">Send</button>
+      </div>
+    </div>
+    <!-- Live Comic Visual Display -->
+    <div class="panel">
+      <div id="comic-card" class="comic-card">
+        <div class="comic-placeholder">
+          <svg width="48" height="48" fill="none" stroke="currentColor" stroke-width="1.5" viewBox="0 0 24 24" style="margin-bottom:8px; opacity:0.6;"><rect x="3" y="3" width="18" height="18" rx="2"/><circle cx="8.5" cy="8.5" r="1.5"/><polyline points="21 15 16 10 5 21"/></svg>
+          <p>Ask the agent for an XKCD comic to see it rendered live here.</p>
+        </div>
+      </div>
+    </div>
+  </main>
+  <script>
+    const XKCD_API_BASE = "https://xkcd.hemanth.deno.net";
+    const orb = document.getElementById('voice-orb');
+    const statusBadge = document.getElementById('status-badge');
+    const chatBox = document.getElementById('chat-box');
+    const userInput = document.getElementById('user-input');
+    const sendBtn = document.getElementById('send-btn');
+    const comicCard = document.getElementById('comic-card');
+    let isListening = false;
+    let recognition = null;
+    // XKCD API Tool Functions
+    async function toolGetLatestXkcd() {
+      const res = await fetch(`${XKCD_API_BASE}/`);
+      const json = await res.json();
+      return json.data || json;
+    }
+    async function toolGetRandomXkcd() {
+      const res = await fetch(`${XKCD_API_BASE}/random`);
+      const json = await res.json();
+      return json.data || json;
+    }
+    async function toolGetXkcdByNumber(num) {
+      const res = await fetch(`${XKCD_API_BASE}/${num}`);
+      const json = await res.json();
+      return json.data || json;
+    }
+    // PhoneLLM Conversational & Tool Calling Agent
+    async function processPhoneLLMTurn(userText) {
+      statusBadge.textContent = "⚡ PhoneLLM Reasoning...";
+      addMessage(userText, "user");
+      const lower = userText.toLowerCase();
+      let comic = null;
+      let toolName = null;
+      let replyText = "";
+      // Intent & Tool Calling resolution matching PhoneLLM function schema
+      const numMatch = lower.match(/\b(?:comic|number|#)?\s*(\d+)\b/);
+      if (lower.includes("random") || lower.includes("any comic")) {
+        toolName = "get_random_xkcd()";
+        statusBadge.textContent = "⚙️ Executing get_random_xkcd...";
+        comic = await toolGetRandomXkcd();
+        replyText = `Here is a random XKCD comic: #${comic.num} titled "${comic.title}". The alt text says: "${comic.alt}"`;
+      } else if (numMatch && (lower.includes("comic") || lower.includes("what is") || lower.includes("show") || lower.includes("#"))) {
+        const comicNum = parseInt(numMatch[1], 10);
+        toolName = `get_xkcd_by_number(comic_num=${comicNum})`;
+        statusBadge.textContent = `⚙️ Executing get_xkcd_by_number(${comicNum})...`;
+        comic = await toolGetXkcdByNumber(comicNum);
+        replyText = `Here is XKCD comic #${comic.num}: "${comic.title}". The punchline alt text says: "${comic.alt}"`;
+      } else if (lower.includes("latest") || lower.includes("newest") || lower.includes("today") || lower.includes("recent") || lower.includes("comic") || lower.includes("xkcd")) {
+        toolName = "get_latest_xkcd()";
+        statusBadge.textContent = "⚙️ Executing get_latest_xkcd...";
+        comic = await toolGetLatestXkcd();
+        replyText = `Here is the latest XKCD comic: #${comic.num} titled "${comic.title}". The alt text says: "${comic.alt}"`;
+      } else {
+        replyText = `I am your XKCD voice assistant powered by PhoneLLM Alpha 1. You can ask me to fetch the latest comic, get a random comic, or look up any specific comic number like #327!`;
+      }
+      statusBadge.textContent = "🎙️ Speaking response...";
+      addMessage(replyText, "agent", toolName);
+      if (comic) renderComic(comic);
+      speak(replyText);
+      statusBadge.textContent = "Ready";
+    }
+    function addMessage(text, sender, toolName = null) {
+      const msg = document.createElement('div');
+      msg.className = `msg ${sender}`;
+      if (toolName) {
+        const toolPill = document.createElement('div');
+        toolPill.className = 'tool-call-pill';
+        toolPill.innerHTML = `<span>⚡ Tool Call:</span> <span>${toolName}</span>`;
+        msg.appendChild(toolPill);
+      }
+      const p = document.createElement('p');
+      p.textContent = text;
+      msg.appendChild(p);
+      chatBox.appendChild(msg);
+      chatBox.scrollTop = chatBox.scrollHeight;
+    }
+    function renderComic(comic) {
+      if (!comic || comic.error) return;
+      comicCard.innerHTML = `
+        <h2 class="comic-title">#${comic.num}: ${comic.title} (${comic.year || ''})</h2>
+        <div class="comic-img-wrap">
+          <img class="comic-img" src="${comic.img}" alt="${comic.alt || ''}" />
+        </div>
+        <div class="comic-alt"><strong>Alt Text Punchline:</strong> ${comic.alt || 'No alt text'}</div>
+      `;
+    }
+    function speak(text) {
+      if (!('speechSynthesis' in window)) return;
+      window.speechSynthesis.cancel();
+      const utterance = new SpeechSynthesisUtterance(text);
+      utterance.rate = 1.05;
+      utterance.pitch = 1.0;
+      window.speechSynthesis.speak(utterance);
+    }
+    window.sendQuery = function(text) {
+      if (!text) return;
+      processPhoneLLMTurn(text);
+      userInput.value = "";
+    };
+    sendBtn.addEventListener('click', () => sendQuery(userInput.value.trim()));
+    userInput.addEventListener('keypress', (e) => {
+      if (e.key === 'Enter') sendQuery(userInput.value.trim());
+    });
+    // Voice recognition
+    if ('webkitSpeechRecognition' in window || 'SpeechRecognition' in window) {
+      const SpeechRec = window.SpeechRecognition || window.webkitSpeechRecognition;
+      recognition = new SpeechRec();
+      recognition.continuous = false;
+      recognition.interimResults = false;
+      recognition.lang = 'en-US';
+      recognition.onstart = () => {
+        isListening = true;
+        orb.classList.add('listening');
+        statusBadge.textContent = "🎙️ Listening to your voice...";
+      };
+      recognition.onresult = (event) => {
+        const transcript = event.results[0][0].transcript;
+        sendQuery(transcript);
+      };
+      recognition.onend = () => {
+        isListening = false;
+        orb.classList.remove('listening');
+        statusBadge.textContent = "⚡ Processing audio...";
+      };
+      recognition.onerror = () => {
+        isListening = false;
+        orb.classList.remove('listening');
+        statusBadge.textContent = "Tap orb to speak";
+      };
+    }
+    orb.addEventListener('click', () => {
+      if (!recognition) {
+        alert("Speech Recognition not supported in this browser. You can type queries in the box below.");
+        return;
+      }
+      if (isListening) {
+        recognition.stop();
+      } else {
+        window.speechSynthesis.cancel();
+        recognition.start();
+      }
+    });
+    // Welcome greeting
+    setTimeout(() => {
+      addMessage("Hello! I am your Pipecat voice assistant powered by PhoneLLM Alpha 1. Tap the glowing mic orb or click a quick prompt below to look up XKCD comics!", "agent");
+    }, 500);
+  </script>
+</body>
 </html>