Building a YouTube AI chat Chrome extension with no backend

Why build a YouTube AI chat Chrome extension

I watch a lot of long technical talks. The usual workflow is bad: pause, copy the URL, paste it into a chat app, wait for it to fetch a transcript, then ask a question that has nothing to do with where I actually am in the video.

I wanted the opposite. Ask "what did he mean by that?" at 34:12 and get an answer about 34:12, with clickable timestamps back into the video. And I didn't want to run a server to do it, because a server means accounts, a database, and me paying for other people's tokens.

So the constraints were:

• Everything runs in the browser, inside YouTube's own page.
• The user's API key never touches the page or any server except OpenAI.
• Every feature works with one key and no setup beyond pasting it.

Architecture: three worlds in one extension

A Manifest V3 extension has separate execution contexts, and the design follows them.

Piece	Runs in	Job
content.jsx	YouTube tab, isolated world	Mounts the React panel in a Shadow DOM inside #secondary-inner
page.js	YouTube's page world	Reads ytInitialPlayerResponse to find caption tracks
background.js	Service worker	Holds the key, calls the OpenAI Responses API, runs tool loops and the WebRTC handshake

The panel talks to the service worker over extension ports. The page never sees the key. That rule shaped almost every later decision, and I wrote it into CONTRIBUTING.md: the service worker is the only place a secret touches the network.

Summaries, highlights and chats are cached per video in chrome.storage.local for the last 30 videos, so you don't pay twice to summarize the same talk.

The playhead is the context

The feature that makes this more than a transcript summarizer is small. Every chat message is wrapped with where the viewer is right now:

export function wrapUser(text, tSec, cues) {
  const window = around(cues, tSec);
  return (
    `WATCHHEAD ${fmt(tSec)} (${Math.floor(tSec)} seconds into the video).\n` +
    (window
      ? `Playing around the viewer right now:\n${window}\n\n`
      : 'No caption line at this exact second.\n\n') +
    `Viewer question:\n${text}`
  );
}

around() takes the caption lines from 45 seconds before to 15 seconds after the playhead and marks the line playing right now with ▶. A usePlayhead() hook listens to the <video> element's timeupdate, seeked, play and pause events, and re-attaches when YouTube swaps the element during navigation.

The model answers with [12:34] style timestamps, and the UI turns them into links that seek the player. It sounds trivial. In practice it is the difference between "summarize this video" and "explain this exact moment", and the second one is what I actually wanted.

The blank-answer bug and the Responses API

The most useful engineering work in this project was not a feature. It was a bug where the chat sometimes showed an empty answer, or spun forever.

I reproduced it with a mocked Chrome port and valid Responses API payloads, and found three separate defects in the old worker:

Input	Old behavior
response.completed with the final text but no text deltas	Reported done, answer thrown away
response.incomplete because of max_output_tokens	Reported as success
Non-streaming JSON with output[].content[].text	Read the wrong field, returned empty text

There were more under those: reasoning items dropped during search continuation, a tool loop that could run out of turns and still claim success, and a Stop button that didn't settle its promise.

The fix lives in src/lib/responses.mjs. It uses eventsource-parser for SSE framing, requires a terminal event before calling a response done, extracts text and refusals from the canonical message items, and turns incomplete or failed streams into errors the user can see. Search continuation now replays every output item, including encrypted reasoning, because the requests use store: false. The search loop is capped at four turns and keeps the last one for the answer. A three-minute deadline stops anything that hangs.

All of it is covered by node:test suites with mocked providers: 28 protocol, worker and bridge tests plus render checks. No test makes a paid call or needs a key. The write-up is in docs/RESPONSE-REPAIR.md.

The lesson I took away: with streaming LLM APIs, "the stream ended" and "the answer is complete" are different events. Treat EOF without a terminal event as a failure.

Live voice over WebRTC

The Live tab is full-duplex voice with OpenAI's Realtime models. You talk, it talks back, and you can interrupt it.

The first attempt failed in an instructive way. Picking a realtime model in the normal chat returned "This is not a chat model", because realtime models live behind a different API. So voice became its own tab with its own model setting.

The transport choice was forced by the browser. The WebSocket version of the Realtime API needs an Authorization header on the handshake, and browsers can't set headers on a WebSocket. WebRTC works: the extension sends an SDP offer to /v1/realtime/calls, and that single key-bearing HTTP call happens in the service worker. Audio then flows over RTP, and control events go over a data channel.

The session also gets the video title, the transcript around the playhead, and the exact second. A Sync button re-sends that context mid-call, so the voice stays on the same moment as the viewer. The mic is torn down on tab switch, video change and unmount, so there is never an orphaned microphone.

Pluggable web search inside MV3 rules

Chat can search the web through a small provider registry:

• DuckDuckGo (default): no key, scraped from the HTML endpoints in the service worker.
• OpenAI native: the hosted web_search tool, billed per search.
• SearXNG: your own instance.
• Off: answers come from the video only.

SearXNG was the interesting one. Its URL is different for every user, so it can't be a static host permission. The manifest declares optional_host_permissions for all hosts, which grants nothing at install time. When you connect your instance, the popup asks Chrome for that one origin, runs a live test query, and only then saves the URL. Once granted, service worker fetches skip CORS, so SearXNG's missing CORS headers don't matter. The repo ships a one-command installer that runs SearXNG bound to loopback with the JSON API on.

What I'd flag before you use it

• DuckDuckGo scraping is fragile by nature. The code detects bot-check pages and says so instead of returning garbage, but it can break when the HTML changes.
• The tests use mocked responses. A real Chrome plus OpenAI run is still the final check for any given model.
• It is unofficial and not affiliated with YouTube or OpenAI.

Building this pushed me deeper into the same patterns I used in the erxes AI agents plugin: tool loops, budgets on turns, and never trusting a model response until it is actually finished. It also showed me how far a "no backend" product can go. Earlier this month I wrote about OpenClaw 2.0 reusing credentials you already have. BYO key is the same idea at a smaller scale: the user owns the account, and the software just borrows it.

FAQ

Is ChatYouTube free?

The extension is free and MIT-licensed. You pay OpenAI directly for what you use with your own key.

Does ChatYouTube send my data to a server?

Only to OpenAI and the search provider you pick. Requests go from the tab to the extension's service worker, then to api.openai.com. There is no backend in between.

How do I install it?

Clone the repo, run npm install and npm run build, then load the dist/ folder in chrome://extensions with Developer mode on.

---

Sources

• Amartuvshins0404/chatyoutube on GitHub, including docs/REALTIME.md, docs/SEARCH.md and docs/RESPONSE-REPAIR.md

Cover image: Shixart1985 / Wikimedia Commons