hey, sanjeet.

Explore Sanjeet’s work

Jump to projects, pages, or hiring context.

All work
Independent logoIndependent
In developmentIndependent · Voice agent platform · Pre-launch prototype v0.1.0

Neo-AI

One voice you can talk to, interview with, learn from and practise a language with.

Neo-AI is a voice-first agent platform with four capabilities on one harness. General: just talk, about the weather, the news, a joke, a story, or 'tell me about Steve Jobs'. Interview: a professional mock interview across technical, coding and system-design rounds, scored against an evidence ledger rather than how the conversation felt. Tutor: a custom course on any topic you pick, with follow-ups that help you upskill and stay consistent. Language learning: Kannada, French, Japanese and more, by speaking. It runs fully local on a Mac with Ollama, Whisper and Piper.

Role
Founder-engineer, independent
Status
Pre-launch prototype, single user, runs locally
Runs on
Ollama · faster-whisper · Piper on an M3 Mac
Codebase
~10.4k lines, Python + TypeScript

~1s

To first spoken word

VAD ≤ 900ms · STT ≤ 400ms · first token ≤ 300ms · TTS ≤ 100ms

18

Competencies

Each with rubric, signals, red flags, seniority range

4

Capabilities

General · Interview · Tutor · Language learning

6

Sandbox languages

Python, JS, TS, Java, Go, SQL

3

LLM providers

Local Qwen, OpenAI-compatible, Claude

0

Audio leaving the machine

Local-first by default, no cloud cost

01The problem

Prep chatbots drill your last answer. Real interviews cover the role.

Candidates cannot practise against a realistic interviewer on demand: human mock interviews are expensive, hard to schedule and inconsistent. Text chatbots do not feel like an interview, which is spoken, timed and switches between talk, code and whiteboard. Generic AI prep keeps following up on whatever was just said instead of covering what the role actually requires, and the feedback is vague.

Neo-AI is designed the other way round: decide what evidence a role needs, run spoken and screen rounds to collect it, score each answer against a rubric into a ledger, and tell the candidate which competency is below the bar and why.

One harness, one voice, one memory of you. Capabilities are modules; the relationship is the product.

The vision behind Neo-AI

02My role

Every layer, explicitly.

No hand-waving about “contributed to”. These are the parts I designed, built and operated.

Product & vision
A voice-first companion that grows by capability: talk, interview, tutor, language. Candidate journey, readiness bands and gate rule for the interview.
Harness
Control packs, context fabric, fail-closed tool bus and runtime that every capability shares.
Capabilities
General conversation, professional interview, custom-course tutor and spoken language learning, each a module with its own control pack.
Voice pipeline
Streaming STT → LLM → TTS over one WebSocket, intent handling before the model, sentence-flush speech.
Interview engine
Role intelligence, 18-competency catalog, planner, question selector, coverage controller, evaluator, ledger, scoring.
Tools & sandbox
Ten tools from competency modelling to code execution, hidden tests and whiteboard diagrams; a local subprocess sandbox.
Frontend
React app with AudioWorklet mic capture, energy VAD, gapless playback, captions and scorecard.
Providers
Factory-built LLM, STT and TTS interfaces so Ollama, OpenAI or Claude swap by configuration.

03What it does

Pick a capability by talking. The interview loop: set up, plan, interview, scorecard.

  1. 01Set up
  2. 02Model
  3. 03Plan
  4. 04Interview
  5. 05Screen rounds
  6. 06Scorecard

Real-time voice pipeline

Mic → AudioWorklet downsampling → energy VAD → streaming speech-to-text → intent check → capability turn → sentence-flush text-to-speech → gapless playback, over one WebSocket per session. About a second to the first spoken word.

  • faster-whisper
  • Piper
  • WebSocket
  • PCM16

General: just talk

Everyday conversation by voice: today's weather and temperature, the news, a joke, a bedtime story, or 'tell me about Steve Jobs'. The capability that makes Neo-AI feel like a presence rather than a tool.

Tutor: custom courses

Pick a topic and Neo-AI designs a course for it, teaches it by voice, follows up on what you found hard, and checks in so you stay consistent while you upskill.

Language learning

Learn Kannada, French, Japanese and other languages by speaking: vocabulary, phrases and conversation practice with spoken correction.

Role intelligence

Role, topic or job description → role family (AI/ML, backend, frontend, data, general) → a competency graph drawn from a catalog of 18 competencies.

Interview planner

A deterministic blueprint of rounds sized to the available time, with question types chosen by seniority: knowledge and coding for juniors, design, critique and ambiguity for staff.

Coverage controller

Picks the next move by the evidence still missing, with at most two follow-ups per question, so the loop covers the role instead of rabbit-holing.

Evaluator & evidence ledger

A separate LLM pass scores each answer against a competency rubric into a ledger that is the single source of truth for scoring.

Scoring & readiness bands

Weighted average over tested competencies only; a critical competency below 60 caps the score at 69. Bands from Not Ready to Exceptional, with strengths, hiring risks and recommended preparation.

Coding sandbox

Real code runs with stdin, public and hidden tests, and an 8-second hand-in. Python, JS, TS, Java, Go and SQL in a temp directory with a stripped environment and hard timeouts.

System-design whiteboard

Nodes for clients, APIs, services, databases, queues, caches, load balancers and CDNs. Tools seed, modify and statically inspect the diagram for dangling edges and missing components.

Harness: Control · Context · Agency · Runtime

Control packs define what a capability may use. The context fabric assembles each turn from ten layers with provenance under a token budget. Agency is the fail-closed tool bus. Runtime is the narrow view a capability gets. General, Interview, Tutor and Language learning all plug into it as modules.

  • control pack
  • context fabric
  • tool bus
  • runtime

Swappable providers

LLM, STT and TTS behind abstract interfaces and build factories. Ollama Qwen by default; OpenAI-compatible or Claude by configuration.

04Architecture

Browser mic to a session conductor to a harness that only sees what it is allowed to.

A React app streams microphone audio over a single WebSocket to a FastAPI session. The session feeds speech-to-text, checks intents, hands the turn to the capability through the harness, and streams synthesised speech back sentence by sentence. Each layer depends only on the one below it through factories and abstract bases: the session never imports a capability, and capabilities never touch Whisper, Piper, the socket or the sandbox.

Client

React 18 + Vite browser app

AudioWorklet mic capture 48 → 16 kHz · energy VAD · gapless playback · voice orb · captions · scorecard

API

FastAPI session + harness ⟨Control, Context, Agency, Runtime⟩

  • STT feed → intent regex → capability turn
  • Sentence-flush TTS → widget flush
  • Control packs: allowed tools, intents, context policy
  • Context fabric: 10 layers, provenance, token budget
  • Agency: fail-closed tool bus
  • Capabilities: general · interview · tutor · language
  • Interview capability state machine
  • 10 tools → sandbox & whiteboard
  • Memory store (in-memory today)

Ollama qwen2.5

default LLM; OpenAI or Claude by config

faster-whisper

streaming speech-to-text

Piper

22.05 kHz text-to-speech

Sandbox runner

6 languages, hard timeouts

Widgets

practice · coding · whiteboard

  • Python 3.11
  • FastAPI
  • Pydantic v2
  • WebSockets
  • Ollama
  • faster-whisper
  • Piper
  • React 18
  • Vite
  • TypeScript
  • Tailwind
  • Zustand
  • pytest

05Skills used

Everything the build touched, by layer.

Voice AI

  • Streaming STT
  • Streaming TTS
  • Energy VAD
  • AudioWorklet capture
  • Gapless playback
  • Latency budgeting

Agent harness

  • Control packs
  • Context fabric
  • Fail-closed tool bus
  • Capability interface
  • Intent routing
  • Memory store

Interview assessment

  • Competency modelling
  • Interview planning
  • Coverage control
  • Rubric evaluation
  • Evidence ledger
  • Readiness scoring

Tools & sandbox

  • Code execution
  • Hidden tests
  • Diagram tools
  • Architecture inspection
  • Process isolation

Frontend

  • React 18
  • Vite
  • TypeScript
  • Zustand
  • Framer Motion
  • WebSocket protocol

Providers & ops

  • Ollama
  • OpenAI-compatible
  • Anthropic Claude
  • pydantic-settings
  • pytest
  • Bootstrap scripts

06Hard problems

Where the engineering actually lived.

  1. 01

    A one-second budget to the first spoken word

    VAD within 900ms, speech-to-text within 400ms, first model token within 300ms, speech synthesis within 100ms per sentence. Sentence-flush TTS and gapless scheduling on the client are what make a local model feel conversational.

  2. 02

    Control phrases that never become content

    Pause, resume, skip, end and 'I don't know' are matched on start-anchored phrases before the model sees the turn. Voice control still works while a coding widget blocks spoken answers.

  3. 03

    Scoring from evidence, not vibes

    A separate evaluator writes rubric scores into a ledger. Scoring averages only the competencies that were actually tested, and a critical competency under 60 caps the result at 69. Untested areas do not drag the score down.

  4. 04

    Coverage over rabbit-holes

    The coverage controller selects the next question by missing evidence and allows two follow-ups at most, so a 30-minute loop covers the role's competency graph.

  5. 05

    Running candidate code honestly and safely

    Code runs for real with hidden tests kept server-side and redacted to pass or fail. The sandbox is process isolation for a single trusted user; a real jail is the gate before any multi-tenant launch.

07Engineering decisions

Why it is built this way.

Why a separate evaluator and an evidence ledger instead of asking the interviewer model for a score?

The conversational model is optimised to keep the interview moving, not to grade. A separate LLM pass scores one competency at a time against its rubric and writes JSON into an evidence ledger. The final score is computed from that ledger, so it reflects what was demonstrated rather than how the conversation felt.

Why cap follow-ups at two per question?

The coverage controller asks 'what evidence is still missing for this role?' rather than 'what follow-up can I ask?'. Limiting follow-ups forces the loop to cover the competency graph instead of going deep on one answer, which is what a real interview loop does and what prep chatbots fail at.

Why run intents as regex before the LLM?

'Pause', 'skip' and 'I'm done' are control, not content. Matching them on start-anchored phrases before anything reaches the model means a control phrase can never be scored as an answer, and voice control keeps working even while a coding widget blocks spoken turns.

Why does the tool bus fail closed?

Each capability declares its allowed tools and intents in a control pack. Only those tools are instantiated; calling anything else raises. The interview capability can run code and edit diagrams, and a future tutor capability simply will not have those tools. Safety comes from construction, not from the prompt.

Why local-first providers behind factories?

The default setup runs Ollama, faster-whisper and Piper entirely on a Mac: no cloud cost per session and no audio leaving the machine. Because LLM, STT and TTS each sit behind an interface with a build factory, switching to OpenAI or Claude is one environment variable, and the latency budget can be tuned by swapping model sizes.

Why keep hidden tests on the server?

Code actually runs, and hidden tests stay server-side with results redacted to pass or fail, so a candidate cannot game them. The widget spec strips hidden tests before anything is sent to the browser. A coding widget only closes when tests pass and the evaluator agrees.

08What I'd tell another builder

Harness over model.

What a capability may know, use and do each turn is decided by the harness. The tutor capability must ship with zero edits to Whisper, Piper, Ollama or the session.

Score from a ledger.

Keep the evidence story visible. No hiring-probability claims until scores are calibrated against real interview outcomes.

One harness, four capabilities.

General talk, interview, tutor and language learning share the voice loop, context fabric, tool bus and memory. A new capability is a module and a control pack, not a new app.

09Ownership

Thesis, harness, voice pipeline, four capabilities, tools, sandbox and frontend: an independent build.

  • Product thesis
  • Harness architecture
  • Voice pipeline
  • General conversation
  • Interview engine
  • Tutor courses
  • Language learning
  • Competency catalog
  • Evaluator & ledger
  • Tool bus & 10 tools
  • Sandbox runner
  • React frontend
  • Provider factories

Contribution

  • Product thesis
  • Agent harness
  • Real-time voice pipeline
  • General, interview, tutor & language capabilities
  • Interview engine
  • Evidence-based scoring
  • Coding sandbox & whiteboard tools
  • React frontend
  • Provider abstraction
Independent logo

Built at Independent

Want this kind of ownership on your team?

Get in touch