Skip to content
Audio Notes

How it works

Audio Notes turns a voice recording into a transcript and a short summary. This page explains how, and why each part is built the way it is. The source code is on GitHub.

How a recording moves through the systemThe browser starts an upload with the API, sends the file straight to storage, and finishes the upload with the API. The API queues a job; a worker has Gnani transcribe the file from a signed storage link and a language model write the summary. The API keeps the job's state in Postgres and the browser polls it.Browserthe web appAPIchecks, records, answersPostgresevery job's stateStorageprivate bucketQueuejobs waitingWorkerone short step at a timeGnanispeech to textGroqwrites the summary13824567signed link

From recording to notes

  1. The browser asks the API to start an upload. The API checks the file type and size, records a new job in Postgres, and returns a signed link to a private storage bucket.
  2. The browser sends the file straight to storage with that link. The file never passes through our servers.
  3. When the last byte is sent, the browser asks the API to finish. The API checks that the file really is in storage at the size that was declared, and only then queues the job.
  4. The job waits in a queue until a worker is free.
  5. The worker takes one short step and records the result before doing anything else. A step is small enough that restarting the worker loses nothing.
  6. The worker gives Gnani a short-lived link to the file, not the file itself, and asks for a transcript. It checks back every 10 seconds until Gnani is done, then saves the transcript.
  7. The worker sends the transcript to a language model, which writes a short summary in a fixed structure. A long transcript is summarised in parts and the parts are merged.
  8. The page asks the API for the job's state every few seconds and shows whatever the database says. A refreshed page rebuilds itself from the same answer.

Why the file goes straight to storage

Recordings can be large, and a server that relays them has to hold every connection open for as long as the upload takes. So the API never touches the audio: it signs a link, and the browser sends the bytes to the bucket itself. The signature covers the file type and exact size, so storage refuses anything that differs from what was checked, and the link expires after 15 minutes. Uploads are limited to 2 GB.

The bucket is private. The only ways to reach a file are a signed link the API hands to you for playback, and the short-lived link given to Gnani for transcription.

Why batch transcription, and how long recordings are handled

Gnani's synchronous speech endpoint accepts about a minute of audio. Its batch service takes recordings up to four hours long and fetches the file from a link, which is what makes the direct-to-storage design work. A transcription can take a while, so nothing waits for it.

Long transcripts are also too big for one request to a language model, and the model has per-minute limits. So the transcript is split at word boundaries, each part is summarised in its own step, and the part summaries are merged a few at a time. Progress is saved after every step, so a restart carries on from the next part.

What happens now and what happens later

Only quick work happens while you wait: checking the file type and size, creating the job, signing a link, and confirming the upload arrived. Transcribing and summarising happen in the background, and no request waits for them. You can close the page and come back; the job keeps going.

How progress is shown

The upload bar is the number of bytes that have actually left your browser. After that, the speech service does not say how far along it is, so the page shows which stage is running (upload, transcribe, summarize, ready) and never a made-up percentage. To show that a long job is still alive, it also shows when the job was last touched by the worker.

When something goes wrong

  • A temporary problem, such as the speech service being busy, is retried a few times with longer pauses, and polling gives up after two hours. A problem that retrying cannot fix, such as a corrupt file or silence, is shown straight away with a plain explanation.
  • Every failure says which stage failed and what you can do. If retrying can help there is a button for it; if not, the page says so.
  • If only the summary fails, the transcript is kept and readable, and retrying repeats the summary without transcribing the recording again.
  • A transcription is never started twice for the same attempt. The worker claims the job first, saves Gnani's job id the moment it has it, and does not blindly repeat a request whose outcome it cannot know.

What is stored

Postgres holds one row per recording: the file name, size, type and language, the job's status, the transcription job's id and status, the transcript, the summary, any error, and timestamps. The audio itself stays in storage, never in the database, and is deleted automatically two hours after the upload begins: a timer is set for each upload, and a cleanup that runs once a day removes any that were missed. The transcript and summary are kept.

There are no accounts. Your browser makes up a random id and keeps it; the server stores only a hash of it and shows you the recordings that match. That keeps visitors' uploads private from each other, but it is not a login: clearing this site's data, or using another browser, starts an empty history.

Where it runs

The web app, the API and the step runner are separate processes that share only Postgres, the queue and storage. This deployment runs them as follows.

  • The web app is a Next.js site on Vercel.
  • The API (api, public) and the step runner (steps, private) are two Google Cloud Run services built from one image, in Mumbai. Only the queue can call steps.
  • The queue is a Google Cloud Tasks queue, job-steps. It calls steps when each step is due and tries again if a step does not answer, so there is no always-on worker: nothing runs, or costs anything, while nobody is using it.
  • A Google Cloud Scheduler job calls steps once a day, at 03:00 India time, to delete any recording that is past its time. It is a safety net: each upload also has its own deletion scheduled in the queue.
  • The database is Neon Postgres in Singapore, the closest region it offers to Mumbai.
  • Recordings are in a private AWS S3 bucket in Mumbai. Your browser uploads to it directly.

In development the same code runs locally with Postgres and Redis in Docker, and a Celery worker takes the place of Cloud Tasks.

Limits

  • Recordings longer than four hours are not supported. They would need to be split into pieces first.
  • This public demo accepts files up to 200 MB and five uploads per browser in 24 hours, so that it cannot be run up by one visitor.
  • Summaries are written by a model from the transcript. They are instructed not to add anything, but they can still be wrong, which is why the transcript is always shown next to them.
  • Transcripts are automatic, so names, numbers and heavily accented speech can be misheard.
  • An upload stops if you close the tab, and cannot be resumed. Start it again.
  • The recording itself is gone after two hours, so it cannot be played or processed again after that. Transcripts and summaries cannot be deleted or renamed yet.
  • The speech service allows about one call a second, so many recordings at once wait their turn.

What would change with real users

  • Accounts and proper authorization in place of the browser id, plus per-user limits.
  • Resumable, multi-part uploads, so a dropped connection does not restart a large file.
  • Webhooks from the speech service instead of checking back on a timer.
  • Splitting very long audio, and cleaning up old files automatically.
  • Scanning uploads, and dashboards and alerts on failures and queue length.