Time to First Token

Time to First Token (TTFT) refers to the time between the moment you send a query to an AI service and the moment the first character of its answer appears on screen. It's similar to timing how long it takes for the first side dish to arrive after you place an order at a restaurant. It doesn't measure how long it takes for the whole meal to come out, just the moment you first sense that something is happening.

The shorter this time is, the more responsive the AI feels to users. Even if generating the full answer takes a while, if the first character appears quickly, the person waiting feels less frustrated. That's why, in real-time interactive services like chatbots or coding assistants, this initial response speed is treated as being just as important as overall response speed.

Throughput and end-to-end latency, on the other hand, are different metrics that look at how fast and how much of an answer is generated overall. Time to First Token isolates specifically the part that corresponds to the user's first impression.

The article mentions measuring "throughput, TTFT (Time to First Token), and end-to-end latency." Here, it's important not to confuse TTFT with how fast the service completes the entire response — it specifically refers only to the time it takes for the first response to appear on screen after a user submits a query.

Try asking a chatbot a short question, and use a stopwatch to measure the time from the moment you hit enter to the moment the first character or cursor movement appears on screen. Then measure how long it takes for the entire answer to finish, and compare the two numbers — you'll get a concrete sense that initial response speed and overall completion speed are two different things.