매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation

arXiv:2608.196112026-08-21

LLM이 답을 낼 때마다 수백 번 다시 굴려보지 않아도, 통계 모델 하나면 '어디서 결정이 갈리는지'를 훨씬 싸게 알아낼 수 있다

LLM에게 같은 질문을 여러 번 풀게 하면 답이 갈리는 지점(포킹 포인트)이 있는데, 이걸 정확히 찾으려면 각 단계마다 수백 번씩 다시 생성시켜야 해서 비용이 엄청나다. 연구진은 적은 횟수만 재생성한 뒤 통계적으로 매끄럽게 다듬는 모델을 만들어, 실제로는 수백 번 재생성한 것과 비슷한 정확도를 훨씬 적은 비용으로 얻어냈다. Llama-3-8B-Instruct와 DeepSeek-R1-Distill-Llama-8B로 tinyMMLU 문제를 풀게 하며 총 17.7억 개 토큰 분량의 데이터를 모아 검증했다.

무엇을 했나

  1. 문제: 모델이 한 문장씩 생성할 때마다 '이후 어떤 답으로 이어지는지' 확인하려면 각 지점에서 수십~수백 번씩 이어쓰기를 시켜봐야 해서 토큰 비용이 막대하다(포킹 패스 분석, Forking Paths Analysis)
  2. 관찰: 재생성 횟수를 아주 많이 늘리면(S=1000) 답이 갈리는 정도가 대부분 구간에서 매끈해지고, 급격히 바뀌는 지점(포킹 포인트)만 뚜렷하게 남는다. 즉 적은 횟수에서 보이는 들쭉날쭉함은 대부분 그냥 샘플링 잡음이었다
  3. 해법: 변화점 탐지(PELT)로 안정적인 구간과 갈리는 지점을 먼저 나누고, 각 구간 안에서는 커널 가중 평균(가우시안 커널로 이웃 데이터를 가중 합산)으로 매끄럽게 다듬는 3단계 통계 모델을 개발했다
  4. 결과: 이 모델을 적용하면 적은 샘플(S=5~30)로도 실질적으로 3.3배~5배 더 많이 뽑은 것과 비슷한 정확도를 얻고, 토큰 예산을 최대 1/8까지 줄여도 오차 증가가 크지 않았다
  5. 한계: tinyMMLU의 객관식 문제와 8B급 모델 두 개에 한정된 실험이며, 큰 답 변화(포킹 임계값 0.20 이상)에서는 오히려 원본 데이터보다 살짝 부정확해질 수 있다
Figure 1: Uncertainty dynamics ot for one question (tinyMMLU question 39; Llama-3-8B-Instruct). Colors represent the fraction of rollouts at t that end with different final answers. (a) When collecting a massive number of resampled rollouts (S=1000) at every token (N=1), the outcome distribution becomes increasingly smooth, except at forking points. (c) Uncertainty dynamics with a lower-sample analysis (S=20, N=2) using 1% of the reference’s sampling cost. (b) Smoothing the lower-sample data ot closely approximates the high-sample data and recovers the same forking points. More examples in Appendix B and our interactive dashboard (https://github.com/ericb-goodfire/forking-fast).
Figure 1: Uncertainty dynamics ot for one question (tinyMMLU question 39; Llama-3-8B-Instruct). Colors represent the fraction of rollouts at t that end with different final answers. (a) When collecting a massive number of resampled rollouts (S=1000) at every token (N=1), the outcome distribution becomes increasingly smooth, except at forking points. (c) Uncertainty dynamics with a lower-sample analysis (S=20, N=2) using 1% of the reference’s sampling cost. (b) Smoothing the lower-sample data ot closely approximates the high-sample data and recovers the same forking points. More examples in Appendix B and our interactive dashboard (https://github.com/ericb-goodfire/forking-fast).
Figure 2: Accuracy versus number of resampled rollouts. We find that with raw data observations ot (dotted lines), greater resample spacing N leads to slightly lower accuracy with equal S, with considerably more difference at higher S. This effect is significantly more pronounced with the smoothed ot (solid lines), which also shows that increasing log⁡S has a consistent linear effect on log-TVD. Larger N having decreasing accuracy for a given S has a relatively simple explanation, since this means fewer samples collected overall. Accuracy on the y-axis is measured by total variation distance (TVD) relative to the S=200 reference ot.
Figure 2: Accuracy versus number of resampled rollouts. We find that with raw data observations ot (dotted lines), greater resample spacing N leads to slightly lower accuracy with equal S, with considerably more difference at higher S. This effect is significantly more pronounced with the smoothed ot (solid lines), which also shows that increasing log⁡S has a consistent linear effect on log-TVD. Larger N having decreasing accuracy for a given S has a relatively simple explanation, since this means fewer samples collected overall. Accuracy on the y-axis is measured by total variation distance (TVD) relative to the S=200 reference ot.
Figure 3: Accuracy versus token cost. We find that overall, with raw ot data, wider spacing between resampled points N leads to better accuracy with lower overall token cost (dotted lines). Our statistical model significantly increases the accuracy for most S and N, but also negates the improvement of larger N increasing accuracy with equal cost (solid lines). TVD between the S=200 reference and each low-sample estimator is shown on the y-axis, with colors representing different N. The black star represents the sampling parameters used in (1).
Figure 3: Accuracy versus token cost. We find that overall, with raw ot data, wider spacing between resampled points N leads to better accuracy with lower overall token cost (dotted lines). Our statistical model significantly increases the accuracy for most S and N, but also negates the improvement of larger N increasing accuracy with equal cost (solid lines). TVD between the S=200 reference and each low-sample estimator is shown on the y-axis, with colors representing different N. The black star represents the sampling parameters used in (1).
Figure 4: Reconstructions for three questions from the full tinyMMLU evaluation set using Llama-3-8B-Instruct. From top to bottom, the rows show the S=200 every-token reference, the Full Model’s reconstruction from a nested S=15 run at 4-token spacing, and Raw Data from the same reduced run.
Figure 4: Reconstructions for three questions from the full tinyMMLU evaluation set using Llama-3-8B-Instruct. From top to bottom, the rows show the S=200 every-token reference, the Full Model’s reconstruction from a nested S=15 run at 4-token spacing, and Raw Data from the same reduced run.

왜 중요한가

모델의 추론 과정에서 어느 단계가 결정적이었는지 분석하는 해석 가능성(interpretability) 연구는 지금까지 막대한 재생성 비용 때문에 소수의 사례에만 적용됐는데, 이 방법으로 그 비용 장벽을 크게 낮출 수 있다. 결과적으로 더 많은 질문, 더 큰 모델에 대해 '왜 이런 답이 나왔는지'를 저비용으로 조사할 길이 열린다.

Figure 5: DeepSeek counterpart to Figure 2. Pooled TVD to the full S=200 reference is plotted against the number of samples per position, S, with one line per observation spacing in sentences. Dotted lines are results for raw data, solid lines are smoothed data.
Figure 5: DeepSeek counterpart to Figure 2. Pooled TVD to the full S=200 reference is plotted against the number of samples per position, S, with one line per observation spacing in sentences. Dotted lines are results for raw data, solid lines are smoothed data.
Figure 6: DeepSeek counterpart to Figure 3. Pooled TVD is plotted against the number of tokens generated per question; colors match Figure 5. Dotted lines are results for raw data, solid lines are smoothed data. The star marks the baseline setting of S=30 at 1-sentence spacing.
Figure 6: DeepSeek counterpart to Figure 3. Pooled TVD is plotted against the number of tokens generated per question; colors match Figure 5. Dotted lines are results for raw data, solid lines are smoothed data. The star marks the baseline setting of S=30 at 1-sentence spacing.
Figure 7: Forking region counterpart to Figure 2. TVD to the S=200 reference is evaluated only at forking regions, using forking thresholds of 0.10 (left) and 0.15 (right). Solid lines show Smoothed Data, and dotted lines show Raw Data; colors match those in the main text.
Figure 7: Forking region counterpart to Figure 2. TVD to the S=200 reference is evaluated only at forking regions, using forking thresholds of 0.10 (left) and 0.15 (right). Solid lines show Smoothed Data, and dotted lines show Raw Data; colors match those in the main text.
Figure 8: Fork-region counterpart to Figure 3. TVD in forking regions is plotted against the number of tokens generated per question at forking thresholds of 0.10 and 0.15. Solid lines show Smoothed Data, and dotted lines show Raw Data.
Figure 8: Fork-region counterpart to Figure 3. TVD in forking regions is plotted against the number of tokens generated per question at forking thresholds of 0.10 and 0.15. Solid lines show Smoothed Data, and dotted lines show Raw Data.

이 논문의 용어

  • 포킹 패스 분석(Forking Paths Analysis) · 생성 도중 각 지점에서 문장을 다시 이어 써보게 하여 답이 어떻게 갈라지는지 추적하는 분석법
  • 포킹 포인트(forking point) · 모델이 내놓는 답의 분포가 갑자기 크게 바뀌는 결정적 지점
  • PELT (Pruned Exact Linear Time) · 시계열 데이터에서 통계적 패턴이 바뀌는 지점(변화점)을 정확하게 찾아내는 알고리즘
  • 커널 풀링(kernel pooling) · 이웃한 데이터를 가우시안 가중치로 합쳐 잡음을 줄이고 매끄럽게 만드는 방법
  • 총변동거리(TVD, Total Variation Distance) · 두 확률분포가 얼마나 다른지를 재는 지표, 값이 작을수록 두 분포가 비슷함
Figure 9: Paired per-question TVD differences between each ablation variant and the Full Model for DeepSeek-R1-Distill-Llama-8B at 1-sentence spacing. Error bars show 95% t-intervals over tinyMMLU, and the dashed zero line denotes the Full Model. Results are shown for pooled accuracy (left) and forking region accuracy at threshold 0.10 (right). Removing segmentation harms accuracy near forks but not pooled accuracy; fixing the detection cost to L2 is indistinguishable from the Full Model in both panels.
Figure 9: Paired per-question TVD differences between each ablation variant and the Full Model for DeepSeek-R1-Distill-Llama-8B at 1-sentence spacing. Error bars show 95% t-intervals over tinyMMLU, and the dashed zero line denotes the Full Model. Results are shown for pooled accuracy (left) and forking region accuracy at threshold 0.10 (right). Removing segmentation harms accuracy near forks but not pooled accuracy; fixing the detection cost to L2 is indistinguishable from the Full Model in both panels.
Figure 10: Ablation overlays for two example questions. Reconstructions use S=30 on the densest spacing N and are evaluated against leave-replicate-out S=200 references; dashed lines mark reference forks. The left column shows Llama-3-8B-Instruct question 46 using the ablation variants, cross-validation, and operating point from the recorded S=200 store. The right column shows the recorded DeepSeek example, where the fork occurs when continuations stop resolving to an answer. Removing segmentation (Kernel Pooling) blurs the forks, whereas removing kernel pooling (Segment+Pool) reduces gradual drift to discrete steps. The L2 detection-cost variant is visually indistinguishable from the Full Model.
Figure 10: Ablation overlays for two example questions. Reconstructions use S=30 on the densest spacing N and are evaluated against leave-replicate-out S=200 references; dashed lines mark reference forks. The left column shows Llama-3-8B-Instruct question 46 using the ablation variants, cross-validation, and operating point from the recorded S=200 store. The right column shows the recorded DeepSeek example, where the fork occurs when continuations stop resolving to an answer. Removing segmentation (Kernel Pooling) blurs the forks, whereas removing kernel pooling (Segment+Pool) reduces gradual drift to discrete steps. The L2 detection-cost variant is visually indistinguishable from the Full Model.
Figure 11: Replicate TVD versus the number of samples per position, S, on log–log axes. The measured curve remains within 2% of the exact i.i.d. multinomial null through S=200. Triangles show the full dataset means for both models.
Figure 11: Replicate TVD versus the number of samples per position, S, on log–log axes. The measured curve remains within 2% of the exact i.i.d. multinomial null through S=200. Triangles show the full dataset means for both models.
Figure 12: Number of detected segments as a function of the PELT penalty for S=30 dense-grid counts on the two development questions. Curves show the exact multinomial cost, a trend-aware cost, and the silently substituted L2 cost. The dashed line marks the ideal segment count.
Figure 12: Number of detected segments as a function of the PELT penalty for S=30 dense-grid counts on the two development questions. Curves show the exact multinomial cost, a trend-aware cost, and the silently substituted L2 cost. The dashed line marks the ideal segment count.

논문 원문 초록 (영문)

LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncertainty. Resampling-based analyses characterize this distribution, revealing which steps of a rollout determine how the model arrives at its answer. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly. Our work strives to make resampling analysis more computationally efficient, while also shedding light on an important scientific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resampling many reasoning chains, uncertainty dynamics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM's sensitivity to each individual token or reasoning step. We develop a statistical model for smoothing noisy low-sample rollout data to better approximate high-sample data, allowing us to significantly cut sampling costs.

저자 · Eric Bigelow, Amir Zur, Satchel Grant, Tal Haklay, Can Rager, Owen Lewis, Thomas McGrath, Jack Merullo, Ekdeep Singh Lub

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Eric Bigelow et al., arXiv:2608.19611, CC BY 4.0