METAL LAB

은하 이미지로 훈련한 AI(AstroPT)를 이용해, 언어모델 해석 도구가 진짜 개념을 찾아내는지 검증할 수 있는 시험장을 만들었다

arXiv:2608.226142026-08-22

What AstroPT knows about galaxies, and what that can teach us about LLMs

은하 이미지로 훈련한 AI(AstroPT)를 이용해, 언어모델 해석 도구가 진짜 개념을 찾아내는지 검증할 수 있는 시험장을 만들었다

연구팀은 수백만 장의 은하 이미지로 훈련한 AstroPT라는 GPT/BERT 방식 모델을 언어모델(LLM) 해석 연구의 검증 도구로 제안했다. 은하 속성은 이미 물리학적으로 난이도와 상호관계가 알려져 있어서, 모델이 훈련 중에 어떤 개념을 먼저 배우는지, 그 순서가 실제 난이도와 맞는지 확인할 수 있다. 실험 결과 은하 속성은 알려진 난이도 순서대로 나타났고, 모델이 배운 방향 벡터들도 실제 물리적 관계의 부호(양/음의 상관)를 정확히 재현했다.

METAL LAB 해설 도표

은하 이미지로 학습한 AstroPT 모델의 학습 과정을 중심에 두고, 픽셀에서 바로 읽히는 겉보기등급은 학습 초반 얕은 층에서 실선으로 먼저 나타나며, 여러 파장을 조합해야 하는 적색이동은 학습 후반 깊은 층에서 점선으로 뒤늦게 약하게 나타난다는 것을 보여준다.
METAL LAB이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. 픽셀에서 거의 직접 읽히는 값(밴드 등급)은 훈련 초반, 신경망의 얕은 층에서 먼저 디코딩 가능해졌고, 여러 파장/스펙트럼을 조합해 추론해야 하는 값(적색이동, 특정 별생성률)은 훈련 후반, 깊은 층에서야 나타났다.
  2. 이 순서는 GPT식 자기회귀(AR) 훈련과 BERT식 마스킹(MAE) 훈련 두 가지 방식 모두에서 동일하게 나타났고, 모델 크기가 커져도 순서는 바뀌지 않고 성능 수치만 올라갔다.
  3. 선형 프로브(모델의 숨겨진 표현에서 특정 속성을 읽어내는 간단한 선형 회귀 장치)가 찾아낸 방향 벡터들을 비교했더니, 광도와 별질량은 강하게 같은 방향, 특정 별생성률과 별질량은 반대 방향, 적색이동과 별질량은 같은 방향으로 정렬돼 있었는데, 이는 실제 은하 물리학에서 알려진 부호 구조와 일치했다.
  4. 광도만으로 예측할 수 없는 별질량의 '잔차'(mass residual)를 따로 뽑아 확인했더니, 이 잔차는 광도 방향과 거의 직각(독립적)으로 표현되어 있어 모델이 질량을 단순히 광도의 대체값으로만 쓰지 않는다는 신호를 보였다.
Figure 1: Overview of AstroPT. Galaxy images are split into ordered patch sequences and used to train a GPT (pictured here) or BERT-style transformer via a patch reconstruction objective. The learned embeddings are then used for downstream probes of galaxy properties.
Figure 1: Overview of AstroPT. Galaxy images are split into ordered patch sequences and used to train a GPT (pictured here) or BERT-style transformer via a patch reconstruction objective. The learned embeddings are then used for downstream probes of galaxy properties.
Figure 2: Concepts emerge in a fixed, difficulty-ordered sequence. Held-out mag​r, redshift, and sSFR R2 as a function of pre-training progress and model depth. Directly recoverable photometric information (r-band magnitude) is decoded earliest and most strongly; redshift follows more weakly; specific star formation rate remains weakly retrievable after one epoch at our model parameter counts. Likewise, our directly recoverable quantities peak in shallow layers, while integrated and inferred quantities peak deeper. The order is fixed across model size, with capacity raising the attained score rather than reordering the properties.
Figure 2: Concepts emerge in a fixed, difficulty-ordered sequence. Held-out mag​r, redshift, and sSFR R2 as a function of pre-training progress and model depth. Directly recoverable photometric information (r-band magnitude) is decoded earliest and most strongly; redshift follows more weakly; specific star formation rate remains weakly retrievable after one epoch at our model parameter counts. Likewise, our directly recoverable quantities peak in shallow layers, while integrated and inferred quantities peak deeper. The order is fixed across model size, with capacity raising the attained score rather than reordering the properties.
Table 1: Final-checkpoint bootstrap metrics across objective and model-size choices. Here ℓr=−0.4​Mr is the absolute r-band log-luminosity proxy, M⋆ is stellar mass, and ϵM|ℓr is the residual mass at fixed luminosity. Intervals are 95% bootstrap intervals.
ConfigR2​(ℓr)R2​(M⋆)R2​(ϵM|ℓr)R2​(log⁡sSFR)R2​(z)
AR 1M0.422 [0.420, 0.424]0.535 [0.533, 0.536]0.101 [0.099, 0.104]0.258 [0.256, 0.260]0.410 [0.408, 0.412]
AR 21M0.499 [0.496, 0.501]0.610 [0.608, 0.612]0.109 [0.104, 0.114]0.298 [0.295, 0.302]0.503 [0.499, 0.506]
AR 100M0.530 [0.526, 0.533]0.639 [0.636, 0.641]0.110 [0.103, 0.116]0.309 [0.304, 0.313]0.573 [0.568, 0.576]
MAE 1M0.356 [0.354, 0.358]0.454 [0.452, 0.456]0.085 [0.082, 0.087]0.248 [0.245, 0.250]0.328 [0.326, 0.330]
MAE 21M0.350 [0.347, 0.354]0.446 [0.443, 0.449]0.070 [0.065, 0.075]0.228 [0.224, 0.232]0.323 [0.319, 0.327]
MAE 100M0.386 [0.381, 0.390]0.488 [0.484, 0.491]0.080 [0.075, 0.085]0.246 [0.243, 0.250]0.349 [0.343, 0.354]
Figure 3: Probe geometry recovers the signs of known galaxy relationships. Across model sizes, luminosity and stellar-mass directions are strongly aligned, sSFR and stellar-mass directions are anti-aligned, redshift and stellar-mass directions are positively aligned, and the mass residual is much less aligned with luminosity than stellar mass is, matching the known sign structure of these relationships.
Figure 3: Probe geometry recovers the signs of known galaxy relationships. Across model sizes, luminosity and stellar-mass directions are strongly aligned, sSFR and stellar-mass directions are anti-aligned, redshift and stellar-mass directions are positively aligned, and the mass residual is much less aligned with luminosity than stellar mass is, matching the known sign structure of these relationships.
Figure 4: Label-space galaxy relationships used as baselines for the representation-geometry analysis. Left: specific star formation rate is anti-correlated with stellar mass. Middle: redshift is positively correlated with stellar mass. Right: absolute r-band luminosity is strongly correlated with stellar mass. These label-space relationships set the expected signs for the probe-direction analysis in the main text.
Figure 4: Label-space galaxy relationships used as baselines for the representation-geometry analysis. Left: specific star formation rate is anti-correlated with stellar mass. Middle: redshift is positively correlated with stellar mass. Right: absolute r-band luminosity is strongly correlated with stellar mass. These label-space relationships set the expected signs for the probe-direction analysis in the main text.

실제로 확인된 결과

  • r밴드 등급(직접 관측 가능한 값)은 훈련 초반과 얕은 층에서 가장 먼저, 가장 강하게 디코딩됐고 적색이동이 그 다음, 특정 별생성률은 한 에폭 내에서 약하게만 디코딩됐다(Fig. 2).
  • 이 난이도 순서는 AR과 MAE 두 훈련 목표 모두에서, 그리고 1M/21M/100M 파라미터 크기 모두에서 동일하게 유지됐으며, 모델이 커질수록 순서 대신 평균 성능만 올라갔다.
  • 프로브 방향 벡터를 비교한 결과 광도-별질량 방향은 강하게 정렬, sSFR-별질량 방향은 반정렬, 적색이동-별질량 방향은 양의 정렬로 나타났고, 이는 실제 은하 물리학의 부호 구조와 일치했다(Fig. 3, Fig. 4 라벨 공간 기준선과 비교).
  • 광도로 설명되지 않는 별질량 잔차(residual)는 광도·별질량보다 훨씬 디코딩하기 어려웠고(R2가 낮음), 잔차 방향은 광도 방향과 거의 직각을 이뤄 별질량 방향보다 훨씬 덜 정렬됐다(Fig. 10, Fig. 11, Table 1).
  • 비슷한 광도지만 질량 잔차가 다른 은하 쌍 사이에서 활성화를 교체(patching)하는 실험에서, 같은 잔차나 무작위 프로브를 쓴 대조군보다 더 자주 잔차 예측이 원본 쪽으로 이동했다(Fig. 12).
Figure 5: Training-time relationship geometry under the AR objective. Left: the sSFR direction is anti-aligned with the stellar-mass direction through training. Right: the redshift direction is positively aligned with the stellar-mass direction through training. The signs are also stable across model sizes, suggesting that the relationships are not confined to our fully pre-trained models.
Figure 5: Training-time relationship geometry under the AR objective. Left: the sSFR direction is anti-aligned with the stellar-mass direction through training. Right: the redshift direction is positively aligned with the stellar-mass direction through training. The signs are also stable across model sizes, suggesting that the relationships are not confined to our fully pre-trained models.
Figure 6: Training-time relationship geometry under the MAE objective. Left: the sSFR direction is anti-aligned with the stellar-mass direction through training. Right: the redshift direction is positively aligned with the stellar-mass direction through training. The signs are also stable across model sizes, suggesting that the relationships are not confined to our fully pre-trained models.
Figure 6: Training-time relationship geometry under the MAE objective. Left: the sSFR direction is anti-aligned with the stellar-mass direction through training. Right: the redshift direction is positively aligned with the stellar-mass direction through training. The signs are also stable across model sizes, suggesting that the relationships are not confined to our fully pre-trained models.

어디에 쓸 수 있나

  • 언어모델에서 개념이 훈련 중 언제, 어느 층에서 나타나는지 조사하는 해석 방법을 먼저 정답이 알려진 도메인(천문학)에서 검증한 뒤 적용해볼 수 있다.
  • 선형 프로브가 찾아낸 방향 벡터가 실제 개념 간 관계를 반영하는지 확인하는 방법론을 다른 과학 기반 파운데이션 모델에도 적용해볼 수 있다.
  • 활성화 패칭(activation patching) 같은 인과 추적 기법을 검증하는 벤치마크로 은하 데이터셋을 활용할 수 있다.
Figure 7: Layer-wise relationship geometry under the AR objective. The luminosity and stellar-mass directions remain strongly aligned across layers, the sSFR and stellar-mass directions remain anti-aligned, and the redshift direction is positively aligned with both mass and luminosity. This suggests the relationship geometry is distributed across layers rather than confined to the final layer.
Figure 7: Layer-wise relationship geometry under the AR objective. The luminosity and stellar-mass directions remain strongly aligned across layers, the sSFR and stellar-mass directions remain anti-aligned, and the redshift direction is positively aligned with both mass and luminosity. This suggests the relationship geometry is distributed across layers rather than confined to the final layer.
Figure 8: Layer-wise relationship geometry under the MAE objective. Luminosity and stellar mass remain strongly aligned across layers, sSFR and stellar mass remain anti-aligned, and redshift remains positively aligned with both mass and luminosity. The same sign structure appears under MAE and AIM, supporting the claim that the recovered geometry is not tied to a single training objective.
Figure 8: Layer-wise relationship geometry under the MAE objective. Luminosity and stellar mass remain strongly aligned across layers, sSFR and stellar mass remain anti-aligned, and redshift remains positively aligned with both mass and luminosity. The same sign structure appears under MAE and AIM, supporting the claim that the recovered geometry is not tied to a single training objective.

한계와 남은 검증

  • 은하는 언어가 아니고 이미지 패치는 언어의 토큰과 다르므로, 천문 데이터 자체가 LLM 사전훈련에 유용하다는 주장은 아니다.
  • 분석은 상관관계 수준이며, 물리적 라벨은 카탈로그 추정값이라 오차를 포함할 수 있다.
  • 사전훈련은 한 에폭, {1M, 21M, 100M} 파라미터라는 비교적 작은 모델 범위에 한정됐다.
  • 선형 프로브는 표현 공간에서 선형적으로 접근 가능한 정보만 드러내며, 비선형적으로 숨겨진 구조는 확인하지 못한다.
  • 질량 잔차에 대한 활성화 패칭 실험은 탐색적 수준이며 추가 검증이 필요하다.
Figure 9: Empirical mass–luminosity relation used to define the residual mass at fixed luminosity. The horizontal axis is a log-luminosity proxy from absolute r-band magnitude, ℓr=−0.4​Mr. The dashed line is the linear fit on the training split, and color shows the residual ϵM|ℓr.
Figure 9: Empirical mass–luminosity relation used to define the residual mass at fixed luminosity. The horizontal axis is a log-luminosity proxy from absolute r-band magnitude, ℓr=−0.4​Mr. The dashed line is the linear fit on the training split, and color shows the residual ϵM|ℓr.
Figure 10: Residual mass-at-fixed-luminosity remains weak across objective choices. Each panel reports final-checkpoint, best-layer R2 for the luminosity proxy ℓr, stellar mass M⋆, and the residual ϵM|ℓr, with error bars showing 95% bootstrap confidence intervals (300 resamples). Luminosity and stellar mass are consistently easier to decode, while the residual remains weak across AR/MAE objectives.
Figure 10: Residual mass-at-fixed-luminosity remains weak across objective choices. Each panel reports final-checkpoint, best-layer R2 for the luminosity proxy ℓr, stellar mass M⋆, and the residual ϵM|ℓr, with error bars showing 95% bootstrap confidence intervals (300 resamples). Luminosity and stellar mass are consistently easier to decode, while the residual remains weak across AR/MAE objectives.

왜 중요한가

언어모델 해석 연구는 '개념이 언제, 어떻게 나타나는가'를 묻지만 언어에는 정답이 없어 검증이 어렵다. 은하 물리학은 난이도와 관계가 미리 알려져 있어, 이 논문은 해석 도구들이 실제로 맞는 답을 찾아내는지 확인할 수 있는 정답지 역할을 한다.

Figure 11: Residual probe directions are separated from the dominant luminosity-mass axis, with error bars showing 95% bootstrap confidence intervals (300 resamples). Across objective choices, luminosity and stellar-mass directions remain strongly aligned. The residual direction is much less aligned with luminosity than the stellar-mass direction is, indicating that the residual probe is not reading out the same luminosity direction. The separation is strongest in the larger AR models and weaker in the MAE setting.
Figure 11: Residual probe directions are separated from the dominant luminosity-mass axis, with error bars showing 95% bootstrap confidence intervals (300 resamples). Across objective choices, luminosity and stellar-mass directions remain strongly aligned. The residual direction is much less aligned with luminosity than the stellar-mass direction is, indicating that the residual probe is not reading out the same luminosity direction. The separation is strongest in the larger AR models and weaker in the MAE setting.
Figure 12: Causal tracing of the mass-luminosity residual. Patching activations between galaxy pairs with similar luminosity but different mass residual moves the residual prediction toward the source more often than same-residual and shuffled-probe controls.
Figure 12: Causal tracing of the mass-luminosity residual. Patching activations between galaxy pairs with similar luminosity but different mass residual moves the residual prediction toward the source more often than same-residual and shuffled-probe controls.

이 논문의 용어

  • AstroPT · 은하 이미지를 패치(조각) 시퀀스로 나눠 GPT나 BERT처럼 자기지도 학습시킨 트랜스포머 모델
  • 선형 프로브 · 모델의 숨겨진 표현에서 특정 속성이 얼마나 쉽게 읽히는지 확인하는 간단한 선형 회귀 장치
  • 적색이동(redshift) · 우주 팽창으로 멀어지는 은하의 빛이 파장이 길어지는 정도, 여러 밴드나 스펙트럼을 조합해야 추정 가능
  • 특정 별생성률(sSFR) · 단위 별질량당 별이 만들어지는 속도, 추론이 필요한 간접적 물리량
  • 코사인 유사도 · 두 방향 벡터가 얼마나 같은 방향을 향하는지 나타내는 지표, 양수면 같은 방향, 음수면 반대 방향

저자 · UniverseTBD

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: UniverseTBD et al., arXiv:2608.22614, CC BY-SA 4.0