ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization
arXiv:2608.12756v1 Announce Type: new Abstract: Adaptive latent tokenization maps a fine-grained input to a shorter sequence of continuous representations associated with input-dependent spans. We introduce ReconSpan, which divides text into chunks that a backward decoder can reconstruct from a single contextual prefix code and retains one such code as the latent token for each chunk. The reconstruction criterion is applied when chunks are formed, allowing one trained autoencoder to produce aver
arXiv에서 원문 보기