Quantifying cross-modal transfer in astronomical transient classification

We investigate whether the classification of observed astronomical transients exhibits cross-modal transfer: the improved performance of models trained with more modalities than are available at inference. We consider models trained with combinations of light curves, images and spectra using different modality drop and fine-tuning strategies. For a sample of observed transients, we confirm that multimodal models show better macro F1 classification performance, although with consistent or marginally worst calibration errors. We find no significant cross-modal transfer using light curve-only inference, and we marginally detect cross-modal transfer using light curves and images at inference. Classic models trained with manually engineered features achieve better classification metrics than light curve-only inference models, but have significantly worst calibration errors.