הכנס הארצי למדעי המחשב והמידע

Abstract

Diffusion models have transformed the landscape of image generation, offering a reliable sampling from the probability density function of images. These models have also been extended to provide posterior sampling capabilities, by conditioning the process on given external controls. This ability has been demonstrated for handling inverse problems in novel ways, and for producing images conditioned on textual prompts, showing in both tasks remarkable results. A very new and fascinating trend in this domain harnesses diffusion models to image compression, showing great potential here as well. This talk presents this story: The essence of diffusion models, their generative abilities, their posterior-sampling extensions, and most importantly, their ability to lead to novel and highly effective compression schemes for images and other data sources. No background is assumed.

Abstract

Grokking is the sudden onset of generalization long after a model has already overfit its training data. In this talk, I will present an end-to-end theoretical account of this phenomenon in classical ridge regression. For over-parameterized linear models trained by gradient descent with weight decay, we prove the full grokking trajectory: early overfitting, a prolonged period of poor generalization, and eventual convergence to arbitrarily small generalization error. Our analysis provides quantitative bounds on the generalization delay, and shows how it can be amplified or eliminated through principled hyperparameter tuning. I will also present experiments suggesting that these predictions extend beyond linear models to nonlinear neural networks. Overall, the results suggest that grokking is not an inherent failure mode of deep learning, but rather a consequence of specific training conditions, and thus does not require fundamental changes to the model architecture or learning algorithm to avoid.

Joint work with Mingyue Xu and Gal Vardi.