Generative modeling & large-scale training
At Apple, my work centers on training foundation models and building the infrastructure around them: select and filter data, design training objectives, implement dataloaders, design tooling, integrate evaluation, baby-sit runs, conduct scaling studies.
This effort trained 3B tri-modal masked-diffusion models on text, image, and audio at 6T-token scale. It covers the tri-modal design space: tokenization, noise schedules, loss weighting, modality balancing, zero-shot hyper-parameter transfer and scaling laws.
Quality filters are a standard part of pretraining pipelines, but what they actually buy is not obvious. Analyzed at scale, most of the gain comes from discarding the low-quality tail rather than from identifying rare high-value data, which has direct consequences for how filters should be built and tuned.
Choosing how much of each data domain to train on is usually done by trial and error. Model performance as a function of mixture proportions follows predictable scaling laws, which can be fit on small runs and extrapolated to choose mixtures for larger training budgets.
Finetuning a pretrained model on new data tends to erase earlier capabilities. Mixing a small fraction of the original pretraining data back into finetuning follows a scaling law that quantifies the trade-off, so forgetting can be predicted and controlled rather than discovered after the fact.
Learning algorithms, robustness & geometry
My earlier and ongoing theoretical work studies how a constraint on the model, in particular its Lipschitz constant, ties together robustness, privacy, optimal transport, and geometry. The same constraint that certifies robustness also gives differential privacy without clipping, well-behaved neural distance fields, and stable explanations.
An energy-based model assigns low energy to plausible data. This energy landscape becomes a Riemannian metric whose shortest paths stay in high-density regions, yielding data-aware distances and interpolations grounded in the learned distribution.
A true signed distance function has unit gradient norm almost everywhere. Parameterizing the field with a 1-Lipschitz network obtains this property by construction rather than through a soft eikonal penalty, giving accurate and stable distance fields for shape representation. Best paper award at the Symposium on Geometry Processing (SGP) 2024.
DP-SGD achieves differential privacy by clipping per-example gradients, which is expensive and hard to tune. Constraining the network to be Lipschitz gives an analytic bound on gradient sensitivity, so privacy guarantees follow without clipping, making private training simpler and more scalable.
One-class classification asks whether a point belongs to the data distribution. Parameterizing a signed distance function with a 1-Lipschitz network gives a classifier whose margin is a certified distance to the boundary, which provides robustness guarantees and a geometric reading of what counts as in-distribution.
Lipschitz-constrained networks are often thought to trade accuracy for robustness. Much of this trade-off is an artifact of the loss function, and with the right loss these networks reach competitive accuracy while retaining certified robustness.