Statistical Learning Research Group (Sailing) 2023
May 22, Could a Large Language Model be Conscious? David Chalmers (NYU) May 15, Transformers and Pretraining , NLP lecture (Stanford University)May 8, Self-Attention and Transformers May 1 , Introduction to self-attention and transformers April 17, GPT-3 & Beyond , Christopher Potts (Stanford University)April 3, Reinforcement Learning 10 (Classic Games Case Study) , Hado van Hasselt (DeepMind)March 27, How to increase certainty in predictive modeling , Emmanuel Candès (Stanford University)March 20, Reinforcement Learning 8 (Advanced Topics in Deep RL) , Hado van Hasselt (DeepMind)March 13, Reinforcement Learning 7 (Planning and Models) , March 6, Reinforcement Learning 6 (Policy Gradients and Actor Critics) , Hado van Hasselt (DeepMind)February 20, Welcome lunch February 13, Reinforcement Learning 4 (Model-Free Prediction and Control) , Hado van Hasselt (DeepMind)February 6, Offline Reinforcement Learning , Sergey Levine, (UC Berkeley)January 30, Reinforcement Learning 2 (Exploration and Exploitation) , Hado van Hasselt (DeepMind). HW: watch the rest of the videoJanuary 23, Reinforcement Learning 1 , Hado van Hasselt (DeepMind). HW: watch the rest of the video
January 16. Introduction to Algebraic Topology , N J Wildberger (University of New South Wales, Australia)January 9, Signal Recovery with Generative Priors , Paul Hand (Northeastern University, Boston)
2022
December 12, Learning-Based Low-Rank Approximations , Piotr Indyk (MIT), paper December 5, General graph problems with neural networks , Soledad Villar (New York University, Current Institute)November 28, The Transformer Network for the Traveling Salesman Problem , Xavier Bresson (NTU)November 25 , Artificial Intelligence in Acute Medecine, From theory to applications ,November 21, Attention, Learn to Solve Routing Problems! Wouter Kool (University of Amsterdam)November 14, Why Did Quantum Entanglement Win the Nobel Prize in Physics? Sixty Symbols – Spooky Action at a Distance (Bell’s Inequality) November 7 , EigenGame PCA as a Nash Equilibrium , Ian Gemp (DeepMind), Deep Semi-Supervised Anomaly Detection October 31, Discovering faster matrix multiplication algorithms with reinforcement learning , Yannic KilcherOctober 17, Compressing Variational Bayes , Stephan Mandt (UC Irvine)October 10, From Machine Learning to Autonomous Intelligence , Yann LeCun (NYU, Meta)October 3, Diffusion Probabilistic Models , Jascha Sohl-Dickstein (Google Brain)September 12 & 26, Attention and Memory in Deep Learning , Alex Graves (DeepMind)September 5, Transformers and Self-Attention , Ashish Vaswani (Adept AI Labs)August 29, Ensuring Safety in Online Reinforcement Learning by Leveraging Offline Data , Sergey Levine (UC Berkeley)August 22, Geometric Deep Learning: The Erlangen Programme of ML , Michael Bronstein (Imperial College, London)August 15, The Devil is in the Tails and Other Stories of Interpolation , Niladri Chatterji (Stanford Unversity)Summer break July 11, Gaussian multiplicative chaos: applications and recent developments , Nina Holden (ETH, Werner Wendelin Group)July 1, Meeting with UFC-LNIT. June 27, Statistical mechanics arising from random matrix theory , Thomas Spencer (IAS)June 20, Stop Explaining Black Box Machine Learning Models Cynthia Rudin (Duke University)May 30 & June 13, Network Calculus , Jean-Yves Le Boudec (EPFL)May 16 & 23, Synthetic Healthcare Data Generation and Assessment: Challenges, Methods, and Impact on Machine Learning , Mihaela van der Schaar (university of Cambridge) and Ahmed Alaa (MIT).
May 9, Combining Reinforcement Learning & Constraint Programming for Combinator… , Louis-Martin Rousseau (Ecole Polytechnique de Montreal)May 2, Deep Reinforcement Learning at the Edge of the Statistical Precipice , Rishabh Agarwal (Google Brain)April 25, Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies , Paul Vicol (University of Toronto)April 11, I Can’t Believe Latent Variable Models Are Not Better , Chris Maddison (University of Toronto)April 4 , From System 1 Deep Learning to System 2 Deep Learning March 14 and 21 , Latent Dirichlet Allocation Online Learning for Latent Dirichlet Allocation , Matthew D. Hoffman, David M. Blei (Princeton University)March 7, On the Expressivity of Markov Reward , David Abel (DeepMind) et al. paper February 28, Continuous Time Dynamic Programming — The Hamilton-Jacobi-Bellman Equation , Neil Walton (University of Manchester)February 21, Computational Barriers in Statistical Estimation and Learning , Andrea Montanari (Stanford University)February 14, Offline Deep Reinforcement Learning Algorithms , Sergey Levine (UC Berkeley)February 7, Infusing Physics and Structure into Machine Learning, Anima Anandkumar (CalTech)January 31, Robust Predictable Control , Benjamin Eysenbach (CMU), web page , paper January 10, 17 and 24, Recent Advances in Integrating Machine Learning and Combinatorial Optimization – Tutorial at AAAI-21
Tutorial webpage with slides
Part 1: Introduction to combinatorial optimization & tutorial overview
This tutorial will provide an overview of the recent impact machine learning is having on combinatorial optimization, particularly under the Mixed Integer Programming (MIP) framework. Topics covered will include ML and reinforcement learning for predicting feasible solutions, improving exact solvers with ML, a software framework for learning in exact MIP solvers, and the emerging paradigm of decision-focused learning.
The tutorial targets both junior and senior researchers in two prominent areas of interest to the AAAI community: (1) Machine learning researchers looking for a challenging application domain, namely combinatorial optimization; (2) Optimization practitioners and researchers who may benefit from learning about recent advances in ML methods for improving combinatorial optimization algorithms.
Presented by: Elias B. Khalil (University of Toronto), Andrea Lodi (Polytechnique Montréal), Bistra Dilkina (University of Southern California), Didier Chételat (Polytechnique Montréal), Maxime Gasse (Polytechnique Montréal), Antoine Prouvost (Polytechnique Montréal), Giulia Zarpellon (Polytechnique Montréal) and Laurent Charlin (HEC Montréal)
2021
December 13 and 20, Attention and Transformer Networks , Pascal Poupart (University of Waterloo, Canada)December 6, Yes, Generative Models Are The New Sparsity , Alex Dimakis (University of Texas at Austin)November 29 . The Knockoffs Framework: New Statistical Tools for Replicable Selections November 15 and 22 . Advanced Machine Learning Day 3: Neural Architecture Search November 8 , La théorie de l’apprentissage de Vapnik et les progrès récents de l’IA November 1, Compositional Dynamics Modeling for Physical Inference and Control , Yunshu Li (MIT)October 25, Safe and Efficient Exploration in Reinforcement Learning , Andreas Krause (ETH)October 18, Special guest today: Luis von Ahn , CEO and co-founder of Duolingo (former professor at CMU)October 11, Contrastive Learning: A General Self-supervised Learning Approach , Yonglong Tian (MIT)
September 27 & October 4, Adversarial Robustness – Theory and Practice , J. Z. Kolter (CMU-Bosch) and A. Madry (MIT)No Lunch Seminar during the summer break
June 21 & 28, Recent Developments in Over-parametrized Neural Networks , Jason Lee (University of Southern California) (43.42-end)June 14, Feedback Control Perspectives on Learning , Jeff Shamma (University of Illinois at Urbana-Champaign)June 7, Self-Supervised Learning & World Models , Yann LeCun (NYU – Courant institute, Facebook)May 31 , Theoretical Foundations of Graph Neural Networks May 17 & 24, Deep Implicit Layers , David Duvenaud (University of Toronto), J. Zico Kotler (CMU), Matt Johnson (Google Brain)May 3 & 10, Bayesian Deep Learning and Probabilistic Model Construction , Andrew Gordon Wilson (Courant institute, New York University)April 26, Learning Ising Models from One, Ten or a Thousand Samples , Costantinos Daskalakis (MIT)April 19, Deconstructing the Blockchain to Approach Physical Limits , Sreeram Kannan (University of Washington)April 12, Federated Learning and Analytics at Google and Beyond , Peter Kairouz (Google)March 22, Equivariant Networks and Natural Graph Networks , Taco Cohen (Qualcomm)March 8 and 15, Parameter-free Online Optimization 1 , 2 , 3 , 4 .February 22 and March 1, Machine Learning with Signal Processing , Arno Solin (Aalto University)February 15 . A Function Approximation of Perspective on Sensory Representations , Cengiz Pehlevan (Harvard). References: Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent , Neural Tangent Kernel: Convergence and Generalization in Neural Networks , On Exact Computation with an Infinitely Wide Neural Net , Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural NetworksFebruary 8: Hopfield Networks in 2021 , Fireside chat between Sepp Hochreiter (Johannes Kepler University of Linz) and Dmitry Krotov (IBM Watson)February 1 : Convergence and Sample Complexity of Gradient Methods for the Model-Free Linear Quadratic Regulator January 25 : TIC Department ConferenceJanuary 18, Influence: Using Disentangled Representations to Audit Model Predictions , Charlie Marx (Haverford College)January 11, Offline Reinforcement Learning , Sergey Levine, UC Berkeley
2020
December 14, Stanford Seminar (part 2) – Information Theory of Deep Learning , Naftali Tishby, Hebrew University of Jerusalem. New Theory Cracks Open the Black Box of Deep Learning , Quanta MagazineDecember 7, Stanford Seminar (part 1) – Information Theory of Deep Learning , Naftali Tishby, Hebrew University of JerusalemNovember 30, Computer vision: who is harmed and who benefits? Timnit Gebru, Google. Links related to the talk: James Landay, Smart Interfaces for Human-Centered AI , Ali Alkhatib, Anthropological/Artificial Intelligence & the HAI , Faception , HireVue , Our Data Bodies ,November 23, Network Telemetry and Analytics for tomorrows Zero Touch Operation Network , Online talk by Swisscom Digital LabNovember 16, Representation Learning Without Labels , Ali Eslami, DeepMindNovember 9, Active Learning: From Theory to Practice , Robert Nowak, University of Wisconsin-MadisonNovember 2 , Stanford Seminar – Machine Learning for Creativity, Interaction, and Inclusion, October 26 (pdf ) , LambdaNetworks: Modeling long-range Interactions without Attention (Paper Explained, ICLR 2021 submission) October 12 , Artificial Stupidity: The New AI and the Future of Fintech September 28, LSTM is dead. Long Live Transformers! Leo Dirac, Amazon. LSTM paper , LSTM Diagrams- Understanding LSTM , Attention is all you need , Illustrated Attention , BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , Deep contextualized word representations , huggingface/transformers September 14, Machine Learning Projects Against COVID-19 , Yoshua Bengio, Université de MontréalNo Lunch Seminar during the summer break
June 22, Kernel and Deep Regimes in Overparameterized Learning , Suriya Gunasekar (TTI-Chicago, Microsoft Research)June 15, Energy-based Approaches to Representation Learning, Yann LeCun (New York University, FaceBook)June 8 , Learning probability distributions; What can, What can’t be done May 25, Generalized Resilience and Robust Statistics , Jacob Steinhardt (UC Berkeley).
May 18, From Classical Statistics to Modern Machine Learning , Mikhail Belkin (The Ohio State University).
References: Mikhail Belkin, Siyuan Ma, Soumik Mandal To Understand Deep Learning We Need to Understand Kernel Learning , Luc Devroye et al., The Hilbert Kernel Regression Estimate , Cover and Hart, Nearest Neighbor Pattern Classification, Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate , Adityanarayanan Radhakrishnan et al., Overparameterized Neural Networks Can Implement Associative Memory , Mikhail Belkin et al., Reconciling modern machine-learning practice and the classical bias–variance trade-off , Madhu S. Advani, Andrew M. Saxe, High-dimensional dynamics of generalization error in neural networks
May 11 , Automatic Machine Learning, part 3 Slides part 3
May 4 , Automatic Machine Learning, part 2 Slides parts 1-2, Slides part 3 .
References: Thomas Elsken et al., Neural Architecture Search: A Survey. J. Bergstra et al., Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , Hector Mendoza et al., Towards Automatically-Tuned Neural Networks , Barret Zoph et. al., Neural Architecture Search with Reinforcement Learning, Thomas Elsken et al., Neural Architecture Search: A Survey , Peter J. Angeline et al., An Evolutionary Algorithm that Constructs Recurrent Neural Networks , Kenneth O. Stanley et al., Evolving Neural Networks through Augmenting Topologies , Risto Miikkulainen et al., Evolving Deep Neural Networks , Esteban Real et al.,Regularized Evolution for Image Classifier Architecture Search , Kevin Swersky et al., Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces , Kirthevasan Kandasamy et al., Neural Architecture Search with Bayesian Optimisation and Optimal Transport , Chenxi Liu et al., Progressive Neural Architecture Search , Arber Zela et al., Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search , Catherine Wong et al., Transfer Learning with Neural AutoML , Tianqi Chen et al., Net2Net: Accelerating Learning via Knowledge Transfer , Tao Wei et al., Network Morphism , Han Cai et al., Path-Level Network Transformation for Efficient Architecture Search , Han Cai et al., Efficient Architecture Search by Network Transformation , Thomas Elsken et al., Simple and Efficient Architecture Search for CNNs , Corinna Cortes et al., AdaNet: Adaptive Structural Learning of Artificial Neural Networks , Shreyas Saxena, Convolutional Neural Fabrics , Gabriel Bender et al., Understanding and Simplifying One-Shot Architecture Search , Hieu Pham et al., Efficient Neural Architecture Search via Parameter Sharing , Andrew Brock et al., SMASH: One-Shot Model Architecture Search through HyperNetworks , Hanxiao Liu et al., DARTS: Differentiable Architecture Search , Mingxing Tan, MnasNet: Platform-Aware Neural Architecture Search for Mobile , Rui Leite et al., Selecting Classification Algorithms with Active Testing , Salisu Mamman Abdulrahman et al., Speeding up algorithm selection using average ranking and active testing by introducing runtime , Martin Wistuba et al., Learning Hyperparameter Optimization Initializations , J. N. van Rijn et al., Hyperparameter Importance Across Datasets , Philipp Probst et al., Tunability: Importance of Hyperparameters of Machine Learning Algorithms , Martin Wistuba et al., Hyperparameter Search Space Pruning – A New Component for Sequential Model-Based Hyperparameter Optimization , C. E. Rasmussen et al.,Gaussian Processes for Machine Learning , Martin Wistuba et al., Scalable Gaussian process-based transfer surrogates for hyperparameter optimization , Matthias Feurer et al., Scalable Meta-Learning for Bayesian Optimization
April 27 , Automatic Machine Learning, part 1 Slides parts 1-2 , Slides part 3 .
References: Part 1: Book, chapter 1, On bayesian methods for seeking the extremum , Nando de Freitas et al., Exponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations , Kenji Kawaguchi et al., Bayesian Optimization with Exponential Convergence , Ziyu Wang et al., Bayesian Optimization in High Dimensions via Random Embeddings , Frank Hutter et al., Sequential Model-Based Optimization for General Algorithm Configuration , Kevin Swersky et al., Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces , Leo Breiman, Random Forests , Jasper Snoek et al., Scalable Bayesian Optimization Using Deep Neural Networks , Jost Tobias Springenberg, Bayesian Optimization with Robust Bayesian Neural Networks , James Bergstra, Algorithms for Hyper-Parameter Optimization , Hans Georg Beyer, Hans Paul Paul Schwefel, Evolution strategies –A comprehensive introduction , Nikolaus Hansen, The CMA Evolution Strategy: A Tutorial , Ilya Loshchilov et al., CMA-ES for hyperparameters optimization of neural networks , Tobias Domhan et al., Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves , Luca Franceschi et al., Bilevel Programming for Hyperparameter Optimization and Meta-Learning , Jelena Luketina et al., Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters , Aaron Klein et al, Learning curve prediction with Bayesian neural networks , Kevin Swersky, Multi-Task Bayesian Optimization , Kevin Swersky, Freeze-Thaw Bayesian optimization , Kirthevasan Kandasamy, Multi-fidelity Bayesian Optimisation with Continuous Approximations , Stefan Falkner et al., BOHB: Robust and Efficient Hyperparameter Optimization at Scale , Github link , Lisha Li et al., Hyperband: Bandit-based configuration evaluation for hyperband parameters optimiation , Kevin Jamieson, Non-stochastic Best Arm Identification and Hyperparameter Optimization , Chris Thornton et al., Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms , Brent Komer et al., Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn , Matthias Feurer et al., Efficient and Robust Automated Machine Learning, Auto-sklearn, GitHub link , Randal S. Olson et al., Automating Biomedical Data Science Through Tree-Based Pipeline Optimization
April 6 , Efficient Deep Learning with Humans in the Loop
References: Davis Liang et al., Learning Noise-Invariant Representations for Robust Speech Recognition , Zachary C. Lipton et al., BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems , Yanyao Shen et al., Deep Active Learning for Named Entity Recognition , Aditya Siddhant et al., Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study , David Lowell et al., Practical Obstacles to Deploying Active Learning , Peiyun Hu et al., Active Learning with Partial Feedback , Shish Khetan et al., Learning From Noisy Singly-labeled Data , Yanyao Shen et al. Deep Active Learning for Named Entity Recognition , Peiyun Hu et al. Active Learning with Partial Feedback , Jonathon Byrd et al., What is the Effect of Importance Weighting in Deep Learning? Jason Yosinski et al., Understanding Neural Networks Through Deep Visualization
March 30 , Studying Generalization in Deep Learning via PAC-Bayes
Few references: G.K Dziugaite, D. Roy, Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data , Huang et al., Stochastic Neural Network with Kronecker Flow , Zhou et al., Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach , Abadi et al., Deep Learning with Differential Privacy , R Herbrich, T Graepel, C Campbell, Bayes point machines , Neyshabur et al., The role of over-parametrization in generalization of neural networks , K Miyaguchi, PAC-Bayesian Transportation Bound
A little bit of background on probably approximately correct (PAC) learning: Probably Approximately Correct Learning , A primer on PAC-Bayesian learning
March 23 , Integrating Constraints into Deep Learning Architectures with Structured Layers
References: Honglak Lee, Roger Grosse, Rajesh Ranganath, Andrew Y. Ng, Convolutional Deep Belief Networksfor Scalable Unsupervised Learning of Hierarchical Representations . Brandon Amos, J. Zico Kolter, OptNet: Differentiable Optimization as a Layer in Neural Networks. Po-Wei Wang, Priya L. Donti, Bryan Wilder, Zico Kolter, SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud, Neural Ordinary Differential Equations . Shaojie Bai, J. Zico Kolter, Vladlen Koltun, Trellis Networks for Sequence Modeling. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Attention Is All You Need
March 2 , Rebooting AI February 24 , Is Optimization the Right Language to Understand Deep Learning? February 17 , Adversarial Machine Learning February 10 , Our Mathematical Universe February 3 , Nobel Lecture: Michel Mayor, Nobel Prize in Physics 2019 January 29 , How to Successfully Harness Machine Learning to Combat Fraud and Abuse
2019
December 16 & January 13, 2019-20, Variational Inference: Foundations and Innovations
December 2 & 9, 2019, Variational Inference: Foundations and Innovations
November 18, On Large Deviation Principles for Large Neural Networks
November 11, 2019, Anomaly Detection using Neural Networks
October 28 & November 4, 2019, Extreme Value Theory
October 7, 2019, On the Optimization Landscape of Matrix and Tensor Decomposition Problems
September 30, 2019, Recurrent Neural Networks
September 23, 2019, When deep learning does not learn,
July 15, 2019, Optimality in Locally Private Estimation and Learning
July 1, 2019. Capsule Networks
June 24, 2019, A multi-perspective introduction to the EM algorithm
June 17, 2019, Theoretical Perspectives on Deep Learning ,
May 27, 2019. 2018 ACM Turing Award Human in the Loop Reinforcement Learning
May 20, 2019. How Graph Technology Is Changing Artificial Intelligence and Machine Learning
May 13, 2019, 2017 Nobel Lectures in Physics
May 6, 2019, Accessorize to a Crime: Real and Stealthy Attacks on State-Of-The-Art Face Recognition paper
April 29, 2019, Build Intelligent Fraud Prevention with ML and Graphs
April 15, 2019, Active Learning: Why Smart Labeling is the Future of Data Annotation
April 8, 2019, Generalization, Interpolation, and Neural Nets
April 1, 2019, Similarity learning using deep neural networks
March 18/25, 2019, Deep Reinforcement Learning ( First lecture of MIT course 6.S091)
March 11, 2019, Ensembles: Boosting
March 4, 2019, Dataset shift in machine learning
February 25, 2019, Could Machine Learning Ever Cure Alzheimer’s Disease?
February 18, 2019, 2015 IAAA Winner Intelligent Surgical Scheduling System
February 11, 2019, Artificial Intelligence Machine Learning Big Data, Exponential Finance
February 4, 2019, Bayesian Deep Learning with Edward (and a trick using Dropout)
January 28, 2019, Ouroboros ,
January 21, 2019, Cosmos Proof of Stake
January 14, 2019, Geometric Deep Learning
January 7, 2019, Deep Generative Networks as Inverse Problems
2018
