ETS: Efficient Tree Search for Inference-Time Scaling

Coleman Hooper, Sehoon Kim, Suhong Moon, Kerem Dilmen, Monishwaran Maheswaran, Nicholas Lee, Michael W Mahoney, Sophia Shao, Kurt Keutzer, Amir Gholami

Preprint, 2025

Code

QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache

Rishabh Tiwari*, Haocheng Xi*, Aditya Tomar*, Coleman Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W Mahoney, Kurt Keutzer, Amir Gholami

Preprint, 2025

Squeezed Attention: Accelerating Long Context Length LLM Inference

Coleman Hooper*, Sehoon Kim*, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

Preprint, 2024

TinyAgent: Function Calling at the Edge

Lutfi Eren Erdogan*, Nicholas Lee*, Siddharth Jha*, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, Amir Gholami

Empirical Methods in Natural Language Processing (EMNLP) Demo Track, 2024

Code

Blog

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

Conference on Neural Information Processing Systems (NeurIPS), 2024

Code

Efficient and Scalable Estimation of Tool Representations in Vector Space

Suhong Moon*, Siddharth Jha*, Lutfi Eren Erdogan, Sehoon Kim, Woosang Lim, Kurt Keutzer, Amir Gholami

Preprint, 2024

Code

Characterizing Prompt Compression Methods for Long Context Inference

Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Kurt Keutzer, Amir Gholami

ICML Workshop on Efficient Systems for Foundation Models (Oral), 2024

Learned Best-Effort LLM Serving

Siddharth Jha, Coleman Hooper, Xiaoxuan Liu, Sehoon Kim, Kurt Keutzer

ICML Workshop on Efficient Systems for Foundation Models, 2024

LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

Nicholas Lee*, Thanakul Wattanawong*, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, Amir Gholami

The Association for Computational Linguistics (ACL), 2024

Code

An LLM Compiler for Parallel Function Calling

Sehoon Kim*, Suhong Moon*, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

International Conference on Machine Learning (ICML), 2024

Code

Talk

SqueezeLLM: Dense-and-Sparse Quantization

Sehoon Kim*, Coleman Hooper*, Amir Gholami*, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer

International Conference on Machine Learning (ICML), 2024

Code

AI and Memory Wall

Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer

IEEE MICRO Journal Special Issue, 2024

Blog

SPEED: Speculative Pipelined Execution for Efficient Decoding

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, Sophia Shao

NeurIPS Workshop on Efficient Natural Language and Speech Processing, 2023

Full Stack Optimization of Transformer Inference: a Survey

Sehoon Kim*, Coleman Hooper*, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, Amir Gholami

Preprint, 2023 (Short Version at ISCA ASSYST Workshop 2023)

Speculative Decoding with Big Little Decoder

Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, Kurt Keutzer

Conference on Neural Information Processing Systems (NeurIPS), 2023

Code

Squeezeformer: An Efficient Transformer for Automatic Speech Recognition

Sehoon Kim*, Amir Gholami*, Albert Shaw†, Nicholas Lee†, Karttikeya Mangalam, Jitendra Malik, Michael W. Mahoney, Kurt Keutzer

Conference on Neural Information Processing Systems (NeurIPS), 2022

Code

A Fast Post-Training Pruning Framework for Transformers

Woosuk Kwon*, Sehoon Kim*, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, Amir Gholami

Conference on Neural Information Processing Systems (NeurIPS), 2022

Code

Learned Token Pruning for Transformers

Sehoon Kim*, Sheng Shen*, David Thorsley*, Amir Gholami*, Woosuk Kwon, Joseph Hassoun, Kurt Keutzer

Conference on Knowledge Discovery and Data Mining (KDD), 2022

Code

Integer-only Zero-shot Quantization for Efficient Speech Recognition

Sehoon Kim, Amir Gholami, Zhewei Yao, Nicholas Lee, Patrick Wang, Anirudda Nrusimha, Bohan Zhai, Tianren Gao, Michael W. Mahoney, Kurt Keutzer

International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2022

Code

Hessian-Aware Pruning and Optimal Neural Implant

Shixing Yu*, Zhewei Yao*, Amir Gholami*, Zhen Dong*, Sehoon Kim, Michael W. Mahoney, Kurt Keutzer

Winter Conference on Applications of Computer Vision (WACV), 2022

Code

A Survey of Quantization Methods for Efficient Neural Network Inference

Amir Gholami*, Sehoon Kim*, Zhen Dong*, Zhewei Yao*, Michael W. Mahoney, Kurt Keutzer

Book Chapter: Low-Power Computer Vision: Improving the Efficiency of Artificial Intelligence, 2021

WindTunnel: Towards Differentiable ML Pipelines Beyond a Single Model

Gyeong-In Yu, Saeed Amizadeh, Sehoon Kim, Artidoro Pagnoni, Ce Zhang, Byung-Gon Chun, Markus Weimer, Matteo Interlandi

International Conference on Very Large Data Bases (VLDB), 2021

Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs

Taebum Kim, Eunji Jeong, Geon-Woo Kim, Yunmo Koo, Sehoon Kim, Gyeong-In Yu, Byung-Gon Chun

Conference on Neural Information Processing Systems (NeurIPS), 2021

I-BERT: Integer-only BERT Quantization

Sehoon Kim*, Amir Gholami*, Zhewei Yao*, Michael W. Mahoney, Kurt Keutzer

International Conference on Machine Learning (ICML, Oral), 2021

Code

Memory-Efficient Hardware Performance Counters with Approximate-Counting Algorithms

Jingyi Xu, Sehoon Kim, Borivoje Nikolic, Yakun Sophia Shao

International Symposium on Performance Analysis of Systems and Software (ISPASS), 2021