Jiachang Liu

fistName.lastName at duke.edu

Duke University

Durham, NC, 27705

I am a Ph.D. candidate studying machine learning at Duke University. My advisor is Professor Cynthia Rudin.

My main research interest is in interpretable machine learning. My goal is to create simple and sparse models that can fit into the palm of a person’s hand but still give accurate predictions. Simple models help us reveal the underyling data pattern and allow people with less technical background to engage in the data science process by bringing with their domain knowledge.

My research is organized around three thrusts:

1) Finding practical ML problems where interpretability matters greatly but performance or computational speed lags behind;

2) Designing efficient discrete and continuous optimization techniques to solve sparse ML problems, which are usually nonconvex and have a combinatorial nature; and

3) Building robust open-source code by leveraging efficient data structures and careful implementation to provide user-friendly softwares/packages for practitioners in the broad ML/statistics community.

Before coming to Duke, I earned my B.S. degree with double majors in physics and mathematics and a minor in computer science from University of Michigan, Ann Arbor in 2018.

news

Oct 5, 2023	Together with Cynthia Rudin, Margo Seltzer, and Chudi Zhong, I was awarded the Bell Labs Prize 2023 at 2nd place for work on trustworthy AI.

Publications

NeurIPS

OKRidge: Scalable Optimal k-Sparse Ridge Regression for Learning Dynamical Systems

Jiachang Liu, Sam Rosen, Chudi Zhong, and Cynthia Rudin

Advances in Neural Information Processing Systems (NeurIPS), Spotlight 2023

Abs PDF Code

We consider an important problem in scientific discovery, identifying sparse governing equations for nonlinear dynamical systems. This involves solving sparse ridge regression problems to provable optimality in order to determine which terms drive the underlying dynamics. We propose a fast algorithm, OKRidge, for sparse ridge regression, using a novel lower bound calculation involving, first, a saddle point formulation, and from there, either solving (i) a linear system or (ii) using an ADMM-based approach, where the proximal operators can be efficiently evaluated by solving another linear system and an isotonic regression problem. We also propose a method to warm-start our solver, which leverages a beam search. Experimentally, our methods attain provable optimality with run times that are orders of magnitude faster than those of the existing MIP formulations solved by the commercial solver Gurobi.
NeurIPS

FasterRisk: Fast and Accurate Interpretable Risk Scores

Jiachang Liu*, Chudi Zhong*, Boxuan Li, Margo Seltzer, and Cynthia Rudin

Advances in Neural Information Processing Systems (NeurIPS) 2022

Abs PDF Code

Over the last century, risk scores have been the most popular form of predictive model used in healthcare and criminal justice. Risk scores are sparse linear models with integer coefficients; often these models can be memorized or placed on an index card. Typically, risk scores have been created either without data or by rounding logistic regression coefficients, but these methods do not reliably produce high-quality risk scores. Recent work used mathematical programming, which is computationally slow. We introduce an approach for efficiently producing a collection of high-quality risk scores learned from data. Specifically, our approach produces a pool of almost-optimal sparse continuous solutions, each with a different support set, using a beam-search algorithm. Each of these continuous solutions is transformed into a separate risk score through a "star ray" search, where a range of multipliers are considered before rounding the coefficients sequentially to maintain low logistic loss. Our algorithm returns all of these high-quality risk scores for the user to consider. This method completes within minutes and can be valuable in a broad variety of applications.
AISTATS

Fast Sparse Classification for Generalized Linear and Additive Models

Jiachang Liu, Chudi Zhong, Margo Seltzer, and Cynthia Rudin

In International Conference on Artificial Intelligence and Statistics 2022

Abs PDF Code

We present fast classification techniques for sparse generalized linear and additive models. These techniques can handle thousands of features and thousands of observations in minutes, even in the presence of many highly correlated features. For fast sparse logistic regression, our computational speed-up over other best-subset search techniques owes to linear and quadratic surrogate cuts for the logistic loss that allow us to efficiently screen features for elimination, as well as use of a priority queue that favors a more uniform exploration of features. As an alternative to the logistic loss, we propose the exponential loss, which permits an analytical solution to the line search at each iteration. Our algorithms are generally 2 to 5 times faster than previous approaches. They produce interpretable models that have accuracy comparable to black box models on challenging datasets.
ACL-W

What Makes Good In-Context Examples for GPT-3?

Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen

In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Association for Computational Linguistics 2022

Abs PDF Code

GPT-3 has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its powerful and versatile in-context few-shot learning ability. Despite its success, we found that the empirical results of GPT-3 depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3’s few-shot capabilities. Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt. Intuitively, the in-context examples selected with such a strategy may serve as more informative inputs to unleash GPT-3’s extensive knowledge. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (41.9% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset). We hope our investigation could help understand the behaviors of GPT-3 and large-scale pre-trained LMs in general and enhance their few-shot capabilities.
TPAMI

Representing Graphs via Gromov-Wasserstein Factorization

Hongteng Xu, Jiachang Liu, Dixin Luo, and Lawrence Carin

IEEE Transactions on Pattern Analysis and Machine Intelligence 2022

Abs PDF

We propose a new nonlinear factorization model for graphs that have topological structures, and optionally, node attributes. This model is based on a pseudo-metric called Gromov-Wasserstein (GW) discrepancy, which compares graphs in a relational way. It estimates observed graphs as the GW barycenters constructed by a set of graph bases with different weights. By minimizing the difference between each observed graph and its GW barycenter-based estimation, we learn the graph bases and their weights associated with the observed graphs. The model achieves a novel and flexible factorization mechanism for graphs, in which both the observed graphs and the learnable graph bases can be unaligned and with different sizes. We design an effective approximate algorithm for learning this Gromov-Wasserstein factorization (GWF) model, unrolling loopy computations as stacked modules and computing gradients with backpropagation. For each observed graph, the weights lead to its permutation-invariant representation. These weights can be parametrized by a graph neural network so that we can extend the proposed transductive GWF model to an inductive version. Compared with state-of-the-art methods, our GWF method can represent graphs with better interpretability and using lower dimensionality, while simultaneously achieving encouraging results in graph clustering and classification task.
ICML

CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin

In International Conference on Machine Learning 2020

Abs PDF Code

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.