Data Science
-
Computational constrained optimization 1: Lagrange multipliers - January 31, 2022
In my post on the principle of maximum entropy I showed how choosing good priors in Bayesian modeling can be expressed as a constrained optimization problem, using (relative) entropy as the objective function. This post introduces the general technique of Lagrange multipliers, as well as the concept of a partition function which simplifies calculations when the optimization objective involves entropy.
-
The principle of maximum entropy - January 27, 2022
What is the correct way to select a prior in Bayesian modeling? This is a deep question which leads naturally to the principle of maximum entropy, a fundamental tool in statistics, machine learning, and beyond.
-
The subtleties of computing quantiles - June 21, 2020
Computing quantiles is a good way to summarize the distribution of a numerical dataset. But confusingly there are nearly a dozen different definitions of quantile that can all claim to be correct, and in all cases it is difficult to actually compute quantiles for very large datasets. I will explain why there are so many definitions, and compare a couple of different strategies for doing computations at scale.
-
Gibbs' Inequality - May 27, 2019
It is a fairly standard fact that relative entropy (KL-divergence) is positive definite, but I was unsatisfied with the proofs of this fact that I saw when I glanced through the literature. In this post I will provide a complete proof which works on a general probability space.
-
Annotator Agreement in Machine Learning Experiments - February 12, 2018
It is standard practice in applied machine learning to use human annotated data to build training and evaluation datasets. But these datasets can be compromised when the annotation task is vague or the annotators disagree on the labels to be applied to an observation. The standard way to assess the quality of the results of an annotation experiment is to measure the rate of agreement between annotators, but this calculation is not as straightforward as one might naively expect.