Logit or Leave It?

By Kristof Zetenyi and Tom Beckford^*

The logit model has recently come under scrutiny due to an order issued by Judge Donato in In re: Google Play Store Antitrust Litigation. Judge Donato excluded the plaintiffs’ merits expert’s analysis due to its reliance on a pass-through formula derived from the logit model, finding that the characteristics of the logit model and its underlying independence from irrelevant alternatives (“IIA”) property were not sufficiently reliable to be admitted under Rule 702.¹ In this article, we take a deeper dive into the logit demand model and its IIA property and discuss what the future might hold for the logit demand model in the context of expert analysis.

I. Background on the Logit Model

The logit demand model was developed in the economic literature in the 1960s and 1970s, and its introduction was considered a methodological breakthrough in economics.² Indeed, the development of the logit model was cited as part of the work for which Daniel McFadden won the Nobel Prize in Economic Sciences in 2000.³ The logit demand model was the first so-called discrete choice model, which is a type of model used to analyze consumer choices when facing a set of distinct product options with defined product attributes.⁴ In discrete choice models, each product is modeled as a set of observable attributes and an unobserved taste component that varies across consumers (and that the econometrician cannot observe). The model attempts to estimate the value – so-called utility – that consumers derive from each of the observable product attributes analyzed.⁵ The structure and assumptions that underlie these models allow economists to predict the share of consumers that will choose each product in the market. Moreover, since a product is modeled as a set of product attributes, discrete choice models like the logit model can predict consumer demand if certain product attributes change, new products are introduced, or existing products are removed from the marketplace.⁶

In a well-known academic paper, Amil Petrin used such a model to analyze the economic impact of the introduction of the minivan.⁷ Petrin estimated a discrete choice model for consumer demand for vehicles, including product attributes for acceleration, vehicle dimensions, drive type, fuel efficiency, air conditioning as standard (to measure luxury), and price, which he then used to estimate the impact of the minivan’s 1984 entry to the market on prices for other vehicles, consumer standard of living, and manufacturer profits.⁸

These discrete choice models lend themselves well to expert analysis of consumer demand in alternative “but-for” worlds. For example, if someone was interested in modeling the demand for smartphones, they might consider studying consumer preferences of product attributes such as price, display type and size, chip speed, camera resolution, storage capacity, battery life, wireless charging capabilities, and waterproofness. Once consumer preferences for these attributes are estimated, one could model various but-for worlds where the product options are altered in some way. For example, if a hypothetical plaintiff had alleged that the use of certain smartphone chips had been in violation of the plaintiff’s patents, a but-for scenario might calculate the value differential to consumers associated with a hypothetical smartphone with a lower-quality chip.

The development of the logit demand model blended economic theory and econometric analysis in a computationally practical way, which was important given the computing power available at the time of its introduction.⁹ To achieve tractability required restrictive assumptions, the most notable being that the unobserved taste component that varies across consumers was assumed to be distributed following a very specific distribution called the Gumbel distribution.¹⁰ This distribution’s mathematical properties reduce the computational burden of estimating these logit demand models, making them an attractive choice for characterizing differences in consumers’ preferences between products in a tractable way. However, it also comes with the drawback of the IIA property.

II. What Is the IIA Property?

The IIA property means, in simple terms, that a consumer’s relative likelihood of choosing product A over product B does not change if product C is added to the choice set.

To illustrate the consequence of the IIA property, consider a football fan deciding how to get to the stadium.¹¹ Suppose that they can go in a taxi or by the blue buses operated by the city, and there is a 50% chance that they will pick either option. Now suppose that the city rolls out red buses as a third transportation option, but blue and red buses are functionally equivalent from the perspective of the football fan who does not care about the color of the bus when choosing their means of transport. Therefore, the choice between a taxi or a bus should be unaffected by the introduction of the new but otherwise irrelevant red bus alternative into the football fan’s choice set – i.e., the football fan should still have a 50% chance of picking either mode of transport.¹²

However, the IIA property requires that relative chances of pairs of alternatives need to be unaffected by a third alternative. Without red buses as an option, the probability of taking a taxi or a blue bus was the same (i.e., 50-50), and so IIA requires that this ratio of probabilities stay the same once red buses become an option. Therefore, with red buses as a third option and with otherwise no functional differences to blue buses, the IIA property implies that the predicted probability of the football fan taking a taxi, the blue bus, and the red bus are all the same (i.e. 33% each).¹³ Thus, a model with the IIA property will predict that the introduction of the red buses results in a 17 percentage point reduction in the probability that the football fan will take a taxi (from 50 to 33%) and an equivalent increase in the probability that the football fan will take the bus (from 50 to 67%). This outcome is inconsistent with economic intuition and common sense.

The implication of the IIA property for the logit demand model is a strong restriction on product substitution patterns in response to some change in prices, product attributes, or the set of products available. The IIA property means that substitution depends only on the market shares of the products, so-called proportional shifting, and not on any of the product attributes or variation in consumer tastes for these attributes – so-called heterogeneity in consumer tastes.¹⁴ Therefore, the central criticism of the logit model is that these substitution patterns do not depend on any measure of how similar the products are relative to other products that are considered – the market shares determine substitution. However, real-life substitutions patterns do not necessarily need to align with market shares, and thus using a model that requires this to be the case can lead to economically unrealistic predictions of substitution patterns.

III. The Use of Logit Demand Models and the IIA Property in Expert Analysis

Judge Donato’s order extensively discussed the impact of the IIA property on plaintiffs’ expert’s analysis of apps in the Google Play Store.¹⁵ The plaintiffs’ expert used as an input into his calculation of aggregate damages a logit-based formula to calculate the pass-through rates for each broad category of apps used by the Google Play Store (such as “Health and Fitness” and “Productivity”). However, the reliability of the pass-through rate depends on whether the plaintiffs’ expert reliably estimated the underlying logit demand systems for each of the categories because, as Judge Donato noted, “[t]here is an economic consensus that if real world demands do not satisfy this property, then the model will yield unreliable results.”¹⁶ To illustrate the IIA problem, Google’s expert argued that some apps within these broad categories will not be substitutes and so specifying a logit model in this way, with the proportional shifting property, is inappropriate.¹⁷ Google’s expert considered two language learning apps in the “Education” category(Rosetta Stone with less than a 5% share and Duolingo with around a 15% share) and another app with completely different functionality but still within the “Education” category (“PictureThis – Plant Identifier” (with around a 20% share).¹⁸ The language learning apps and PictureThis – Plant Identifier would not logically be substitutes, but the IIA property of the logit model imposed a substitution pattern such that if the price of the Rosetta Stone app was raised, customers would substitute more to PictureThis – Plant Identifier than to Duolingo because it had a higher share within the “Education” category.¹⁹ This makes no economic sense and illustrates the potential flaw with the logit model and its IIA property if the model is not used in an appropriate context. Ultimately, Judge Donato ruled that the plaintiffs’ expert’s pass-through model “is not within accepted economic theory and literature [] and does not give a jury a sound basis to make a reasoned and reasonable judgement about antitrust impact and damages in a product market that does not show proportional substitution across alternatives.” 20

Many commonly used models similarly assume an underlying Gumbel distribution and as such also suffer from the restrictions resulting from the IIA property. For example, antitrust agencies and merging parties have often used a second-score auction (“SSA”) model to evaluate the potential effects of a merger or acquisition on competition.²¹ An SSA consists of a single round of sealed bids in which all suppliers participate. An SSA assigns scores to each prospective supplier based on the suppliers’ bids and generates a ranking of suppliers based on these scores. The transaction price is determined using both the winning and the runner-up suppliers’ bids (hence the name second-score auction).

The standard form of the SSA model was originally developed by Nathan Miller to study a merger of suppliers in a market that is assumed to have many buyers.²² The standard and most commonly used form of the SSA model assumes an underlying Gumbel distribution, and as such features the IIA property as well. In this context, the IIA property plays a critical role in the determination of the frequency with which merging parties are the winner and the runner up in the SSA, which in turn is pivotal in assessing the likely competitive effects of a proposed merger.

For example, suppose that in the transportation business supplier, A has a 10% market share and supplier B has a 20% market share. In the context of the SSA model, a merger between suppliers A and B will harm buyers if suppliers A and B were winner and runner up in an auction.²³ Rather than using actual data on the frequency with which suppliers A and B were winner and runner up, the SSA model relies on the IIA property to predict, using market shares alone, the frequency with which suppliers A and B were winner and runner up. In our example, the IIA property implies that supplier A is the winner 10% of the time (because its market share is 10%), and in those instances supplier B is runner up 22% of the time (that is, 20%/(100%-10%)). Similarly, the IIA property implies that supplier B is the winner 20% of the time (because its market share is 20%), and in those instances supplier A is runner up 12.5% of the time (that is, 10%/(100%-20%)). The actual frequency of this occurring can range from zero (say, if supplier A specializes exclusively on long-haul transport and supplier B specializes in short distances) to 100% (say, if suppliers A and B are the only suppliers catering to college students), and anywhere in between. The impact of the distributional assumption underlying this SSA model is therefore similar to its impact on the logit model – the model is tractable and its predictions are simple to calculate, but it relies on strong and potentially unrealistic assumptions.²⁴

IV. What Is the Future of the Logit Model in Expert Analysis?

As the discussion above illustrates, the use of the logit model conveys tractability but can have limitations depending on context due to the strong assumptions it requires.²⁵ In light of Judge Donato’s recent decision, a key question for any expert who is considering the use of a logit model is whether their particular logit model as specified is appropriate for the products and consumer preferences being analyzed. To evaluate whether their logit model is appropriate, experts have a couple of options.

First, an expert evaluating their reliance on a logit model with the IIA property can perform statistical tests developed in the economic literature to assess whether the IIA property is a close enough approximation of the products and consumer preferences being analyzed.²⁶ One such test, if the data allow, essentially checks the IIA property in reverse – that is, instead of considering adding a choice alternative, as explained in the example above, the test eliminates one or more alternatives to see if consumer behavior obeys the IIA property, and assesses whether these two results are statistically different.²⁷ Another test adds so-called cross-alternative variables to the regression, which are attributes from one alternative that enter the utility of another alternative. The IIA property can be rejected if the attributes of a third alternative affect the ratio of probabilities of two other alternatives.²⁸

Second, an expert can evaluate whether relaxing the assumptions that lead to the IIA property results in meaningfully different results. Extensions and more complex discrete choice models have been developed that relax the IIA property while retaining other key insights of the logit demand model. Such extensions are well understood in the economic literature, and a number of studies compare the performance of these models to the standard logit model, at times highlighting unrealistic substitution patterns resulting from the IIA property.²⁹ Several of these alternative discrete-choice models are described below.

Nested logit models separate choices into “nests” or groups of choices. The IIA property continues to hold for choices within a nest but is relaxed between nests.³⁰ For example, a worker when commuting might choose between transit, consisting of either a bus or a train, or driving, consisting of either driving alone or carpooling.³¹ Progressively more complicated nested logit models can have increased levels of nests or overlapping nests, which subsequently allow the IIA property to be further relaxed, but such nested structures require additional parameters to be estimated.³²

Mixed (or random-coefficients) logit models are generalized versions of logit, where the ratio of probabilities for two choices depends on data for all available choices, and thus the model does not exhibit the IIA property.³³ These models can produce demand elasticities that are more realistic than the basic logit model, but at the cost of increased computational complexity. Therefore, applications of utilizing the full power of the mixed logit became more common starting in the 1990s, as computational power increased and improved simulation methods of estimating these models were developed.³⁴

Mixed logit models that rely only on market-level data have also been developed.³⁵ Among the most well-known models of this category is that developed by Berry, Levinsohn, and Pakes, or as it is often called, the BLP model.³⁶ The BLP model builds on the mixed logit model by only using market-level price and quantity data and dealing with the endogeneity of prices.³⁷ As such, it retains the benefits of the mixed logit model by relaxing the assumptions that lead to the IIA property and allowing for more realistic substitution patterns, but it may have more demanding data requirements and additional econometric considerations that need to be overcome.

More broadly, in light of Judge Donato’s recent decision, it may be prudent for economic experts to perform a careful assessment of the appropriateness of relying on any models that exhibit the IIA property. Whether or not the reliance on any model with the IIA property is appropriate for the marketplace being studied is an empirical question. Tests are widely available to help with this assessment. For example, in the context of the SSA model, one can empirically check if the SSA model provides a plausible prediction. Miller (2014) proposed comparing model predictions to the actual margins of individual firms.³⁸ Alternatively, the sensitivity of the SSA model’s predictions can be evaluated if alternative distributional assumptions are used.³⁹ Finally, customer substitution surveys or win/loss data could also be used to check for consistency with the IIA property, but of course such sources of data should also be carefully assessed for reliability.⁴⁰

V. Conclusion

Judge Donato’s opinion cast the spotlight on a well-known weakness of the logit demand model: the IIA property. Models that exhibit the IIA property can generate unrealistic predictions due to the proportional shifting substitution patterns that the IIA property engenders. We discussed a number of statistical tests of the appropriateness of the IIA property, as well as extensions to the simple logit model that generate more realistic substitution patterns. Economic experts considering the use of models relying on the IIA property should perform statistical tests to assess whether the IIA property⁴¹ is a close enough approximation of the products and consumer preferences being analyzed or should assess whether relaxing the assumptions that lead to the IIA property results in meaningfully different results.

Click here for a PDF version of the article

* Kristof Zetenyi is Vice President of Analysis Group. Tom Beckford is a Manager at Analysis Group.

1 See, Order re Merits Opinions of Dr. Hal J. Singer, In re Google Play Store Antitrust Litigation, 21-md-02981-JD (N.D. Cal. Aug. 28, 2023), at 4, 17.

2 See, Charles F. Manski, Daniel McFadden and the Econometric Analysis of Discrete Choice, 103 The Scandinavian Journal of Economics 217, 217-224 (2001).

3 See, Daniel L. McFadden Facts, The Nobel Prize (2024), https://www.nobelprize.org/prizes/economic-sciences/2000/mcfadden/facts/.

4 See, e.g., Kenneth E. Train, Discrete Choice Methods with Simulation 3-5, (Cambridge University Press, 2^nd Ed. 2009); Manski, supra note 2, at 220.

5 See, Manski, supra note 2, at 220-223; Daniel L. McFadden, Conditional Logit Analysis of Qualitative Choice Behavior, in FRONTIERS IN ECONOMETRICS 105 (Paul Zarembka ed., 1974).

6 See, Manski, supra note 2, at 219-223; Aviv Nevo, A Practitioner’s Guide to Estimation of Random-Coefficients Logit Models of Demand, 9 Journal of Economics and Management Strategy 513, 517-521 (2000); Amil Petrin, Quantifying the Benefits of New Products: The Case of the Minivan, 11 Journal of Policial Economy 705 (2002).

7 See, Petrin, id.

8 See, Petrin, supra note 6, at 712, 718-726.

9 See, Manski, supra note 2, at 119-220, 223.

10 See, Manski, supra note 2, at 223-225; Daniel L. McFadden, Regression-Based Specification Tests for the Multinomial Logit Model, 34 Journal of Econometrics 108 109-114 (1987)

11 The inappropriateness of the IIA property was first pointed out by Chipman and Debreu. The following example of the inappropriateness of the IIA property follows the famous red bus blue bus illustration. Train, supra note 4, at 50.

12 Because the sports fan doesn’t care about the color of bus, the probabilities should be a 50% chance of going by taxi, a 25% chance of taking a red bus, and a 25% chance of taking a blue bus.

13 See, Train, supra note 4, at 46. See also, Gerald Debreu, Review of Individual Choice Behavior: A Theoretical Analysis by R. Duncan Luce, 50 American Economic Review 186 (1960).

14 See, e.g., Train, supra note 4, at 47 (“This pattern of substitution, which can be called proportionate shifting, is a manifestation of the IIA property. The ratio of probabilities for alternatives i and k stays constant when an attribute of alternative j changes only if the two probabilities change by the same proportion.”); Nevo, supra note 6, at 515 (“A problem with this [logit demand] model is the strong implication of some of the assumptions made. Due to the restrictive way in which heterogeneity is modeled, substitution between products is driven completely by market shares and not by how similar the products are.”).

15 Order re Merits Opinions of Dr. Hal J. Singer, supra note 1, at 10-17.

16 Id., at 11.

17 Id., at 9-10, 13.

18 See, Id., at 13.

19 Id..

20 Id., at 16-17.

21 See, e.g., United States of America, et al., v. Anthem, Inc., et al., 16-1493 (D.D.C. 2016); Federal Trade Commission, et al. v. Sysco Corporation, et al. 1:15–cv–00256 (D.D.C. 2015); Federal Trade Commission v. Wilhelm Wilhelmsen, et al. 18-cv-00414-TSC (D.D.C. 2018); Federal Trade Commission v. Rag-Stiftung, Evonik, et al. 19-cv-02337 (D.D.C. 2019); United States of America v. United States Sugar Corporation, et al. 21-1644 (D. Del. 2021); United States of America v. Bertelsmann SE & Co. KGaA, Penguin Random House, LLC, et al. 21-cv-02886(D.D.C. 2021); In the Matter of IQVIA Holdings Inc. and Propel Media, Inc. 9416 (S.D. N.Y. 2024).

22 See, Nathan Miller, Modeling the Effects of Mergers in Procurement, 37 International Journal of Industrial Organization 201, 203 (2014).

23 For a detailed explanation of the mechanics of the SSA model, see e.g., Chanont “Big” Banternghansa, Maria Eugenia Garibotti, & Kristof Zetenyi, Efficiencies in the Second-Score Auction Model, 2-2023 Concurrences 1, 2 (2023) (“Merging suppliers are typically assumed to continue offering the products offered by each standalone supplier. Knowing the value and cost of each product, the merged supplier will have an incentive to submit only one bid—the one with the highest surplus, which is the mechanism through which there can be reduced competition. […] In this way, a reduction in competition can be seen when the second-best alternative worsens as a result of the merger”).

24 See, Miller, supra note 20, at 206.

25 We note that a logit model can capture systematic taste variation that is related to observed characteristics of the decision marker, but it cannot capture random taste variation. To some extent, logit models can also capture the dynamics of repeated choices. Train, supra note 4, at 42.

26 See, McFadden, supra note 5, at 64 (“This [IIA property] is a strong restriction on the structure of response probabilities which should be tested in most applications.”). See also, Train, supra note 4, at 49-50; and Jerry Hausmann & Daniel L. McFadden, Specification Tests for the Multinominal Logit Model, 52 Econometrica, 1219, 1219-20 & 1238-39 (1984).

27 These are tests of the hypothesis that the parameters on the subset of alternatives are the same as the full set of alternatives, and thus is a test of IIA. Train, supra note 4, at 53-54; Hausmann & McFadden, supra note 24, at 1220; Kenneth A Small and Cheng Hsiao, Multinomial Logit Specification Tests, 26 International Economic Review 619 (1985). The mixed logit discussed below can also be used as test of IIA because the simple logit model is a special case of the mixed logit model, which occurs when the variance of the mixing distribution of the mixed logit is zero. Therefore, testing whether the variance of the mixing distribution of the mixed logit is zero is a test of IIA. Train, supra note 4, at 54.

28 See, Train, supra note 4, at 54.

29 See, e.g., David Brownstone and Kenneth Train, Forecasting New Product Penetration with Flexible Substitution Patterns, 89 Journal of Econometrics 109-129 (1999); Laura Grigolon and Frank Verboven. Nested Logit or Random Coefficients Logit? A Comparison of Alternative Discrete Choice Models of Product Differentiation, 96 The Review of Economics and Statistics 916-35 (2014).

30 See, Train, supra note 4, at 81-83.

31 See, Train, supra note 4, at 81-83.

32 See, Train, supra note 4, at 90-96. Another discrete choice model that has been used by economists is the probit model. Probit models utilize a multivariate normal distribution for the unobserved components of utility, which relaxes the IIA property, as well as allowing for random tasted variation. However, probit models have another known and notable weakness for estimating consumers’ price sensitivity, whereby the model allows for some consumers to prefer more expensive alternatives, all else equal. See, Train, supra note 4, at 101. This property violates the law of demand and is considered exceptionally rare. Such goods are called Giffen goods, which may have been observed only once or twice in history. See, e.g., N. Gregory Mankiw, Principles of Economics, 6 Cengage Learning, 453-454 (2012).

33 See, Train, supra note 4, at 138, 145.

34 See, Train, id.; Manski, supra note 2, at 226.

35 See, e.g., Nevo, supra note 6.

36 See, Steven Berry, James Levinsohn, and Ariel Pakes, Automobile Prices in Market Equilibrium, 63 Econometrica 841-90 (1995).

37 See, Nevo, supra note 6, at 515-516, 526-528.

38 See, Miller, supra note 20, at 206.

39 See, Martino De Stefano, Keler Marku, and Yianis Sarafidis, The Effect of Functional Form Assumptions on Merger Price Effects in Second-Score Auction Models, SSRN Electronic Journal (2020) 10.2139/ssrn.3639703.

40 See, Miller, supra note 20, at 206.