1 1 UNITED STATES DISTRICT COURT 2 FOR THE EASTERN DISTRICT OF MICHIGAN 3 SOUTHERN DIVISION 4 5 BARBARA GRUTTER, 6 For herself and all others 7 Similarly situated -- 8 Plaintiff, 9 -v- Case Number: 97-CV-75928 10 LEE BOLLINGER, JEFFREY LEHMAN, 11 DENNIS SHIELDS, and REGENTS OF 12 THE UNIVERSITY OF MICHIGAN, 13 Defendants, 14 And 15 KIMBERLY JAMES, et al., 16 Intervening Defendants. 17 ------------------------------------/ VOLUME 12 18 BENCH TRIAL 19 BEFORE THE HONORABLE BERNARD A. FRIEDMAN 20 United States District Judge 21 238 U.S. Courthouse & Federal Building 22 231 Lafayette Boulevard West 23 Detroit, Michigan 24 SATURDAY, FEBRUARY 10, 2001 25 2 1 APPEARANCES: 2 3 FOR PLAINTIFF: Kirk O. Kolbo, Esq. 4 R. Lawrence Purdy, Esq. 5 6 FOR DEFENDANTS: John Payton, Esq. 7 Craig Goldblatt, Esq. 8 Stuart Delery, Esq. 9 On behalf of Defendants. 10 11 George B. Washington, Esq. 12 Miranda K.S. Massie, Esq. 13 On behalf of Intervening Defendants. 14 15 COURT REPORTER: Joan L. Morgan, CSR 16 Official Court Reporter 17 18 19 20 21 22 23 24 25 3 1 I N D E X 2 3 WITNESS: PAGE: 4 5 ORAL ARGUMENTS RE: WITNESS, GAIL HERIOT 6 By Mr. Goldblatt 5 7 By Mr. Washington 10 8 By Mr. Kolbo 11 9 10 REBUTTAL WITNESS CALLED ON BEHALF OF PLAINTIFF 11 KINLEY LARNTZ, PhD 12 Direct Examination by Mr. Kolbo 19 13 Cross Examination by Mr. Delery 56 14 Cross Examination by Ms. Massie 101 15 16 17 E X H I B I T S 18 MARKED RECEIVED 19 Trial Exhibit 225 54 20 Trial Exhibit 226 54 21 Trial Exhibit 227 54 22 23 24 25 4 1 Detroit, Michigan 2 Saturday, February 10, 2001 3 (At or about 8:33 a.m.) 4 -- --- -- 5 THE COURT: Let's talk first, if we might just take 6 a second, and we have -- 7 MR. WASHINGTON: Could we wait just a minute? I 8 think Ms. Massie is having car trouble, and she is going to 9 be handling today, so if we could wait for a minute, I'd 10 appreciate it. 11 THE COURT: Oh, I'm sorry. I didn't notice she 12 wasn't here. Of course, it may go much quicker. 13 MR. WASHINGTON: I understand. 14 THE COURT: I'm kidding. No, we will definitely 15 wait. 16 Can we argue the motion in terms of the witness, 17 Gail Heriot, or is Ms. Massie handling that, also? 18 MR. WASHINGTON: I can argue that, I suppose. 19 THE COURT: It's up to you. 20 MR. WASHINGTON: We are really supporting the 21 University's position, so, yeah, fine. We would be happy 22 to do that. 23 MR. PAYTON: Your Honor, Mr. Goldblatt will be 24 arguing this motion. 25 MR. GOLDBLATT: Good morning, Your Honor. 5 1 THE COURT: Good morning. 2 MR. GOLDBLATT: The Defendants have brought this 3 motion to exclude the proffered expert testimony of Gail 4 Heriot. She is a proposed expert submitted by the 5 Plaintiffs. She has submitted an expert report in this 6 matter. I believe it's Proposed Exhibit 136, Your Honor, 7 and that's a report about diversity. 8 It's a report about the opinions of Professor Heriot 9 that policies like the ones at issue here that involve the 10 consideration of race as a factor in admissions don't bring 11 about benefits, don't improve, in sum, the educational 12 experience. 13 As the Court is aware, the Defendants have prepared 14 a case directed exactly at that issue. We believe it is the 15 most rigorous, comprehensive, thorough-going empirical case 16 proving that students who are brought together on campuses 17 that are diverse in many ways, including with respect to 18 race, receive a better education. They learn better, the 19 schools prepare better citizens, the law schools prepare 20 better lawyers. 21 And that interest, that diversity is compelling, not 22 just as that term is used as a matter of legal doctrine, but 23 in its plain English sense that that interest is compelling. 24 It's a case, Your Honor, that the Defendants and the 25 University are quite proud of. We would be delighted to put 6 1 on that case. 2 And we had, of course, a number of witnesses 3 prepared to speak to that issue, Professor Garin, Darren 4 Bok, Thomas Shagru, Albert Camareo. The Court hasn't seen 5 any of those witnesses, and the reason for that, of course, 6 Your Honor, is that this Court's ruling on December 22nd 7 said that that question was out of this trial. 8 Our testimony, the reports that we put in, are part 9 of the summary judgment record, and everyone agrees, they 10 are going to travel with this case as it makes its way 11 through the Federal Judiciary, and that's the Court's ruling 12 and we will, of course -- 13 THE COURT: You mean everyone is not accepting my 14 ruling in this case? 15 I'm just kidding, go on. I thank God for the Court 16 of Appeals every day, so. 17 MR. GOLDBLATT: Your Honor, this is what Mr. Purdy 18 said when we were talking about this issue. He said, "And 19 of course, Your Honor, as everyone has made clear, no one is 20 contesting that there are educational benefits of diversity. 21 It is simply not an issue in this case." 22 Now, what Professor Heriot's report says is that 23 there aren't sufficient benefits to justify policies like 24 these, and in that sense, what the Plaintiff is purporting 25 to do is to put on a witness who is going to say that which 7 1 Mr. Purdy told this Court isn't true. And Your Honor, we 2 just think that the Plaintiffs should be stopped from doing 3 that. 4 But there is another reason, also, Your Honor. It's 5 that Professor Heriot isn't an expert within the meaning of 6 the Federal Rules of Evidence. She shouldn't be allowed to 7 offer opinion testimony in this case. She certainly has 8 opinions, but there are cases, as this Court is of course 9 aware, like Daubert and Cumo Tire that make clear that 10 before those opinions become evidence in a case this Court 11 plays a gatekeeping role, and before she is permitted to 12 offer opinion testimony there are certain standards that 13 need to be satisfied. 14 Professor Heriot doesn't have any degree that's 15 relevant to her testimony at all. She is a lawyer. She has 16 the same degree that many of us have. She teaches court 17 law. She says that she has, quote, "studied the policy and 18 legal aspects of the issue of racial preferences." That's 19 from her deposition, Your Honor. And she said those two 20 things, the policy aspects and the legal aspects, are 21 interconnected. But this isn't a subject that she has 22 even taught. 23 The law is clear, Your Honor, that the testimonial 24 latitude, I think that's a quote from Daubert, that expert 25 witnesses get that allow them to say more than just their 8 1 firsthand factual testimony isn't something a witness earns 2 just by having an opinion. There are standards, and with 3 respect to the specialized knowledge to which an expert 4 witness will testify there needs to be evidence that the 5 testimony is reliable, that it's more than just one person's 6 opinion. 7 This Court mentioned yesterday that there was a copy 8 of the Daubert opinion on the Bench, and it's actually an 9 encapsulation of that opinion and subsequent Supreme Court 10 case that I mentioned, it's the Cumo Tire case, and on 11 pages, I think it's 149 and 150 of that case, the Supreme 12 Court tells us exactly what the questions are that we're 13 supposed to ask here. There are essentially four questions. 14 Is there a theory or a technique that's been tested? 15 THE COURT: Does it say -- I don't have the case 16 here, I have the same thing you have, the four questions 17 right there. 18 MR. GOLDBLATT: Exactly, Your Honor. 19 Is there a theory or a technique that's been 20 tested? Have the opinions been subject to peer review 21 and publication? Are there standards we can look to to 22 determine that the opinions are reliable? Is there some 23 technique or theory that's gained general acceptance within 24 a relevant scientific community? 25 Your Honor, the answer to every single one of those 9 1 questions is no. This isn't a close case under Daubert and 2 Cumo Tire. 3 Another subject that Professor Heriot discusses in 4 her report is Proposition 209 and its effects, but it should 5 be noted, Your Honor, that Professor Heriot doesn't even 6 teach at a public university in the University of California 7 system. She teaches at a private university, the University 8 of San Diego Law School, a university that, by the terms 9 of Prop 209, isn't affected. It's not, Your Honor, the 10 University of California at San Diego that Dean Garcia was 11 discussing, it's a private university not affected by 12 Proposition 209. 13 And in this case, the Daubert qualifications of the 14 witness with respect to that question, the effects of 209, 15 aren't any better than they are with respect to the benefits 16 of diversity. All four questions, again, all four Daubert 17 questions are answered no. 18 Your Honor, this is really just one person's 19 opinion. It's exactly the kind of testimony that cases like 20 Daubert and Cumo Tire say aren't evidence, aren't evidence 21 in the Federal Court under the Federal Rules. I don't doubt 22 that these opinions are genuine, that they are sincerely 23 held, but that doesn't make them competent evidence and 24 Professor Heriot should be precluded from offering them 25 here. 10 1 THE COURT: Thank you. 2 MR. WASHINGTON: Your Honor, we certainly agree with 3 Mr. Goldblatt's remarks on this. I would just add a couple 4 more. 5 I think insofar as Professor Heriot purports to 6 testify as to the effects of 209 on the educational system 7 of the State of California, there is no evidence that she 8 has ever taught at the University of California other than 9 in speeches or forums organized by the proponents of 209. 10 There is not even any indication that she has ever had any 11 connection with the University of California. There is no 12 indication that she has published any studies other than 13 popular articles about the University of California. 14 Her report contains at most some alleged statements 15 about grades at the public university in the City of San 16 Diego, which as I understand is a city of something like 17 three million people, and there is no indication where or 18 how she got that or how she has any particular expertise 19 at all. 20 There is no technique or theory here that this 21 witness has. She has not submitted anything to any peer 22 review. There is no kind of background in education, 23 anything of that nature, and in that regard, I think this 24 is really the rankest of lay testimony by a partisan in the 25 debate brought here to say things which aren't true, but 11 1 more relevant than that, are not competent as testimony. 2 So we certainly think this witness should not be 3 allowed to testify. 4 MR. KOLBO: Your Honor, I guess my response in a 5 few words is, surely they jest. They must be making these 6 arguments, in part, tongue in cheek, and I want to address, 7 first of all, the relevance issue. 8 It's true that we made some objections early in this 9 case to any testimony about the value of diversity, and I 10 suspect if there hadn't been any testimony about the value 11 of diversity, and there has been a lot in this case from 12 both the Defendants and the Intervenors, that we wouldn't 13 be at least putting on Professor Heriot to talk about that 14 issue. 15 And we understand that the Trial Court's issue is 16 limited, but this case, as has been mentioned, is going 17 to be heard on appeal and we think it would just not be 18 appropriate for there to be lots of evidence on the 19 Defendants' side of the case, on the Intervenors' side of 20 the case, about the value of diversity, but none, even 21 from one witness, the one witness that the Plaintiffs have 22 offered in rebuttal on that particular issue. We think that 23 would be just highly inappropriate. 24 And just so I can illustrate my point, Your Honor, 25 when Mr. Goldblatt says that they had a case that they 12 1 wanted to bring with respect to diversity and they didn't 2 bring it, it's true, there are some witnesses that they had 3 on that issue that they haven't called, but they elicited 4 very directly and clearly and over our objection a number, a 5 substantial amount of testimony on the issue of the value of 6 diversity, what critical mass is, and why it's important. 7 They obtained that testimony, Your Honor, not 8 necessarily always from qualified or formally qualified 9 experts, but from people who are educators. 10 President Bollinger testified why he believes 11 diversity is important, why critical mass is important. 12 Dean Lehman testified about why diversity is important, why 13 critical mass is important and how you go about achieving 14 it. Professor Lempert testified on that issue, as well, 15 at length, and of course, they called Professor Severud 16 on that issue, really only on that issue in terms of the 17 importance of diversity and how you get it and so forth. 18 So it just seems absurd to me, Your Honor, to 19 suggest that it's not an issue in the case, and therefore, 20 the Plaintiffs shouldn't be able to call a witness even in 21 rebuttal on those issues. It just doesn't make -- it 22 doesn't pass the smile test, Your Honor, is I guess what 23 I would say about that. 24 With respect to the substance of Professor Heriot's 25 testimony, she is not going to testify just about diversity 13 1 and whether it's important. Her report, Your Honor, is 2 before the Court and I think you have a sense about what 3 she is going to testify about. Her view is, there is 4 nothing wrong with diversity, certainly diversity, 5 intellectual diversity is a good thing. You'd rather have 6 it. 7 But one of the things she is going to testify to is 8 how you go about getting it, and what she says, you don't 9 have to work very hard to get diversity in the classroom, 10 it pretty much comes naturally. If you're going to have a 11 classroom of 30 students, people are individuals, they bring 12 by their very nature different experiences, outlooks and 13 backgrounds, so you don't have to do much to bring it and 14 you certainly don't have to use racial preferences to 15 achieve diversity in the schools. 16 We think that's relevant to the issues in the case 17 and it's certainly, it's certainly relevant rebuttal 18 testimony to the voluminous testimony that the Defendants 19 have brought. And I haven't even mentioned the Intervenors, 20 Your Honor. They have marched student after student up here 21 to talk about their classroom experiences and so forth. We 22 had Professor Garcia yesterday talking about diversity on 23 campuses that he's not even involved with at the University 24 of California system. 25 So I think it's just frankly a little absurd to 14 1 say that the issue isn't in the case, and therefore, the 2 Plaintiffs shouldn't be able to bring a witness to respond 3 to their witnesses. What they are really trying to do, 4 Your Honor, is to suggest that there should only be one side 5 heard on this issue. Their side has been heard and our side 6 should be heard, and from the one witness that we intend to 7 call on this issue. 8 If I may then, Your Honor, address the issue of 9 qualifications, again, I have to say that this brings a 10 little bit of a smile to my face when I hear that Professor 11 Heriot, first of all, is not qualified to testify on the 12 subject of legal education, the very same subject that, 13 again, President Bollinger testified about, Professor and 14 Dean Lehman testified about, Professor Lempert testified 15 about, Professor Severud testified about. 16 Our witness, Your Honor, Gail Heriot, is a professor 17 of law and a full professor of law. She has been teaching 18 since 1989. She was Dean for a year at George Mason 19 University. When we're talking about diversity, as I 20 understand, what I have heard from the witnesses on the 21 other side, we're talking about what goes on in the 22 classroom, what's important, what does diversity bring to 23 the classroom, what does it bring out of the students, how 24 does it affect the teacher's performance and so forth. 25 I don't know how anybody could bring more expertise 15 1 to that subject than someone who is, in fact, a legal 2 educator and I think that's all that's required to pass the 3 expert testimony threshold in this case. That's all it took 4 for their experts on that subject. That's all it took for 5 their lay witnesses to testify on the subject of the value 6 of diversity. So there is just -- it's just not a serious 7 argument to suggest that our witness, a full professor of 8 law and a teacher of education and one who has thought about 9 and researched some of these issues, doesn't have the basic 10 threshold qualifications to present testimony. 11 Now, I understand they think their experts are 12 better qualified. They think more of Professor Severud than 13 they think of Professor Heriot in terms of credentials and 14 they are certainly entitled to think that and they can 15 certainly argue in terms of the weight of the testimony on 16 that. We think Professor Heriot is very highly qualified 17 and her opinions will be or should be well received by the 18 Court, so it's just not a matter of throwing her testimony 19 out. 20 And finally, Your Honor, with respect to the issue 21 of California, again, I have to smile. They are suggesting 22 that a professor in California is not appropriate to testify 23 about the effects of Proposition 209 in California. I 24 didn't hear exactly what Professor or President Bollinger's 25 or Dean Lehman's or Professor Lempert's or Dean Severud's 16 1 qualifications were to testify about California, and they 2 did testify about what they thought would be the dire 3 consequences in Michigan, based on California, if race 4 couldn't be used as a factor, so it's just not a serious 5 argument to suggest that their Michigan professors are 6 better able to tell this Court about California than a 7 California professor. 8 And finally, Your Honor, with respect to that issue, 9 they seem to misunderstand the point that we thought has 10 already been brought out in this case, which is the 11 consequences of Proposition 209 aren't just limited to 12 the Berkeley campus and they aren't just limited to the 13 University of California system, and they are not even just 14 limited to the public schools of California. 15 One of the things that Professor Heriot is going to 16 testify concerning is that Proposition 209 has had positive 17 consequences, and those positive consequences extend to 18 schools that aren't even in the California system, schools 19 like hers, where she has seen improvements in the quality 20 of classroom discussion and so forth as a result of 21 Proposition 209 and what she believes should be attributed 22 to it. So there is a great deal of relevance there and she 23 certainly has foundation to testify in these matters, Your 24 Honor. 25 THE COURT: In this matter, I know, Mr. Goldblatt, 17 1 you would like to say some more, but let me just -- no use 2 taking a lot of time on it. 3 As to qualifications, I will have to listen to what 4 the qualifications are. I hear both sides arguing what the 5 qualifications may or may not be, but I'll tell you in no 6 uncertain terms that I'm going to use the same standard 7 that I have used throughout this case, and there have been 8 witnesses that have testified that have done no studies, 9 there have been witnesses that have testified that they have 10 had no formal teaching and so forth, but that the Court has 11 ruled that they could testify, because the reason being is, 12 that they do have special knowledge because of their 13 position and so forth. 14 For instance, I think Mr. White was very helpful 15 after we heard his testimony and I think he was very 16 credible in terms of what he had to say and the statistics 17 and so forth, though he had no formal training, hadn't done 18 any personal studies or anything of that nature, and I am 19 going to use the same kind of test. 20 Even Dean Severud, obviously, had no formal 21 training, but he had been around a long time and had taught 22 courses, he had done things that perhaps, judicial Daubert, 23 you know, you would say, well, that wouldn't apply. 24 I only give you these examples because I know that 25 this person is coming from a long way and so forth, but I'm 18 1 saying that I can't rule on her qualifications until I hear 2 them, until I have had an opportunity to have the Plaintiff 3 present them and the Defense and the Intervenors voir dire 4 the witness and then I can make that determination. It's 5 hard to do in the abstract, though. 6 As I indicated, I have listened to both sides and 7 I want both sides to know that I'm going to use the same 8 standards that I have used for everybody at this point. 9 My other issue is that certain matters have arisen 10 that were important and I wanted to hear them, and I have 11 heard them, that perhaps the Plaintiff was not anticipating 12 or so forth, and this is rebuttal, and I'm going to treat 13 it as rebuttal and it's going to be limited to -- her 14 testimony, if permitted, is going to be limited to rebut 15 those things that have been put into the record by the 16 Defense and the Intervenors. 17 So with that said, I suspect the Plaintiffs will 18 call and we will have some voir dire as to qualifications 19 and I will make that ruling, and my ruling, should I allow 20 her to testify, will be that it's limited to rebut those 21 issues that have been raised by the Defense or by the 22 Intervenors. 23 Okay. Mr. Goldblatt? 24 MR. GOLDBLATT: No, thank you, Your Honor. 25 MR. KOLBO: Your Honor, we appreciate the Court and 19 1 the Parties' agreement to take our first rebuttal witness 2 out of order. We will recall Professor Kinley Larntz to 3 the stand, Your Honor. 4 THE COURT: Welcome. 5 K I N L E Y L A R N T Z, P h D, 6 having been previously called as a witness herein, and after 7 having been first duly sworn to tell the truth, the whole 8 truth, and nothing but the truth, was examined and testified 9 as follows: 10 DIRECT EXAMINATION 11 BY MR. KOLBO: 12 Q Dr. Larntz, thank you for coming again. 13 In the interim, I think it's been a couple of weeks 14 or so since your testimony, you understand that Professor 15 Steven Raudenbush offered testimony with resect to the same 16 issues, in some respects, in response to your testimony? 17 A Yes, I understand that. 18 Q Now, you weren't here in the courtroom for that 19 testimony, were you? 20 A I was not. 21 Q But have you been able to review a transcript of 22 Dr. Raudenbush's testimony? 23 A I have. 24 Q And do you understand that Dr. Raudenbush took issue 25 with some of -- some aspects of your testimony and report? 20 1 A Yes. 2 Q Have you also reviewed your own transcript, trial 3 transcript testimony in this case? 4 A Yes. 5 Q Were you able to ascertain from reading, perhaps 6 rereading or reading your testimony, as well as 7 Dr. Raudenbush's testimony, were you able to ascertain 8 whether you had already addressed or covered some of the 9 issues that Dr. Raudenbush took issue with in his testimony? 10 A Well, certainly. I mean, we exchanged our reports 11 before and I, in my direct testimony, offered information 12 on a number of criticisms that he talked about in his 13 testimony, yes. 14 Q And did you, in reviewing your testimony -- you 15 understood that you had already responded to some of those 16 or anticipated some of those in your direct testimony? 17 A A good number, I think, yes. 18 Q Did you also come to the conclusion that there were 19 some aspects of Dr. Raudenbush's testimony that you hadn't 20 really addressed or hadn't been explained in your direct or 21 cross examination that occurred? 22 A There are a few points, yes. 23 Q Okay. I want to just limit, obviously, our questions 24 and answers this morning to that, so please keep your 25 answers limited to that respect. 21 1 And I'm going to ask you about some criticisms that 2 Dr. Raudenbush raised and to the extent that you don't 3 believe they were addressed either in your prior direct or 4 in your cross examination, I would like you to respond to 5 some of these criticisms, if you have some opinions. 6 First of all, do you recall that Dr. Raudenbush had 7 a criticism with respect to the fact that there are -- that 8 in determining or ascertaining odds ratios, there are high 9 odds ratios that don't reflect large differences in 10 probability; that is, at higher ranges of probabilities, so 11 they don't reflect large odds ratios? 12 A There were cases presented by Dr. Raudenbush, and also 13 I think in my cross examination relating to odds ratios that 14 may be relatively high where the difference in probabilities 15 is not -- well, depending on how you think, not great. 16 Q Do you understand what the criticism is of your report 17 and testimony in that area? 18 A Well, I think my understanding is that the -- while 19 there may be relatively high odds ratios, there isn't much 20 of a difference in probability, and I think it's important, 21 I think it's very important that the Court understand and 22 that everyone understand the effect of odds ratios on 23 probabilities. 24 And I tried originally to provide some illustrations 25 and they provided some illustrations and I just want to make 22 1 sure that we're very clear on the relationship from a 2 baseline probability how a particular odds ratio would 3 translate into another probability. 4 So for instance, a baseline probability, I think in 5 my direct examination we talked about a baseline probability 6 of 10 percent going to 90 percent from an odds ratio of 81 7 and so I wanted to -- I wanted to make sure we were clear 8 about how that works over the whole range of probabilities. 9 Q And did you read in the reports of Dr. Raudenbush's 10 testimony where he compared relative probabilities of, say, 11 90 versus 99 percent and the effect of going from 90 to 99 12 to 999 and so forth? 13 A Yes, yes, I did read that. 14 Q And what is demonstrated by that? 15 A It's again illustrating for a certain baseline 16 probability how relative odds of a particular value would 17 translate into another probability for another group. 18 Q And I think you have testified, you indicated this 19 morning that you have prepared something to illustrate the 20 range of probabilities? 21 A What I did, what I did is it's in the booklet that I 22 think you have, and the last page of the booklet, what I did 23 was prepare just a table, and we have it as a slide. 24 I prepared a table for a range of probabilities over 25 the whole range, rather than just choose a certain point, 23 1 where I used 10 percent as a base, and they, Dr. Raudenbush, 2 used 90 percent for the whole range of probabilities, going 3 from a base of one percent, a baseline probability of one 4 percent up here in the corner, all the way down to, I think 5 it's 99, down in the bottom. 6 And so what I did is I did it for various relative 7 odds and I chose values that we have seen results of, 5, 10, 8 20, 50, 100, 200. I have chosen -- what I have done is I 9 have said, how does this baseline probabilities translate 10 into the probability of another group that has a relative 11 odds of a given amount. 12 So for instance, for instance, Dr. Raudenbush had 13 90 percent, and I think was looking at 90 versus 99, and 14 that's a relative odds of 11. I don't have 11 on this 15 table, but I have 90. I have 10, and 90 goes to what, 98.9. 16 And then talking about 111, I think, was one of the ones he 17 used, and 90 goes to -- he had 111, I have 100, and 90 goes 18 to 99.89. 19 So these are -- I'm referring to them as fractions, 20 but we often talk in terms of percentages. And other 21 values, well, for instance, 10 percent, I had reported 22 10 percent with an odds ratio of 81 going to 90. Well, we 23 can see 10 percent here going, well, at 100, goes to 91.7. 24 And all I have done on this table is just to make sure we're 25 clear what the effect of a relative odds is on a baseline 24 1 probability. And that's the only purpose of the table, is 2 for clarification. 3 Q Let me ask, Doctor, Dr. Larntz, do you disagree with 4 Dr. Raudenbush's premise that when you start with a high 5 baseline probability like 90 and then have a comparative 6 probability of 99, that odds ratios go up rapidly as 7 those probabilities diverge? 8 A Well, sure, the odds do, the odds ratios do go up 9 mathematically. They are what they are, and that's what 10 this table is trying to just illustrate, what the 11 mathematics is, and I think that's clear. 12 In point of fact, in that particular case, if you 13 look at the percentage denied, obviously they go up or 14 they go from 10 percent to one percent, so that's a big 15 difference in that probability scale. 16 Q So as I understand it, you agree with Dr. Raudenbush 17 on that fundamental principal? 18 A On the calculation, absolutely, there is no 19 disagreement on the calculation. That's why I provided 20 the table, so you can see the whole range of possible 21 calculations. 22 Q And does that mathematical fact, does that change 23 anything with respect to your opinions on the extent to 24 which race is taken into account in the admissions process? 25 A No, no. 25 1 Q Dr. Larntz, were you also -- did you also understand 2 that Professor Raudenbush in his testimony took issue with 3 your -- or criticized you for what he called discarding some 4 of the data in your analysis or selectively attending to the 5 data? 6 A Yes, I did read that, yes. 7 Q Do you have an opinion with respect to the validity of 8 that criticism? 9 A You mean the fact that different parts of the data 10 provide different amounts of information? I certainly have 11 an opinion that that's true, and I'm not quite sure -- in 12 the sense that certain cells in the grid provide more 13 information than other cells, that's true. 14 Q What would be an example of that? 15 A Well, I mean, obviously cells that have no one 16 admitted would provide no information about admission rates, 17 in comparison of admission rates. Cells that have everyone 18 admitted provide no comparative information. 19 And that's what we're talking about here, because 20 we're trying to compare the admissions of minority students 21 to majority students. So the different cell grids in fact 22 do provide different amounts of information. 23 Q Okay. And as I understand it, Dr. Raudenbush has 24 suggested that you did not consider certain cells in the 25 data. 26 1 A Well, in the calculation, every cell entered the 2 calculation and the computer made the judgment of how much 3 information would be -- you know, how much information was 4 there. There are some cells that the computer found that 5 there was no comparative information and other cells where 6 it found comparative information and used all the cells that 7 had comparative information in deriving the summary relative 8 odds ratios that we talked about. 9 Q What would be the kinds of cells that would provide no 10 comparative information, what categories would those fall 11 into, into terms of what you saw? 12 A Well, cells in a grid that would have no one admitted, 13 okay, there is no comparative information, everyone 14 admitted. And there are very few cells like that, but there 15 are a few, okay? Cells, if there are no people in the 16 cells, there is no comparative information. 17 And in fact, the one additional category is if 18 everyone in a cell was a member of one of the ethnic groups, 19 just one, so if everyone in the cell, for instance, if 20 everyone in the cell were caucasian and there were no 21 minority students in that cell, or no member of any other 22 ethnic group, then that wouldn't provide any comparative 23 information, because everyone would be the same. 24 Q What would be an example of a cell that provided some, 25 but not very much information? 27 1 A Oh, there are cells, for instance, a cell where 2 virtually everyone is admitted, but not everyone, then that 3 cell would provide -- may provide comparative information, 4 but the amount of information in how much it contributes to 5 the estimation of the overall relative odds would be small. 6 Q Have you prepared some charts to illustrate what cells 7 provided comparative information in your analysis? 8 A What I have done is for each of the years, and this is 9 also in the book, for each of the years what I have done 10 is -- and we have the first one for 1995. What I have done 11 for each of the years is looked at the original grid, this 12 is for all applicants, and I have highlighted, I have 13 highlighted in yellow the cells that provide comparative 14 information. 15 So all the cells that provide comparative 16 information are highlighted in yellow and the cells that 17 are not, the blank cells, except for the permanent cells, 18 the blank cells in the balance of the table for which there 19 is no comparative information, that is, in the sense that 20 the computer would check and say, well, there isn't anything 21 I can add based on these cells, those are left blank. 22 Q Would there be a way of including the cells that have 23 no applicants, first of all? 24 A Cells that have no applicants? 25 Q Yes. 28 1 A Whatsoever? 2 Q I'm sorry, that have no -- for across racial lines 3 there are no admits, or let's take, for example, no 4 admissions. 5 A Well, there -- if there is no admits, then there is no 6 comparative information, so they -- the computer makes a 7 judgment that there is no comparative information there and 8 so it attributes -- it doesn't actually contribute to the 9 estimation. 10 Q Did Dr. Raudenbush in his analysis use those cells in 11 some fashion, cells in which there are applicants, but no 12 admissions from any racial group? 13 A Oh, I believe that in some of his models he used those 14 cells, yes. 15 Q And does that have any effect, as far as you're 16 concerned, on the validity of the results? 17 A Well, with respect to providing comparative 18 information you would have to have a model that says how 19 they enter, and so based on how his model works, that model, 20 whether that model is true or not depends on how that 21 information would enter. I'll just say it that way, 22 that's true. 23 Q Would it provide a more accurate her less accurate 24 picture of the -- as far as you're concerned -- the extent 25 to which race is taken into account, like including these 29 1 cells in which there are no admissions from any racial 2 group, or would there be a distortion there or what 3 would be -- 4 A Well, I actually don't know what the effect is, I 5 mean, as far as that goes. With respect to the statistical 6 principle of looking at cells with comparative information, 7 they don't provide comparative information, so that's what 8 I know. 9 Q And so you have got in front of you what we have up 10 here, 1995? 11 THE COURT: They don't provide any comparative 12 information, in your opinion. If you included them, would 13 there be a downside to including them, even though they have 14 no information or is there just no reason for it or -- 15 THE WITNESS: Well, I mean, what you have to do 16 is you have to model the whole grid, okay, and that to me, 17 that's a hard process, okay. I looked at it and took the 18 comparative information and compared cell by cell. To me 19 it's harder to model the whole grid. It's just a more 20 difficult problem and your results would depend upon how 21 you did that modeling. 22 THE COURT: So there would be another level of 23 discretion, because that new level of discretion is now an 24 extended modeling of the whole thing? 25 THE WITNESS: That's the way I looked at it. 30 1 THE COURT: I mean, is that pretty much -- I 2 remember you talked about the less that you have to do in 3 terms of discretion, the better the results sometimes are. 4 THE WITNESS: That's right. 5 THE COURT: That's what you told us. 6 THE WITNESS: Right. The kind of modeling would be, 7 there is some kind of -- you know, I'll try to be as clear 8 as I can. You basically have a surface that you're modeling 9 here. You're trying to model a response surface over this 10 whole grid and you have to make assumptions about how to do 11 that, and what I did is, I didn't do that, and then what 12 you -- I have to use two hands, sorry. 13 THE COURT: That's okay. 14 THE WITNESS: You model, say, the majority students 15 one way and the minority students in another and sort of the 16 difference here becomes the odds ratio, right? So depending 17 on how you do that modeling, depending on how you do that 18 modeling, you could get different answers. 19 THE COURT: And the more modeling you do, the more 20 assumptions you make? 21 THE WITNESS: Yeah, well, sure, yeah. 22 THE COURT: Your position is that you did -- you 23 tried to do as little modeling and as little assumptions as 24 you can to actually get the figures? 25 THE WITNESS: Right. And exactly in this way, what 31 1 I did is I said, let's not try to model the whole surface, 2 let's just look at it one point at a time, and look at the 3 difference one point at a time. And that's basically what 4 I did, was look at the difference one point at a time. 5 THE COURT: Creating a situation where you have to 6 make fewer assumptions? 7 THE WITNESS: I think there are fewer assumptions 8 in that, surely there are fewer assumptions in that, yes. 9 BY MR. KOLBO: 10 Q And you have done this for each one of the years in 11 question? 12 A That's right. 13 Q Can we just take a look at 1996? 14 A I think so. 15 Q Same pattern? 16 A Oh, same pattern. The cells that were no admitted 17 were the predominant cells that were left out and they are 18 the ones that have very low test scores and very low GPA's 19 and those students just, you know, they didn't admit them. 20 So that's true universally. There are a few other cells 21 that don't provide very good information for the other 22 reasons I said, but predominantly, they are the lower grade 23 point averages and the lower test scores. 24 Q And 1997, can we see that same pattern here? 25 A Same pattern. 32 1 THE COURT: And '98, perhaps -- 2 THE WITNESS: Same pattern. 3 BY MR. KOLBO: 4 Q And then '99. I wanted to ask you about a cell or a 5 couple of cells for 1999. I don't think these are -- these 6 kinds of cells show up on the earlier versions, but there is 7 a cell, for example, under grade point 3.0 to 3.24 and 170 8 and above in which there are nine applicants and three 9 admissions; correct? 10 A That's correct. That's the cell over on the -- let me 11 see, it's the cell right over here in this corner, 170 and 12 above, and this nine and three here that's not shaded. 13 Q And it's not shaded, meaning that it didn't provide 14 comparative information? 15 A That's right. 16 Q And why in that case was there no comparative 17 information? 18 A Well, I don't recall exactly the group, but all nine 19 of those applicants were from one ethnic group, so they were 20 from one ethnic group. 21 Q That shows no comparative information across racial 22 lines? 23 A That's right. 24 Q Did you -- there has been some testimony, I think, 25 from Dr. Raudenbush about the number of cells or percentage 33 1 of cells or so forth that were used in your analysis or that 2 were -- that, as you have testified, provide comparative 3 information. Can you -- do you have some idea in terms 4 of the number of applicants that contributed comparative 5 information to your analysis across these different years? 6 A Yes, I did calculate the percentage of applicants that 7 are in the shaded areas and I don't remember exactly from 8 each year, but it's between 84 and 88 percent across the 9 six years. 10 Q That all provided comparative information? 11 A That's right. 12 Q Dr. Larntz, then if I can ask you, do you recall 13 that in reading Dr. Raudenbush's testimony that he had a 14 criticism with respect to your reporting of uniform odds for 15 composite odds ratio assumptions? 16 A Yes. 17 Q Do you understand what his criticism is in that 18 regard? 19 A Well, I understand the technical, statistical aspect 20 of the criticism, sure. 21 Q Do you agree or disagree with it? 22 A I disagree on its importance, and I am not sure if 23 that assumption is technically satisfied or not, but in 24 fact, the importance of the assumption is that my composite 25 odds ratios, if in fact that assumption is not perfectly 34 1 satisfied, and no assumptions are ever perfectly satisfied, 2 if that assumption is not perfectly satisfied and there are 3 some cells that have higher odds ratios than others, then 4 the number I gave you would be a composite that would be 5 lower than they would be for the ones that are high and 6 a little bit higher than the ones that are lower. 7 Q Do you have an opinion as to whether Dr. Raudenbush's 8 criticism -- does it change any of your opinions with 9 respect to the extent to which you believe race is 10 considered in the admissions process as reflected by 11 your analysis? 12 A No, it doesn't change my opinion with respect to 13 that at all, no. 14 Q Can you explain why not? 15 A Well, I mean, well, first of all, I think the odds 16 ratios we see are a direct reflection of the cell. We can 17 compare the grid cells and see high odds ratios in the grid 18 cells and the summary numbers that I provided, I think, are 19 a reflection of those, of those values. 20 Q Do you recall Dr. Raudenbush testifying that there are 21 a lot of large odds ratios in the middle ranges of the 22 grids? 23 A Oh, I believe that he said that there are very high -- 24 or there are high odds ratios in the middle, yes. 25 Q And do you agree or disagree with that? 35 1 A Yes, if there weren't higher odds ratios in the middle 2 we wouldn't have composite odds ratios. If there weren't 3 high odds ratios someplace, and they're mostly in the middle 4 where there is a difference in admission rates, then we 5 wouldn't -- the composite odds ratios wouldn't be as large 6 as they were. 7 Q Do you understand Dr. Raudenbush's testimony to be 8 that the odds ratios are smaller at the upper end of the 9 grid? 10 A I think I -- I think the implication of what he said 11 was they were smaller in the upper end, that's right. 12 Q Do you agree or disagree with that? 13 A Well, I think in the upper end it's very hard to tell, 14 because there is really -- when almost everyone is admitted, 15 there is -- you can estimate odds ratios, but there is -- 16 the amount of comparative information up there is relatively 17 so small, so they don't contribute a great deal to the 18 composite odds ratio, because if almost everyone is admitted 19 the comparative information there isn't great, so I think 20 it's very hard to determine the relative odds up in the 21 corners unless you create some kind of model. 22 Q Where does most of the comparative information come 23 from him? 24 A Well, most of the comparative information is down in 25 this area here where there are clear differences and where 36 1 there is some discretion being made with respect to the 2 admissions process where not everyone is admitted. 3 Q The middle of the, primarily, shaded area? 4 A Well, actually, it's the lower part of the shaded 5 area, if you want to call it, the middle of the grid. 6 Q And then do you recall reading a criticism of 7 Dr. Raudenbush with respect to what he termed, I think, 8 the stability of the estimates over the years? 9 A Yes, I do remember reading that. 10 Q And do you have an opinion as to whether you disagree 11 or agree with his criticisms there? 12 A Well, I certainly think that they are relatively 13 stable over the years and I think that we have to -- I think 14 we have to do -- probably do a little bit of work, a little 15 bit of statistical work in the courtroom, to show that. 16 Q So you disagree with Dr. Raudenbush's conclusions? 17 A I do disagree that there is a -- that there is any 18 kind of major problem with stability over the years, yes, 19 I disagree completely. 20 Q And could you explain the bases, and using the board 21 if the Court will permit? 22 THE COURT: Sure. 23 A Well, a great deal was made of a comparison -- a great 24 deal was made of a comparison between African American 25 relative odds in 1997 and African American relative odds in 37 1 the year 2000, and I think that a great deal was made of 2 that comparison, and I think we need to talk about African 3 American relative odds. I have to say, if you select two 4 years out then you might get a different comparison, so I 5 want to look at all of them across all six years, okay, and 6 so I want -- can I go to the board? 7 Q Sure. Have you prepared some notes? 8 A I have a little note that summarizes things that I had 9 from before. 10 THE COURT: Can I make a suggestion? 11 THE WITNESS: Yes, sure. 12 THE COURT: I didn't know where you were, I thought 13 you were sitting. 14 If we put the board over here just for purposes 15 of this, then everybody can still have a seat and sit and 16 see it. 17 MR. KOLBO: In fact, I think we're done with the 18 projector. 19 MS. MASSIE: Thanks, Judge. 20 (Discussion off the record at 9:15 a.m.) 21 THE WITNESS: What I want to do is, I want to look 22 at all the years, so 1995, 1996, 1997, 1998, 1999 and 2000. 23 And I want to talk about relative odds and I want to talk 24 about comparing relative odds, and I said I have to -- I 25 have to do a little bit of statistics and a little bit of 38 1 math here, so that's just because it's been raised and I 2 want to make sure that we all understand. 3 So the relative odds, and I'll just call it the 4 estimated relative odds that we calculated, the estimated 5 relative odds from the report for 1995 was 257.93. That's 6 1995. This is, we're looking at the relative odds for 7 preference for African Americans, and that's always to a 8 baseline of caucasian. 9 For 1996, it was 313.59. For 1997 it was 53.49. 10 And for 1998 it was 132.16. For 1999, it was 206.45. And 11 for 19 -- whatever it is -- 2000, it was 443.26. 12 So what was talked about, what was talked about was 13 the comparison of this value, the 53, to the 443. Now, I 14 presume, I presume those were chosen because, well, if you 15 look down the list, what's the smallest one? It's 53. And 16 what's the largest one? It's 443. Would there be -- would 17 there always be a largest and a smallest? Well, that's 18 mathematics. There will be a smallest and a largest. And 19 if you're doing comparisons, well, depending, you would 20 typically look at the -- might look at the extreme, that's 21 fine, okay. 22 Now, I also had a number of standard deviations, a 23 Z score, and I need to look at that, and I'll talk about 24 that in a second, but I just want to record the one. These 25 are straight out of the reports and we saw these before, and 39 1 the standard deviation, and I'll need these, so 14.40, 2 13.18, 13.96, 13.46, 12.64, and 12.53 -- or 51, excuse me, 3 51. So this is how this is reported. These were reported 4 in the first report that I did, these first four, and these 5 were in two supplemental reports. And these are the 6 standard deviations which measure the level of statistical 7 significance of how far these are away from one, okay. So 8 this is all summary information. 9 Now, how do we actually do the calculations? Where 10 do they come from? They come from logistic regression. If 11 you recall, that's the technique we used. And logistic 12 regression works in terms, like every regression, works 13 in terms of regression coefficients. 14 And what are the regression coefficients that are 15 behind these numbers, what are the regression coefficients? 16 Well, in fact, these are not the regression coefficients 17 themselves, there is -- in the logistic regression there are 18 coefficients that we can use to calculate these. In fact, 19 the coefficients, the coefficients themselves turn out to 20 be, to be careful, it's the natural logarithm of these 21 values, the natural logarithm, not the log to the base ten, 22 but the natural logarithm. On your calculators, that's the 23 LN function, right? 24 So in fact, the regression coefficients, the 25 coefficients, the coefficients themselves, the coefficients 40 1 themselves are what you use for doing all the inferences. 2 And so what I'm going to do is write down what the 3 coefficients are, and that's what you use for comparison, 4 are the coefficients, because that's the basic underlying 5 calculation mechanism, are these coefficients. 6 And so the coefficients are, and I'll put them in 7 order and say them, the coefficients are, and I'm going 8 to round them to two places, the coefficients are -- for 9 corresponding to 57.93, the natural logarithm is 5.55. For 10 313.59, the natural logarithm is 5.75. For 53.49, the 11 natural logarithm is 3.98. And for 132.16, the natural 12 logarithm is 4.88. And for 206.45 the natural logarithm 13 is 5.33. And for 443.26, the natural logarithm is 6.09. 14 So these are the actual coefficients in the computer 15 output that you would find in the logistic regression 16 computer output. 17 Now, a relative odds of one means no preference. If 18 you take the logarithm of one, you get zero, that's just the 19 way it works, and so these need to be compared to zero. And 20 in fact, the way these standard deviations are calculated 21 is they take the coefficient, this is out of this, and 22 divide it by something called the standard error of the 23 coefficient, and you get the number of standard deviations. 24 Now, I think it's informative to plot these values, 25 because this is the scale for comparison, the comparisons in 41 1 the regression coefficients, so I need to plot these values, 2 and I'll do the best I can, okay. Zero is here, and then we 3 have got one, two, three, four, five, six, and you can see 4 if I did a reasonable job of making those equally spaced. I 5 tried, okay. 6 And then what I would do is plot values, plot the 7 values here, so I better give myself a code, one, two, 8 three, four, five. So 5.55 is here. And 5.75 is here. 9 And 3.98 is here. And 4.88 is here. And 5.33 is here. 10 And 6.09 is here, okay. 11 So this, these are the six years. This is '97 and 12 this is 2000, these are the extremes, but this is where the 13 coefficients fall in terms of the regression coefficients 14 and this is the scale for comparison. 15 Now, statistically what you're doing is comparing 16 each of these values to zero. You're comparing each of 17 these to zero, because that's in the regression coefficient 18 scale and clearly these are clustered relatively far away 19 from zero. 20 In fact, in fact, the standard deviation tells us 21 that for -- well, this standard deviation number tells us 22 how many standard deviations we are away from zero and so, 23 for instance, the year 2000 one is 12.5 standards deviations 24 away relatively. Each one of these has a standard error 25 comparative measurement attached to it, and so we're fairly 42 1 always far away. So when I say that these results are 2 statistically relatively stable, I'm making the statement 3 of how they are in the regression coefficient scale. So 4 that's how to illustrate and say that the variation from 5 year to year is within reasonable statistical standards. 6 Now, there are a couple of more things we could do 7 with this, and I'm not sure how far we should go, but one 8 more thing we could do is we could, for instance, provide 9 confidence limits for these relative odds if we wanted to. 10 I was asked about, could you provide confidence limits, and 11 I think I was criticized for not providing confidence 12 limits, I'm not quite sure. It's not hard to do. 13 So I could provide confidence limits, and if I do 14 one of them, if I do one of them and write down the results 15 of the -- for the others, we can see how that goes and I'm 16 going to give -- so what I need to do to provide confidence 17 limits for the relative odds is I need to determine the 18 standard error of the coefficient, because the way I would 19 provide confidence intervals is, I have to provide a 20 confidence interval for the coefficient and then I have to 21 retranslate it into relative odds. 22 Remember, the coefficient is the logarithm, so 23 what I have to do is provide confidence intervals for the 24 coefficient and then retranslate it, okay. 25 So to get standard errors, I actually -- the 43 1 standard deviation is the coefficient divided by the 2 standard error. It turns out if we take the coefficient 3 and divide it by the standard deviation, we have to get the 4 standard error. So the way the standard can be calculated 5 is we will take this coefficient value and divide by the 6 standard deviation, so 5.55 divided by 14.40, and that will 7 actually -- this is, we're doing this backwards in some 8 sense, because this is actually in the computer output, but 9 just to show where it comes from, this is the standard error 10 of the coefficient, 0.39. That's this value divided by that 11 (indicating). And we can do the same for all of these. So 12 we get 0.44 for 1996, 0.29 for 1997, 0.36 for 1998, 13 0.42 for 1999, and 0.49 for the year 2000. 14 And from this, from those standard errors, we 15 can calculate confidence intervals. The usual way of 16 calculating confidence intervals, I'm going to do it 17 approximately, because it won't make any difference, is to 18 get 95 percent confidence bounds, we take the confidence 19 coefficient, the regression coefficient, and take a plus or 20 minus two standard errors and that gives us 95 percent 21 confidence amounts. 22 So I could do that for this, and so, for instance, 23 in 95 percent confidence, 95 percent confidence interval, 24 then, for the regression coefficient, take 5.55 and subtract 25 two times this, and the lower bound, then, is 4.77 and the 44 1 upper bound is 6.33. So that's a confidence interval for 2 the true value of the coefficient. 3 And I can do that for each of these, and I'm not 4 going to write them all down right now unless you would like 5 me to, but what I can do now is that I can translate these 6 into confidence intervals for the relative odds. And 7 what I do is I use the, let me see, inverse logarithm, 8 exponentiation, so I undo, to get into terms of relative 9 odds, and in terms of relative odds, the confidence interval 10 runs from 118, that's the lower bound, to 561. 11 I think I had indicated earlier that these numbers, 12 I don't believe these numbers exactly, and maybe, you know, 13 in some sense I should never put two decimal points down on 14 something I don't believe too accurately, but we can see 15 that this is a confidence interval for the relative odds. 16 These are all high numbers. You know, I think they are all 17 high values, but these are confidence intervals for the 18 relative odds. 19 So if I might, I think I'll just -- could I write 20 down -- I'll write down the confidence interval for relative 21 odds for each of these, without going through all the other 22 calculations. 23 So the confidence interval for 1996 then becomes 24 130 to 757. Notice they are wide, and they are actually, 25 because of the way it works, they have to be wider on the 45 1 high end, because the relative odds, as they go up, there 2 isn't that much -- as much difference as they go higher and 3 higher. 4 For the 53.49, lower bound is 30, again, and the 5 upper bound is 96, so there is a spread, but these lower 6 bounds are all what I consider big relative odds. 7 THE COURT: Even 30? 8 THE WITNESS: Oh, sure. Oh, sure. Ten is big, 9 okay. I think I said that before. 10 THE COURT: You did. 11 THE WITNESS: Okay. 12 And 64 to 270, 89 to 478, and the last one is 13 166 to 1,176, okay. 14 So the lower bounds are all, you know, in what I 15 consider large relative odds. The upper bounds, of course, 16 are -- well, you know, they just become harder to estimate 17 out at higher relative odds. 18 So they are all -- so this gives you an idea of some 19 variation. So in my estimation, these are all indicating 20 the same thing, that in fact, there is a substantial, a very 21 large allowance for African American applicants for each of 22 the years, and they are consistent. 23 They are consistent, not technically statistically 24 consistent, but substantively consistent, but substantively 25 consistent from year to year. There may be a technical 46 1 difference between them. 2 So that's something I wanted to make sure we 3 demonstrate, the stability in the appropriate scale, which 4 is the regression coefficient scale, and the corresponding 5 confidence intervals that we could generate for relative 6 odds. 7 Q So in short, in a short sentence here, what you have 8 just explained is the basis for your opinion that you 9 disagree with Dr. Raudenbush's conclusion that the relative 10 odds ratios are unstable across years? 11 A Well, that's part of it, yes. That's certainly the 12 basis. This is my thinking of how I would look at this 13 and certainly I believe they are certainly substantively 14 stable, if not technically statistically stable across 15 years. 16 Q Now, I think Dr. Raudenbush criticized you for not 17 reporting confidence intervals as you have just illustrated 18 here for these years for African Americans. Could 19 Dr. Raudenbush have calculated confidence intervals for your 20 relative odds based on the data that he had available to 21 him? 22 A Oh, sure. This is not -- this is straightforward 23 calculations and the computer output was all provided. 24 Everything was there to do this, yes. 25 Q Now, you -- 47 1 A But you can do it from the reports, actually, as we 2 just did from that information. 3 Q And you have reported that all of these, as far as 4 you're concerned for these years for African Americans, all 5 of these show very large preferences? 6 A Oh, sure, yes. 7 Q Are you able to tell whether that's true or not for 8 the other races that you did odds ratios analysis on for all 9 these years? 10 A Well, if you computed confidence intervals for 11 relative odds, they would all be, you know, relatively -- or 12 distributed around the estimated values, but they would all 13 show large preferences, yes. 14 Q You know, Dr. Raudenbush testified, I believe, that he 15 believed that the standard deviation -- I probably won't say 16 this correctly so you will have to correct my question, 17 perhaps, and then answer it. I believe he testified that 18 there were 11 standard deviations in different separated 19 odds ratios between 1997 and 2000. Did you understand that 20 he testified along those lines? 21 A Yes, I remember being asked that in cross and I read 22 his testimony to that effect that the difference was about 23 11, I think he said, almost 11 standard deviations. 24 Q Do you disagree with that opinion? 25 A Oh, yes. That's wrong. Excuse me. 48 1 Q Do you have any understanding based on what you have 2 seen from Dr. Raudenbush's testimony and the data that you 3 have as to how he could have come to that conclusion? 4 A Well, if I might, can I do the calculation for the 5 Court, do the calculation in the regression coefficient in 6 the appropriate scale and then show you what the difference 7 is? And then I obviously don't know exactly how he did 8 the calculations, but I have an idea of how he did the 9 calculations, okay? 10 THE COURT: Certainly. Why don't we -- just for the 11 record, why don't we just put some identification on that. 12 MR. KOLBO: I was going to ask if we could mark it, 13 Your Honor, and offer it, as well. 14 THE COURT: Why don't you at least mark it and then 15 we can talk about offering it. 16 MR. KOLBO: 226. 17 THE COURT: What's the next number? 18 MR. KOLBO: Do you want to write 226? 19 THE COURT: Ex. 226, just so we can talk about it, 20 then you can move it later. Just so we have something for 21 the record that we know where it is, okay. 22 THE WITNESS: I need to do the comparison. I have 23 to do the -- compare standard errors. 24 MR. DELERY: We have another easel with another pen. 25 THE WITNESS: That's okay, let me record them, then 49 1 we will do it. 2 1997, the coefficient was what, 3.98, and the 3 standard error was .29. That's okay. We're fine. And it 4 was 6.09, is that right, and the standard error was 0.4, 5 okay. That's all we need. 6 THE COURT: And then if you'd just put on the 7 right-hand bottom corner again, 227, Ex. 227, and the record 8 will reflect that he is now working on 227, proposed. Okay. 9 THE WITNESS: Well, from these values we can see how 10 different they are. And I am going to bring this back just 11 to show you, then I'll move it back over. 12 You can see this value here is what, 12 standard 13 deviations from zero, right, 12 standard deviations from 14 zero, and this one, this one here is what, 14 standards 15 deviations from zero, and what Dr. Raudenbush has testified 16 to is that if we compare these two, we get something close 17 to 11. 18 Well, I mean, statistically, and I think I hope 19 I reacted appropriately when someone asked me, would it 20 surprise you that the difference between these two is 11, 21 given this one all the way down here is twelve and this one 22 all the way down here is 14, would it surprise you that 23 this difference is 11, and the answer is, it would totally 24 surprise me, and the reason is that we're carrying this to a 25 fixed value of zero, and these are compared to each other, 50 1 and each of them has a standard error attached to it, so it 2 doesn't -- it's not reasonable that this difference should 3 be 11. 4 So I went back and did the calculation. And I'm 5 going to show you how to do the calculation now and show 6 you what the number is, okay. What I need is I'm going to 7 compare these two coefficients, so I'm going to compare 8 6.09, the difference between 6.09 and 3.98. That's what I 9 need to compare. 10 In order to compare this difference, I have to get 11 what's called the standard error for the difference, and 12 this is something that we teach in our first course in 13 statistics. And if we have got two standard errors and we 14 combine them to the standard error for the difference, 15 assuming they are computed from different data sets -- which 16 they are, I didn't put anything together, '97 was done 17 completely separate from 2000 -- so the standard error of 18 the difference, actually the way we do it in statistics, we 19 always have to do things in the variance scale, so we have 20 to square these things. So we put them together, 0.29, we 21 square it, and 0.49, we square it, and then to get the 22 standard error we have to go back and we have to take the 23 square root of that. So that's the math. We square these 24 two numbers, add them up, and take the difference. 25 Now, I should have a calculator. 51 1 MR. DELERY: It's .31. 2 THE WITNESS: You got .31? Let's see. 3 THE COURT: He did it in his head, see. 4 THE WITNESS: But he is not testifying. 5 THE COURT: That's okay. For your profession, the 6 calculator is the most important tool. For lawyers, it's 7 their business card. 8 THE WITNESS: Okay. Let's just make sure I do it 9 right. So I have to make sure I do it right, and the value 10 I get, this one is .49, it can't be any smaller than that, 11 it's got to be bigger than that, bigger than the total, each 12 one separately, and it turns out to be 0.57, 0.57, okay. 13 And so we divide this by 0.57. That's the standard 14 error of the difference. And this tells us how many 15 standard deviations these two numbers are apart, and that 16 number turns out to be 3.7, 3.7. 17 Now, that's in the range of big. Two or three is 18 big. This is a big number, but it's not 11. Am I concerned 19 that this number is this big? Well, in fact, I think there 20 probably are some year to year differences, and I think I 21 testified to that before. There probably are. We don't 22 expect these coefficients to be the same year to year. 23 There is variation. There might be differences in the way 24 things are done. 25 Is this a really large number? In the context of 52 1 looking, what, at the most extreme case, right? 1997 versus 2 2000, those are the most extreme cases. Is this a large 3 number? The answer is, it's not large when you consider 4 that it's the most extreme and the fact that we don't expect 5 it to be exactly the same year to year. So this is a number 6 that I think is consistent with the kind of variation we 7 saw on the previous plot, that these are not 11 apart or 8 almost 11. 9 THE COURT: And 11 would be a large number, you 10 would agree with that? 11 THE WITNESS: I think I agreed in my cross 12 examination that I thought 11 was -- I was surprised. 13 I couldn't believe it was 11, okay, but I didn't dare do the 14 calculations right there on the spot. So 3.7, that's given 15 where we are, that's the value. Well, it is the value, so 16 it's reflected in the plot. 17 Q Dr. Larntz, do you have any -- based on what you 18 understand from the data, is there any way you could 19 understand how someone could conclude that the standard 20 deviation was 11? 21 A Well, I don't know exactly what Dr. Raudenbush did, 22 but if, if you didn't go into the regression coefficient 23 scale, if you tried to work in the relative odds scale, if 24 you didn't go to the basic logistic regression coefficient 25 scale and tried to do these directly and used this standard 53 1 deviation, but didn't take this, that we had to work in 2 coefficient scale, if you did the math, the same math that 3 we did over here, but you just took these numbers and 4 assumed that you could take the standard error of this one 5 by dividing by that, if you did that, if you did that, 6 which is -- 7 THE COURT: That's not an acceptable statistical 8 procedure? 9 THE WITNESS: Well, it's not right. It's not right. 10 It's not right, because these have more variations. We saw 11 that from the confidences. These have more variations than 12 that. I mean, the standard deviation associated with this 13 isn't about ten, it's much more variation than that, or 11 14 or 15 or whatever. 15 So, but if you did that, if you did the math that 16 way, and did the same, to my understanding, did the same 17 thing, you would get 10.94 if you did it that way. So I 18 don't know what was done, but I know if you did that, you 19 would get 10.94. 20 THE COURT: And you know if that was the way it was 21 done, it's not done directly, in your opinion? 22 THE WITNESS: Oh, absolutely, that's right. 23 BY MR. KOLBO: 24 Q Is that something statisticians could reasonably 25 disagree with, whether it's appropriate? 54 1 A There is no disagreement about that at all. I mean, 2 the logistic regression coefficient scale is the scale for 3 comparison, not the -- and the relative odds are derived 4 from those directly for purposes of understanding the model. 5 Q I just have a few more questions, Dr. Larntz. 6 I think Dr. Raudenbush testified -- I understood him 7 to testify that your analysis really demonstrates only that 8 a race factor was used as a factor in the admissions 9 process. It doesn't demonstrate anything about the extent 10 to which race was used in the admissions process. Do you 11 agree or disagree with that? 12 A Well, I would disagree, and shortly -- can I just 13 briefly describe? 14 I disagree in the sense that these are big 15 representative odds. There is a tremendously large 16 allowance given to race, and it's hard to believe that any 17 other factor could explain that away, unless it were just a 18 surrogate for race. 19 MR. KOLBO: I have nothing further, Your Honor. 20 I would like to offer the exhibits, Your Honor. 21 THE COURT: Any objection? 22 MR. DELERY: No objection, Your Honor. 23 THE COURT: Fair enough. 24 MR. KOLBO: It's 225, 226 and 227. 225 is the first 25 handout. 55 1 MR. DELERY: Oh, the charts, okay. All right. 2 THE COURT: You know, if you remind me Monday, we 3 may be able to see if Court Services -- actually, when we 4 have those and admit them, they have a Polaroid, I think 5 they still have it, and we can take some Polaroids of that 6 so that everybody will -- I don't know how else to get it to 7 a form so everybody will be able to have it, so if somebody 8 reminds me Monday, I'll try to see if they have a Polaroid 9 still up there and take some pictures. If not, I'm not sure 10 how we will do it. 11 MR. PURDY: Just a suggestion, if the Court would 12 prefer, we're happy to have these typed up. We would be 13 happy to do that and then just -- 14 THE COURT: That's even better. Nobody objects? We 15 will keep these. You can all sign off on them. 16 MR. PURDY: We won't even take them out of the 17 courtroom. We will just copy them and present them. 18 MR. DELERY: Or mark Dr. Larntz' notes if they are 19 the same thing. 20 THE COURT: I think we should use what he used and 21 preserve those for the official record and type them up and 22 if for some reason you can't do that, or if you can't do it, 23 we will take the numbers, we will get them down here to take 24 pictures. 25 In Judge Taylor's courtroom, they have the white 56 1 board now that you write on and then you push a button and 2 the thing comes out. 3 CROSS EXAMINATION 4 BY MR. DELERY: 5 Q Good morning, Dr. Larntz. 6 A Good morning. 7 Q Welcome back. 8 Like Mr. Kolbo, I think I'm going to -- certainly 9 hope that I stick to what you have covered here this 10 morning. We're not going to go back over the ground that we 11 covered last month when you were here the first time, and I 12 think I'll start where we left off and then work back from 13 there. 14 MR. DELERY: If I may approach, Your Honor? 15 THE COURT: Absolutely. 16 BY MR. DELERY: 17 Q The first two columns here in Exhibit 226 were taken 18 from your expert reports; is that correct? 19 A That's correct. 20 Q The first relative odds numbers came from model one 21 for each year; is that right? 22 A Yes, I think that's right. That's the first set of 23 analyses, that's true. 24 Q You reported three composite odds ratios for each 25 year; correct? 57 1 A That's true. That's true. I was responding to the 2 criticism of this particular comparison. 3 Q Right. I just want to get it clear what these numbers 4 are. These were the model one numbers from each year? 5 A Sure. 6 Q Okay. And then the second column here that you have 7 headed SD are what your report called the standard 8 deviations on the same tables from model one? 9 A That's right. 10 Q And the standard deviations were reported in your 11 report on the log scale; correct? I mean, these are 12 standard deviations in the log scale? 13 A They are standard deviations for the regression 14 coefficient, that's right. 15 Q And the regression coefficients are in the log scale? 16 A Well, the logistic, l-o-g, regression, does analysis 17 on the log odds, so that's correct. 18 Q And the odds ratios, the estimated relative odds in 19 the first column, are not on the log scale? 20 A Oh, that's true, yes. 21 Q And is there a name for not on the log scale, is there 22 a term for that? 23 A You mean on just the normal scale? 24 Q Is that what it's called? 25 A The real scale? I mean, relative odds scale, the 58 1 relative odds scale. It's the relative odds scale. 2 Q It's the normal scale; is that -- 3 A I would call it the relative odds scale. 4 Q Okay. 5 A As opposed to the log odds scale. 6 Q And -- okay. The scales are not -- if you line them 7 up, they work differently; correct? 8 A I mean, there is a direct translation one to another, 9 but obviously the relative odds are it's a trans -- they are 10 a direct translation of each other. 11 Q So just so I'm clear, the two numbers reported in your 12 reports for each of these model results were on different 13 scales? 14 A Well, I guess I wouldn't consider it that way. I 15 would consider that, in fact, I gave the relative odds, 16 that's an appropriate way to summarize the value, and this 17 is the significance to understand the relative odds. So 18 I think the standard deviations are directly related to 19 the significance of relative odds. 20 Q The standard deviations are not the standard 21 deviations of the relative odds; is that right? 22 A Oh, no, no. They are an indication of the statistical 23 significance of the relative odds. 24 Q But they are actually the standard deviations for the 25 regression coefficient? 59 1 A For the regression coefficient, that's exactly how you 2 have to do that, yes. 3 Q Okay. And that's the way you reported it in your 4 report? 5 A The report, I reported with those two columns, that's 6 exactly right. 7 Q And then it was the calculations that you have done 8 here that led to the third column for the regression 9 coefficient; is that right? I guess that's actually the 10 fourth column on this page. 11 A The numbers there, I re -- I back calculated, but they 12 are the numbers from the -- if you looked at the computer 13 output for logistic regression, that those are the numbers 14 that you would see in the computer output, yes. 15 Q And so the relative odds is just the -- well, I'll go 16 the other way. To get the regression coefficient, you just 17 take the log of the relative odds? 18 A Of course, the way you get the relative odds is you 19 exponentiate the regression coefficient, that's how, but the 20 regression coefficient is the basic number that you work 21 from and then you exponentiate that to get it in relative 22 odds. That's a standard output in logistic regression, is 23 to report both of those. 24 Q So you can move back -- what you are saying is you can 25 move back and forth between the log scale and the relative 60 1 odds scale? 2 A Well, they represent the same thing. 3 Q Turning now to Exhibit 227, the standard error of 4 the difference here between the odds ratio -- or the 5 coefficient, I'm sorry -- for 1997 and 2000, you calculate 6 as 3.7? 7 A That's right. 8 Q That's a fair statement of what's on this page? 9 A I mean, someone can check the math, but -- 10 Q Okay. Am I right that anything over two is considered 11 statistically significant? 12 A Without a selection bias in the sense of looking at 13 extremes, anything over two would have a five percent 14 statistical significance, but we're not doing just any two 15 here, we're doing the most extreme two, so we have selected 16 the two we're looking at out of a group, and I would say 17 that in that case you need a value, I don't know what the 18 value is, but it certainly would range, when you're 19 selecting out of, you know, the most extreme cases, that two 20 doesn't go anymore. You have got to take account of the 21 fact that you're selecting from the most extreme cases. 22 And so typically values of -- you know, you wouldn't be 23 surprised if you got values of three or four when they are 24 the most extreme. 25 Q Okay. Putting aside the most extreme context, two 61 1 ordinarily would be statistically significant? 2 A I mean, you want me to ignore the fact that -- where 3 the data came from? 4 Q For this purpose, I'm just asking you, if -- 5 A If I ignore where the data came from, then you have 6 about a five percent chance of getting a value outside the 7 ranges of two if the data comes from a normal distribution. 8 Q So greater than two ordinarily would be considered 9 statistically significant? 10 A In the context of something that wasn't generated as a 11 selection -- 12 Q All right. 13 A -- that would be true, in something that wasn't 14 generated as a selected. 15 Q And greater than three would ordinarily be considered 16 highly statistically significant, is that fair to say? 17 A In the same context, where you're not -- where you're 18 just looking straight away, not where you're looking at 19 extremes. 20 Q Okay. Let's go back, if we could, to Exhibit 226. 21 Your view is that all of the odds ratios that you have 22 reported here, both in the first column from your report 23 under the RO column, and then in the confidence interval 24 parentheses, are very large, is that your testimony? 25 A By the standard of statistical practice, they are very 62 1 large, absolutely. 2 Q So in the year 2000, for example, what your results 3 show is that the relative odds for African Americans as 4 opposed to whites could be anywhere from 166 to 1,176? 5 A The data are consistent with those values, yes, at 6 the 95 percent, at the 95 percent confidence level. 7 Q And so we can't really be sure where in that range it 8 falls, but you're fairly confident that it's somewhere in 9 that range? 10 A I mean, the interpretation of a confidence interval 11 now, you want to do that? Okay. 12 Q I mean, have I fairly -- 13 A Those are the values that are consistent with the 14 data at the 95 percent confidence interval, confidence 15 coefficient level, that's right. 16 Q Okay. And so we can do the same thing for the other 17 years, 1999, it's somewhere between 89 and 478? 18 A Well, knowing that those statements are made with 19 error rates that are five percent, that's right. 20 Q But the conclusion you draw is that all of these are 21 large, so these differences don't trouble you? 22 A Oh, that's right. They are all substantively 23 consistent, that's what I said, and that's exactly right. 24 Q And then when you plotted it here, you said that the 25 reason that these differences, the differences in the first 63 1 odds ratios that you reported don't trouble you is because 2 they are all quite far away from zero? 3 A They are all quite far away from zero and they cluster 4 together. 5 Q So again, here zero means a relative odds of one, am I 6 right? 7 A That's right. 8 Q Which means that the members of both groups have an 9 equal likelihood of being admitted? 10 A That's true. 11 Q So from these odds ratios you can be quite sure that 12 you have disproved a contention that both groups have the 13 same likelihood of being admitted as a percentage basis? 14 A You certainly can disprove that and you can disprove a 15 lot of other odds ratios, because the lower bounds are far 16 away from one, that's right. 17 Q So you think that you can be confident of more than 18 just that the so-called null hypothesis is disproved from 19 these? 20 A Oh, certainly, certainly, certainly, and from the 21 size of the standard deviations, the number of standard 22 deviations and the corresponding confidence, yes, certainly. 23 Q You believe, in fact, that you have quantified, 24 I believe that you said, the role that race plays in 25 admissions based on these numbers; is that right? 64 1 A Well, I am just a statistician, so I'll be careful of 2 what I did, okay? What I have done is I have described what 3 the admissions office did, okay? And so this quantifies the 4 admissions decisions, okay? And so what I have done is 5 described, quantified the admissions decisions, and this is 6 from their data and this is what their data tells me about 7 that. 8 Q And I want to be clear about that, because we 9 obviously had some fairly lengthy discussions of this issue 10 when you were here last month. 11 Today, Mr. Kolbo asked you some questions about 12 whether anything that Dr. Raudenbush said in his testimony 13 changed your opinions concerning the effect that race plays 14 in the admissions process. Do you remember those questions 15 from Mr. Kolbo? 16 A I'm not sure of the exact wording, but I took that to 17 mean if was there any change in my opinion from previously, 18 and I answered that there was no change in my opinion 19 previously. 20 Q Okay. So putting aside that, now I want to ask the 21 follow-up question. Do you believe that you have expressed 22 an opinion concerning the extent -- and I'm sorry, I guess I 23 may have misspoken a moment ago. I think Mr. Kolbo used the 24 word extent, not effect, if I misspoke, but do you believe 25 that you have expressed an opinion concerning the extent to 65 1 which race is taken into account in the admissions process? 2 A What I have done is expressed an opinion of the size 3 of the allowance that is shown in the data that's from the 4 admissions office for individuals that have the similar 5 credentials, the advantage that's given based on ethnic 6 groups in those for individuals with similar credentials, 7 and that's what I testified to, I hope, and that's really 8 what the conclusion is, is that I have quantified 9 statistically the size of the allowance that's given 10 for individuals with similar credentials. 11 Q And just so we're clear, by size of the allowance, do 12 you mean by an admissions officer sitting down to read a 13 file in making the decision? 14 A I mean describing the admissions decisions, what show 15 up in their -- from their data. 16 Q What shows up as a result of the decisions? 17 A As a result of the decisions, absolutely. It's the 18 results of the decision. I don't think I have ever said 19 or didn't mean to say that I did anything other than to 20 describe. It's their decision. I'm describing what the 21 results are. 22 Q Right. But again, just so we're clear, that's 23 different from describing how the decisions were made; 24 correct? 25 A These are the results of the decisions. 66 1 Q So is the answer to my question yes? 2 A Well, I don't know exactly how decisions are made. 3 Q Okay. And these data don't say anything about how 4 decisions are made? 5 A They say a good deal about the results of those 6 discussions and that's what I would say. 7 Q But not how the decisions were made? 8 A The mechanism for making decisions, that's not 9 statistics. 10 Q The process? 11 A That's not statistics. 12 Q I believe that you said earlier in asking -- answering 13 the questions for Mr. Kolbo that you're confident in the 14 substantive stability of your results, but that the odds 15 ratios were not technically statistically stable across the 16 years, did I get that right? 17 A I think that what I would say is that the substantive 18 results are clearly consistent, okay, clearly consistent. 19 There may be, and I think there probably is, I would expect 20 there to be some technical, in the sense of statistical, 21 variation from year to year. I wouldn't expect the 22 variation from year to year to year, so what I was 23 responding to is I think that the variation from year to 24 year is probably not zero. There is probably some -- in 25 the underlying true scale, that there probably is some year 67 1 to year variation, but it's not large enough to change the 2 substantive conclusions, that's right. 3 Q So the differences among the odds ratios across years 4 don't lead you to alter the basic conclusions that you have 5 reached, that's the bottom line of what you're saying? 6 A Substantively they are the same from year to year, 7 that's what I said. 8 Q I think maybe now I would like to go, if we could, 9 back to the last page of the packet that you have. It's 10 Exhibit 225, I believe, if you could look at that. 11 A I have it. 12 Q The last page was the chart of the relative odds 13 effect on baseline probabilities. 14 A That's correct. 15 Q Am I right to understand that this is basically a way 16 to translate odds to probabilities, is that what this is 17 doi