1 1 UNITED STATES OF AMERICA 2 IN THE UNITED STATES DISTRICT COURT 3 FOR THE EASTERN DISTRICT OF MICHIGAN 4 SOUTHERN DIVISION 5 BARBARA GRUTTER, 6 For herself and all others 7 Similarly situated, 8 Plaintiff. 9 -vs- Case Number: 10 97-CV-75928 11 LEE BOLLINGER, JEFFREY LEHMAN, 12 DENNIS SHIELDS, and REGENTS OF 13 THE UNIVERSITY OF MICHIGAN, 14 Defendants, 15 -and- 16 KIMBERLY JAMES, et. al., 17 Intervening Defendants. 18 ______________________________________/ VOLUME IV 19 BENCH TRIAL BEFORE THE HONORABLE BERNARD A. FRIEDMAN 20 United States District Judge 238 U.S. Courthouse & Federal Building 21 231 Lafayette Boulevard West Detrot, Michigan 48226 22 Friday, January 19, 2001 23 APPEARANCES: 24 FOR PLAINTIFF: Kirk O. Kolbo, Esq. 25 R. Lawrence Purdy, Esq. 2 1 APPEARANCES (CONTINUING) 2 FOR DEFENDANTS: John Payton, Esq. 3 Craig Goldblatt, Esq. 4 Stuart Delery, Esq. 5 On behalf of the Defendants 6 Bollinger, et. al. 7 8 George B. Washington, Esq., 9 Miranda K.S. Massie, Esq. 10 On behalf of Intervening. 11 Defendants. 12 COURT REPORTER: JOAN L. MORGAN, CSR 13 Official Court Reporter. 14 15 Proceedings recorded by mechanical stenography. 16 Transcript produced by computer-assisted 17 transcription. 18 19 20 21 22 23 24 25 3 1 I N D E X 2 WITNESS PAGE 3 STEPHEN W. RAUDENBUSH 4 Direct Examination by Mr. Delery 5 5 Cross-Examination by Ms. Massie 118 6 Cross-Examination by Mr. Kolbo 121 7 Redirect Examination by Mr. Delery 160 8 DENNIS SHIELDS 9 Direct Examination by Mr. Payton 162 10 Cross-Examination by Mr. Purdy 193 11 Redirect Examination by Mr. Payton 215 12 Recross-Examination by Mr. Purdy 218 13 E X H I B I T S 14 NUMBER IDENTIFICATION ADMITTED 15 145 Expert Witness Report of S. Raudenbush 12 16 146-150 Supp. Expert Witness Rep. of S. Raudenbush 12 17 151 Raudenbush Curriculum Vitae 9 18 184-194 Charts of S. Raudenbush 108 19 5 Gospel According to Dennis 188 20 21 22 23 24 25 4 1 Detroit, Michigan 2 January 19, 2001 3 * * * 4 THE COURT: Good morning, everyone. On the 5 motions, I have nothing else on the agenda this case, why 6 don't we start the case and when we take it a break sometime 7 we'll argue those motions. 8 MS. MASSIE: That sounds great. 9 THE COURT: Is that good for everybody? I'm all 10 prepared, but I just don't want to waste your time this 11 morning. I know you have a witness. This is yours? 12 MR. DELERY: Yes. Good morning, Your Honor, 13 Stewart Delery, Your Honor, again for the university and 14 the individual defendants. 15 THE COURT: How are you. 16 MR. DELERY: If you're ready to proceed. 17 THE COURT: I'm ready. If you're ready, I'm 18 ready. We call. 19 MR. DELERY: We call as our next witness, Stephen 20 Raudenbush. 21 THE COURT: For evidence? 22 MR. DELERY: Thank you, Your Honor. 23 S T E P H E N W. R A U D E N B U S H. 24 was called as a witness and after having been 25 sworn was examined and testified as follows: 5 1 DIRECT EXAMINATION 2 BY MR. DELERY: 3 Q Could you please state your name and address for the 4 record. 5 A Stephen W. Raudenbush, 7 Harvard Place, Ann Arbor, 6 Michigan. 7 Q And where do you work? 8 A I work at the University of Michigan. 9 Q What's your job there? 10 A I'm a professor in the School of Education and the 11 Department of Statistics, and I also have a joint 12 appointment as a Senior Research Scientific at the Survey 13 Research Center. 14 Q How long have you been at the University of Michigan? 15 A I've been at Michigan since January 1 of 1998. 16 Q And where were you before that? 17 A For fourteen years before that I was at Michigan State 18 University. 19 Q Well, Professor Raudenbush, could you please, please 20 briefly describe your education, or educational background 21 for the Court. 22 A Sure. I received my bachelor's degree from Harvard 23 College in 1968 and my doctoral degree from Harvard 24 University in 1984. 25 Q Has your work at the University of Michigan and before 6 1 that at Michigan State focused on any particular areas? 2 A Yes, it has. It's, primarily my work is involved 3 applications of statistics in education, studying student 4 learning, studying student transitions into college, 5 studying how schools and classrooms effect academic 6 achievement. And also looking at other aspects of human 7 development. 8 Q Okay. And have you published in these fields? 9 A Yes, I have. 10 Q About how many publications have you had? 11 A Well, I guess if you count the second edition of our 12 book on Hierharchical Linear Models, if you count the 13 second edition of our book on Hierarchical Linear Models. 14 THE COURT: Do you want it spelled? 15 (Whereupon an off-the-record discussion was 16 had.) 17 A H-i-e-r-h-a-r-c-h-i-c-a-l. Okay. There would be, if 18 you count that one, there would be four books and quite a 19 large number of referee journal articles and book chapters 20 that I've published over the years. I'm not sure exactly 21 how many but I publish about four to six articles and 22 chapters a year. 23 Q Okay. This may be a relative question, but are any of 24 those publications particularly widely known? 25 A Well, the book I mentioned, I won't mention the title 7 1 again, has become very, very widely used in education 2 because it deals with the problem of students being nested 3 within classrooms, classrooms within schools. Those kinds 4 of problems become very widely used. And other aspects of 5 social science where we have people in neighborhoods, or we 6 have small groupings of people, basically, which has some 7 relevance to this case. 8 Q Are you a member of any professional organizations? 9 A I am. I'm a member of the American Statistical 10 Association, the American Educational Research Association. 11 I'm a member of the National Academy of Education. 12 Q What's the National Academy of Education? 13 A Well, the National Academy of Education is an honorary 14 association limited to 125 people in the United States who 15 are involved in education and educational research. 16 Q Have you held any editorial positions for journals or 17 other publications in your field? 18 A I have. I've been an Associate Editor of the Journal of 19 Educational and Behavioral Statistics for quite a large number 20 of years. I was the Chair of the Management 21 Committee of that journal for six years. I have served on 22 the Publications Management Committee of the American 23 Statistical Association. I'm also the Associate Editor for 24 the American Journal of Sociology, Educational Evaluation 25 and Policy Analysis and actually several other journals. I 8 1 won't list them all. 2 Q Okay. Have you received any teaching or other honors 3 in your field? 4 A I have. I received, while I was at Michigan State, I 5 received three teaching awards. I've also received several 6 awards for outstanding publications in education and 7 sociology. 8 Q Okay. Are there any awards or honors that you think 9 are particularly significant? 10 A I think perhaps the one that I'm, maybe most proud of 11 is that in 1993 I received the Early Career Award for the 12 American, from the American Educational Research 13 Association, which is a very large group of educators and 14 educational researchers around the country. 15 Q What about national panels or symposia? Have you 16 participated in any of those? 17 A Yes. In the last, within the last three years, I 18 served on the National Academy of Sciences' panel on the 19 assessment of children in conjunction, basically testing, in 20 conjunction with the Title I Program, which is a 21 compensatory education program. I also served on the 22 National Academy of Science panel on early childhood 23 science, which has just distributed a new book on childhood 24 science with implications for policy and practice. 25 Q Okay. Professor Raudenbush, I'd like to ask you to 9 1 look at Exhibit 151, which is, I think in binder six, Your 2 Honor. 3 A Okay. I see it. 4 Q Okay. Is that a current copy of your CV? 5 A It does indeed appear to be that, yes, a current copy. 6 Q And does it include a current list of your 7 publications and honors and professional experiences? 8 A Yes, it does. 9 MR. DELERY: Your Honor, at this time, we'd offer 10 Exhibit 151 into evidence? 11 THE COURT: Received. 12 Q Now, Professor Raudenbush, how would you come to be 13 involved in this case? 14 A Actually, you asked me if I'd be willing to serve as 15 an expert in this case. Can you hear me? 16 Q Yes, I can. 17 THE COURT: If anybody can't, let us know. 18 A Yeah. I had to move this because I can't turn the 19 page. 20 THE COURT: Yeah, that's correct. 21 Q And what was the purpose of your involvement in the 22 case? 23 A Well, I started by looking at some of the expert 24 reports written by Professor Kinley Larntz and I then got 25 involved in looking at the database myself, in trying to 10 1 understand some of the issues involved in this controversy. 2 Q Okay. Were you present here in court for Dr. Larntz' 3 testimony on Wednesday? 4 A Yes, I was. 5 Q And you were here for the entire day for all the 6 testimony? 7 A I was. 8 Q And what about on Thursday morning, yesterday morning 9 when he returned? 10 A I was here then too, yes, correct. 11 Q Dr. Larntz at one or two points said that he was 12 responding to some criticism of his work. Do you recall 13 that? 14 A I do. 15 Q Were you the author of that criticism? 16 A I'm quite sure that I was. 17 Q And before this week, had you ever met Dr. Larntz? 18 A No. 19 Q Are you being compensated for your work in this case? 20 A No, I'm not. 21 Q And have you ever served as an expert witness before? 22 A No, I have not. 23 Q Have you prepared expert reports, based on your work 24 in this case? 25 A Yes, I have. 11 1 Q Okay. If you could look in the same binder there, 2 binder six, I'd like for you to look at Exhibit 145 to 150 3 and tell the court whether those are the expert reports 4 that you submitted in this case? 5 A Yes, these are, these are the expert reports. 6 Q What information did you consider in preparing your 7 expert reports? 8 A Well, I read the law school admission policy, which 9 was dated 1992. And I examined data from the database made 10 available by the law school. 11 Q Did you review the expert reports of Dr. Larntz? 12 A Yes, I read also each, each expert report that Dr. 13 Larntz wrote. 14 Q Okay. And what about any deposition testimony in the 15 case, did you review any of that? 16 A Yes. I read Dr. Larntz' deposition. Of course I read 17 my own. 18 Q Did anybody help you with your work in this matter? 19 A Yes. Julia Smith, who was at that time a 20 post-doctoral fellow at Michigan, helped me. She's now an 21 assistant professor. And in certain aspects of the work 22 the, basically the diversity of context for learning part, 23 I received some help from two graduate students at the 24 University of Michigan. 25 MR. DELERY: Your Honor, at this point we'd offer 12 1 Exhibit 145 through 150 into evidence. 2 THE COURT: Any objection? Received. 3 MR. DELERY: We'd also at this point offer 4 Professor Raudenbush as an expert in the application of 5 statistical methods to education. 6 THE COURT: Any objection? 7 MR. PAYTON: No. 8 THE COURT: Okay. 9 Q All right, Professor Raudenbush. I believe you 10 mentioned that you reviewed Dr. Larntz' work in this 11 matter. 12 A That's correct. 13 Q Do you have an opinion concerning, now just as a 14 summary matter, we'll get into it in more detail. But do 15 you have an opinion concerning the reasonableness of the 16 approach that Dr. Larntz took and his results? 17 A I do. 18 Q And what is that opinion? 19 A I'm actually quite skeptical for two reasons. Dr. 20 Larntz attempted to construct a statistical model that 21 could tell us the extent to which race is taken into 22 account in admissions. And I'm convinced that it's not 23 logically possible to answer that question with such a 24 statistical model. 25 Moreover, certain methodological decisions made by 13 1 Dr. Larntz, I believe, have led to a misleading impression 2 about the strength of association between minority status 3 and admissions at the law school. 4 Q Okay. Now, you indicated that in addition to 5 reviewing Dr. Larntz' work, you did some things of your 6 own. What did you do in your analysis? 7 A Well, as I implied, I think it's, it's not possible, 8 given the data at hand, to organize a statistical analysis 9 that's going to tell us the extent to which race is taken 10 into account in admissions. What we can do, however, and 11 what I think is very useful, is to do a causal analysis of 12 the impact of using race in admissions on those who apply 13 to the university or to the law school. 14 Q Okay. And what are the basic conclusions again, as a 15 summary matter that you draw from your work in that 16 context? 17 A What we did, and we'll go into some detail on this, is 18 we compared the current policy, which does use race as a 19 factor in admissions to an alternative policy that would 20 not use race as a factor. And we estimated how that 21 difference in policies would effect the average probability 22 of admission of various people who apply, various 23 sub-groups of people who apply at the University of 24 Michigan. 25 And essentially what we found, first, of course, 14 1 is that a change in the policy would effect people 2 differently, depending on grades and test scores. It would 3 also effect people differently, depending on ethnic 4 minority status. A switch from the current policy to a 5 so-called race-blind policy would have a fairly substantial 6 effect, negative effect, on the probability of admission on 7 minority candidates. 8 On the other hand, such a change from, again the 9 current policy to a race-blind policy, would have a 10 comparatively modest effect on the positive effect, that 11 is, on the average probability of admission of majority 12 students. 13 Q And from your work, do you draw any conclusions about 14 the likely effect on the diversity of the law school class 15 of moving to a race-blind admissions policy? 16 A Yes. We can then take the admissions probabilities 17 under the current policy, as compared to an alternative 18 policy. And from that data, we're able to project the 19 number of applicants, not only who will be admitted, but 20 then using yield statistics, how many would then, in fact, 21 attend the law school. And then we can have an estimate of 22 how the class composition would look of the first-year 23 students at the law school. And so we're then able to make 24 some statements about the likely diversity with that class. 25 Q And what do you conclude? 15 1 A And what we conclude is that switch from the current 2 policy to a so-called race-blind policy would, would quite 3 dramatically reduce the fraction of students who are from 4 underrepresented minority backgrounds, and we'll define 5 that as we go, and to try to understand the practical 6 implications of that, we then took a look at how that would 7 translate into the composition of various contexts for 8 learning that occur in the law school. And, again, the, 9 how different classrooms and other context for learning 10 would look under the current policy versus an alternative 11 policy is really quite different. 12 Q Well, with that sort of basic overview in mind, let's 13 go back and talk in more detail about how you arrived at 14 these conclusions. 15 A Okay. 16 Q What was, basically, the first thing that you did when 17 you approached these data? 18 A Well, the first thing we did, and we did this for each 19 year between 1995 and 2000, was just to take a look at the 20 basic data; who applied at the law school, who was 21 admitted, who, how many people who were admitted decided to 22 come to the law school, and then what was the composition 23 of the first year class for each of those years. 24 Q Okay. And I think we've prepared a chart of an 25 illustration of that, is that right? 16 1 A Yes. 2 Q Is that right? And I'd like to put up, if I could, 3 Your Honor, Exhibit 184, the series of exhibits, I think, 4 is in the supplemental exhibit file. And the lights on 5 would be fine, because they're just words today, no screen. 6 A Your Honor, may I stand up and explain what's on the 7 screen? 8 THE COURT: You may absolutely stand up and 9 explain, yes, or we can move it closer to you so you can 10 sit. 11 A Yeah. 12 THE COURT: You're a professor, you're used to 13 standing and talking. 14 A That's right. Either that or I'll have to get new 15 bifocals. 16 THE COURT: Yeah, whatever. 17 A That's fine, thank you. 18 THE COURT: I've got a pointer here if you'd like 19 one, too, however you got to promise to give it back. 20 A Right. 21 THE COURT: Because the government, again, we can 22 get almost anything we want, but pointers. They're hard to 23 come by these days. 24 A It will be hard to walk away with this. 25 THE COURT: Yeah. 17 1 Q All right. So this chart is of the 2000 admissions 2 data, is that right? 3 A That's right. 4 Q Why don't you explain what's here and what you find 5 significant about these numbers? 6 A Well, the basic idea behind this chart is that it 7 shows quantitatively how a pool of applicants gets 8 translated into people who actually attend the law school. 9 And the thing to illustrate that, I'll just use 10 the top row of the chart in 2000. And we break this down 11 by ethnic groups. So just to take the first group here in 12 2000, there were 262 African-American applicants and that 13 constituted about 7.4 percent of the applicant pool. 14 And of those 262 people, 36.3 percent were 15 admitted. And that led to 95 offers of admission for that 16 group. Now, of those people who were offered admission, 17 only a minority, 40 percent, decided to come to the law 18 school. So if you multiple 40 percent times the 95 who 19 were admitted, then you get the number of African-Americans 20 who actually were attending the law school in 19, in 2000, 21 and that turns out to be 38. 22 So what you, basically, see is that this number on 23 the left which is 262, ultimately becomes 38, through whose 24 admitted and whether they decide to attend. That's the 25 basic idea on the chart. 18 1 Now, what we've done is, is to, to make this 2 clear, and I think in conformity with the law school policy 3 of admissions, is we've taken three groups; 4 African-Americans, Hispanics and Native Americans and 5 combined their data in the lower panel here to the data, to 6 a group that we label those of underrepresented minority 7 status. So that -- 8 Q That's UMS? 9 A And that's called UMS in this table. And then we have 10 taken data from the Caucasian group, Caucasian American, 11 and those, those whose ethnicity is unknown, and again, 12 that's in accord with our understanding of the how the 13 policy works. And we've taken their data and combined them 14 into another group that we call them non-UMS. They are the 15 ones who are not in the underrepresented minority status. 16 Now, that leads one group that I haven't 17 mentioned, and that's the other group, the non-citizen 18 group, and that group, we do not include in this table. We 19 could have looked at underrepresented minority status, yes 20 or no, and foreign or foreign students. But the numbers, 21 in fact, there were only three foreign students attending 22 in 2000, are really too small to do much with. And it 23 seemed that whatever was happening with minority status and 24 non-represented minority status was somewhat different 25 because this group is ethically very diverse, the foreign 19 1 group, and yet they're not in these categories so we didn't 2 include that small number of applicants. 3 So then down at the bottom what we, basically, 4 have are underrepresented minority students and 5 non-underrepresented minority students and then a total. 6 Q Do you find anything significant about the pattern of 7 the numbers here on the bottom half of the chart? 8 A Yeah. There's several significant features of this 9 table. One is we just start by just looking at the 10 applicant pool. So we see that there are 484 applicants 11 who are minority. I'm just going to use the word 12 minority's" and "non", I think, because it gets hard in 13 saying. 14 THE COURT: That would be great. We all 15 understand. 16 A And I'll try, and I often may use the word "race", and 17 I don't necessarily mean "race". We know there's ethnicity 18 and it's complex, but I'll use it because it gets hard to 19 use so many words. 20 But so 484 minority applicants, and in contrast to 21 2,871, majority applicants, or non-majority applicants. 22 And so the pool sizes are very different. There's a much 23 smaller number of minority applicants than non. So that's 24 one factor that we -- it's very important in 25 understanding -- the dynamics of this whole system is just 20 1 the different sizes of that applicant pool. 2 The next feature that's very important to look at 3 is just the percentage admitted, because that's a crucial 4 factor in, who ends up being in law school. And what we 5 see is that 35.1 percent of the minority applicants and 40 6 percent of the non-minority applicants are admitted. 7 And these numbers are quite reflective of what 8 happens year to year. Only a minority of people, of the 9 overall applicant pool is admitted. The numbers are pretty 10 similar. In general, the fraction admitted is smaller for 11 the minority group than for the non-minority group. 12 Q And is that true in each of the years from 1995 to 13 2000? 14 A That is true. The general pattern is true each year. 15 These numbers will fluctuate but the general pattern is 16 true. And the, from the point of view of promoting 17 diversity, ethnic diversity, which is one of the goals 18 stated in the admissions policy, these two facts; there is 19 the small pool size and the comparatively small fraction of 20 those admitted has important implications for the diversity 21 of the class. Because if this number is lower much, the 22 number of people who actually attend can get very small. 23 Specifically, in this case, with 35.1 percent of 24 the minority applicants admitted, and then with the yield 25 of 34.1 percent out of the 484 applicants who are minority, 21 1 what we see as actually attending, 58. So 484 goes down to 2 58. 3 And if you're, you know, if you're interested in 4 diversity, the size of this applicant pool, the fraction 5 admitted and the yield are going to strongly effect this 6 number, and I guess this percentage admitted is under -- 7 obviously under the direct control of the law school. And 8 if we shadow where we're going with our analysis, if this 9 number were reduced significantly, this number 58 would 10 begin to go down. 11 I mean, if this number were cut in half, then we'd 12 have only 29 minority students, so, and that would assume 13 that the number of applicants and the yield would remain 14 constant. Do you see my point? That if we cut this number 15 in half, hold everything else constant, we're down to 29. 16 THE COURT: Or double it and it may go up? 17 A Or double it and it will be go up to 116 if we double 18 it. So whatever we do here has big effects on this number, 19 but we also need to take into account the possibility that 20 changing this number could change this number. It could 21 change the number of people who apply. It could also 22 change this number, the number of people once admitted who 23 might then decide to attend. 24 So in particular, if this number were lower, this 25 number could, would likely -- it probably wouldn't stay the 22 1 same. A more likely outcome, if you lowered the 2 probability of admission of a group, it might encourage 3 fewer people to apply. That's, we don't know. And our 4 analysis won't assume that, but the law school would have 5 to take that into account as a possibility. And lowering 6 this number might also end up lowering the yield because if 7 you, if you reduce this number substantially you would be 8 left with, under a race-blind policy, the people who would 9 be here would be extreme. 10 Q "Here" being the number admitted? 11 A The number admitted would be an extremely highly 12 qualified group, in terms of grades, test scores and so 13 forth. And the yield for such a group may be, may be lower 14 because there may be significant competition, among law 15 schools for those people. So changing this number could 16 impact these numbers. And with, with large effects on 17 this, this relatively small number, 58, so that's, that's 18 the key thing that's happening. 19 Q Right. So this chart is of the 2000 data. Does your 20 report include similar information for the other years? 21 A It does. 22 Q For the various reports? 23 A We have a similar flow chart for each year from 1995 24 to 2000. 25 Q And is the 2000 data unusual, compared to the other 23 1 years? 2 A The 2000 data are pretty similar in virtually all 3 regards. There's one slight difference here. The yield 4 for African-Americans candidates in 2000 was 40 percent, 5 which is, which is higher than it had generally been in the 6 other years. So that number is a little higher than 7 average, but other than that it looks. 8 Q If we compared the, this data, including the number of 9 applicants to some similar charts in Dr. Larntz' reports, I 10 think there may be some slight differences, is that right? 11 A I looked at those numbers. The, the exact numbers are 12 not identical and I don't really know why. 13 Q You worked from the same database? 14 A We worked from the same database. 15 THE COURT: Are they significantly different? 16 A They're not significantly different. 17 Q Okay. 18 A The patterns that I'm describing are very similar, I 19 mean, they're virtually identical in the two sets of 20 figures. 21 Q Now, in addition to your point about how, how the 22 various percentages, in particular, can effect the number 23 attending in each year, do you take any other basic 24 conclusions away from looking at this basic descriptive 25 data? 24 1 A There are a couple of other conclusions. While, 2 remember, I mentioned that a change in this percentage 3 would lead to, perhaps, fairly large changes in this 4 number; that is, the number admitted and also this number, 5 the number attending, and that's for minority applicants. 6 If changes in this number that are small would 7 have comparatively modest effects, if, let's say, half of 8 these people were rejected instead of, let's say, that 9 would be, that would be, we have 170. That would be 85 10 people. If those 85 places became available to the 11 majority students and then these 2,871 would compete for 12 those 85 places, and so that change, which is big here, 13 that is in the minority row, would have a comparatively 14 modest chain effect on the majority role, so that's one 15 additional piece of evidence from this. 16 Q And the comparison or the percentage admitted of the 17 two groups, I think, what also might be called the average 18 probability of admission, is that right? 19 A Yes. 20 Q Does that comparison tell you anything about the 21 impact of considering race in admissions? 22 A Well, this, we call it a, yeah, we call this bivariate 23 association. There are two variables. There's the race of 24 the candidate, and then there's the admission decision, and 25 when we look at these two proportions, that gives us 25 1 evidence about that bivariate association. Is there an 2 association between race and admissions? And we see a very 3 small bivariate association, actually which favors the 4 majority applicants. 5 Now, we use, in statistics, we tend to look at 6 these bivariate associations as a first take on what's 7 going on, just simple data, there's no model, just look at 8 the data. And so we see this relationship. And in 9 conjunction with other bivariate relationships, my 10 conclusion from this was that it, it leads one to be 11 skeptical of a claim that race is a powerful predictor of 12 the admissions decision. 13 Q Not the end of the analysis but a starting point? 14 A It's not the end of the analysis, but, let me expand a 15 little bit. If we look at, let's say, just the association 16 between grades and admissions, there's a very strong 17 relationship, even with higher grades are more likely to be 18 admitted. If we look, and we don't have to control for 19 race to see that. We just see that relationship. If we 20 look at the relationship between test scores, LSAT and the 21 probability of being admitted, we see a very strong 22 relationship, we don't have to control for anything else to 23 see that. We look at the relationship between race and the 24 probability of being admitted, we see very little 25 relationship. 26 1 So that, that tells us that race is unlikely to be 2 a powerful predictor of the outcome. It doesn't mean that 3 race and admissions are not related controlling for other 4 factors, but it does suggest that race will not be a 5 powerful predictor for the admissions decision. 6 Q Okay. I think at this point you can probably take 7 your seat again. 8 A Thank you. Your Honor. 9 THE COURT: No, just hold on to it. 10 A Yeah, I need it again. 11 MR. DELERY: I think we may need it again. 12 THE COURT: Maybe you can move the chart just so 13 the folks in the audience can see. 14 MR. DELERY: Sure. 15 THE COURT: Great. Thank you. 16 MR. DELERY: I apologize. 17 Q Now, in addition to the examination of the basic 18 descriptive data, what did you do as part of your analysis 19 in the case? 20 A Well, as I mentioned, I'm convinced, and I think will 21 explain why a little later. But I'm convinced that we 22 can't develop a statistical model that's going to tell us 23 the extent to which race is taken into account in 24 admissions. What we can do and what I think is useful to 25 do is to do a causal analysis. What's the impact of the 27 1 policy that the university has of using race in admissions 2 on the people who apply. And that causal analysis is 3 something that we can do with a minimum of assumptions. 4 And so that's what I decided to do, and I thought that that 5 would be informative. 6 Q Okay. Have you prepared a chart to sort of explain 7 that causal connection? 8 A Yes, I have. 9 Q Okay. I think for this one, you can probably just 10 stay where you are with the easel where it is. 11 A Especially with this. 12 Q This is Exhibit 185, right, exactly with the long 13 stick? 14 A Right, with the long stick. I don't have to get up. 15 Q So this chart is called conception for causal link 16 between race and admissions? 17 A Right. 18 Q What do you mean by that? 19 A Well, in causal analysis and statistics, the way we 20 think is that we've got, let's say, two alternative 21 treatments. We've got treatment A and treatment B. 22 Now, for each person that we're interested in, we 23 imagine the following, that that person has an outcome 24 under treatment A and an outcome under treatment B, and the 25 difference between the two outcomes is defined, 28 1 statistically, as the causal effect of the treatment. 2 So if I, if one person has, let's say, I could 3 randomly assign a person to have surgery for heart problem 4 or I could randomly assign to have medicine, and the, and 5 the person would have one outcome under the first 6 treatment, another outcome under the second treatment. 7 Causal effect is the difference between the two outcomes. 8 So we applied that basic idea to the, to the scenario here. 9 What we have on the left, what we have up here is, is a 10 person, an applicant which and this person. 11 Q You can tell we're not artistic. 12 A Right. I wouldn't want to be that person, but we have 13 that person. And this person is going to apply to the law 14 school and that person might apply under policy A. Policy 15 A is the current policy, according to the admissions 16 policy. 17 And in that policy, it states a number of factors 18 that are going to be taken into account, and I guess, I'll 19 read them. I don't know if you can see them all; 20 undergraduate grades, the law school aptitude test, 21 Michigan residency, minority status, gender is, could be 22 considered, I assume as a force, a form of diversity. The 23 quality of the undergraduate school, the curriculum; that 24 is the courses that the applicant took, trend in grades, 25 not just are they how or were they going up, relationship 29 1 with family members who are alumni. There are essays that 2 are required, letters of recommendation and leadership 3 experience. A person may have displayed other unique 4 experiences and talents and then unusual circumstances. So 5 this -- there's this list of factors that could be taken 6 into account. 7 Q And these are all things, if I could interrupt you for 8 a second? 9 A Yes. 10 Q That are reflected in the policy, as you read it? 11 A That's right. I, I got these right out of the policy 12 document itself. And so our applicant comes and applies 13 under policy A. All of these characteristics are taken 14 into account and the results is this person has a certain 15 probability of admission. We call it a probability because 16 there's some uncertainty in what's actually going to happen 17 here. There's subjective judgments being made and there's 18 some probability of admissions. So we call that 19 probability A. So that's policy A. 20 Now, if our same applicant were to apply under a 21 different policy, and we're going to call that policy B, 22 the result might different. Policy B is, we label a 23 race-blind admissions policy. And the way we're, the way 24 we're defining that is that all of the same factors that 25 were taken into account under policy A would be also taken 30 1 into account under policy B with one exception, and that is 2 underrepresented minority status. That would not be 3 considered. So we call that a race-blind policy. 4 So our applicant comes along now, low and behold, 5 policy B is in effect. These are taken into account, these 6 factors, and the result is that our applicant has a 7 probability of admission, a piece of B. 8 And so with that scenario in mind, we can define 9 the causal effect of policy A versus policy B as being the 10 difference in the two probabilities of admission. So if, 11 let's say our applicant applied under policy A and got a 12 piece of A, probability under B, a piece of B. 13 Suppose those two probabilities were the same, 14 identical, there would be no causal effect of a change in 15 policy on that person. Suppose, on the other hand, that 16 these probabilities were very different. A person was, 17 let's say, you know, very unlikely to get in under policy A 18 and very likely to get in under policy B, big causal effect 19 of the policy. So that's, basically, how we defined the 20 causal effect. And that was what set up our analysis. 21 Q Now, why do you think it's important to look at this 22 contrast between two policies in this case? 23 A There are two reasons. One is that a change from 24 policy A to policy B could effect the diversity of the 25 incoming class and that's one of the goals stated in the 31 1 admissions policy is to have an ethically diverse class, 2 and so we can use this framework to assess the effect, 3 causal effect on the change of policy on the diversity of 4 the class. 5 The other reason that it's important is that it, 6 it's a way of gauging the causal effect of, on those who 7 apply, I mean, I think that a person who applied to the 8 university, or to the law school, would be very concerned 9 about, are my probabilities going to be very different 10 under these two, under these two policies. If they were, 11 that would have important effect on behavior of people who 12 apply and it's just an important issue and it gauges the 13 extent to which the current policy is strongly effecting 14 the outcomes of people who apply. 15 Q Okay. And is this kind of comparison between 16 alternative policies the standard way in your field to get 17 at causal questions? 18 A This has become the, essentially, the consensus in how 19 we think about causation in statistics, two alternative 20 policies, an outcome under each for each person and the 21 causal effect being defined, as I mentioned. 22 Q Now, how, if at all, does this conception, this 23 approach, differ from what Dr. Larntz did? 24 A Okay. In Dr. Larntz' analysis, he's analyzing the 25 data that were generated under policy A and computing 32 1 correlations or associations and trying to use those to 2 make strong causal inferences. And, as I mentioned, I'm 3 convinced that that's not logically possible to do in this 4 case. This kind of analysis -- 5 THE COURT: You say in this case, in any case? 6 A With, well, I think part of the problem is the amount 7 of available information. With, if, with a great deal of 8 information, one might be able to make a better, I think 9 that's an important constraining piece, if there were 10 enough information, but we really had very limited 11 information about the people who apply, numerical 12 information, so I think that's a key constraint on the, on 13 a correlational approach. Generally. 14 THE COURT: Well, you say. 15 A Sure. 16 THE COURT: Limited numerical information. What 17 other, on your list, there's only certain things that can 18 be equated to numbers. 19 A Right. And that's one of the difficulties in drawing 20 a causal inference from numerical data is the -- 21 THE COURT: Oh, I see. 22 A If the important, if many of the important factors are 23 not co-indentifiable. 24 THE COURT: I see. Thank. 25 A That would be a good reason why we didn't have that 33 1 information. 2 Q Now, with this conception for the causal analysis in 3 mind, what did you do next in your analysis? 4 A What we tried to do then was to compare policy A and 5 B, and I think we have an exhibit that displays how we 6 approach that. 7 Q Okay. Let's put up Exhibit 186 now, the next chart. 8 Does this chart illustrate how you approached your 9 analysis? 10 A It does. Simulating would happen under policy A was 11 very easy because we actually didn't have to simulate it. 12 We have the data from the years '95 to 2000. So we just 13 actually used, we used the actual reported admissions 14 results to compute probabilities of admission, average 15 probabilities, of admission for various sub-groups who 16 applied, and those were just based strictly on the data. 17 Policy B posed us with a more challenging problem. 18 We don't know what the effect will be on the probability of 19 admission under policy B, because it's never been 20 implemented. So we have to make some assumptions. 21 Essentially what we did was we had data on grades, 22 on test scores, Michigan residency and gender. And we can 23 develop, based on past data a prediction equation that 24 would predict the probability of admission, based on past 25 data. And then from that we can simulate what's happening 34 1 under policy B. The problem we face is the same problem 2 that Professor Larntz faced. There's a lot of information 3 that we don't have. We don't know anything about the 4 undergrad school curriculum, etc., essays, recommendations, 5 all these other things, these long list of factors. We 6 don't have any numerical data. 7 Q When you say "we don't know about those things", you 8 mean that, as a statistician looking at the data you don't 9 know? 10 A Exactly. As a statistician analyzing the numerical 11 database, I only have access to a small fraction of the 12 relevant information used in make admissions decisions, so. 13 Q The admissions officers have more information than you 14 have? 15 A Exactly. And that's why, that's one of the reasons 16 why it's difficult to model those decisions. They know a 17 lot more than we do. And we have to make assumptions about 18 what we don't know. In order to do this simulation, we 19 have to assume, essentially, that all of these factors that 20 we don't know anything about are not associated with the 21 factors that are in our model. 22 THE COURT: So you have quite a few there? 23 A That's right. 24 THE COURT: And Dr. Larntz testified that the 25 fewer assumptions you make, and I'm not saying you have to 35 1 agree or not agree, but I'd like your opinion on it. He 2 testified that the fewer assumptions you make, the better 3 your results are. That when you start making assumptions, 4 that it may skew it to subject -- I don't think you used 5 the word, subjective, but at least it's more extensive. In 6 your model you're making assumptions, at least, as to one, 7 two, three, four, five, six, seven, eight, nine, ten areas? 8 A That's right, exactly. 9 THE COURT: So do you disagree with him? 10 A Oh, I agree with him on that, absolutely, yes. We're 11 very concerned about the impact of the possible falsehood 12 of these assumptions. And there are almost certain to be 13 some falsehoods here. The question is the falseness of 14 these assumptions, the question is to what extent does that 15 effect the result. 16 We know we're not going to really have the model 17 right, but to the extent we have it wrong, to what extent 18 does that have some effect on our results. And that's what 19 we then had to do in this was to, what we actually did was 20 we did this simulation. 21 We looked at the results. We repeated the 22 simulation a couple of other ways, but actually, this is, 23 in some ways that I believe the great strength of the 24 causal analysis. We can put bounds on the errors of your 25 our estimates that require virtually no assumptions, so we 36 1 can actually assess the extent to which errors in our 2 assumptions effect our results in a very sure-minded way, 3 and I'll try to explain how we did that as we go. 4 So the way, the way it works is, is you do an 5 analysis, based on assumptions, you look at the results, 6 you try another analysis, generally, that's based on maybe 7 some different, slightly different assumptions. But then 8 you try to bound the error in your results as a function of 9 your assumptions, and we we'll show how we do that. 10 MR. DELERY: I think it will be easier to see 11 that, Your Honor. 12 THE COURT: That's fine. 13 MR. DELERY: After we see the results. 14 Q But before we leave this point, while we're on 15 assumptions and just so we're clear, what, what is, or what 16 are the assumptions about the factors below the line on the 17 chart, as related to the factors above the line, just so we 18 have that in mind? 19 A Right. Basically the assumption is that if any of the 20 factors below the line are correlated with, with the 21 factors above the line, then our estimate of the effects of 22 the factors above the line will be biased. 23 Q So -- 24 A And if they're biased, the predictions, the predicted 25 probabilities will be potentially biased as well. 37 1 Q I think we'll come back to that as to how you dealt 2 with that, is that right? 3 A Yes. 4 Q All right. But before we go to look at the results, 5 let me just ask you a couple questions about exactly what 6 you did. Did you, just as a general matter, did you use 7 any particular kind of, of analysis to undertake the 8 simulation? 9 A We did. We used -- the first method we used was 10 called, logistic regression. And I think we've had a 11 discussion of that. You have a binary outcome which is 12 admitted, yes or no, and then you have a number of what we 13 call explanatory variables, which are the ones here above 14 the line. And you are able to estimate an equation that, 15 that estimates the relative weights of these factors on the 16 probability, the log odds of admission, and ultimately we 17 can translate like that into the probability of admission. 18 Q So - 19 A We've talked about that in court. And I assume we 20 don't need to necessarily say much more about it. I think 21 Professor Larntz explained what that was. 22 Q And so Dr. Larntz also used logistic regression, of 23 course, as part of his analysis? 24 A Yes. 25 Q And we'll get back to Dr. Larntz' regression models. 38 1 But are there general things that you can say about how 2 your regression analysis differed from, in addition to the 3 conception from what Dr. Larntz did? 4 A We, yes. We actually estimated our models separately 5 for minority and majority applicants. And the reason we 6 did that was that we found that the association between 7 minority status and admissions was strongly dependent on 8 grades and test scores; that is, we found that, for 9 example, applicants who had very high grades and test 10 scores, for those applicants minority status has a very 11 small effect, or very small association. And for 12 applicants in other cells the association is considerably 13 stronger. So because the association between minority 14 status and these factors varied, what statisticians then do 15 is, they say we can't estimate one model for everybody, we 16 then do the models separately. 17 Q Did you exclude any of the applicants for which you 18 had data from your analysis? 19 A No. We used -- oh, I should say, we did exclude 20 people, a very small number of people have have no grades. 21 There's just, they don't have grades in the database. It's 22 a tiny fraction, or they don't have LSATs, so those people 23 we excluded. But we excluded no cases based on their 24 outcomes. 25 And this is a very important point. When you 39 1 start excluding cases from an analysis based on the outcome 2 of the admissions decision, you get into some significant 3 biases and we did not do that. 4 (Whereupon an off-the-record 5 discussion was had.) 6 Q All right. So with the simulation model or regression 7 model, how did you conduct your simulation? 8 A So what we did was we actually, for each year, we did 9 the analysis I mentioned, we did it separately for majority 10 and minority applicants. We actually used the majority 11 equation in predicting the probabilities of admission under 12 the race-blind policy. We assumed that under the so-called 13 race-blind policy that the majority equation, which has 14 more cases involved in the estimation would be more like, I 15 mean, the average equation would be more like that. So we 16 used that equation. 17 Q Okay. And with that equation, what did you do? 18 A Well, based on that equasion we could compute the 19 predicted probability of admission under policy B for any 20 applicant, and then we could combine those within ethnic 21 groups to predict the average probability of admission for 22 any sub-group of applicants in this case, as a function of 23 ethnicity. 24 Q And so from that you can estimate how, what the 25 percentages admitted would look like? 40 1 A Exactly. From that we're able to compute the average 2 probability of admissions for ethnic, for minority and 3 majority applicants, and compare it to the observed 4 probability of admission under the current policy. 5 Q All right. I'm going to ask just one other thing 6 about the simulations. Are you able to, are you able to 7 say, based on the simulation, what would happen to any 8 particular applicant under the alternative policy? 9 A No, we're not. And this is one of the ironies of 10 causal inference and causal modeling. For any person, 11 we'll never know the two probabilities. In order to do 12 that -- we can't even imagine how to do it. We'd have to 13 have both policies in operation and we'd have to have them 14 implied under both policies and see all the results. But 15 we can't do that. And that's generally true in causal 16 inference. We can't compute the causal effect for any 17 specific case. What we can compute is called the average 18 causal effect. In this case, it would be the average 19 probability of admission under policy A, minus the average 20 under policy B for sub-groups of applicants. 21 Q Now, let's look at, if we could what happened in your 22 simulations. I think the next Exhibit is 187 in the same 23 category. 24 A Now, mind you -- 25 Q Yeah, why don't you first tell us what the columns 41 1 are. 2 A Right. 3 Q And then -- 4 A Yeah. 5 Q Explain what the results are? 6 A Let me just preface it by saying that the results of 7 policy B are going to be those based on the model I just 8 described, but we also replicated this analysis using 9 another, actually a couple of different regression models 10 we tried. But we also used another method, which we can 11 describe a little bit later. But under the method that I 12 just described -- 13 Q Can I, let me just ask you -- 14 A Yeah. 15 Q Are the results under the other methods substantially 16 different? 17 A They're not substantially different. They're somewhat 18 different but in the main, they're very, very similar. 19 Q All right. So why don't you explain what you have on 20 the chart and then what the results showed. 21 A Okay. What we have on the chart are two columns, 22 policy A, again, that's the current policy; policy B, this 23 is the so-called race-blind policy that I mentioned. 24 Q And just so we're clear, the number in policy A is the 25 actual observed data? 42 1 A Right. And so we have for minority and non-minority 2 applicants, and for each year, the predicted -- well, in 3 this case under policy A, the actual observed average 4 probability of admissions. And then under policy B, the 5 average probability of admission for that same group. 6 A So again looking at 2000, we've been looking at 2000. 7 The average probability of admission in 2000 for minority 8 applicants was .35. We project that under policy B the 9 average probability of admissions would be .10, which is, 10 which is quite a large difference. And that type of result 11 occurs in each year. They're pretty similar. There's some 12 exceptions. 13 It turns out that 1995 is a bit extreme in terms 14 of the change in the probabilities for the minority group. 15 But, but it follows the same pattern. It's, and the other 16 years are very similar to, to the year 2000. So we see 17 then in some, a quite sharp reduction in the average 18 probability of admission of the minority applicants under 19 policy A and policy B. 20 A Now, if we move down to the bottom panel, we have the 21 results for the non-minority applicants under each year. 22 So, again let's just take a look at, for illustration of 23 the year 2000 under policy A the average observed, average 24 probability of admission was .40, 40 percent of those who 25 applied were admitted. We project that under policy B, 43 1 this is a race-blind policy, that would increase. It would 2 increase from .40 to .44. So it would be rather marked, 3 small or marginal increase in the average probability of 4 admission, .40 to .44. 5 Q And are the results similar for the other years? 6 A And the results are very similar for other years. It 7 tends to be, .99 goes 41 to 45, again the difference being. 8 .04. In some cases it's .05. I think the actual biggest 9 one we see is in '95, not surprisingly, which is .06, .28 10 up to.34. 11 Q Now, why is it, Professor Raudenbush, that the change 12 in the average probability of admission is fairly large for 13 the minority students and fairly small for the non-minority 14 students? 15 A It's a very straight-forward result of the difference 16 in the sizes of the applicant pools. There are relatively 17 few minority applicants, a small -- a large change in the 18 probability, a large reduction in the probability of 19 admission of those candidates translates into a very small 20 increase in the probability of admission of the majority 21 group, because it has so many more applications; basically, 22 any extra, sort of admission seats, if you will, or admits, 23 could become available by reducing this probability, will 24 be competed for by a large number of people. 25 Q Now, as Judge Friedman alluded to earlier. 44 1 A Right. 2 Q These simulation results are based on regression 3 models which involve assumptions, correct? 4 A Right. 5 Q How can you be confident, given those assumptions 6 about these results here? 7 A Right. Well, the first thing we did, as I mentioned, 8 was we did use an alternative method to do the simulation, 9 and as you asked me were the results similar and the answer 10 was, yes, they were very similar. 11 Q So the fact that you got similar results says what 12 about these? 13 A From an approach that did not use logistic regression 14 at all, and I'll explain exactly what we did a little bit 15 later. But the most important way that we can bound our 16 error, if you will, is much more straight forward and 17 requires an absolute minimum of assumptions. And I think I 18 can maybe demonstrate that with a different exhibit. 19 Q All right. Why don't we go. 20 THE COURT: Let me ask you one question? 21 A Sure. 22 THE COURT: You can also conclude from that chart 23 that by having a race-blind policy that, looking at 2000, 24 for example, that there's a 25, obviously a 25 percent 25 difference, so that. 45 1 A Right. 2 THE COURT: That's right. So you could also say, 3 could you not, that the effect is, the effect having a 4 policy that's not race blind is about 25 percent? 5 A A difference in probabilities of .25, yes, right. And 6 people do this in different ways. We talk about odds, 7 ratios of probability. Sometimes differences in 8 probabilities are the most straight-forward way of 9 interpreting the results. It depends on the situation. 10 Q Let me ask a related question. 11 THE COURT: Well, go on. 12 MR. DELERY: Please. 13 THE COURT: You ask, I'll get mine later. He may 14 answer. If he doesn't. 15 Q Before we look at the bounding point, in your view, do 16 these numbers here, the results of your simulation analyses 17 say anything about the extent to which race is considered 18 by admissions officers in making their decisions? 19 A They don't. 20 Q And why is that? 21 A Let me explain what could generate this difference in 22 probabilities. If you have a large, a much larger 23 applicant pool than can be admitted, so you have many more 24 people apply than you can accept; and if grades and test 25 scores are very important, play an extremely important role 46 1 in the admissions decision, then a very small difference 2 between two groups can lead to a large difference in the 3 probability of being admitted. 4 And so under this, under our simulation of the 5 race-blind policy, grades and test scores are playing a 6 very important, and extremely important role because we 7 don't have any other data, basically. We know that there 8 are many more applicants than there are seats. And we know 9 that there's a small difference between minority and 10 non-minority applicants. And that explains why this 11 difference turns out to be big. 12 Q So. 13 A And it doesn't. 14 THE COURT: Turns out to be big? 15 A Big, yes, these numbers are quite different. That 16 doesn't depend on how heavily the admissions officer weigh 17 race. It's a function of the fact that you're heavily 18 weighing a factor on which two groups have a different 19 mean. 20 THE COURT: A different what? 21 A A different mean, a different average. 22 THE COURT: Mean. 23 A Right. 24 Q Just so I have a sense of the terminology here, is it 25 your view that there's a difference between measuring the 47 1 effect or impact of the policy on the one hand? 2 A Right. 3 Q And the extent to which a particular factor is 4 considered in an admissions process on the other? 5 A There's a great deal of difference. And I might add, 6 especially in this case, the causal impact of the policy is 7 much more excessible to statistical investigation than is 8 an attempt to discern how people who are making decisions 9 about admissions are weighing one of many factors, when we 10 don't have any information about most of the factors. It's 11 just a very difficult thing to do, statistically. We 12 basically can't do it. 13 So, but we can assess the impact of what they do. 14 We don't know why it has that impact. You see, there's a 15 big difference between finding a causal effect and 16 explaining the causal effect, knowing why it happens. 17 There are lots of things in social science, 18 medical science, where we know there's an impact on 19 something, but there's so many possible explanations. And 20 we don't have the information to explain the explanation. 21 So this analysis can be conducted with a minimum of 22 assumptions and with a considerable amount of confidence, 23 whereas the more, the much more challenging task of trying 24 to use statistical information to discern how people who 25 have much more information than we do, how they think. 48 1 This is much more difficult. 2 Q I think we'll come back to this question of extent a 3 little bit with some additional illustrations, but let's 4 return to the bounding point? 5 A Right. 6 Q That you were on, if we could. And I think the next 7 exhibit is 188. 8 MS. MASSIE: Judge Friedman, I don't know if this 9 is, if we could take a quick break, that would be great. 10 THE COURT: Of course, how much do you want? 11 MS. MASSIE: Five minutes. 12 THE COURT: Okay. We'll take a five-minute break. 13 (Whereupon an off-the-record 14 discussion was had.) 15 THE COURT: Okay. You may be seated. Thank you. 16 MR. DELERY: Thank you, Your Honor. 17 Q Professor Raudenbush, I believe we had been talking 18 about the simulation results for the minority students on 19 the one hand and the non-minority students on the other 20 hand and the bounding issue that you? 21 A Yes. Just to recreate where we were, the key result 22 here was that the effect of going from policy A to policy B 23 was quite big for the minority students. Like in 19, in 24 2000 it was 25 percentage points, whereas the effect going 25 from policy A to policy B on the non-minority students was 49 1 quite small. 2 So in 2000, going from forty, .40 to .44, so going 3 up on four percentage points. So that's where we were, and 4 the question is the problems with this model. 5 As we discussed, policy A, policy B is based on a 6 simulation. It's based on a model. The model has to make 7 assumptions. The assumptions, not might be, but probably 8 are wrong, and so how far off might we be, as a result of 9 failure of those assumptions, and that was our next step. 10 Q Okay. And here, are you talking about the assumptions 11 that the factors not in the model are unrelated to the 12 factors in the model? 13 A Correct. 14 Q Did Dr. Larntz' model include the same assumptions? 15 A Yes. 16 Q Well, why don't you move to the next chart, actually, 17 and tell us what you did. The next chart will be 188. 18 Tell us what you did to evaluate how reasonable your 19 results were, in light of the assumptions. 20 A What we did was we used an idea that has a fancy name 21 but it's a real simple idea. The fancy name is, these are 22 non-parametric upper and lower bounds on causal effects. 23 The simple idea is how, how small could the effect be and 24 how big could it logically be. And here's how simple it 25 really is. 50 1 Again, let's just focus on 2000. And we're 2 looking at majority students here. And we see that in 2000 3 40 percent of them were admitted. How small could the 4 effect be of going to policy B? Well, logically it seems 5 that the smallest the effect could be would be there 6 probability would stay the same. 7 In other words, we go to a race-blind policy and 8 there's no impact. It goes from .40 to .40. It logically, 9 it logically can't really go down. It's hard to imagine 10 how eliminating race as a factor would make things worse 11 for, for majority students. So .40 is the lower bound for 12 the effect. So zero percentage points, .40 to .40. The 13 upper bound is, is constructed, again, very simply; how big 14 could the effect be. The biggest it possibly could be 15 would be if every minority students were rejected under 16 policy B. If you eliminate race as a factor and every 17 single minority students were rejected, then that means 18 that's the biggest effect it could be. 19 And under that scenario, the upper bound is .46, 20 so that means the difference between the lower bound and 21 the upper bound is .06. That's six percentage points. Our 22 estimate, based on our simulation is .04. It's kind of in 23 between the lower bound and the upper bound. So our .44 is 24 undoubtedly wrong, to some degree, but to what degree can 25 it be wrong, the upper and lower bound tell us, it can't 51 1 be -- the lower bound is a .04 error, the upper bound is a 2 .02 error and those bounds don't require me to make any 3 assumptions about what's in the model, what's not in the 4 model. Those are logical upper and lower bounds. 5 Q So based on the bounds that you found and, as compared 6 to the simulation results, do the bounds give you 7 confidence in, in your models and in your analysis? 8 A They give us confidence in the causal effect of the 9 policy change on the majority students. 10 Q And that's what this chart shows? 11 A That's what this chart shows. Now, I should add that 12 the bounds on the causal effect for the minority students 13 are wider because like when, I think in 19 -- in 2000 we 14 went from, I think it was something like .34 to ten. The 15 extreme bound would be to zero. So from .34 to zero. So 16 they were a little bit wider. 17 There's a little more uncertainty as to how the 18 switch in policy would effect the minority students. But 19 there's a great deal more -- I should say a great deal less 20 uncertainty about how the change in policy would effect the 21 majority students. 22 Q Did Dr. Larntz do any kind of similar bounding 23 analysis on the results of his regression model? 24 A I didn't see any evidence of it in the reports. And I 25 didn't hear him put an upper and lower bounds or a 52 1 confidence interval on the odds ratios. 2 A By the way, a confidence interval is a weaker bound, 3 much weaker than a non-parametric up upper and lower bound 4 because this bound has virtually no assumptions. The only 5 real assumption I'm making is that going from policy A to 6 policy B wouldn't hurt the majority students, and that seem 7 indisputable. 8 Q So these results tell us what the expected 9 probabilities of admission are for, on this chart, the 10 majority students and on the earlier chart also, the 11 minority students? 12 A Correct. 13 Q Did you take that analysis any further? 14 A Yes, I did. Once we have predicted probabilities of 15 admission or average probabilities of admission for 16 sub-groups, we can then develop a picture of what the 17 composition of the first-year class would look like under 18 policy B. Of course we already know the composition of the 19 class under policy A. It's what we observed. 20 And to do this is really very straight forward. 21 We take the probabilities of admission under policy B. We 22 multiple that by the yield which is what fraction of people 23 who were admitted decided to come to Michigan, the one that 24 was actually observed. And that can then give us the 25 expected number of people in each, of each group for each 53 1 year. 2 Q Okay. I think we have a chart showing those results. 3 A Yes. 4 Q It's Exhibit 129. Just so I'm clear about your last 5 point, Professor Raudenbush, you're assuming in this part 6 of the analysis that the yield rate would not change? 7 A That's correct. 8 Q If the university moved to a race-blind admissions 9 policy? 10 A Exactly. We're, it could arguably go down if this 11 change were made, in which case our results would 12 understate the impact on diversity. 13 We're also assuming, as years go by, that the size 14 of the minority applicant pool would not be effected by a 15 sharp reduction in the probability of admission, which is, 16 which is another conservative assumption. It seems 17 reasonable that if the probability of admission goes down, 18 the number of people who would take the time and effort and 19 pay the price of climb might well go down, but we didn't 20 assume that that would happen. 21 Q Why don't you look at this chart, Exhibit 189, and 22 tell us what it shows about this next step of your 23 simulation analysis? 24 A Okay. Again, it's divided. As we go down the, down 25 the rows, we see the years. We have under policy A and 54 1 under policy B and in each case what's in here is the is 2 the composition of the class. So for policy A it's going 3 to be the actual composition that happened in that year. 4 Under policy B, it's what we would predict, based on the 5 simulation. 6 And again, why don't we just, for illustration, 7 stick with 2000. Under the current policy, 170 minority 8 students were admitted and based on the yield, 58 actually 9 attended. And that was, that turned out to be 14.5 percent 10 of the class. 11 Q Those numbers were taken from the first chart that we 12 saw today? 13 A That's right. Those are just the actual observed 14 numbers. Under policy B, we, we would predict that only 46 15 minority students would be admitted. And then applying the 16 yield, that would lead to 16 attending. So only 16 17 minority students, from 58 down to 16, and then that would 18 be four percent of the class, so our, our analysis would, 19 would predict a reduction in the fraction of students who 20 are minority from 14.5 percent to 4.0 owe percent. 21 Q So what, if anything, do these results, I guess I 22 should back up and ask, is 2000 unusual in this respect, 23 or? 24 A The basic pattern of 2000 appears each year. We see 25 very similar results. Again, there's a little more extreme 55 1 result in 1995, but it's basically in the same direction, 2 same pattern, and the other years are very similar. 3 Q So what did these results tell you, if anything, about 4 the expected diversity of the law school class under a 5 race-blind admissions system? 6 A Right. So we did see that under this simulation, that 7 the overall composition of the class, which, in 2000 was 8 14.5 percent minority, would be very substantially less 9 diverse with only four percent of the students being from 10 minority background. 11 Q I think you indicated that there would be somewhat 12 over a hundred fewer minority students admitted, your model 13 predicts, under the alternative race-blind policy? 14 A Right. 15 Q What would happen to the spaces in the class that, I 16 guess, those students had accounted for under the current 17 policy. 18 A Right. Well, -- 19 MR. KOLBO: Object to the form, basis, Your Honor. 20 THE COURT: I think it's a pretty obvious answer, 21 but why don't you rephrase it. 22 MR. DELERY: Okay. I'll rephrase it. 23 Q Can you tell us anything about what the model predicts 24 about where the hundred-plus spaces that had been under the 25 current policy given to admitted minority students? What 56 1 would happen to those spaces under your alternative 2 simulation? 3 A Right. Under our alternative simulation, those places 4 which look to be approximately 134 places would be competed 5 for by all of the non-minority students; that is, 6 approximately three, 2,800, whatever the number was, of 7 students that would compete for those places. That's the 8 way we've constructed the simulation. 9 Q Okay. Now, using these numbers, the predicted 10 composition of the law school class as a whole under your 11 alternative policy, did you do anything to look at how that 12 would translate into the more day-to-day activities of the 13 law school? 14 A Yes, I did. And I believe we have an exhibit that 15 displays that. Essentially, what we -- 16 Q Why don't we put the exhibit up, if we could. 17 A What we did, while that's being put up -- 18 Q -- This is 190, by the way. 19 A People at the law school supplied me with a list of 20 some of the important contexts for learning that arise at a 21 law school. They're listed here and they range in size. 22 The first-year section is the biggest one, 85 students are 23 in the first-year section where students take many of 24 their, several of their required classes. The smallest is 25 a moot court team which is just pairs of people in a moot 57 1 court, and, and there are other contexts. Each one has a 2 size. And what we did next was to ask questions about the 3 likely composition of these contexts for learning under 4 policy A, which is the current policy again; and policy B. 5 And that's essentially what we did. And I think we have an 6 exhibit that displays the results. 7 Q Okay. In your view, these, these contexts were 8 representative? 9 A I was told by the people that supplied these, actually 10 through your office, that these were the representative 11 contexts. And they cover the range of sizes of various 12 contexts. And what's really important from the point of 13 view of statistics here is the size of the context and how 14 does that then look, in terms of its ethnic and 15 composition. 16 Q Why don't we put up the next chart, if we could. 17 That's 191. What does this chart represent, Professor 18 Raudenbush? 19 A Okay. So what we've been done is asked questions 20 about the expected composition of each learning context, 21 from the standpoint of a majority student and from the 22 standpoint of, we just picked African-American students We 23 wanted to have a definite type of person, rather than a 24 minority student in mind when we thought about this. And 25 we didn't do it for all of the contexts. 58 1 We picked three represent -- three that were sort 2 of across the range of sizes. We picked the first-year 3 section, which has 85, then the second row is the half 4 section. And then the residential dormitory entryway. 5 This is an entryway of a dormitory and approximately 25 6 students would be in that entryway. 7 Q And the results for the other contexts are reflected 8 in your report? 9 A They're in my report, right. And I think you, this 10 basically captures what's going on here. I don't think 11 it's necessary to go through all these numbers. I might 12 just pick one of them and kind of explain. The first-year 13 section, the biggest context, let's take it from the point 14 of view of the majority student. 15 What's the probability that that would be 16 segregated in the sense that that would be no minority 17 students under policy A and policy B. And the answer is 18 it's a very small like likelihood. Under either policy 19 it's unlikely that there would be no minority students. 20 It's actually .00 versus .03. 21 But then let's ask another question, well, what's 22 the probability that there would be at least, at least 23 three minority students. And it could be nearly certain, 24 which is, approximately, pushing toward 1.0 under policy A, 25 whereas under policy B that would only happen two thirds of 59 1 the time. There would be a one-third chance of not having 2 as many as three in that section. 3 And then for, what's the probability that there 4 would be, at least three African-American students and at 5 least three Hispanic students in that group of 85. Under 6 policy A it's almost certain to occur. Under policy B, 7 approximately one time out of four. So it's actually not 8 likely to have that agree of diversity. That's the biggest 9 section. The effects of the policy are more pronounced 10 when we go to smaller-size sections. 11 For example, for example, just take, take the, the 12 residential dormitory, what's the probability of having at 13 least three minority students, .75 in that residential 14 dormitory, to picture, 25 people who live in the dormitory, 15 .75 probability that at least three of those people would 16 be minority under policy A. Under policy B, .08, a very 17 unlikely matter. So that kind of demonstrates what's going 18 on from the point of view of the majority student. 19 Things are a little bit different from the point 20 of view of an African-Americans student because, you know, 21 the African-American has to be in the context before we can 22 ask what's happening. So given that there is an 23 African-American, we ask questions, the following 24 questions; what's the probability that you'd be the only 25 African-American student in that context, or, you know 60 1 what's the probability of three or more of those. 2 So just, we could say, again, take, take the 3 residential dormitory example, under policy A, that's the 4 current policy -- there's a pretty small chance that you'd 5 be the only African-American student, .18, in this 6 residential dormitory. Under policy B, .69, it's very 7 likely that you would be the only African-Americans student 8 in the dormitory. And the probability of at least three, 9 at least two other African-American students would be, 10 would be relatively high under policy A, .56, at least 11 better than half, and very low, .07, under policy B. 12 So I think this gives some flavor of our 13 expectations about what would happen to the diversity of 14 certain contexts for learning under a change in policies. 15 Q All right. Now, taking all of these simulation 16 analyses together, the overall picture that you've 17 presented here this morning, what conclusions, if any, do 18 you draw about the impact of using race in law school 19 admissions at the university? 20 A I draw several conclusions. The first is that the 21 impact on the probability of admission of minority 22 candidates would be quite substantial. There would be 23 quite a sharp reduction in the probability of admission. 24 The second conclusion would be that the impact on majority 25 applicants would be modest, by comparison. There would be 61 1 a small increase in the average probability of admission 2 for majority candidates. And about that conclusion, I feel 3 considerable confidence. 4 Q And again, why do you think there is that difference? 5 A And the reason that that's, that difference occurs, 6 that is, you know, why does it effect minority students 7 more than majority students, it's simply a result of the 8 smaller pool of applicants of the underrepresented minority 9 group than of the majority group. 10 Q Now, so by giving these views and these estimates of 11 the impact of considering race and admissions, are you 12 saying anything about the extent to which the race of an 13 applicant is considered by the admissions people? 14 A No. We're not making any inferences about how heavily 15 this is being weighed by the people who are making the 16 admissions decisions. We don't have information about that 17 question. But we do have information about the impact. 18 Q And are these impacts, estimates, telling anything 19 about the relative weights of any of the factors in the 20 admissions process? 21 A No. They're not quantifying the relative weights of 22 anything in the process. 23 Q Okay. So as I think you indicated before, this 24 simulation results, simulation analysis, I should say, is a 25 different approach from the approach that Professor Larntz 62 1 took? 2 A Correct. 3 Q Is that your view? 4 A Correct. 5 Q Is that your view? 6 A That's right. 7 Q How does your simulation analysis bear on an 8 evaluation of Dr. Larntz' work? 9 A Well, I think that the simulation analysis gives a 10 framework of a policy framework. We've looking at policy 11 options faced by the law school that we can use to 12 understand the reasonableness of some of the results 13 results of Professor Larntz' work. 14 Q And in your opinion does Dr. Larntz' work provide an 15 accurate or realistic picture of the role that race plays 16 in law school admissions? 17 A And of course the answer is, no. As I stated at the 18 outset, Professor Larntz attempted to construct a 19 statistical model that could tell us the extent to which 20 race played a role. And I don't believe that we have 21 information that can enable us to do that. 22 Q And on its own terms, do you believe that Dr. Larntz' 23 approach was appropriately executed? 24 A Well, I believe that certain key methodological 25 choices that Dr. Larntz made led to a, an exaggerated 63 1 impression about the association between minority status and 2 admissions. 3 Q And what were those? 4 A Well, they're essentially -- 5 Q Just briefly and then we'll get into them a little 6 more? 7 A I'll give you three types, and I know we'll talk about 8 some of the details. 9 The first was that his analysis selectively 10 attended to the data; that is, it discarded data based on 11 the outcomes of the admission process. And it discarded 12 data that was, in fact, discrepant with the hypothesis that 13 there is a strong correlation between race and admissions. 14 That was the first. 15 The second was that his analysis was based on 16 strong assumptions, as our policy via regression, as I 17 explained the same kinds of assumptions that we had. 18 And that in one important case, I did an analysis 19 that showed that a key assumption that he made and was an 20 important one, was not true. And in the second case, the 21 other, another key assumption is like what I described 22 before. It's probably not true, almost certainly not true, 23 the problem being we don't know the impact. We can't gauge 24 the impact of the falsehood of the assumption on the 25 validity of the results. 64 1 And thirdly, the results of his analysis were 2 extremely unstable. They were very different from year to 3 year, and the size of the differences from year to year 4 really can't be explained by the process, or by the data at 5 hand. And so my conclusion is that there are aspects of 6 the methodological approach that create the instability, 7 not the admissions policy or the data. 8 Q Before we talk about those problems that you found 9 with Dr. Larntz' work in more detail, I'm wondering if you 10 could give us a sense of, of how his overall approach, his 11 conceptual framework differed from your's? 12 A Right. Well, his, his conceptual framework was, 13 again, the idea of constructing a model that would tell us 14 about the role of admissions, the extent to which they're 15 taken into account by the admissions people, which I view 16 as a very challenging thing. You have to have tremendous 17 amount of information to assess peoples' thinking and the 18 extent to which they're weighing factors. My question is 19 actually a more limited one but one that I think we can 20 approach with minimal assumptions through statistical 21 inference and still get some very useful information. It 22 doesn't tell you, it doesn't give us the answer to that 23 question, but it gives us extremely important information 24 about the impact of taking race into account. 25 Q Now, obviously you were here the other day when Dr. 65 1 Larntz testified, and there was a lot of discussion about 2 odds ratios, yes. 3 A Right. 4 Q Obviously we all remember that. 5 A Right. I'm just glad I don't have to explain what 6 they are. 7 Q Is, well, I'm going to ask you to give some examples 8 in a second. 9 A Okay. I couldn't get out of that one. 10 Q No such luck. I guess my first question, though, 11 about this is, is computing odds ratios an accepted method 12 of statistical analysis? 13 A It is. It's widely accepted. It's widely used. 14 Q And in what context is it appropriately used? 15 A Well, the thing about odds ratios is that typically an 16 odds ratio by itself doesn't tell us what we need to know. 17 It's a piece of information. But to interpret the meaning 18 of the odds ratios, we, odds ratios, we really need to know 19 something about the probabilities that went into computing 20 the odds ratio because depending on what the probability, 21 you know, an odds ratio controls a function of the 22 probabilities for each group. And depending on what those 23 two probabilities are, the odds ratio could be very, very 24 different things. So my, my general rule of thumb is to 25 always keep in mind the probabilities as well as the odds 66 1 ratios, for that reason. 2 Q And you have used odds ratios in your work? 3 A Oh, yes. 4 Q Is that right? 5 A Yes, I have. 6 Q Okay. In your opinion, do odds ratios provide an 7 accurate or appropriate way to look at the role that ratios 8 make in the law school admissions process? 9 A There's some problems with using, there's some huge 10 problems with using them alone, again, without, without 11 accompanying them with other information. Generally what 12 happens to the odds ratio is that it becomes very unstable 13 when one group or the other has a probability or, of either 14 nearly one or nearly zero. 15 Q Do you have some illustrations of that effect? 16 A Well, I thought we might actually just revisit some of 17 the odds ratios we looked at. Was that, the day before 18 yesterday I think it was, right. The day before yesterday. 19 And maybe we could even just quickly review those. I don't 20 know if we still have those charts or if we need to 21 scribble down those things again. 22 Q I think we do. I think the page that we have before 23 is gone. 24 A May I. 25 MR. DELERY: I'll move the easel out a little bit 67 1 here. 2 A Thank you. I think what we had the other day was we 3 had a group, some group. Let's call this group one, that 4 had a probability of admission of .99. And then we had 5 group two that had a probability of admission of .90, and 6 the odds ratio turned out to be eleven. 7 So, basically, this was saying group one had 8 eleven times the odds of admission of group two. And then 9 we had another example where group one had a probability of 10 admission of .999. Group two still had a probability of 11 .90. And what happened to the odds ratio was that it 12 became 111. And then just, you can see the pattern here. 13 If group one had a probability of admission of .9999 and 14 group two system had a probability of admission of .91, the 15 odds ratio went to 1,111. Now, those are, those are facts. 16 There's no problem with that. 17 The only problem is, if all we saw, if I hid these 18 probabilities, and all I saw were the odds ratios, I might 19 get the impression that those are three extremely different 20 results. Eleven times the odds, 111 times the odds, 1,111 21 times the odds. These look so different. But when I look 22 at the probabilities of admission from a practical point of 23 view, if I'm a candidate, and my probability is .99 versus 24 .90, that's about ten percentage points. 25 And I'm nearly certain to be admitted. If I go up 68 1 to .999 versus .91 it's still about ten percentage points. 2 I'm still nearly certain, but yet my odds ratio went up by 3 ten, a factor of ten. And then another factor of ten as we 4 go to .999. So all I'm saying is the odds ratio by itself 5 can create a misleading impression if you don't also see 6 these numbers. 7 Q Is there something about the mathematical 8 characteristic of the odds ratio that causes this, I mean, 9 is that the reason? 10 A The basic problem is that an odds ratio requires 11 division. And if one of the probabilities is either near 12 one or near zero, we encounter something called division by 13 zero which is prohibited, mathematically. We can't have a 14 fraction that has zero and nine. 15 Q And so what's the results of that? 16 A And so as the denominator goes towards zero, the 17 fraction increases without bound to incredibly large 18 numbers. If we keep adding nines, this thing keeps going 19 up and up and up. 20 Q And does the same pattern happen when you're talking 21 about small probabilities at the other end? 22 A Exactly the same pattern happens, so, for example, if, 23 I just switch it around. If group one had a probability of 24 admission of .10, and group two had a probability of .10, 25 the odds ratio would be eleven. 69 1 If I went from, again, group one, .0 to group two 2 .001, 111, .10 to .0001, 1,000, 111. So again, group one, 3 ten percent chance of getting in, group two, very small 4 .10, very small, very small. Ten percentage point 5 difference leads to very, very different odds ratios. 6 Q Do you have an example of a situation in which two 7 people might have similar probabilities of something 8 happening, but very different odds or a real world example? 9 A Yes, actually, I did think of one. It actually 10 involved the lottery. Suppose that, you know -- I get 11 excited about the lottery and I buy a lottery ticket. And 12 you say, well I'm going to outdo you, I'm going to buy 13 fifty lottery tickets. 14 So what would happen is your odds would be roughly 15 fifty times, mine. But yet both of us would have near zero 16 probability of winning the lottery. I mean, it's wise, 17 you'd say, I'm going to be really smart and go buy 18 thousands of tickets to the lottery. Everybody would be 19 buying. Of course they are, but. 20 THE COURT: Actually this week it's fifty-nine 21 million. There's a sign on my way home. Every time I keep 22 looking. 23 A They're doing it. They're rapidly increasing their 24 odds, but what they don't know is their probability is 25 staying right almost exactly at zero. 70 1 Q All right. Okay. If you could take the stand. 2 Professor Raudenbush, in your view does this pattern that 3 you've just described to us examples have any relevance to 4 the data we have in this case? 5 A They do. There are combinations of grade point 6 average and LSAT where the probability of admission of 7 anyone who applies to the law school is extremely high. I 8 mean, people who have near A averages who are up in the 9 upper 160's or 170's on their LSAT have an extremely high 10 probability of admission. 11 Of course in the data what we see is that the 12 proportions are something like 1.0 for minority applicants, 13 and something in the .9 range, or in a very high range for 14 majority applicants. And so in that sense, the examples 15 that I was presenting were not unusual. And something 16 similar can also, and does appear at the lower end of 17 people who have fewer qualifications where the differences 18 may be small in probability terms, but the odds ratios may 19 be big. 20 Q With that background in mind, I'd like to ask some 21 questions about the cell-by-cell analysis, that Dr. Larntz 22 conducted. 23 A Okay. 24 Q Just, let's start with a general question. What's 25 your opinion about the of the appropriateness or the 71 1 validity of that approach? 2 A Well the problem, well, one of the problems with that 3 approach is that it requires that an odds ratio be 4 computable for every single one of the hundred plus cells 5 that appear in any year in Professor Larntz' reports. And 6 since the odds ratio is not computable in a number of 7 cases, what this leads to is a discarding of data in those 8 cases where there can't, where no odds ratios is 9 computable. And this ends up discarding considerable 10 evidence that are relevant to how the university is 11 handling the admissions decisions. 12 Q And I believe we had some examples? 13 A Yes. 14 Q Of those situations? 15 A We do. 16 Q I think this is Exhibit 192. If you'd put that up. 17 Maybe, David, if you could put the easel back where it was. 18 Can you read it from there? 19 A Yeah, I can see the numbers from there. 20 Q Okay. There's very small, actually. 21 A I'll -- 22 Q I think I need to come closer. 23 A Okay. All right. 24 MR. DELERY: If that's all right, Your Honor. 25 THE COURT: Of course. 72 1 Q So I guess let me first just ask, I take it this page 2 here on the left is a little image of a page from one of 3 Dr. Larntz' reports? 4 A Yes. That's page six of six from the March 20, 2000 5 report. And we selected that page. It was just convenient 6 because it had three examples that I wanted to say 7 something about, because it has a bearing on what we're 8 discussing, and they all came from the same page. 9 And the first example, actually, the first two 10 examples involve cases in which the admissions process 11 treated people the same, in terms, they had the same 12 admission decision regardless of minority status. So in 13 the first cases, and we're looking here at, at students who 14 have relatively low grade point average. It's down 2.25 to 15 2.49, but relatively high LSAT's, 161 to 163. There was 16 one minority applicant in that, in, who had those 17 characteristics. And that person was rejected. There were 18 two majority applicants and they were rejected because both 19 people were rejected. Of course, what we know is they both 20 had the same admissions decision. There was no different 21 decision for the minority and majority applicants. But 22 because none of them were admitted, we can't compute the 23 odds ratio. So if you've developed a statistical approach 24 that requires cell-by-cell computation of odds ratios, you 25 can't compute the odds ratio. 73 1 Basically what happens is you have to discard 2 their cell. But when you discard this cell, you're 3 discarding information that's relevant to the decision of, 4 it's relevant to the decision made by the admissions 5 committee. That is, essentially, you're waiting to see 6 what the admissions committee decides. 7 And if they make it a certain decision, which in 8 this case is treating everybody the same by rejecting them, 9 discard the data. If the admissions decision had been 10 different, if, if someone had been admitted, then the cell, 11 the data would have gone into the analysis. So that means 12