1 1 UNITED STATES OF AMERICA 2 IN THE UNITED STATES DISTRICT COURT 3 FOR THE EASTERN DISTRICT OF MICHIGAN 4 SOUTHERN DIVISION 5 BARBARA GRUTTER, 6 For herself and all others 7 Similarly situated, 8 Plaintiff. 9 -vs- Case Number: 10 97-CV-75928 11 LEE BOLLINGER, JEFFREY LEHMAN, 12 DENNIS SHIELDS, and REGENTS OF 13 THE UNIVERSITY OF MICHIGAN, 14 Defendants, 15 -and- 16 KIMBERLY JAMES, et. al., 17 Intervening Defendants. 18 ______________________________________/ VOLUME IV 19 BENCH TRIAL BEFORE THE HONORABLE BERNARD A. FRIEDMAN 20 United States District Judge 238 U.S. Courthouse & Federal Building 21 231 Lafayette Boulevard West Detrot, Michigan 48226 22 Friday, January 19, 2001 23 APPEARANCES: 24 FOR PLAINTIFF: Kirk O. Kolbo, Esq. 25 R. Lawrence Purdy, Esq. 2 1 APPEARANCES (CONTINUING) 2 FOR DEFENDANTS: John Payton, Esq. 3 Craig Goldblatt, Esq. 4 Stuart Delery, Esq. 5 On behalf of the Defendants 6 Bollinger, et. al. 7 8 George B. Washington, Esq., 9 Miranda K.S. Massie, Esq. 10 On behalf of Intervening. 11 Defendants. 12 COURT REPORTER: JOAN L. MORGAN, CSR 13 Official Court Reporter. 14 15 Proceedings recorded by mechanical stenography. 16 Transcript produced by computer-assisted 17 transcription. 18 19 20 21 22 23 24 25 3 1 I N D E X 2 WITNESS PAGE 3 STEPHEN W. RAUDENBUSH 4 Direct Examination by Mr. Delery 5 5 Cross-Examination by Ms. Massie 118 6 Cross-Examination by Mr. Kolbo 121 7 Redirect Examination by Mr. Delery 160 8 DENNIS SHIELDS 9 Direct Examination by Mr. Payton 162 10 Cross-Examination by Mr. Purdy 193 11 Redirect Examination by Mr. Payton 215 12 Recross-Examination by Mr. Purdy 218 13 E X H I B I T S 14 NUMBER IDENTIFICATION ADMITTED 15 145 Expert Witness Report of S. Raudenbush 12 16 146-150 Supp. Expert Witness Rep. of S. Raudenbush 12 17 151 Raudenbush Curriculum Vitae 9 18 184-194 Charts of S. Raudenbush 108 19 5 Gospel According to Dennis 188 20 21 22 23 24 25 4 1 Detroit, Michigan 2 January 19, 2001 3 * * * 4 THE COURT: Good morning, everyone. On the 5 motions, I have nothing else on the agenda this case, why 6 don't we start the case and when we take it a break sometime 7 we'll argue those motions. 8 MS. MASSIE: That sounds great. 9 THE COURT: Is that good for everybody? I'm all 10 prepared, but I just don't want to waste your time this 11 morning. I know you have a witness. This is yours? 12 MR. DELERY: Yes. Good morning, Your Honor, 13 Stewart Delery, Your Honor, again for the university and 14 the individual defendants. 15 THE COURT: How are you. 16 MR. DELERY: If you're ready to proceed. 17 THE COURT: I'm ready. If you're ready, I'm 18 ready. We call. 19 MR. DELERY: We call as our next witness, Stephen 20 Raudenbush. 21 THE COURT: For evidence? 22 MR. DELERY: Thank you, Your Honor. 23 S T E P H E N W. R A U D E N B U S H. 24 was called as a witness and after having been 25 sworn was examined and testified as follows: 5 1 DIRECT EXAMINATION 2 BY MR. DELERY: 3 Q Could you please state your name and address for the 4 record. 5 A Stephen W. Raudenbush, 7 Harvard Place, Ann Arbor, 6 Michigan. 7 Q And where do you work? 8 A I work at the University of Michigan. 9 Q What's your job there? 10 A I'm a professor in the School of Education and the 11 Department of Statistics, and I also have a joint 12 appointment as a Senior Research Scientific at the Survey 13 Research Center. 14 Q How long have you been at the University of Michigan? 15 A I've been at Michigan since January 1 of 1998. 16 Q And where were you before that? 17 A For fourteen years before that I was at Michigan State 18 University. 19 Q Well, Professor Raudenbush, could you please, please 20 briefly describe your education, or educational background 21 for the Court. 22 A Sure. I received my bachelor's degree from Harvard 23 College in 1968 and my doctoral degree from Harvard 24 University in 1984. 25 Q Has your work at the University of Michigan and before 6 1 that at Michigan State focused on any particular areas? 2 A Yes, it has. It's, primarily my work is involved 3 applications of statistics in education, studying student 4 learning, studying student transitions into college, 5 studying how schools and classrooms effect academic 6 achievement. And also looking at other aspects of human 7 development. 8 Q Okay. And have you published in these fields? 9 A Yes, I have. 10 Q About how many publications have you had? 11 A Well, I guess if you count the second edition of our 12 book on Hierharchical Linear Models, if you count the 13 second edition of our book on Hierarchical Linear Models. 14 THE COURT: Do you want it spelled? 15 (Whereupon an off-the-record discussion was 16 had.) 17 A H-i-e-r-h-a-r-c-h-i-c-a-l. Okay. There would be, if 18 you count that one, there would be four books and quite a 19 large number of referee journal articles and book chapters 20 that I've published over the years. I'm not sure exactly 21 how many but I publish about four to six articles and 22 chapters a year. 23 Q Okay. This may be a relative question, but are any of 24 those publications particularly widely known? 25 A Well, the book I mentioned, I won't mention the title 7 1 again, has become very, very widely used in education 2 because it deals with the problem of students being nested 3 within classrooms, classrooms within schools. Those kinds 4 of problems become very widely used. And other aspects of 5 social science where we have people in neighborhoods, or we 6 have small groupings of people, basically, which has some 7 relevance to this case. 8 Q Are you a member of any professional organizations? 9 A I am. I'm a member of the American Statistical 10 Association, the American Educational Research Association. 11 I'm a member of the National Academy of Education. 12 Q What's the National Academy of Education? 13 A Well, the National Academy of Education is an honorary 14 association limited to 125 people in the United States who 15 are involved in education and educational research. 16 Q Have you held any editorial positions for journals or 17 other publications in your field? 18 A I have. I've been an Associate Editor of the Journal of 19 Educational and Behavioral Statistics for quite a large number 20 of years. I was the Chair of the Management 21 Committee of that journal for six years. I have served on 22 the Publications Management Committee of the American 23 Statistical Association. I'm also the Associate Editor for 24 the American Journal of Sociology, Educational Evaluation 25 and Policy Analysis and actually several other journals. I 8 1 won't list them all. 2 Q Okay. Have you received any teaching or other honors 3 in your field? 4 A I have. I received, while I was at Michigan State, I 5 received three teaching awards. I've also received several 6 awards for outstanding publications in education and 7 sociology. 8 Q Okay. Are there any awards or honors that you think 9 are particularly significant? 10 A I think perhaps the one that I'm, maybe most proud of 11 is that in 1993 I received the Early Career Award for the 12 American, from the American Educational Research 13 Association, which is a very large group of educators and 14 educational researchers around the country. 15 Q What about national panels or symposia? Have you 16 participated in any of those? 17 A Yes. In the last, within the last three years, I 18 served on the National Academy of Sciences' panel on the 19 assessment of children in conjunction, basically testing, in 20 conjunction with the Title I Program, which is a 21 compensatory education program. I also served on the 22 National Academy of Science panel on early childhood 23 science, which has just distributed a new book on childhood 24 science with implications for policy and practice. 25 Q Okay. Professor Raudenbush, I'd like to ask you to 9 1 look at Exhibit 151, which is, I think in binder six, Your 2 Honor. 3 A Okay. I see it. 4 Q Okay. Is that a current copy of your CV? 5 A It does indeed appear to be that, yes, a current copy. 6 Q And does it include a current list of your 7 publications and honors and professional experiences? 8 A Yes, it does. 9 MR. DELERY: Your Honor, at this time, we'd offer 10 Exhibit 151 into evidence? 11 THE COURT: Received. 12 Q Now, Professor Raudenbush, how would you come to be 13 involved in this case? 14 A Actually, you asked me if I'd be willing to serve as 15 an expert in this case. Can you hear me? 16 Q Yes, I can. 17 THE COURT: If anybody can't, let us know. 18 A Yeah. I had to move this because I can't turn the 19 page. 20 THE COURT: Yeah, that's correct. 21 Q And what was the purpose of your involvement in the 22 case? 23 A Well, I started by looking at some of the expert 24 reports written by Professor Kinley Larntz and I then got 25 involved in looking at the database myself, in trying to 10 1 understand some of the issues involved in this controversy. 2 Q Okay. Were you present here in court for Dr. Larntz' 3 testimony on Wednesday? 4 A Yes, I was. 5 Q And you were here for the entire day for all the 6 testimony? 7 A I was. 8 Q And what about on Thursday morning, yesterday morning 9 when he returned? 10 A I was here then too, yes, correct. 11 Q Dr. Larntz at one or two points said that he was 12 responding to some criticism of his work. Do you recall 13 that? 14 A I do. 15 Q Were you the author of that criticism? 16 A I'm quite sure that I was. 17 Q And before this week, had you ever met Dr. Larntz? 18 A No. 19 Q Are you being compensated for your work in this case? 20 A No, I'm not. 21 Q And have you ever served as an expert witness before? 22 A No, I have not. 23 Q Have you prepared expert reports, based on your work 24 in this case? 25 A Yes, I have. 11 1 Q Okay. If you could look in the same binder there, 2 binder six, I'd like for you to look at Exhibit 145 to 150 3 and tell the court whether those are the expert reports 4 that you submitted in this case? 5 A Yes, these are, these are the expert reports. 6 Q What information did you consider in preparing your 7 expert reports? 8 A Well, I read the law school admission policy, which 9 was dated 1992. And I examined data from the database made 10 available by the law school. 11 Q Did you review the expert reports of Dr. Larntz? 12 A Yes, I read also each, each expert report that Dr. 13 Larntz wrote. 14 Q Okay. And what about any deposition testimony in the 15 case, did you review any of that? 16 A Yes. I read Dr. Larntz' deposition. Of course I read 17 my own. 18 Q Did anybody help you with your work in this matter? 19 A Yes. Julia Smith, who was at that time a 20 post-doctoral fellow at Michigan, helped me. She's now an 21 assistant professor. And in certain aspects of the work 22 the, basically the diversity of context for learning part, 23 I received some help from two graduate students at the 24 University of Michigan. 25 MR. DELERY: Your Honor, at this point we'd offer 12 1 Exhibit 145 through 150 into evidence. 2 THE COURT: Any objection? Received. 3 MR. DELERY: We'd also at this point offer 4 Professor Raudenbush as an expert in the application of 5 statistical methods to education. 6 THE COURT: Any objection? 7 MR. PAYTON: No. 8 THE COURT: Okay. 9 Q All right, Professor Raudenbush. I believe you 10 mentioned that you reviewed Dr. Larntz' work in this 11 matter. 12 A That's correct. 13 Q Do you have an opinion concerning, now just as a 14 summary matter, we'll get into it in more detail. But do 15 you have an opinion concerning the reasonableness of the 16 approach that Dr. Larntz took and his results? 17 A I do. 18 Q And what is that opinion? 19 A I'm actually quite skeptical for two reasons. Dr. 20 Larntz attempted to construct a statistical model that 21 could tell us the extent to which race is taken into 22 account in admissions. And I'm convinced that it's not 23 logically possible to answer that question with such a 24 statistical model. 25 Moreover, certain methodological decisions made by 13 1 Dr. Larntz, I believe, have led to a misleading impression 2 about the strength of association between minority status 3 and admissions at the law school. 4 Q Okay. Now, you indicated that in addition to 5 reviewing Dr. Larntz' work, you did some things of your 6 own. What did you do in your analysis? 7 A Well, as I implied, I think it's, it's not possible, 8 given the data at hand, to organize a statistical analysis 9 that's going to tell us the extent to which race is taken 10 into account in admissions. What we can do, however, and 11 what I think is very useful, is to do a causal analysis of 12 the impact of using race in admissions on those who apply 13 to the university or to the law school. 14 Q Okay. And what are the basic conclusions again, as a 15 summary matter that you draw from your work in that 16 context? 17 A What we did, and we'll go into some detail on this, is 18 we compared the current policy, which does use race as a 19 factor in admissions to an alternative policy that would 20 not use race as a factor. And we estimated how that 21 difference in policies would effect the average probability 22 of admission of various people who apply, various 23 sub-groups of people who apply at the University of 24 Michigan. 25 And essentially what we found, first, of course, 14 1 is that a change in the policy would effect people 2 differently, depending on grades and test scores. It would 3 also effect people differently, depending on ethnic 4 minority status. A switch from the current policy to a 5 so-called race-blind policy would have a fairly substantial 6 effect, negative effect, on the probability of admission on 7 minority candidates. 8 On the other hand, such a change from, again the 9 current policy to a race-blind policy, would have a 10 comparatively modest effect on the positive effect, that 11 is, on the average probability of admission of majority 12 students. 13 Q And from your work, do you draw any conclusions about 14 the likely effect on the diversity of the law school class 15 of moving to a race-blind admissions policy? 16 A Yes. We can then take the admissions probabilities 17 under the current policy, as compared to an alternative 18 policy. And from that data, we're able to project the 19 number of applicants, not only who will be admitted, but 20 then using yield statistics, how many would then, in fact, 21 attend the law school. And then we can have an estimate of 22 how the class composition would look of the first-year 23 students at the law school. And so we're then able to make 24 some statements about the likely diversity with that class. 25 Q And what do you conclude? 15 1 A And what we conclude is that switch from the current 2 policy to a so-called race-blind policy would, would quite 3 dramatically reduce the fraction of students who are from 4 underrepresented minority backgrounds, and we'll define 5 that as we go, and to try to understand the practical 6 implications of that, we then took a look at how that would 7 translate into the composition of various contexts for 8 learning that occur in the law school. And, again, the, 9 how different classrooms and other context for learning 10 would look under the current policy versus an alternative 11 policy is really quite different. 12 Q Well, with that sort of basic overview in mind, let's 13 go back and talk in more detail about how you arrived at 14 these conclusions. 15 A Okay. 16 Q What was, basically, the first thing that you did when 17 you approached these data? 18 A Well, the first thing we did, and we did this for each 19 year between 1995 and 2000, was just to take a look at the 20 basic data; who applied at the law school, who was 21 admitted, who, how many people who were admitted decided to 22 come to the law school, and then what was the composition 23 of the first year class for each of those years. 24 Q Okay. And I think we've prepared a chart of an 25 illustration of that, is that right? 16 1 A Yes. 2 Q Is that right? And I'd like to put up, if I could, 3 Your Honor, Exhibit 184, the series of exhibits, I think, 4 is in the supplemental exhibit file. And the lights on 5 would be fine, because they're just words today, no screen. 6 A Your Honor, may I stand up and explain what's on the 7 screen? 8 THE COURT: You may absolutely stand up and 9 explain, yes, or we can move it closer to you so you can 10 sit. 11 A Yeah. 12 THE COURT: You're a professor, you're used to 13 standing and talking. 14 A That's right. Either that or I'll have to get new 15 bifocals. 16 THE COURT: Yeah, whatever. 17 A That's fine, thank you. 18 THE COURT: I've got a pointer here if you'd like 19 one, too, however you got to promise to give it back. 20 A Right. 21 THE COURT: Because the government, again, we can 22 get almost anything we want, but pointers. They're hard to 23 come by these days. 24 A It will be hard to walk away with this. 25 THE COURT: Yeah. 17 1 Q All right. So this chart is of the 2000 admissions 2 data, is that right? 3 A That's right. 4 Q Why don't you explain what's here and what you find 5 significant about these numbers? 6 A Well, the basic idea behind this chart is that it 7 shows quantitatively how a pool of applicants gets 8 translated into people who actually attend the law school. 9 And the thing to illustrate that, I'll just use 10 the top row of the chart in 2000. And we break this down 11 by ethnic groups. So just to take the first group here in 12 2000, there were 262 African-American applicants and that 13 constituted about 7.4 percent of the applicant pool. 14 And of those 262 people, 36.3 percent were 15 admitted. And that led to 95 offers of admission for that 16 group. Now, of those people who were offered admission, 17 only a minority, 40 percent, decided to come to the law 18 school. So if you multiple 40 percent times the 95 who 19 were admitted, then you get the number of African-Americans 20 who actually were attending the law school in 19, in 2000, 21 and that turns out to be 38. 22 So what you, basically, see is that this number on 23 the left which is 262, ultimately becomes 38, through whose 24 admitted and whether they decide to attend. That's the 25 basic idea on the chart. 18 1 Now, what we've done is, is to, to make this 2 clear, and I think in conformity with the law school policy 3 of admissions, is we've taken three groups; 4 African-Americans, Hispanics and Native Americans and 5 combined their data in the lower panel here to the data, to 6 a group that we label those of underrepresented minority 7 status. So that -- 8 Q That's UMS? 9 A And that's called UMS in this table. And then we have 10 taken data from the Caucasian group, Caucasian American, 11 and those, those whose ethnicity is unknown, and again, 12 that's in accord with our understanding of the how the 13 policy works. And we've taken their data and combined them 14 into another group that we call them non-UMS. They are the 15 ones who are not in the underrepresented minority status. 16 Now, that leads one group that I haven't 17 mentioned, and that's the other group, the non-citizen 18 group, and that group, we do not include in this table. We 19 could have looked at underrepresented minority status, yes 20 or no, and foreign or foreign students. But the numbers, 21 in fact, there were only three foreign students attending 22 in 2000, are really too small to do much with. And it 23 seemed that whatever was happening with minority status and 24 non-represented minority status was somewhat different 25 because this group is ethically very diverse, the foreign 19 1 group, and yet they're not in these categories so we didn't 2 include that small number of applicants. 3 So then down at the bottom what we, basically, 4 have are underrepresented minority students and 5 non-underrepresented minority students and then a total. 6 Q Do you find anything significant about the pattern of 7 the numbers here on the bottom half of the chart? 8 A Yeah. There's several significant features of this 9 table. One is we just start by just looking at the 10 applicant pool. So we see that there are 484 applicants 11 who are minority. I'm just going to use the word 12 minority's" and "non", I think, because it gets hard in 13 saying. 14 THE COURT: That would be great. We all 15 understand. 16 A And I'll try, and I often may use the word "race", and 17 I don't necessarily mean "race". We know there's ethnicity 18 and it's complex, but I'll use it because it gets hard to 19 use so many words. 20 But so 484 minority applicants, and in contrast to 21 2,871, majority applicants, or non-majority applicants. 22 And so the pool sizes are very different. There's a much 23 smaller number of minority applicants than non. So that's 24 one factor that we -- it's very important in 25 understanding -- the dynamics of this whole system is just 20 1 the different sizes of that applicant pool. 2 The next feature that's very important to look at 3 is just the percentage admitted, because that's a crucial 4 factor in, who ends up being in law school. And what we 5 see is that 35.1 percent of the minority applicants and 40 6 percent of the non-minority applicants are admitted. 7 And these numbers are quite reflective of what 8 happens year to year. Only a minority of people, of the 9 overall applicant pool is admitted. The numbers are pretty 10 similar. In general, the fraction admitted is smaller for 11 the minority group than for the non-minority group. 12 Q And is that true in each of the years from 1995 to 13 2000? 14 A That is true. The general pattern is true each year. 15 These numbers will fluctuate but the general pattern is 16 true. And the, from the point of view of promoting 17 diversity, ethnic diversity, which is one of the goals 18 stated in the admissions policy, these two facts; there is 19 the small pool size and the comparatively small fraction of 20 those admitted has important implications for the diversity 21 of the class. Because if this number is lower much, the 22 number of people who actually attend can get very small. 23 Specifically, in this case, with 35.1 percent of 24 the minority applicants admitted, and then with the yield 25 of 34.1 percent out of the 484 applicants who are minority, 21 1 what we see as actually attending, 58. So 484 goes down to 2 58. 3 And if you're, you know, if you're interested in 4 diversity, the size of this applicant pool, the fraction 5 admitted and the yield are going to strongly effect this 6 number, and I guess this percentage admitted is under -- 7 obviously under the direct control of the law school. And 8 if we shadow where we're going with our analysis, if this 9 number were reduced significantly, this number 58 would 10 begin to go down. 11 I mean, if this number were cut in half, then we'd 12 have only 29 minority students, so, and that would assume 13 that the number of applicants and the yield would remain 14 constant. Do you see my point? That if we cut this number 15 in half, hold everything else constant, we're down to 29. 16 THE COURT: Or double it and it may go up? 17 A Or double it and it will be go up to 116 if we double 18 it. So whatever we do here has big effects on this number, 19 but we also need to take into account the possibility that 20 changing this number could change this number. It could 21 change the number of people who apply. It could also 22 change this number, the number of people once admitted who 23 might then decide to attend. 24 So in particular, if this number were lower, this 25 number could, would likely -- it probably wouldn't stay the 22 1 same. A more likely outcome, if you lowered the 2 probability of admission of a group, it might encourage 3 fewer people to apply. That's, we don't know. And our 4 analysis won't assume that, but the law school would have 5 to take that into account as a possibility. And lowering 6 this number might also end up lowering the yield because if 7 you, if you reduce this number substantially you would be 8 left with, under a race-blind policy, the people who would 9 be here would be extreme. 10 Q "Here" being the number admitted? 11 A The number admitted would be an extremely highly 12 qualified group, in terms of grades, test scores and so 13 forth. And the yield for such a group may be, may be lower 14 because there may be significant competition, among law 15 schools for those people. So changing this number could 16 impact these numbers. And with, with large effects on 17 this, this relatively small number, 58, so that's, that's 18 the key thing that's happening. 19 Q Right. So this chart is of the 2000 data. Does your 20 report include similar information for the other years? 21 A It does. 22 Q For the various reports? 23 A We have a similar flow chart for each year from 1995 24 to 2000. 25 Q And is the 2000 data unusual, compared to the other 23 1 years? 2 A The 2000 data are pretty similar in virtually all 3 regards. There's one slight difference here. The yield 4 for African-Americans candidates in 2000 was 40 percent, 5 which is, which is higher than it had generally been in the 6 other years. So that number is a little higher than 7 average, but other than that it looks. 8 Q If we compared the, this data, including the number of 9 applicants to some similar charts in Dr. Larntz' reports, I 10 think there may be some slight differences, is that right? 11 A I looked at those numbers. The, the exact numbers are 12 not identical and I don't really know why. 13 Q You worked from the same database? 14 A We worked from the same database. 15 THE COURT: Are they significantly different? 16 A They're not significantly different. 17 Q Okay. 18 A The patterns that I'm describing are very similar, I 19 mean, they're virtually identical in the two sets of 20 figures. 21 Q Now, in addition to your point about how, how the 22 various percentages, in particular, can effect the number 23 attending in each year, do you take any other basic 24 conclusions away from looking at this basic descriptive 25 data? 24 1 A There are a couple of other conclusions. While, 2 remember, I mentioned that a change in this percentage 3 would lead to, perhaps, fairly large changes in this 4 number; that is, the number admitted and also this number, 5 the number attending, and that's for minority applicants. 6 If changes in this number that are small would 7 have comparatively modest effects, if, let's say, half of 8 these people were rejected instead of, let's say, that 9 would be, that would be, we have 170. That would be 85 10 people. If those 85 places became available to the 11 majority students and then these 2,871 would compete for 12 those 85 places, and so that change, which is big here, 13 that is in the minority row, would have a comparatively 14 modest chain effect on the majority role, so that's one 15 additional piece of evidence from this. 16 Q And the comparison or the percentage admitted of the 17 two groups, I think, what also might be called the average 18 probability of admission, is that right? 19 A Yes. 20 Q Does that comparison tell you anything about the 21 impact of considering race in admissions? 22 A Well, this, we call it a, yeah, we call this bivariate 23 association. There are two variables. There's the race of 24 the candidate, and then there's the admission decision, and 25 when we look at these two proportions, that gives us 25 1 evidence about that bivariate association. Is there an 2 association between race and admissions? And we see a very 3 small bivariate association, actually which favors the 4 majority applicants. 5 Now, we use, in statistics, we tend to look at 6 these bivariate associations as a first take on what's 7 going on, just simple data, there's no model, just look at 8 the data. And so we see this relationship. And in 9 conjunction with other bivariate relationships, my 10 conclusion from this was that it, it leads one to be 11 skeptical of a claim that race is a powerful predictor of 12 the admissions decision. 13 Q Not the end of the analysis but a starting point? 14 A It's not the end of the analysis, but, let me expand a 15 little bit. If we look at, let's say, just the association 16 between grades and admissions, there's a very strong 17 relationship, even with higher grades are more likely to be 18 admitted. If we look, and we don't have to control for 19 race to see that. We just see that relationship. If we 20 look at the relationship between test scores, LSAT and the 21 probability of being admitted, we see a very strong 22 relationship, we don't have to control for anything else to 23 see that. We look at the relationship between race and the 24 probability of being admitted, we see very little 25 relationship. 26 1 So that, that tells us that race is unlikely to be 2 a powerful predictor of the outcome. It doesn't mean that 3 race and admissions are not related controlling for other 4 factors, but it does suggest that race will not be a 5 powerful predictor for the admissions decision. 6 Q Okay. I think at this point you can probably take 7 your seat again. 8 A Thank you. Your Honor. 9 THE COURT: No, just hold on to it. 10 A Yeah, I need it again. 11 MR. DELERY: I think we may need it again. 12 THE COURT: Maybe you can move the chart just so 13 the folks in the audience can see. 14 MR. DELERY: Sure. 15 THE COURT: Great. Thank you. 16 MR. DELERY: I apologize. 17 Q Now, in addition to the examination of the basic 18 descriptive data, what did you do as part of your analysis 19 in the case? 20 A Well, as I mentioned, I'm convinced, and I think will 21 explain why a little later. But I'm convinced that we 22 can't develop a statistical model that's going to tell us 23 the extent to which race is taken into account in 24 admissions. What we can do and what I think is useful to 25 do is to do a causal analysis. What's the impact of the 27 1 policy that the university has of using race in admissions 2 on the people who apply. And that causal analysis is 3 something that we can do with a minimum of assumptions. 4 And so that's what I decided to do, and I thought that that 5 would be informative. 6 Q Okay. Have you prepared a chart to sort of explain 7 that causal connection? 8 A Yes, I have. 9 Q Okay. I think for this one, you can probably just 10 stay where you are with the easel where it is. 11 A Especially with this. 12 Q This is Exhibit 185, right, exactly with the long 13 stick? 14 A Right, with the long stick. I don't have to get up. 15 Q So this chart is called conception for causal link 16 between race and admissions? 17 A Right. 18 Q What do you mean by that? 19 A Well, in causal analysis and statistics, the way we 20 think is that we've got, let's say, two alternative 21 treatments. We've got treatment A and treatment B. 22 Now, for each person that we're interested in, we 23 imagine the following, that that person has an outcome 24 under treatment A and an outcome under treatment B, and the 25 difference between the two outcomes is defined, 28 1 statistically, as the causal effect of the treatment. 2 So if I, if one person has, let's say, I could 3 randomly assign a person to have surgery for heart problem 4 or I could randomly assign to have medicine, and the, and 5 the person would have one outcome under the first 6 treatment, another outcome under the second treatment. 7 Causal effect is the difference between the two outcomes. 8 So we applied that basic idea to the, to the scenario here. 9 What we have on the left, what we have up here is, is a 10 person, an applicant which and this person. 11 Q You can tell we're not artistic. 12 A Right. I wouldn't want to be that person, but we have 13 that person. And this person is going to apply to the law 14 school and that person might apply under policy A. Policy 15 A is the current policy, according to the admissions 16 policy. 17 And in that policy, it states a number of factors 18 that are going to be taken into account, and I guess, I'll 19 read them. I don't know if you can see them all; 20 undergraduate grades, the law school aptitude test, 21 Michigan residency, minority status, gender is, could be 22 considered, I assume as a force, a form of diversity. The 23 quality of the undergraduate school, the curriculum; that 24 is the courses that the applicant took, trend in grades, 25 not just are they how or were they going up, relationship 29 1 with family members who are alumni. There are essays that 2 are required, letters of recommendation and leadership 3 experience. A person may have displayed other unique 4 experiences and talents and then unusual circumstances. So 5 this -- there's this list of factors that could be taken 6 into account. 7 Q And these are all things, if I could interrupt you for 8 a second? 9 A Yes. 10 Q That are reflected in the policy, as you read it? 11 A That's right. I, I got these right out of the policy 12 document itself. And so our applicant comes and applies 13 under policy A. All of these characteristics are taken 14 into account and the results is this person has a certain 15 probability of admission. We call it a probability because 16 there's some uncertainty in what's actually going to happen 17 here. There's subjective judgments being made and there's 18 some probability of admissions. So we call that 19 probability A. So that's policy A. 20 Now, if our same applicant were to apply under a 21 different policy, and we're going to call that policy B, 22 the result might different. Policy B is, we label a 23 race-blind admissions policy. And the way we're, the way 24 we're defining that is that all of the same factors that 25 were taken into account under policy A would be also taken 30 1 into account under policy B with one exception, and that is 2 underrepresented minority status. That would not be 3 considered. So we call that a race-blind policy. 4 So our applicant comes along now, low and behold, 5 policy B is in effect. These are taken into account, these 6 factors, and the result is that our applicant has a 7 probability of admission, a piece of B. 8 And so with that scenario in mind, we can define 9 the causal effect of policy A versus policy B as being the 10 difference in the two probabilities of admission. So if, 11 let's say our applicant applied under policy A and got a 12 piece of A, probability under B, a piece of B. 13 Suppose those two probabilities were the same, 14 identical, there would be no causal effect of a change in 15 policy on that person. Suppose, on the other hand, that 16 these probabilities were very different. A person was, 17 let's say, you know, very unlikely to get in under policy A 18 and very likely to get in under policy B, big causal effect 19 of the policy. So that's, basically, how we defined the 20 causal effect. And that was what set up our analysis. 21 Q Now, why do you think it's important to look at this 22 contrast between two policies in this case? 23 A There are two reasons. One is that a change from 24 policy A to policy B could effect the diversity of the 25 incoming class and that's one of the goals stated in the 31 1 admissions policy is to have an ethically diverse class, 2 and so we can use this framework to assess the effect, 3 causal effect on the change of policy on the diversity of 4 the class. 5 The other reason that it's important is that it, 6 it's a way of gauging the causal effect of, on those who 7 apply, I mean, I think that a person who applied to the 8 university, or to the law school, would be very concerned 9 about, are my probabilities going to be very different 10 under these two, under these two policies. If they were, 11 that would have important effect on behavior of people who 12 apply and it's just an important issue and it gauges the 13 extent to which the current policy is strongly effecting 14 the outcomes of people who apply. 15 Q Okay. And is this kind of comparison between 16 alternative policies the standard way in your field to get 17 at causal questions? 18 A This has become the, essentially, the consensus in how 19 we think about causation in statistics, two alternative 20 policies, an outcome under each for each person and the 21 causal effect being defined, as I mentioned. 22 Q Now, how, if at all, does this conception, this 23 approach, differ from what Dr. Larntz did? 24 A Okay. In Dr. Larntz' analysis, he's analyzing the 25 data that were generated under policy A and computing 32 1 correlations or associations and trying to use those to 2 make strong causal inferences. And, as I mentioned, I'm 3 convinced that that's not logically possible to do in this 4 case. This kind of analysis -- 5 THE COURT: You say in this case, in any case? 6 A With, well, I think part of the problem is the amount 7 of available information. With, if, with a great deal of 8 information, one might be able to make a better, I think 9 that's an important constraining piece, if there were 10 enough information, but we really had very limited 11 information about the people who apply, numerical 12 information, so I think that's a key constraint on the, on 13 a correlational approach. Generally. 14 THE COURT: Well, you say. 15 A Sure. 16 THE COURT: Limited numerical information. What 17 other, on your list, there's only certain things that can 18 be equated to numbers. 19 A Right. And that's one of the difficulties in drawing 20 a causal inference from numerical data is the -- 21 THE COURT: Oh, I see. 22 A If the important, if many of the important factors are 23 not co-indentifiable. 24 THE COURT: I see. Thank. 25 A That would be a good reason why we didn't have that 33 1 information. 2 Q Now, with this conception for the causal analysis in 3 mind, what did you do next in your analysis? 4 A What we tried to do then was to compare policy A and 5 B, and I think we have an exhibit that displays how we 6 approach that. 7 Q Okay. Let's put up Exhibit 186 now, the next chart. 8 Does this chart illustrate how you approached your 9 analysis? 10 A It does. Simulating would happen under policy A was 11 very easy because we actually didn't have to simulate it. 12 We have the data from the years '95 to 2000. So we just 13 actually used, we used the actual reported admissions 14 results to compute probabilities of admission, average 15 probabilities, of admission for various sub-groups who 16 applied, and those were just based strictly on the data. 17 Policy B posed us with a more challenging problem. 18 We don't know what the effect will be on the probability of 19 admission under policy B, because it's never been 20 implemented. So we have to make some assumptions. 21 Essentially what we did was we had data on grades, 22 on test scores, Michigan residency and gender. And we can 23 develop, based on past data a prediction equation that 24 would predict the probability of admission, based on past 25 data. And then from that we can simulate what's happening 34 1 under policy B. The problem we face is the same problem 2 that Professor Larntz faced. There's a lot of information 3 that we don't have. We don't know anything about the 4 undergrad school curriculum, etc., essays, recommendations, 5 all these other things, these long list of factors. We 6 don't have any numerical data. 7 Q When you say "we don't know about those things", you 8 mean that, as a statistician looking at the data you don't 9 know? 10 A Exactly. As a statistician analyzing the numerical 11 database, I only have access to a small fraction of the 12 relevant information used in make admissions decisions, so. 13 Q The admissions officers have more information than you 14 have? 15 A Exactly. And that's why, that's one of the reasons 16 why it's difficult to model those decisions. They know a 17 lot more than we do. And we have to make assumptions about 18 what we don't know. In order to do this simulation, we 19 have to assume, essentially, that all of these factors that 20 we don't know anything about are not associated with the 21 factors that are in our model. 22 THE COURT: So you have quite a few there? 23 A That's right. 24 THE COURT: And Dr. Larntz testified that the 25 fewer assumptions you make, and I'm not saying you have to 35 1 agree or not agree, but I'd like your opinion on it. He 2 testified that the fewer assumptions you make, the better 3 your results are. That when you start making assumptions, 4 that it may skew it to subject -- I don't think you used 5 the word, subjective, but at least it's more extensive. In 6 your model you're making assumptions, at least, as to one, 7 two, three, four, five, six, seven, eight, nine, ten areas? 8 A That's right, exactly. 9 THE COURT: So do you disagree with him? 10 A Oh, I agree with him on that, absolutely, yes. We're 11 very concerned about the impact of the possible falsehood 12 of these assumptions. And there are almost certain to be 13 some falsehoods here. The question is the falseness of 14 these assumptions, the question is to what extent does that 15 effect the result. 16 We know we're not going to really have the model 17 right, but to the extent we have it wrong, to what extent 18 does that have some effect on our results. And that's what 19 we then had to do in this was to, what we actually did was 20 we did this simulation. 21 We looked at the results. We repeated the 22 simulation a couple of other ways, but actually, this is, 23 in some ways that I believe the great strength of the 24 causal analysis. We can put bounds on the errors of your 25 our estimates that require virtually no assumptions, so we 36 1 can actually assess the extent to which errors in our 2 assumptions effect our results in a very sure-minded way, 3 and I'll try to explain how we did that as we go. 4 So the way, the way it works is, is you do an 5 analysis, based on assumptions, you look at the results, 6 you try another analysis, generally, that's based on maybe 7 some different, slightly different assumptions. But then 8 you try to bound the error in your results as a function of 9 your assumptions, and we we'll show how we do that. 10 MR. DELERY: I think it will be easier to see 11 that, Your Honor. 12 THE COURT: That's fine. 13 MR. DELERY: After we see the results. 14 Q But before we leave this point, while we're on 15 assumptions and just so we're clear, what, what is, or what 16 are the assumptions about the factors below the line on the 17 chart, as related to the factors above the line, just so we 18 have that in mind? 19 A Right. Basically the assumption is that if any of the 20 factors below the line are correlated with, with the 21 factors above the line, then our estimate of the effects of 22 the factors above the line will be biased. 23 Q So -- 24 A And if they're biased, the predictions, the predicted 25 probabilities will be potentially biased as well. 37 1 Q I think we'll come back to that as to how you dealt 2 with that, is that right? 3 A Yes. 4 Q All right. But before we go to look at the results, 5 let me just ask you a couple questions about exactly what 6 you did. Did you, just as a general matter, did you use 7 any particular kind of, of analysis to undertake the 8 simulation? 9 A We did. We used -- the first method we used was 10 called, logistic regression. And I think we've had a 11 discussion of that. You have a binary outcome which is 12 admitted, yes or no, and then you have a number of what we 13 call explanatory variables, which are the ones here above 14 the line. And you are able to estimate an equation that, 15 that estimates the relative weights of these factors on the 16 probability, the log odds of admission, and ultimately we 17 can translate like that into the probability of admission. 18 Q So - 19 A We've talked about that in court. And I assume we 20 don't need to necessarily say much more about it. I think 21 Professor Larntz explained what that was. 22 Q And so Dr. Larntz also used logistic regression, of 23 course, as part of his analysis? 24 A Yes. 25 Q And we'll get back to Dr. Larntz' regression models. 38 1 But are there general things that you can say about how 2 your regression analysis differed from, in addition to the 3 conception from what Dr. Larntz did? 4 A We, yes. We actually estimated our models separately 5 for minority and majority applicants. And the reason we 6 did that was that we found that the association between 7 minority status and admissions was strongly dependent on 8 grades and test scores; that is, we found that, for 9 example, applicants who had very high grades and test 10 scores, for those applicants minority status has a very 11 small effect, or very small association. And for 12 applicants in other cells the association is considerably 13 stronger. So because the association between minority 14 status and these factors varied, what statisticians then do 15 is, they say we can't estimate one model for everybody, we 16 then do the models separately. 17 Q Did you exclude any of the applicants for which you 18 had data from your analysis? 19 A No. We used -- oh, I should say, we did exclude 20 people, a very small number of people have have no grades. 21 There's just, they don't have grades in the database. It's 22 a tiny fraction, or they don't have LSATs, so those people 23 we excluded. But we excluded no cases based on their 24 outcomes. 25 And this is a very important point. When you 39 1 start excluding cases from an analysis based on the outcome 2 of the admissions decision, you get into some significant 3 biases and we did not do that. 4 (Whereupon an off-the-record 5 discussion was had.) 6 Q All right. So with the simulation model or regression 7 model, how did you conduct your simulation? 8 A So what we did was we actually, for each year, we did 9 the analysis I mentioned, we did it separately for majority 10 and minority applicants. We actually used the majority 11 equation in predicting the probabilities of admission under 12 the race-blind policy. We assumed that under the so-called 13 race-blind policy that the majority equation, which has 14 more cases involved in the estimation would be more like, I 15 mean, the average equation would be more like that. So we 16 used that equation. 17 Q Okay. And with that equation, what did you do? 18 A Well, based on that equasion we could compute the 19 predicted probability of admission under policy B for any 20 applicant, and then we could combine those within ethnic 21 groups to predict the average probability of admission for 22 any sub-group of applicants in this case, as a function of 23 ethnicity. 24 Q And so from that you can estimate how, what the 25 percentages admitted would look like? 40 1 A Exactly. From that we're able to compute the average 2 probability of admissions for ethnic, for minority and 3 majority applicants, and compare it to the observed 4 probability of admission under the current policy. 5 Q All right. I'm going to ask just one other thing 6 about the simulations. Are you able to, are you able to 7 say, based on the simulation, what would happen to any 8 particular applicant under the alternative policy? 9 A No, we're not. And this is one of the ironies of 10 causal inference and causal modeling. For any person, 11 we'll never know the two probabilities. In order to do 12 that -- we can't even imagine how to do it. We'd have to 13 have both policies in operation and we'd have to have them 14 implied under both policies and see all the results. But 15 we can't do that. And that's generally true in causal 16 inference. We can't compute the causal effect for any 17 specific case. What we can compute is called the average 18 causal effect. In this case, it would be the average 19 probability of admission under policy A, minus the average 20 under policy B for sub-groups of applicants. 21 Q Now, let's look at, if we could what happened in your 22 simulations. I think the next Exhibit is 187 in the same 23 category. 24 A Now, mind you -- 25 Q Yeah, why don't you first tell us what the columns 41 1 are. 2 A Right. 3 Q And then -- 4 A Yeah. 5 Q Explain what the results are? 6 A Let me just preface it by saying that the results of 7 policy B are going to be those based on the model I just 8 described, but we also replicated this analysis using 9 another, actually a couple of different regression models 10 we tried. But we also used another method, which we can 11 describe a little bit later. But under the method that I 12 just described -- 13 Q Can I, let me just ask you -- 14 A Yeah. 15 Q Are the results under the other methods substantially 16 different? 17 A They're not substantially different. They're somewhat 18 different but in the main, they're very, very similar. 19 Q All right. So why don't you explain what you have on 20 the chart and then what the results showed. 21 A Okay. What we have on the chart are two columns, 22 policy A, again, that's the current policy; policy B, this 23 is the so-called race-blind policy that I mentioned. 24 Q And just so we're clear, the number in policy A is the 25 actual observed data? 42 1 A Right. And so we have for minority and non-minority 2 applicants, and for each year, the predicted -- well, in 3 this case under policy A, the actual observed average 4 probability of admissions. And then under policy B, the 5 average probability of admission for that same group. 6 A So again looking at 2000, we've been looking at 2000. 7 The average probability of admission in 2000 for minority 8 applicants was .35. We project that under policy B the 9 average probability of admissions would be .10, which is, 10 which is quite a large difference. And that type of result 11 occurs in each year. They're pretty similar. There's some 12 exceptions. 13 It turns out that 1995 is a bit extreme in terms 14 of the change in the probabilities for the minority group. 15 But, but it follows the same pattern. It's, and the other 16 years are very similar to, to the year 2000. So we see 17 then in some, a quite sharp reduction in the average 18 probability of admission of the minority applicants under 19 policy A and policy B. 20 A Now, if we move down to the bottom panel, we have the 21 results for the non-minority applicants under each year. 22 So, again let's just take a look at, for illustration of 23 the year 2000 under policy A the average observed, average 24 probability of admission was .40, 40 percent of those who 25 applied were admitted. We project that under policy B, 43 1 this is a race-blind policy, that would increase. It would 2 increase from .40 to .44. So it would be rather marked, 3 small or marginal increase in the average probability of 4 admission, .40 to .44. 5 Q And are the results similar for the other years? 6 A And the results are very similar for other years. It 7 tends to be, .99 goes 41 to 45, again the difference being. 8 .04. In some cases it's .05. I think the actual biggest 9 one we see is in '95, not surprisingly, which is .06, .28 10 up to.34. 11 Q Now, why is it, Professor Raudenbush, that the change 12 in the average probability of admission is fairly large for 13 the minority students and fairly small for the non-minority 14 students? 15 A It's a very straight-forward result of the difference 16 in the sizes of the applicant pools. There are relatively 17 few minority applicants, a small -- a large change in the 18 probability, a large reduction in the probability of 19 admission of those candidates translates into a very small 20 increase in the probability of admission of the majority 21 group, because it has so many more applications; basically, 22 any extra, sort of admission seats, if you will, or admits, 23 could become available by reducing this probability, will 24 be competed for by a large number of people. 25 Q Now, as Judge Friedman alluded to earlier. 44 1 A Right. 2 Q These simulation results are based on regression 3 models which involve assumptions, correct? 4 A Right. 5 Q How can you be confident, given those assumptions 6 about these results here? 7 A Right. Well, the first thing we did, as I mentioned, 8 was we did use an alternative method to do the simulation, 9 and as you asked me were the results similar and the answer 10 was, yes, they were very similar. 11 Q So the fact that you got similar results says what 12 about these? 13 A From an approach that did not use logistic regression 14 at all, and I'll explain exactly what we did a little bit 15 later. But the most important way that we can bound our 16 error, if you will, is much more straight forward and 17 requires an absolute minimum of assumptions. And I think I 18 can maybe demonstrate that with a different exhibit. 19 Q All right. Why don't we go. 20 THE COURT: Let me ask you one question? 21 A Sure. 22 THE COURT: You can also conclude from that chart 23 that by having a race-blind policy that, looking at 2000, 24 for example, that there's a 25, obviously a 25 percent 25 difference, so that. 45 1 A Right. 2 THE COURT: That's right. So you could also say, 3 could you not, that the effect is, the effect having a 4 policy that's not race blind is about 25 percent? 5 A A difference in probabilities of .25, yes, right. And 6 people do this in different ways. We talk about odds, 7 ratios of probability. Sometimes differences in 8 probabilities are the most straight-forward way of 9 interpreting the results. It depends on the situation. 10 Q Let me ask a related question. 11 THE COURT: Well, go on. 12 MR. DELERY: Please. 13 THE COURT: You ask, I'll get mine later. He may 14 answer. If he doesn't. 15 Q Before we look at the bounding point, in your view, do 16 these numbers here, the results of your simulation analyses 17 say anything about the extent to which race is considered 18 by admissions officers in making their decisions? 19 A They don't. 20 Q And why is that? 21 A Let me explain what could generate this difference in 22 probabilities. If you have a large, a much larger 23 applicant pool than can be admitted, so you have many more 24 people apply than you can accept; and if grades and test 25 scores are very important, play an extremely important role 46 1 in the admissions decision, then a very small difference 2 between two groups can lead to a large difference in the 3 probability of being admitted. 4 And so under this, under our simulation of the 5 race-blind policy, grades and test scores are playing a 6 very important, and extremely important role because we 7 don't have any other data, basically. We know that there 8 are many more applicants than there are seats. And we know 9 that there's a small difference between minority and 10 non-minority applicants. And that explains why this 11 difference turns out to be big. 12 Q So. 13 A And it doesn't. 14 THE COURT: Turns out to be big? 15 A Big, yes, these numbers are quite different. That 16 doesn't depend on how heavily the admissions officer weigh 17 race. It's a function of the fact that you're heavily 18 weighing a factor on which two groups have a different 19 mean. 20 THE COURT: A different what? 21 A A different mean, a different average. 22 THE COURT: Mean. 23 A Right. 24 Q Just so I have a sense of the terminology here, is it 25 your view that there's a difference between measuring the 47 1 effect or impact of the policy on the one hand? 2 A Right. 3 Q And the extent to which a particular factor is 4 considered in an admissions process on the other? 5 A There's a great deal of difference. And I might add, 6 especially in this case, the causal impact of the policy is 7 much more excessible to statistical investigation than is 8 an attempt to discern how people who are making decisions 9 about admissions are weighing one of many factors, when we 10 don't have any information about most of the factors. It's 11 just a very difficult thing to do, statistically. We 12 basically can't do it. 13 So, but we can assess the impact of what they do. 14 We don't know why it has that impact. You see, there's a 15 big difference between finding a causal effect and 16 explaining the causal effect, knowing why it happens. 17 There are lots of things in social science, 18 medical science, where we know there's an impact on 19 something, but there's so many possible explanations. And 20 we don't have the information to explain the explanation. 21 So this analysis can be conducted with a minimum of 22 assumptions and with a considerable amount of confidence, 23 whereas the more, the much more challenging task of trying 24 to use statistical information to discern how people who 25 have much more information than we do, how they think. 48 1 This is much more difficult. 2 Q I think we'll come back to this question of extent a 3 little bit with some additional illustrations, but let's 4 return to the bounding point? 5 A Right. 6 Q That you were on, if we could. And I think the next 7 exhibit is 188. 8 MS. MASSIE: Judge Friedman, I don't know if this 9 is, if we could take a quick break, that would be great. 10 THE COURT: Of course, how much do you want? 11 MS. MASSIE: Five minutes. 12 THE COURT: Okay. We'll take a five-minute break. 13 (Whereupon an off-the-record 14 discussion was had.) 15 THE COURT: Okay. You may be seated. Thank you. 16 MR. DELERY: Thank you, Your Honor. 17 Q Professor Raudenbush, I believe we had been talking 18 about the simulation results for the minority students on 19 the one hand and the non-minority students on the other 20 hand and the bounding issue that you? 21 A Yes. Just to recreate where we were, the key result 22 here was that the effect of going from policy A to policy B 23 was quite big for the minority students. Like in 19, in 24 2000 it was 25 percentage points, whereas the effect going 25 from policy A to policy B on the non-minority students was 49 1 quite small. 2 So in 2000, going from forty, .40 to .44, so going 3 up on four percentage points. So that's where we were, and 4 the question is the problems with this model. 5 As we discussed, policy A, policy B is based on a 6 simulation. It's based on a model. The model has to make 7 assumptions. The assumptions, not might be, but probably 8 are wrong, and so how far off might we be, as a result of 9 failure of those assumptions, and that was our next step. 10 Q Okay. And here, are you talking about the assumptions 11 that the factors not in the model are unrelated to the 12 factors in the model? 13 A Correct. 14 Q Did Dr. Larntz' model include the same assumptions? 15 A Yes. 16 Q Well, why don't you move to the next chart, actually, 17 and tell us what you did. The next chart will be 188. 18 Tell us what you did to evaluate how reasonable your 19 results were, in light of the assumptions. 20 A What we did was we used an idea that has a fancy name 21 but it's a real simple idea. The fancy name is, these are 22 non-parametric upper and lower bounds on causal effects. 23 The simple idea is how, how small could the effect be and 24 how big could it logically be. And here's how simple it 25 really is. 50 1 Again, let's just focus on 2000. And we're 2 looking at majority students here. And we see that in 2000 3 40 percent of them were admitted. How small could the 4 effect be of going to policy B? Well, logically it seems 5 that the smallest the effect could be would be there 6 probability would stay the same. 7 In other words, we go to a race-blind policy and 8 there's no impact. It goes from .40 to .40. It logically, 9 it logically can't really go down. It's hard to imagine 10 how eliminating race as a factor would make things worse 11 for, for majority students. So .40 is the lower bound for 12 the effect. So zero percentage points, .40 to .40. The 13 upper bound is, is constructed, again, very simply; how big 14 could the effect be. The biggest it possibly could be 15 would be if every minority students were rejected under 16 policy B. If you eliminate race as a factor and every 17 single minority students were rejected, then that means 18 that's the biggest effect it could be. 19 And under that scenario, the upper bound is .46, 20 so that means the difference between the lower bound and 21 the upper bound is .06. That's six percentage points. Our 22 estimate, based on our simulation is .04. It's kind of in 23 between the lower bound and the upper bound. So our .44 is 24 undoubtedly wrong, to some degree, but to what degree can 25 it be wrong, the upper and lower bound tell us, it can't 51 1 be -- the lower bound is a .04 error, the upper bound is a 2 .02 error and those bounds don't require me to make any 3 assumptions about what's in the model, what's not in the 4 model. Those are logical upper and lower bounds. 5 Q So based on the bounds that you found and, as compared 6 to the simulation results, do the bounds give you 7 confidence in, in your models and in your analysis? 8 A They give us confidence in the causal effect of the 9 policy change on the majority students. 10 Q And that's what this chart shows? 11 A That's what this chart shows. Now, I should add that 12 the bounds on the causal effect for the minority students 13 are wider because like when, I think in 19 -- in 2000 we 14 went from, I think it was something like .34 to ten. The 15 extreme bound would be to zero. So from .34 to zero. So 16 they were a little bit wider. 17 There's a little more uncertainty as to how the 18 switch in policy would effect the minority students. But 19 there's a great deal more -- I should say a great deal less 20 uncertainty about how the change in policy would effect the 21 majority students. 22 Q Did Dr. Larntz do any kind of similar bounding 23 analysis on the results of his regression model? 24 A I didn't see any evidence of it in the reports. And I 25 didn't hear him put an upper and lower bounds or a 52 1 confidence interval on the odds ratios. 2 A By the way, a confidence interval is a weaker bound, 3 much weaker than a non-parametric up upper and lower bound 4 because this bound has virtually no assumptions. The only 5 real assumption I'm making is that going from policy A to 6 policy B wouldn't hurt the majority students, and that seem 7 indisputable. 8 Q So these results tell us what the expected 9 probabilities of admission are for, on this chart, the 10 majority students and on the earlier chart also, the 11 minority students? 12 A Correct. 13 Q Did you take that analysis any further? 14 A Yes, I did. Once we have predicted probabilities of 15 admission or average probabilities of admission for 16 sub-groups, we can then develop a picture of what the 17 composition of the first-year class would look like under 18 policy B. Of course we already know the composition of the 19 class under policy A. It's what we observed. 20 And to do this is really very straight forward. 21 We take the probabilities of admission under policy B. We 22 multiple that by the yield which is what fraction of people 23 who were admitted decided to come to Michigan, the one that 24 was actually observed. And that can then give us the 25 expected number of people in each, of each group for each 53 1 year. 2 Q Okay. I think we have a chart showing those results. 3 A Yes. 4 Q It's Exhibit 129. Just so I'm clear about your last 5 point, Professor Raudenbush, you're assuming in this part 6 of the analysis that the yield rate would not change? 7 A That's correct. 8 Q If the university moved to a race-blind admissions 9 policy? 10 A Exactly. We're, it could arguably go down if this 11 change were made, in which case our results would 12 understate the impact on diversity. 13 We're also assuming, as years go by, that the size 14 of the minority applicant pool would not be effected by a 15 sharp reduction in the probability of admission, which is, 16 which is another conservative assumption. It seems 17 reasonable that if the probability of admission goes down, 18 the number of people who would take the time and effort and 19 pay the price of climb might well go down, but we didn't 20 assume that that would happen. 21 Q Why don't you look at this chart, Exhibit 189, and 22 tell us what it shows about this next step of your 23 simulation analysis? 24 A Okay. Again, it's divided. As we go down the, down 25 the rows, we see the years. We have under policy A and 54 1 under policy B and in each case what's in here is the is 2 the composition of the class. So for policy A it's going 3 to be the actual composition that happened in that year. 4 Under policy B, it's what we would predict, based on the 5 simulation. 6 And again, why don't we just, for illustration, 7 stick with 2000. Under the current policy, 170 minority 8 students were admitted and based on the yield, 58 actually 9 attended. And that was, that turned out to be 14.5 percent 10 of the class. 11 Q Those numbers were taken from the first chart that we 12 saw today? 13 A That's right. Those are just the actual observed 14 numbers. Under policy B, we, we would predict that only 46 15 minority students would be admitted. And then applying the 16 yield, that would lead to 16 attending. So only 16 17 minority students, from 58 down to 16, and then that would 18 be four percent of the class, so our, our analysis would, 19 would predict a reduction in the fraction of students who 20 are minority from 14.5 percent to 4.0 owe percent. 21 Q So what, if anything, do these results, I guess I 22 should back up and ask, is 2000 unusual in this respect, 23 or? 24 A The basic pattern of 2000 appears each year. We see 25 very similar results. Again, there's a little more extreme 55 1 result in 1995, but it's basically in the same direction, 2 same pattern, and the other years are very similar. 3 Q So what did these results tell you, if anything, about 4 the expected diversity of the law school class under a 5 race-blind admissions system? 6 A Right. So we did see that under this simulation, that 7 the overall composition of the class, which, in 2000 was 8 14.5 percent minority, would be very substantially less 9 diverse with only four percent of the students being from 10 minority background. 11 Q I think you indicated that there would be somewhat 12 over a hundred fewer minority students admitted, your model 13 predicts, under the alternative race-blind policy? 14 A Right. 15 Q What would happen to the spaces in the class that, I 16 guess, those students had accounted for under the current 17 policy. 18 A Right. Well, -- 19 MR. KOLBO: Object to the form, basis, Your Honor. 20 THE COURT: I think it's a pretty obvious answer, 21 but why don't you rephrase it. 22 MR. DELERY: Okay. I'll rephrase it. 23 Q Can you tell us anything about what the model predicts 24 about where the hundred-plus spaces that had been under the 25 current policy given to admitted minority students? What 56 1 would happen to those spaces under your alternative 2 simulation? 3 A Right. Under our alternative simulation, those places 4 which look to be approximately 134 places would be competed 5 for by all of the non-minority students; that is, 6 approximately three, 2,800, whatever the number was, of 7 students that would compete for those places. That's the 8 way we've constructed the simulation. 9 Q Okay. Now, using these numbers, the predicted 10 composition of the law school class as a whole under your 11 alternative policy, did you do anything to look at how that 12 would translate into the more day-to-day activities of the 13 law school? 14 A Yes, I did. And I believe we have an exhibit that 15 displays that. Essentially, what we -- 16 Q Why don't we put the exhibit up, if we could. 17 A What we did, while that's being put up -- 18 Q -- This is 190, by the way. 19 A People at the law school supplied me with a list of 20 some of the important contexts for learning that arise at a 21 law school. They're listed here and they range in size. 22 The first-year section is the biggest one, 85 students are 23 in the first-year section where students take many of 24 their, several of their required classes. The smallest is 25 a moot court team which is just pairs of people in a moot 57 1 court, and, and there are other contexts. Each one has a 2 size. And what we did next was to ask questions about the 3 likely composition of these contexts for learning under 4 policy A, which is the current policy again; and policy B. 5 And that's essentially what we did. And I think we have an 6 exhibit that displays the results. 7 Q Okay. In your view, these, these contexts were 8 representative? 9 A I was told by the people that supplied these, actually 10 through your office, that these were the representative 11 contexts. And they cover the range of sizes of various 12 contexts. And what's really important from the point of 13 view of statistics here is the size of the context and how 14 does that then look, in terms of its ethnic and 15 composition. 16 Q Why don't we put up the next chart, if we could. 17 That's 191. What does this chart represent, Professor 18 Raudenbush? 19 A Okay. So what we've been done is asked questions 20 about the expected composition of each learning context, 21 from the standpoint of a majority student and from the 22 standpoint of, we just picked African-American students We 23 wanted to have a definite type of person, rather than a 24 minority student in mind when we thought about this. And 25 we didn't do it for all of the contexts. 58 1 We picked three represent -- three that were sort 2 of across the range of sizes. We picked the first-year 3 section, which has 85, then the second row is the half 4 section. And then the residential dormitory entryway. 5 This is an entryway of a dormitory and approximately 25 6 students would be in that entryway. 7 Q And the results for the other contexts are reflected 8 in your report? 9 A They're in my report, right. And I think you, this 10 basically captures what's going on here. I don't think 11 it's necessary to go through all these numbers. I might 12 just pick one of them and kind of explain. The first-year 13 section, the biggest context, let's take it from the point 14 of view of the majority student. 15 What's the probability that that would be 16 segregated in the sense that that would be no minority 17 students under policy A and policy B. And the answer is 18 it's a very small like likelihood. Under either policy 19 it's unlikely that there would be no minority students. 20 It's actually .00 versus .03. 21 But then let's ask another question, well, what's 22 the probability that there would be at least, at least 23 three minority students. And it could be nearly certain, 24 which is, approximately, pushing toward 1.0 under policy A, 25 whereas under policy B that would only happen two thirds of 59 1 the time. There would be a one-third chance of not having 2 as many as three in that section. 3 And then for, what's the probability that there 4 would be, at least three African-American students and at 5 least three Hispanic students in that group of 85. Under 6 policy A it's almost certain to occur. Under policy B, 7 approximately one time out of four. So it's actually not 8 likely to have that agree of diversity. That's the biggest 9 section. The effects of the policy are more pronounced 10 when we go to smaller-size sections. 11 For example, for example, just take, take the, the 12 residential dormitory, what's the probability of having at 13 least three minority students, .75 in that residential 14 dormitory, to picture, 25 people who live in the dormitory, 15 .75 probability that at least three of those people would 16 be minority under policy A. Under policy B, .08, a very 17 unlikely matter. So that kind of demonstrates what's going 18 on from the point of view of the majority student. 19 Things are a little bit different from the point 20 of view of an African-Americans student because, you know, 21 the African-American has to be in the context before we can 22 ask what's happening. So given that there is an 23 African-American, we ask questions, the following 24 questions; what's the probability that you'd be the only 25 African-American student in that context, or, you know 60 1 what's the probability of three or more of those. 2 So just, we could say, again, take, take the 3 residential dormitory example, under policy A, that's the 4 current policy -- there's a pretty small chance that you'd 5 be the only African-American student, .18, in this 6 residential dormitory. Under policy B, .69, it's very 7 likely that you would be the only African-Americans student 8 in the dormitory. And the probability of at least three, 9 at least two other African-American students would be, 10 would be relatively high under policy A, .56, at least 11 better than half, and very low, .07, under policy B. 12 So I think this gives some flavor of our 13 expectations about what would happen to the diversity of 14 certain contexts for learning under a change in policies. 15 Q All right. Now, taking all of these simulation 16 analyses together, the overall picture that you've 17 presented here this morning, what conclusions, if any, do 18 you draw about the impact of using race in law school 19 admissions at the university? 20 A I draw several conclusions. The first is that the 21 impact on the probability of admission of minority 22 candidates would be quite substantial. There would be 23 quite a sharp reduction in the probability of admission. 24 The second conclusion would be that the impact on majority 25 applicants would be modest, by comparison. There would be 61 1 a small increase in the average probability of admission 2 for majority candidates. And about that conclusion, I feel 3 considerable confidence. 4 Q And again, why do you think there is that difference? 5 A And the reason that that's, that difference occurs, 6 that is, you know, why does it effect minority students 7 more than majority students, it's simply a result of the 8 smaller pool of applicants of the underrepresented minority 9 group than of the majority group. 10 Q Now, so by giving these views and these estimates of 11 the impact of considering race and admissions, are you 12 saying anything about the extent to which the race of an 13 applicant is considered by the admissions people? 14 A No. We're not making any inferences about how heavily 15 this is being weighed by the people who are making the 16 admissions decisions. We don't have information about that 17 question. But we do have information about the impact. 18 Q And are these impacts, estimates, telling anything 19 about the relative weights of any of the factors in the 20 admissions process? 21 A No. They're not quantifying the relative weights of 22 anything in the process. 23 Q Okay. So as I think you indicated before, this 24 simulation results, simulation analysis, I should say, is a 25 different approach from the approach that Professor Larntz 62 1 took? 2 A Correct. 3 Q Is that your view? 4 A Correct. 5 Q Is that your view? 6 A That's right. 7 Q How does your simulation analysis bear on an 8 evaluation of Dr. Larntz' work? 9 A Well, I think that the simulation analysis gives a 10 framework of a policy framework. We've looking at policy 11 options faced by the law school that we can use to 12 understand the reasonableness of some of the results 13 results of Professor Larntz' work. 14 Q And in your opinion does Dr. Larntz' work provide an 15 accurate or realistic picture of the role that race plays 16 in law school admissions? 17 A And of course the answer is, no. As I stated at the 18 outset, Professor Larntz attempted to construct a 19 statistical model that could tell us the extent to which 20 race played a role. And I don't believe that we have 21 information that can enable us to do that. 22 Q And on its own terms, do you believe that Dr. Larntz' 23 approach was appropriately executed? 24 A Well, I believe that certain key methodological 25 choices that Dr. Larntz made led to a, an exaggerated 63 1 impression about the association between minority status and 2 admissions. 3 Q And what were those? 4 A Well, they're essentially -- 5 Q Just briefly and then we'll get into them a little 6 more? 7 A I'll give you three types, and I know we'll talk about 8 some of the details. 9 The first was that his analysis selectively 10 attended to the data; that is, it discarded data based on 11 the outcomes of the admission process. And it discarded 12 data that was, in fact, discrepant with the hypothesis that 13 there is a strong correlation between race and admissions. 14 That was the first. 15 The second was that his analysis was based on 16 strong assumptions, as our policy via regression, as I 17 explained the same kinds of assumptions that we had. 18 And that in one important case, I did an analysis 19 that showed that a key assumption that he made and was an 20 important one, was not true. And in the second case, the 21 other, another key assumption is like what I described 22 before. It's probably not true, almost certainly not true, 23 the problem being we don't know the impact. We can't gauge 24 the impact of the falsehood of the assumption on the 25 validity of the results. 64 1 And thirdly, the results of his analysis were 2 extremely unstable. They were very different from year to 3 year, and the size of the differences from year to year 4 really can't be explained by the process, or by the data at 5 hand. And so my conclusion is that there are aspects of 6 the methodological approach that create the instability, 7 not the admissions policy or the data. 8 Q Before we talk about those problems that you found 9 with Dr. Larntz' work in more detail, I'm wondering if you 10 could give us a sense of, of how his overall approach, his 11 conceptual framework differed from your's? 12 A Right. Well, his, his conceptual framework was, 13 again, the idea of constructing a model that would tell us 14 about the role of admissions, the extent to which they're 15 taken into account by the admissions people, which I view 16 as a very challenging thing. You have to have tremendous 17 amount of information to assess peoples' thinking and the 18 extent to which they're weighing factors. My question is 19 actually a more limited one but one that I think we can 20 approach with minimal assumptions through statistical 21 inference and still get some very useful information. It 22 doesn't tell you, it doesn't give us the answer to that 23 question, but it gives us extremely important information 24 about the impact of taking race into account. 25 Q Now, obviously you were here the other day when Dr. 65 1 Larntz testified, and there was a lot of discussion about 2 odds ratios, yes. 3 A Right. 4 Q Obviously we all remember that. 5 A Right. I'm just glad I don't have to explain what 6 they are. 7 Q Is, well, I'm going to ask you to give some examples 8 in a second. 9 A Okay. I couldn't get out of that one. 10 Q No such luck. I guess my first question, though, 11 about this is, is computing odds ratios an accepted method 12 of statistical analysis? 13 A It is. It's widely accepted. It's widely used. 14 Q And in what context is it appropriately used? 15 A Well, the thing about odds ratios is that typically an 16 odds ratio by itself doesn't tell us what we need to know. 17 It's a piece of information. But to interpret the meaning 18 of the odds ratios, we, odds ratios, we really need to know 19 something about the probabilities that went into computing 20 the odds ratio because depending on what the probability, 21 you know, an odds ratio controls a function of the 22 probabilities for each group. And depending on what those 23 two probabilities are, the odds ratio could be very, very 24 different things. So my, my general rule of thumb is to 25 always keep in mind the probabilities as well as the odds 66 1 ratios, for that reason. 2 Q And you have used odds ratios in your work? 3 A Oh, yes. 4 Q Is that right? 5 A Yes, I have. 6 Q Okay. In your opinion, do odds ratios provide an 7 accurate or appropriate way to look at the role that ratios 8 make in the law school admissions process? 9 A There's some problems with using, there's some huge 10 problems with using them alone, again, without, without 11 accompanying them with other information. Generally what 12 happens to the odds ratio is that it becomes very unstable 13 when one group or the other has a probability or, of either 14 nearly one or nearly zero. 15 Q Do you have some illustrations of that effect? 16 A Well, I thought we might actually just revisit some of 17 the odds ratios we looked at. Was that, the day before 18 yesterday I think it was, right. The day before yesterday. 19 And maybe we could even just quickly review those. I don't 20 know if we still have those charts or if we need to 21 scribble down those things again. 22 Q I think we do. I think the page that we have before 23 is gone. 24 A May I. 25 MR. DELERY: I'll move the easel out a little bit 67 1 here. 2 A Thank you. I think what we had the other day was we 3 had a group, some group. Let's call this group one, that 4 had a probability of admission of .99. And then we had 5 group two that had a probability of admission of .90, and 6 the odds ratio turned out to be eleven. 7 So, basically, this was saying group one had 8 eleven times the odds of admission of group two. And then 9 we had another example where group one had a probability of 10 admission of .999. Group two still had a probability of 11 .90. And what happened to the odds ratio was that it 12 became 111. And then just, you can see the pattern here. 13 If group one had a probability of admission of .9999 and 14 group two system had a probability of admission of .91, the 15 odds ratio went to 1,111. Now, those are, those are facts. 16 There's no problem with that. 17 The only problem is, if all we saw, if I hid these 18 probabilities, and all I saw were the odds ratios, I might 19 get the impression that those are three extremely different 20 results. Eleven times the odds, 111 times the odds, 1,111 21 times the odds. These look so different. But when I look 22 at the probabilities of admission from a practical point of 23 view, if I'm a candidate, and my probability is .99 versus 24 .90, that's about ten percentage points. 25 And I'm nearly certain to be admitted. If I go up 68 1 to .999 versus .91 it's still about ten percentage points. 2 I'm still nearly certain, but yet my odds ratio went up by 3 ten, a factor of ten. And then another factor of ten as we 4 go to .999. So all I'm saying is the odds ratio by itself 5 can create a misleading impression if you don't also see 6 these numbers. 7 Q Is there something about the mathematical 8 characteristic of the odds ratio that causes this, I mean, 9 is that the reason? 10 A The basic problem is that an odds ratio requires 11 division. And if one of the probabilities is either near 12 one or near zero, we encounter something called division by 13 zero which is prohibited, mathematically. We can't have a 14 fraction that has zero and nine. 15 Q And so what's the results of that? 16 A And so as the denominator goes towards zero, the 17 fraction increases without bound to incredibly large 18 numbers. If we keep adding nines, this thing keeps going 19 up and up and up. 20 Q And does the same pattern happen when you're talking 21 about small probabilities at the other end? 22 A Exactly the same pattern happens, so, for example, if, 23 I just switch it around. If group one had a probability of 24 admission of .10, and group two had a probability of .10, 25 the odds ratio would be eleven. 69 1 If I went from, again, group one, .0 to group two 2 .001, 111, .10 to .0001, 1,000, 111. So again, group one, 3 ten percent chance of getting in, group two, very small 4 .10, very small, very small. Ten percentage point 5 difference leads to very, very different odds ratios. 6 Q Do you have an example of a situation in which two 7 people might have similar probabilities of something 8 happening, but very different odds or a real world example? 9 A Yes, actually, I did think of one. It actually 10 involved the lottery. Suppose that, you know -- I get 11 excited about the lottery and I buy a lottery ticket. And 12 you say, well I'm going to outdo you, I'm going to buy 13 fifty lottery tickets. 14 So what would happen is your odds would be roughly 15 fifty times, mine. But yet both of us would have near zero 16 probability of winning the lottery. I mean, it's wise, 17 you'd say, I'm going to be really smart and go buy 18 thousands of tickets to the lottery. Everybody would be 19 buying. Of course they are, but. 20 THE COURT: Actually this week it's fifty-nine 21 million. There's a sign on my way home. Every time I keep 22 looking. 23 A They're doing it. They're rapidly increasing their 24 odds, but what they don't know is their probability is 25 staying right almost exactly at zero. 70 1 Q All right. Okay. If you could take the stand. 2 Professor Raudenbush, in your view does this pattern that 3 you've just described to us examples have any relevance to 4 the data we have in this case? 5 A They do. There are combinations of grade point 6 average and LSAT where the probability of admission of 7 anyone who applies to the law school is extremely high. I 8 mean, people who have near A averages who are up in the 9 upper 160's or 170's on their LSAT have an extremely high 10 probability of admission. 11 Of course in the data what we see is that the 12 proportions are something like 1.0 for minority applicants, 13 and something in the .9 range, or in a very high range for 14 majority applicants. And so in that sense, the examples 15 that I was presenting were not unusual. And something 16 similar can also, and does appear at the lower end of 17 people who have fewer qualifications where the differences 18 may be small in probability terms, but the odds ratios may 19 be big. 20 Q With that background in mind, I'd like to ask some 21 questions about the cell-by-cell analysis, that Dr. Larntz 22 conducted. 23 A Okay. 24 Q Just, let's start with a general question. What's 25 your opinion about the of the appropriateness or the 71 1 validity of that approach? 2 A Well the problem, well, one of the problems with that 3 approach is that it requires that an odds ratio be 4 computable for every single one of the hundred plus cells 5 that appear in any year in Professor Larntz' reports. And 6 since the odds ratio is not computable in a number of 7 cases, what this leads to is a discarding of data in those 8 cases where there can't, where no odds ratios is 9 computable. And this ends up discarding considerable 10 evidence that are relevant to how the university is 11 handling the admissions decisions. 12 Q And I believe we had some examples? 13 A Yes. 14 Q Of those situations? 15 A We do. 16 Q I think this is Exhibit 192. If you'd put that up. 17 Maybe, David, if you could put the easel back where it was. 18 Can you read it from there? 19 A Yeah, I can see the numbers from there. 20 Q Okay. There's very small, actually. 21 A I'll -- 22 Q I think I need to come closer. 23 A Okay. All right. 24 MR. DELERY: If that's all right, Your Honor. 25 THE COURT: Of course. 72 1 Q So I guess let me first just ask, I take it this page 2 here on the left is a little image of a page from one of 3 Dr. Larntz' reports? 4 A Yes. That's page six of six from the March 20, 2000 5 report. And we selected that page. It was just convenient 6 because it had three examples that I wanted to say 7 something about, because it has a bearing on what we're 8 discussing, and they all came from the same page. 9 And the first example, actually, the first two 10 examples involve cases in which the admissions process 11 treated people the same, in terms, they had the same 12 admission decision regardless of minority status. So in 13 the first cases, and we're looking here at, at students who 14 have relatively low grade point average. It's down 2.25 to 15 2.49, but relatively high LSAT's, 161 to 163. There was 16 one minority applicant in that, in, who had those 17 characteristics. And that person was rejected. There were 18 two majority applicants and they were rejected because both 19 people were rejected. Of course, what we know is they both 20 had the same admissions decision. There was no different 21 decision for the minority and majority applicants. But 22 because none of them were admitted, we can't compute the 23 odds ratio. So if you've developed a statistical approach 24 that requires cell-by-cell computation of odds ratios, you 25 can't compute the odds ratio. 73 1 Basically what happens is you have to discard 2 their cell. But when you discard this cell, you're 3 discarding information that's relevant to the decision of, 4 it's relevant to the decision made by the admissions 5 committee. That is, essentially, you're waiting to see 6 what the admissions committee decides. 7 And if they make it a certain decision, which in 8 this case is treating everybody the same by rejecting them, 9 discard the data. If the admissions decision had been 10 different, if, if someone had been admitted, then the cell, 11 the data would have gone into the analysis. So that means 12 the data goes into the analysis conditional on the decision 13 of the university. 14 If the university makes a decision to treat 15 everybody the same, we throw the data out. If the 16 university decides to treat them differently, the data go 17 in and we -- we don't like that situation in statistical 18 analysis. This, we don't wait to find the outcomes of the 19 data and then decide whether to use the data. We decide 20 what data we're going to use prior to, to, to investigating 21 the outcomes, or without any attention paid to what the 22 outcomes are that we're trying to discuss. 23 Q Okay. And what about the second cell here? 24 A Well, the second cell is another example of the same 25 thing but it's at the upper end of the distribution. 74 1 In this case we have people whose grades are 3.75 2 and above. This is very high. These people are getting 3 basically A's, maybe a few A minuses. Their LSATs are also 4 very high. They're 167 to 169, which I think is very high 5 up in the percentiles of that distribution so these are, in 6 terms of just grades and test scores this is a very able 7 group of applicants. 8 In 1999 there were two minority applicants. They 9 were both admitted. There were 106 majority applicants. 10 They were all admitted. So you look at those data, and I 11 think reasonable people would say, did race play a factor 12 in the decision for those people. And the answer seems to 13 be no. They had very high grades and very high test 14 scores. They all had the same decisions. The decisions 15 weren't different. However, can't compute the odds ratio, 16 throw out the data. 17 Q Well, let me ask you about that, because Dr. Larntz 18 said that these cells don't have comparative information in 19 them, as I understand it, and so a principle or fair 20 comparison should mean that you would discard them. You 21 would look at only cells where you have different results. 22 Do you disagree with that? 23 A I strongly disagree with that, and I'll try to explain 24 why. We only know after the fact that these people had the 25 same treatment. To then say, well, because they had the 75 1 same treatment we're going to throw them out, no, you can't 2 do that. The admissions decision could have gone the other 3 way. And that's what we have to think about in statistics. 4 THE COURT: Everything in every cell is after the 5 fact? 6 A Right. But we don't use or not use data, depending on 7 what we see in terms of who was admitted. The principle, 8 the actual principle, statistical principle is you use all 9 of the information in the data. 10 THE COURT: But all of the information in the data 11 that he says, at least, that he wanted to use, and that was 12 necessary was after the fact. He didn't combine after the 13 fact with before the fact. 14 A He decided which, which parts of the data to use after 15 he saw, based on the results of the admissions decision. 16 We don't, we don't decide, well, I'm not going the analyze 17 these because these people were admitted or rejected. We 18 don't. That's not the way we do it. I mean, these are 19 results that are discriminate with his hypothesis. 20 THE COURT: You were here for his testimony? 21 A Yeah. 22 THE COURT: He said as to these cells there was 23 nothing to analyze? 24 A And that's not true. That's simply not true. I did 25 the analysis myself. I used every scrap of data that there 76 1 was. We can analyze all of it. We must analyze all of the 2 data. 3 THE COURT: So you disagree with him? 4 A I strongly disagree. And I say that the reason he 5 discarded those data was because he was committed to a 6 cell-by-cell computation of odds ratios, and they can't be 7 computed. 8 Q So in other words, a different methodology would have 9 allowed all of the data to be include, is that right? 10 A That is absolutely right. 11 THE COURT: Did you do that? 12 A I did. 13 THE COURT: I expect we'll know your results? 14 A Yes. Actually all of the analyses I've reported so 15 far never, we never selected cases for the analysis as a 16 function of whether people were admitted or not. We always 17 analyzed whatever data came to us. 18 Q Now, as a statistician, if you had selected a 19 methodology and then you saw that it was leading to the 20 exclusion of a number of cases from the data, would that 21 cause you to think about your methodology in any way? 22 A It would very much cause me concern. And let me, 23 maybe I can explain a little bit in a very simple 24 straight-forward way how this could be so consequential. 25 Suppose another statistician came along and had 77 1 never seen Larntz, the, the report of Professor Larntz, but 2 had the database and decided to create cells. But suppose 3 that this statistician decided to create bigger cells. 4 Let's say, let's take everybody from, instead of 5 just LSAT from 161 to 163, I'm going to create larger 6 cells. I'm going to take everybody from 161 to 165. So 7 you'd have bigger sample sizes. What would inevitably 8 happen is that you'd have a larger fraction of the cells 9 where you could compute the odds ratios. 10 And, in fact, you could define the cells big 11 enough so that you could compute an odds ratio for every 12 cell. So what would happen, statistician No. 2 would come 13 along and define the cells somewhat differently, using the 14 same methodology, would throw away different cases, fewer 15 cases and get different results, quite different results, 16 in fact. 17 Statistician No. 3 comes along and says, I don't 18 like really, these cells, they're too big. I like really 19 small cells. I'm going to define cells that only go LSAT 20 from 161 to 162 because I want to equate people really 21 closely to every LSAT point. So I'm going to have, and 22 that would create basically roughly twice as many cells. 23 What would happen to statistician No. 3, 24 statistician No. 3 would see a lot more small cells where 25 nobody was admitted or everybody was admitted, and would 78 1 have many fewer computable odds ratios than Professor 2 Larntz and would throw out a great deal more of the 3 information. 4 So now what we have is using this methodology of 5 constructing cells and then for each cell computing odds 6 ratios, we have three statisticians using the same 7 methodology but they define the cells differently. They 8 never talked about it. They just define them differently. 9 And they're analyzing different data sets with different 10 subsets of cases that have different outcomes and they're 11 coming up with different results. That is not what we, 12 what we aim for in statistical practice. 13 Q And in your opinion, does that fact, the fact that 14 different size cells would, would lead to different 15 results, does that tell you anything about the 16 appropriateness of the cell by cell approach, in general? 17 A Exactly. Because it's the cell-by-cell computation of 18 the odds ratios that, that, which involves division, which 19 we can't divide by zero, or, or to paraphrase, Professor 20 Larntz, we can't divide infinity by infinity or zero by 21 zero. And so if you're committed to that strategy, you 22 have to discard data from cells that, where you can't 23 compute the odds ratio, but it turns out that the cells 24 you're discarding are the ones where people are being 25 treated the same; in many cases the same, as a function of 79 1 ethnicity. 2 Either they're being rejected or they're all being 3 accepted. The key thing we have to keep in mind as a 4 statistician is it could have gone the other way. In a 5 different world with a different policy, some of these 6 people would have been rejected, and we have to, our models 7 have to anticipate that the world might be telling us a 8 different story. 9 Q So would you choose a cell-by-cell approach? 10 A No, I would not. 11 Q Why don't we look at the third cell that you have here 12 on this chart. Tell us what you find significant about 13 that one? 14 A Well, this cell, actually exemplifies something we 15 discussed earlier and how an odds ratio by itself can be 16 misleading. 17 What we have here are candidates who have grades 18 in the sort of B plus range, but very high LSATs. And 19 there was one minority applicant, there was one minority 20 applicant and one minority admit. So one person applied 21 and was admitted. There were 75 majority applicants and 73 22 admits. So in terms of proportions, 100 percent of the 23 minority applicants were admitted, and 907 percent of the 24 majority applicants were admitted. 25 When we compute the odds ratio, we come up with an 80 1 infinite odds. We can't really compute that number. We 2 can't really divide by zero. But if labeled infinite, and 3 it conveys the impression that minority applicants were 4 much more likely to be admitted, and yet if we look at the 5 proportions, it's 1.0 versus .97. Those look very similar 6 and I think most people would say on balance looking at 7 that cell, applicants were treated similarly. 8 Q Would you call an odds ratio for a cell that turns out 9 to be infinity a calculable odds ratio? 10 A No, I would not because it involves division by zero 11 which we can't do. The computers won't let us do it. 12 Q Ms. Massie, the other day, or yesterday, I think, 13 asked Dr. Larntz whether infinity is an irrational number 14 or an imaginary number. What is it? 15 A Well, actually, it's not a number. In fact, if you 16 try to, if, in many computer programs, if you try to divide 17 by zero, it will print out, n-a-n, not a number. 18 Q All right. Given the, well, what do you take away 19 from the fact that the cell-by-cell approach of Dr. Larntz 20 generated odds ratios of infinity in this way? 21 A I take away that, that analyzing many, many, many 22 small subsets of data, using this method is not the right 23 way to go, and it will lead to distortions. It will lead, 24 in fact, to an exaggerated estimate of the association 25 between minority status and ethnicity, both in terms of 81 1 which data are discarded and which data are analyzed, and 2 in terms of these unstable odds ratios which veer to become 3 increasingly large, depending on the denominator and in the 4 division. 5 Q Okay. Is it fair to say that the pool sizes, in other 6 words, the number of applicants in a number of these cells 7 are quite small? 8 A Yes. That's correct. 9 Q Do you find that fact significant in any way in 10 evaluating the appropriateness of this approach? 11 A It's because they're small that so much of the data 12 are discarded using this approach. And that's really the 13 key. That's a key issue. And it's also because they're 14 small that the odds ratios become so unstable. 15 Q Now, it is the case, I mean we have put up here 16 examples of cells in which the minority and majority 17 applicants were treated quite similarly? 18 A Right. 19 Q That's what we've selected. It's also the case, isn't 20 it, that there are cells where the probabilities of 21 admission are quite different, or the proportions of 22 admission are quite different? 23 A That's true. 24 Q Where large proportions of minority students are 25 admitted and very small proportion of the majority students 82 1 are admitted? 2 A That's correct. 3 Q Have you looked at those cells in any way? 4 A We have. We've looked at those cells and I think we 5 actually have a display of one of them that will reveal 6 some of the features of those cells. 7 Q Okay. This is Exhibit 193, I believe. So am I right, 8 Professor Raudenbush, that this sample cell is taken from a 9 page of Dr. Larntz' report, March 20 of 2000, the same 10 report as the page we just saw? 11 A Yes, and it's page five of six. 12 Q Okay. Well, why don't you tell us what the cell 13 shows, and then what you take away from it? 14 A Okay. The cell is, includes people whose grades were 15 3.50 to 3.74, which is in the B plus to A minus range. 16 Their LSAT scores are 156 to 158, which is, which is 17 comparatively high. It's in the seventy-first to 18 seventy-eighth percentile so they're pretty high up in the 19 percentiles of the LSAT. They were non-residents. There were 20 seven minority applicants, and of those six were 21 admitted, six out of seven. 22 There were 73 majority applicants, and of those 23 one was admitted. So one out of 73, obviously, six out of 24 seven looks quite different from one out of 73, and 25 Professor Larntz' computed an odds ratio of 432 and a 83 1 probability value of point, less .0001. 2 Q And what does that probability number stand for? 3 A That is a test of the, what we call the null 4 hypothesis. The null hypothesis is that in this cell 5 there's no association between race and admissions. And so 6 if the null hypothesis were true, there's no association, 7 how likely is it that we would see results like this, and 8 the answer is not very likely. We, therefore, reject the 9 null hypothesis and infer there is a statistical 10 association between race and admissions in this cell. 11 Q Okay. And that's what, that's also what the odds 12 ratio indicates, is that right? 13 A The odds ratio is, it's hard to know what the odds 14 ratio really indicates by itself, 432. I mean we've seen 15 some cases where a number like that might not mean much at 16 all, but in some cases it might mean a lot. 17 In this case we can see that six out of seven is 18 different from one out of 73 and that's, in proportion 19 terms, those are pretty big differences. Of course we 20 don't know how big that difference is, we can't put a good 21 confidence interval because the sample size of minority 22 applicants is small. 23 What I mean is it's hard for us to say just how 24 big the effect is. We know there's affect. But to really 25 bound it is difficult because of the small size of the 84 1 sample of the minority applicants. 2 Q Have you looked at another way to think about the 3 effect of race in this cell? 4 A Yes. What we've done with this cell is just to do a 5 little mini-simulation, just to do the causal analysis that 6 we did earlier, but only with this cell, and it's really 7 pretty straight forward. In this cell, there were eighty 8 applicants overall and seven were admitted. So the common 9 probability of admission observed here is the seven divided 10 by eighty, which is .0875. So here's a very simple way of 11 simulating a race-blind policy. 12 Suppose that common probability of admission were 13 applied to the majority applicants and the minority 14 applicants. How many admits would we then expect under the 15 race-blind policy. So we multiple .0875 by 73, and we get 16 six. We round off, we can't admit half a person. So we 17 have to round off six majority admits and then the same for 18 minorities. We take the common probability of admission. 19 .0875, multiple by seven, and we get one minority admit. 20 So here's kind of a real simple way in which the 21 simulation works. What we saw in reality seven minority 22 admits, I'm sorry, seven minority applicants, six admits, 23 and one admit for majority. Under the race-blind policy it 24 would switch, six majority admits and only one minority 25 admit. 85 1 We can then compute the change in probabilities 2 for the majority students under the race-blind policy. 3 Under the current policy, one out of 73 was admitted. 4 That's 1.4 percent. Race-blind policy, six out of 73. 5 That's 8.2 percent. 6 So what that gauges is the causal, is -- it's an 7 estimate with uncertainty. But it's an estimate of the 8 causal impact of changing to a race-blind policy for the 9 majority students. Their probability of admission goes up 10 from 1.4 percent to 8.2 percent, which is definitely an 11 increase. It's an increase of about 7 percentage points. 12 Under either policy their probability of admission is less 13 than one in ten. And that's a way of quantifying what's 14 going on in this cell, and I guess it just shows, this is 15 kind of how we're quantifying what's happening in the cell, 16 as opposed to simply using a number of an odds ratio 432. 17 Q Do you think that 432 odds ratio quantifies how much 18 race has been considered in the admission process for the 19 applicants in this cell? 20 A No, I don't have. 21 Q And why is that? 22 A Well, the idea, to make an inference about the role of 23 race, the extent to which race was taken into account in 24 admissions, we would have to infer or assume that everyone in 25 this cell is identical, in terms of their other 86 1 credentials. If that quantifies the impact, we're assuming 2 that these people are the same. 3 Actually, let me back up. We're assuming that 4 other factors are unrelated to grade point average and test 5 score. But the basic idea is we're assuming that these 6 people are very similar in terms of their credentials. 7 Q Well, Dr. Larntz, as I understood it, said that his 8 general approach was to try to identify similar students, 9 and then look at the relative? 10 A Right. 11 Q Relative odds of their acceptance. 12 THE COURT: Using similarly as to those who 13 factors? 14 MR. DELERY: Right. 15 THE COURT: Grade point and the exam. 16 MR. DELERY: Right. 17 Q Well let me ask you -- 18 A Well actually there were a couple analyses; one 19 controlled for just grade point and LSAT, another 20 controlled for residents, gender, fee waiver, several other 21 factors. 22 THE COURT: Yeah. That was a separate analysis? 23 A That was another analysis, yeah. 24 THE COURT: But his main premise that he used. 25 A Right. 87 1 THE COURT: Only the two that, I think we're on 2 the same wave length? 3 A Yes, that's correct. 4 THE COURT: Only used those two and he explained 5 the reasons? 6 A And those are the big ones in terms of predictive 7 power, right. 8 Q Let me ask you this, if we had data for all of the 9 factors that are considered by the admissions process, and 10 we know that we don't, as we discussed earlier, but 11 assuming, hypothetically, that we had statistical data on 12 all of those factors, so that you could identify students 13 who were exactly the same. 14 A Right. 15 Q On all of the factors that the admissions office 16 considers, except that they differed by whether they were a 17 minority or not? 18 A Yes. 19 Q What would happen then to the odds ratio? 20 A In that case, it would have to be infinite. 21 Q And why is that? 22 A It would have to be infinite for this reason. Let's 23 just say there are ten factors that can, that can account 24 for admissions. And we have people who are identical on 25 all nine, nine of those ten, but they're different by the 88 1 last factor. Then that last factor must determine the, any 2 outcomes that weren't already determined by the previous 3 nine. It just, logically, has to be true. 4 Any admissions decisions that were not dictated by 5 the nine would have to be then made by that last factor, 6 and so because it's the only thing that can explain what's 7 left, the odds ratio would have to go to infinity. 8 Q So in my hypothetical example, would race have to be 9 taken into account a lot to yield the infinity odds ratio? 10 A And the answer is no. It would not have to be taken 11 into account a lot. If it were taken into account a little 12 or a lot, if it's the last, the only last thing that could 13 be effecting this decision, you would still have an 14 infinite odds ratio. 15 Q Do you have an example that could illustrate that in 16 some way? 17 A Yeah. I tried to think of something that would make 18 this point sort of clear. If I have a scale with two sides 19 on it and it's in a balance and I put something on that, on 20 one side of that scale and I see it go, one side go down, I 21 can't infer how heavy the thing was that I put on that 22 scale. 23 I mean, I could have had, there could have been 24 one pound on this side and one pound on this side and I 25 added a pound to this side and it went down. I could have 89 1 had a thousand pounds on this side and a thousand pounds on 2 this side. I could have put an ounce on this side. It 3 would have gone down, so knowing that the scale went down 4 decisively, cannot tell us how big the weight was that made 5 it go down. 6 THE COURT: Can you tell us how big it wasn't, 7 though? You can tell us it wasn't a feather or two 8 feathers or six feathers? 9 A Well, theoretically, any weight, if there was an exact 10 tie, any weight would have made the scale, even an ounce. 11 THE COURT: I'm talking about degrees? 12 A But what I'm saying is if these are in actual, in 13 absolute balance, these two sides, and any weight 14 whatsoever, no matter how small is put on this side, it has 15 to go down. So the only thing we can infer is that there's 16 more weight on this, that something was put on the side. 17 We know that something happened, that this last thing was 18 taken into account, but we can't tell how much we put. 19 THE COURT: That's right, but what I'm saying is 20 your analogy if they're both equal, this one goes down, you 21 put something on this one and it goes down just a little 22 bit, you know, you just put a little bit on if it goes down 23 a little bit more go. 24 A Well, if they're really imbalanced, it will go down, 25 it will go all the way down. 90 1 Q Can I say, to bring us back to what we're talking 2 about? 3 A Yes, a scale. A balance beam is maybe a better 4 analogy as to what I'm saying. 5 THE COURT: I don't know. It's funny, I've seen 6 them a hundred times in court. I've never used one? 7 A Let's say like a teeter-totter. I don't do those any 8 more, but, you know, it's kind of sitting there and it 9 might just be sitting there and, basically, if someone sits 10 on one end it goes down to the ground. 11 THE COURT: I got you. 12 A Now, that could be a little child or it could be, you 13 know, a huge football player. It would still go down to 14 the ground. The fact that it's on the ground doesn't tell 15 us the size of the person who's sitting on that 16 teeter-totter. 17 Q In the admissions decision we're talking about a yes, 18 no decision? 19 A Yes. 20 Q Is that right? 21 A Right. 22 Q Right. Not one of degree? 23 A Right. 24 Q If you come back to the cell here that we were looking 25 at, I think Dr. Larntz would say that he could infer 91 1 something about the extent to which race was taken into 2 account by how big that odds ratio number is, that because 3 it's so big, that must mean that race was a big factor in 4 the decisions? 5 A Right. 6 Q Do you disagree with that? 7 A I disagree with that for the reasons we've just 8 described. Knowing that the proportion went up for one 9 group doesn't tell, has no information about the extent to 10 which the people making the admissions decisions were 11 relying on that factor. It, the analysis suggests that 12 there is, that it is being taken into account, and the idea 13 that it's not, that it's absolutely irrelevant is the null 14 hypothesis. We rejected that, but the extent to which it's 15 being taken into account you can't determine from this 16 analysis. 17 Q I'd like to turn now from the cell-by-cell approach 18 that we've been talking about to the composite odds ratios 19 that Dr. Larntz generated? 20 A Okay. 21 Q Do you have an opinion concerning the meaningfulness 22 of those composite or global odds ratios? 23 A I do. 24 Q And what is that? 25 A And that is that I, I do not view them as a valid 92 1 assessment of the association between race and admissions, 2 given test scores and GPA, which is a narrow way of 3 defining what the analysis was intending to estimate. 4 Q And in your view, does the choices that Dr. Larntz 5 made about the cells that we discussed earlier, do those 6 choices have any implications for your evaluation of the 7 composite odds ratios? 8 A They do. And first let me just mention again the, the 9 question of making a methodology decision that then 10 influences which outcome data we throw away and which 11 outcome data we pay attention to. 12 In general, using logistic regression we do not 13 need to discard any cases from this, from -- there are, 14 there are no cases, no people whose data needs to be 15 rejected. And that's, I mean, in a nutshell, we can handle 16 that. 17 What Professor Larntz did was to construct for 18 every cell in the matrix that had any data at all a 19 predictor variable. Well, I should say, what he did was 20 construct for every cell in the matrix that had what he 21 defines as comparative information. He gave a very clear 22 definition of that yesterday. He defined for those cells a 23 predictor variable. That means that in his logistic 24 regression model he had approximately 100 predictor 25 variables, one for each cell that had the kind of data that 93 1 he found to be useful, or one that could be used in that 2 context, one that had the features that he described which, 3 you know, was a good description of what it was. Having 4 the 100 predictor variables, one for each cell, requires 5 that you discard cells of the type we looked at that were 6 just also discarded cell-by-cell analysis. And, and so 7 that was a decision to construct the many, many, many 8 predictor variables, one for each cell that had the same 9 consequence it did in the cell-by-cell analysis. 10 Q Okay. What about the cells that would generate on a 11 cell-by-cell basis the infinity odds ratios that Dr. Larntz 12 found? What happened to those cells in his analysis? 13 A Yeah. Well, those cells, and he pointed this out, 14 it's not as if we'd be averaging infinity with three other, 15 with a hundred other numbers. 16 But what we would tend to be averaging in many 17 cases, or combining, would be numbers that are very, very 18 high, as a function of the number of predictor variables in 19 the model, and the number and the small size of those 20 samples. And when you create a composite across a hundred 21 numbers, many of which are very high for those reasons, you 22 get an unstable composite estimator, and that's what we see 23 in this case. When I say unstable, I mean it varies from 24 year to year much more than we would expect, given the data 25 at hand. 94 1 Q I'd like to come back to the instability point. 2 A Right. 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 95 1 BY MR. DELERY: 2 Q Before we get to the instability point, I would like 3 to talk about the assumptions issues that you 4 mentioned a few minutes ago. 5 A Right. 6 Q What were the assumptions that Dr. Larntz made that 7 you found significant, and what do they tell you 8 about his analysis? 9 A Well, there were two really central assumptions in 10 this kind of analysis. When what you're trying to 11 do is characterize the difference between two 12 groups, or an odds ratio that expresses this 13 difference controlling for a large number of other 14 factors. 15 The assumption is that the size of 16 the difference is, or in this case the size of the 17 odds ratio, is invariant across all of the cells of 18 the matrix. 19 That is literally all of the cells 20 have the same true odds ratio. That's a very 21 important assumption for this analysis. 22 And I actually did some analysis to 23 check that assumption, and found that it was easily 24 rejected. Indeed, the size of the odds ratio varies 25 significantly across the cells of the matrix. 96 1 Q And by the matrix, what do you mean? 2 A The matrix meaning the GPA by LSAT grid, which is 3 what Professor Larntz was using in the analysis. 4 And this is a case where the higher 5 arc of the linear model actually is very useful, 6 because we have many, many small subsets of data. 7 It's almost like we have children in classrooms 8 and we actually have applicants and cells. 9 Q And that's what your book was about? 10 A That's what my book was about. 11 Q That's what we're talking about. 12 A With how you handle data where we have many, many 13 small subsets of data. How do we combine the 14 information in such a way, that we are not required 15 to discard the information. 16 And what we do is we have the 17 following conception. That every cell has its own 18 true odds ratio and that they have variability 19 across the cells. They randomly vary across the 20 cells. 21 The beauty of that is we only have to 22 estimate this one parameter, how much variability is 23 there across the cells. We don't have to estimate 24 each cells odds ratio. 25 And when we do that analysis what we 97 1 find is that there is very substantial variability 2 across the cells of the matrix in terms of the odds 3 ratio. 4 Now, this is something that you can 5 see by looking at the data, you can actually look at 6 the data, you can see that in the upper ends where 7 people have very high grades and test scores, 8 they're being treated very similarly. 9 And in the more middle ranges like 10 the cell we are now looking at still, I guess, the 11 one that had the odds ratio here, that the odds 12 ratios become quite large. 13 So, they do vary or the cells of the 14 matrix, and that contradicts an assumption that's 15 very important. And means that we can't 16 characterize the association between race and 17 admissions with a single odds ratio. 18 Q And just so we're clear, could you sort of expand a 19 little bit on why it's significant that this 20 assumption is wrong. What does that mean about the 21 usefulness of Dr. Larntz's odds ratio? 22 A Well, substantively one feature of it is that if the 23 actual difference in probabilities, or the odds 24 ratio is bigger in some areas than others, that's a 25 very different story then saying every person who 98 1 applies to the law school is going to be subjected 2 to this odds ratio, that's one thing. But it has 3 certain technical results. 4 If, in fact, the odds ratios vary 5 across the cells and you think they don't, what 6 happens is the standard error of your estimates 7 becomes too small. And any confidence intervals and 8 test of significance become questionable. 9 Q In practical terms what does that mean about how you 10 interpret the odds ratio over the cells? 11 A In practical terms what that means is that we can't 12 bound the size of the quantity we're trying to 13 estimate. We can't put upper and lower bounds on 14 the size of the relationship that we're trying to 15 estimate. 16 Q So it could be larger or it could be smaller than 17 the results that Dr. Larntz report? 18 A It could be larger or smaller. Generally when we 19 report a result, a number, like if I say the odds ratio 20 is 432, I typically would say, well, but what 21 are the upper and lower bounds of the possible odds 22 ratios that we might have gotten, because we don't 23 believe it's actual exactly 432. 24 To do that we need a standard error 25 that's reliable, and we can't get a standard error 99 1 that's reliable if that assumption fails. 2 Q And as a statistician if you can't get a standard 3 error to put a bounds around the number like 432, 4 what does that tell you about how much weight you 5 can put on a number like that? 6 A Well, it tells that, basically it tells you that the 7 odds ratio is greater than one, it's significantly 8 greater than one. But it doesn't quantify the 9 extent to which it's greater than one. 10 Q So, in other words, is it fair to say you can reject 11 and nullify hypothesis, but can't quantify the 12 extent beyond that? 13 A That's right. 14 Q Now, we talked earlier when we talked about your 15 models and also about Dr. Larntz's, that you both 16 had to assume that factors that you couldn't put in 17 your models were unrelated to the factors that were 18 in your models? 19 A Correct. 20 Q In your review of Dr. Larntz's work, does that 21 assumption mean anything about the significance of 22 his results? 23 A Well, anytime we estimate a logistic regression 24 equation, we are almost always required to make this 25 assumption, because it's almost always true that 100 1 there are a lot of things we don't know that are 2 important. 3 And in that regard as I mentioned, my 4 logistic regressions are vulnerable to the 5 criticism, we've discussed that. 6 The problem here was that we had no 7 way of checking to see the extent to which the 8 failure of that assumption might have affected the 9 results. We can't put upper or lower bounds. 10 We know that the assumption is false, 11 at least to some degree. It may be trivial, it may 12 be a large degree, but we can't assess the extent to 13 which the falsehood of the assumption might have 14 affected the results. 15 Q Why can't you or Dr. Larntz put a bound on his 16 numbers in the same way that you did on yours? 17 A Well, the method that I'm using--well, you could 18 actually create a bound, but it would be so wide it 19 would go from zero to infinity. I mean it would be 20 extremely wide. 21 There's just no way in this context 22 to have a strong sense of what the upper and lower 23 bounds are. 24 What we would want minimally would be 25 a confidence interval, the validity of which would 101 1 still be contingent upon the assumptions. But, at 2 least, that would be a way of bounding the quantity. 3 But we don't have that here. 4 Q You had mentioned an instability point earlier and I 5 deferred you on that, I would like to return to it. 6 What is your point about the 7 instability, I think you called it, of Dr. Larntz's 8 odds ratio? 9 A When we discussed some of the sources of it, the 10 cell by cell analysis, the instability of the odds 11 ratio itself, the choices as to which data are used 12 and not used could possibly feed into it. 13 What I did was I simply looked across 14 the years at the results from Professor Larntz's 15 reports and I look at the odds ratios. I think we 16 have an exhibit. 17 Q Okay. Why don't we turn to that, Exhibit 194. 18 Looking first at the left side of the chart, if we 19 could. Am I right that these are odds ratios that 20 were taken from Dr. Larntz's various reports? 21 A That's correct. 22 Q Okay. And there are three columns here for Model 23 One, Model Two and Model Three, what did those mean? 24 A Well, as Professor Larntz presented models, he 25 presented results from models that controlled only 102 1 for grades and test scores. 2 He had a second model that in 3 addition controlled for Michigan residents, gender, 4 fee, fee status waiver. And also numerical 5 discrepancies in GPAs, it was more elaborate model. 6 And then the third model was one 7 where we used the selection index as opposed to the 8 GPA and LSAT. 9 Q And that was a third model that we didn't hear about 10 during his testimony, is that right? 11 A Right, correct. 12 Q So then you have listed the odds ratio from his 13 reports under the three models? 14 A That's true. 15 Q What do you conclude based on the pattern of numbers 16 here across the years? 17 A Well, when I look at the numbers across the years 18 within a model, we can just take model one. There 19 is really very large variability in these numbers. 20 For example, in 1997 the odds ratio 21 for African Americans was 53.9, whereas in 2000 it 22 was 443.26. 23 Now, if we took those numbers 24 literally, it would imply that the relative 25 advantage of African Americans was basically nine 103 1 times as great in 2000 as it was in 1997. 2 Which would imply a very big change 3 in the policy. It would imply that the data would 4 look different. 5 And, of course, looking at these 6 numbers and knowing that the odds ratio itself can 7 be unstable, my first impulse was to assume, and I 8 think this was Professor Larntz's, that these 9 numbers were varying by chance. They're big odds 10 ratios. That we saw that odds ratios can become 11 unstable. 12 However, I took another step which 13 was to look at the standard errors of the 14 differences between any pair of odds ratios. These 15 standard errors are basically in the report that 16 Professor Larntz--in his report. 17 Q You can derive them from information? 18 A The standard errors from each year are derivable 19 from the report. And we can then easily compute a 20 standard error for the difference between any two 21 odds ratios. 22 And what I found was-- 23 Q Let me just interrupt for a second. 24 A Sure. 25 Q That's a standard statistical technique that you 104 1 performed? 2 A Yes. 3 Q Yes. 4 A Anytime we want to know how big is the difference 5 between two numbers, we compute what we call a 6 standard error of the difference, and how many 7 standard errors are they apart. Professor Larntz 8 referred to these as standard deviations. 9 Q Okay. 10 A What I found was that the standard deviations in 11 2000 were--I'm sorry, the odds ratio 443 in 2000 was 12 eleven standard deviations bigger than the odds 13 ratio in 1997. 14 Q And, in your opinion, what's the significance of 15 that fact? 16 A Well, that kind of a difference could stem from 17 a difference in the policy. It would have to be a 18 very big difference, leading to a very big 19 difference in how the basic data looked. Or it 20 would have to simply be a function of the 21 methodology. 22 And so if you look at the law 23 school's policy, the same policy was in effect from 24 1992 to the present. There's no reason to believe 25 that it dramatically changed, that there was 105 1 tremendously increased weight put on African 2 American admissions. 3 When we look at the data, what we 4 have on the right panel, is the percentage of people 5 admitted, African Americans versus Caucasian, those 6 number are stable, they're similar. They're not 7 that different. 8 In '97 it was 34 percent for African 9 Americans, 39 percent Caucasian. 2000 it was 36 10 versus 41, those are very similar. 11 And so my conclusion is, that the 12 instability in the result stems not from changes in 13 the policy, nor from changes in the basic data. But 14 must, in fact, be a result of the methodology. 15 Q And based on your experience as a statistician, 16 would that kind of instability in the results cause 17 you to call the model that you have chosen into 18 question? 19 A It would. 20 Q And why is that? 21 A If the process you're studying stays stable, you 22 have fairly large sample sizes for every year, the 23 data look very similar. One would expect the result 24 of the analysis also to be stable. 25 If they're not, then they must not be 106 1 reflecting the data or the policy, something else 2 must be going on. I mean you want to know what it 3 was. 4 Q Let me now sort of bring this discussion full circle 5 and ask you, you know, now that we've looked at your 6 views on Dr. Larntz's work, how do your simulations 7 and your results bear on your views of the results 8 that Dr. Larntz reported? 9 A My results using the simulations, of course, are 10 asking a much less challenging question. We're 11 asking what's the causal impact of the policy on 12 those who apply, rather than asking how much are the 13 people who are doing the admissions weighing 14 different factors. So it's a more modest question. 15 And the results, however, I think are 16 very informative about the potential consequences of 17 policy changes both for those who apply and for the 18 overall diversity of the class. 19 The results are very stable over 20 years, they can be bounded with truly minimum 21 assumptions. I mean essentially the only assumption 22 that we're making is that the probability of 23 admission for majority candidates will not go down 24 if race is abolished as a factor in the admissions. 25 So, with minimal assumptions, we have 107 1 stable results that I think are very informative on 2 the question of how using race affects the people 3 who apply. 4 In Professor Larntz's case, he was 5 trying to answer a much, much more difficult 6 question. Which is to use these limited statistical 7 data, to try to make inferences about how people 8 were making decisions, when the people who were 9 making the decisions have a great deal of 10 information that we just don't have access to. 11 And so his results in addition as we 12 see, because of methodological reasons, using 13 certain subsets of data, not using others, creating 14 unstable results, are somewhat problematic. But 15 that's not really the big point here. 16 The big point is that we can't really 17 answer the question he posed with the data at hand. 18 And I think that's the key, that's at least the 19 story. 20 Q In your view, would it be fair to say based on the 21 data, that race is a predominate factor in the 22 admissions process? 23 A No. The data do not suggest that race is a 24 predominate factor in the admissions process. 25 Q And, in your view, what do the data show about the 108 1 impact that considering race has on the admissions 2 process? 3 A They show that the impact on minority candidates 4 would be quite substantial, we suspect. And the 5 impact on majority candidates would be very modest. 6 Q And that's the impact of changing to an alternative 7 race blind policy? 8 A Correct. 9 MR. DELERY: Your Honor, at this 10 point I would move Exhibits 184 through 194, the 11 charts we used, into evidence. 12 MR. PURDY: No objection. 13 THE COURT: Received. 14 MR. DELERY: And no further 15 questions. 16 THE COURT: Does the Intervenors have 17 any questions? 18 MS. MASSIE: Yes, we will. It might 19 be a good time to break for lunch though. 20 THE COURT: Okay. We'll break for 21 lunch and you'll still get an hour and 15 minutes. 22 Why don't we argue those motions before lunch, it's 23 not going to take but a couple of minutes to do that 24 and then we'll break for lunch. 25 LEt the record reflect that we have 109 1 Plaintiff's motion in limine to exclude certain 2 Intervenor witnesses. 3 Have the Intervenors, rather than 4 spending a lot of time arguing about those you 5 intend to call, have you made a decision as to any 6 of those that have been objected to at this point? 7 MS. MASSIE: Just one second. What 8 we know for sure is that we won't be calling all the 9 ones. We won't be calling all the ones who have 10 been object ed to. 11 For example, at most we anticipated 12 calling one of the four law professors who are 13 listed as fact witness. Those being Margaret 14 Montoya, Sumi Cho, Marjorie Schultz and Charles 15 Lawrence. 16 Frankly I don't think we're going to 17 have a chance to call anyone of those four people, 18 but if we do call one it will be one. 19 In other words, there's no chance 20 that all four of them are being called. Beyond that 21 I think it's extremely unlikely of the triad of John 22 Hope Franklin, Thomas Sugrue and Eric Foner, that we 23 would seek to call more than two of those three 24 witnesses. And we might, in fact, call only one of 25 them. 110 1 THE COURT: Okay. I just thought we 2 were wasting time on that. 3 MR. KOLBO: Counsel. Your Honor, 4 Kirk Kolbo again for the Plaintiff. Our concern is 5 if I can just be brief about this. 6 In conversations with counsel for the 7 University, I think I've learned that they're 8 probably going to finish with their case next Monday 9 or so. So we'll spend about five trial days 10 altogether between the Plaintiff's case and the 11 University's case. 12 And Ms. Massie had listed, I 13 understand now, it's being diminished, but I think 14 some 27 witnesses. We're concerned about that for a 15 number of reasons. 16 And I'm not here today to talk about 17 cumulative testimony, I think that might be 18 appropriate at some point. 19 But even given the fact the court has 20 given each side 30 hours just seems to be at some 21 point testimony, I think, in any particular subject 22 can become cumulative. But that again is not really 23 what my concern today is. 24 For the witnesses that I have 25 mentioned here, it seems to me that they are, as far 111 1 as I can tell, they are addressed subjects that are 2 clearly outside the scope of the Court's order in 3 the trial of the case. 4 And, in fact, I think I've been 5 somewhat conservative in this. I think I can 6 actually find some of their other witnesses who 7 appears to me, at least, can only be relevant on 8 matters that are outside the scope of the trial 9 here, and I tried to focus on a few that I thought 10 made this point best. 11 For the most point, the witnesses 12 here that we have mentioned--and I'm not going to go 13 through them one by one, your Honor, unless you 14 would like me to. 15 THE COURT: No, you don't have to. I 16 have read everything. 17 MR. KOLBO: They seem to fall into 18 two categories. One is witnesses who will testify, 19 they're all academics, I think, in one fashion or 20 another. 21 And I'm not trying to at this point 22 exclude any of the Intervenors, we don't think that 23 their testimony given the scope of the trial is any 24 more relevant then Ms. Grutter's is at this stage, 25 we're not objecting to their testifying in court. 112 1 I am concerned about the academics 2 that seem to be offered on matters that are outside 3 the scope of the trial in two areas in particular. 4 One, it appears that the Intervenors 5 plan to have experts testify very generally about 6 historical race relations in this country, history 7 of discrimination. 8 Those aren't matters in dispute, your 9 Honor, we don't dispute Plaintiffs in this case 10 there's a long history of discrimination against 11 minorities in this country. 12 We think given the Supreme Court's 13 precedence those kinds of important issues simply 14 can't rise to a compelling governmental interest to 15 justify racial preferences. 16 And for that reason, that kind of 17 testimony isn't needed or relevant here. And these 18 experts really, even though they're talking about 19 discrimination generally, they're not experts, as I 20 understand it, your Honor, that are offered as 21 experts on standardized testing or cultural bias 22 with respect to grades and test scores. 23 They're much more general then that. 24 And it seems to me we just don't need to spend time 25 with that. 113 1 The other general category, your 2 Honor, and I'm actually more concerned about the 3 second category than the first, because it seems to 4 me this open up all kinds of possibilities, as far 5 as where this trial might head. 6 Ms. Massie has identified a number of 7 experts who would be really experts on the question 8 of whether diversity has educational value. 9 And we all know that that issue has 10 been taken under advisement by the court as a matter 11 of law. But a number of these witnesses, as far as 12 I could tell, could only be relevant on that 13 subject. 14 And a number of these professors, for 15 example, a larger majority of the group we may only 16 hear one from. But it just seems to me that that's 17 not the issue that's before the court. We have not 18 prepared ourselves to try it at that level at 19 this point. 20 And the other thing that concerns me 21 on that, your Honor is, I did see the University 22 file and I guess our response as well to our motion 23 with respect to the Intervenors. 24 And they indicated that if 25 Ms. Massie, if the Intervenors get to put on 114 1 evidence of the educational value of diversity, well 2 then they ought to be able to too. And it seems to me 3 that we're on a whole different ball game at that 4 point. 5 THE COURT: That's Professor Franklin 6 and Montoya and so forth? 7 MR. KOLBO: Yes, I think a number of 8 them tend to cross into that area. All of the last 9 four Montoya, Sumi Cho, Marjorie Schultz, 10 understanding that only one may be called now but 11 here is what each are supposed to testify about. 12 Why it's necessary to have a critical 13 mass of minority students for those students who 14 achieve their full potential. 15 That's just isn't one of the issues 16 as I understand it that we're trying in that narrow 17 scope of the trial. Those are our concerns, your 18 Honor. 19 THE COURT: Thanks. 20 MS. MASSIE: Thank you, Judge. Let 21 me say first that we don't intend to have any of 22 these witnesses if they're called to testify about 23 the educational benefits of diversity. 24 THE COURT: Good, because I was going 25 to rule in the Plaintiff's favor, because it's not 115 1 an issue here. 2 MS. MASSIE: We know that. 3 THE COURT: And the University take 4 exception to it also, because they feel that they 5 have those put some witnesses on, that they have 6 strong witnesses in which they have not put them on 7 and they were not limited. Go on. 8 MS. MASSIE: That's understandable 9 and we have no dispute with any of that. All of 10 these witnesses will go to questions that you've 11 identified at the trial. 12 They'll go to why there's a score gap 13 in the LSAT. For example, they're go to why you 14 have to take the kind of breaks in admissions to 15 move toward fairness and equality in law school 16 education. That will go to the extent to which you 17 have to take in account of race in admissions. 18 In that regard, they're not 19 completely unlike Syverud on the question of extent. 20 Mr. Kolbo was just objecting to the idea that 21 critical mass is still an issue in this case. 22 But you the other day ruled that 23 Kent Syverud could testify based in part on his 24 testimony on critical mass, which has to do with the 25 extent to which race has to be taken into account in 116 1 the law school admissions process. 2 THE COURT: Well, I've limited it to 3 a very narrow--well, I didn't limit it, but they 4 intended to call him for the very limited and narrow 5 issues. But, go on. 6 Well, I'll tell you how I'm going to 7 rule, because it's not a secret. I am going to 8 again indicate, and I just indicated to you, that as 9 to those issues that are not relevant here shouldn't 10 be presented. 11 I'm not going to tell you exactly how 12 to present your case and I don't intend to do that. 13 You have 30 hours total, I don't know how many you 14 have used so far. 15 However, I will accept any during the 16 testimony before you spend the money bringing these 17 folks in, remember that I'm ongoing to allow that 18 which is really relevant. 19 And again history, the history of 20 discrimination in this country the Plaintiffs are 21 not disputing the effects of that and so forth. 22 So, those issues that are before me, 23 and again I don't know everything that these 24 witnesses are going to testify to. I think I have a 25 total of maybe 20 pages here between all parties in 117 1 relation to this particular motion. 2 So I'm going to deny the motion. 3 However, with the understanding that both Plaintiff 4 as well as the University may make objections as to 5 relevance. If it's not relevant, I'm going to take 6 a pretty hard line on that. 7 MS. MASSIE: That makes sense. 8 THE COURT: Okay. We'll be back at 9 1:15. 10 (A brief recess was taken.) 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 118 1 (Court back in session.) 2 THE COURT: Okay, you ready? 3 MS. MASSIE: Yes, your Honor. 4 5 CROSS-EXAMINATION 6 BY MS. MASSIE: 7 Q Hi, Dr. Raudenbush. 8 A Hi. 9 Q I never introduced myself earlier, we've never met 10 before today, is that right? 11 A That's right. 12 Q I'm going to ask you, Dr. Raudenbush to turn to your 13 original report in this matter which is date 14 January 22, 1999, and I believe it is at 145, do you still 15 have it in front of you? 16 A No, I don't actually. 17 MS. MASSIE: Is it okay if I 18 approach? 19 THE COURT: Please. 20 BY MS. MASSIE: 21 Q And this is the same report you were looking at 22 earlier today? 23 A Yes, this is my first report. 24 Q Got you. Could you turn for me to page seven, 25 please. 119 1 A Okay. 2 Q What I would like to do here is if-- 3 MS. MASSIE: Judge, do you have page 4 seven in front of you? 5 THE COURT: I sure do. 6 MS. MASSIE: I'm not going to bother 7 about projecting it then. 8 THE COURT: I have it. 9 BY MS. MASSIE: 10 Q I'm going to ask you to read the paragraph that 11 begins with the word nor, close to the bottom of 12 that page if you would. And I'll just ask you to 13 read the full paragraph, and then I'm going to ask 14 you to explain a little bit what you mean by it? 15 A Okay. "Nor does the report-- 16 Q (Interposing) Can I stop you right there, I 17 apologize. When you say the report, you mean? 18 A The report of the--the first report, I believe it 19 was, of Professor Larntz. 20 Q Thanks. 21 A "Nor does the report consider the possibility that a 22 given value on the index has a different meaning on 23 average for different ethnic groups. 24 A candidates score on the LSAT and on 25 GPA may be viewed as reflecting motivation, aptitude 120 1 and prior educational opportunities. 2 Presumably, if one person has had 3 more opportunities than another and if both have the 4 same index score, the second person must have a 5 higher level of aptitude plus motivation. 6 Admissions officers have some 7 information on each component and can use that 8 information to ensure that those accepted for 9 admission are uniformly capable with regard to 10 motivation and aptitude, but diverse not only with 11 respect to ethnicity, but also with respect to prior 12 educational opportunity. 13 A sensical statistical analysis of 14 the admissions process should use all of the 15 available data to explore whether and by what means 16 the University has been able to achieve such goals. 17 But statistical analysis that equate 18 test scores in prior cases with aptitude and 19 motivation to learn law, would overstate the 20 predictor of validity of LSAT in particular. 21 That model will also be predicated on 22 the assumption that prior educational opportunity 23 had no role to play, or that the access of minority 24 applicants to prior educational opportunities is, on 25 average, equal to the prior educational 121 1 opportunities of Caucasian." 2 Q Dr. Raudenbush, what do you mean by aptitude, do you 3 mean in borne intellectual capacity of some kind? 4 A Not necessarily. It's hard to nail down the things 5 that cause people to have different aptitudes for a 6 particular subject or area of study. 7 Q So you're not necessarily referring to something in 8 a person from birth? 9 A Not necessarily. 10 Q And do you still agree with the views that you 11 expressed in this paragraph? 12 A Yes. 13 MS. MASSIE: That's all I have. 14 Thanks. 15 16 CROSS-EXAMINATION 17 BY MR. KOLBO: 18 Q Good afternoon, Dr. Raudenbush. 19 A Good afternoon. 20 Q We met once, I think, before telephonically doing 21 your deposition in the midst of a blizzard, I think? 22 A That's right. It was the longest phone call I ever 23 had. 24 Q Just for the record, my name is Kirk Kolbo and I 25 represent the Plaintiff. One of the lawyers 122 1 representing the Plaintiff in this particular 2 lawsuit. 3 Am I correct that you have, first of 4 all, you've been, I think, very careful several 5 times to indicate that you have not done any 6 statistical analysis for the purposes of quantifying 7 the extent to which race is used in the admissions 8 process at Michigan Law School, correct? 9 A That's correct. 10 Q But you have conducted a statistical analysis to 11 assess whether race is an important factor in the 12 Michigan Law School process, correct? 13 A An important predictor in terms of just the 14 correlation, not in terms of a factor. It depends 15 on what you mean by factor. But statistically to 16 predict. 17 Q I think you used the word earlier today, causal 18 factor? 19 A We look at the causal not of race per se, but of 20 using race. Of a policy that uses race relative to 21 another policy that doesn't. We look at the causal 22 effect of those two policies. 23 Q And you look to determine whether that was the 24 important causal effect in this case, correct? The 25 use of race, that is? 123 1 A The use of race, yes. 2 Q And you have concluded, have you not as a general 3 matter, that race is important causal effect with 4 respect to admissions decisions that are made at the 5 Michigan Law School, correct? 6 A It's not quite that simple, if I may explain. That 7 the magnitude of the affect is quite large for 8 minority applicants on average, but not for majority 9 applicants. 10 Q But you have concluded, have you not, that there 11 would be very important consequence for the racial 12 composition of the Michigan Law School if race were 13 not a factor in the admissions process, correct? 14 A That's correct. 15 Q And your testimony has been there would be a larger 16 impact for the group of minority students, relative 17 to the group of non-minority students, correct? 18 A That's correct. 19 Q And you, in fact, have attempted to quantify the 20 extent to which that is true, correct? 21 A I have. 22 Q And you have concluded that there would be, as I 23 understand it, very dramatic consequences in terms 24 of the reduction of minority students at Michigan 25 Law School if we went from the current policy to a 124 1 hypothetical race neutral policy, correct? 2 A Correct. 3 Q You have, I think, used terms like substantial and 4 sharp in terms of introduction? 5 A Yes. 6 Q Would you agree that that would--I think Ms. Munzel 7 the other day, I don't know if you were here for her 8 testimony, but she suggested, she's the current 9 Admissions director, that there would be a 10 devastating drop in minority admissions if the law 11 school were to go to a race neutral system and 12 everything else in the system remained the same? 13 A I wasn't present during that testimony. 14 Q Would you agree with that characterization? 15 A Well, the word devastating, I'm sure people might 16 disagree as to what would be devastating. Some 17 people might be devastated and others might not. 18 I think statistically the numbers, 19 the expected reduction in the the average 20 probability of admission which I showed, is quite 21 substantial. 22 Q Just to use one more lay person's term. Would it be 23 fair to say that the consequences would be enormous? 24 A Again, statistically the word enormous, it's a very 25 subjective word. I'd rather just stick with the 125 1 numbers and language I used in my own reports. 2 Dramatic is about as far as I went, I 3 don't think I necessarily used the word enormous. 4 Q Maybe dramatic but not enormous? Substantial and 5 sharp? 6 A Substantial and sharp. I think the size of it is 7 quite clear. 8 Q And you did as you say, I've just been using terms 9 that I understand. I'm a history major normal not a 10 statistical. I'm not sure I even took a course in 11 statistics, to tell you the truth. 12 But you didn't rely simply on English 13 language, but you quantified your findings and you 14 spent some time doing that that afternoon? 15 A That's correct. 16 Q And I want to ask you about some of that a little 17 bit later as well. 18 A Okay. 19 Q If I may use the Defendant's board over here. This 20 is Exhibit 184. This was the first slide or, I 21 guess, the first graphic that you displayed this 22 morning, Dr. Raudenbush. And I want to just ask you 23 a couple of questions about it. 24 This is a display of what, I guess, 25 would be descriptive statistics, correct? 126 1 A Yes, these is descriptive statistics. 2 Q And one of the things that you demonstrated through 3 this is for the year 2000, for example, there were 4 about 14.4 applicants who were minority students 5 UMS. I will use the same shorthand. Of all 6 minorities, you only know Asian Americans, for 7 example, are not included. 8 Generally what we're talking about is 9 UMS or minority students in the context of your 10 testimony, correct? 11 A That's correct. They're not classified as being 12 underrepresented minority students, because the law 13 school policy doesn't include them in the category 14 of people who, as I remember, have been historically 15 discriminated against and would likely be 16 underrepresented. 17 Q And if I could just have the same understanding that 18 you had with Mr. Delery this morning, unless I say 19 otherwise, when I talking about minority students, 20 I'm talking about the underrepresented minority 21 student groups that we talked about earlier, okay? 22 A Fair enough. 23 Q And the year 2000 about 14.4 percent minority 24 students applied, and about 35 percent of those were 25 admitted, correct? 127 1 A That's right. 2 Q And about the same percentage of the enrollments 3 following the yield 14.5 percent enrolled about the 4 same as the total number of applicant pool, correct? 5 A That's right. 6 Q And then you got the numbers up there for Caucasians 7 as well? 8 A Right. Actually for non-minority. 9 Q Non-minority, right. Which does include Asian 10 Americans, I think? 11 A Yes. As well as those whose ethnicity us unknown. 12 Q I just want to understand that this analysis, for 13 example, the total numbers of minority students who 14 are admitted, that's without regard to any 15 consideration of relative qualifications, correct? 16 A That's right. These are just simply descriptions of 17 who was admitted. 18 Q This involves no analysis that compares the 19 credentials of those two groups, minority students 20 and non-minority students? 21 A That's correct. 22 Q Now, would you agree that grades and test scores are 23 very important predictors for all applicants at the law 24 school? 25 A Yes, I would. 128 1 Q Grades and test scores are very important predictors 2 for minority students? 3 A Yes. 4 Q Grade and test scores are very important predictors 5 for non-minority students? 6 A Yes. 7 Q And is it fair to say then that based on what you 8 have seen in the data, the law school certainly uses 9 grades and test scores to make decisions with 10 respect to all applicants? 11 A Certainly test scores and grades play a very heavy 12 role for all subgroups of applicants. I don't know 13 about each individual applicant, but certainly for 14 all the ethnic groups, absolutely. 15 Q Speaking in terms of groups though, we can certainly 16 say that the law school uses, looks at and makes 17 decisions based on grades and test scores of all 18 applicants. 19 And that's a true statement with 20 respect to minorities, and that's a true statement 21 with respect to majority students, correct? 22 A Correct. 23 Q Is it true though that you have also found in 24 looking at the data, that the relative importance of 25 those factors, grades and test scores, in at least 129 1 the decisions that are made out of the Admissions 2 Office. 3 The relative importance is different 4 for different racial groups? 5 A The regression co-efficients are different for 6 different groups. Which doesn't necessarily imply, 7 however, that their relative to importance in making 8 the decisions is different. 9 The regression co-efficients don't 10 necessarily reflect the process that makes the 11 decisions. It's a statistical association that 12 we're looking at here. 13 Q Could you get Exhibit 146 for Dr. Raudenbush. It's 14 one of your reports, maybe you have it in front of 15 you. 16 A I have it. 17 Q It's your report dated March 3, 1999? 18 A I have got it. 19 Q And I'm on page five. 20 A Okay. 21 Q I'm on the last full paragraph. I'm just going to 22 read, I may stop to make sure that we can--correct 23 me if I'm reading things wrong, and I also want to 24 ask you if these are true statements in your report. 25 First sentence, "The evidence of 130 1 these effects", and I assume the evidence means the 2 data you looked at, correct? 3 A Now, you're starting in the middle here. Okay, I 4 see. Go ahead. 5 Q The evidence of these effects, and when we're 6 talking about evidence here we're talking about the 7 data you looked at, right? 8 A Right. 9 Q "Is presented in detail a series of logistics 10 regressions in the appendix." And you attach an 11 appendix to your report here, correct? 12 A Correct. 13 Q And continuing on, "These analysis show that for all 14 applicants grades and test scores are extremely 15 important as predictors of mission to the law 16 school." 17 And we just agree that that's true, 18 correct? 19 A That's correct. 20 Q And then you go and state, "Other factors are also 21 important for all applicants, however, the relative 22 importance of each factor differs as a function of 23 underrepresented minority status. 24 Test scores, grades, Michigan 25 residents and gender play quite different roles in 131 1 determining the probability of admissions for these 2 two groups." 3 Are those all true statements? 4 A Yes. Now, these are statements of statistical 5 association, not statements of what people are 6 thinking about when they make their decisions. But, 7 yes, that's true. 8 Q Just to quote again, "The relative importance of 9 each factor differ as a function of underrepresented 10 minority status." 11 Is that a true statement? 12 A That's a true statement. You need to though look at 13 the sentence just above it which talks about 14 important as predictors of admission. That's all I 15 mean to say. Otherwise I agree. 16 Q And again I don't want to get into too much 17 technical jargon, because I won't be able to 18 understand it or ask the right question. 19 But we're talking predictors. There 20 you're assuming that, first of all, grades and test 21 stores were used as predictors in your regression 22 analysis, correct? 23 A That's correct. 24 Q And we can fairly assume, can we not, that they're 25 predictors because the Admissions office is using 132 1 grades and test scores to make admissions decisions, 2 correct? 3 A It seems very unlikely that they would be as strong 4 predictors as they are if the Admissions people were 5 completely ignoring them. 6 And everything I know about the 7 policies says that they're supposed to be important 8 predictors. So I guess in that sense the data are 9 consistent with the policy. 10 Q So, if they are strong predictors, grades and test 11 scores, and if we can assume that the Admissions 12 Office is using them, and if we can conclude that 13 they have a different relative importance for 14 different racial groups, can't we draw a conclusion 15 that the Admissions Office is attaching a different 16 relative importance to those factors in considering 17 minority applicants? 18 A No, we can't. We definitely cannot draw that 19 conclusion from these data. I can explain that if 20 you like, but we can't. 21 Q No, I just was curiously looking for an answer. 22 A Okay. 23 Q Now, I guess I would like to go next, jumping over a 24 number of of your exhibits. I would like to go to 25 the--maybe I should just put it up, because I'm 133 1 going to be talking about it if not immediately then 2 soon, Exhibit 187. 3 And I'm not going to draw your 4 attention to this immediately, but I hope to lead up 5 to it. 6 As I understand it, you use logistic 7 regression, the same mode of analysis that 8 Dr. Larntz used. You don't have an objection to the 9 use of that mode of analysis to form some 10 comparative analysis, do you? 11 A No, I don't. 12 Q And as I understand it, you made a choice--you have 13 to make a choice in using logistic regression about 14 what predictor variables we're going to use? 15 A That's correct. 16 Q You makes some assumptions about the fact that 17 they're probably--they probably have predictor 18 value, correct? 19 A You do, yes. You start by hypothesizing what things 20 might actually predict the outcome, yes. 21 Q And in this case you choose grades and test scores? 22 A Yes. 23 Q And it's not clear to me, maybe you can just explain 24 this to me. I think you got one regression model 25 that just uses grades and test scores. And then 134 1 another one that uses a few others like residents 2 and gender and so forth, is that correct? 3 A That's correct. 4 Q Which of those regression analysis are equations 5 that you used in coming up with the comparative data 6 that you got there with Exhibit 187? 7 A We used the one that had the other factors, 8 including gender and residents. 9 Q Now, you didn't use any factors that Dr. Larntz did 10 not use, did you? 11 A No, I don't think we did. 12 Q You used nothing in addition to the factors that 13 Dr. Larntz used? 14 A That's correct. 15 Q In constructing your regression analysis? 16 A That's correct. 17 Q And I think you made this clear, and I think your 18 report makes it clear. There are a lot of factors 19 other than grades and test scores that go into 20 admissions decision making, correct? 21 A That's correct. 22 Q Nobody disputes that? 23 A Nobody disputes it, but the policy seems to list 24 quite a large number of things. 25 Q And you didn't take account any of those other 135 1 factors other than what you did, correct? 2 A That's right. 3 Q And as I understand it, you came to your conclusions 4 about what would happen to minority admissions by 5 formulating a regression equation that best explains 6 admissions decisions for majority students, correct? 7 A That's what generated these numbers, yes, that's 8 right. We checked it with another methodology, but 9 we did do what you said, correct. 10 Q And when I say, again, I apologize if I'm not very 11 precise, it's just because I can't be in some of 12 these with my limited facility of statistics. 13 But with respect to the regression 14 analysis for majority students, what you were trying 15 to do there is to come up with an equation with 16 grades, test scores and these few other factors as 17 predictors, that best explains admissions decisions 18 for majority students? 19 A That's correct. Actually we did estimate those for 20 both majority and minority, but in generating these 21 we actually used the resulting majority equation. 22 Q And the equation--to get the equation that best 23 explains the result, what you're doing, are you not, 24 you're kind of working backwards to try and figure 25 out how much weight in the equation one would have 136 1 to give factors, the predictor variables like grades 2 or test scores, in order to get the best explanation 3 for the result, correct? 4 A That's true. But I'm not actually creating those 5 weights. Those weights are being estimated from the 6 data. I just want to make sure. 7 Q Absolutely, I understand. We're working backwards, 8 you're trying to explain, you see the outcome and 9 now you're trying to work backwards with an equation 10 to understand how that outcome can be best 11 explained? 12 A Can be best predicted. 13 Q It's not a perfect prediction, because there are 14 other factors? 15 A That's correct. 16 Q But you're trying to come up with an equation that 17 best predicts what happens, correct? 18 A That's correct. 19 Q And to do that, or ultimately looking to get the 20 result, you end up with weights being assigned to 21 whatever the predictor values are? 22 A That's correct. Those are estimated from the data 23 and then we use those to predict the probabilities 24 of admission, correct. 25 Q So, there's a weight that you end up with through 137 1 this equation, following based on whatever effects 2 however, that would be assigned to grades and test 3 scores? 4 A Correct. 5 Q As well as the other few objective factors that you 6 looked at? 7 A That's right. 8 Q And that was just for majority students though, 9 right? 10 A We did it for both, but as I said the one we 11 actually used for these are listed in the equation, 12 correct. 13 Q So, there's a separate regression equation that 14 would best explain, best predict I guess is the 15 word, admission outcome for the minority students 16 considering the same predictor variables, correct? 17 A That's correct. 18 Q And it's different because, again, you found that 19 there is a different relative importance, at least, 20 in terms of the effects with respect to how grades 21 and test scores are considered for these two racial 22 groups, correct? 23 A That's correct. 24 Q Could I ask Wayne to put up on the board, I guess at 25 this point, I think it's the second to the last page 138 1 of Dr. Raudenbush's report. It's Appendix A1. 2 A Same report we're looking at here? 3 Q This is the same report and it's Table A1, it's the 4 second to the last page. 5 A I have got it. 6 Q Believe me, I'm not going to ask a lot about this 7 because I'll be totally lost very soon. 8 But do I understand here you have 9 reported the equations for the two different racial 10 categories that you designed, or I should say 11 figures out the regression equation for? 12 A That's correct. 13 Q On the left-hand column on the left side there is 14 the regression equation for whites and Asians? 15 A That's correct. Yes, that actually does include 16 Native Americans. I checked on that, it include 17 actually white--I'm sorry, whites and Asians and 18 blacks, Hispanics and Native Americans, yes. 19 Q Whites and Asian America? 20 A That's correct. 21 Q That's the non-minority group? 22 A Right. 23 Q That's the the equation on the left? 24 A Yes. 25 Q That you used to assess your--to form your 139 1 conclusion about what would happen under a race 2 neutral system, correct? 3 A That's correct. 4 Q And on the right is a separate equation, correct? 5 A Correct. 6 Q A different equation. And that's the equation that 7 best predicts admissions decisions for the minority 8 students, at least, considering these predictor 9 variables? 10 A Correct. 11 Q And they're different again because we have 12 concluded in looking at the data that the relative 13 importance of these factors is different, correct? 14 A Yes. The intent of which they actually predict the 15 outcome is different. Of course you can see that 16 they are, as we said before, very important for both 17 groups. 18 Q Right. 19 A But they're different. 20 Q Very important and very different, correct? 21 A They're different. 22 Q Okay. 23 A The extent of the difference, of course, is somewhat 24 different in different years. But there tend to be 25 some difference. 140 1 Q One can measure the extent of the difference, 2 correct? 3 A In all of my reports, in fact, I give them year by 4 year as you can see. 5 Q Right. You have, I think, you just said it, but you 6 did very similar analysis for each of the admissions 7 data, years that we have, correct? 8 A That's right. 9 Q Your supplemental report dated March whatever one 10 you're looking at, March 3, 1999, this is for these 11 particular years. Later reports, I think, include 12 this data at least for '99 and 2000, I think, 13 correct? 14 A That's right. 15 Q Did you do these--I don't even remember, did you 16 construct these separate regression equations for 17 all of the years in question? 18 A I did. 19 Q Okay. In all cases they were different equations? 20 A They were sometimes more similar and sometimes they 21 generally were statistical different. They were 22 different more often then they were the same or 23 similar. 24 Q Am I correct that one of the premises in your 25 opinions, is that you use the term in your report 141 1 and I'm looking at page seven, if you want to take a 2 look at this. 3 Underrepresented minority students 4 are disadvantaged with respect to GPA, test scores 5 and other factors relative to other racial groups, 6 that's language that you use there? 7 A Yes, that's language that I used, right. 8 Q When you used the word disadvantage there, is that a 9 statistical term? 10 A Yes. 11 Q And does that mean that they have lower test scores 12 and grades? 13 A They have lower means on something that's related to 14 the outcome, correct. In this case grades and test 15 scores. 16 Q And it was because--and you have got a reference to 17 other factors here. What other factors did you 18 determine that minority students were disadvantaged 19 with respect to relative to other racial groups? 20 A Actually in this data set I actually don't, in the 21 law school data set, I can't think of any other 22 factors where I have evidence that they were 23 disadvantaged with possible exceptions of alumni 24 status. 25 This was perhaps a typo, because I 142 1 wrote a similar report in the undergraduate case 2 where there were other facts. 3 Q I understand that, that's happened to me a lot. 4 A It's embarrassing, but that seems to be what's here. 5 Q And do I understand that there was a significance 6 for your comment on that in your report, that there 7 was a disadvantage with respect to these factors. 8 The significance of that was that you 9 concluded that to eliminate minority status as a 10 consideration while maintaining these other criteria 11 in place, I assume to be test scores and grades, 12 will presumably reduce the probability of admission 13 of minority students possibility substantially? 14 A That's correct. 15 Q So, that was, at least, one of the reason you 16 thought that might occur? 17 A That's right. 18 Q And you, in fact, did an analysis that confirmed 19 your conclusion in that regard, is that correct? 20 A That's correct. 21 Q Correct. I think that's all I'm going to need for 22 that. You may want to put up again the graphic I had 23 up, Exhibit 187. 24 Now, that we have got some foundation 25 for it, I've had a chance to ask you some questions 143 1 about the equation, the non-minority equation that 2 you used to assemble your differences and 3 probability. I want to ask you some questions about 4 the conclusions. 5 You have concluded for the year 1995 6 that if one were to take the--if one were to run the 7 minority admissions, the data that you got to the 8 majority regression equation, one would see a drop 9 in minority student admissions from .26 to .04, is 10 that correct? 11 A In '95. 12 Q In '95? 13 A Right. 14 Q That's about, what, about a 85 percent drop? 15 A It's something like that, I guess. 16 Q Very substantial? 17 A Yes, very substantial. It's actually the biggest 18 one in all the years. 19 Q And that's just as a result of changing one factor 20 in the admissions process, correct? 21 A That's correct. 22 Q That's just under the assumption that you remove 23 race as a factor in admissions? 24 A Correct. 25 Q And the other part of the assumption, or at least 144 1 one other assumption is, that everything else stays 2 the same in the admissions process, correct? 3 A Well, we have to assume that everything else stays 4 the same. What we're able to actually control for 5 unfortunately is simply grades and test scores. And 6 Michigan residents and gender which are not very 7 important as predictors. 8 So, essentially we're using grades 9 and test scores, and we're forced to assume that the 10 other things are operating in the same way and not 11 correlated with those. 12 Q As we talked about earlier, there's a lot of factors 13 that are considered? 14 A Right. 15 Q Grades and test scores are important for everybody? 16 A Right. 17 Q They're very important. And your assumption would 18 by running the minority applicants through the 19 majority equation, that the importance to the LSAT, 20 for example, would stay the same, correct, as it is 21 today? 22 A That's right. The predictive power of it would be 23 the same, correct. Not the importance of the 24 admissions decision, but the statistical importance 25 in doing the predictions would stay the same. 145 1 Q It would remain as important a factor under your 2 comparative analysis as it is today, at least, for 3 the majority students, correct? 4 A Correct. 5 Q And other factors--that's true with all the other 6 factors as well, grades would have the same? 7 A Grade will, yes. The other factor we is data on, 8 yes. 9 Q Everything is held constant? 10 A Right. 11 Q With only one exception? 12 A Right. 13 Q And that's the removal of the consideration of race 14 in the process? 15 A Right. 16 Q And just with that one factor you get this very 17 substantial drop in the admissions, correct? 18 A Correct. 19 Q Now, you've indicated that there would be a change 20 also with respect to the non-minority students, and 21 you've indicated that whereas with minority students 22 there would be a negative impact. 23 With respect to non-minority 24 students, there would be a positive impact with 25 respect to more offers of admission? 146 1 A The average probability of admission would go up 2 with majority students if race were eliminated as a 3 factor. 4 Q And you've made clear that the difference in the 5 average probabilities would be substantial, that is 6 a much greater impact with respect to the minority 7 groups versus the majority? 8 A Yes, their probabilities would go down quiet 9 substantially. The majority probabilities would go 10 up, but not very much. 11 Q But I want to be clear about something else. That's 12 comparing the two groups, correct? 13 A Correct. 14 Q Now, with respect to individuals, the change is 15 going to be--there's going to be a number of 16 individuals who are minority students who are not 17 going to be admitted because of this change to a 18 race neutral system, correct? 19 A Correct. 20 Q And do I understand your analysis to be, or the 21 consequence of your analysis to be, that those seats 22 in the class will then be filled by non-minority 23 students? 24 A What actually would happen would be that there would 25 be a small addition to the number of seats that 147 1 about 3000 people would be competing for. 2 Q But the absolute numbers would be on a one to one 3 ratio, wouldn't they? 4 A The absolute numbers in terms of the composition of 5 the final student body? 6 Q For every minority? 7 A For those admitted, yes. The number admitted, under 8 our simulation the numbers of minority 9 students--this is an assumption, that the numbers of 10 minority students--the difference of the number 11 admitted would be equal to the gain in the number of 12 majority students admitted. I think that's your 13 point. 14 Q The absolute numbers are on one to one ratio, 15 correct? 16 A Correct. 17 Q So, for every minority student who is out, there's 18 presumably a majority students who wins? 19 A Somewhere out there somebody will win that extra 20 seat that about 3000 people competing for. 21 Q We just don't know who that is? 22 A That's correct. 23 Q Just like we don't know which minority student is 24 going projected? 25 A Right. We assume that the credentials of those 148 1 people would be evaluated, and the people with the 2 best credentials would be the ones who win the seat. 3 Q Can we just, maybe, take an example. I don't 4 suppose you have a calculator with you? 5 A I do. 6 Q You may not need it for this one. 7 A I hope I don't, but I have one. 8 Q I think the first slide we had up there with the 9 total numbers, yes, let's take that one. And I'll 10 keep in front of me, I think it would be hard to 11 have two of them up there at the same time. 12 But in 1995 we had 262--actually, 13 let's go down to the bottom. 14 A These are all 2000 data. 15 Q I'm sorry, 2000. Let's go to the bottom of the 16 chart you have 484 minority applicants, correct? 17 A Correct. 18 Q And what I see, at least, on my Exhibit 187 is that 19 under the current system, 35 percent were offered 20 admission? 21 A Right. 22 Q And under, at least, in Exhibit 187 if you go to 23 Policy B the race neutral system we go to four 24 percent admitted. 25 A Okay. No, that wasn't in 2000. 2000 it was ten 149 1 percent. 2 Q I'm sorry, I'm getting confused. Ten percent. 3 A Okay. 4 Q Ten percent under the race neutral system in the 5 year 2000? 6 A Right. 7 Q And so that's actually 35 percent is actually two 8 and a half times ten percent, right? 9 A Three and a half, I think. 10 Q I told you I would embarrass myself what that math. 11 We have even got to the statistical. 12 Can you calculate what that would 13 mean in the year 2000 in terms of the number of 14 minority students? 15 A If the proportion admitted were ten percent out of 16 the 484 who applied, there would be 48 admissions. 17 Q Okay. 18 A 48.4. But we can't admit that point four person, so 19 we make it 48. 20 Q I understand. And that's about 120 21 fewer admissions? 22 A Right. 23 Q And so presumably there would be 120 more offers of 24 admissions to other racial groups? 25 A Correct. 150 1 Q Did you ever assess what the overall effect on the 2 minority composition, and here I'm going to use the 3 term differently. I'm going to use the term, let's 4 include Asian Americans and any other groups other 5 than UMS students. 6 A So majority you're referring to? 7 Q Well, Asian Americans and I presume there are other 8 racial groups that are not necessarily majority 9 groups. Caucasians, Asian Americans? 10 A Yes. 11 Q Did you do any analysis-- 12 A (Interposing) Maybe we should take Asians so I can 13 understand where you're going. 14 Q Okay. Let me back up a little bit. I'm wondering 15 whether you did any analysis to determine what the 16 overall impact would be on minority admissions at 17 the law school, as a result of going to a race 18 neutral system, including Asian American in the 19 definition of minority students? 20 A No, I didn't do that. I just used the definition of 21 underrepresented minority students that was in the 22 1992 policy. 23 Q Is it fair to presume that because Asian Americans 24 are sort of considered as majority students for 25 these purposes here, that some of those seats that 151 1 would open up, would be seats which would be 2 competed for Asian Americans? 3 A I think I now understand the analysis that you might 4 be suggesting. I did do an analysis, I didn't 5 record it in my reports, but I did do an analysis of 6 how the average probability of admission would 7 change for Asian Americans under Policy A and 8 Policy B. And the increase is exactly the same as 9 it is for Caucasians. 10 Q Actually I was just trying to figure out what the 11 total minority population would be of the law school 12 under the race neutral system that you have 13 suggested as a hypothesis? 14 A No, I didn't use any definitions of minority other 15 than the underrepresented minority status definition 16 that appeared in the 1992 policy. 17 Q Did you consider using any other frame of reference 18 to assess what would happen under a race neutral 19 system, other than the current system? 20 A No. You know, our Policy A was always based on the 21 data that we had. And Policy B was the simulated 22 alternative. 23 Q Would it have been possible to have assigned 24 different values to LSAT and grade points as 25 predictors, and determine what the different effect 152 1 would be for different values? 2 A It would be very possible to do that. 3 Q You didn't do that though? 4 A I didn't do that. 5 Q And one could, I suppose even, eliminate the LSAT 6 and then make some predictions based on what would 7 happen to minority enrollment, correct? 8 A Well, the problem with that would be, if the current 9 policy used both LSAT and GPA and then we came up 10 with an alternative policy, how would we know what 11 the association would be between GPA and the 12 probability of admission. 13 See we always base that on the data 14 that were available, and the data that was available 15 were based on the current policy that uses both LSAT 16 and GPA. 17 Certainly you could say, let's just 18 assigned a weight of 1.0 to the grade point average 19 generate predicted values and then do the 20 simulation, and then compare that to what we do now. 21 Q Well. I understand that point. In other words, 22 it's kind of a purely academic exercise to 23 understand where you might want to be with respect 24 to different values, correct, different weights? 25 A Well, generally what I would try to do as simulation 153 1 would be to make sure that if I'm comparing two 2 policies, Policy A and B, would be make sure that 3 I'm simulating important policy relevant 4 alternatives. 5 And that's how I would construct the 6 A and B in this case. We tried to tailor it just in 7 such a way that it would really be relevant to the 8 policy options that are kind of at issue. 9 Q Well, one possibility is if the Admissions Office 10 decided to use LSAT scores and grades in a less 11 important way then they are today, and they were 12 admissions decisions generated as a result of that 13 process, one could then construct a new regression 14 equation that best predict outcomes under that new 15 system, correct? 16 A Yes. In general we ought to be able to try to, if 17 we had a realistic policy alternative, try to 18 simulate what would happen as long as we have the 19 data that are relevant to that alternative. Some 20 data that are, at least, relevant for that 21 alternative policy. 22 Q You made, in your testimony you made some statements 23 or assumptions about what might happen to the yield 24 under the hypothetical system you proposed? 25 A Yes. I speculated that changing the average 154 1 probability admission for a particular group could 2 effect the yield. Although, to highlight really the 3 point in our simulations, we assume that the yield 4 would be the same. Which may well not be true. 5 Q That's something that's kind of outside your 6 expertise, isn't it, like how the yield might be 7 effected? 8 A Not entirely outside of my expertise. I know that 9 in graduate admissions that I'm involved in that 10 people who have the highest grades and test scores 11 generally have the lowest probability of accepting 12 an offer. 13 And we assume, and it seems to be 14 true when they check into it, that they have offers 15 from other highly prestigious universities. 16 Q I have just a couple of questions for you. Well, 17 let me back up. 18 You've indicated, I think, you've 19 acknowledged that there are problems with these 20 models regression equations, because there are all 21 of these other factors that are out there, and one 22 can't take account of all of them? 23 A That's correct. 24 Q But my understanding is, you're pretty confident 25 about your conclusions here with respect to what 155 1 would be the consequences of going to a race neutral 2 system, am I correct in that assumption? 3 A We're able to check, to create what we call bounds, 4 upper and lower bounds, on the causal effect for the 5 majority students, that don't depend at all on the 6 assumptions of the regression equation. 7 And in that sense to sort of bound 8 how much uncertainty get through into the system by 9 the fact that we don't know all of this stuff. 10 Q And as a result, I think you said you did another 11 check on this as well, for example? 12 A Yes. 13 Q And as a result of these checks that you get, you're 14 pretty confident then about your results here, in 15 terms of what would be the relative probability 16 changes in going to a race neutral system? 17 A Yes. And I might add just to reiterate something I 18 did mention this morning. We're more confident 19 about the bounds for majority students under the two 20 policies then we are minorities students, because 21 the bounds are narrower. 22 Q Are you confident that there will be a--all other 23 things being held equal, and if one were to go to 24 the race neutral system under the hypothesis that 25 you have explained, are you pretty confident that 156 1 there would be these very dramatic substantial sharp 2 reductions in the admission of minority students? 3 A Yes, I'm quite confident that they would be 4 substantial. The bound is just--there's a little 5 more uncertainty there because the bound is wider, 6 but, yes, 7 Q And just to be clear on this. This decline that you 8 have testified to it, the only thing that accounts 9 for that, at least, in your statistical analysis is 10 the removal of race as a factor in the admissions 11 process, correct? 12 A Yes. Well, the question of why is it that there 13 would be a substantial reduction is more complex 14 then what you just said. But the policy change 15 that's generating it is this change, that's right. 16 I mean again, it's contingent 17 upon--basically there's two other factors that are 18 critical in making that, in effect, large. 19 One is the fraction of all people who 20 are admitted. The fact that this is a selective law 21 school, lots of people apply, most people are 22 rejected. 23 And secondly, the fact that there is 24 a strong association between grids and test scores 25 on the admissions decision. And if those two things 157 1 are true, then differences between two groups 2 minority and majority in those grids and test scores 3 can translate into--even if those differences 4 aren't very large, can translate into big 5 differences in the probability of admission under 6 the new policy. 7 So, I just wanted to create a little 8 context there to understand why this is occurring 9 when you change the policies, because of other 10 conditions in the system. 11 Namely the selectivity of it and the 12 reliance on grades and test scores that make that 13 happen. 14 Q And the relevant importance of those factors for 15 race is relevant as well, correct? 16 A Not for race, it's just that minority and majority 17 students have different means on two variables that 18 are very strongly predictive of admissions. 19 If those variables weren't so 20 strongly predicted of admissions, then you wouldn't 21 see such a big difference. 22 Also if the school were less 23 selective, if the number of people admitted were 24 more similar to the number who apply, you wouldn't 25 see those differences. 158 1 So, to understand those differences, 2 you really have to take into account the dynamics of 3 the system. So you change one variable and it has 4 that effect because of how the system works. 5 Q Just a couple of questions on odds ratios. Do you 6 use odds ratios in the statistical work that you do? 7 A I do. 8 Q You sometimes find odds ratios report calculated 9 value of infinity? 10 A I don't. 11 Q You've never seen one of those? 12 A Typically to have an odds ratio that has an infinity 13 you have to have a very small sample size. You have 14 to have one or the other of the two groups. You 15 have either all have the event occur, or none of 16 them have the event occur. 17 And then the data I worked at, I have 18 never found a data set where--I mean I don't usually 19 analyze data where, let's say, everyone drops out of 20 high school, or everyone goes to college, those 21 kinds of data. I've never analyzed data that have 22 those kinds of numbers. So I wouldn't see those 23 large odds ratios in those kind of data. 24 Q Well, just let me ask you. If you were to see that, 25 let's say, just forget about law school for a 159 1 minute. If you were to see a series of odds ratios 2 analysis, perhaps we've all used a drug, for 3 example. 4 Which ten patients were administered 5 a drug and ten of them were cured. And in another 6 hospital a placebo was administered to 50 people and 7 one of them was cured. 8 Would one be able to compute relative 9 odds for those two groups? 10 A Yes, you could. 11 Q Wouldn't relative odds be infinity? 12 A Generally if we had those data we wouldn't do a 13 statistical analysis, because if everybody is cured 14 we don't need statistics to know it. 15 Q Well, as a matter of statistical principals, do 16 those figures yield comparative information? 17 A Those figures? 18 Q Yes. 19 A Sure. If I had a drug that everybody was cured and 20 then basically nobody was cured, that would be 21 statistical information, right. 22 Q It yields comparative statistical information? 23 A Yes. 24 Q If it could calculate the value of the odds ratio, 25 of the relative odds infinity? 160 1 A Yes. 2 MR. KOLBO: May I confer with my 3 colleague, your Honor? 4 THE COURT: Of course. 5 MR. KOLBO: Your Honor, I have no 6 further questions. 7 THE COURT: Defense has any other 8 questions? 9 MR. DELERY: Just a couple of 10 questions, your Honor. 11 THE COURT: Sure. 12 13 REDIRECT EXAMINATION 14 BY MR. DELERY: 15 Q Professor Raudenbush, Mr. Kolbo used the term weight 16 several times when talking about the co-efficients 17 in your regression equations? 18 A That's correct. 19 Q Do you recall that? Am I right that weight has a 20 technical term? 21 A It does. 22 Q I mean a technical meaning in that sense? 23 A It does. 24 Q Do those co-efficients correspond to the weight that 25 the Admission officers give the various factors when 161 1 they're making admissions decisions? 2 A No. 3 Q And when you use the term relative importance as a 4 factor in the discussion earlier, am I right that 5 you were not talking about the relative importance 6 that Admissions officers gave the factors when they 7 were making admissions decision? 8 A You're correct, I was not doing that. 9 Q We saw a moment ago the two regression equations 10 that Mr. Kolbo discussed with you? 11 A Yes. 12 Q Can you conclude anything from the fact that there 13 are two equations about how the factors are actually 14 being considered by the Admissions officers when 15 they're making the decisions? 16 A No, you can't. 17 MR. DELERY: No further questions, 18 your Honor. 19 THE COURT: Okay, you may step down. 20 (Witness excused.) 21 THE COURT: Thank you. I forgot, who 22 is your next witness? 23 MR. PAYTON: My next witness is 24 Dennis Shields, I think he stepped out. 25 THE COURT: No problem. 162 1 MR. PAYTON: I'll go get him. 2 THE COURT: We can take a little 3 break now and go from there. Take our afternoon 4 break. 5 (Court in recess.) 6 (Court back in session.) 7 THE COURT: You maybe seated. 8 Dean Shields. 9 DENNIS SHIELDS, 10 was called as a witness at approximately 2:40 p.m. 11 after having been first duly sworn to tell the 12 truth, the whole truth and nothing but the truth. 13 14 DIRECT EXAMINATION 15 BY MR. PAYTON: 16 Q Would you state your name for the record? 17 A Dennis J. Shields. 18 Q Mr. Shields, where do you currently live? 19 A I live in Durham, North Carolina. 20 Q And what do you currently do? 21 A I'm the assistant Dean for Admissions and Financial 22 Aide at Duke University School of Law. 23 Q And how long have you been at Duke? 24 A I've been at Duke three years. I moved there in 25 January of 1998. 163 1 Q It is just about the anniversary? 2 A Yes. 3 Q And what did you do before you were the director of 4 Admissions and Financial Aide at Duke? 5 A I was the assistant dean and director of Admissions 6 at the University of Michigan Law School. 7 Q And when did you come to the University of Michigan 8 Law School? 9 A I started as of July of 1991. 10 Q And before you were the director of Admissions at 11 the University of Michigan Law School, what did you 12 do? 13 A I was the assistant dean for Admissions and 14 Financial Aide at the University of Iowa Law School. 15 Q Okay. And when did you start at the University of 16 Iowa in Admissions? 17 A I just started as a third year law student in 1981. 18 Q You went to Iowa Law School? 19 A Yes. 20 Q You graduated from Iowa Law School? 21 A Yes, I did. 22 Q So, you started as a third year law student, when 23 did you start after law school, when was the first 24 time you started in the Admissions Office? 25 A Right after I graduated, that May. 164 1 Q And at what point did you become in charge of 2 Admissions at Iowa? 3 A I believe that was 1985. 4 Q Is it fair to say that you have been in law school 5 Admissions for about 20 years? 6 A This is my 20th year. 7 Q A little scary, isn't it? 8 A Yes. 9 Q And you've been in charge of Admissions at three 10 schools, Iowa, Michigan and Duke for 15 years? 11 A Yes. 12 Q Okay. Are you in any professional organizations 13 that relate to law school admissions? 14 A Well, I've had extensive affiliations over time with 15 the MBA, the Law School Admissions Council. I 16 currently serve as council member on the Council of 17 Legal Education and Opportunity. I'm a member of 18 the National Bar Association. 19 Q The Law School Admissions Council, what is that? 20 A Well, that's the entity that is essentially 21 responsible for the administration of the law school 22 admissions test, and the law school data assembly 23 service. 24 Q Okay. 25 THE COURT: Is the name of that 165 1 organization is the law? 2 A Law School Admissions Council. 3 THE COURT: Thank you. 4 5 BY MR. PAYTON: 6 Q And what has been your affiliation with it? 7 A I served on a number of different committees, 8 Minority Affairs Committee, I was chair of the Audit 9 Committee. I was a member of the board of the 10 Law School Admissions Council for a total of six 11 years, I believe. 12 Q I want to focus your attention on your tenure as the 13 director of Admissions at Michigan. 14 How did you come to be the director 15 of Admissions? 16 A I believe the associate dean for Student Affairs, 17 Susan Ekland, wrote me a letter in late 1990 or 18 early 1991, and invited me to submit a resume for 19 consideration. 20 Q They found you? 21 A Yes. 22 Q And you then underwent a process--you heard 23 Professor Lempert and President Bollinger discussing how you 24 came to actually be hired? 25 A Yes. 166 1 Q You were present in court for that testimony? 2 A Yes, I was. 3 Q Was that accurate? 4 A Yes. 5 MR. PAYTON: I'm not going to go over 6 that again. 7 THE COURT: Yes, that's fine. 8 BY MR. PAYTON: 9 Q Did you know Allan Stillwagon? 10 A Yes, we knew each other. Not well, but knew each 11 other. 12 Q How did you know Allan Stillwagon? 13 THE COURT: I have one question 14 before you get into that. How did you happen to get 15 into admissions, just fall into it, or was it 16 like--I'm curious? 17 A My mentor who was then the dean of Admissions at 18 Iowa, and is now the dean of the law school at 19 Ohio State asked me if I wanted, he had a half time 20 position, and asked me if I wanted to do it. 21 And I actually thought when I 22 graduated he had made it a full time job that I 23 would do it for a couple of years until I decided 24 what I wanted to be when I grow up. 25 So, now it's 20 years later area and 167 1 either I haven't grown up, or I haven't decided what 2 I want to be. 3 THE COURT: Or it's something you 4 really like. Again, I am just amazed because, you 5 know, it's such an important position in the 6 acadame. But most as I have heard so far have come 7 from areas out of the acadame. 8 BY MR. PAYTON: 9 Q How did you know Allan Stillwagon? 10 A Well, he was the director of Admissions at the 11 University of Michigan. There is a lot of 12 recruiting travel which you do, we be at the same 13 events, the annual meeting of the law school 14 Admissions Council. So we bump into one another. 15 Q So you knew him before you came to Michigan? 16 A Yes. 17 Q Did you ever have a conversation with him about how 18 he did admissions at Michigan? 19 A No. 20 Q Once you came to Michigan in the summer of 1991, did 21 you call him up and ask him what had been going on? 22 A No. 23 Q Actually have you talked to Allan Stillwagon since 24 you became the dean at Michigan in the summer of 25 1991? 168 1 A We had pleasantries here in the courtroom. 2 Q That is on Tuesday? 3 A On Tuesday. Other than that, I have not even laid 4 eyes on him since 1990 maybe. 5 Q Now, one of the first things that happened when you 6 came to Michigan, was that the dean, then 7 Dean Bollinger, put you on the faculty Admissions 8 Committee that was charged with coming up with a new 9 policy, is that right? 10 A Yes. 11 Q I'm not going to go into how the committee 12 functioned either, you heard Professor Lempert and 13 you heard Dean Bollinger. 14 Did they accurately describe how that 15 happened and how the committee functioned? 16 A Yes. I appreciated that. 17 Q But I do want to ask you this which is, what the 18 opportunity to serve on that committee looked like 19 to you having just arrived at the University of 20 Michigan Law School? 21 A Well, it was a tremendously exciting time for me. 22 I was coming to what is already believed to be one of 23 the finest law schools in the country. I had been 24 asked to take on a major role. And I think 25 President Bollinger admitted this is an important 169 1 aspect of the law school life. 2 I was going to be working on this 3 committee with a very distinguished group of faculty 4 members that had expertise in areas that I knew less 5 about then they did you. 6 But it was also an opportunity for me 7 to walk into a situation where in many ways I had 8 more expertise then they had. And to help them 9 think through this very important subject for the 10 law school. It was very exciting. 11 Q The expertise that you had was about admissions? 12 A Absolutely. 13 Q About law school admissions? 14 A About law school admissions. 15 Q And you were the person that had the expertise on 16 the committee in that area, is that right? 17 A I don't think there's anybody else on that committee 18 that had one-tenth the experience I had actually in 19 Admissions. 20 Q Okay. I actually don't intend to go over the policy 21 again either, I think we've had enough of that. But 22 I do want to ask you about some of your 23 contributions to what's in the policy. 24 In the policy we have heard testimony 25 and we have seen summaries about Student X, I think 170 1 there's two Student Xs, a Y and a Z. And we heard 2 those were actual student files, is that correct? 3 A Yes. 4 Q And how did those student files come to the 5 attention of the Committee? 6 A Well, I selected them along with a whole host of 7 others. When I arrived here I discovered that the 8 faculty had actually not read files in years, over 9 decades. 10 And as part of the process it was 11 important for them, I thought, to get an actual feel 12 for what it was like to review a file. What was in 13 it, what kind of things to consider and that kind of 14 thing. 15 So I selected a whole range of files 16 for them to peruse. 17 Q At the very end of the policy, actually it's the 18 attachment to the policy there's a grid, you know 19 what I'm talking about? 20 A Yes, I do. 21 Q And there's been some testimony about that format. 22 And that format of the grid, I think, was what 23 Mr. Larntz used to create his model of cells? 24 A Yes. 25 Q You were present for this? 171 1 A Yes. 2 Q Let me ask you this, the grid that's at the end of 3 the policy, was it designed with the idea of being 4 able to show how race played a role in any 5 Admissions process or decision? 6 A No, not at all. 7 Q I want to talk to you a little bit about how you 8 went about implementing this Admissions policy. 9 When you were figuring out how to 10 implement this policy, and how to train people in 11 your office about how they should go about their 12 jobs once we're after the adoption of the policy and 13 that's in the spring of 1992, you created a document 14 which I believe is entitled Gospel According To 15 Dennis? 16 A Yes, that's right. 17 Q What can I say? 18 A Little did I know. 19 Q Now, we have got Dennis. Could you hold that out. 20 THE COURT: Is that Exhibit 5? 21 MR. PAYTON: I believe it's 22 Exhibit 5. 23 BY MR. PAYTON: 24 Q You recognize this document? 25 A Yes, I do. 172 1 Q The Gospel According To Dennis? 2 A Yes. 3 Q Written October 3rd, 1992, what was the purpose of 4 this document? 5 A Well, I had in that year, I anticipated in the 6 future that I would have people who had never read 7 the law school Admissions files in the past, that 8 would be involved in evaluating and providing some 9 assessment of files for me as I went about my 10 business in making decisions. 11 So, this was a document I created for 12 them, as part of their preparation for that process. 13 Q Okay. I take it you read a lot of files yourself? 14 A Yes, a lot. 15 Q Is it fair to say that you read most of the files? 16 A Most of the files. 17 Q Okay. And who else would read files in your office? 18 A There was always a number two person in my office, 19 the assistant or associate director of Admissions. 20 And then there were, depending on the year and the 21 staffing kinds of things, up to one or two other 22 people on my staff that read files. 23 Q And you would give them this document? 24 A I would give them this document, as well as a copy 25 of the Admissions Policy. 173 1 Q Okay. Is that all you give them? 2 A Yes. 3 Q That's it? 4 A Well, they'd look at the bulletin and that's it. 5 Q Did you ever tell them that we're trying to get X 6 percent of underrepresented minorities? 7 A I don't think I've ever said that to anyone. 8 Q Okay. That just never--nothing at all? 9 A Absolutely not. 10 Q You gave them these two documents. Who else got 11 these two documents if there were going to be 12 members of the Admissions Committee that would read 13 files, would they get these documents? 14 A I thought it might be a little presumptuous for me 15 to give this kind of document to a law school 16 faculty member. I'm sure you're familiar with--I 17 think you even taught law school. So you would know 18 how they would receive a document like this. 19 If I have known it was going to be 20 used in something like this, I might have 21 appreciated that. 22 Q Well, before I go into this then, let me just ask 23 you a few questions about how you got along with the 24 faculty in implementing the policy. 25 You served on the Committee and I 174 1 take it that was some relationship with the faculty, 2 is that right? 3 A Right. I worked very close with the faculty members 4 on the Committee. I might back up, I think when I 5 arrived at Michigan since I was new to the 6 institution, it was very important for me in my job 7 as the dean in charge of Admissions, to get to know 8 the institution well. 9 And so I made all kinds of effort to 10 interact with faculty. I sat in on faculty meetings 11 as a member of the dean's staff, 12 Q How often did faculty meetings happen? 13 A I couldn't give you a precise number, but probably 14 at least two-thirds of the Fridays of every term had 15 a faculty meeting. 16 Q Okay. 17 A And I would go to probably half of them when I was 18 in town. 19 Q Okay. 20 A And I would go to lunch with faculty members, I 21 would make an effort to go to social events that 22 were for faculty members so I could get to know 23 them. Several of the faculty members invited me 24 over to dinner at their homes. 25 And so I thought as an ongoing basis 175 1 it was my job to stay in tune with various aspects 2 of the law school, and provide plenty opportunity 3 for them to interact with me about what I was doing. 4 Q This is a two-way relationship? 5 A Yes, that's the way I viewed it. 6 Q Now, Ms. Munzel testified that you had trained her 7 in how to review files and how to do Admissions, is 8 that accurate? 9 A Yes, that's true. 10 Q You hired her? 11 A I hired her. 12 Q And she started reading files, she was your No. 2? 13 A Right. 14 Q So she knew this document pretty well? 15 A Well, she was supposed to have read it. 16 Q Now, I take it it's not going to disappoint you to 17 learn that the Gospel According To Dennis has been 18 retired? 19 A Not at all. I would assume that somebody else would 20 put it a little different spin on it. 21 Q The Gospel According to Dennis, it begins on this 22 first page. 23 A Yes. 24 Q You see--it actually starts, this is the first page 25 but it says four at the top, but this is the first 176 1 page? 2 A Right. 3 Q So, starting on what says page four, the first page 4 of this up at the top, it talks about we are trying 5 to select, do you see that? 6 A Is that the first paragraph? 7 Q It says under Philosophy. 8 A Okay. 9 Q You see right here, "We are trying to select from 10 the specially well credential pool of candidates 11 those that show the most promise." 12 A Yes. 13 Q Is that why this is a tough thing to do? 14 A Absolutely. Making decisions on candidates, 15 particularly at a school like Michigan, you have a 16 pool of candidates that are very, very strong in 17 almost every way. 18 Q And the end of that same paragraph it says, rather, 19 do you see that, "Rather we must begin with the 20 numbers and go forward from there to scrutinize the 21 essays and letters of recommendation." 22 A Yes. 23 Q "As well as considering extracurricular and work 24 experience, to look for candidates that show 25 intellectual talents, leadership ability and 177 1 academic acumen which augers for a lively 2 intellectual educational community and important 3 contributions to the profession." 4 Do you see that? 5 A Yes. 6 Q That's what you wanted everyone to be able to pick 7 out when they did their job here? 8 A Everyone that read files, that was the purpose, the 9 mission of the endeavor. 10 Q Go to the next page, page five. You see the first 11 full paragraph that says, given all this? 12 A Yes. 13 Q "I try to read each file with an open mind and try 14 to find something that distinguishes the candidate 15 and provide some reason to consider them 16 affirmatively for admission." Okay? 17 A Yes. 18 Q Is that how you went about doing it? 19 A Yes. Look, I would suspect that there are people 20 who think the process is one where you always look 21 for--first, look for a reason not to admit someone. 22 I tend to want to think positively about each 23 candidate, to try to find some reason to act in 24 their favor. 25 Q Okay. If you go down to the bottom you'll see it 178 1 says the basic approach and it just list things? 2 A Yes. 3 Q Things that you look at, the LSAT, GPA, 4 undergraduate institution, trends in grades? 5 A Yes. 6 Q And then it goes on to it says comparative rank. 7 And then it says, "If the index shows a significant 8 improvement from freshman/sophomore to junior/senior 9 academic performance." 10 You see that? 11 A Yes. 12 Q What's index mean there? 13 A Well, the index I think we heard a little bit about 14 it earlier. 15 Q Was this about the index score, because you don't 16 have an index score for freshman/sophomore? 17 A No, I'm basically talking about the trend in the 18 grades there. That if it's going upward, if it's 19 going downward, if it's sort erratic, that kind of 20 stuff. 21 Q Let me sort of go to where you were just about to 22 go. The index score, which is some formula that 23 relates LSAT, GPA with first year grades? 24 A Yes. 25 Q Do you use that in actually reviewing the individual 179 1 file? 2 A No. 3 Q Is it in the file? 4 A No. 5 Q Why won't you use it? 6 A Well, there are better tools to assess that in each 7 file, that you actually have to look at the academic 8 record. You have to look at the transcript, look at 9 the law school data assembly report that gives you a 10 wealth of information about each candidate in the 11 undergraduate institution. That kind of thing. 12 So, it's really not a particularly 13 useful number to look at when you're assessing a 14 file. 15 Q Okay. When Ms. Munzel testified, and she went over 16 a file in some detail, and actually I was going to 17 show you a file but now I'm not. 18 I just want to ask you, is that the 19 way you were reviewing files and you trained people 20 to review files what you saw her do? 21 A Yes. 22 Q This document the Gospel, it's written right after 23 the policy went into effect in October of 1992? 24 A Right. 25 Q Did you ever revise it or this one just stayed? 180 1 A That was it. I usually had other things to do, to 2 sort of look back at this. 3 Q Okay. I have looked through the entire document in some 4 detail, and there is no mention of race in here 5 at all? 6 A That's correct. 7 Q Why not? 8 A Because I think when you're reading a file, that's 9 not the primary consideration as you're making 10 judgments about it. It's the things that I talk 11 about in there. 12 Q Okay. Now, the admissions policy that we have all 13 spent some time looking at, the 1992 policy. That's 14 a policy about all of the admissions, isn't it? 15 A Right. 16 Q And the Gospel is also a document about all 17 admissions, isn't it? 18 A Absolutely. 19 Q And so you use the Gospel and the policy to guide 20 you in making all admissions decisions, is that 21 right? 22 A That's absolutely right. 23 Q Was there some minimum criteria for grades and test 24 scores that you needed before you would read a file? 25 A No. Every file deserved to be read. And so that's 181 1 what we did. 2 Q Now, we also heard some testimony, I believe from 3 Ms. Munzel, about a comment sheet that was filled 4 out at the end of a file. I think the second file 5 she looked at she had someone else's comments, you 6 remember that? 7 A Yes. 8 Q And she actually read off some of the information on 9 the comment sheet, so we heard what kind of 10 information was on there. 11 Did you keep comment sheets? 12 A No. 13 Q What happened to them? 14 A I had them thrown away. 15 Q When did you throw them away? 16 A At the end of the admissions year, that was my 17 instruction. Whenever the files were going to leave 18 our direct control for the admitted students 19 that ended up matriculating, the file went down to the 20 Registrar's Office. 21 And for those students who applied 22 and either were denied or chose not to come, they 23 went to a storage area. And when they left our 24 immediate control, those comment sheets were 25 removed. 182 1 Q And thrown away? 2 A Yes. 3 Q Was that just a standard policy you had? 4 A Yes. 5 Q By the way, was it the same policy you had at Iowa? 6 A Yes. 7 Q Now, I think we have seen some numbers about the 8 application flow that comes through the office, it's 9 3000 to 4000, some number like that? 10 A Yes. 11 Q You're telling me that every year you and some small 12 number of people on your staff read all three to 13 4000 files? 14 A Absolutely. 15 Q And reviewed them to make judgments? 16 A Absolutely. 17 Q And how did you decide which factors made a 18 difference, I mean there's a whole range of things 19 that are in the policy and in your Gospel memo about 20 things you ought to look at, how did you decide? 21 A We had to sit down and read the whole file and make 22 a judgment based on everything that you saw there. 23 There was no one thing, you had to look at the 24 transcripts, contemplate the test scores, think 25 about the undergraduate institution, read the essays 183 1 that were there, read the letters of recommendation 2 and arrive at an overall conclusion about an 3 individual file. 4 Q There's a lot of discretion that goes into this, is 5 that right? 6 A Yes, there is. 7 Q Is that a good thing? 8 A I think it is a good thing. With the guidance that 9 you have from the faculty of the law school, you're 10 implementing what they want to do. 11 And it's important to look at the 12 whole person in making a judgment about whether or 13 not to admit them to law school. 14 Q Now, how did you take race into account? You're 15 reading a file, how did you take race into account? 16 A Well, you read the whole file. It was one of 17 several, a number of factors you might take into 18 account. 19 Just as if you might take into 20 account the trend in grades, the rigor of the 21 curriculum. There was no specific way that you took 22 it into account. 23 Q Would it be taken into account the same way in every 24 minority, underrepresented minority applicant's 25 file? 184 1 A In every file. 2 Q Would it be taken into account in the same way? 3 A Not in the same way. Look, the assessment of any 4 individual file is never precisely the same way, 5 there are a lot of different things that you're 6 looking at. 7 And any one factor in there, for 8 example, an especially remarkable essay may be 9 dispositive in a particular case. 10 An exceptionally strong rigorous 11 academic record may be dispositive in any given 12 case. It might be the thing that tips the balance. 13 A particularly strong LSAT score in 14 some cases, might be the thing that tips the balance 15 in favor of a candidate. 16 So, in any given file the weight that 17 you might give to any particular aspect of it, would 18 vary from other files. 19 Q Now, as I understand it, from time to time you would 20 have a conversation with the dean, whether that be 21 Dean Bollinger or eventually Dean Lehman? 22 A Yes. 23 Q About how many Michigan residents you're looking 24 for? 25 A Yes. 185 1 Q Did you ever have a conversation with either one of 2 them about how many underrepresented minorities you 3 were looking for? 4 A No. 5 Q You ever have a conversation with him about what the 6 range of underrepresented minorities were that you 7 were looking for? 8 A No. Absolutely not. 9 Q Was the manner in which race was taken into account 10 different from the manner in which, let's say, 11 the essays or leadership ability, or any of the other 12 factors were taken into account? 13 A Well, an individual file it may carry more weight in 14 one file, it may carry less weight into another 15 file. And that was true about anything you might 16 think about in the file. In the grandest scheme of 17 things, no, it wasn't treated any different. 18 Q Okay. Now, another of your responsibilities as dean 19 and I think we have the impression that all you did 20 everyday was sit down and read files. 21 I take it another of your 22 responsibilities was to do all the things you have 23 to do to recruit students to file the applicants in 24 the first place, okay? 25 A Absolutely. 186 1 Q What did you do to do that? 2 A Well, if I can. My job is to create an entering 3 class every year. And in order to do that, you have 4 to have people apply, and you have to have good 5 people apply. And if you're interested in 6 diversity, you have to have a diverse pool of people 7 to select from. 8 And so we traveled extensively to 9 college campuses, to law school recruitment fairs. 10 I made contacts, maintained my contacts with the 11 pre-law advisors on different campuses across the 12 country and corresponded regularly with them. 13 We made it a point to visit a whole 14 range of different types of institutions. We did a 15 lot of direct mail to students that we thought were 16 competitive for admission. 17 Locally I established a Minority Law 18 Day for freshmen and sophomores on the campus of the 19 University of Michigan. 20 I regularly interacted with the 21 pre-law advisors on Michigan's campuses and the 22 various student organizations that were aimed at 23 ultimately applying to law schools in the 24 undergraduate pre-law call, that kind of things. 25 Maintained the same kind of contacts 187 1 with a number of other organizations on other 2 campuses around the country. 3 Q Now, were these things that, not all of them, but 4 were some of these things, new things that you did, 5 that as far as you know hadn't been done before to 6 recruit students? 7 A Absolutely. I took as my charge when I arrived, to 8 reenergize and to be innovative about the kinds of 9 things that we did to attract candidates for 10 admission. 11 Q And did you do special things to try to recruit 12 underrepresented minorities to apply to the law 13 school? 14 A Absolutely. 15 Q What did you do? 16 A We would visit colleges, universities where there 17 was significant population, or where that was the 18 particular mission, so to speak. Historically black 19 colleges, universities that had significant 20 representations of Hispanics, Asian Americans. 21 Other campuses we made a point to go 22 there, to find out who the undergraduate 23 organizations were that we could work with, talk to, 24 make presentations to. That kind of thing. 25 Q And were these new things? 188 1 A Yes. 2 Q I'd like to show you some exhibits that were used 3 with Ms. Munzel and that I used in my opening. 181, 4 182, 183 and 184. 5 A Okay. 6 MR. PAYTON: While we're doing this, 7 your Honor, I want to offer into evidence Exhibit 5, 8 I believe, which is the Gospel According to dennis. 9 MR. PURDY: No objection, your Honor. 10 THE COURT: Received. 11 BY MR. PAYTON: 12 Q These are charts that show data from 1997, and they 13 show all of the applicants and all of the admitted 14 students. And then separately they show it for 15 underrepresented minorities admitted, 16 underrepresented minorities rejected. Majority 17 students admitted, and majority students rejected. 18 You can put up any one. Put up any 19 one of the charts so I can just ask him 20 These are non-admitted majority 21 applicants, do you see that, Mr. Shields? 22 A Yes. 23 Q There were some testimony about--actually it wasn't 24 testimony, it was a representation by me about the 25 fact that there are scores on the LSAT, non-standard 189 1 scores is how I referred to them, that show as zeros 2 in the data that is reported by Law Services? 3 A Right. 4 Q Okay. Was I right? 5 A Absolutely. 6 Q Could you explain what a non-standard score on the 7 LSAT is? 8 A Well, the law school admissions test is a 9 standardized test. That is it's supposedly everyone 10 who takes it, takes it under the same conditions, et 11 cetera, et cetera. 12 Well, in fact, there are some, not a 13 whole lot, but there are some who take it under 14 non-standard conditions. 15 Most often or probably the only way 16 that that happens, is if they have some documented 17 disability. And when that happens they get more 18 time, or they get different kind of test. 19 For example, someone who has a vision 20 problem might actually have someone read the exam to 21 them. And because their taking it under nonstandard 22 conditions, the scores reported on their LSAT 23 report. 24 But in terms of the data since it's 25 non-standard they get a zero when it comes done to 190 1 accounting for statistics. 2 Q Do you see this, for example, on this chart right 3 here on the bottom axis which is zero, if you look 4 to the right and across you see a number of, I call 5 them hits, little points, those are zeros? 6 A Right. 7 Q Non-standard? 8 A Test takers. 9 Q Test takers who show on here as zeros because in the 10 data frame they show a zero? 11 A Right. 12 Q And that's simply how Law Services deals with what 13 you call the non-standard test? 14 A Right. For example, when you see an LSAT report, on 15 most of the LSAT reports taken under standard 16 conditions, you get a score and an identification of 17 the percentile positioning of that score on the 18 scale. 19 In the non-standard setting, you get 20 a score but you get no percentage. And that's why 21 it shows up as a zero. 22 Q Could you put on top of that, I don't know what this 23 exhibit is, it's 182. Can you put the companion 24 which is the non-admitted minority on that. 25 Mr. Shields, you were the director of 191 1 Admissions in 1997, is that correct? 2 A Yes, I was. 3 Q So this is data that relates to how you ran the 4 office? 5 A Absolutely. 6 Q And do you see that when you look at the majority 7 and the minority plots up there, they have some 8 substantial overlap, those are my words. But do you 9 agree that they have substantial overlaps? 10 A Yes, absolutely. 11 Q And these were the students who did not get in? 12 A Right. 13 Q Are you surprised by that? 14 A Not at all. 15 Q Could you put up the other two. And these are the 16 admitted students, and if you pull them apart so he 17 can see the difference underneath. 18 At the top now is the majority 19 students who were admitted, and now placed on top of 20 them right now are the minority students who were 21 admitted. 22 The same scale, you see the overlap 23 there? 24 A Yes. 25 Q Okay. And that picture of what the class of 192 1 admitted students look like in 1997, does that 2 surprise you in anyway? 3 A Not at all. 4 Q Did you admit, in your opinion, really good classes 5 of students, minority and majority alike? 6 A I'm very proud of the classes that I admitted to the 7 University of Michigan Law School. 8 Q Mr. Shields, I want to ask you about how you look 9 back on what you accomplished at the University of 10 Michigan Law School, with respect to this policy and 11 its implementation. 12 What's your reflection on how this 13 policy and your implementation work? 14 A I think that the policy, I'm very proud of the role 15 that I had in developing it, I'm very proud of the 16 final policy. 17 I think my implementation of it and 18 attempts to accomplish what the policy asked of me. 19 I'm very proud of all of that. I don't think anyone 20 else could have done it better. 21 MR. PAYTON: Thank you, your Honor. 22 THE COURT: Intervenors, any 23 questions? 24 MS. MASSIE: None. 25 THE COURT: Plaintiffs. 193 1 MR. PURDY: Thank you, your Honor. 2 Your Honor, Larry Purdy again for the Plaintiff. 3 THE COURT: Mr. Purdy. 4 5 CROSS-EXAMINATION 6 BY MR. PURDY: 7 Q Good afternoon, Dean Shields? 8 A How you doing? 9 Q Good. Let me go through first, I've got a couple of 10 questions that I want to try and get to just a 11 little bit later, but let me try and walk through it 12 if I could just briefly some of the testimony that 13 you've given to us. 14 First, with regard to Exhibit 5 your 15 Gospel According To Dennis Shields? 16 A Yes. 17 Q Would it be fair to say that this philosophy applies 18 to every candidate regardless of his or her race 19 ethnicity? 20 A Yes. 21 Q And do I assume that it was your intention to apply 22 this philosophy equally to every applicant that came 23 across your desk regardless of his or her race and 24 ethnicity? 25 A The purpose of the document was to give guidance to 194 1 people reading files for me. 2 Q Sure. And you didn't vary the way you approached 3 any file depended upon the person's race or 4 ethnicity, would that be a fair statement? 5 A That's a fair statement. 6 Q You know, there was a question from Mr. Payton about 7 the index scores. And I believe you said that index 8 scores is not used in the review, do you recall 9 that? 10 A Yes. 11 Q I believe I wrote it down and if I'm wrong correct 12 me. But I believe you said it's not useful number, 13 do you recall that? 14 A Yes. 15 Q But, in fact, doesn't the policy itself talk about 16 the importance of the index and the admissions 17 process? 18 A Well, that's a short hand term for looking at the 19 law school admissions test score, and the 20 undergraduate academic record. 21 And it's not useful because it 22 doesn't give you very complete information. For 23 example, as we evaluate an academic record, the 24 quality of the school that one attends, the rigor of 25 the undergraduate curriculum that one has pursued, 195 1 is not in anyway captured in that index. 2 Q Just out curiosity because there's been so much 3 discussion about it, I actually went through the 4 policy last night and counted the number of times 5 where the word index appears. 6 Would it surprise you that the word 7 index appears in the Admissions policy 20 times or 8 more? 9 A Not at all. Not at all. 10 Q And we have gone over this, Exhibit 5 it doesn't 11 reference race at all, correct? 12 A Correct. 13 Q But the Admissions policy does? 14 A Yes, it does. 15 Q And, of course, you told us this afternoon that you 16 were guided in your admissions decisions by the 17 Gospel and the policy, correct? 18 A Right. Well, the Gospel was something that I wrote 19 for the benefit of the people in my office that 20 would be reading files and providing evaluations of 21 those files, summaries of them with the files when I 22 was ready to make a judgment on them. 23 Q I appreciate the distinction. I think what you're 24 trying to tell us is actually you were guided in 25 your decisions by the policy? 196 1 A Yes. 2 Q There was also a question from Mr. Payton about 3 whether or not you periodically had discussions with 4 Dean Bollinger and then subsequently Dean Lehman 5 about the percentage of residents that the school 6 may be seeking each year, you recall that? 7 A Yes. 8 Q And, of course, it's clear that every year you did 9 have discussions trying to figure out where you 10 wanted to be, in terms of resident matriculants 11 within the class? 12 A Right. 13 Q And as I recall from looking at the documents, and 14 again correct me if I'm wrong. But I believe that 15 it consistently felt a third of the class plus or 16 minus were residents? 17 A Well, it depends on what time frame you're talking 18 about. Because most of what we've been talking 19 about was from 1995 forward. 20 Q Let me back up. From 1992 until you left in 1998, 21 did the percentage of residents that the school 22 sought to admit roughly fall in the one-third range? 23 A Yes, give or take five percent probably either way. 24 Q And the policy specifically mentions the preference 25 that they want to consider for Michigan residents, 197 1 does it not? 2 A Yes. 3 Q In fact, it uses the language honoring the special 4 claims of Michigan residents to a Michigan law 5 school education, correct? 6 A Yes. 7 Q And, of course, that wasn't divided by race or 8 ethnicity, it's all Michigan residents? 9 A Right. 10 Q And then Mr. Payton asked you whether or not you 11 recalled any discussion with either Dean Bollinger 12 or Dean Lehman about whether or not there was a 13 target range for race and you told us there wasn't, 14 correct? 15 A Right. 16 Q But you had-- 17 A (Interposing) I didn't have any conversations with 18 them about that. 19 Q You didn't have any conversations. But you were 20 aware of discussions about certain percentages of 21 certain underrepresented minority groups, were you 22 not? 23 A I'm not sure what you're referring to. You mean in 24 the creation of the policy? 25 Q Yes, sir. 198 1 A I'm familiar with those discussions. 2 Q All right. And, in fact, if I could ask you to turn 3 to Exhibit 34. 4 A Okay. 5 Q Dean Shields, I am just going to ask you, when you 6 were part of the faculty policy, I'm sorry, sure the 7 policy that you were creating was the faculty 8 Admissions policy. 9 Did you periodically get copies of 10 the drafts and make your own comments? 11 A I don't think I ever wrote comments. I saw the 12 drafts, but I generally--you have to understand what 13 my falls are like. 14 I typically visit anywhere from 20 to 15 30 different campuses, et cetera. So the time I 16 have to scratch down notes is rather limited. 17 So, usually I would try to come to 18 the meetings having read it, and then react to what 19 I had read. 20 Q I appreciate that. And, of course, we took your 21 deposition, what, two years ago or something like 22 that? 23 A Yes. 24 Q And you didn't recall having any drafts and we have 25 not found any, so I'll tell you I'm not going to 199 1 spring a draft of yours. Unlike counsel who sprung 2 the Gospel of Dennis on you. 3 But if you look on page 13 of 4 Exhibit 34, and this was an initial draft of the 5 policy. And while you're getting there, you do 6 recall seeing various drafts of the policy as it was 7 underway, correct? 8 A Yes, I do recall. I don't know if I saw this 9 specific one marked up like this. 10 Q Sure. Let me just ask you to look at the bottom of 11 the full paragraph on page 13, and I'm just going to 12 read the last sentence real quickly. In fact, I'll 13 even paraphrase it. 14 It just notes in the past we have 15 achieved the kinds of benefits that we associate 16 with racial and ethnic diversity from classes in 17 which the proportion of African American, Hispanic 18 and Native Americans members has been between eleven 19 percent and 17 percent of total enrollees." 20 Do you recall reading that from other 21 drafts of the policy while you were serving on the 22 committee? 23 A You know, I don't know that I recall reading it. I 24 know that we talked about numbers in that process. 25 Q All right. And let me ask you to also turn to 200 1 Exhibit 32, it should be just two exhibits in front 2 of that, if you would, please. 3 A Yes. 4 Q And here is a memorandum from Professor Regan. You 5 know Professor Regan, do you not? 6 A Yes, I do. 7 Q And, in fact, this indicates that you got a copy of 8 this particular memorandum, does it not? 9 A Yes. 10 Q Do you recall reading this memorandum back in this 11 time frame? 12 A I don't have a specific recollection, but I'm sure 13 that I have no question about whether I saw it. 14 Q And so you recall Professor Regan, at least 15 reviewing at some point Professor Regan's comments 16 about whether or not to leave numbers in or take 17 them out of the policy? 18 A Yes. 19 Q And you recall Professor Regan's suggestion that for 20 a variety of reasons, including candor, I incline to 21 prefer to keep the numbers in and try to explain 22 what they really signify, do you recall that? 23 A Yes. 24 Q So, you were, at least, aware of what the faculty's 25 views were about the percentage of underrepresented 201 1 minorities, or at least the size of the class that 2 they would like to attract each year? 3 MR. PAYTON: You mean the committee? 4 MR. PURDY: The committee, I'm sorry. 5 BY MR. PURDY: 6 Q The committee? 7 A This was Professor Regan's view point at this point 8 in time. I wouldn't want to attribute any of his 9 position to any other member of the committee, 10 okay. 11 Q I understand that the memo that we're looking at, 12 Exhibit 32? 13 A That's Don Regan's, Professor Regan's take on things 14 at that point. 15 Q He was commenting, was he not however, on the 16 percentages that we see in Exhibit 34, the eleven to 17 17 percent? 18 A I guess, I don't know. 19 Q Do you recall discussions in the faculty meetings 20 that you attended about this policy where the 21 numbers eleven to 17 percent was discussed? 22 A Yes, I was a very active part of those discussions. 23 Q Dean Shields, every year while you were at Michigan, 24 you would receive numerous applications from various 25 minorities who presented stellar academic 202 1 credentials, correct? 2 A Yes. 3 Q These were minority applicants who graduated from 4 the same range of schools as did your white nation 5 American applicant, would that be a fair statement? 6 A There was a significant overlap in where the 7 minority students went to school that applied, and 8 where the majority students went to school that 9 applied. 10 Q Sure. You would have minority students who came 11 from schools from the Ivy League, and you would have 12 some that came from school that I'm sure you knew to 13 be outstanding small liberal arts schools? 14 A Very few white candidates from historically black 15 colleges. 16 Q Any? 17 A None that I know of. 18 Q Okay. And you would have those that came out of a 19 lot of public institutions, and I'm talking about 20 you have minority applicants with great credentials 21 who came from public institutions like Michigan 22 State and University of Michigan, correct? 23 A Yes. 24 Q Just like you would white students and African 25 Americans? 203 1 A Sure. 2 Q Minority students who followed the same tough 3 curricula, took the same tough courses, that would 4 have impressed you? 5 A Sure. 6 Q And if I may, I'm going to, if I could, I'm just 7 going to put page 13 of the Policy which guided your 8 work. And if you can't read it, it's Exhibit 4. 9 Q I'm not trying to embarrass you so suggest that you 10 can't see over there. 11 A Well, I tell you, you know, I'd hate to admit it, 12 there's probably a day I could see it. 13 Q It's page 13, and if I could just have you turn to 14 that. And certainly you would agree, would you not, 15 Dean Shields, that there were people who were 16 members of underrepresented minority groups who you 17 would admit every year without reference, without 18 reference to their minority status, correct? 19 A There were some, yes. 20 Q And, of course, we know that you told told us yield 21 is a very tough problem with all applicants, 22 particularly in the upper grid cells? 23 A I would characterize that any applicant that's 24 particularly remarkable presents a challenge in 25 convincing them to come to Michigan as opposed to 204 1 other very fine law schools. 2 Q That's interesting. Why did you have that 3 particular problem getting them to come to Michigan, 4 as opposed to other schools where they would 5 typically also be accepted? 6 THE COURT: The weather. 7 A Well, that was some of the problems. 8 BY MR. PURDY: 9 Q What were some of the problems Michigan faces in 10 recruiting the same kids that get accepted, to say, 11 Harvard or Yale or Chicago or UCLA or Berkley? 12 A Those are all good options. And for a whole--well, 13 precisely for the reason I heard people think very 14 carefully about where they attend law school. 15 The size of the law school, where 16 it's located, what their long term career ambitions 17 might be. Each candidate makes sort of independent 18 choices about how that matches up with where they 19 want to go to school. 20 Q Well, just so there's no misunderstanding. Every 21 year there--while you were here, we'll just confine 22 it to the five years, was it about five year? 23 A Six and a half years. 24 Q Six and a half years, I'm sorry. Every year while 25 you were serving as the dean of Admissions in 205 1 Michigan, you had underrepresented minority 2 applicants who you admitted who required no 3 consideration of their race in order to obtain that 4 admission, correct? 5 A Yes. 6 Q And, indeed, I assume it's a fair statement that if 7 you could achieve the law school goal of enrolling a 8 critical mass of these students from this group that 9 didn't need any consideration of race, you'd happily 10 do that, would you not? 11 A I would admit anybody who I thought was a remarkable 12 candidate. 13 Q Sure. And if you could achieve a critical mass of 14 specifically underrepresented minority students 15 without referring to race, that would be a wonderful 16 achievement too, correct? 17 A I hope that day comes. 18 Q We do, we all do. That's actually the ultimate goal 19 of the policy, is it not, I mean that's what they 20 talk about on page 13. That hopefully that will be 21 exhausted at some point? 22 A There's a lot of other goals talked about in the 23 policy, and I think that it's important to keep in 24 mind that the Policy governs overall admissions. 25 And the task that we were to undertake in that year 206 1 was to rethink all of the admissions. 2 But one of the goals would be 3 to--well, I don't think it would ever be a goal with 4 this policy to not have a diverse class, okay. 5 If we ever get to the point where we 6 can achieve that without any consideration of race, 7 I think this country would be a happier place. 8 Q But one of the goals in the policy and I won't pour 9 through the blowups and try and find it, but you 10 recall it no doubt. I think it's a previous page 11 twelve. 12 One of the goals is to enroll a 13 critical mass of underrepresented minorities? 14 A Critical mass is part of the goal, sure. 15 Q Sure. And in order to achieve that critical mass of 16 minority students the practice was and the policy 17 called for, a willingness to admit minority students 18 from generally lower academic qualifications then 19 majority students, isn't that a fair statement? 20 A I think that's a fair statement. 21 Q Do you have Exhibit 15, and if you don't have the 22 book we'll get it for you. Actually, you know, 23 before I get to that and I apologize, but you've got 24 the book. 25 Could you look at page ten, I'm 207 1 sorry, Exhibit 10 first, it's a daily report. 2 A Yes. 3 Q Just so it's clear, you would get these periodically 4 during the admissions cycle, would you not, and this 5 would help you determine where you were in terms of 6 these offers that had been made, and where you sat 7 in terms of admissions offers that had been accepted 8 in terms of the possible things of that nature? 9 A Yes. Early in the season I might look at this, this 10 is something I could just punch a button on my 11 computer and it will crank this out in about 15 12 minutes. 13 And in December I might get it once, 14 in January I might get it two or three times. And 15 by the time you get to April and May when the 16 deposits are rolling in, I may want to see it daily. 17 Q I was going to say, I think I recall in your 18 deposition you talked about that it's certainly your 19 use of these types of reports increase from, let's 20 say, early March until the end of May that your 21 class was really starting to come together? 22 A Right. 23 Q And these reports were broken down by race so that 24 you could tell where you sat in terms of the 25 admissions from each end? 208 1 A Well, they're broken down a number of different 2 ways. 3 Q Sure. 4 A By race, gender, residency, non-residency, et 5 cetera. 6 Q My only point is you did have the ability and, in 7 fact, took advantage of the ability to see how the 8 class was shaping up as you went along? 9 A Yes. 10 Q And part of that was to see how the class was 11 shaping up in terms of its racial and ethnic maybe 12 up, correct? 13 A Yes. 14 Q Let me now ask you to turn to Exhibit 15, if I 15 could, please, sir, that's where I was directing you 16 initially. 17 A Okay. 18 Q You know actually let me ask you, I apologize, let 19 me for a moment. 20 Do you have a view as you sit here 21 today what percent of underrepresented minority 22 students would constitute a critical mass? 23 A No, not really. You mean you're looking for a 24 particular number or percentage? 25 Q Even a rough percentage? 209 1 A Not really. 2 Q Would five percent underrepresented minority 3 students constitute a critical mass in your view, as 4 an admissions expert? 5 A I don't think so, I don't know though. I mean part 6 of that is not just my sort of assessment, the 7 assessment of other people in the law school where 8 I'm working. That kind of thing. 9 Q And I don't mean to--would ten percent constitute a 10 critical mass? 11 A It might, I don't know. It could. 12 Q Looking at Exhibit 15, and I believe this is in 13 evidence. 14 THE COURT: I think it is. 15 MR. PURDY: If it's not, we'll offer 16 it. 17 THE COURT: I suspect it's in. 18 BY MR. PURDY: 19 Q This is a copy of a grid showing all of the 20 applicants, of course, all of the LSAT and grade 21 point ranges, do you recall seeing documents like 22 this while you served at Michigan as the dean? 23 A Yes. 24 Q And you would use these reports each year kind of as 25 a--to compare how your current class was shaping up 210 1 in comparison to your last year class, is that the 2 way you would use it? 3 A No. 4 Q How would you use this document? 5 A I would look at this just to see what had happened 6 in the proceeding year. Probably--well this one-- 7 Q (Interposing) This is 1995. This was for the class 8 that entered in the fall of 1995? 9 A The fall and summer of '95. I would typically look 10 at this kind of stuff after that class had been put 11 to bed, so to speak. Early in the fall or later. 12 And that would probably be about the only time I 13 would look at this information. 14 Q And, in fact, this is a class that you selected, so 15 we're looking at decisions you made, correct? 16 A Yes. 17 Q If I could ask you to turn to the third page of this 18 document, it's the grid that shows the--in fact, 19 I'll just read it to. It's page three of 20 Exhibit 15, it's the University of Michigan Law 21 School Admissions Office, admissions grid of LSAT 22 and GPA for African Americans. 23 And I'm going to direct you down to 24 the line that begins with grade point 3.2. And 25 we're just picking it because you've been through it 211 1 before, so I thought it was the easiest. 2 3.25 through 2.49 and we'll start 3 over under the LSAT score of 151 to 153, and you 4 understood that to be about the 50th percentile? 5 A Around that, I'm not sure exactly where it was for 6 that year. But about that. 7 Q All right. And it shows that in terms of just the 8 African Americans and we're going to start with that 9 and we're going to move up the scale on LSAT keeping 10 the grade point constant. 11 But you had seven applicants and 12 three were admitted, correct? 13 A That's correct. 14 Q The next, moving up to 154 to 155, five applicants 15 four were admitted, correct? 16 A Yes. 17 Q And then moving on to the 156 to 158. Ten 18 applicants, ten admitted, correct? 19 A Yes. 20 Q 159 to 160, three applicants, three admitted, 21 correct? 22 A Yes. 23 Q And 161 to 163, four applicants four admitted, 24 correct? 25 A Yes. 212 1 Q And, in fact, if we go all the way across the scale 2 goes up that all the applicants obviously were 3 admitted. Let's turn to the next page, 4 Dean Shields, if we could. 5 And I want to direct you to the same 6 line, and we're going to start with the same LSAT 7 and GPA grid position. 8 MR. PURDY: And for the record, your 9 Honor, this is page four. This is the admissions 10 grid for Caucasian Americans. 11 BY MR. PURDY: 12 Q And you will see that under the 151 to 153 where we 13 had seven African Americans, three admitted. You 14 have 24 Caucasian who applied, and zero admits, 15 correct? 16 A Yes. 17 Q And moving to the next column where we had 18 previously five African Americans applicants and 19 four admits, we have 21 Caucasian applicants and 20 again zero admits, correct? 21 A Yes. 22 Q Going up to the 156 to 158 where previously you had 23 ten African Americans applicants, all ten admitted. 24 Here there was 51 Caucasian Americans who applied 25 and one was admitted, correct? 213 1 A Yes. 2 Q And the next column where there were three out of 3 three African Americans accepted, there was 61 4 Caucasians who applied and one was admitted, 5 correct? 6 A That's correct. 7 Q And going over to the next column, 126 Caucasian 8 applicants, five admitted, do you see that? 9 A Yes. 10 Q Dean Shields, would it be fair to assume, is it 11 accurate to assume, I'm not asking you about any of 12 the final decisions you made within these grids, but 13 the average, the difference that we see in terms of 14 the decision making with respect to African 15 Americans in these cells and Caucasians, can 16 generally be explained by the extent to which race 17 is taken in account in the admissions process, would 18 that be a fair statement? 19 A I'm not willing to go all the way there with you 20 without reviewing the files or having the files to 21 look at. Because I couldn't be certain without 22 seeing those files again. But, at least, some of it 23 could be attributed to that. 24 Q Let me just ask you, do you have your deposition 25 handy in front of you? 214 1 A Yes. 2 Q If I could ask you just to turn to page 154, just 3 for a moment? 4 A That's in the thicker one? 5 Q It's the thick one, yes, sir. I'm going to direct 6 your attention to line 15 and I'm just going to read 7 two questions and two answers that were given to 8 you. 9 This was back on December 7, 1998 10 when my partner Mr. Kolbo who is sitting back there 11 took your deposition. 12 A Yes. 13 Q And let me preface and I apologize. You had just 14 gone through the same analysis with the grids as we 15 just went through? 16 A Sure. 17 Q Okay. 18 "Q Would it be fair to assume, is it accurate 19 to assume and I'm not asking you about any 20 individual's files here, but the average 21 here, the difference here in terms 22 of decision making with respect to African 23 Americans in these cells and Caucasians, 24 can generally be explained by the extent 25 to which race is taken into account in the 215 1 admissions process? 2 A. Generally, yes. 3 Q. There might be something else in a. 4 particular applicant's file, but on a. 5 whole that is the explanation? 6 A. Generally that's probably true." 7 Do you recall being asked those 8 Questions and giving those answers? 9 A Well, they're here. 10 Q But I mean those were your answers to those 11 questions, correct? 12 A Sure. 13 MR. PURDY: Your Honor, I have 14 nothing further. 15 THE COURT: Mr. Payton. 16 MR. PURDY: Thank you, very much. 17 18 REDIRECT-EXAMINATION 19 BY MR. PAYTON: 20 Q Mr. Shields, Mr. Purdy asked you about drafts and 21 memoranda about drafts of the 1992 policy, in which 22 there was a reference to eleven to 17 percent. And 23 you said you remembered discussions about that. 24 Do you remember your position in 25 those discussions about that? 216 1 A Absolutely. 2 Q What was it? 3 A I thought it was entirely inappropriate for there to 4 be numbers included because--and I said this during 5 the deliberations. 6 If, in fact, we had a pool of 7 candidates where we could not admit any specific 8 number, then that's just the way it would be. 9 And that my job was to assure that we 10 had a stronger pool of candidates, in part my job 11 was to have a stronger pool of candidates available, 12 and that we should not constrain ourselves. It was 13 fine to have an aspiration, but we should not 14 constrain ourselves to that by that. 15 So, that if there were 50 percent 16 minority in the class, that should not be looked at 17 as some sort of violation of the policy. Nor, if it 18 was less than that, it should be considered some 19 violation of the policy. 20 That, in fact, what we were trying to 21 do is make individual decisions about individual 22 candidates. 23 Q Now, when you read a file, when you read a file when 24 you were at the University of Michigan Law School 25 and you're looking through a file, I understand 217 1 there's no document that says a number. 2 But in your mind as you're going 3 through the file, do you have in your head a number 4 that you're trying to hit with respect to 5 underrepresented minorities? 6 A Absolutely not. Absolutely not. As I read a file, 7 I'm making an independent judgment about that 8 candidate. And I may look back at the gross numbers 9 at some point in time and think, well, we're doing 10 pretty good here or we're not doing so well here to 11 whatever. 12 But as you make a decision about 13 individual files, you're not keeping in mind any 14 sort of specific target. 15 Q Okay. Now, with respect to every single student you 16 admitted at the University of Michigan Law School, 17 and with respect to the overall classes that you 18 admitted at the University of Michigan Law School, 19 do you believe today that they were a very well 20 qualified group of students individual by 21 individual? 22 A Absolutely. Remarkable classes. 23 MR. PAYTON: Thank you, your Honor. 24 MR. PURDY: Just briefly, your Honor. 25 THE COURT: Okay. 218 1 RECROSS-EXAMINATION 2 BY MR. PURDY: 3 Q Just very briefly to follow-up on what Mr. Payton 4 said. 5 It was expressly set forth in the 6 policy, was it not, that you no matter, what you 7 would offer admission to no applicant who you didn't 8 believe could succeed and complete the law school 9 curriculum without serious academic problems, 10 correct? 11 A Right. 12 Q And so if for whatever reason your applicant pool 13 didn't present you with enough residents, for 14 example, who you believe base on your review of the 15 whole file could complete the course without serious 16 academic problems, you weren't going to admit those 17 kids, correct? 18 A Right. 19 Q Okay. So, constrained by that, obviously you 20 wouldn't be admitting kids, you wouldn't bring 21 applicants in to the school who you didn't believe 22 could complete the program, correct? 23 A Right. 24 Q All right. At anytime, Dean Shields, during the six 25 and a half years that you were there, did the 219 1 underrepresented minority enrollment ever drop below 2 eleven percent? 3 A I'm not absolutely certain, but I don't think so. 4 MR. PURDY: That's all I got your 5 Honor. 6 THE COURT: Thank you, Dean. 7 (Witness excused.) 8 THE COURT: Who's your next witness? 9 MR. PAYTON: This is all I have for 10 today, as I said. My next witness Monday looks like 11 this. We're calling Kent Syverud and we're going to 12 call Dean Lehman. Those our last two witnesses. 13 THE COURT: Great. We'll recess 14 until Monday morning at nine. We'll see you Monday 15 morning at nine o'clock. 16 (Court adjourned at 3:50 p.m.) 17 18 19 20 21 22 23 24 25 220 1 2 CERTIFICATE 3 I, JOAN L.MORGAN, Official Court Reporter for the United 4 States District Court for the Eastern District of Michigan, 5 appointed pursuant to the provisions of Title 28, United States 6 Code, Section 753, do hereby certify that the foregoing 7 proceedings were had in the within entitled and numbered 8 cause of the date hereinbefore set forth; and I do further 9 certify that the foregoing transcript has been prepared by me 10 or under my direction. 11 12 ____________________ JOAN L. MORGAN, CSR 13 Offical Court Reporter 14 15 Date: __________________ 16 17 18 19 20 21 22 23 24 25