The opinion of the court was delivered by: John T. Curtin United States District Judge
In Second Amended Complaint "B" in this case, plaintiffs Men of Color Helping All ("M.O.C.H.A.") Society, Inc., claim that the City of Buffalo's use of a statewide promotional examination given in 1998 to generate a list for promoting Buffalo firefighters to the rank of lieutenant (the "1998 Lieutenant's Exam") had a disparate impact against African-American firefighters, in violation of Title VII of the Civil Rights Act of 1964, 42 U.S.C. § 2000e-2 (see Item 54). An evidentiary hearing was held by the court over the course of five days in June - August 2008 for the limited purpose of determining whether the 1998 Lieutenant's Exam was "job related for the position in question and consistent with business necessity," as required under Title VII to validate the use of the Exam notwithstanding disparate impact. See 42 U.S.C. § 2000e-2(k)(1)(A)(i).
Based on the testimony and evidence presented at the hearing, and considering the arguments set forth in the parties' post-hearing submissions, the court makes the following findings of fact and conclusions of law.
In December 1997, the City of Buffalo sent a request to the New York State Department of Civil Service for services related to the preparation and scoring of an examination for promotion of firefighters to the position of Fire Lieutenant (see "Request for Examination Assistance," Joint Exhibit 9; see also Tr. 363-64).*fn1 The request was made in accordance with Section 23 of the New York Civil Service Law, which provides:
The state civil service department, upon the request of any . . . municipal commission, shall render service relative to the announcement, review of applications, preparations, construction, and rating of examinations, and establishment and certification of eligible lists for positions in the classified service under the jurisdiction of such municipal commission.
N.Y. Civil Service Law § 23(2).
The Department of Civil Service, Testing Services Division, is the State agency responsible for providing examination assistance to local jurisdictions (Tr. 204-05). The Testing Services Division employee responsible for overseeing the development of the 1998 Lieutenant's Exam was Associate Personnel Examiner Wendy J. Steinberg, Ph.D., under the supervision of Testing Services Director Paul D. Kaiser (Tr. 14, 107-08). The official title of the Exam is the "Lower Level Fire Promotion Series" (Tr. 118). It is used by some jurisdictions for promotion to the rank of Captain (id.).
The City administered the Exam on March 14, 1998. A total of 179 White firefighters and 89 African-American firefighters took the Exam. Of those who took the Exam, 133 White candidates passed and 46 failed, for a pass rate of 74.3 percent; and 38 African-American candidates passed and 51 failed, for a pass rate of 42.6 percent (see Plaintiffs' Exhibit 25).
M.O.C.H.A. originally filed this action in February 1998 (before the March 1998 administration of the Exam) seeking declaratory relief and damages on their own behalves and as representatives of a proposed class of all African-American firefighters employed by the City during the three prior years, based upon allegations relating to the City's overall policy for promoting firefighters to the rank of lieutenant, as well as the City's drug-testing policy as implemented within the Buffalo Fire Department. Due to the complex nature of the original pleading, the court directed plaintiffs to file separate amended complaints setting forth the claims relating to the drug-testing program (Complaint "A") and the promotional practices (Complaint "B"), and subsequently granted leave to file second amended complaints dealing with each claim.
Following protracted discovery and motion practice, M.O.C.H.A. twice moved for summary judgment on Second Amended Complaint B arguing that the statistical results of the City's administration of the Lieutenant's Exam in March 1998 constituted prima facie evidence that the Exam had a disparate impact on African-American firefighters, and that the City failed to meet its burden to demonstrate that the Exam was valid--i.e., that the Exam was job-related and consistent with business necessity. The court denied these motions, finding on each occasion genuine issues of material fact with respect to the validity of the Exam, see M.O.C.H.A. Society, Inc. v. City of Buffalo, 2005 WL 589834 (W.D.N.Y. February 28, 2005); M.O.C.H.A. Society, Inc. v. City of Buffalo, 2007 WL 3354211 (W.D.N.Y. November 9, 2007), setting the stage for the evidentiary hearing which took place during the summer of 2008.
At the hearing, Dr. Steinberg and Mr. Kaiser testified about the services they performed as the Civil Service Department officials charged with the task of developing the 1998 Lieutenant's Exam. The court also heard testimony from plaintiffs' employment testing expert, Kevin R. Murphy, Ph.D., and from the City's Director of Civil Service, Ms. Olivia Licata.
The hearing testimony is summarized here as follows:
Dr. Steinberg was called as a witness by both parties to testify about the steps she took to develop the 1998 Lieutenants' Exam. Her educational background includes extensive course work in the development and validation of employment tests, culminating in a Ph.D. in Educational Psychology and Statistics from the State University of New York at Albany (Tr. 97-99). She was employed by the Testing Services Division from 1979 to 1998, and was eventually promoted to the position of Associate Personnel Examiner (Tr. 13-14, 99, 107). During her tenure as Associate Personnel Examiner, she was responsible for supervision and oversight of the development of several employment examinations, including the 1998 Lieutenants' Exam (Tr. 14, 107-08).
Dr. Steinberg's oversight duties included determining the validity of the Exam, relying on the following professional standards (which she referred to as the "joint standards"): the American Psychological Association Standards for Educational and Psychological Tests (the "APA Standards") (Defendants' Exhibit 132); the Principles for the Validation and Use of Personnel Selection Procedures, authored by the Society for Industrial and Organizational Psychology (the "SIOP Principles") (Defendants' Exhibit 133); and the Uniform Guidelines on Employee Selection Procedures (the "Uniform Guidelines") adopted by the Equal Employment Opportunities Commission ("EEOC") (Tr. 108-11). She also relied on comments from the Fire Advisory Committee, a standing committee of fire personnel that oversees the Civil Service Department's efforts to develop tests for fire-related jobs (Tr. 122-23). During the time Dr. Steinberg was engaged in the development of the Exam, the Fire Advisory Committee consisted of personnel from the Office of Fire Prevention and Control, the Office of Career Chiefs, and the New York State Professional Firefighter Association. The committee also included one member from a large fire department, one member from a small fire department, and one member representing racial or ethnic diversity review issues (Tr. 120-21). Dr. Steinberg testified that the Fire Advisory Committee's primary role was to assist in the development of the job analysis for the new examination, summarized in a document she prepared entitled "1995-97 Fire Service Job Analysis" (Joint Ex. 7).
As an initial step in conducting the job analysis, Dr. Steinberg prepared a "task rating list" describing the different duties of fire personnel in all positions, from fire truck driver to fire chief. In developing the task rating list, Dr. Steinberg reviewed the job specifications submitted by Buffalo and various other jurisdictions, along with task rating lists from previous job analyses, and submitted the information to the Fire Advisory Committee for further suggestions on the tasks to be included on the list. The completed list of 190 tasks was sent out as a survey in April 1995 to all full-time paid incumbent fire personnel in all fire departments in New York State (outside of New York City), asking them to rate such things as the frequency they perform the listed tasks, the importance of those tasks, the amount of time they spend on those tasks, and when they needed to perform those tasks (Tr. 123-24, 126-29).
The next step in the job analysis process was to prepare and distribute to incumbents a survey of the skills, knowledges, abilities, and personal characteristics ("SKAPs") employees in each job title needed to perform the tasks identified in the task survey. The SKAP survey was prepared in a manner similar to the task survey, with review and editing by the Fire Advisory Committee. The final version listing 150 SKAPs was sent out to incumbents in October 1996 (Tr. 129-33; Joint Ex. 15). Dr. Steinberg also sent surveys to 13 other jurisdictions nationwide, including large municipalities such as Los Angeles, Baltimore, Miami, Denver, New York City, the District of Columbia, and Chicago, seeking information regarding the minimum qualifications and test plans they employ for all level of competitive fire positions under their jurisdictions (Tr. 135-38; Joint Ex. 7, p. 12). According to Dr. Steinberg, the test plans used by these jurisdictions were very similar to the 1998 Lieutenants' Exam (Tr. 138).
Dr. Steinberg reported in her job analysis summary that several larger jurisdictions, including Buffalo, refused to participate in the task/SKAP survey at a meaningful level (see Joint Ex. 7, pp. 6, 11). When questioned about this by plaintiffs' counsel, Dr. Steinberg testified that she sent Buffalo directly over 900 surveys to cover 833 fire positions, and followed up with two additional mailings through Civil Service and the statewide union office, but she received only 68 responses to the task survey and no responses to the SKAP survey (Tr. 54-55; 59-62: see also Joint Exs. 13, 17). According to Dr. Steinberg, this was insufficient data to determine whether tasks needed for the fire lieutenant job in Buffalo required the same knowledge, skills, and abilities as in other jurisdictions, so she compared the responses she received to the responses from other large fire departments in New York State and nationwide, and then compared that data to information about the fire lieutenant job obtained from the Fire Advisory Committee (Tr. 67-68). She also testified that while she was not aware of any studies showing that the fire lieutenant job in Buffalo is so unique that it differs significantly from other large fire departments statewide and nationwide, there are several "validity generalization" studies available to show that tasks for similar positions require the same knowledge, skills, and abilities regardless of where they are performed (Tr. 69-70).
Dr. Steinberg testified that after collecting the responses to the task rating and SKAP surveys, the next step in her job analysis was to analyze the data to determine what topics to include on the Exam. This included determining how many responses had been received, determining the average values for each of the questions asked about each of the tasks, and assigning an overall value to each task. She then consulted with the Fire Advisory Committee to set an appropriate cutoff value for determining the tasks to be included on the Exam, and to determine which tasks should be grouped into similar areas to develop appropriate subtests (Tr. 139-40).
Dr. Steinberg identified two methods for developing the subtests: rational grouping, which involves the use of ordinary judgment to combine similar tasks for inclusion in a subtest, and factor analysis, which is a statistical technique used to group survey data by similarity of response. Dr. Steinberg testified that the subtest groupings developed from the two methods matched "almost perfectly" (Tr. 140).
Dr. Steinberg identified several documents related to her job analysis. Defendants' Exhibit 107 is a copy of the task listing sent out to all incumbents, with Dr. Steinberg's handwritten indication of the overall values assessed for tasks considered critical to performing the job of lieutenant. Defendants' Exhibit 123 is a document entitled "Fire Service Task Criticality Ratings" which Dr. Steinberg described as the guidelines she used to combine the different factors of importance, frequency, and consequence of error into an overall rating for each task. Defendants' Exhibit 108 is a handwritten chart entitled "Task Criticality Ratings" which indicates overall importance, frequency, and consequence of error ratings for each of the 190 tasks and for each job title (Tr. 140-43). Joint Exhibit 21 is a series of computer printout sheets identified in Dr. Steinberg's handwriting as "Factor Analysis for Critical Day 1 Tasks" for the job title of fire lieutenant. Dr. Steinberg testified that she utilized the information on these computer printouts to determine which tasks should be grouped together to develop the questions for the fire-related subtests of the 1998 Lieutenants' Exam (Tr. 145-46). Defendants' Exhibit 106 is a document entitled "Fire Service Task-SKAP Linkage," which identifies the SKAPs needed to do each of the listed tasks (Tr. 147-48). This document was developed upon consultation with a panel of Testing Services Division personnel and was approved by the Fire Advisory Committee as another way of determining the appropriate subtest areas for the Exam (Tr. 148).
Joint Exhibit 20 is a document entitled "Final Scope," which identified six subtest areas for the 1998 Lieutenants' Exam: (1) fire attack and suppression, (2) fire prevention, (3) rescue and first responder, (4) understanding and interpreting written material, (5) training practices, and (6) supervision. Subtests (1), (2), and (3) are referred to as the "fire-related" subtests, and subtests (4), (5), and (6) are referred as the "generic" or "cross-occupational" subtests (Tr. 149-50). Dr. Steinberg testified that she included the three generic subtests on the Exam because the ratings of the tasks associated with these activities were very high, and consultation with the Fire Advisory Committee and the Office of Fire Prevention and Control confirmed that these activities were important parts of the fire lieutenant's job. These subtest areas were also included in the test plans for the title of fire lieutenant from other jurisdictions nationwide, and were identified as knowledges, skills, and abilities ("KSAs") required for the job of fire lieutenant on the job specifications provided by the statewide municipalities, including Buffalo (Tr. 150-54). The questions for the generic subtests were developed by Civil Service Department units specializing in cross-occupational testing upon request of Dr. Steinberg, with reference to the results of the job analysis, the listings of required tasks and KSAs, and the particular municipality's job specifications (Tr. 158-60).
Dr. Steinberg also testified that she recruited subject matter experts ("SMEs") from all jurisdictions across New York State for input to assist in the development of the questions to be included on the fire-related subtests. Defendants' Exhibit 119 is a copy of the Department of Civil Service memorandum, dated October 21, 1997, that was sent to local civil service agencies statewide requesting nominees and listing the qualifications for SMEs to help develop questions for the fire-related subtests. Dr. Steinberg held a two-day meeting with the SMEs in December 1997 at the Fire Academy in Montour Falls, at which the SMEs were given instructions about how to write test questions and were provided with a list of proposed topics (Defendants' Ex. 118). The SMEs drafted questions, which were reviewed and edited first by Dr. Steinberg and her supervisor Paul Kaiser, and then by the Fire Advisory Committee (Tr. 154-58; see also Defendants' Exs. 113, 114, and 115).*fn2 Although invited to participate, the Buffalo Fire Department did not send any SMEs to the December 1997 meeting in Montour Falls (Tr. 86-87).
Paul Kaiser has been employed by the New York State Civil Service Department since 1972 (Tr. 180-81). He is currently Director of the Testing Services Division, which provides examination services to all State agencies, and upon request to local jurisdictions, for development and scoring of written examinations to be used in the selection of employees for competitive class positions (Tr. 204-05).
Mr. Kaiser worked his way up through the ranks of the Testing Services Division, and is familiar with all aspects of the examination development process. The entry level position is Personnel Examiner Trainee. Upon satisfactory completion of the traineeship, the employee is eligible for promotion to the position of Senior Personnel Examiner, with primary responsibility for the review of the material to be included on a particular examination. Upon successful completion of an examination, the employee is eligible for promotion to the supervisory position of Associate Personnel Examiner, which is the position held by Dr. Steinberg at the time the 1998 Lieutenant's Exam was developed. The next position is Principal Personnel Examiner, with responsibility for supervision of the Associate Examiners and their units, followed by the positions of Chief Personnel Examiner, Assistant Director, and finally Director (Tr. 205-06).
Mr. Kaiser testified that all examinations developed by the Testing Services Division are subject to rigorous internal review, guided by the joint standards (i.e., the APA Standards, the SIOP Principles, and the EEOC's Uniform Guidelines) (Tr. 208). At a minimum, the material to be included in a particular examination is reviewed by either the Senior Personnel Examiner or the Associate Personnel Examiner (in this case, Dr. Steinberg), who would have primary supervisory responsibility for developing the examination. The material is also subjected to a process called "pre-rating review," whereby candidates who have sat for an examination have an opportunity to review and object to test questions prior to final scoring. The examination may be exempted from pre-rating review if it has received prior approval from the Civil Service Commission. Under the prior approval process, the exam questions are reviewed initially by a Senior Personnel Examiner, then by an Associate Personnel Examiner, and then by a section head, who is either a Principal or a Chief Personnel Examiner. The exam material is then submitted for approval by the Assistant Director and, finally, by the Director (Tr. 209-10). The 1998 Lieutenant's Exam received prior approval from the Civil Service Commission after being subjected to these numerous levels of internal review (Tr. 211).
Mr. Kaiser testified that the Testing Services Division is organized by occupational specialty, with certain units designated as cross-occupational to develop examination material which focuses on competencies that are common across numerous occupations rather than on knowledge unique to a particular field (Tr. 211-12). He testified at some length about the Division's routine practices for selecting questions to be included on cross-occupational subtests, and explained the different factors that would be considered by the different subunits responsible for the three subtests which were included on the 1998 Lieutenant's Exam (Tr. 212-19). For example, the supervisory subunit would consider such factors as the level of supervisory responsibilities of the job, and whether the supervision involved a white-collar (or clerical) or blue collar (out in the field) work environment. The subunit would also look at the history of the subtest to determine how well the questions have functioned on previous examinations (Tr. 218-19; see also Defendants' Ex. 135).
Mr. Kaiser testified that the overall process of validating a vocational examination involves the accumulation of evidence which provides information that the test is appropriate for its purpose (Tr. 226). This evidence includes the historical data about past use of the subtests, objections from the jurisdictions requesting the exams, and information obtained from individuals with experience in the jobs for which the exams are being developed (Tr. 227-34).
As reflected in his considerable curriculum vitae (Plaintiffs' Ex. 24), Dr. Murphy has a Ph.D. in Industrial/Organizational Psychology. He has taught, lectured, and published extensively in the areas of employment testing, including job analysis, performance assessment, and validation, and has provided testimony in several employment testing cases over the course of his 25-year career (Tr. 250-51).*fn3
Dr. Murphy testified that he prepared a "Statistical Analysis of the Scores by White and African-American Test Takers on the 1998 Buffalo Fire Lieutenant Examination" (Plaintiffs' Exhibit 25), which reported pass rates of 74.3% for White examinees, and 42.6% for African-American examinees (Tr. 252-53). This resulted in an "adverse impact ratio" of .573, significantly lower than the "four-fifths" or "eighty percent" rule adopted by the EEOC's Uniform Guidelines, which allows an inference of adverse impact to be drawn where the passing rate for a disadvantaged group is less than 80 percent of the passing grade of the highest scoring group (Tr. 253-54).
Dr. Murphy also performed a statistical analysis of the scores on the Exam as a whole and on each of the separate subtests of the Exam, using standardized formulas for eliminating random chance or sampling error (Tr. 254-55). According to Dr. Murphy, this analysis revealed significant systematic differences in favor of White applicants, indicating that the Exam "had a very substantial effect on the employment opportunities of black versus white examinees." (Tr. 255).
In addition to conducting a statistical analysis of the test scores, Dr. Murphy's duties as plaintiffs' employment testing expert included evaluating the validity of the 1998 Lieutenant's Exam. He testified that he performed his evaluation in accordance with the joint standards--including the APA Standards and the SIOP Principles, which he helped develop--and concluded that there was insufficient credible evidence ...