The opinion of the court was delivered by: JOSEPH M. MCLAUGHLIN
McLAUGHLIN, Circuit Judge*
Plaintiffs -- states, cities, citizens' groups, and individual citizens and taxpayers -- seek a judgment: (1) vacating former Secret of Commerce Robert Mosbacher's July 15, 1991 decision that the 1990 census would not be statistically adjusted; (2) ordering that such an adjustment be made; and (3) allowing plaintiffs to use and publicize certain data generated by the Census Bureau, and already produced, subject to a protective order, to the plaintiffs during this litigation. For the reasons set forth below, the Court holds that the decision against adjustment shall not be disturbed, but grants the plaintiffs' request to use and publish the Census Bureau data. The following constitute the Court's findings of fact and conclusions of law in accordance with Federal Rule of Civil Procedure 52.
Just to recount the facts of this case is arduous, given its four-year history, the number of parties involved, and the complicated statistical evidence lying at the core of the dispute. Many of the material facts have been set forth in two prior published opinions -- City of New York v. United States Dep't of Commerce, 713 F. Supp. 48 (E.D.N.Y. 1989) ("City of New York I"), and City of New York v. United States Dep't of Commerce, 739 F. Supp. 761 (E.D.N.Y. 1990) ("City of New York II") -- some familiarity with which is assumed.
The Constitution requires a decennial census. Article I, Section 2, Clause 3 states that "the actual enumeration shall be made [every ten years], in such manner as [the Congress] shall by Law direct." Congress has, in turn, delegated to the Secretary of Commerce the duty of taking the census "in such form and content as he may determine, including the use of sampling procedures and special surveys." 13 U.S.C. § 141(a) (1982). The Bureau of the Census, an agency within the Department of Commerce, actually conducts the census. See 13 U.S.C. § (1982).
The results of the census are used for a galaxy of purposes. The federal government uses them to calculate how to dispense program funds among the states. States use the counts for political redistricting. Sociologists and historians study them for more esoteric purposes. None of this obscures the central truth that the "basic constitutional purpose" of the census is "to determine the apportionment of Representatives among the States." Carey v. Klutznick, 653 F.2d 732, 736 (2d Cir. 1981).
The first census of the American population was in 1790. Thomas Jefferson, who was in charge of it, complained of an undercount. There have been 20 subsequent censuses. Each of them has also resulted in an undercount. More troubling than the undercount itself, however, is that racial and ethnic minorities are undercounted to a greater degree than the population as a whole. This problem, known antiseptically as the "differential undercount," has skewed every census since the Bureau started measuring it in 1940.
Taking the census has always been a daunting task, and the 1990 count was no exception. The Bureau began preparing in 1983, seeking to improve the techniques that it had used in prior censuses. Among other things, it consulted with state and local governments, planned an extensive advertising campaign, designed a more ethnically inclusive census questionnaire, and increased the amount of automation used, including the use of an automated geographic control system, which assured accurate and timely maps and geographic files for the 1990 census. While the parties may disagree on the quality of the census counts achieved in 1990, the four-step procedure used to conduct the census is largely undisputed.
First: an address list of housing units was compiled. This list was crucial because it indicated every household in the nation to which the Bureau would send questionnaires. Since the Bureau relies on the mail return of those questionnaires to count a majority of the population, an accurate and comprehensive list was vitally important. In constructing the list, the Bureau relied primarily on commercial mailing lists, supplemented by extensive field research and collaboration with the Postal Service. Then, numerous quality controls were instituted to improve the accuracy of the list.
Second: census questionnaires were mailed to each housing unit. Householders were asked to complete and return the questionnaires to the local census district office on or before April 1, 1990.
This is called the "mail out/mail back" phase. The effort to get individuals to participate in the mail out/mail back phase was extensive. In addition to the Census Bureau's general advertising campaign, it also conducted campaigns specifically targeted at African-Americans, Asians, Hispanics, and Native Americans. In addition, the Bureau published specialized, foreign-language brochures encouraging public participation in the census. It also maintained a set of toll-free numbers (in eight languages) for anyone who had questions regarding the census questionnaire, and every census form advised Spanish speakers that they could call a toll-free 800 number to obtain a census form in Spanish. Finally, the Census Bureau employed different methods in areas where it was believed that the normal procedure would be particularly ineffective. See Secretary of the Department of Commerce, Decision on Whether or Not a Statistical Adjustment of the 1990 Decennial Census of Population Should be Made for Coverage Deficiencies Resulting in an Overcount or Undercount of the Population, July 15, 1991 (the "Decision"), at 4-5-4-6.
Third: because the return rate of census questionnaires is obviously never 100%, and in 1990 was only 63%, see Transcript of Trial ("Tr.") at 1823, the Census Bureau embarked on an extensive follow-up campaign. Second mailings were sent to households that failed to return the initial form, and in census districts with particularly low return rates, the Bureau remailed census forms to all residents. Tr. at 1730-31.
After the enumeration was completed, post-enumeration "Coverage Improvement Programs"
were implemented, with the result that 5.4 million people were added to the counts. Decision at 4-7. The result of all of these efforts was that 249,632,692 people were counted during the 1990 census. Decision at 4-2.
The Differential Undercount
Despite the herculean efforts of the Census Bureau, it is undisputed that the 1990 Census was not -- and could not realistically be -- successful in its goal of achieving an exact count of the nation's population. Given the nature of the task, it is not surprising that the census fails to count some individuals ("omissions") and also adds persons into the count erroneously ("erroneous enumerations"). Tr. at 80-82.
The "net undercount" is the difference between omissions and erroneous enumerations. It is undisputed that the 1990 census, like all previous censuses, resulted in a net national undercount. Decision at 1-1. It is similarly uncontroverted that African-Americans and other minorities have been persistently undercounted to a greater degree than non-Hispanic whites in all censuses since 1940 when the Bureau began measuring such differences, and that this anomaly is perpetuated in the 1990 census. The difference between the undercount rate for non-Hispanic whites and that for minority populations is known as the "differential undercount." Tr. at 91-92. According to the Secretary, "Blacks appear to have been undercounted in the 1990 census by 4.8%, Hispanics by 5.2%, Asian-Pacific Islanders by 3.1%, and American Indians by 5.0%, while non-Blacks appear to have been undercounted by 1.7%." Decision at 1-1.
Possibility of Statistical Adjustment
The Census Bureau has been aware of the existence of a differential undercount since the 1950's. The intractable problem has been how to fix it. Following the 1980 census, concerns over the persistence of the differential undercount, its deleterious effects on the accuracy of census counts, and the unfair results arising from such inaccuracy, prompted the Bureau to start a research program aimed at developing statistical techniques to ameliorate the problem in the 1990 census.
Tr. at 525, 1291-92. By 1984, the Bureau had developed a timetable for internal Bureau research that would ultimately lead to a decision whether to adjust the 1990 census statistically in an effort to reduce the differential undercount. Two task forces were created to consider the undercount problem as it related to the upcoming 1990 census: The Undercount Steering Committee ("USC") was responsible for planning undercount research and policy development. The Undercount Research Staff ("URS") conducted the actual research. Other divisions at the Bureau also conducted research on the undercount and the possibility of adjustment. Tr. at 517-25, 1292-93. In addition, the Bureau sought the opinions of outside experts and organizations, such as the American Statistical Association and the National Academy of Science, regarding the possibilities for adjustment.
After considering the alternatives, the Bureau settled upon the PES as the best tool to statistically adjust the census through the use of "dual system estimation" ("DSE"). Tr. at 559-61. Dual system estimation or, in more pedestrian terms, "capture/recapture," is, as relevant here, an approach that uses a second measurement to ascertain the quality of the estimate obtained by an initial measurement, and then uses that information to provide a purportedly more accurate, dual system estimate.
Here, the original enumeration, the census, was followed by a second measurement, the PES, which attempted to measure the rate at which people were omitted and erroneously enumerated by the census, in order to determine a net undercount rate.
While the Bureau has used post-enumeration surveys in a variety of ways since 1950, it has never statistically adjusted based on DSE. The Bureau worked throughout the 1980's to design the PES to make it an effective tool for census adjustment. Tr. at 572. For example, correlation bias, which may occur when residents become confused by an overlap between the census and the PES, was addressed by distinctly separating the two procedures. Tr. at 578-82. Another species of correlation bias, which arises when individuals who have different probabilities of being counted ("capture probabilities") in the census are grouped together in the PES, was reduced by the use of "poststratification." Tr. at 205-208.
In addition, statistical "smoothing" was chosen to address anomalous results in the PES.
By the Spring of 1987, after much testing and fine-tuning, the Census Director, John Keane, had decided that the Bureau should proceed with plans to adjust the 1990 census data through the use of DSE, if the PES results met a certain quality standard. Dr. Keane met with his superior, Robert Ortner, the Under Secretary of the Department of Commerce, to tell him that such a decision had been made and that a press conference to that effect was imminent. Six days later, Keane met again with Ortner and other Commerce Department officials, who informed Keane that they had decided against adjustment. Shortly thereafter, Commerce Department officials instructed their Census Bureau officials not to disclose that a decision had been made. Tr. 629-30, 1330. On October 30, 1987 the Department of Commerce announced its decision against adjustment, and this lawsuit was born.
History of This Litigation
In November, 1988, plaintiffs sued to enjoin the 1990 census, challenging the methodology by which it would be taken, and seeking to reverse the decision against adjustment. Defendants -- the Department of Commerce, its Secretary, President Bush, and other officials within the Department of Commerce and its subsidiary, the Bureau of the Census -- moved to dismiss the application for the injunction. This Court denied the dismissal motion, holding that the plaintiffs had standing to challenge the census on constitutional grounds;
the Court also ruled that it would consider the Commerce Department's decision against adjustment under the "arbitrary and capricious" standard of review of the Administrative Procedure Act, 5 U.S.C. § 706(2)(A) (1982) (the "APA"). City of New York I, 713 F. Supp. at 54.
When the dismissal motion was denied, a hearing was scheduled on the injunction. It was set to go forward in the Summer of 1989, when, at the eleventh hour, the parties entered into a stipulation (the "Stipulation" or the "Stip."). The Stipulation vacated the Commerce Department's 1987 decision against adjustment and agreed that the new Commerce Secretary, Robert Mosbacher, would consider de novo and "with an open mind," whether adjustment was warranted. Stip. at 2-3. The Stipulation also agreed that the program to gather the statistical data necessary for adjustment would proceed, that the Secretary would decide whether to adjust by July 15, 1991, and that his decision would be consistent with certain procedures, including the promulgation of "guidelines" articulating what the defendants believed to be the relevant technical and policy considerations affecting the decision. It also mandated the creation of an eight-member Special Advisory Panel (the "Panel")
of statistical and demographic experts to advise the Secretary on whether to adjust. Stip. at 4-5.
The defendants adopted and promulgated the required guidelines, but the plaintiffs challenged them as inadequate, and they also sought a declaratory judgment that a statistical adjustment would not violate the Constitution or any federal statute. Defendants countered that the plaintiffs' challenge to the census presented a non-justiciable political question. This Court rejected the defendants' political question claim, and concluded that statistical adjustment, per se, would not violate either the Constitution or the laws of the United States. City of New York II, 739 F. Supp. at 767-68. This Court noted that, while the guidelines were vague, they did satisfy, albeit just barely, the defendants' obligations under the Stipulation. Id. at 770.
The Post-Enumeration Survey
Following this Court's decision in City of New York I, the Bureau resumed work on its plans to implement the PES, and implemented it in 1990. In the first step of the PES, the Bureau methodically selected approximately 5000 blocks
in an effort to attain an appropriate sample size for each poststratum. In February 1990, Bureau employees visited each sample block and listed all the housing units they found, identifying approximately 170,000 households. In July 1990, Census Bureau interviewers returned to each address to obtain information regarding the residency status of those households on Census Day. The Bureau found that those blocks contained approximately 400,000 people. Tr. at 208. After collecting the PES data, the Bureau matched it to the information collected in the original enumeration for those same sample blocks. From this matching, the Bureau endeavored to estimate, for each poststratum, rates of omission and erroneous enumeration, and from these calculated a net undercount rate for each poststratum. Tr. at 221.
The Bureau used these results to develop an "adjustment factor" for each poststratum, i.e., the number by which the population count as indicated by the census had to be multiplied so that the entire census would reflect the variations found in the PES. Accordingly, the 1,392 poststrata resulted in 1,392 corresponding adjustment factors. One further statistical twist to the use of the PES was the employment of "smoothing."
After smoothing, the Bureau used the smoothed adjustment factors to produce adjusted counts down to the block level, which were then aggregated to provide population estimates for cities, counties, states, and the nation. Tr. at 224-25; Decision at 4-18.
A number of quality control checks were made to test the results of the PES. First, the Bureau conducted or commissioned more than twenty formal research projects, called "P-Studies," to study the potential sources of error within the PES. The results of these P-Studies regarding particular sources of error were then combined in the "total error model" that summarized the overall quality of the PES data. Tr. at 652-59.
The final result of the PES was that the census enumeration was estimated to have undercounted the population by 5,269,917, or 2.07%. In terms of the differential undercount, the PES indicated that the census undercounted Hispanics by 5.2%, African-Americans by 4.8% and Asian/Pacific Islanders by 3.1%. The PES-calculated undercount for non-African-Americans was 1.7% and 1.2% for non-Hispanic whites, with a total national undercount of 2.1%.
The Bureau also conducted a number of "loss function analyses" to compare the quality of enumeration counts to the adjusted counts. A loss function analysis is a systematic way of assessing the consequences flowing from a particular decision. In the context of the adjustment decision, the Bureau used loss function analysis to determine whether the adjusted data were expected to be more accurate than the unadjusted data. Tr. at 1941-42. This Court is satisfied that for most purposes the PES resulted in a more accurate -- or to be statistically fashionable, a less inaccurate -- count than the original census.
The Secretary's Decision and The Trial
On July 15, 1991, in accordance with the Stipulation, Secretary Mosbacher went on national television to announce his decision not to adjust. Simultaneously, he produced the Decision, a 178-page report giving the reasons for his decision. The decision revitalized the case and discovery resumed. Claiming that the Secretary's decision violated the Constitution, the APA, and the Stipulation, the plaintiffs requested a trial. More specifically, they alleged that the administrative record proffered by the Secretary as the basis of his decision is a self-serving, post-hoc compilation of documents assembled for the purpose of strengthening the defendants' litigation position and that the Secretary's decision was tainted by partisan political influence. Over the defendants' objection, this Court ordered a trial, which consisted almost exclusively of expert testimony in the fields of demographics and statistics, and continued for thirteen trial days.
The expert witnesses expressed their opinions as to whether the Secretary considered all the factors specified in the guidelines in making his decision, and also analyzed at length the conclusions that the Secretary reached in the Decision. Plaintiffs' direct case consisted of the testimony of nine witnesses, including all four of the plaintiffs' designees to the Panel. It also included the introduction of hundreds of exhibits and numerous deposition transcripts from other witnesses.
Defendants' evidence was similarly grand in scope. They presented five expert witnesses, including one Panel member. They also introduced the deposition transcripts of other witnesses and numerous exhibits. Of these, Exhibit 1, denominated as the Administrative Record by the defendants, and skeptically dubbed "the so-called Administrative Record" by the plaintiffs, contains over 12,000 documents and occupies 18,000 pages. The trial transcript exceeds 2,600 pages.
Plaintiffs allege that the Secretary's decision not to adjust the census count violates the APA, the Constitution, and the Stipulation.
They also argue that the process the Secretary used to make his decision was a sham.
They seek a order directing the Secretary of Commerce to make the adjustment and they ask for permission to use Census Bureau data provided to ...