Microsoft Word - Final Report on Mass, Computer-Generated, and Fraudulent Comments (Final 06-01-2021).docx

REPORT FOR THE

ADMINISTRATIVE CONFERENCE OF THE UNITED STATES

MASS, COMPUTER-GENERATED, AND FRAUDULENT

COMMENTS

Steve Balla

George Washington Regulatory Studies Center

Reeve Bull

Administrative Conference of the United States

Bridget Dooling

George Washington Regulatory Studies Center

Emily Hammond

George Washington Law School

Michael Herz

Cardozo School of Law

Michael Livermore

University of Virginia School of Law

Beth Simone Noveck

The Governance Lab

This report was prepared for the consideration of the Administrative Conference of the United States. It does not

necessarily reflect the views of the Conference (including its Council, committees, or members).

Recommended Citation

Steve Balla, Reeve Bull, Bridget Dooling, Emily Hammond, Michael Herz, Michael Livermore, & Beth Simone

Noveck, Mass, Computer-Generated, and Fraudulent Comments (June 1, 2021) (report to the Admin. Conf. of the

U.S.).

Executive Summary

This report explores three forms of commenting in federal rulemaking that have been

enabled by technological advances: mass, fraudulent, and computer-generated comments. Mass

comments arise when an agency receives a much larger number of comments in a rulemaking than

it typically would (e.g., thousands when the agency typically receives a few dozen). The report

focuses on a particular type of mass comment response, which it terms a “mass comment

campaign,” in which organizations orchestrate the submission of large numbers of identical or

nearly identical comments. Fraudulent comments, which we refer to as “malattributed comments”

as discussed below, refer to comments falsely attributed to persons by whom they were not, in fact,

submitted. Computer-generated comments are generated not by humans, but rather by software

algorithms. Although software is the product of human actions, algorithms obviate the need for

humans to generate the content of comments and submit comments to agencies.

This report examines the legal, practical, and technical issues associated with processing

and responding to mass, fraudulent, and computer-generated comments. There are cross-cutting

issues that apply to each of these three types of comments. First, the nature of such comments may

make it difficult for agencies to extract useful information. Second, there are a suite of risks related

to harming public perceptions about the legitimacy of particular rules and the rulemaking process

overall. Third, technology-enabled comments present agencies with resource challenges.

The report also considers issues that are unique to each type of comment. With respect to

mass comments, it addresses the challenges associated with receiving large numbers of comments

and, in particular, batches of comments that are identical or nearly identical. It looks at how

agencies can use technologies to help process comments received and at how agencies can most

effectively communicate with public commenters to ensure that they understand the purpose of the

notice-and-comment process and the particular considerations unique to processing mass comment

responses. Fraudulent, or malattributed, comments raise legal issues both in criminal and

Administrative Procedure Act (APA) domains. They also have the potential to mislead an agency

and pose harms to individuals. Computer-generated comments may raise legal issues in light of

the APA’s stipulation that “interested persons” are granted the opportunity to comment on

proposed rules. Practically, it can be difficult for agencies to distinguish computer-generated

comments from traditional comments (i.e., those submitted by humans without the use of software

algorithms).

While technology creates challenges, it also offers opportunities to help regulatory officials

gather public input and draw greater insights from that input. The report summarizes several

innovative forms of public participation that leverage technology to supplement the notice and

comment rulemaking process.

The report closes with a set of recommendations for agencies to address the challenges and

opportunities associated with new technologies that bear on the rulemaking process. These

recommendations cover steps that agencies can take with respect to technology, coordination, and

docket management.

Introduction

In 2015, the Environmental Protection Agency (EPA) issued a rule defining the “Waters

of the United States.” During the course of the rulemaking, the EPA received more than one million

comments.

Over ninety percent of the comments were submitted as part of mass comment

campaigns. At about the same time, a large number of mass comment campaigns was also

submitted in response to the EPA’s proposed “Clean Power Plan.” These and other similar

occurrences signaled the regularity of mass comment campaigns, at least in the context of highly

salient, controversial rulemakings.

In February 2020, the Subcommittee on Oversight and Investigations of the U.S. House of

Representatives Committee on Financial Services investigated allegations that the Securities and

Exchange Commission (SEC) may have received fraudulent comments from “an Army veteran, a

Marine veteran, a single mom, and a couple of retirees.”

During the committee hearing, witnesses

alleged that those public comments were sent by advocacy groups and signed with the names of

people who never saw the comments or who did not exist at all. Although these allegations have

been disputed, the possibility of fraudulent comments in the government’s rulemaking process has

prompted questions and concern.

In 2017, the Federal Communications Commission (FCC) “Restoring Internet Freedom”

(i.e., net neutrality) rulemaking attracted a record number of public comments: almost 22 million

by the official close of the comment period, with another three million arriving after the fact.

Although about six percent of the comments were unique, the rest were submitted multiple times,

in some cases hundreds of thousands of times.

On nine different occasions, more than 75,000

comments were dumped into the docket at the very same second.

The comments “included

comments from stolen email addresses, defunct email accounts and people who unwittingly gave

See Steven J. Balla et al., Lost in the flood?: Agency Responsiveness to Mass Comment Campaigns in Administrative

Rulemaking, REGUL. & GOVERNANCE (May 20, 2020), https://onlinelibrary.wiley.com/doi/abs/10.1111/rego.12318;

Rachel Augustine Potter, More than Spam?: Lobbying the EPA through Public Comment Campaigns, BROOKINGS

INST. (Nov. 19, 2017), https://www.brookings.edu/research/more-than-spam-lobbying-the-epa-through-public-

comment-campaigns/.

Fake It Till They Make It: How Bad Actors Use Astroturfing to Manipulate Regulators, Disenfranchise Consumers,

and Subvert the Rulemaking Process: Hearing Before the H. Fin. Servs. Comm., 116th Congress (Feb. 6, 2020)

(hereinafter Fake It Till They Make It).

In 2015, the FCC issued its “net neutrality rule,” prohibiting broadband Internet providers from blocking, degrading,

or interfering with Internet traffic. Protecting and Promoting the Open Internet, 80 Fed. Reg. 19737 (June 12, 2015).

The rule was upheld in United States Telecomm. Ass’n v. FCC, 825 F.3d 674 (D.C. Cir. 2016). In 2017, the FCC

proposed repealing the 2015 rule. Restoring Internet Freedom, Notice of Proposed Rulemaking, 32 FCC Rcd 4434

(2017). The Notice of Proposed Rulemaking was released in May and published in the Federal Register in June. 82

Fed. Reg. 25,568 (June 2, 2017). The Final Rule was promulgated in early January 2018. See Restoring Internet

Freedom, 83 Fed. Reg. 7852 (Feb. 22, 2018). For information on the comments, see Paul Hitlin, Kenneth Olmstead,

& Skye Toor, Public Comments to the Federal Communications Commission About Net Neutrality Contain Many

Inaccuracies and Duplicates, PEW RES. CTR. (Nov. 29, 2017), https://www.pewresearch.org/internet/wp-content/

uploads/sites/9/2017/11/PI_2017.11.29_Net-Neutrality-Comments_FINAL.pdf.

Id.

permission for their comments to be posted.”

A consulting firm later determined that about a third

of the comments were sent from temporary or disposable email domains, and about 10 million

were from senders of multiple comments.

FCC Commissioner Jessica Rosenworcel has stressed

that 500,000 or so comments came from Russia.

The New York Attorney General concluded that

9.3 million comments were submitted with false identities, including 7 million from a single

submitter.

In sum, the nature of public participation—the mass occurrence of identical and near

duplicate comments, the malattribution of identities, and the apparent automation of comment

submission—called into question elements of the process behind the FCC’s regulation.

Partly in response to issues posed by such rulemakings, in 2019 the U.S. Senate Permanent

Subcommittee on Investigations issued a staff report entitled, “Abuses of the Federal Notice-and-

Comment Rulemaking Process.”

The report identified problems associated with mass,

fraudulent, and computer-generated comments, including a lack of agency processes and policies

aimed at identifying, managing, and addressing such comments.

This attention underscores the importance of comments for the rulemaking process, which

generates thousands of regulations every year that touch many aspects of economic and social

life.

The Administrative Conference of the United States (ACUS) has also been tracking these

issues for several years. In 2018, ACUS held a forum addressing mass and fraudulent comments.

At the time, the general consensus of the panelists, which included prominent academics with

James V. Grimaldi & Paul Overburg, Millions of People Post Comments on Federal Regulations. Many Are Fake.,

WALL ST. J. (Dec. 13, 2017), https://www.wsj.com/articles/millions-of-people-post-comments-on-federal-

regulations-many-are-fake-1513099188. A video on the newspaper’s website summarizes: “[T]he Wall Street Journal

uncovered thousands of comments from fake email addresses, abandoned or defunct email accounts, posted on behalf

of unwitting participants. For example, 818,000 identical comments on the FCC site favor repealing the rules. In a

random sample of people whose emails were used for those posts, 72% said they had nothing to do with them. Jack

Hirsch was one of them. “I was horrified. Knowing that this is actually an issue that I cared enough to write my

representatives about, and knowing that my information had been falsified to support a completely opposing view, it

was really frustrating, and honestly, I felt like there was no recourse.” Thousands of Fake Comments on Net Neutrality:

A WSJ Investigation, WALL ST. J. (Dec. 12, 2017, 12:02 PM), https://www.wsj.com/video/thousands-of-fake-

comments-on-netneutrality-a-wsj-investigation/8E52172E-821C-4D89-A2AA-2820F30B8648.html.

FCC Restoring Internet Freedom Docket 17-108: Comments Analysis, EMPRATA 2 (Aug. 30, 2017),

https://www.emprata.com/emp2017/wp-content/uploads/2017/08/FCC-Restoring-Internet-Freedom-Comments-

Analysis.pdf.

Nicholas Confessore and Jeremy Singer-Vine on Request for Inspection of Records, 33 FCC Rcd. 11808 (adopted

Nov. 7, 2018) (Rosenworcel, dissenting).

N.Y. STATE OFF. OF THE ATTORNEY GENERAL LETITIA JAMES, FAKE COMMENTS: HOW U.S. COMPANIES &

PARTISANS HACK DEMOCRACY TO UNDERMINE YOUR VOICE (2021). See also Letter from Eric Schneiderman, Att’y

Gen., N.Y., to Thomas M. Johnson, Jr., Gen. Counsel, FCC (Dec. 13, 2017), https://ag.ny.gov/sites/default/files/ltr_t

o_fcc_gen_counsel_re_records_request.pdf (noting 8 million comments filed under false identities).

PERMANENT SUBCOMMITTEE ON INVESTIGATIONS, U.S. SENATE COMMITTEE ON HOMELAND SECURITY AND

GOVERNMENT AFFAIRS, STAFF REPORT, ABUSES OF THE FEDERAL NOTICE-AND-COMMENT RULEMAKING PROCESS

(2019), https://www.hsgac.senate.gov/imo/media/doc/2019-10-24%20PSI%20Staff%20Report%20-

%20Abuses%20of%20the%20Federal%20Notice-and-Comment%20Rulemaking%20Process.pdf (hereafter

“Subcommittee Report”).

CONG. RSCH. SERV., R43056, COUNTING REGULATIONS: AN OVERVIEW OF RULEMAKING, TYPES OF FEDERAL

REGULATIONS, AND PAGES IN THE FEDERAL REGISTER (2019).

expertise in rulemaking and governmental officials working in the rulemaking space, was that

agencies were mostly well-equipped to handle mass comments and that fraudulent comments,

though unseemly, were relatively uncommon and didn’t merit any extensive efforts on the part of

agencies to combat them. Deputy Office of Information and Regulatory Affairs (OIRA)

Administrator Dominic Mancini emphasized that mass comment responses were rare.

And

Deputy Assistant Attorney General Matthew Miner noted that a criminal prosecution for

submitting false information to the government could be difficult to sustain and potentially raise

First Amendment concerns.

Under the 1946 APA, the public has a right to participate by commenting on draft

regulations in the rulemaking process, which is why it is often referred to as notice-and-comment

rulemaking.

Such participation in rulemaking enhances both the legitimacy and the quality of

regulations by enabling agencies (and the executive offices and congressional committees that

oversee them) to obtain information from a wide audience of stakeholders, interest groups,

businesses, nongovernmental organizations (NGOs), academics, and interested individuals.

Participation also provides an accountability check on the rulemaking process by ensuring public

scrutiny prior to rules going into effect. Overall, the process facilitates the reason-giving

requirement that is necessary for a rule to survive judicial review.

The shift over the last two decades to a digital process, in which participants submit

comments via the Internet, has made commenting easier. Expanding the ability to participate in an

important governance process is an overwhelmingly positive change. Rulemaking has long been

criticized as an insiders’ game, invisible to the general public, with modest levels of participation,

almost entirely by organizations rather than individuals. But, although online participation has the

potential to increase the quantity and diversity of participation, it has also inadvertently opened

the floodgates to mass comment campaigns, malattributed comments, and computer-generated

comments, potentially making it harder for agencies to extract the information needed to inform

decision making and undermining the legitimacy of the rulemaking process.

As this discussion indicates, the technology of electronic commenting has enabled three

kinds of potentially problematic comments that are the subject of this report:

1. Mass comments (sometimes orchestrated as a campaign by one or more entities);

2. Fake, fraudulent, or what this report below refers to as “malattributed” comments; and

Dominic Mancini, Deputy Adm’r, Off. of Info. & Reg. Affs., Keynote Address at the Administrative Conference

of the United States Symposium on Mass and Fake Comments in Agency Rulemaking 14 (Oct. 5, 2018) (transcript

available at https://www.acus.gov/sites/default/files/documents/10-5-

18%20Mass%20and%20Fake%20Comments%20in%20Agency%20Rulemaking%20Transcript.pdf).

Matthew Minor, Deputy Assistant Att’y Gen., Dep’t of Justice, Remarks at the Administrative Conference of the

United States Symposium on Mass and Fake Comments in Agency Rulemaking 113-14 (Oct 5, 2018) (transcript

available at https://www.acus.gov/sites/default/files/documents/10-5-

18%20Mass%20and%20Fake%20Comments%20in%20Agency%20Rulemaking%20Transcript.pdf).

CONG. RSCH. SERV., supra note 11.

Journalistic and popular attention has focused on comments that fall into all three categories simultaneously, i.e.

mass computer-generated malattributed comments. But these three distinct characteristics do not necessarily coincide.

Each presents distinct practical and normative issues. While this report examines all three types of comments, it is

careful to disaggregate them and take into account the important ways in which they differ. These three types of

comments are defined in further detail below. See section II.A.

3. Computer-generated comments.

All three of these types of comments can generate serious challenges to agencies, raising a

pressing set of questions concerning how best to respond while ensuring the functioning of the

informal rulemaking process. The task of this report, therefore, is to evaluate whether and to what

extent such submissions are problematic, and to make recommendations for how rulemaking

agencies should respond using legal, policy, and technological strategies.

Our overarching conclusion is that agencies should adopt both low- and high-tech measures

to limit the negative impact of these sorts of comments. Mass, malattributed, and computer-

generated comments, however, do not represent a crisis for the regulatory state at this time. They

have not been found to violate federal law and do not generally undermine the integrity of notice-

and-comment rulemaking, and we are not aware of evidence of widespread substantive harms in

particular rulemaking efforts or to the rulemaking system overall. However, appropriate responses,

especially those that take advantage of new technology, could reduce the cost and negative impacts

of technology-enabled comments.

Adopting such techniques could, for example, afford agency officials more time to improve

the opportunity for a diverse public to participate in the rulemaking process meaningfully and to

augment current practices with new forms of citizen engagement. Indeed, in addition to exploring

how new technologies–the very same technologies that enable mass, fraudulent and computer-

generated comments–can also help with analyzing those comments, we also explore throughout

how technology can help regulatory officials make sense of public input and draw greater insights

from public comments of all kinds. Finally, other jurisdictions at the state and local level and

internationally are turning to the use of new technology to enable innovative forms of public

participation to improve the quality of rule- and policymaking. As detailed in Part VI, these

illustrate hopeful opportunities for future experimentation.

Written by seven professionals with expertise in administrative law, rulemaking practice,

and new technologies, this report is informed by a review of the Permanent Subcommittee’s

report,

relevant law, related legal and social scientific scholarship, and a set of interviews with

agency personnel with background in the rulemaking process at agencies with substantial

rulemaking dockets during the summer and fall of 2020. The interviews, which were not meant to

capture the views of a representative or random sample, were with staff of the EPA, the Consumer

Finance Protection Bureau (CFPB), the Department of Transportation (DOT), and the FCC, as

well as with officials from the General Services Administration (GSA) responsible for developing

and maintaining the Federal Docket Management System (FDMS). A technical advisory group of

experts drawn from government, private industry, and academia also provided feedback to the

report authors, as did an additional online roundtable of agency officials with experience in the

notice-and-comment process.

The report is divided seven parts. Part I provides a general introduction to notice-and-

comment rulemaking and the role of technology in that process. Part II discusses recent

technological development that have contributed to the growth of mass, malattributed, and

computer-generated comments, and describes some of the challenges associated with these types

The Subcommittee Report addressed a handful of additional topics, including obscenity and copyrighted materials

in public comments, that are not addressed in the current report. ACUS will provide a copy of the Subcommittee

Report to the Committee on Rulemaking as they begin deliberations on this project.

of comments. Parts III, IV, and V focus on each of these comment types in turn. Part VI discusses

technological opportunities, with a focus on current available tools that can be used to facilitate

the processing of information from the notice-and-comment process or enhance supplements to

the notice-and-comment process. Part VII offers draft recommendations based on the findings in

this report.

I. Technology and Notice-and-Comment Rulemaking

During the latter half of the twentieth century, there was considerable growth in the use of

informal rulemaking by administrative agencies.

The procedures for informal rulemaking are set

out in section 553 of the APA. A cornerstone of this process is the opportunity for members of the

public to submit comments on rulemaking proposals,

which is why it is often referred to as

notice-and-comment rulemaking. For decades, organizations and individuals have availed

themselves of this opportunity to help inform the process of regulatory development. Public

comments on agency rulemakings take a wide variety of forms that include detailed submissions

by sophisticated repeat players, short expressions of support or opposition from members of the

public, signed form letters in response to solicitations from NGOs, and technical reports from

unaffiliated experts.

There is considerable variation in the level of public participation from one rulemaking to

another. The vast majority of rulemakings are relatively unremarked upon by the public, with—at

most—participation by the stakeholders most affected by a rule.

This level of participation is not

surprising given the often highly technical and specialized nature and low visibility of many

rulemakings. While federal agencies publish the opportunity to participate in the Federal Register

(effectively, the newspaper of the federal government), they generally do not advertise

rulemakings elsewhere and the public tends to have little knowledge of the right to engage unless

a third party promotes the opportunity. In a small percentage of well-publicized rulemakings with

particular public salience—such as those highlighted above—public participation can be orders of

magnitude above the norm, with the number of comments ranging from thousands to millions.

See Christopher DeMuth, Can the Administrative State Be Tamed?, 8 J. LEGAL ANALYSIS 121, 126–27 (2016).

Cary Coglianese, Citizen Participation in Rulemaking: Past, Present, and Future, 55 DUKE L.J. 943, 945 (2006)

(“participation in rulemaking is one of the most fundamental, important, and far-reaching of democratic rights”).

See Michael A. Livermore, Vladimir Eidelman & Brian Grom, Computationally Assisted Regulatory Participation,

93 NOTRE DAME L. REV. 977 (2018).

David M. Shafie, Participation in E-Rulemaking: Interest Groups and the Standard-Setting Process for Hazardous

Air Pollutants, 5 J. INFO. TECH. & POL. 399, 403–405 (2008). See also Stuart Shapiro, When Do Agencies Change

Their Proposed Rules (Nov. 10, 2007) (presented at APPAM: Association for Public Policy Analysis & Management

Conference) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1026066 (finding that

fewer than 10% of rulemakings received more than 100 comments); John M. de Figueiredo, E-Rulemaking: Bringing

Data to Theory at the Federal Communications Commission, 55 DUKE L.J. 969 (2006). Other agency actions that are

subject to public commenting follow as similar pattern. An analysis focusing on the fifty draft guidance documents

published for public comment by the Environmental Protection Agency (EPA) from 2011-2014 found that just eight

received more than 5,000 comments, with five exceeding 40,000 comments. Nicholas R. Parrillo, Should the Public

Get to Participate Before Federal Agencies Issue Guidance? An Empirical Study, 71 ADMIN. L. REV. 57, 97-98 (2019)

[hereinafter Parrillo, Should the Public Get to Participate]; Nicholas R. Parrillo, Federal Agency Guidance: An

Institutional Perspective (Oct. 12, 2017) (report to the Admin. Conf. of the U.S.).

The APA sets forth the key elements of notice-and-comment rulemaking. Subject to certain

exceptions, agencies first must publish a general notice of the proposed rulemaking in the Federal

That notice “shall include—

(1) a statement of the time, place, and nature of public rule making proceedings;

(2) reference to the legal authority under which the rule is proposed; and

(3) either the terms or substance of the proposed rule or a description of the subjects and

issues involved.”

The APA further provides that “[a]fter notice required by this section, the agency shall give

interested persons an opportunity to participate in the rule making through submission of written

data, views, or arguments with or without opportunity for oral presentation. After consideration of

the relevant matter presented, the agency shall incorporate in the rules adopted a concise general

statement of their basis and purpose.”

“Person” is broadly defined to include “an individual,

partnership, corporation, association, or public or private organization other than an agency.”

Consistent with the broad ideals underlying the commenting process, courts read this provision

expansively.

Courts have also elaborated that the purpose of the notice requirement is to facilitate

meaningful comments.

For example, agencies must disclose in their notice any scientific or

technical details on which they base their proposed rules in order to give the public a fair

opportunity to react and comment thereon.

Agencies not only must provide notice and an opportunity for public comments but also

must then “consider[] . . . the relevant matter presented” in those comments.

The courts have

interpreted this language to require that, in the notice of the final rule, agencies respond to

“significant” comments—those that, “if true, . . . would require a change in [the] proposed rule.”

Failure to so respond is grounds for remand.

Under § 706(2)(A) of the APA, agency rulemakings

will be set aside if they are “arbitrary, capricious, an abuse of discretion, or otherwise not in

accordance with law.”

The failure to acknowledge and respond to substantive concerns raised in

5 U.S.C. § 553(b).

Id.

5 U.S.C. § 553(c).

5 U.S.C. § 551(2).

E.g., O’Rourke v. U.S. Dep’t of Justice, 684 F. Supp. 716, 718 (D.D.C. 1988) (holding “person” includes non-

citizens and collecting cases); Neal-Cooper Grain Co. v. Kissinger, 385 F. Supp. 769, 776 (D.D.C. 1974) (foreign

government or instrumentality thereof is a “person”).

Portland Cement Ass’n v. Ruckelshaus, 486 F. 2d 375, 393 (D.C. Cir. 1976).

Id.

5 U.S.C. § 553(c).

Am. Mining Cong. v. EPA, 907 F.2d 1179, 1188 (D. C. Cir. 1990); Home Box Office, Inc. v. FCC, 567 F.2d 9, 35-

36 (D.C. Cir. 1977) (per curiam). Without such an obligations, courts have said, the opportunity to comment would

be “meaningless.” Id; see Carlson v. Postal Regulatory Comm’n, 938 F.3d 337, 351 (D.C. Cir. 2019); ACLU v. FCC,

823 F.3d 1554, 1581 (D.C. Cir. 1987); St. James Hosp. v. Heckler, 760 F.2d 1460, 1470 (7th Cir. 1985).

E.g., La. Fed. Land Bank Ass’n v. Farm Credit Admin., 336 F.3d 1975 (D.C. Cir. 2003).

5 U.S.C. § 706(2)(A).

the rulemaking process is one of the grounds for a court to find that an agency’s rulemaking fails

under the arbitrary or capricious standard.

Although the requirement to respond to comments is serious, it is not absolute. The “APA

requirement of agency responsiveness to comments is subject to the common-sense rule that a

response [is not always] necessary.”

Comments that “are purely speculative and do not disclose

the factual or policy basis on which they rest require no response.”

In recognition of the potential for information and communication technologies to facilitate

broader participation in the regulatory process, Congress enacted the E-Government Act in

October 2002.

Among other things, the George W. Bush administration established the e-

Rulemaking Program to spearhead the creation of an online system for conducting the notice-and-

comment process at agencies throughout the federal government.

To the extent deemed

practicable, each agency must post information required to be published in the Federal Register

online, maintain online rulemaking dockets, and allow for electronic submission of comments

accepted under § 553(c). To better facilitate this public online access, regulations.gov was created

in January 2003.

All executive agencies were required to join the e-Rulemaking initiative.

of this writing, many but not all independent agencies also use regulations.gov for their

rulemakings; those that do not prominently include the FCC and the SEC.

Filing comments, electronically or by mail, is not the only way that individuals and

organizations may participate in agency rulemaking. Agencies also occasionally hold public

hearings, consult with experts in advisory committees, and work with interest group stakeholders

in negotiated rulemakings. The Negotiated Rulemaking Act encouraged agencies to use a dispute

resolution process for soliciting stakeholder comment to enhance the informal rulemaking

process.

Public participation also occurs after-the-fact through intervention in agency

adjudications, citizens and groups informal monitoring activities, and litigation. But notice-and-

comment rulemaking remains a central mechanism for public participation in agency policy

making.

See, e.g., Bus. Roundtable v. SEC, 647 F.3d 1144, 1152 (D.C. Cir. 2011) (discussing agency failure to address cost

issues raised in comments).

NRDC v. EPA, 859 F.2d 156, 188–89 (D.C. Cir. 1988).

Home Box Office, Inc. v. FCC, 567 F.2d 9, 35, n.58 (D.C. Cir. 1977) (per curiam); see also Pub. Citizen, Inc. v.

FAA, 988 F.2d 186, 197 (D.C. Cir. 1993).

See E-Government Act § 206, 44 U.S.C. § 3501 note.

Regulations.gov was initially managed by the Environmental Protection Agency. The General Services

Administration (GSA) assumed the role of managing partner of the e-Rulemaking Program at the beginning of October

2019; see About the eRulemaking Initiative, REGULATIONS.GOV, https://www.regulations.gov/about (last visited Mar.

31, 2021).

See Cynthia R. Farina, Recent Development: Achieving the Potential: The Future of Federal E-Rulemaking, Report

of the Committee on the Status and Future, 62 ADMIN. L. REV. 279, 282 (2009).

A complete list of participating and nonparticipating agencies appears at https://www.regulations.gov/agencies.

Negotiated Rulemaking Act, 5 U.S.C. § 561 et seq. (1994) (the purpose of the Act is to encourage agencies to use

this innovative dispute resolution process for soliciting stakeholder comment to enhance the informal rulemaking

process). See also Admin. Conf. of the U.S., Recommendation 2017-2, Negotiated Rulemaking and Other Options for

Public Engagement, 82 Fed. Reg. 31039 (June 16, 2017).

The adoption of e-rulemaking by agencies, along with broader associated technological

developments, has led to fundamental changes in notice-and-comment rulemaking. What was once

a paper process that was difficult to access and generally dominated by a small number of repeat

players has become more visible and therefore more accessible. The elimination of barriers to

participation has brought with it meaningful change. Much of this is for the better. The move online

has increased participation, made for better-informed agencies, made rulemaking more

transparent, and provided commenters, stakeholders, and rulewriters within the agencies with

easier access to materials in the docket. It has also enabled certain forms of commenting that lack

obvious benefits and that may do affirmative harm. The remainder of this report examines these

opportunities and challenges.

II. The New World of Technology-Enabled Comments

For much of its history, the APA’s notice-and-comment process involved a particular form of

commenting: an individual or entity with “data, views, or arguments” relevant to the draft rule

produced a bespoke comment that reflected that individual’s or entity’s expertise or concerns. One

person, one comment, and every comment made a unique contribution because it was from a

unique submitter.

The three sorts of technology-enabled comments addressed in this report do not fit this model.

Many commentators believe that this departure from the past is a problem. One example: the U.S.

Senate Permanent Subcommittee on Investigations’ 2019 report pointed to a variety of issues

associated with mass, malattributed, and computer-generated comments (in addition to

highlighting other problems associated with comments such as the inclusion of obscenity and

copyrighted materials).

It saw these types of comments as contributing to “abuses” that “reduce[]

the effectiveness of the notice-and-comment process; cost[] taxpayer funds to mitigate; allow[]

identity theft-related crimes to go unaddressed; and leave[] the rulemaking process vulnerable to

disruptive activity.”

This report offers, among other things, a response to some of the concerns

raised by the Staff Report and many other observers. The widespread outrage caused by FCC’s net

neutrality rulemaking—with its massive total number of comments as well as fake or

“malattributed” (a comment filed under the name of someone who did not in fact prepare, approve,

or submit it) and computer-generated comments—suggests that technology-enabled comments

may be a major problem but, as we shall explore, the challenges are surmountable.

We begin this section with a taxonomy, identifying the particular sort of commenting

activity that is our focus. We then turn to a discussion of the overall issues that these sorts of

comments raise. In the following sections, we consider each of the three types of comments in

particular.

Subcommittee Report, supra note 10.

Id. at 1.

A. Three Types of Technology-Enabled Comments

1. Mass Comments

In a relatively small number of rulemakings, agencies receive an unusually large number

of comments (e.g., hundreds or thousands, as opposed to the few dozen, or fewer, that are

typical).

We designate such situations as a “mass comment response.” We do not define a specific

threshold for the number of comments to qualify as a mass comment response, as the threshold

will tend to vary from agency to agency and rule to rule. As a general matter, a “mass comment

response” will feature at least an order of magnitude increase (e.g., 10x) in the number of

comments received vis-à-vis a typical rulemaking for that agency.

We use the term “mass

comment campaign” to refer to the special case of a mass comment response in which one or more

organizations has successfully urged a large number of individuals or groups to submit comments

to the agency or allow those organization(s) to submit comments in their names.

Mass comment

responses and mass comment campaigns have grown in frequency and scope as information and

communication technologies including email and the internet have reduced the cost of

participating in the notice-and-comment process.

2. Malattributed Comments

Much of the outraged reaction to the net neutrality rulemaking focused not on the sheer

number or duplicativeness of the comments but on the fact that millions were submitted under

false names, purporting to be from someone who either did not exist or had no awareness of the

comment. We term comments falsely attributed to persons by whom they were not in fact

submitted “malattributed comments.”

Malattributed comments are similarly facilitated by new technologies. Easy access to very

large data sets of personal information makes the task of malattributing comments much easier

than in the past. In addition, it is possible to automate the malattribution of comments, using simple

software applications coupled with publicly available information such as DMV listings or voter

registration data.

See supra note 18; Hearing of the Permanent Subcommittee on Investigations and the Subcommittee on Regulatory

Affairs and Federal Management, S. Homeland Sec. & Governmental Affs. Comm., October 24, 2019 (statement of

Dominic J. Mancini, Deputy Administrator, Off. of Info. & Regul. Affs.) (noting that internal analysis “suggested that

about 80 percent of proposed rules receive 10 or fewer comments”).

Admittedly, this definition is somewhat arbitrary. There is no accepted definition of a “mass comment response,”

and we adopt this working definition for purposes of this report based on our interviews with agency officials.

See Balla et al., supra note 1.

The more common term is “fraudulent comment.” For a more complete discussion of possible labels, including

“fraudulent,” “fake,” “pseudonymous,” “fabricated,” “inauthentic,” and “misattributed,” see Michael Herz,

Fraudulent Malattributed Comments in Agency Rulemaking, 42 CARDOZO L. REV. 1, 10-12 (2020). A false name can

be seen as just one instance of false statements in comments generally. We address that larger problem briefly below,

but our focus is on the false identity issue alone. The question of how agencies should respond to false information

included in the body of a comment is also important, but it is only tangentially related to challenges created by new

technologies, which are the focus of this report. The malattribution problem, by contrast, has been greatly accentuated

by the development of new technologies.

3. Computer-Generated Comments

The notice-and-comment process invites interested persons to submit their views on

proposed rulemakings. A tacit assumption of this invitation is that the text contained within a

public comment will be written by a human commenter. This assumption can be violated when a

software program is used to generate the text. We define “computer-generated comments” as those

that are generated by a software algorithm, thus replacing both human content generation and

human interaction with the agency. Although a human must create the software to accomplish this

task, once automated, that individual need not further engage in the commenting process and,

instead, the software can submit comments via regulations.gov, possibly even repeatedly.

Computer-generated comments are enabled by advances in automated text creation. To date,

computer-generated comments have been fairly crude cut-and-pastes that are easy to detect.

However, researchers in the field of natural language processing (NLP) continue to make striking

progress. Although the first computer program attempting to mimic human conversation was

introduced in the mid-1960s,

in recent years more sophisticated software is making it possible to

produce comments that seem to be unique and written by humans, when they are actually produced

by machines. Contemporary algorithms have achieved results that are difficult to distinguish from

human writers.

If similar (or more advanced) systems were used to generate public comments,

they could overwhelm an agency with an arbitrarily large number of human-quality comments.

The creators of a recently developed artificial intelligence technique for generating human-like

text known as GPT-3 recognized this risk, including “abuse of legal and governmental processes”

among the potential misuses of text generation tools.

A recent experiment involving computer-

generated comments submitted in response to a proposed Medicaid waiver, which we describe

below, gives a glimpse of the potential of this technology to produce realistically human comments

in the rulemaking process.

Such advances need not necessarily be problematic. One can envision computer-generated

comments that add value to the rulemaking process. For example, a program could review an

agency’s published proposed rule for spelling errors, broken links, and other simple errors and

then file a comment in the relevant docket to provide a report of these errors. In this example the

program is not providing input on policy views; it is merely providing technical assistance.

In addition, there may be more sophisticated ways to bring the benefits of big data and artificial

intelligence to the comment process. In the same way that credit card companies can offer fraud alerts

to individual cardholders when purchasing activity is unusual, perhaps there are ways for people to

Many malattributed comments are also computer-generated comments, but neither is a full subset of the other. Some

concerns—for example, that they make it look as if more people are submitting comments or taking a particular

position than is in fact the case—may apply to both. Our discussion of malattributed comments, however, focuses

specifically on the malattribution, not the fact that it is often done by a computer. And our discussion of computer-

generated comments focuses on the source of the comment, not the fact that it may well attach a false identity to the

submission.

Joseph Weizenbaum, ELIA—A Computer Program for the Study of Natural Language Communication between

Man and Machine, 9 COMMUNC’NS OF THE ACM 37 (1966).

See e.g., TOM B. BROWN, ET AL., LANGUAGE MODELS ARE FEW SHOT LEARNERS 5 (July 2020),

https://arxiv.org/pdf/2005.14165.pdf (discussing the GPT-3 autoregressive language model).

Id. at 35.

leverage technology to assist them in producing draft comments based on the interests they have

expressed through their purchasing habits or other administrative data. There are plenty of operational,

privacy, and other issues to consider, but we raise it here as a possibility that may caution against overly

restrictive approaches to computer-generated comments. To the extent that these types of tools could

help overcome the collective action problems that inhibit public participation in the rulemaking

process, they could offer a useful path forward.

B. Challenges Posed by Technology-Enabled Comments

All three types of comments we discuss in this report affect the rulemaking process. We

explore in more detail below the specific challenges posed by mass, malattributed, and computer-

generated comments. But, first, it is important to note that these technology-enabled comments are

not an entirely new phenomenon.

With respect to mass comments, there were high-salience rulemakings that generated

substantial numbers of public comments prior to the advent of online commenting. In the 1995-96

rulemaking in which it first asserted regulatory authority over tobacco cigarettes, the Food and

Drug Administration received over 700,000 paper comments, many of which were identical, so-

called “postcard comments.”

Yet orchestrating a campaign to mail in masses of comments was much more burdensome

and expensive than it is now. Once it became possible to comment with the click of a button, mass

participation, both spontaneous or orchestrated, became much more straightforward as did the

potential for duplicative comments.

Similarly, it has always been the case that a commenter could sign a phony name to a

comment or include other falsehoods in the comment. But the digital availability of personal

information, automation tools, and online submission makes it much easier to submit such

comments at scale. And the difficulty of identifying misleading or mislabeled comments is

heightened in a deluge of comments.

Even computer-generated comments could, in theory, have been submitted on paper.

Automated text-generation software has existed since the 1960s, and it would be a trivial task to

print out computer-generated text and place it in the mail. Again, however, the shift to electronic

submissions—alongside tremendous advances in artificial intelligence and NLP in recent years—

facilitates and reduces the costs of submitting comments written by computers.

All three of these types of comments can generate serious challenges to agencies, raising a

pressing set of questions concerning how best to respond while preserving a functional rulemaking

Food and Drug Admin., Regulations Restricting the Sale and Distribution of Cigarettes and Smokeless Tobacco to

Protect Children and Adolescents, 61 Fed. Reg. 44396, 44418 (1996). As the FDA described it:

Altogether, the agency received more than 700,000 pieces of mail, representing the views of nearly

1 million individuals. Most of the submissions were form letters or post cards. The agency identified

more than 500 different types of form letters. Others were petitions with sometimes hundreds of

signatures. More than 95,000 submissions expressed individual comments on the 1995 proposed

rule, including more than 35,000 from children who were overwhelmingly supportive.

Id.

Beth Simone Noveck, The Electronic Revolution in Rulemaking, 53 EMORY L.J. 433 (2004) (expressing concern

that new technology might open the floodgates to “notice-and-spam” rulemaking).

process. We now turn to exploring what is at stake with mass, malattributed, and computer-

generated comments.

1. Information

Agency rulemakings often touch on important areas of social and economic life, and can

have complex and difficult-to-anticipate effects. Agencies bring considerable internal expertise to

the task of crafting rules, and officials can also often draw on published research. However, there

is often useful information that an agency might not have readily at hand during its deliberations,

and an important part of the value of notice-and-comment rulemaking is the public’s opportunity

to bring such information to the agency’s attention.

Often, the most useful information for an agency will be technical or operational. This type

of information includes scientific or engineering studies, relevant data, analysis about how well

the proposed regulatory change will address the problem being solved, what kind of changes

compliance will require, or legal or policy analysis. Technical or operational information

empowers agency decision makers to anticipate the consequences of the choices they face in the

design of regulations. Information along these lines facilitates higher quality rulemakings, where

quality is understood in terms of technical or operational proficiency.

A second type of information concerns conclusions drawn by stakeholders or members of

the public concerning the desirability of a rulemaking. Under the APA, agencies are bound to

consider relevant substantive arguments offered by commenters in support of their conclusions.

But the status of the ultimate evaluation offered by a commenter is less clear. There is a debate

among administrative law scholars concerning the extent to which agencies should consider

commenter preferences. Cynthia Farina, for example, has argued that the deliberative and technical

nature of the rulemaking process makes consideration of pure expressions of preference

inappropriate, especially if they are not informed or representative.

Nina Mendelson, by contrast,

has argued that expressions of preference can and should be considered by agencies, at least in

some contexts, because agencies are often called on to “decide values and policy questions left

unresolved by their authorizing statutes.”

For Mendelson, concerns about representativeness, for

Cynthia R. Farina et al., Rulemaking vs. Democracy: Judging and Nudging Public Participation That Counts, 2

MICH. J. ENVTL. & ADMIN. L. 123, 137–39 (2012).

Nina A. Mendelson, Rulemaking, Democracy, and Torrents of E-Mail, 79 GEO. WASH. L. REV. 1343, 1350–51

(2011) [hereinafter Mendelson, Rulemaking, Democracy]; see also Nina A. Mendelson, Should Mass Comments

Count?, 2 MICH. J. ENVTL. & ADMIN. L. 173, 1818 (2012) (“[A]gency officials might pay attention to large volumes

of comments, for example, to help gauge public resistance or anticipate significant opposing views.”). This debate

tracks, in certain respects, the scholarly conversation on whether courts should accept “political reasons” as

justifications for agency decisions. Jodi L. Short, The Political Turn in American Administrative Law: Power,

Rationality, and Reasons, 61 DUKE L.J. 1811 (2012); Nina A. Mendelson, Disclosing “Political” Oversight of Agency

Decision Making, 108 MICH. L. REV. 1127 (2010); Kathryn A. Watts, Proposing a Place for Politics in Arbitrary and

Capricious Review, 119 YALE L.J. 2 (2009). More broadly, the debate about the appropriate role for consideration of

expressions of preference in public comments tracks alternative views about the legitimate foundations of

administrative decision making. See generally Vanessa Duguay, Views or Votes: The Challenge of Mass Comments

in Rulemaking, 26 GEO. MASON L. REV. 625 (2018); Steven P. Croley, Public Interest Regulation, 28 FLA. ST. U. L.

REV. 7, 7 (2000); Mark Seidenfeld, A Civic Republican Justification for the Bureaucratic State, 105 HARV. L. REV.

1511 (1992).

example, must be balanced against the drawbacks of ignoring the sentiments of those who have

taken the time to comment.

Without settling this debate, it is worth noting that there are often substantive limits on the

kinds of information that agencies may consider in the course of rulemaking.

Under the State

Farm formulation, “[n]ormally, an agency rule would be arbitrary and capricious if the agency has

relied on factors which Congress has not intended it to consider.”

There are many examples

where courts have found limits on agencies’ ability to consider certain factors. For example, in

Whitman v. American Trucking, the Court found that the EPA may not consider costs when setting

the National Ambient Quality Standards under the Clean Air Act.

Accordingly, public

preferences concerning whether the benefits of more stringent air quality standards outweigh the

costs are not relevant under American Trucking.

Mass, malattributed, and computer-generated comments can make it difficult to extract

both technical/operational and preference information from the notice-and-comment process, but

the potential challenges for preference information are greater. With respect to

technical/operational information, the identity of the commenter (even whether the commenter is

a human being) and how frequently that information appears in the record will often not be

relevant. The primary challenge raised by mass, malattributed, and computer-generated comments

is that useful technical/operational information may be difficult to find within a large flood of

comments. Preference information, by contrast, would only be relevant inasmuch as it relates to

the views of a genuine person, making it necessary to separate out bot and malattributed comments

from those that are genuinely submitted by a person. Further, for rules that result in a mass

comment response, agencies face a range of difficult questions (discussed in more detail below)

concerning the representativeness of the pool of commenters and the role of intermediary groups

that conduct mass comment campaigns.

Finally, information is valuable only if it is accurate. When a comment contains false or

erroneous statements, intentional or otherwise, it is at least a delay and distraction. If the falsehood

is important and undiscovered, it could negatively affect the substance of the final rule. From the

outset, one concern about e-rulemaking has been that it would lead to agencies being deluged with

misinformation.

The agency personnel we interviewed reported that misinformation in

comments has not been a major problem to date. Presumably this is for at least two reasons. First,

agencies are repositories of significant expertise. That means they will often recognize substantive

errors in comments or at least know enough to realize that further investigation is required. Second,

broad participation is a prophylactic against misinformation; the false submission might be

countered by a true one, often multiple true ones. This is not to dismiss all concerns about false

See Stuart M. Benjamin, Evaluating E-Rulemaking: Public Participation and Political Institutions, 55 DUKE L. J.

893, 906–907 (2006) (“Many statutes leave no room for an agency to consider public sentiment.”).

Motor Vehicle Mfrs. Assn. of United States v. State Farm Mut. Automobile Ins. Co., 463 U.S. 29, 43 (1983)

(emphasis added).

Whitman v. Am. Trucking Ass’ns, 531 U.S. 457 (2001).

See, e.g., Daniel C. Esty, Environmental Protection in the Information Age, 79 N.Y.U. L. REV. 115, 172 (2004)

(“The promise of cyberdemocracy with a fully informed and engaged populace could give way to ‘spam,’

misinformation, and dialogue among the uninformed that diminishes thoughtful deliberation.”).

submissions, which may be seen as a growing concern, but it is a reminder that a falsehood in a

comment is only a small first step toward a substantive error in a final rule.

2. Legitimacy

The concept of legitimacy is complex and its full exegesis is beyond the scope of this

report. We focus on the potential effects of mass, malattributed, and computer-generated

comments on positive (or sociological) legitimacy, which concerns empirical questions related to

public acceptance of the exercise of government power,

as distinct from normative or moral

legitimacy.

There is a considerable body of behavioral and social scientific research on the causes of

positive legitimacy.

One important thread of that literature concerns perceptions of “procedural

fairness” and the components of decision-making processes that tend to enhance or undermine

those perceptions. In a recent OECD report by E. Allan Lind and Christiane Arndt summarizing

some of this literature, the authors identify “[t]hree general elements of process . . . [that] stand out

in terms of their impact on whether a citizen will feel fairly treated in his or her interactions with

government [. . .]: voice, polite and respectful treatment, and explanations.”

The authors define

voice as “a chance [for affected people] to present their views” along with “some indication that

the input was actually given consideration.”

According to this review of the literature, voice

“remains the most extensively researched and arguably the most powerful antecedent of perceived

procedural fairness.”

The relationship between voice and the notice-and-comment process is obvious—the APA

requirement that agencies solicit and consider the views of interested persons maps exactly onto

the definition of voice offered by Lind and Arndt. Based on current research in the field, there is

reason to believe that the notice-and-comment process enhances the positive legitimacy of agency

rulemaking, particularly when compared to an imagined counterfactual in which there is no

Questions concerning the sociological legitimacy of the state reach back to the foundations of contemporary social

sciences. See Max Weber, Die drei reinen Typen der legitimen Herrschaf, 187 PREUSSISCHE JAHRBÜCHER 1 (1922)

(appearing later as Max Weber, The Three Types of Legitimate Rule, 4 BERKELEY PUBLICATIONS IN SOC’Y &

INSTITUTIONS 1 (1958) (Han Gerth trans.)). These questions arise for all governmental bodies, but are particularly

pressing for those with more tenuous relationships with the electoral process. Cf. Jeffery J. Mondak, Policy Legitimacy

and the Supreme Court: The Sources and Contexts of Legitimation, 47 POL. RES. Q. 675, 690 (1994); James L. Gibson,

Understandings of Justice: Institutional Legitimacy, Procedural Justice, and Political Tolerance, 23 LAW & SOC.

REV. 469, 471 (1989).

Cf. Richard H. Fallon, Legitimacy and the Constitution, 118 HARV. L. REV. 1787, 1790 (2005); Paul Weithman,

Legitimacy and the Project of Rawls’s Political Liberalism, in RAWLS’S POLITICAL LIBERALISM 73 (T. Brooks and

M. Nussbaum eds. 2012).

See, e.g., James L. Gibson & Michael J. Nelson, The Legitimacy of the US Supreme Court: Conventional Wisdoms

and Recent Challenges Thereto, 10 ANN. REV. L. SOC. SCI. 201 (2014).

E. Allan Lind & Christiane Arndt, Perceived Fairness and Regulatory Policy: A Behavioural Science Perspective

on Government-Citizen Interactions 20 (OECD Regulatory Policy Working Paper No. 6, 2016).

Id.

consistent opportunity for the public to comment or such comments are not considered by agency

decision makers.

Malattributed and computer-generated comments may undermine the confidence of

members of the public in their ability to have their voices heard. Observers reasonably worry that

computer-generated comments submitted at scale could drown out comments from real persons,

create confusion on relevant issues, or prompt an agency to ignore even legitimate comments,

and malattributed comments could be perceived as hijacking or expropriating a person’s voice.

These issues will be discussed in more detail below.

Mass comments present other questions concerning their interaction with voice and

perceptions of procedural fairness. Lower costs of submitting comments and broader public

participation creates more widespread opportunities for voice. But if comments contain

information that agencies may not or do not consider—including expressions of preference—it is

not clear that the process will ultimately enhance perceptions of procedural fairness. As noted by

Lind and Arndt “research on voice makes it clear that it is not enough just to allow for more raw

input or comment: There must also be some indication that the input was actually given

consideration.”

A mismatch between commenter expectations and agency treatment of

comments raises serious concerns, which are discussed in more detail below.

It is worth emphasizing that for purposes of positive legitimacy, perceptions of procedural

fairness matter, irrespective of how well those perceptions map onto reality. For example, even if

agencies are able to easily sort through bot or malattributed comments, these phenomena could

lead to misimpressions about the integrity of the system that undermine public confidence in the

process.

Experience with the FCC’s net neutrality rulemaking demonstrates that the public

comment process can become publicly salient without warning, with the associated risks of

sensational commentary and the potential for people to draw inferences about the entire process

based on an exceptional example.

See, e.g., Fake It Till They Make It, supra note 2, at 11 (statement of Paulina Gonzalez-Brito, Executive Director,

California Reinvestment Coalition) (stressing the need “to ensure that community voices are not drowned out by

fabricated comments fraudulently submitted in favor of industry”); Bob Barr, Massive Fraud in Net Neutrality Process

Is a Crime Deserving of Justice Department Attention, MARIETTA DAILY J. (Dec. 25, 2017),

https://www.mdjonline.com/opinion/bob-barr-massive-fraud-in-net-neutrality-process-is-a-crime-deserving-of-

justice-department/article_87a01d86-e9c5-11e7-af34-3bc55501c7a0.html (“[B]efore too long, the voices of real

people, expressing genuine opinions on regulations, will be drowned out and ignored all together by those in power.”);

Eric T. Schneiderman, An Open Letter to the FCC, MEDIUM (Nov. 21, 2017),

https://medium.com/@NewYorkStateAG/an-open-letter-to-the-fcc-b867a763850a (objecting that “the perpetrator or

perpetrators attacked what is supposed to be an open public process by attempting to drown out and negate the views

of the real people, businesses, and others who honestly commented on this important issue”).

Lind & Arndt, supra note 61, at 20.

“The last thing we need is a common view that essentially the entire rulemaking process is being gamed by a variety

of machines and shadowy players.” Nicole Ogrysko, GSA Launches Public Campaign to Battle Bots, Fake Comments

from Online Rulemaking Forums, FED. NEWS NETWORK (Jan. 31, 2020) (quoting Michael Fitzpatrick, head of global

regulatory affairs for Google), https://federalnewsnetwork.com/management/2020/01/gsa-launches-public-campaign-

to-battle-bots-fake-comments-from-online-rulemaking-forums/.

In addition to perceptions of procedural fairness, scholars have identified an alternative

source of positive legitimacy: it flows from the outcomes of government decisions themselves.

The basic idea is that high quality, effective government decision making leads to public

acceptance. If mass, malattributed, or computer-generated comments reduce regulatory quality by,

for example, making it more difficult for agencies to extract useful information from the notice-

and-comment process, then, over time, they could erode confidence in agency decision making.

3. Processing Costs

It takes time and resources to review, analyze, and respond to comments and use the

insights to recraft the rule. When there is a small number of comments, those costs are relatively

low. As the number of comments grows, processing costs can be expected to increase. For

example, agencies sometimes hire outside contractors to help process comments, which helps

alleviate the processing burden, but adds to the overall expense. When they do this work in-house,

the review process, albeit important, consumes significant staff power. Similarly, if agencies must

spend resources to identify malattributed or computer-generated comments, that only further

increases processing costs. Time spent sorting out mass, malattributed, and computer-generated

comments also delays the process and takes time away from other productive policymaking

activities.

When the informational and legitimacy-conferring benefits of comments are high, then the

time invested may be well worth their processing costs. Nevertheless, the direct financial and delay

costs of spurious or low-quality comments are nontrivial and worth keeping in mind.

III. Mass Comments

Large volumes of public comments present both opportunities and challenges for agencies.

On the one hand, current participation in the notice-and-comment process demonstrates substantial

interest in agency rulemaking, which creates occasions for meaningful engagement between

agencies and the public. Public comments can also contain helpful information that agencies can

use to improve their rulemakings.

On the other hand, large volumes of comments are burdensome to process and digest,

increase the risk of missing important arguments or information, and may make it more difficult

to extract overall patterns in the content of comments.

As noted above, for purposes of this report, we distinguish between a mass comment

response and a mass comment campaign. The latter is a special case of a mass comment response

in which an individual or organization successfully urges a large number of individuals to file

comments that express a similar set of views or positions. Often, comments made in response to a

mass solicitation contain identical or nearly identical language. The soliciting organization may

post a sample comment and encourage the submitter to file the comment verbatim but include a

sentence or two at the end explaining how the rule personally affects the submitter. It is also

possible that soliciting organizations may encourage submissions of unique—if substantively

See, e.g., FRITZ SCHARPF, GOVERNING IN EUROPE: EFFECTIVE AND DEMOCRATIC? 11 (1999) (distinguishing “input-

oriented” and “output-oriented” forms of democratic legitimacy).

similar—comments. In such cases, it may be difficult to disentangle a mass comment campaign

from a more spontaneous mass response.

A. Legal Issues Raised by Mass Comments

Courts have had many opportunities to visit the question of how agencies must consider

information generated during the notice-and-comment process. In one early and important

formulation, the D.C. Circuit’s Home Box Office v. FCC opinion provided the following standard:

In determining what points are significant, the “arbitrary and capricious” standard

of review must be kept in mind. Thus only comments which, if true, raise points

relevant to the agency’s decision and which, if adopted, would require a change in

an agency’s proposed rule cast doubt on the reasonableness of a position taken by

the agency. Moreover, comments which themselves are purely speculative and do

not disclose the factual or policy basis on which they rest require no response. There

must be some basis for thinking a position taken in opposition to the agency is

true.

A few years previously, in the canonical Portland Cement decision, Judge Leventhal

expressed a similar sentiment, writing that “comments must be significant enough to step over a

threshold requirement of materiality before any lack of agency response or consideration becomes

of concern.”

More recently, the D.C. Circuit stated the point this way: “An agency is not obliged

to respond to every comment, only those that can be thought to challenge a fundamental

premise.”

Courts have repeatedly noted that agencies are required to consider the substance of

comments: “An agency need not respond to every comment, but it must ‘respond in a reasoned

manner to the comments received, to explain how the agency resolved any significant problems

raised by the comments, and to show how that resolution led the agency to the ultimate rule.’”

There is no obligation to respond to comments per se.

Rather, “[t]he failure to respond to

Home Box Office, Inc. v. FCC, 567 F.2d 9, 35 n.58 (D.C. Cir. 1977); see Pub. Citizen, Inc. v. FAA, 988 F.2d 186,

197 (D.C. Cir. 1993) (citing HBO for standard); Nat’l Shooting Sports Found., Inc. v. Jones, 716 F.3d 200, 215 (D.C.

Cir. 2013) (same).

Portland Cement Ass’n v. Ruckelshaus, 486 F.2d 375, 394 (D.C. Cir. 1973). This language continues to be cited by

courts to express the relevant standard. See, e.g., Am. Great Lakes Ports Ass’n v. Zukunft, 296 F. Supp. 3d 27, 53

(D.D.C. 2017).

MCI WorldCom, Inc. v. FCC, 209 F.3d 760, 765 (D.C. Cir. 2000); see also Am. Mining Cong. v. U.S. EPA, 907

F.2d 1179, 1187-88 (D.C. Cir. 1990) (“[I]n assessing the reasoned quality of the agency’s decisions, we are mindful

that the notice-and-comment provision of the APA . . . ‘has never been interpreted to require [an] agency to respond

to every comment, or to analyse [sic] every issue or alternative raised by comments, no matter how insubstantial.”)

(quoting Thompson v. Clark, 741 F.2d 401, 408 (D.C. Cir. 1984)).

Action on Smoking & Health v. CAB, 699 F.2d 1209, 1216 (D.C. Cir. 1983) (quoting Rodway v. U.S. Dep’t of

Agric., 514 F.2d 809, 817 (D.C. Cir.1975)); see South Carolina ex rel. Tindal v. Block, 717 F.2d 874, 885 (4th Cir.

1983) (“The purpose of allowing comments is to permit an exchange of views, information, and criticism between

interested persons and the agency.”).

Sherley v. Sebelius, 689 F.3d 776, 784 (D.C. Cir. 2012) (“[A]n agency’s failure to address a particular comment or

category of comments is not an APA violation per se.”); United States v. Nova Scotia Food Prods. Corp., 568 F.2d

comments is significant only insofar as it demonstrates that the agency’s decision was not ‘based

on a consideration of the relevant factors.’”

A corollary to this focus on the substance of comments has been a tendency to deemphasize

the importance of the number of comments received. The D.C. Circuit has stated directly that

agencies are under “no obligation to take the approach advocated by the largest number of

commenters.”

There is a broad consensus that the public comment process is “not a vote.”

Courts have sometimes explicitly considered the number of comments received by an agency,

but these are outside the rulemaking context and not to evaluate the adequacy of the agency’s

explanation or the support in the record for its conclusion..

Although courts have emphasized the importance of substance over volume in evaluating

agency responses to public comments, there is no general bar against agencies relying on

information contained in form comments. For example, the plaintiffs in Resident Councils v.

Leavitt argued that a regulation was invalid because the vast majority of the supportive comments

on the proposed rule were form letters and the agency’s reliance on them was therefore

unwarranted. The court disagreed, stating that “there is no reason the Secretary was not entitled to

rely on such letters in promulgating the regulations.”

The court followed up by stating that just

240, 252 (2d Cir. 1977) (“‘We do not expect the agency to discuss every item of fact or opinion included in the

submissions made to it in informal rulemaking.’”) (quoting Auto. Parts & Accessories Ass’n v. Boyd, 407 F.2d 330,

338 (D.C. Cir. 1968)).

Thompson, 741 F.2d at 409 (quoting Citizens to Pres. Overton Park v. Volpe, 401 U.S. 402, 416 (1971)); see also

Covad Commc’ns Co. v. FCC, 450 F.3d 528, 550 (D.C. Cir. 2006).

U.S. Cellular Corp. v. FCC, 254 F.3d 78, 87 (D.C. Cir. 2001). See also Nat. Res. Def. Council, Inc. v. EPA, 822

F.2d 104, 122 n.17 (D.C. Cir. 1987) (noting that rulemaking is not a process where “the majority of commenters

prevail by sheer weight of numbers”).

Bridget C.E. Dooling, Legal Issues in E-Rulemaking, 63 ADMIN. L. REV. 893, 901 n.33 (2011); Michael Herz,

“Data, Views, or Arguments”: A Rumination, 22 WM. & MARY BILL RTS. J. 351, 369-74 (2013); Tips for Submitting

Effective Comments, REGULATIONS.GOV (“The comment process is not a vote.”) (on file with authors)

For example, in North Carolina Growers’ Ass’n v. United Farm Workers, the court found that a truncated comment

period with substantial content restrictions was inadequate, in part relying on the small number of comments received

compared to a prior related rulemaking. N.C. Growers’ Ass’n v. United Farm Workers, 702 F.3d 755, 770-71 (4th

Cir. 2012); see also California ex rel. Becerra v. U.S. Dep’t of the Interior, 381 F. Supp. 3d 1153, 1176-1178 (N.D.

Cal. 2019). Some courts have also looked to the number of comments received during the National Environmental

Policy Act review process to determine whether a project is “controversial” and therefore requires a full environmental

impact statement, although there is disagreement over the relevance of the scale of public reaction to that inquiry.

Sierra Club v. Bosworth, 510 F.3d 1016, 1032 (9th Cir. 2007) (“Given the large number of comments, close to 39,000,

and the strong criticism from several affected Western state agencies, we cannot summarily conclude that the effects

of the Fuels CE are not controversial.”); Greenpeace Action v. Franklin, 14 F.3d 1324, 1333-34 (9th Cir. 1992) (noting

that an “outpouring of public protest” along with a “substantial dispute . . . as to size, nature, or effect” of a proposed

action can demonstrate that the action is controversial and therefore requires an EIS (citations omitted)); Emily M.

Slaten, Note, “We Don’t Fish in Their Oil Wells, and They Shouldn’t Drill in Our Rivers”: Considering Public

Opposition Under NEPA and the Highly Controversial Regulatory Factor, 43 IND. L. REV. 1319 (2010).

Resident Councils of Wash. v. Leavitt, 500 F.3d 1025, 1030 n.5 (9th Cir. 2007).

because numerous people “share the same opinion and pooled their efforts does not undermine

their intended show of support.”

B. Policy Issues Raised by Mass Comments

There are a number of policy issues raised by mass comment responses (whether or not

part of a mass comment campaign). Many researchers have found that a large percentage of the

comments received in mass comment responses are not highly substantive, but rather contain

general statements of support or opposition.

As mentioned above, there is some debate

concerning whether and to what the extent to which agencies should consider comments that

contain only statements of preference.

The Permanent Subcommittee on Investigations of the U.S. Senate’s Homeland Security

and Governmental Affairs Committee has recommended that Congress consider amending the

APA to provide guidance to agencies on the extent to which they should consider the volume of

comments in favor of or in opposition to a proposed rule.

Guidance from Congress could be

helpful to agencies in deciding when, if ever, they should take the number of comments and the

sentiment expressed in the comments into account when finalizing a rule.

Inasmuch as public opinion is relevant for a rulemaking, comments generally do not

provide a reliable metric of the views of the broader public. Commenters are an entirely self-

selected group, and there is no reason to believe that they are in any way representative of the

larger public. Relatedly, the group of commenters may represent a relatively privileged group, with

less advantaged members of the public less likely to engage in this form of political participation.

There are also questions related to the actual influence of mass comment responses on

agency decision making. There is a considerable social science literature that examines the public

comment process and how it affects regulatory outcomes.

An important early paper by Marissa

Id. Morales v. Lyng, 702 F. Supp. 161 (N.D. Ill. 1988) also discusses the role of form comments. In this case, the

Department of Agriculture was found to have acted in an arbitrary and capricious manner by choosing to generally

ignore certain comments. The Secretary of Agriculture argued that the ignored comments were “endless clones of

conclusory statements.” Id. at 163. However, the court found that by choosing to ignore these comments that offered

a differing view than that chosen by the agency, the agency had impermissibly “failed to consider important aspects

of the administrative record and hence the issue itself.” Id.

Thomas A. Bryer, Public Participation in Regulatory Decision-Making: Cases From Regulations.gov, 37 PUB.

PERFORMANCE & MGMT. REV. 263, 263 (2014) (analysis of EPA and HHS rulemakings finding that many comments

were “emotional, illogical and lacking in credibility”); Kimberly D. Krawiec, Don’t “Screw Joe the Plummer”: The

Sausage-Making of Financial Reform, 55 ARIZ. L. REV. 53, 58 (2013) (contrasting industry comments on the

Securities and Exchange Commission’s Volcker Rule, which were “meticulously drafted, argued, and researched”

with “citizen letters [which were] short and provide little evidence that citizen commenters even understand, or care,

what proprietary or fund investment is, much less the ways in which agency interpretation of the Volcker Rule’s

complex and ambiguous provisions might govern such activities”); Stuart W. Shulman, The Case Against Mass E-

mails: Perverse Incentives and Low Quality Public Participation in U.S. Federal Rulemaking, 1 POL’Y & INTERNET

23 (2009) (arguing the many comments lack substantive merit).

Subcommittee Report, supra note 10, at 3.

For an overview of this literature, see Susan Webb Yackee, The Politics of Rulemaking in the United States, 22

ANN. REV. POL. SCI. 37 (2019). An important general point is that other mechanisms for interested parties to affect

agency decision making, such as ex-parte communications during the pre-proposal stage, may be more influential than

Golden found that business interests tended to dominate the rulemaking process, but that the

overall influence of comments was low.

Subsequent work has found that, at least under certain

conditions, agencies sometimes do make changes in response to comments.

Among the factors

that have been found to affect commenter influence is the degree of sophistication in the comments

and the source of the comment.

Studies of mass commenting in particular have found that

agencies tend to be less responsive to mass comment campaigns,

and to refer to the number of

comments received in favor of or in opposition to a rule in opportunistic ways.

Some have argued

that the reality that agencies are unlikely to alter rules in response to less substantive comments

provides a reason to discourage, or at least “not actively facilitate public participation” of this

sort.

Direct influence may not be the only motivation behind comments, and advocacy groups

may solicit mass comments for many different reasons. Research on mass comment campaigns

suggests that that different groups carry out such campaigns to promote a range of goals, including

the public comment process. See Jeffrey J. Cook, Crossing the Influence Gap: Clarifying the Benefits of Earlier

Interest Group Involvement in Shaping Regulatory Policy, 42 PUB. ADMIN. Q. 466 (2018); Susan Webb Yackee, The

Politics of Ex Parte Lobbying: Pre-Proposal Agenda Building and Blocking during Agency Rulemaking, 22 J. PUB.

ADMIN. RES. & THEORY 373 (2012).

Marissa Martino Golden, Interest Groups in the Rule-Making Process: Who Participates? Whose Voices Get

Heard?, 8 J. PUB. ADMIN. RES. & THEORY 245 (1998). That study involved analysis of comments received by three

agencies (EPA, HUD, and NHTSA) to a set of eleven rulemakings. Generally, Golden finds that “business

commenters” dominated the public comment process, as “[b]etween 66.7 percent and 100 percent of the comments

received were submitted by corporations, public utilities, or trade associations.” Id. at 252–53. However, Golden did

not find a large substantive impact from the business community’s participation; she attributed this lack of influence

in part due to the fact that “business did not present a united front[;] . . . [t]here were frequently divisions within the

business community.” Id. at 262.

Amy McKay & Susan Webb Yackee, Interest Group Competition on Federal Agency Rules, 35 AM. POL. RES. 336

(2007); Susan Webb Yackee, Sweet-Talking the Fourth Branch: The Influence of Interest Group Comments on

Federal Agency Rulemaking, 16 J. PUB. ADMIN. RES. & THEORY 103 (2006).

Mariano-Florentino Cuéllar, Rethinking Regulatory Democracy, 57 ADMIN. L. REV. 411 (2005) (identifying three

instances in which agencies modified proposals in light of submissions from non-business commenters); Jason Webb

Yackee & Susan Webb Yackee, A Bias Towards Business? Assessing Interest Group Influence on the U.S.

Bureaucracy, 68 J. POL. 128, 133–35 (2006) (finding that agencies consistently alter proposals to reflect comments

from business interests but not others). One article observes: “The relatively high value placed on hard data in

comments is best summed up by the interviewee who stated, ‘We look at every comment; we consider every comment.

But unless there is data supporting the position, it’s just not that useful in the rulemaking process.’” Keith Naughton

et al., Understanding Commenter Influence During Agency Rule Development, 28 J. POL’Y ANALYSIS & MGMT. 258,

270 (2009).

Balla et al., supra note 1 (finding that agencies give mass comments limited attention in the preambles to final rules

and that “regulations are generally not consistent with changes requested in comments, a lack of association that holds

especially for mass comment campaigns.”). Some observers have identified cases where mass comments (at least

arguably) influenced regulatory outcomes. See Lauren Moxley, E-Rulemaking and Democracy, 68 ADMIN. L. REV.

661, 692–95 (2016) (attributing change in FCC’s 2015 final net neutrality rule to large number of and consensus

among commenters).

See Herz, supra note 75, at 372–73 (“When [the agencies’] conclusion has strong support in the [mass] comments

they tend to note that fact, and when it does not they tend to glide over it.”); Parrillo, Should the Public Get to

Participate, supra note 20, at 71.

Farina et al., supra note 52, at 150 (“A democratic government should not actively facilitate public participation

that it does not value.”).

calling public attention to a rulemaking.

Others have pointed to internal organizational goals,

such as increasing membership and financial contributions and moving members up the “ladder of

engagement” towards greater involvement, as a motivation for efforts to mobilize actions like

petition-signing and sending public comments.

Scholars have identified several potential problems with mass comment campaigns. Some

have argued that they may be used to distort regulators’ perception of public opinion, and may

lead to agency cynicism about the public comment process.

Mass comment campaigns often

involve many duplicate, or near duplicate comments.

Such duplicate comments impose various

real costs on agencies, without adding new substantive content to the rulemaking record.

It is

worth noting that, the expert consensus that the public comment process is “not a vote,” appears

to conflict with “widely held views among participating individuals, advocacy groups, and

journalists that the public expression of preferences should and does carry some weight, entirely

apart from whatever substantive justification for those preferences is offered.”

Cynthia Farina

Steven J. Balla et al., Where’s the Spam? Interest Groups and Mass Comment Campaigns in Agency Rulemaking,

11 POL’Y & INTERNET 460 (2019). Balla et al. find that that campaigns organized by regulated entities are more

substantive than campaigns organized by regulatory beneficiaries. Regulatory beneficiaries sponsored 73% of mass

comment campaigns analyzed (87% of campaigns with 1000+ comments), whereas regulated entities sponsored 27%

of mass comment campaigns (13% of those over 1000 comments). Campaigns sponsored by regulatory beneficiaries

were larger, averaging 15,783 comments, whereas those by regulated entities received an average of 4,345 comments.

Regulatory beneficiaries stated in interviews that during the Obama administration mass comment campaigns were

used to help the EPA justify proposed actions, whereas during the Trump-era the campaigns were used to cause the

administration to “feel pain” in the media and public opinion. Regulated entities, on the other hand, stated in interviews

that they use mass comment campaigns to try and counterbalance the mobilization by regulatory beneficiaries.

Farina, supra note 52, at 141; David Karpf, Online Political Mobilization from the Advocacy Group’s Perspective:

Looking Beyond Clicktivism, 2 POL’Y & INTERNET 1, 35 (2010) (“[Organizations use] email to mobilize member

interest around their top campaign priority, as a first step in a ladder-of-engagement.”); Shulman, supra note 79, at

27-30.

Sara R. Jordan & K. Lee Watson, Reexamining Rulemaking in an Era of Internet-Enabled Participation, 42 PUB.

PERFORMANCE & MGMT. REV. 836, 856 (2019) (“At the level of regulatory politics, manufactured salience is the

generation by politically or economically motivated actors of a large number of comments . . . in order to alter the

perceptions of regulators’ ascribed level of salience of a position on a rule.”); David Schlosberg et al., Deliberation in

E-Rulemaking? The Problem of Mass Participation, in ONLINE DELIBERATION: DESIGN, RESEARCH, AND PRACTICE

133, 143 (Todd Davies & Seeta Peña Gangadharan eds., 2009) (“Interviews with agency rule writers show that

agencies do not value and often openly resent form letters. The EPA, in fact, simply prints and stores an inaccessible

hard copy of all but one example of each identical or similar mass email.”); Stuart W. Shulman, The Internet Still

Might (But Probably Won’t) Change Everything, 1 I/S 111, 111–12 (2005) (raising concern that agency personnel

would become cynical about mass comment campaign).

Stuart Shulman, Whither Deliberation? Mass E-Mail Campaigns and U.S. Regulatory Rulemaking, 3 J. E-GOV’T

41, 58 (2006) (finding that very small percentage of mass campaign-generated comments include unique substantive

information); see also Cary Coglianese, Citizen Participation in Rulemaking: Past, Present, and Future, 55 DUKE L.J.

943, 952–59 (2006) (raising concern that e-rulemaking will increase number of comments but not range of

viewpoints).

Benjamin, supra note 54, at 904–05 (discussing costs associated with duplicative comments); see also Jeffrey S.

Lubbers, A Survey of Federal Agency Rulemakers’ Attitudes About E-Rulemaking, 62 ADMIN. L. REV. 451 (2010)

(describing a survey of officials involved in rulemaking that found a widespread view that e-rulemaking increased

total amount of participation but that there was rarely useful information or new arguments in the additional

comments).

Livermore, Eidelman & Grom, supra note 19, at 992.

argues that “powerful cultural patterns,” including “the popular equation in the United States of

democratic voice with casting a vote,” reinforce this “plebiscite assumption.”

The conflict

between public and expert perception could lead to some commenters operating under a false

understanding of the weight that will be given to their views.

The interviews provided a variety of perspectives concerning how agencies respond to

mass comment campaigns and to expressions of opinion contained in public comments more

generally. The agency officials that we interviewed were uniform in their position that the notice-

and-comment process is not a vote (i.e., agencies don’t tabulate comments “pro” and “anti” and

then choose the more popular position), but they have a wide array of approaches to addressing

opinions expressed in comments. Specifically, there seem to be three approaches taken by different

agencies: (a) opinions expressed in comments are irrelevant—only the factual content matters; (b)

opinions expressed in comments are relevant to the political perception of the rule and may affect

agencies’ activities on the Hill, color how agency leadership thinks about the viability of a

proposed rule, or affect how agencies roll out a rule if there is significant opposition; and (c)

opinions expressed in comments are relevant insofar as they express popular sentiment, and agency

decision makers (especially agency leadership) may consider that in deciding how to proceed,

though it should never be the sole factor in deciding whether or not to pursue a particular policy.

Some agencies appear not to track the overall number of comments received or the number

of times a particular comment was received. Agencies are well aware that organizations orchestrate

mass comment campaigns, and it is obvious that these campaigns will affect the comments the

agency receives. Some of the agency personnel we interviewed viewed the opinions expressed in

mass comment campaigns as mostly amounting to statements of preference.

C. Technological Responses to Mass Comments

Technologies have emerged that help agencies grapple with the large quantities of

duplicative comments that can result from mass or computer-generated comment campaigns.

Submissions in response to mass comment campaigns often include many duplicate comments,

which existing software can easily identify. The most important relevant software is known as “de-

duplication” (or “de-duping”) software. A de-duping program is software that scans each comment

and then compares it to every other comment that the agency has received.

The program will

then identify the degree of overlap between each of the comments and group those comments that

appear to derive from a common source. For instance, if the content of a comment is 90% identical

to another comment or more and the comments are more than a few words long, it is safe to assume

that the submitters either coordinated with each other when preparing their comments or that they

both used a common source document that one or both of them slightly modified (or, as we shall

discuss below, that the comment was computer-generated). The vast majority of the time, this

pattern arises when an organization has supplied text to its members and urged them to either

Cynthia R. Farina et al., Rulemaking 2.0, 65 U. MIAMI L. REV. 395, 431 (2011).

A de-duping program can, of course, only process comments that are in electronic form. With respect to paper

comment submissions, an agency may, in theory, scan the comments and then use an optical character recognition

(OCR) program to convert the file into an electronic form. The electronic version can then be run through a de-duping

program. The integrity of the OCR process depends upon the quality of the underlying physical document.

submit the comment verbatim or modify it slightly so that each comment is largely, if not entirely,

identical.

De-duping programs allow the agency to set the threshold at which a comment is flagged

as likely being part of a mass comment campaign. For instance, if the agency sets the threshold at

90% identity, any comment that has 90% or more of overlapping content will be grouped with

other such comments; any comment that is less than 90% identical will not. De-duping software

only focuses on the actual words in each document and the order in which they appear. For

example, if a submitter took a form comment and changed most of the words to synonyms (e.g.,

“happy” to “glad”), the deduping program would not recognize the comment as being duplicative.

By batching identical and nearly identical comments in this way, de-duping software

greatly reduces an agency’s burden in processing comments in rulemakings involving mass

comment campaigns. For example, consider the following hypothetical scenario. An agency issues

a proposed rulemaking regulating Issue XYZ. Organization ABC supports the rule. ABC sends an

email to its members with a request to submit a public comment in support of the rule to

Regulations.gov. The email includes a four-paragraph sample comment and also asks that the

commenters include a sentence or two that explains how Issue XYZ relates to them. 60,000

members file a comment, and about half of them add the extra sentence or two. Organization LMN

opposes the rule and also sends out an email to its members, providing them with draft text to

submit in opposition to the rule and encouraging them to explain precisely how the regulation

would harm them. 200,000 members file a comment. Of these, 100 submit LMN’s text verbatim,

80 reproduce that text and add a sentence or two on how the regulation will hurt them, and 20

reproduce the comment and provide an extensive analysis on exactly how the regulation would

cause a specific set of harms.

When the agency processes the comments received, the de-duping software will

immediately identify these two separate campaigns and batch the comments. For the ABC

campaign, it can simply ignore the 30,000 comments that are 100% identical. Rereading the same

text 30,000 times would be an extravagant waste of time and taxpayer dollars and contribute no

new information to enhance the rule-writing process. And the software can make short work of the

other 30,000 nearly identical comments, having an agency official take a quick look at the added

language in each comment and decide if it adds substantive, new information.

The process for the LMN campaign is slightly more complicated. The 100 identical

comments can be ignored, and the 80 nearly identical comments can be quickly processed,

assuming that the added sentences contain little by way of empirical data. For the 20 comments

that contain extensive additional information, the agency will need to spend more time with each

one. Indeed, they may not even be flagged by the de-duping software, depending on the extent of

the changes (e.g., if the submitter pastes in a 500-word comment and then adds 500 additional

words, the de-duping program will not flag the comment unless it is set at 50% overlap or lower).

This simple example illustrates two key points. First, de-duping software can massively

decrease the processing burden for agencies. Second, unique or partially unique comments are

much more challenging for agencies to process than identical or very nearly identical comments.

For the latter, de-duping software can reduce the marginal processing time for each iteration of a

mass comment to zero (if the comments are identical) or close to zero (if the comments are almost

identical).

Such de-duplication software has been in use for well over a decade. Stuart Shulman

created some of the earliest deduplication software for the rulemaking context as a result of

research funded by the National Science Foundation.

For example, Shulman de-duplicated the

public comment dataset from a 2013 school nutrition rulemaking in order to be able to quickly

reveal the substance of the comments and analyze them. Shulman de-duplicated the polar bear

rulemaking in 2007 (660,000 comments) and the national monuments rulemaking in 2017 (3.3

million comments).

Our interviews suggested that today most agencies use some sort of de-duplication tool,

though there is significant variation in how they do so. First, some agencies use a tool built into

FDMS, the federal toolkit for searching, viewing, downloading, and reviewing comments on

proposed federal rules. Others have their own program, others use contractors that have de-

duplication programs, and still others allow individual comment processors to use de-duplication

tools but don’t have any agency-wide prescribed tool.

While software makes it easy to spot identical or near-identical comments, agencies still

need to set the policies by which they decide how much overlap must exist between comments

before they qualify as being part of a mass comment campaign (and therefore do not review them).

De-duplication programs let an agency set the level of overlap (e.g., 90%), and different agencies

use different thresholds, though none appears to require 100% overlap for something to qualify as

being part of a mass comment campaign.

To the extent that a participant in a mass comment campaign adds unique information (e.g.,

submits the organizer’s form comment but then adds a sentence saying “I am personally supportive

of this rule because . . . .”), agencies generally review the unique information even if the identical

content is batched and treated as a group. Some agencies do not post all iterations of comments

received on Regulations.gov if they qualify as being part of a mass comments campaign. For

instance, certain agencies just post a representative example.

IV. Malattributed Comments

Malattributed comments differ from mass and computer-generated comments in two

critical ways that raise particular legal and policy issues. First, they involve a direct falsehood; the

submitter makes an assertion about its identity that is untrue. Second, they may cause harm not

just to the notice-and-comment process but to some individuals outside that process, namely, those

whose names have been used. Some have gone so far as to characterize malattributed comments

as a form of identity theft. In this section we examine these and other claims.

A. Legal Issues Raised by Malattributed Comments

Two sorts of legal issues arise with regard to malattributed comments. First, many people

might assume, and some have asserted, that submitting a malattributed comment is illegal, indeed

criminal.

Several members of Congress have requested the Department of Justice to undertake

Shulman, supra note 79.

See, e.g., In the Matters of Nicholas Confessore, 33 FCC Rcd. 11808 (2018); id. at 11821 (Rosenworcel, Comm’r,

dissenting) (noting that in the net neutrality rulemaking millions of “people had their identities stolen and used to file

criminal investigations and prosecutions (though it has not done so); at least two state Attorneys

General launched criminal investigations into the FCC net neutrality rulemaking; and the common

phrase “fraudulent comment” is itself an assertion that the submitter has violated the law.

However, some of this rhetoric may have gotten out ahead of legal realities.

Whether this activity constitutes a crime is central to the appropriate agency response for

three reasons. First, if it does, then one important federal response would be prosecution, and it

would be incumbent on the agency to refer significant examples to the Department of Justice.

Second, if the activity is criminal, then the agency has a stronger obligation to discourage or

prevent it than would be the case if it is problematic but unregulated. And third, if this is criminal

activity, that signals a societal judgment that the problem here is serious indeed.

The second set of legal issues arise under the Administrative Procedure Act. Here the

question is whether the APA requires an agency to rely on malattributed comments, forbids it to

do so, or just has nothing to say on the matter.

Our role is not to reach definitive legal conclusions, particularly as the directly relevant

caselaw is non-existent. Rather, we flag the critical questions that agencies and prosecutors must

confront.

1. Criminal Prohibitions

A number of possible criminal prohibitions might conceivably apply to malattributed

comments—fraud, making false statements, computer crime, identity theft.

The following

discussion touches on the two theories that seem most frequently mentioned.

(a) Fraud

The standard definition of fraud has five elements: (1) a false statement of a material fact,

(2) knowledge on the part of the defendant that the statement is untrue, (3) intent on the part of the

defendant to deceive the alleged victim, (4) justifiable reliance by the alleged victim on the

statement, and (5) injury to the alleged victim as a result.

fake comments, which is a crime under both federal and state laws.”); Catherine Sandoval, Reply Comments, In the

Matter of Restoring Internet Freedom, FCC 17-60, at 8 (Aug. 30, 2017) (“False filings based on stolen identities are

neither anonymous speech, nor protected speech; they constitute federal and state crimes.”).

Indeed, the Permanent Subcommittee on Investigations report lamented that “[o]nly one agency contacted by the

Subcommittee—the CFTC—said that it had referred suspicious activity to the Federal Bureau of Investigation (‘FBI’).

Other agencies, including the CFPB, the Department of Labor, and the FCC, all were aware of comments submitted

under false identities regarding their rules, but took little action to address them.” Subcommittee Report, supra note

10, at 17. In addition, it may be unlikely that the Department of Justice will undertake criminal prosecutions in this

space. See Matthew Minor, Remarks at the Administrative Conference of the United States Symposium on Mass and

Fake Comments in Agency Rulemaking, supra note 13 at 131-133.

Possibly relevant federal statutes include 18 U.S.C. § 1341 (2018) (mail fraud); 18 U.S.C. § 1343 (2018) (wire

fraud); 18 U.S.C. § 1037 (2018) (email fraud); 18 U.S.C. § 1030 (2018) (Computer Fraud and Abuse Act); 18 U.S.C.

§ 371 (2018) (conspiracy to defraud the United States); 18 U.S.C. § 1505 (2018) (obstructing agency proceedings);

18 U.S.C. § 1001 (2018) (making false statements “in any matter within the jurisdiction of the executive . . . branch”);

and 18 U.S.C. §§ 1028, 1028A (2018) (identity theft). For a full, and skeptical, discussion of the possible applicability

of these provisions, see Herz, supra note 45, at 33–55.

Malattributed comments clearly meet some elements of this definition—for example, they

involve a false statement of fact. But several elements raise real issues. First, the alleged defrauded

party is the agency. But it will be rare that an agency will “rely on” the false identity set out in a

comment. For run of the mill comments, agencies do not “rely on” the identity of the commenter;

they consider it irrelevant and so ignore it. In general, there will be nothing to rely on; reading a

comment that purports to be from “John Smith,”

100

how could the agency “rely on” it really being

from John Smith. Where the false identity is recognizable (Barack Obama, Elvis Presley, a well-

known NGO), then it is possible that the agency could conceivably rely on the putative identity of

the submitter, taking it more or less seriously in light of its source. But it will not do so because it

will recognize the falsehood. Finally, if the submitter claims to be someone with relevant personal

experience, then (a) the relevant falsehood is not the name but the content of the submission, and

(b) it would not be reasonable for the agency to meaningfully rely on such assertions without

further investigation or confirmation.

For similar reasons, it could be difficult to satisfy the fifth prong, actual injury to the

defrauded party. A loss of public confidence in the rulemaking process is arguably an injury to the

agency, but not the sort of tangible harm to person or property that fraud generally requires.

Finally, fraud requires that the false statement of fact be material. Again, if the actual identity of

the commenter does not matter, which is often the case, then a malattribution is not material.

The analysis is similar under the various federal fraud statutes, of which there are many.

Submission of a comment will almost always involve the use of a wire or of the mails, thus coming

within the ambit of the wire fraud

101

and mail fraud statutes.

102

But both these statutes require that

the perpetrator be attempting to obtain “money or property” from the defrauded party. Even if

there is an ultimate financial interest, someone attempting to influence agency policymaking is not

trying to obtain money from the agency.

103

There is one specific fraud provision that in some circumstances might reach malattributed

comments. For a century and a half, federal law has made it a crime to conspire to defraud the

United States. The current version, 18 U.S.C. § 371, makes it a crime to “conspire either to commit

any offense against the United States, or to defraud the United States, or any agency thereof in any

manner or for any purpose.” The courts have read the italicized language broadly, going beyond

common-law fraud and extending to “any conspiracy for the purpose of impairing, obstructing, or

defeating the lawful function of any department of Government.”

104

There still needs to be some

sort of trick or deceit, but using false identities could be such. If the submitter is attempting to

influence the final decision by making the agency think, for example, that a raft of non-existent

individuals support a particular outcome, the submission can be seen as an attempt to impair the

100

As 3,997 submissions in the Net Neutrality rulemaking were. Hitlin, Olmstead, & Toor, supra note 4, at 4.

101

18 U.S.C. § 1343.

102

18 U.S.C. § 1341.

103

See Cleveland v. United States, 531 U.S. 12, 26–27 (2000) (concluding “that § 1341 requires the object of the fraud

to be ‘property’ in the victim’s hands” and that a state license does not qualify). If the effort to get the government to

give the supposed fraudster a license is not an attempt to obtain money or property, an effort to get the government to

adopt a particular regulation is not a fortiori.

104

Haas v. Henkel, 216 U.S. 462, 479–80 (1910) (upholding convictions under this provision where the defendants

had submitted false information to the Department of Agriculture, thereby skewing its published statistics).

lawful functions of the agency. Even if it is unlikely to succeed in doing so, that failure is irrelevant

to the existence of a conspiracy to impair, obstruct, or defeat lawful functions. And if the agency

is influenced by false information in a submission, its functions have been impeded or impaired in

the sense that it failed to reach the “right” result. Furthermore, submission of a huge number of

comments—whether malattributed or not—with the purpose of slowing down the agency could

perhaps be understood as an effort to “obstruct.”

On the other hand, this setting is quite different than those in which prosecutions under

§ 371 are generally brought. These prosecutions—so-called “Klein conspiracies”—are most

common in tax cases; they also arise where there is a legal obligation to disclose information that

a private party has withheld. Such conspiracies directly obstruct agency activities in a way that

phony names on comments do not. Additionally, the difficulty of showing substantial harm to the

federal government or an agency will disincline any prosecutor to pursue such a case. Finally, the

charge here is conspiracy, not fraud, so the usual elements of conspiracy would have to be shown.

(b) Making False Statements

Moving away from fraud-based crimes, the obvious basis for a possible prosecution is the

prohibition on making “false, fictitious, or fraudulent” statements to a federal agency found at 18

U.S.C. § 1001. This is a sweeping prohibition; unlike fraud, it does not require a showing of

financial or property loss to the government or reliance by the government.

105

Most elements of

the crime seem satisfied here.

1. Though not “fraudulent,” a malattributed comment does make a “false” and “fictitious”

“statement or representation” in asserting that it is submitted by someone other than the

actual submitter.

2. The falsehood is knowing or willful. It is true that whoever is programming the computer

or authorizing the submissions does not know of each specific misidentification, but that

person does know that the misidentifications are being made.

3. A notice-and-comment rulemaking would seem to be “a matter within the jurisdiction” of

the agency conducting the rulemaking.

One issue remains: the false statement has to be material, meaning it “has a natural tendency

to influence, or [is] capable of influencing, the decision of the decisionmaking body to which it

was addressed.”

106

Use of a false name is not “material” unless the effect or influence of a comment

hinges on who submitted it. If the agency is taking account only of the content of the comment,

not the identity of its author, then the malattribution seems immaterial. Using random names from

the phone book to misidentify the source of a comment would in that case not be a material

misstatement.

107

105

See United States v. Richmond, 700 F.2d 1183, 1188 (8th Cir. 1983); United States v. Lichenstein, 610 F.2d 1272,

1278 (5th Cir. 1980).

106

Neder v. United States, 527 U.S. 1, 16 (1999).

107

See also Matthew Minor, Remarks at the Administrative Conference of the United States Symposium on Mass and

Fake Comments in Agency Rulemaking, supra note 13 at 125 (stating that “proving fraud here is hard to do” under

the materiality requirement in 18 U.S.C. § 1001).

A subcategory of malattributed comments could potentially violate § 1001, however.

Suppose a comment falsely claims to be from someone with extensive relevant expertise and

experience—a Ph.D. research chemist, a twenty-year line employee in the relevant industry, a user

of a product the agency proposes to ban, the owner or renter of property in the neighborhood of a

regulated facility. Because the person’s supposed unique, relevant experience would give the

comment more weight, that misstatement would be material. And if simply by using a particular

person’s name that information about background could be communicated, then just the false name

could be material.

It seems likely that such comments have been filed in federal agency rulemakings, though

instances would be hard to identify. A recent SEC rulemaking does provide an example, though it

arose prior to, rather than in comments on, the issuance of a proposed rule. In 2018, the SEC held

a roundtable regarding proxy rules and invited follow-up submissions, of which it received about

five hundred. In announcing the proposed rule, the Commission Chair, Jay Clayton, invoked

several of these:

Some of the letters that struck me the most came from long-term Main Street

investors, including an Army veteran and a Marine veteran, a police officer, a

retired teacher, a public servant, a single Mom, a couple of retirees who saved for

retirement, all of whom expressed concerns about the current proxy process.

108

Later reporting put those letters in a different light. All had been assembled, organized, and

written by an industry group funded by supporters of the SEC proposal.

109

The retired teacher did

sign her letter but had not written it; the veterans were the brother and cousin of the chair of the

industry group; the single mom did not write her letter; the retired couple were the in-laws of the

head of the industry group and when contacted had no recollection of ever writing any such letter;

the public servant reported that she had been contacted by a public affairs firm, that she did not

know what a proxy adviser is, and “[t]hey wrote [the letter], and I allowed them to use my name

after I read it. I didn’t go digging into all of this.”

110

Whether the reporting is accurate, and whether any of this violated § 1001, the incident

does flag the possibility of bespoke comments that make material misstatements. Those may or

may not include the name of the submitter. The sort of malattributed comments that have generated

attention and concern to date are quite different. They are duplicative and generic, make no

representations as to background or expertise, and do not use recognizable names of experts.

Finally, one of the ways in which malattributed comments can be misleading, as we discuss

below, is that they make it appear that more commenters hold a particular view than is the case.

108

Statement of Jay Clayton, SEC Chairman, at Open Meeting on Proposals to Enhance the Accuracy, Transparency

and Effectiveness of Our Proxy Voting System (Nov. 5, 2019), https://www.sec.gov/news/public-statement/statement-

clayton-2019-11-05-open-meeting.

109

Zachary Mider & Ben Elgin, SEC Chairman Cites Fishy Letters in Support of Policy Change, BLOOMBERG (Nov.

19, 2019, 10:03 AM), https://www.bloomberg.com/news/articles/2019-11-19/sec-chairman-cites-fishy-letters-in-

support-of-policy-change [https://perma.cc/ER4B-NZ7P].

110

Id. All in all, the interest group got about two-dozen people with connections to the organization to submit letters.

The group’s president insisted, by the way, that his mother- and father-in-law had known about the letter they

supposedly submitted: “They are 80-some-years-old. This happened months ago. I’m sure it’s not top of their minds.”

Id.

(This is true of large-scale, computer-generated comments generally; most malattributed

comments are just an example of that phenomenon in which the computer has attached a random

name to the comment.) A malattribution could perhaps be a false statement that is material not

because the agency cares who submitted the comment but because it cares that someone did. This

theory turns on the complicated question, discussed above, of whether and how an agency should

give weight to the number of comments taking a particular position.

2. The Administrative Procedure Act

Two separate issues arise under the APA. First, some have argued that it violates the APA

for agencies to accept malattributed comments and/or fail to remove them from the docket. Put

most strongly, the argument is that if there are a significant number of malattributed comments in

the docket the rulemaking is fatally tainted and must be abandoned. Presumably, on this reading,

the APA imposes an affirmative duty on agencies to prohibit submission of malattributed

comments and to police that prohibition conscientiously. The second argument is the opposite; it

holds that agencies have an obligation to accept malattributed comments and to give them

whatever weight they deserve. On this reading, it would violate the APA to do exactly what the

first reading says the APA requires.

Of course, it could be that both of these readings are mistaken and the APA is silent on this

matter. On this understanding, an agency could prohibit their submission and refuse to consider

them; but it could also instead choose to consider them along with all the other comments they

receive, giving each whatever weight it is due.

(a) An Obligation to Accept and Consider Malattributed Comments?

The APA requires agencies not just to accept but also to consider comments.

111

Courts

have broadened this obligation by requiring that when issuing a final rule agencies respond to all

significant comments.

112

Arguably, these obligations extend to malattributed comments just like

any other comment. To be sure, an agency will frequently conclude that the comment does not

require a response,

113

and its malattribution, if detected, could be one factor supporting that

conclusion.

114

But under one reading, the APA would require that an agency accept, review, and,

if there is something important and substantive in the comment, consider and respond to a

malattributed comments just like any other comment. And if that is true, agencies could not

prohibit submission of malattributed comments or remove malattributed comments from the

docket.

111

5 U.S.C. § 553 (2018) (providing that agency shall issue a final rule “[a]fter consideration of the relevant matter

presented”).

112

See, e.g., Del. Dep’t of Nat. Res. & Envtl. Control v. EPA, 785 F.3d 1, 15 (D.C. Cir. 2015); Cement Kiln Recycling

Coal. v. EPA, 493 F.3d 207, 225 (D.C. Cir. 2007); Grand Canyon Air Tour Coal. v. FAA, 154 F.3d 455, 468 (D.C.

Cir. 1998).

113

See Thompson v. Clark, 741 F.2d 401, 408 (D.C. Cir. 1984) (explaining that APA § 553(c) “has never been

interpreted to require the agency to respond to every comment . . . no matter how insubstantial”).

114

Cf. Mendelson, Rulemaking, Democracy, supra note 53, at 1378 (noting that an agency could “announc[e] that

anonymous comments will receive less weight, particularly when such comments purport to be informed by an

individual’s own experience”).

On the other hand, agencies unquestionably are free to impose reasonable requirements on

the form and content of public comments.

115

The most obvious example is that notice-and-

comment rulemaking always includes a comment deadline. An agency might consider late-filed

comments—different agencies have different practices. But it is universally accepted that an

agency can ignore a late-filed comment simply because it is a late-filed comment.

116

To take a

more directly relevant example, agencies can, and many do, prohibit anonymous comments.

117

Prior ACUS recommendations have not taken a position on whether agencies should or should not

accept anonymous comments, but one recommendation urges each agency to set a clear public

policy.

118

The premise of this recommendation, of course, is that it is up to the agency whether it

will accept or reject anonymous comments. And if that is the case with regard to anonymous

comments, then it would seem to be the case for malattributed comments.

The foregoing assumes that an agency has a clearly stated policy regarding the

permissibility of malattributed comments. The argument that the agency must consider such

submissions is more plausible if the agency has never indicated that it will not do so.

(b) An Obligation to “Cleanse” the Docket of Malattributed Comments?

The opposite argument would be that the APA prohibits agencies from considering

malattributed comments, or even that it imposes an affirmative obligation to weed them from the

docket. Such a claim was made by many observers regarding the net neutrality rulemaking.

119

115

See Dooling, supra note 75, at 905–15 (discussing such requirements, including civility and not revealing

confidential information).

116

See, e.g., Mont. Sulphur & Chem. Co. v. U.S. EPA, 666 F.3d 1174, 1195 n.12 (9th Cir. 2012) (“EPA was not

required to consider these untimely comments . . . .”); Reytblatt v. U.S. Nuclear Regulatory Comm’n, 105 F.3d 715,

723 (D.C. Cir. 1997) (concluding that agency need not respond to late comments even if it had indicated that it would

consider them); Bd. of Regents of the Univ. of Wash. v. EPA, 86 F.3d 1214, 1222 (D.C. Cir. 1996); JEFFREY S.

LUBBERS, A GUIDE TO FEDERAL AGENCY RULEMAKING 279 (4th ed. 2006).

117

In a 2011 ACUS report on agency practices regarding public comments, Professor Balla found that of the 25

agencies he studied, 15 (including EPA and several components of the Department of Transportation) allowed

anonymous comments and 10 (including the FCC, SEC, and Federal Reserve) required commenters to identify

themselves. See Steven J. Balla, Public Commenting on Federal Agency Regulations 23 (Mar. 15, 2011) (draft report

to the Admin. Conf. of the U.S.), https://www.acus.gov/sites/default/files/COR-Balla-Report-Circulated.pdf.

118

Admin. Conf. of the U.S., Recommendation 2011-2, Rulemaking Comments, ¶ 4, 76 Fed. Reg. 48789, 48791 (Aug.

9, 2011) (“The eRulemaking Project Management Office and individual agencies should establish and publish policies

regarding the submission of anonymous comments.”).

119

See, e.g., Klint Finley, FCC’s Broken Comments System Could Help Doom Net Neutrality, WIRED (Sep. 2, 2017),

https://www.wired.com/story/fccs-broken-comments-system-could-help-doom-net-neutrality (quoting Gigi Sohn as

stating that the agency might have an obligation under the APA to remove fake comments from the docket and that

“[a]t a bare minimum, they should investigate these comments and if they can’t actually remove the comments, they

can and should disregard them as part of their consideration of record”); Letter from Ellen F. Rosenblum, Or. Att’y

Gen. et al. to FCC 1 (Dec. 13, 2017), https://www.doj.state.or.us/wp-content/uploads/2017/12/ag_letter_12-13-

2017.pdf (letter from eighteen state Attorneys General urging the FCC to delay action because “the well of public

comment has been poisoned by falsified submissions,” which makes it impossible to “listen to the public” as the APA

requires); Letter from Allen S. Hammond, IV & Catherine J.K. Sandoval to Sen. Roger Wicker et al. 2 (Dec. 13, 2017)

(“The FCC’s apparent tolerance of allegedly criminal behavior in its comment process [in the form of acceptance of

malattributed comments] falls far below the required standard of reasoned decision-making under the Administrative

In general, if a comment makes no contribution, the agency does not reject it, it just ignores

it. This makes sense. Removing useless comments would require effort, would be inconsistent

with the public’s opportunity to comment in the first place, and would make it impossible for a

reviewing court to review the full record and determine, among other things, whether the agency

had in fact considered and responded to all significant comments.

120

If an agency must purge the

docket of malattributed comments, that must be because such comments are “worse than useless,:

that by their mere presence in the docket they cause affirmative harm that irrelevant or pointless

comments do not. It is hard to pin down precisely what this harm would be. The problematic aspect

of the comment—the false name—is easily ignored by the agency.

121

If there is a harm under the

APA, it would not flow from the malattribution per se, but from legitimate comments being

drowned out or misled by the level of actual public support for a particular position. It is not clear

that either of those things actually occurs. More particularly, it is not clear that they would occur

if the agency is sufficiently aware of the dubious provenance of certain comments to be in a

position to purge them from the docket in the first place.

Perhaps the most potentially problematic malattributed comment would be a bespoke,

sophisticated comment rather than a mass comment that happens to have a phony name. Consider

a comment that purports to come from the General Counsel of a leading industry player or the head

of a prominent civil rights organization and reads as if it could be legitimate. Such a “deep fake”

comment could sow confusion at the agency and among other commenters and prompt a perceived

need among other stakeholders to respond. Because of the impact on other commenters, this is the

situation where the argument that the APA requires the agency to remove the comment—assuming

it is aware of the malattribution—seems strongest.

No court has considered any issue regarding malattributed comments and the APA, even

when such a challenge could have been raised. When the FCC issued its final net neutrality order,

opponents promised that they would challenge the order in court on grounds related to the

malattributed comments.

122

Several petitions for review asserted that one of the Order’s legal

Procedures [sic] Act.”); see also Katherine Krems, Note, Crowdsourcing, Kind of, 71 FED. COMM. L.J. 63, 76 (2018)

(“When there is false information on the record, this information overshadows real public comments that reflect public

sentiment and contravenes the APA’s procedures meant to properly inform agencies of public opinion in decision-

making processes.”).

120

See Dooling, supra note 75, at 917–20. Of course, an agency can set reasonable requirements—for example of

civility or not revealing confidential information—for comments and police those requirements. Id. at 905–15. But

comments in violation of those requirements are not merely unhelpful, they do affirmative harm. The affirmative harm

from malattributed comments is much less clear, as discussed earlier.

121

Some of the objections to the presence of malattributed comments in the rulemaking docket rest not on the fact that

the comments bear an incorrect name but that they are computer-generated or fake mass comments—what looks like

submissions from thousands of people is in fact from only one—and therefore misleading with regard to public

sentiment.

122

See, e.g., Brian Fung, FCC net neutrality process ‘corrupted’ by fake comments and vanishing consumer

complaints, officials say, WASH. POST: THE SWITCH (Nov. 24, 2017), https://www.washingtonpost.com/news/the-

switch/wp/2017/11/24/fcc-net-neutrality-process-corrupted-by-fake-comments-and-vanishing-consumer-complaints-

officials-say/ (quoting Evan Greer of Fight for the Future as stating that “this will absolutely show up in court if we

get there”); see also Karl Bode, The FCC Is Blocking a Law Enforcement Investigation Into Net Neutrality Comment

Fraud, VICE (Dec. 12, 2017), https://www.vice.com/en_us/article/wjzjv9/net-neutrality-fraud-ny-attorney-general-

investigation (“Expect the agency’s failure to police comment fraud to play a starring role in these legal arguments to

come.”).

defects was that it “conflicts with the notice-and-comment requirements of 5 U.S.C. § 553”

123

but

no party actually raised the issue of malattributed comment to the court during briefing.

124

The final question is whether it would violate the APA for an agency to read or to rely on

a comment submitted under a false name. The rulemaking provision itself, § 553, in no way

restricts what the agency can consider or who it can listen to. Rather, any such restriction would

rest on the requirement of “reasoned decision-making” embedded in the prohibition on arbitrary

and capricious agency action.

125

Such determinations are case-specific. It would not be reasoned

decisionmaking to rely on malattributed comments as a measure of public sentiment or to rely on

a comment that purported to be from an authority in a relevant field when it was not. But if a

comment is relevant, factually accurate, and communicates something of value, there is nothing

arbitrary and capricious in an agency making use of what it has to offer, regardless of whether the

sender put someone else’s name on it. Thus, there would seem to be no per se rule allowing or

prohibiting agencies to rely on malattributed comments.

A. Policy Issues Raised By Malattributed Comments

In this section, we elaborate on the concerns raised by malattributed comments in particular

and also discuss how agencies can discourage submission of malattributed comments and handle

malattributed comments once they are discovered. At the same time, we acknowledge that a

malattributed comment may nevertheless contain useful content.

1. Misleading the Agency

Because malattributed comments, by definition, contain a falsehood, an obvious concern

is that the agency may be misled. The misleading might take either of two forms: the agency could

be misled with regard to the identity of the commenter, and the agency might be misled as to public

opinion, mistakenly viewing the phony comments as indicators of broader public support for a

particular position than actually exists.

(a) Commenter Identity

With regard to the first, in general, the agency simply will not notice the name of the

commenter. If the agency receives 10,000 very similar, computer-generated comments, no one is

paying attention to the names under which they are submitted, whether they are false or real. One

cannot be misled by something of which one is unaware. If the submission is not computer-

generated—a unique comment filed under a false name—the falsehood is irrelevant for purposes

123

Pet. for Review at 2, Ctr. for Democracy and Tech. v. FCC, No. 18-1068 (D.C. Cir. Mar. 5, 2018); Pet. for Review

at 2, New York v. FCC, No. 18-1055 (D.C. Cir. Feb. 22, 2018).

124

The D.C. Circuit’s opinion, which was largely but not entirely in the agency’s favor, does not mention the

“fraudulent comments” issue. See Mozilla Corp. v. FCC, 940 F.3d 1 (D.C. Cir. 2019).

125

See 5 U.S.C. § 706(2)(a).

of the agency’s deliberation. The agency will take the comment for what it is worth; the name adds

nothing to its weight and will not affect how it is treated.

126

Now suppose the name is one that someone in the agency recognizes. This is the “deep

fake” scenario described above. For example, it may be an important researcher or advocate in a

relevant field, or the general counsel of a prominent regulated entity. In this situation, the identity

of the commenter may matter for the agency’s deliberations. The agency could give particular

weight to such a comment. But it would be highly unlikely for the falsehood to go unnoticed. The

very background knowledge that makes the name recognizable will make it hard for someone to

pull the deception off. This is especially the case in a situation where the purported commenter’s

interests are well known to the agency, perhaps because of repeat interaction. However, for an

agency that does not regulate often, or that regulates only in certain domains infrequently, this

might be harder to ensure.

The malattributions that often grab observers’ attention involve using the name of a famous

(sometimes dead) person. But these are not misleading because it is apparent that the name is false.

For example, in the net neutrality rulemaking, there were multiple submissions from “Barack

Obama” and from “Ajit Pai.”

127

This does not result in any actual deception; no rulemaking official

would think that the former President or the FCC Chair had submitted the comment. Same for

submissions from “Elvis Presley.”

Our discussions with agency officials are consistent with the foregoing. Their own sense

is that consequential instances of pseudonymous submissions are extremely rare, if not

nonexistent. Of course, we have not done a thorough study and by definition the victim of a

successful deception is unaware of having been deceived. Nonetheless, we credit these statements

because they reflect actual experience, they are consistent with what one would expect, and we are

unaware of a single demonstrated instance to the contrary.

There is one possible setting, however, in which the concerns about the agency being

misled may be more serious. Suppose a comment is not from a recognizable name but asserts that

the submitter has particular experience that appropriately goes to the weight of the comment. For

example, the commenter claims to have done research in the area, or to possess “situated

knowledge,”

128

or to have had direct personal experience, or to be a person who will be directly

regulated or benefitted by the proposed rule. All of those people possess information that members

126

See, e.g., Letter from Ajit Pai, Chairman, FCC, to Rep. Michael E. Capuano (Apr. 12, 2018) (“Despite any

suggestion that the public comment process was somehow ‘flawed’ or ‘tampered with’ by the alleged submission of

comments under false names, any such activity did not affect the Commission’s actual decision-making . . . .”); Letter

from Thomas M. Johnson, Jr., Gen. Counsel, FCC, to Eric Schneiderman Att’y Gen., New York (Dec. 7, 2017) (“[T]he

Commission does not make policy decisions merely by tallying the comments on either side of a proposal to determine

what position has greater support, nor does it attribute greater weight to comments based on the submitter’s identity.”).

127

See, e.g., “Barack Obama,” ID 1051157755251, Restoring Internet Freedom, WC Docket 17-108 (May 11, 2017),

https://ww.fcc.gov/ecfs/filing/1051157755251 (submission from “Barack Obama” of “1600 Pennsylvania Avenue,

Washington, DC,” objecting to the “unprecedented regulatory power the Obama Administration imposed on the

internet” and “Obama’s . . . power grab”).

128

See Cynthia R. Farina, Dmitry Epstein, Josiah Heidt, & Mary J. Newhart, Knowledge in the People: Rethinking

“Value” in Public Rulemaking Participation, 47 WAKE FOREST L. REV. 1185, 1187–88, 1197 (2012) (describing

“situated knowledge” as “information about impacts, ambiguities and gaps, enforceability, contributory causes,

unintended consequences, etc. that is known by participants because of their lived experience in the complex reality

into which the proposed regulation would be introduced”).

of the general public do not and that the agency may find valuable. They may also have a stake

that should counsel caution in taking their assertions at face value. For both reasons, the agency

would want to know who the source of the comment is. An anonymous comment that claimed to

be from a person in such categories would be somewhat suspect; a signed comment may carry

more weight. If the name is a malattribution, and the actual submitter does not have the

qualifications claimed, there is a real risk of inappropriate reliance on the comment. Moreover,

suppose a rulewriter found that comment helpful but wanted to double-check its provenance. An

internet search might reveal the falsehood, but it might reveal nothing, or might appear to confirm

the biographical claims made in the comment.

This risk seems real but slim. We are not aware of real-world examples of such

submissions. That does not mean they have not occurred. In the real world, the malattributed

comments that have gotten attention were duplicative rather than bespoke; they do not make

individualized claims about the submitter. In addition, the real problem here is not the

malattribution so much as the biographical misrepresentation. The malattribution may make it

harder to uncover the relevant falsehood but is not itself misleading. Thus, the problem here is

actually the distinct one of accuracy in the assertions within comments. It is entirely possible for

commenters submitting under their own name to misrepresent their experiences, expertise, or even

views. The SEC proxy rule proceeding is an example.

129

(b) The Weight of Public Support or Opposition to the Proposed Rule

The second concern is that the agency will be misled as to public sentiment. Malattributed

comments are often, though not necessarily, a form of computer-generated comments. Such was

the case in the Net Neutrality rulemaking, for example. Millions of individuals did not sit down

and prepare comments that they submitted under someone else’s name. A handful may have done

so, but presumably almost all the malattributed submissions involved a computer taking a prepared

text, or writing a text, and then randomly attaching actual names and email addresses to the

comment. As with computer-generated comments, part of what observers object to here is that

what looks like a set of mass comments submitted by millions of concerned individuals is in fact

just the effort of a single submitter. To the extent this is the concern, the malattribution is largely

irrelevant. Perhaps, however, an agency might think the 100,000 identical comments with different

names are more likely to be from different individuals than are 100,000 identical anonymous

comments, in which case malattributed computer-generated comments are more misleading. This

is especially problematic to the extent that public comments are understood by agencies as

providing insight into public sentiment.

2. Harms to Individuals

Unlike mass and computer-generated comments, malattributed comments can have

impacts outside the agency and the rulemaking process, imposing harms on the people whose

names and email addresses are used without permission. Many or most will never be aware that

they have supposedly submitted a comment in a federal rulemaking, and many or most may not

129

See supra text accompanying notes 108-110.

care. Even if using someone’s name and address on a comment does not constitute identity theft

under federal law,

130

it still may be harmful to the person whose name is used in this manner.

Two sorts of harms can be imagined. The first is psychological. It would be understandable

that a person who learned that their name was used to submit a comment would be annoyed or

angry, especially if they disagreed with the content of the comment. The harm is somewhat

abstract; unlike standard identity theft, the victim’s bank account is intact. But for some people,

the distress or anger will be quite real.

131

The second possible harm is reputational. For a

malattributed comment in a regulatory docket to cause reputational harm, it would have to be

noticed by someone who changes their opinion of the purported commenter for the worse. The

obscurity of the rulemaking process may make this unlikely, and we are not aware of any instance

in which it has occurred. Still, all it takes is one viral tweet by someone with a large following

about a comment considered benighted or outrageous to do serious harm to the ostensible author

of the comment. Quantifying these harms may well be impossible. Individual views on how

seriously to take them will vary. While some people will not care or perceive themselves to be

harmed at all, others may see themselves as victims of identity theft.

3. Discouraging Malattributed Comments

The e-rulemaking program has taken several recent steps to discourage the submission of

malattributed comments. For example, the user notice on regulations.gov now includes the

following under the heading “Terms of Participation”:

Public comments help agencies develop regulations; we encourage comments from

all viewpoints. Comments submitted to Regulations.gov should be the submitter’s

own comments or be submitted with the commenter’s permission. The development

of federal regulations is within the jurisdiction of the U.S. Government’s executive

branch agencies. It is a violation of federal law to knowingly and willfully make a

materially false, fictitious, or fraudulent statement or representation including false

statements about your identity or your authority to submit a comment on someone

else’s behalf, in relation to the development of such federal regulations, including

through comments submitted on Regulations.gov. See 18 U.S.C. § 1001.

Subject to 18 U.S.C. § 1028(c), it is also a violation of federal law to knowingly

use, without lawful authority, a means of identification of another person in

130

See Herz, supra note 45, at 51–55.

131

For examples, see Letter from Brittany Ainsworth et al., to Ajit Pai, Chairman, FCC (May 25, 2017) (letter from

27 individuals whose names and email addresses were used to submit comments without their involvement or

permission complaining that “someone stole our names and addresses, publicly exposed our private information

without our permission, and used our identities to file a political statement we did not sign onto” and calling on the

agency to remove these “fraudulent comments” from the docket and notify “all proper authorities”); Bode, supra note

122 (complaining that “the agency told me there was nothing it could do after someone hijacked my identity to claim

I falsely supported killing net neutrality protections”), https://www.vice.com/en/article/wjzjv9/net-neutrality-fraud-

ny-attorney-general-investigation; Press Release, A.G. Schneiderman Releases New Details On Investigation Into

Fake Net Neutrality Comments (Dec. 13, 2017), https://ag.ny.gov/press-release/2017/ag-schneiderman-releases-new-

details-investigation-fake-net-neutrality-comments (quoting, among others, unidentified individual as saying “I’m

sick to my stomach knowing that somebody stole my identity and used it to push a viewpoint that I do not hold”).

connection with the violation of any federal law or the commission of a felony

under state or local law. See 18 U.S.C. § 1028(a)(7).

By clicking the submit button, you are verifying that you are not making any

materially false, fictitious, or fraudulent statement or representation regarding

your identity or your authority to submit on someone else’s behalf with regard to

the comment you are submitting on Regulations.gov, and that you are not using,

without lawful authority, a means of identification of another person, real or

fictitious, in connection with any comment you are submitting on

Regulations.gov.

132

This notice implies that a malattributed comment could be a federal crime. This may be

part of a deterrence strategy on the part of the government to discourage anyone from sending

malattributed comments. Whether users are likely to see this, in a user notice that contains several

other paragraphs of policies, is uncertain. Moreover, although it may discourage some individual

submitters from using a false name, it is unlikely to have any impact on large-scale operations.

Agencies have to make decisions about how to treat malattributed comments, once

suspected or discovered. Because of the novelty of this issue, many agencies do not have protocols

for how to resolve whether a comment is malattributed and, if so, policies on how to handle that

comment in the docket. Questions that such policies would address include whether an agency

should strive to resolve a question about a comment’s provenance, or merely flag the potential

issue. Also, and in line with the user notice described above, to the extent that criminal action is

under consideration for particular malattributed comments, agencies may need to make staff

available to assist with any investigations or prosecutions.

B. Technological Responses to Malattributed Comments

The technology readily exists to authenticate users and is in widespread use in many

contexts. Common techniques include secure login, two-factor authentication, biometric

authentication using facial recognition or fingerprint, answering security questions or verifying

names against a database (as is the case in voter registration), or clicking an additional “I agree”

button to acknowledge and agree to terms of service. However, agencies currently do not have the

technology in place to authenticate those filing comments in the way a government department

authenticates someone applying for a driver’s license or a commercial website authenticates

someone buying a product to prevent credit card fraud.

While tools are not in place to authenticate someone’s identity, agencies do use tools to

ensure that a commenter is a human instead of a bot. These tools are primarily a response to

computer-generated comments but, by imposing a “speed bump” on the commenting process, they

may also help to reduce malattributed comments.

132

User Notice, REGULATIONS.GOV, https://www.regulations.gov/user-notice (emphases added).

The addition of reCAPTCHA to regulations.gov is intended to help to “improve[] the

integrity of the commenting process.”

133

CAPTCHA is an example of a “Turing Test”—a thought

experiement developed by Alan Turing to evaluate artificial intelligence—and stands for

“Completely Automated Public Turing Test to tell Computers and Humans Apart.”

134

With

CAPTCHA, users are presented an image of a set of visually distorted letters and numbers and

asked to enter the same characters into a textbox. When CAPTCHA was invented nearly two

decades ago, it was believed that machines would not be able to complete this task since only

humans would be able to interpret what the distorted characters were. With advances in computing

power this is no longer true and techniques to defeat CAPTCHA have been created. CAPTCHA,

however, has also been reinvented to protect against these attacks. In 2018, Google announced

“reCAPTCHA v.3” which eliminates the need for any human interaction with CAPTCHA at all.

By using risk analysis algorithms that assign a “risk score” to every person browsing a website

using the tool, the software alerts administrators if fraudulent activity is detected.

135

Also, regulations.gov now includes a comment application programming interface (API)

to allow authorized entities to post mass comment campaigns to Regulations.gov if they have been

verified by GSA using a commercial identity validation service. In the press release announcing

these changes, GSA indicated that this was “to assure such entities ‘are who they say they are.’”

136

The service does not aim to verify the identities of individual commenters, however.

The public prominence of malattributed comments prompts a fresh look at whether

agencies should verify commenter identity, either on the front-end or after either an internal or

external review flags a comment as potentially malattributed. While authentication is a common

practice and technically straightforward in many circumstances, the practice would be in tension

with agency policies to permit anonymous comments.

V. Computer-Generated Comments

A. Legal Issues Raised by Computer-Generated Comments

The APA requires agencies to provide an opportunity to comment to “interested

persons.”

137

The term “interested” is undefined and is generally understood not to limit the scope

of potential commenters.

138

The term “persons” is defined as follows: “person includes an

individual, partnership, corporation, association, or public or private organization other than an

133

Press Release, Gen. Servs. Admin., GSA Launches Updated Regulations.gov to Improve the Integrity (Feb. 17,

2021), https://www.gsa.gov/about-us/newsroom/news-releases/gsa-launches-updated-regulationsgov-to-improve-

the-integrity-of-public-commenting-02172021#:~:text=WASHINGTON%20%E2%80%94%20The%20U.S.%20

General%20Services,gov%20launching%20February%2018%2C%202021.&text=%E2%80%9CThe%20new%20R

egulations.gov%20re,and%20mobile%2Dfriendly%20interface.%E2%80%9D.

134

The “imitation game” experiment proposed by Turing was invented as a way of approaching the hard question of

“Can machines think?”. See A.M. Turing, Computing Machinery and Intelligence, 59 MIND 433 (1950).

135

reCAPTCHA: Easy on Humans, Hard on Bots, https://www.google.com/recaptcha/intro/v3.html?ref=techmoon

(last visited Apr. 1, 2021).

136

See Gen. Servs. Admin., supra note 130.

137

5 U.S.C. § 553(b).

138

See Herz, supra note 75, at 357–58.

agency.”

139

When Congress passed the APA, it would not have contemplated that a computer

might send a comment. But the definition is instructively broad; it is not limited to natural persons,

and courts have read the word capaciously.

140

Moreover, because a person must set a computer-

generated comment in motion, the section 551 definition is arguably met in any event.

As described above, agencies are required to respond to significant issues raised in

comments. As of this writing, no courts appear to have interpreted this requirement in light of

computer-generated comments. During the interviews, agency staff expressed skepticism that a

computer-generated comment would bring content or issues to the rulemaking docket that were

not otherwise raised by other comments. But these staff also expressed their commitment to

reviewing all comments, regardless of origin, to ensure compliance with their obligations to

consider and respond to comments.

It is theoretically possible—if highly unlikely at this time—that a person would challenge

an agency action on the basis of its failure to adequately account for the substance of a computer-

generated comment that was not otherwise presented in other comments. Should such

circumstances arise, courts may determine whether the suit should move forward based on factors

such as whether the petitioner can demonstrate the reliability or authenticity of the computer-

generated comment.

141

It is also possible that authentication technology might exclude either

computer-generated comments or ordinary comments that raise unique significant issues. If the

agency’s obligation to consider and respond to significant comments does not change in such

circumstances, technological means of identifying computer-generated comments would have to

account for this overarching obligation.

B. Policy and Technical Issues Raised by Computer-Generated Comments

The policy issues raised by computer-generated comments overlap significantly with those

already identified for mass and malattributed comments. For example, the presence of computer-

generated comments may undermine public confidence in the rulemaking process or draw down

agency resources. Many of the issues presented by computer-generated comments, however are

technical. First, one issue is the ability of agencies to identify computer-generated comments. In

2019, an experiment demonstrated the ease with which bots mimic human speech, therefore

making it difficult to distinguish computer-generated comments from comments directly submitted

by persons.

142

The focus of this experiment was a comment period on a waiver from federal

requirements requested by the Idaho Medicaid program. A text generation model was utilized to

139

5 U.S.C. § 551(1).

140

See e.g., O’Rourke v. U.S. Dep’t of Justice, 684 F. Supp. 716, 718 (D.D.C. 1988) (holding that “person” includes

non-citizens and collecting cases); Neal-Cooper Grain Co. v. Kissinger, 385 F. Supp. 769, 776 (D.D.C. 1974) (foreign

government or instrumentality thereof is “person”).

141

In the evidentiary context, courts have managed to assess the admissibility of electronically stored information

(which may include computer-generated information) on the basis of the Federal Rules of Evidence; for example,

proponents must demonstrate the information’s relevance, reliability, authenticity, and so on. See, e.g., Lorraine v.

Markel Am. Ins. Co., 241 F.R.D. 534, 538 (D. Md. 2007). This is not to suggest that the Federal Rules apply to

administrative records; rather, this example is offered to demonstrate that courts may find useful analogies that may

be applied consistently with their equitable powers and authority under the APA.

142

Max Weiss, Deepfake Bot Submissions to Federal Public Comment Websites Cannot Be Distinguished from Human

Submissions, TECH. SCI. (Dec. 17, 2019), https://techscience.org/a/2019121801/.

submit one-thousand comments on the proposed waiver. The inputs for this model were thousands

of comments submitted in response to Medicaid waivers previously requested by a number of other

states. These inputs were used to train the model to employ search-and-replace techniques as a

means of generating comments, which were submitted automatically to the Centers for Medicare

and Medicaid Services at random intervals.

Following the submission of the computer-generated comments, subjects were recruited to

judge whether particular comments in the docket were submitted by a bot or human. On average,

the respondents—all of whom had previously demonstrated competency in identifying

conspicuous bot texts—correctly classified less than half of the comments. Performance was

particularly poor in the context of computer-generated comments, in that less than one-third were

correctly recognized. These results indicate that the computer-generated comments were as a

general matter plausibly human, therefore making consistent sorting of such submissions a non-

trivial exercise for the agency. At the conclusion of the experiment, the researcher revealed the

computer-generated comments and requested that CMS withdraw the bot submissions from

consideration.

These results are consistent with assessments of computer-generated comments that have

occurred outside of the context of experimentation. A variety of analyses have emphasized that

search-and-replace algorithms, and the resulting comment-to-comment variation in content,

enhance the difficulty of identifying computer-generated comments. As a result, “analysts have

struggled to pinpoint” the precise frequency with which computer-generated comments occur.

143

Are there approaches for identifying computer-generated comments in a systematic

manner? The FCC’s net neutrality policy is a good place to turn in this regard, as researchers have

expended considerable energy identifying computer-generated comments that were submitted in

this particular rulemaking. Note that these approaches entail identifying computer-generated

comments in hindsight, as opposed to screening for such comments during the intake process.

One analysis focused on the text of net neutrality comments, searching for expressions

regularly contained in submissions.

144

The analysis discovered combinations of phrases consistent

with the automated deployment of search-and-replace algorithms. Take, for example, the

following comment excerpt: “Americans, as opposed to Washington bureaucrats, deserve to enjoy

the services they desire.” This sentence repeatedly appeared in comments in numerous other

permutations, with “Americans” replaced by terms such as “people like me” and “individual

citizens.” Similarly, “the FCC,” “so-called experts,” and other analogous phrases substituted for

“Washington bureaucrats.” One result of this automation was the submission of large numbers of

comments that, while not identical, conveyed essentially equivalent sentiments. Another

characteristic of this process was the brevity of the resulting computer-generated comments.

145

Increases in comment length multiply opportunities “for the appearance of ‘tells’ (e.g., repeated

143

Katherine Krems, supra note 119, at 71.

144

Jeff Kao, More Than a Million Pro-Repeal Net Neutrality Comments Were Likely Faked, HACKERNOON (Nov. 22,

2017), https://hackernoon.com/more-than-a-million-pro-repeal-net-neutrality-comments-were-likely-faked-

e9f0e3ed36a6.

145

See Krems, supra note 119, at 75.

words, incorrect grammar, nonsensical sentiment) that the comment was not created by a

human.”

146

Other analysis has examined over-time patterns of the submission of net neutrality

comments with identical and near-duplicate content, an approach that is useful for identifying mass

comment campaigns (regardless of human or computer submission). One such pattern is the receipt

of large numbers of comments at precisely the same moment.

147

Researchers discovered, for

example, that on “nine different occasions, more than 75,000 comments were submitted at the very

same second—often including identical or highly similar comments.”

148

Another pattern is

embodied by the submission of the following comment excerpt: “The unprecedented regulatory

power the Obama Administration imposed on the internet is smothering innovation, damaging the

American economy and obstructing job creation.” This text occurred in approximately a half-

million comments.

149

These comments were submitted at near-constant rates for given periods,

which were punctuated by interludes during which no such comments were received.

150

This cycle

suggests that bots were turning on and off at specified intervals.

Another indication of the submission of computer-generated comments was repetition in

email addresses, in particular domains and locations exhibiting behavior inconsistent with human

messaging activity. The FCC, for example, determined that millions of comments were the product

of websites that produce one-off emails and are unable to receive messages. The agency also

discovered that hundreds of thousands of emails originated “from the same address in Russia.”

151

The regular submission of computer-generated comments was also suggested by the nature

of the information submitted along with the comments themselves. When humans fill out

information, the resulting inputs are typically inconsistent. For example, name, address, and email

fields are often left blank, and individuals utilize varying formats. In the context of the

“unprecedented regulatory power” comments referenced earlier, however, fewer than ten

submissions failed to contain complete information. Furthermore, these names, addresses, and

emails exhibited unusual similarity in presentation. Finally, exceedingly few comments requested

that the FCC provide email confirmation of receipt. These attributes suggest that algorithms, rather

than humans, were the immediate sources of the submitted information.

152

1. Current Agency Practices

In the interviews, agency staff expressed their awareness of computer-generated comments

having been submitted in a few rulemaking proceedings. Despite this awareness, the staff we

146

Weiss, supra note 142.

147

DAVID FREEMAN ENGSTROM ET AL., GOVERNMENT BY ALGORITHM: ARTIFICIAL INTELLIGENCE IN FEDERAL

ADMINISTRATIVE AGENCIES 60 (2020), https://www-cdn.law.stanford.edu/wp-content/uploads/2020/02/ACUS-AI-

Report.pdf.

148

Hitlin, Olmstead, & Toor, supra note 4, at 3.

149

See Chris Sinchok, An Analysis of the Anti-Title II Bots, MEDIUM, https://medium.com/@csinchok/an-analysis-of-

the-anti-title-ii-bots-463f184829bc.

150

See id.

151

Grimaldi & Overberg, supra note 6.

152

See Sinchok, supra note 149.

interviewed did not report systematic approaches to identify computer-generated comments. One

agency discovered computer-generated comments through a Wall Street Journal report on the

rulemaking, as well as the rulemaking team’s identification of a number of unusual comments.

These comments consisted of strings of nonsensical words, which made the agency suspicious that

the submissions were not generated by humans.

Despite the availability of tools discussed earlier under mass and malattributed comments,

implementing approaches to systematically identify computer-generated comments was not a high

priority for the agencies we interviewed. Agency staff characterized the discovery of computer-

generated comments as requiring substantial effort, a resource-intensive undertaking that is not

worth the dedication of agency bandwidth. In general, the interviews revealed that agencies are

not focused on the issue of computer-generated submissions in and of themselves. Rather, they

indicated greater concern about mass and malattributed comments whose detrimental attributes

may be deepened by computer generation. Despite this concern, the agency staff we interviewed

reported as their primary concern the need to identify and respond to significant issues that

comments raise, regardless of a given comment’s source.

One reason for the lack of attention to computer-generated comments in and of themselves

may be that agencies are already using de-duplication tools to address mass comments. Computer-

generated comments, in other words, are not seen as creating problems in rulemaking proceedings

other than increasing the volume of comments received by agencies—thereby turning the matter

into one of mass comment management. As an agency official put it during an interview,

computer-generated comments essentially present agencies with a de-duplication task. With the

utilization of de-duplication software, the unique content in computer-generated comments can be

readily identified.

As this perception indicates, agency staff generally expressed that the generation of

comments by computers is not, in and of itself, an important attribute of submissions. The point

was repeatedly made during the interviews that it is the substance of comments that matters, as

opposed to the identity of submitters or the volume of comments. Agencies were not overly

concerned that computer-generated comments convey insights that have not separately been

communicated through human comments. That said, agencies emphasized that they would not

exclude a computer-generated comment on the basis of the source of the submission, but would

consider whether it raised significant issues requiring agency consideration.

2. Threat Versus Practice

In sum, there is currently a disjuncture between conceptions of computer-generated

comments from the vantage points of technologists and the agency staff we interviewed.

Technologists warn of a present—and especially future—in which computer-generated comments

effectively mimic human content, thereby making prevention and detection an impossibility. The

agency staff we interviewed, by contrast, were not overly concerned with such scenarios at this

time. They saw de-duplication tools as being adequate for the task and did not seem anxious to

experiment with additional technologies to streamline the comment review process.

Notwithstanding current perceptions, in the years ahead, it will be important to monitor whether

the technologies that enable mass, malattributed, and computer-generated comments threaten to

undermine the perceived legitimacy of the notice-and-comment process and the ability of agency

officials to make sense of and consider comments thoughtfully.

VI. Innovations to Enhance Participation and Commenting

The foregoing discussion has identified some of the risks associated with mass,

malattributed, and computer-generated comments. These risks are real, and agencies must

undertake appropriate measures to ensure that they protect the integrity and value of the notice-

and-comment process. But technology can present opportunities as well as challenges. As we have

already seen, agencies extensively use de-duplication software to help them process mass comment

campaigns. And this is only a preview of what agencies can accomplish with newly emerging

technologies. This section explores technologies that are not yet in widespread use but that might

enhance and supplement the notice-and-comment process.

153

A. Summarization Technologies and Enhancing the Value of Public Commenting

Although our focus has been on mass, malattributed, and computer-generated comments,

an additional salient and urgent opportunity for regulators is using new technologies to enhance

the process of reviewing public comments.

154

While our interviews found that agencies are not currently using such tools, affordable

technology is on the horizon to help agencies more easily make sense of public comments, helping

those reading the rules, not simply to save time, but to better analyze, spot patterns in, and

understand public comments.

The GSA’s innovative technology unit 18F comments: “It takes enormous amounts of staff

time, resources, and taxpayer dollars to manually analyze written public comments submitted to

agencies through various committees and many other channels.” Their experts go on to

recommend the need to “explore if Natural Language Processing and Artificial Intelligence can

semi-automate, streamline, and expedite the public comment process, and whether any additional

policy or guidance might be required to create a standard approach.”

155

NLP techniques can help comment reviewers both summarize and sort comments, helping

them to extract the most important substantive information from the comments.

156

For example,

these techniques can be used to identify those parts of comments that bear on questions that are of

particular interest to rule-writers, or that contain relevant legal, technical, or operational

information.

157

153

See generally Beth Simone Noveck, The Innovative State, DAEDALUS (forthcoming 2021) (special issue on the

administrative state in the United States in the twenty-first century).

154

See Fake It Till They Make It, supra note 2 (testimony of Beth Simone Noveck),

https://financialservices.house.gov/uploadedfiles/hhrg-116-ba09-wstate-noveckb-20200206.pdf.

155

Aditi Rao, Ben Peterson, & Andrew Suprenant, Synthesizing Public Comments: Phase 2 Report (on file with

authors).

156

See Livermore, Eidelman, & Grom, supra note 19, at 980 (discussing “needle-in-the-haystack” and “forest-for-

the-trees” challenges of mass rulemakings).

157

The field of NLP encompasses a wide range of technologies that use computational tools to convert natural

language artifacts into a format that can be processed and analyzed using computational and statistical tools. NLP

techniques include deduping software as well as:

While still a challenging task, researchers and entrepreneurs have developed tools for

summarization, including shortening and extracting the most relevant portions of documents. To

sort information, one technique that can be used is topic modeling. In brief, a topic model is a

computational text analysis technique that extracts patterns in the semantic content in a corpus of

documents, generating a list of topics (which are distributions over the vocabulary in a corpus) and

characterizing every document as a distribution over those topics.

158

Topic modeling makes it

possible to automatically and quickly sort textual information into semantic categories.

Both Google and Microsoft announced in 2019 that they had built systems capable of

summarizing an enormous range of texts, including news, fictional stories, instructions, emails,

patents, and legislative bills.

159

The MIT Center for Constructive Communication conducted

research on large-scale Twitter data sets.

160

Its Electome project, for example, extracts semantic

• Flesch-Kincaid Readability: a measure of the difficulty or clarity of written English. The readability

score of a text is based on the average number of words per sentence and the average number of

syllables per word. Other readability metrics include the Gunning Fog Index and the Spache Index.

• Linguistic Inquiry and Word Count (LIWC): a software application that counts “words in psychology-

relevant categories,” such as whether words are associated with honesty or deception or track

individual thinking styles.

• Plagiarism Detection: a technique for detecting similarity in written texts, with the goals of identifying

plagiarism or copyright infringement.

• Automated Document Summarization: an application that processes larger texts, or multiple texts, as

inputs with the goal of generating summary texts that convey a condensed version of the original input

texts.

• Sentiment Analysis: a measure of words based on positive or negative valence, as a way to estimate the

opinions or attitudes expressed in a written text.

• Topic modeling: a family of computational tools used to discover the latent thematic structure within a

collection of documents.

• Word Embeddings: a technique for mapping words or phrases into a vector space that compactly

represents semantic content. One technique for generating word embeddings involved “skip-gram”

where a model is trained to use a word to predict surrounding words in a document.

158

For key works in the relevant NLP literature, see, e.g., Kincaid et al., Derivation of New Readability Formulas

(Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel, Research

Branch Report 8-75 (1975); Adam Feldman, Opinion Clarity in State and Federal Trial Courts, in LAW AS DATA:

TEXT, COMPUTATION, AND THE FUTURE OF LEGAL ANALYSIS 407, 415–18 (Michael A. Livermore & Daniel N.

Rockmore eds., 2019); Yla R. Tausczik & James W. Pennebaker, The Psychological Meaning of Words: LIWC and

Computerized Text Analysis Methods, 29J. LANGUAGE & SOC. PSYCH. 24 (2010); BING LIU, SENTIMENT ANALYSIS:

MINING OPINIONS, SENTIMENTS, AND EMOTIONS (2015). David M. Blei, Andrew Y. Ng, & Michael I. Jordon, LATENT

DIRICHLET ALLOCATION, 3 J. MACH. LEARNING RES. 993 (2003); Mikolov et al., Distributed Representations of

Words and Phrases and Their Compositionality, in ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 26,

3136 (2013).

159

Patrick Fernandes, Miltiadis Allamanis, & Marc Brockschmidt, Structured Neural Summarization, published at

Int’l Conf. on Learning Representations (Feb. 2019), https://arxiv.org/pdf/1811.01824.pdf; see also Peter Liu & Xin

Pan, Text summarization with TensorFlow, GOOGLE AI BLOG (Aug. 24, 2016),

https://ai.googleblog.com/2016/08/text-summarization-with-tensorflow.html.

160

The Laboratory for Social Machines, which carried out some of this research, was incorporated into the MIT Center

for Constructive Communication in 2021. See LABORATORY FOR SOCIAL MACHINES, MIT MEDIA LAB,

https://www.media.mit.edu/groups/social-machines/overview/.

content from the entire corpus of Twitter data—billions of tweets—in order to summarize the core

political messages of the day and help drive election coverage.

161

Such summarization and sorting processes sometimes combine automation with human

intelligence to make quick work of large data stores and overcome the biases that arise from using

automation alone. Journalists took advantage of such tools, for example, when they needed to

rapidly sift through the 13.4 million documents that comprised the so-called “Paradise Papers.”

162

Public institutions have also used natural language data analytical techniques to make sense of

social media data. To help UNICEF and other actors craft more effective pro-immunization

messaging programs, researchers set out to monitor social media networks, including blogging

platforms, forums, Facebook, Twitter, Tumblr, and YouTube. They sought to analyze prevalent

conversation themes according to volume, types of engagement, and demographics; to identify

influencers across languages and platforms; and to develop specific recommendations for

improving messaging strategies across languages, platforms, and conversation themes. The

research methodology involved scraping text from conversations on social media platforms in

English, Russian, Polish, and Romanian, in order to be able to summarize them and identify key

patterns.

163

A recent State Department project offers a simple illustration of how agencies could make

sense of rulemaking comments using a combination of artificial intelligence and human oversight.

In 2016, the State Department sought to improve its passport application and renewal process in

anticipation of an increase in the number of passport application and renewal forms. It ran an

online public engagement process to ask people what improvements they wanted, receiving almost

1,000 suggestions. In order to make rapid sense of those submissions, it used a third-party software

company, which applied a text-mining algorithm that scanned the highlighted text for responses

containing similar keywords in order to create summaries. The public was invited to proof and

make suggestions for how to improve those highlights, adding accountability but in a way that is

efficient. The combination of human and machine intelligence made it faster and easier to

summarize content than using an algorithm alone.

To date, application of NLP to public comments in administrative rulemaking has been

largely limited to de-duplication. While still under development, more advanced NLP techniques

could eventually assist agency personnel in identifying relevant substantive content within

comments and summarizing the information presented across a broad spectrum of comments. One

of the challenges for deploying summarizing technology in the context of rulemaking is that there

is often domain specific language that requires retraining the relevant models. However, for

important rulemakings likely to receive a large number of comments, this investment may well be

worth it. Furthermore, the addition of human oversight can provide a check on the performance of

machine learning applications, making it possible to evaluate and confirm the reliability of new

tools for summarization. While NLP tools can be used to augment, rather than replace, human

161

See THE ELECTOME, http://www.electome.org/.

162

Fabiola Torres López, How They Did It: Methods and Tools Used to Investigate the Paradise Papers, GLOB.

INVESTIGATIVE JOURNALISM NETWORK (Dec. 4, 2017), https://gijn.org/2017/12/04/paradise-papers/.

163

STEFAAN G. VERHULST & ANDREW YOUNG, THE POTENTIAL OF SOCIAL MEDIA INTELLIGENCE TO IMPROVE

PEOPLE’S LIVES (Sept. 24, 2017) (report for The Governance Lab), http://www.thegovlab.org/static/

files/publications/social-media-data.

review, the agency staff we interviewed expressed concerns related to how the use of new

technologies might interact with their legal obligation to review and respond to comments. This

legal uncertainty creates the risk that agencies may innovate slowly. Depending on their risk

tolerance, it may prevent them from adopting these technologies at all.

B. CrowdLaw: Innovations in Equitable Participation

In addition to improving the commenting process ex-post using new technologies, agencies

could also explore using complementary platforms and processes—ones already well-honed and

tested by other governments—to create new opportunities for public engagement, especially to

solicit information and expertise from more diverse and varied audiences as a complement to

notice-and-comment. Building on ACUS’ earlier work, we conclude our discussion of public

participation in rulemaking by looking at several contemporary examples of how governments are

enhancing citizen participation using new technology.

164

Over the last decade, federal agencies have expanded citizen engagement through the use

of prize-backed challenges or what is sometimes called open innovation via the Challenge.gov

website. Since 2011 with the reauthorization of the America Competes Act,

165

a hundred federal

agencies have run online challenges to tap the intelligence and expertise of the public.

166

NASA

has regularly used prize-backed challenges to spur crowdsourcing of innovative solutions from the

public. The Asteroid Grand Challenge, for example, was focused on finding all asteroid threats to

human populations.

167

Prize-backed challenges require agencies to articulate and define exactly

what information they need from the public and provide very transparent and specific criteria for

evaluating public submissions. With ten years of experience with prized backed challenges, there

may be useful insights for federal agencies to draw about how to improve public participation in

agency decision making.

Challenge.gov is one example of institutionalized public engagement or what is sometimes

referred to as “CrowdLaw,” namely the use of technology to engage the public in law-, rule-, or

policy-making. It is the idea that public institutions work better when they increase citizen

engagement by using new technologies to obtain diverse sources of information, insight and

164

As Michael Herz wrote in a 2013 ACUS report:

[T]he online world in general has come to be increasingly characterized by participatory and

dialogic activities, with a move from static, text-based websites to dynamic, multi-media platforms

with large amounts of user-generated content. At the heart of this move to “Web 2.0” have been

social media, blogs, Twitter, Facebook, YouTube, IdeaScale, wikis, Flickr, Tumblr, and the like.

Outside the rulemaking setting, federal, state, and local governments have enthusiastically jumped

on the social media bandwagon.

Michael Herz, USING SOCIAL MEDIA IN RULEMAKING: POSSIBILITIES AND BARRIERS (Nov. 2013) (report to Admin.

Conf. of the U.S.), https://www.acus.gov/sites/default/files/documents/Herz%20Social%20Media

%20Final%20Report.pdf.

165

America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science

Reauthorization Act of 2010, Pub. L. No. 111-358, 124 Sta. 3982 (Jan. 4, 2011).

166

About, CHALLENGE.GOV, https://www.challenge.gov/about (last visited Apr. 1, 2021).

167

Asteroid Grand Challenge, NASA.GOV, https://www.nasa.gov/content/asteroid-grand-challenge (last visited Apr.

1, 2021).

expertise at each stage of the law and policymaking cycle to improve the quality as well as the

legitimacy of the resulting laws, regulations, and policies, especially by engaging with

underrepresented communities.

168

CrowdLaw does not describe one form of participation. Rather,

it describes a variety of different methods, tools and platforms that institutions use.

Expert sourcing, where officials crowdsource expert advice, is one example of how

government bodies are implementing more citizen engagement. The Federation of American

Scientists’ Congressional Science Policy Initiative invites hundreds of scientists to help draft

questions for Members of Congress to ask of committee witnesses. Such crowdsourcing, facilitated

by new technology, helps beleaguered staffers write more informed questions.

169

The Governance

Lab at NYU uses videoconferencing to help coordinate online dialogues among experts to advise

government officials on a variety of topics. In Fall 2020, for example, it ran six deliberative

sessions at the behest of seven governments in Latin America to help them develop implementable

strategies for responding to specific public health challenges, including the improvement of mental

health services and combating misinformation.

170

Some jurisdictions have used online collaborative drafting processes and platforms to write

policies and rules with the public, especially with expert members of the public. Instead of an

advisory committee or hearing with a handful of experts or writing rules entirely behind closed

doors, online collaborative annotation makes it possible to hear from a broader and deeper range

of experts and to focus their participation on specific comments on a document. In 2018, the

German government used an annotation platform to “expert source” feedback on its draft artificial

intelligence policy. By putting the draft on Hypothes.is, a free and open-source annotation tool,

the German Chancellor’s Office, working in collaboration with Harvard University’s Berkman

Center for Internet and Society, was able to solicit the input of global legal, technology and policy

experts. Using an annotation platform also made it possible for people to see one another’s

feedback, instead of a series of disconnected comments. One could envision an agency using

collaborative annotation to invite experts to annotate and comment on the text of a draft rule.

Many governments are experimenting with the use of random samples of members of the

public as a mechanism to obtain more legitimate forms of participation. New technology is making

it easier to assemble these representative samples of citizens, known as mini-publics, to weigh in

on a governing process. Small groups are known as citizen juries while larger random samples are

called citizen assemblies. For example, in the Brussels-Capitol region, a random sample of citizen

representatives serves on each parliamentary committee. Citizens ask questions and provide

168

Victòria Alsina & José Luis Martí, The Birth of the CrowdLaw Movement: Tech-Based Citizen Participation,

Legitimacy and the Quality of Lawmaking, 40 ANALYSE & KRITIK 337 (2018); see also Beth Simone Noveck,

CrowdLaw, in THE PALGRAVE ENCYCLOPEDIA OF INTEREST GROUPS, LOBBYING AND PUBLIC AFFAIRS (Phil Harris et

al. eds.) (forthcoming 2021).

169

CONGRESSIONAL SCIENCE POLICY INITIATIVE, https://fas.org/congressional-science-policy-initiative/ (last visited

Apr. 1, 2021).

170

See Smarter Crowdsourcing: Coronavirus, THE GOVERNANCE LAB, https://coronavirus.smartercrowdsourcing.org

(last visited Apr. 1, 2021).

advice.

171

These processes could also be designed to elicit expertise and know-how relevant to

agency decision makers.

Similarly, some have suggested ideas such as administrative agencies empaneling a

thousand randomly selected citizens to provide oversight over agency decisionmaking.

172

variation on this idea would use citizen juries to solicit information on agency agenda setting and

priorities,

173

providing the citizen jurors with background materials generated by deliberative

polling before their discussions.

174

Finally, instead of selecting a random sample, other institutions have relied on self-selected

participation using a variety of tools. In month-long online exercises known as “Evidence Checks,”

UK parliamentary committees invite experts, stakeholders, and members of the public to comment

on the validity of evidence on which a policy is based. The process begins when government

departments supply information to their respective committees about an issue. Each committee

publishes the information on a parliament.uk web page, and it is scrutinized by a wider pool of

invitees. The committee also presents specific questions and problems that it would like

participants to address. In contrast to a representative sample, this process allows a group of people

with relevant experience and expertise to identify gaps in research that require further review.

175

Another example of self-selected participation was initiated by the New Jersey Department

of Education in March 2021 when that agency invited students, parents and educators across the

state to help inform the Department’s policymaking by responding to questions via All Our Ideas,

a free platform developed at Princeton University. All Our Ideas has been used in over 18,000

citizen engagement projects.

176

The owner of the consultation uses the platform to write a series

of statements that are then randomly presented to the participant. People select the response they

prefer (or “I can’t decide” as a third answer) or they may submit their own response. As people

are repeatedly selecting between two randomly generated options, it is a faster and easier

mechanism for responding to a series of questions. This so-called “wiki survey” method of

showing people two pieces of information and having them choose between them and/or submit a

new item offers efficiency benefits over open-ended commenting and can be designed to draw on

participant expertise.

171

The Governance Lab, Belgian Sortition Models: Institutionalizing Deliberative Democracy, CROWDLAW FOR

CONGRESS, https://congress.crowd.law/case-belgian-sortition-models.html (last visited Apr. 1, 2021).

172

See David R. Arkush, Direct Republicanism in the Administrative Process, 81 GEO. WASH. L. REV. 1458 (2013).

173

BETH SIMONE NOVECK, SMART CITIZENS, SMARTER STATE: THE TECHNOLOGIES OF EXPERTISE AND THE FUTURE

OF GOVERNING 220 (2015).

174

See Reeve T. Bull, Making the Administrative State “Safe for Democracy”: A Theoretical and Practical Analysis

of Citizen Participation in Agency Decisionmaking, 65 ADMIN. L. REV. 611 (2014).

175

Nesta, UK Parliament Evidence Checks, https://www.nesta.org.uk/feature/six-pioneers-digital-democracy/uk-

parliament-evidence-checks (last visited Mar. 28, 2021).

176

MATTHEW SALGANIK, BIT BY BIT: SOCIAL RESEARCH IN THE DIGITAL AGE 111–15 (Princeton Univ. Press,

2017).

Recommendations

To help facilitate committee deliberations, we offer the following draft recommendations. The

general categories are technology, coordination and training, docket management, and

transparency. We could conceive of many variations of the ideas below, but offer these to help

the committee formulate its views.

Based on our research, we believe that mass, malattributed, and computer-generated comments

do not, at least currently, fundamentally undermine the notice-and-comment process. However,

such comments raise issues of sufficient significance that steps can and should be taken to

mitigate the difficulties emanating from them. Technology also presents opportunities for

enhancing public engagement in rulemaking.

The following recommendations lay out a variety of immediate and long-run actions for reducing

the challenges of mass, malattributed, and computer-generated comments and taking advantage

of technology-enabled participation.

Technology

Mass, malattributed, and computer-generated comments raise challenges for agencies at two

stages of the rulemaking process.

The first stage is comment submission. At this stage, one difficulty stems from the submission

of large numbers of comments. The GSA has recently taken important actions to help agencies

manage mass comments. For example, the current version of Regulations.gov includes an API

that facilitates the submission of comments in bulk. Another difficulty concerns the

authentication of the identity of the commenter. In this regard, Regulations.gov has implemented

two features. One feature is identity validation in the API, which enables authorized users to

submit comments in bulk. The other feature is reCAPTCHA, which is designed to screen out

commenters who are not humans.

We recommend that agencies, both those that use Regulations.gov and those that do not,

consider utilizing bulk submission, identity validation, reCAPTCHA, or similar tools in their

comment submission processes.

We recommend that agencies and relevant coordinating bodies stay abreast of developments in

the submission of mass, malattributed, and computer-generated comments, so that approaches to

combating difficulties arising from such developments can be implemented as needed.

The second stage at which mass, malattributed, and computer-generated comments raise

challenges is the processing of comments. One challenge in this regard is the submission of large

numbers of duplicate or near-identical comments. Our research indicates that it is commonplace

for agencies to use de-duplication software to identify the unique content. An additional

challenge concerns the identification of comments submitted under false identities and by

computers. Our research suggests that agencies to this point have devoted little attention to

identifying comments in their dockets that are malattributed or computer-generated.

We recommend that agencies continue to (or, if they have not already, begin to) utilize de-

duplication software to identify the unique content in submitted comments.

We recommend that agencies publish policies regarding the posting of duplicate and near-

identical comments. These policies should balance values such as user-friendliness,

transparency, and informational completeness. Options that could be considered include: posting

a single representative example with the count of the duplicates received and an option to view

all comments; breaking out non-identical content; and providing enhanced search options based

on the unique information content of comments.

We recommend that agencies and relevant coordinating bodies should encourage and stay

abreast of technology for identifying malattributed and computer-generated comments in the

docket.

We recommend that agencies and relevant coordinating bodies stay abreast of technologies that

can facilitate public participation outside the notice-and-comment process. Agencies often find

that supplemental public participation processes can be useful, and a wide range of technologies

can be used to structure meaningful dialogue between agencies and relevant publics.

Coordination and Training

We recommend that agencies and relevant coordinating bodies share best practices and relevant

innovations for addressing challenges and opportunities connected with mass, malattributed, and

computer-generated comments, and technologies related to supplemental public participation

processes.

We recommend that agencies work closely with relevant coordinating bodies to improve

existing technologies and develop new technologies to address issues associated with mass,

malattributed, and computer-generated comments.

• The e-Rulemaking Program should provide a common de-duplication platform for

agencies to use, though agencies should be free to modify it or use another platform

as appropriate.

• The e-Rulemaking Program and other relevant coordinating bodies should work with

agencies and private sector experts and vendors to develop technologies that respond

to common issues associated with mass, malattributed, and computer-generated

comments.

We recommend agencies offer opportunities for ongoing training and staff development to

respond to the rapidly evolving nature of technologies related to mass, malattributed, and

computer-generated comments, and supplemental public participation processes.

Docket Management

We recommend agencies post comments promptly to the rulemaking docket. If a comment is to

be included in the docket, it is important that it be posted promptly upon receipt. Evaluation and

control of mass, malattributed, and computer-generated comments depends in part on public

responses to them. Most obviously, someone whose identity has been used in a malattributed

comment may notice and flag the malattribution. Such external correction cannot occur unless

the problematic submission is made available.

We recommend that, if an agency decides to exclude or remove some or all duplicate,

malattributed, or computer-generated comments from the docket, it articulate such a policy in

advance [or provide a reasoned explanation after excluding]. The presumption is that an agency

must place all comments in the docket. To exclude a document requires an affirmative

justification that is articulated in law or in an agency policy. Existing examples include certain

agency prohibitions on threatening language or profanity, on anonymous comments, and on

copyrighted material.

We recommend that an agency policy against submission of malattributed comments provide

that if the agency is aware that it has received such a comment, it either retain the comment in

the docket but remove the malattribution (i.e., render it an anonymous submission) or remove the

comment from the docket altogether. Even if agencies adopt technological barriers to the

submission of malattributed comments, those methods are not likely to be perfect. Our analysis

finds that agencies do not have an obligation to affirmatively search the docket for malattributed

comments. But agencies are free to set reasonable policies concerning the public comment

process and reject comments that violate their policies. Agencies can also rely on comments that

violated their commenting policies (e.g. late comments) in some circumstances. If an agency

determines that a malattributed comment will remain in the docket, anonymization should be

used to protect the person whose identity has been used.

We recommend that agencies not discard computer-generated comments that it receives,

although those comments may be segregated and treated separately.

We recommend that any duplicative, malattributed, or computer-generated comment on which

an agency actually relies be placed and retained in the rulemaking docket. As noted, agencies

may choose to anonymize malattributed comments, and to segregate or flag computer-generated

comments, that are retained in the docket.

We recommend that, to the extent practicable, agencies should provide opportunities (including

potentially after the comment deadline) for individuals whose names have been attached to

comments they did not submit to identify and request removal of such comments from the

docket.

We recommend that agencies consider taking affirmative steps to identify comments that are

malattributed [or computer-generated] comments.

We recommend that, if an agency flags a comment as malattributed [or computer-generated] or

removes such a comment from the docket and the submitter provided electronic contact

information, the agency should notify the submitter of the agency’s action.

Transparency

We recommend agencies and relevant coordinating bodies consider providing materials that

explain to prospective commenters what information is useful to an agency in a public comment.

This could include various formats to reach different audiences (e.g., videos, FAQs).

We recommend that in NPRMs, NOIs, and ANPRs agencies ask specific questions and identify

particular information that would be useful in developing the proposal.

We recommend that, when publishing a final rule, agencies state whether they removed from

the docket any malattributed and/or computer-generated comments.