Overview | Rating
System Development | The Rating System
A. OVERVIEW.
One of the goals of the TIPA is to develop a structure which
can be used across the nation to encourage candidates and campaigns
to run ads that say something meaningful, to encourage them to say
it honestly, to create incentives for them to say it accurately,
and to get them to say it intelligibly.
The TIPA Rating System will evolve over
time and several versions are anticipated to be released over time
as we revise it based on the input we receive from our Advisory Panel,
advertising and political professionals, and academics. While it attempts
to be simple and direct, we realize the development of such a rating
system is an extraordinarily difficult and complex task. To make
it easy to understand and to make it visually attractive for news
organizations, the Project will deploy a three-part rating system – part
one for factual accuracy,
part two for fairness, and part three for relevance.
One of the biggest obstacles to any Rating System, of course, is
the role personal biases and prejudices play in such a process. The
TIPA is attempting to address this challenge head-on.
The ratings will be depicted in a graphically-attractive
manner, using an “Accuracy Meter” and
a “Fairness
Meter” as well as a separate “Relevance
Scale.” These will be used as the primary images conveyed
to media organizations and in the Project’s own communications.
The Rating Systems will be applied on an individual advertisement-by-advertisement
basis as well as for an averaging approach which then can be used to
characterize each campaign.
The Accuracy Rating is based on whether the advertisement
delivers factual information about the candidates and real issues,
or whether it distorts the facts and takes events and other information
out of context to paint a negative, inaccurate, and even intentionally
misleading image of a candidate. Its basis is a traditional fact-checking
approach.
The Fairness Rating is based on whether the advertisement
paints a fair picture of the candidates and their positions on the
issues, or whether it is a smear designed to unfairly assault the character
and/or position of one or more candidates. While more subjective than
the Accuracy Rating, the TIPA is confident the moral compass most people
have will allow them to reasonably and accurately gauge the fairness
of an advertisement.
The Relevance Scale is based
on a straightforward subjective evaluation by the participant who
simply decides whether or not the advertisement includes information
he or she believes is relevant to the current contest. While one
could argue only certain issues – e.g., financial, economic, environmental, security,
educational – may be “relevant,” actual relevance
definitely is in the mind of the beholder. What may be relevant to
one person easily can be irrelevant to another. Rating relevance, therefore,
is an opportunity for the evaluator to say what is important to him
or her at that particular point in time.
The role of political partisanship looms large
in any rating of relevance. One could argue, “Why rate relevance if it likely will be a totally
partisan consideration?” The answer lies in the information provided
by those participants who take their assignment seriously and are willing
to be critical of advertisements in support of candidates from their
own party – as opposed to those who simply will respond with
a knee-jerk rating paralleling their own partisanship. By analyzing
this subgroup, the TIPA may be able to explore issues of how fair or
objective partisan participants can be (assuming the TIPA can determine
what objective ratings of relevance should be for each advertisement – which
may be difficult to achieve).
Combined with the TIPA's diverse panel of academic, political, media,
and community experts, the rating system is designed to provide the
most powerful and easy-to-understand profile of each individual advertisement
as well as the overall advertising character of each campaign.
B. RATING SYSTEM DEVELOPMENT.
Developing a Rating System for political
advertising is no easy task. The TIPA’s offering today is little
more than a suggested starting point. Here is how the TIPA is addressing
the challenge.
- MULTIPLE RATINGS. Rather than using a single
rating, the TIPA has elected to divide the ratings into three
categories. The first category addresses the factual accuracy
of the advertisement. The second category rates its fairness.
Obviously, the latter is more subjective than the former. The
third category is relevance and is very subjective. Relevance
can refer to the relationship the ad content has to the candidates,
their respective campaigns, or any other factor. One will expect
supporters of a candidate to always rate their candidate’s
advertisements as relevant so the revealing information will
be the rare occasion when the opposite happens.
- DETAILED DESCRIPTIONS. Rather than simply using
a scale with minor explanations for the numerical ratings,
the TIPA’s approach
is to use a highly detailed description for each number or range.
Also, rather than having a relatively short scale (e.g., 1 to 3),
the TIPA deploys a 10 position scale. A half negative/half positive
scale was considered (e.g., -5 to +5) but was not deployed due to
arithmetic preference for simple averaging as well as the preference
to avoid a potential rating (“0”) which appeared
to grant content neutrality to an advertisement.
- NONPARTISANSHIP. The TIPA knows a more traditional approach seeking
accuracy in a rating system is to exclude the involvement of participants
with prejudice in the actual evaluation segment of a research project.
Normally, one seeks to avoid investigators or participants at similarly
influential levels with prejudices which could affect the way any
study is conducted (as opposed to the subjects of a study, who are
expected to have biases and prejudices -- which often, in fact, are
what is being studied). The TIPA believes the content of political
advertising is so emotionally charged and is so difficult to study
objectively that the best way to guarantee nonpartisan results is
to include absolutely partisan people in the evaluation phases of
the effort.
- PROTECTIVE MECHANISMS. One of the ways the
integrity of the Project will be ensured is that there always
will be one of the following conditions imposed on and occurring
with every analysis: (a) there will be an equal number of partisan
participants on each political side of an evaluation or (b) where
there is not an equal number, opinions will be weighted proportionately
to guarantee a “level
playing field.” The primary protective mechanism, however,
is to balance the sample for each evaluation so there is a “wash” between
those supporting one candidate and the other. A secondary mechanism
will be used to add an additional layer of protection in the
event gamesmanship occurs and individual participant are taking
extreme positions so as to skew group averages. This can be statistically
addressed in a number of ways which any expert in robust analysis
can deploy.
C. THE RATING SYSTEM.
The Truth In Political Advertising Project depends most of all on
the integrity of its special Advisory Panel participants. The Executive
Director, Aaron Harber, personally knows the vast majority of Panel
members and is confident they will evaluate each advertisement as honestly
and as fairly as possible. As a rule, these are extraordinarily honest
people with great integrity who care about the health of the political
process and sincerely want to contribute to its improvement.
By including partisan participants yet seeking
balance, the TIPA’s
Rating System is designed to incorporate -- rather than forego or even
attempt to avoid -- the expertise political operatives, candidates,
party officials, and similarly experienced people have. The TIPA understands
these people, more than any others, pay more attention to political
advertising than anyone else and have a deeper understanding of it.
These people also have grave concerns about the political system in
which they operate and most, if not all, at times sincerely wish the
political battles in which they are engaged were fought fairly and
on principles and about relevant issues, rather than via the pure gamesmanship
which dominates politics today.
The Accuracy Rating is based on each participating
evaluator’s
sense of the factual correctness of an advertisement. The TIPA
will strive to consistently provide information about each advertisement
which addresses the factual statements in the ad and their accuracy
but separately is interested in every participant’s perception
of factual accuracy.
Each evaluator will bring his or her own expertise, knowledge, information
resources, experience, perspective, and sentiments to that evaluation.
The TIPA will use the ratings of each Advisory Panel member and contrast
those ratings with its own assessment of the factual accuracy of each
advertisement. By using this parallel tracking mechanism, the TIPA
will be able to develop its own internal model for determining the
degree of impartiality each Advisory Panel member presents.
The Accuracy Rating is
on a one to ten scale per the following legend:
01 = Grossly inaccurate
with total disregard for the truth; an advertisement which intentionally
misstates facts to definitively create an impression which is the
opposite of the truth and which omits key facts which otherwise would
lead the observer to the opposite conclusion; viewed as truly despicable
by anyone with a conscience or a modicum of decency.
02 = Extremely inaccurate
with “facts” fabricated and/or distorted so as to give
a false impression of the truth; a damning indictment of misleading
advertising.
03 = Generally inaccurate
with a plurality of key facts misstated and/or intentionally omitted,
giving the observer the impression something false is true or vice
versa.
04 = A mix of accurate
and inaccurate statements (e.g., 50- 50) with more than one significant
fact misstated and/or with key, known or well-established facts omitted;
a factually misleading advertisement.
05 = Superficially
accurate (approximately 70% of the time) with one or more significant
facts intentionally misstated or erroneous; clearly not meeting any
kind of minimal standards for overall accuracy but appearing to do
so as part of an effort to mislead the observer.
06 = Nominally accurate
(approximately 80%) but with one to three key facts subject to misinterpretation
or misunderstanding and such misinterpretation or misunderstandings
likely to have been intentional as part of a “smokescreen of
truth” used to pursue a falsified argument.
07 = Generally accurate
(approximately +90%) with one or two key facts subject to misinterpretation
or misunderstanding – and they clearly should have been corrected
before the advertisement was used.
08 = Very accurate
(approximately +95%) with one or two facts unintentionally unclear
or subject to misinterpretation or misunderstanding.
09 = Almost 100%
accurate, with some minor unintentional misinterpretation or misunderstanding
of one or two facts possible but unlikely; an advertisement obviously
created with the pursuit of truth in mind.
10 = Absolutely
and totally accurate from a factual perspective.
The TIPA staff contemplated simpler 1-to-3 and
1-to-5 scales but felt such limited choices would force evaluators
to select options which were more extreme than their true feelings.
By using a 1-to-10 scale, the TIPA allows each participant to more
closely express the degree of his or her true feelings. With the
broader scale, there is more opportunity for nuance and differentiation
as opposed to being forced into a “harder” category which
may not accurately express the true sentiment of the evaluator (e.g.,
where hesitance or uncertainty may exist).
A 1-to-10 scale also gives the TIPA the option
of later combining evaluation categories (e.g., adding the 1’s and 2’s and
comparing them to the 9’s and 10’s) for cross-tabulation
and other analytical purposes. The larger numerical scale also should
be more meaningful arithmetically when group averages are calculated.
Furthermore, by having the explanation for each rating be so well-defined,
Advisory Panel members and public participants can refer to the legend
when performing their ratings for ease of use and for consistency in
their own application of the Rating System as well as in the application
of the ratings by all other participants.
* * * * * * * * * * * * * * *
The Fairness Rating:
01 = So terribly
mean-spirited, evil, despicable, and/or unfair that the candidate
should be ashamed off him- or herself and resign from the election;
this rating is so low that voters supporting the candidate sponsoring
the advertisement or intended to benefit from the advertisement should
cast their vote for his or her opponent; contains multiple, horribly
vicious personal attacks, clearly intended to ruin the personal life
of an opponent or opposed candidate; the ultimate in scummy.
02 = Extremely unfair
and/or inaccurate with “facts” fabricated and/or so extraordinarily
distorted to vilify or otherwise severely harm the opposing candidate;
containing false or highly personal yet irrelevant material which
could unfairly damage the targeted candidate for life; contains one
or more vicious personal attacks; truly reprehensible.
03 = Grossly misleading
and unfair effort to stain the reputation of an opponent, meant to
severely harm him or her.
04 = Scurrilous
personal attack intentionally distorting the truth to give a false
impression of an opponent, yet mixed with enough relevant or reasonable
claims to soften what otherwise would be a brutal attack.
05 = Contains an
unfounded or unjustified personal attack on a candidate which is
patently unfair and which is not fair game.
06 = Hard-hitting
but fair and reasonable; hard-edged and perhaps a bit too tough yet
just within the bounds of what should be considered acceptable.
07 = Very fair;
portrayal of the opponent might be negative but is not exploitative
or unreasonable in any way; shows a contrast which is accurately
and fairly represented.
08 = Extremely well-balanced
and reasonable; does nothing to unfairly portray the opponent in
a negative light but still may use one very relevant negative to
contrast the candidates or ballot issues yet does so in an acceptable
manner.
09 = Bends over
backwards to be fair and kind to an opponent; intentionally portrays
the opponent in a good light; makes no statements which could be
unfairly construed.
10 = Extraordinarily
fair to the point of being magnanimous to the other side; if every
candidate were this generous, negative advertising would not exist;
one might think the opposition developed and sponsored this advertisement.
There is no question that the Fairness Rating is
the most complex of the three components of the TIPA rating System.
Fairness is very subjective and a number of different tests can be
used when considering what is “fair.” Is it “the
right thing to do?” Is it “how we would want to be treated?” (Think
of the Golden Rule as a constant in the evaluation process.) Given
the world of politics and the rules by which it is played, is the advertisement
being evaluated n the bounds of fair play, good sportsmanship and/or
good taste? How would the sponsoring or benefiting candidate or organization
feel if the tables were turned and the advertisement’s approach
were directed back at him, her or them? Would a principled person consider
the advertisement fair? Is the advertisement honest and not deceptive
in any way? These are the kinds of questions Panelists and members
of the public will have to ask each and every time when rating an advertisement.
In addition to reflecting on all of these issues
and variables, each evaluator will need to stay cognizant of his
or her personal biases. These prejudices often work subconsciously
to excuse bad behavior. This is why a partisan person in favor of
a candidate or ballot issue is likely to excuse or not even recognize
behavior or actions which, if they had been taken or deployed by
the opposition, would be seen as unfair, improper, and even scurrilous.
Trying to maintain objectivity during the peak of bias in an emotional
political setting – especially
as Election Day draws near – is the difficult but quite doable
assignment of TIPA’s participants.
Furthermore, it is common for attacks against
ballot issues to be even more mean-spirited and disingenuous. This
is because such attacks are not “personal” in nature although they can be even
more vicious than any other. That is, such attacks typically are not
directed against a particular individual but, instead, are directed
against a more amorphous or difficult-to-identifyt group. The “issue” rather
than a “person” usually is the subject of such attacks
and, as a result, there is greater opportunity to be extreme in such
instances without being held responsible for any personal attack.
* * * * * * * * * * * * * * *
The Relevance Rating is
a corollary measure of fairness and accuracy in that the use of irrelevant
material is indicative of a candidate and/or campaign interested in
misdirecting voters. There is an inherent dishonesty in candidates
from any party or group who seek to focus the public’s attention
on non-issues – transforming
them into issues so as to gain favor and discreetly submerge the true
and key issues of the day below the surface of political waters. The
misdirection play is common in politics and offers great convenience
to those on the “wrong side” of an issue as well as those
who find great discomfort at the prospect of otherwise being forced
to confront difficult issues. This rating also is on a one to ten scale
per the following legend:
01 = So totally irrelevant
that the sponsoring candidate or the person or group for whom the
ad is intended to benefit should be tarred and feathered (figuratively
speaking) for his, her or their attempts to use these issues in the
campaign; the transgression is so egregious that voting for the opponent
of the sponsoring or candidate or organization or other beneficiary
is a must so the target of the irrelevant ads ultimately benefits.
02 = Extremely irrelevant
with issues or “facts” selected, fabricated, deployed
and/or otherwise used in an intentional manner to redirect voters
away from the real issues of the day but done so in a manner which
serves to attack the opposition via distortions, vilification, et
cetera, often with the simultaneous intent to severely harm the opposing
candidate; containing misleading or false material or one or more
vicious personal attacks which are designed to suppress discussion
of what truly are the relevant issues of the day.
03 = Grossly misleading
effort to redirect voters, possibly including an attempt to stain
the reputation of an opponent, meant to severely harm him or her;
scurrilous personal attack intentionally distorting the truth to
give a false impression of an opponent and avoid addressing the issues
voters see as key in a race.
04 = Contains subject
matter which has little to do with the campaign or contest at hand;
obviously created to focus on a subject which casts the sponsoring
or benefiting candidate in a positive light although the subject
matter has little or no relationship to the political race.
05 = A slight majority
of the material is not relevant (e.g., +50%) and, instead, unintentionally
obscures those issues which are on the minds of most voters; this
advertisement may be sloppier in its construction than intentionally
devious (although the ultimate ramification may involve voter confusion).
06 = More relevant
than not (i.e., +65%) but containing a confusing mix of topics, subjects,
and themes (in terms of their relevance) -- some of which apply to
the contest at hand and others which simply do not apply at all (hence
the confusion).
07 = Almost all
(80%) understandably applicable and relevant subject matter and material
which is related to one or more key issues of the day for a plurality
of voters and which is articulated clearly.
08 = Quite relevant
and appropriate (+90%); the subject matter of the advertisement involves
one or more topics which are critically important at this time and
are on the minds of most voters but something minor and irrelevant
may still have been included in the advertisement.
09 = Very relevant
material (i.e., approaching 100% relevance); the subject matter of
the advertisement is right on target and highlights the issues which
clearly are important to most people today; this advertisement “hits
the nail on the head” when it comes to topic selection.
10 = Extremely relevant
(definitely 100%)); could not have selected more appropriate and
fitting topics, information, and claims; this is what this campaign
should be all about; a model for this and other political campaigns;
this candidate or campaign should be lauded and supported for the
willingness to address what truly are the key issues of the day.
Relevance is important but, again, it is so subjective
that differences may be hard to ascertain in the data once core partisanship
is factored out of any analysis. What “fits” a particular situation
in a political campaign at a particular point in that campaign is likely
to be seen differently by opposing partisans. What is “appropriate” or
what truly “applies” to a particular political race or
a specific point of time – i.e., what is truly relevant? -- are
questions that can be honestly and reasonable answered quite differently.
With the above factors in mind, the most interesting finding may
be the degree to which evaluators can or cannot separate their personal
perspectives from an honest evaluation of the true issues of the day
when evaluating relevance. Similarly, examining how their answers correspond
with responses for the other ratings may generate fascinating results. Despite
these complexities, the TIPA Project staff is confident that, because
of the integrity of those involved on the Advisory Panel, they consistently
will make fair, reasonable, accurate, and objective evaluations of
the advertisements put before them.
* * * * * * * * * * * * * * *
An Overall Rating or Score will be considered by
the TIPA. It likely will be weighted so as to emphasize accuracy. In
this manner, the most subjective elements of the Rating System are
somewhat ameliorated. By intentionally using the same 1 (most negative)
to 10 (most positive) scale, the demonstration project Rating System
lends itself to a single scoring system (i.e., this makes the most
logical Overall Rating system compatible with scores in the 1 to 10
range, with similar connotations related to each scoring level). The
initial Overall Rating scoring system would take an average of the
three ratings and incorporate them as follows:
Accuracy Rating: 45% of the Overall Score.
Fairness Rating: 35% of the Overall Score.
Relevance Rating: 20% of the Overall Score.
Other distributions will be considered and may
be a function of how well each component of the Rating Systems works
as the Rating System actually is used. At a 20% weight, for example,
one might argue the Relevance Rating could become irrelevant itself
(i.e., only an extremely positive or negative rating would affect
and Overall Rating or score). With this in mind, would partisan evaluators
then tend to score Relevance in an extreme way (i.e., simply to have
their opinions “register” when
they know those opinions otherwise would not)? The range of weighting
originally and still being considered was and is as follows, with any
final combination (as detailed above) obviously having to add up to
100%:
Accuracy Rating: 33% to 60% of the Overall Score.
Fairness Rating: 33% to 40% of the Overall Score.
Relevance Rating: 10% to 33% of the Overall Score.
The TIPA admits that the development of any Rating System is a significant
challenge. We have been struggling with this for some time and recognize
it is a work-in-progress. Your comments regarding how the Rating System
can be improved as well as complete alternates to the TIPA Rating System
would be warmly welcomed. Your criticism can only make the TIPA Project
more successful in the long run. Please tell us what we are doing wrong,
where our Rating System does not make sense, how to improve and better
differentiate each scoring group, level, and description, and how to
make the Rating System more accurate, more relevant, more understandable,
and easier to use. The TIPA staff sees this initial Rating System only
as a starting point and is depending on your critique to help us improve
it.
The TIPA staff is aware and is concerned that
the structure of the ratings themselves may drive respondents towards
certain answers not just in terms of the aforementioned potential
gamesmanship but due to our own failures to make adequate descriptions
or distinctions. For example, in some cases, the descriptive difference
between two adjacent ratings is minimal. This may result in slight
biases moving whole numbers when any analysis is done (i.e., almost
everyone rates an advertisement a “2” rather than a “3” because
there is no discernible difference or it is too small).
A separate phenomenon based on distinction failures also may be that
campaigns consider the ratings when developing ads and tone down the
negative aspects of some of them. Hence, the less-than-desired variation
in ratings may promote campaigns to change their tactics so as to avoid
a low rating. While the reality of campaigns makes this phenomenon
unlikely, this could become a long-term impact of the TIPA Project.
There remains the separate issue but critically important issue of
accountability and how it is assigned by the electorate. With the plethora
of independent expenditure committees (most notably the approximately
15,000 527 committees which have been registered in the United States),
there already is a great need to address the issue of how voters should
respond electorally when a third party entity inaccurately, unfairly,
and/or irrelevantly attacks the opposition.
When one candidate attacks another in such a manner,
it is easy to hold the attacking candidate responsible simply by
voting against him or her (i.e., and in favor of the candidate suffering
the attack). When a third party makes that attack, however, while
the understandable electoral response initially may be the same,
the additional question becomes, “Is it reasonable for voters
to hold responsible a candidate who may otherwise benefit from the
attack but who did not make the attack?”
This then moves the discussion into the intrigue
of third party participants, their actual relationships with the
campaigns they seek to support, et cetera. It even can involve devious
and complex third party efforts to defame a candidate the third party
supports in an effort to generate sympathetic support for him or
her. These are issues the TIPA and voters ultimately will need to
address although in Colorado, where “clean” campaigning
has been prevalent for decades and where voters tend to punish bad
campaign behavior, these have not yet come to the forefront. In time,
they will (and that time may be short.)
* * * * * * * * * * * * * * *
Members of the TIPA Advisory Panel are asked to simultaneously e-mail
their ratings and comments to Zachary@TIPAP.org and to Aaron@TIPAP.org as
soon as they are notified there is a new advertisement to review or
as soon as they see a new advertisement on the TIPA Web site. Please
note the advertisement being reviewed as well as your three 1-to10
numerical ratings for Accuracy, Fairness, and Relevance. Everyone is
welcome to send in their ratings – including members of the public.
Advisory Panel member ratings will be calculated separately from the
general public and, when appropriate, the two sets of ratings will
be compared. A rating submission should appear as follows. Use the
following format via a cut-and-paste approach or use the form ( click
here) available for Advisory Panel members and members of the public.
Advertisement Title: ________________________________
Accuracy Rating: _________
Fairness Rating: _________
Relevance Rating: _________
Comments (optional):
That’s all there is to it. If you want to
add information such as where and when you saw the advertisement,
that can be included in the Comment section. This would be helpful
as some advertisements are broadcast in a highly targeted and/or
limited manner. While most campaigns have agreed to cooperate with
the TIPA, vigilant supporters of our Project will help us stay current.
Thanks for your help doing this.
If any evaluator – whether an Advisory Panel member or a member
of the general public -- sees an advertisement not on our Web site,
please click here and send an e-mail message notifying us immediately
of the existence of the advertisement. Again, while participating campaigns
have agreed to provide us with every advertisement as soon as it goes
into rotation anywhere in the State of Colorado but it is entirely
possible that, from time to time, one may be missed or just slips through
the cracks – either at a campaign or at the TIPA Project.
Finally, the ratings of the Advisory Panel as well as those
made by individual members of the public always will be kept confidential. Your
name never will be associated with your own rating of any advertisement.
Your responses will be reported only as part of the total set of
responses received. And your e-mail address and any other contact
information will never be shared with any other organization. Anonymity
is crucial to our success and we need you to always have confidence
your individual, identifiable responses will not be disclosed.
Thank you very much for participating in our pilot project and for
helping begin the process of developing mechanisms to bring integrity
to the political process. The TIPA is just the beginning of a long-term
effort and every contribution made of time, expertise, and other assets
is greatly appreciated by the entire TIPA staff. With your help, we
will change politics in America.
