Citation:
Colin Rule, Designing a Global Online Dispute
Resolution System: Lessons Learned from eBay, 13 U. St.
Thomas L.J. 354 (2017)
Content downloaded/printed from HeinOnline
Tue Nov 27 03:15:44 2018
-- Your use of this HeinOnline PDF indicates your
acceptance of HeinOnline's Terms and Conditions
of the license agreement available at
https://heinonline.org/HOL/License
-- The search text of this PDF is generated from
uncorrected OCR text.
-- To obtain permission to use this article beyond the scope
of your HeinOnline license, please use:
Copyright Information
Use QR Code reader to send PDF
to your smartphone or tablet device
ARTICLE
DESIGNING A GLOBAL ONLINE DISPUTE
RESOLUTION SYSTEM:
LESSONS LEARNED FROM EBAY
COLIN RULE*
I. INTRODUCTION
If you have spent much time surfing the web, you've probably spent
time on eBay. After it was founded in a basement in September 1995, eBay
quickly grew into one of the internet's biggest success stories.' Over the
past twenty years, eBay has become the largest eCommerce marketplace in
the world, with local sites in more than twenty-five countries.2
eBay now
sells more than $80 billion worth of items per year, and at any given moment
there are more than 800 million items for sale within the eBay Mar-
ketplace.3
eBay users buy and sell every kind of item imaginable: a pair of
shoes every two seconds, a cell phone every four seconds, and a car or truck
every four minutes.4 eBay users conduct more than 250 million searches
every day,5 making the daily volume on eBay greater than the daily volume
on the NASDAQ.' As you might imagine, all of these transactions generate
a lot of disputes. Even though only about 1 percent of purchases on eBay
generate a problem, the incredible volume on the site means eBay has to
handle more than 60 million disputes a year.'
* Co-founder, COO, and Chairman of the Board for Modria.com, the world's leading online
dispute resolution platform.
1. Pierre Omidyar, How PierreOmidyar TurnedAn IdealisticNotion Into Billions Of Dollars,
INC. MAGAZINE, http://www.inc.com/magazine/201312/pierre-omidyar/ebay-inspirationmore-effective-than-delegation.html
(last visited June 15, 2016).
2. See Craig Smith, By the Numbers: 37 Amazing eBay Statistics, DMR, http://expandedramblings.com/index.php/ebay-stats/
(click Next to go to Page 2 of 3) (last visited Mar. 3,
2016).
3. Id. at 1.
4. Id. at 3; 20 Fast FactsAbout eBary Worldwide, EBAY.COM (Aug. 28, 2015), http://www
.ebay.com/gds/20-Fast-Facts-About-eBay-Worldwide-/10000000205665282/g.html.
5. Id. at 1.
6. Daily Market Summary, NASDAQTRADER, http://www.nasdaqtrader.com/Trader.aspx?id
=DailyMarketSummary (last visited June 15, 2016).
7. eBay-style online courtscould resolve smallerclaims, BBC NEWS (Feb. 16, 2015), http://
www.bbc.com/news/uk-31483099.
354
LEARNING LESSONS FROM EBAY
This volume of disputes puts eBay at the forefront of the development
of the field of online dispute resolution (ODR). In fact, The Perfect Store
(Adam Cohen's book about the early days of eBay) describes how dispute
resolution was a part of eBay in the first months after Pierre Omidyar
launched the site. As Cohen explains, eBay's first customer support employee
"spent a lot of time doing what Omidyar hated: stepping in and
trying to resolve disputes."
In 1999, Professor Ethan Katsh launched a pilot program with eBay to
resolve disputes between buyers and sellers. After a link was put on a relatively
obscure eBay help page urging people to report issues to Katsh's
Online Ombuds Center at the University of Massachusetts Amherst, the
Center was flooded with cases. 9
That pilot program evolved into the startup
company SquareTrade.com, which became the internet's most successful
online mediation service."o Over the next eight years, SquareTrade
mediators would resolve several million eBay disputes over Feedback
(eBay's reputation management system in which buyers and sellers rate the
performance of their trading partners). But there were many other nonFeedback-related
disputes that SquareTrade could not address as an independent,
third-party service provider outside of the eBay network. As a result,
eBay made the decision to bring ODR in-house in 2003, and that is
when I became eBay's first full-time employee devoted to dispute
resolution.
In this article, I will share some of the lessons I learned in building
eBay's ODR platform. I begin by presenting the common characteristics of
eBay disputes, and then analyzing the design we devised to resolve them.
This design is then abstracted to a broader framework for ODR. We then
examine several implementation issues we encountered in deploying the design,
including: integration with payment systems, buyer and seller perspectives,
and the relationship between reputation and resolutions. We then
conclude with some lessons learned that may be helpful for future ODR
systems designers, both inside and outside of eCommerce.
II. CHARACTERISTICS OF EBAY DISPUTES
When I arrived at eBay, I presumed that all eBay disputes pretty much
looked the same. I also thought that eBay was well-positioned to resolve
disputes, because it was neither the buyer nor the seller in each transaction.
As a marketplace administrator with no stake in the outcome of the dispute
8. ADAM COHEN, THE PERFECT STORE: INSIDE EBAY 36 (2002).
9. Ethan Katsh et al., E-Commerce, E-Disputes, and E-Dispute Resolution: In the Shadow
of "eBay Law", 15 OHIo ST. J. Disp. RESOL. 705, 709 (2000).
10. See Steve Abernathy, Building Large-ScaleOnline Dispute Resolution & Trustmark Systems,
PROCEEDINGS OF THE UNECE FORUM ON ODR 2003, http://www.mediate.com/Integrating/
docs/Abernethy.pdf (last visited June 10, 2016).
2017] 355
UNIVERSITY OF ST. THOMAS LAW JOURNAL
other than a desire to maintain trust in the marketplace, I reasoned that eBay
could credibly play the role of mediator.
I learned quickly that the truth was much more complicated. In reality,
eBay has a wide variety of disputes, and each is fraught with its own specific
complexities. Some disputes are initiated by sellers (such as payment
disputes), while others are initiated by buyers (such as item quality disputes).
Some focus on reputation (Feedback disputes) and others on intellectual
property (owner rights disputes).
Most of these disputes are not over very large amounts of money. They
can be for as little as $5, such as a magazine, or as much as $50,000, such
as a car, but the average is $100." (However, as dispute resolvers know
well, dollar amount is usually not an accurate barometer of passion among
disputants.) And because eBay users are spread all over the world, eBay
disputes can involve cultural misunderstandings, language barriers, and
class differences. A single eBay purchase may involve a buyer in Australia,
a seller in France, and a drop-shipper in China, all transacting on a USbased
website that refers to California law in its Terms and Conditions.
This can lead to many possible points of confusion.
I also learned that eBay has a very personal relationship with each
user. eBay decides what fees will be charged, how listings should be regulated,
and when accounts should be limited or suspended. eBay's decisions
about which listings to promote, either in search results or in marketing
materials, can result in huge swings in sales (and profits) for sellers. As
such, eBay is rarely seen as an impartial figure. 12
The first challenge to tackle in designing the ODR system was the
overwhelming volume of cases. Depending on how you count, the number
of annual disputes within eBay's marketplace exceeds the volume of the
U.S. civil court system.13
With tens of millions of disputes, the math was
obvious: even if eBay had built a staff of 1,000 skilled online mediators, we
would not be able to get through the torrent of cases. We called it "drinking
from the fire hose." It was self-evident that the process needed to be as
automated as possible.
Another major issue that had to be addressed was the culture around
conflict on eBay. The way the eBay community approached transaction
problems was a big part of that challenge. First of all, almost no one used
the word "dispute." The language instead was very legalistic: case, complaint,
or claim. If a buyer did not pay, that user was often referred to as a
11. Notes and Analysis by Colin Rule of eBay's Internal Data on Global Dispute Volume
(Mar. 2010).
12. See Maz Iqbal, eBay: biassed, incompetent, indiferent or all three?, THE CUSTOMER
&
LEADERSHIP BLOG (Sept. 11, 2011), https://thecustomerblog.co.uk/tag/ebay-bias-against-sellers/
(last visited June 13, 2016).
13. See generally U.S. DEP'T. OF JUSTICE, CIVIL JUSTICE SURVEY OF STATE COURTS 2005,
http://doi.org/10.3886/ICPSR23862.v2.
356 [Vol. 13:2
LEARNING LESSONS FROM EBAY
"non-paying bidder" or "deadbeat buyer." If a seller did not deliver an item,
the only option was for the buyer to file a "fraud alert." Obviously, this
language made it very difficult to follow Roger Fisher and William Ury's
well-known admonition to "separate people from the problem." 4
III. THE FORMATIVE PROCESS
It was readily apparent that we would need to forge a new way to
describe these situations so as to optimize the chance that they would be
resolved. In this section, we examine the design process we utilized to build
a system to better handle these disputes, as well as presenting a broader
framework for ODR systems that extends the overall design we employed
to other ODR volumes.
A. Designing an ODR System for eBay
Ethan Katsh and Janet Rifkin's book Online Dispute Resolution had
introduced the concept of the fourth party, in which technology could play a
dispute resolution role separate and apart from the human third party.15
In
line with this concept, we decided to write a software program to assist the
parties in resolving their disputes, and to involve human neutrals only on an
exceptional basis.
How to do it was the key question. Some things that are particularly
powerful offline (e.g., apologies) are much less effective online, due to the
specific characteristics of computer-mediated communications. However,
technology can also open new opportunities that are impossible when working
face-to-face. For instance, in-person dispute resolution services are usually
sought only after a dispute has reached a point of impasse. Most
mediators and arbitrators actually have little opportunity to shape the perspectives
of the parties at the earliest stages of the issue.
On eBay, however, we were in direct communication with the buyer
and seller from the very beginning-from the purchase all the way through
to resolution. That enabled us to work with the parties even before they
really understood whether they had a problem. For instance, if a buyer
made a purchase and the item had not arrived within three or four days, he
or she might start to get concerned. eBay could then step in to reassure the
buyer that the average delivery time for a package going from the seller's
location to his or her home was seven days. Then the buyer's anxiety would
ease, and the package would later arrive on schedule. In a sense, we were
able to resolve the issue before it even became a dispute.
14. ROGER FISHER & WILLIAM URY, GETTING TO YES: NEGOTIATING AGREEMENT WITHOUT
GIVING IN 10 (2d. ed. 1983).
15. See generallyETHAN KATSH & JANET RIFKIN, ONLINE DISPUTE RESOLUTION: RESOLVING
CONFLICTS IN CYBERSPACE (Jossey-Bass 2001).
2017] 357
UNIVERSITY OF ST. THOMAS LAW JOURNAL
An upside existed to the incredible volume of cases coming through
the eBay platform: we understood a lot about what these disputes looked
like. For example, we knew that most item-related disputes fell into one of
two categories: non-receipt (the buyer paid but never got anything) or notas-described
(the item arrived, but it was different than what the buyer expected).
Then, within these dispute types we could again specialize to
pinpoint the problem: How did the seller ship the item? Was shipping insurance
purchased? In the case of a not-as-described item, was the item broken?
Was it counterfeit? Was the difference a small one (e.g., wrong color),
or a big one (e.g., the buyer bought a new laptop and got an old, broken one
instead)?
We also renamed the processes on the site to promote resolution.
"Fraud alerts" became "Item Not Received" disputes. We built a "Dispute
Console" so that users could see all their transaction problems in one place,
communicate easily with their transaction partners, and track them to resolution.
Soon after, we started to see community members using this language
to talk about their transaction problems in the discussion forums.
Several years later the Dispute Console turned into the Resolution Center. 1
This new language helped to shape the way users thought about their transaction
problems.
B. Building a Frameworkfor ODR
In designing the eBay dispute resolution system it became clear that
the process would have to provide a resolution for every issue reported. A
purely facilitative model that left the outcome up to the parties would generate
a lot of frustration, and it would leave many of the toughest cases
unresolved. Also, some parties had an interest in not reaching an agreement
(for instance, a seller in the case of a not-as-described dispute), so there was
a strong incentive for those users to stonewall or not negotiate in good faith.
We decided to take a staircase approach, beginning with problem diagnosis
(where we worked with the complainant to identify the root of the
problem and likely solutions), then escalating to direct negotiation assisted
by technology, and finally moving to an evaluation phase where eBay and
its payment system provider, PayPal, would decide the case if the transaction
partners could not do so. Each party could decide unilaterally when it
wanted the process to move on to the next phase. The goal of the system
was to prevent as many disputes as possible, amicably resolve as many as
possible, and then decide the remainder as quickly and fairly as possible. As
you can see in Figure A, each stage acted like a filter, with the objective
being to minimize the flow of cases that made it to the end.
16. Resolution Center, EBAY.COM, http://resolutioncenter.ebay.com/ (last visited June 15,
2016).
358 [Vol. 13:2
LEARNING LESSONS FROM EBAY
Figure A: ODR Filters
It was very clear that eBay users did not want to spend a lot of time in
transformative processes intended to build a long-term trustworthy relationship.
Most eBay transactions were between strangers, and most buyers did
not buy more than one item from any individual seller, so there was little
interest in approaches that did not match this purely transactional orientation.
What users wanted was communication, transparency, efficiency, and
a fair outcome, in as little time as possible.
We discussed filing fees for our dispute processes, but it is difficult to
convince a disputant to pay $30 to resolve a $50 dispute, and near impossible
to convince him or her to pay $50 to resolve a $50 dispute. The disputant
might as well just give the money to the other side and save the time
associated with the process. Moving online dispute resolution inside eBay
was a much more natural way to address the funding problem. Instead of
thinking about the costs on a per-case basis, as one does with a third-party
provider, eBay evaluated the cost for the system on a revenue enablement
basis, which made much more economic sense. eBay's internal economic
analysis has shown that the savings from reduced contacts with customer
service, retained activity from users, and increased transaction activity more
than justify the investments in ODR.
From the beginning, we designed eBay's ODR processes to be learning
systems. There was no shortage of data available; eBay has total visibility
into each user's usage patterns, history, and account data. Also, eBay
routinely surveys users to gauge their satisfaction with the resolution
processes they have used. eBay uses these data to monitor the performance
of the resolution systems and to improve them as marketplace conditions
change.
2017] 359
UNIVERSITY OF ST. THOMAS LAW JOURNAL
IV. ISSUES IN IMPLEMENTATION
Having an overall design and framework for our solution was one
thing, but actual implementation was a different challenge altogether.
Many issues arose during the launch of our new ODR systems, and we had
to evolve our approach to match the specific needs of individual dispute
types. In this section, we examine some of the specific issue types we
needed to address, including payment disputes, buyer vs. seller disputes,
and independent feedback review.
A. Payment Disputes
In 2003, the dispute type that was generating the most frustration was
Unpaid Item disputes. The issue of consumers bidding on an item, winning
the auction, and not following through to pay was causing great consternation
in the marketplace, particularly with sellers. The system in place to
deal with those issues was very manual, and sellers were quite upset at what
they saw as eBay taking money from them for no reason, because buyers
never followed through and paid. At the 2004 eBay Live conference in New
Orleans, there were sessions with hundreds of angry sellers expressing great
aggravation with the existing processes.
On eBay, sellers pay two times when they sell an item. They pay a
small insertion fee to list the item, and then a small "Final Value Fee" when
the item is purchased." The Final Value Fee is based on the final sales
price. So if a seller lists a Ferrari on eBay and a teenager bids on the car as a
joke with a very high number, when the auction closes the seller thinks the
item is sold, even though the buyer will never follow through and pay. eBay
charges the seller the Final Value Fee as soon as the item closes. The seller
then has to wait for the buyer to follow through and pay. If the buyer
doesn't pay, the seller is out the Final Value Fee.1
The name used in the community for this process was the "deadbeat
bidder" process. In official documents, it was called the "Non-Paying Bidder"
process. The language used to describe the process went a long way
toward defining how users thought of it. Our first action was to change the
name of this process, because it is not wise to use a name that indicates who
is at fault (in this case, the bidder) and possibly even insults them (in the
case of "deadbeat"). There are legitimate reasons for a buyer not to follow
through and pay for an item they committed to buy on eBay (e.g., suppose a
seller changed the shipping price after the auction closed, or refused to include
a component that was advertised in the original listing). In these
17. Standardselling fees, EBAY.COM, http://pages.ebay.com/help/sell/fees.html (last visited
June 10, 2016).
18. What to do when a buyer doesn't pay, EBAY.COM, http://pages.ebay.con/help/sell/un
paid-items.html (last visited June 10, 2016).
360 [Vol. 13:2
LEARNING LESSONS FROM EBAY
cases, the buyer shouldn't actually follow through and pay. But the name of
the process still insulted them for doing the right thing.
Sellers were aggrieved because they felt that eBay should not be profiting
off of transactions that weren't completed.19
They felt the process to
receive reimbursements for Final Value Fees was overly onerous, with lots
of hidden deadlines which maximized the chance that sellers would miss
their filing windows and lose their chance to get a refund. They also felt
that they should not receive feedback from buyers who did not follow
through and pay for items they had purchased. The seller rationale was, if
the buyer didn't complete the transaction by paying, they should not have
the right to leave a public comment on the seller's profile.
We made the decision to rename the process from "deadbeat bidder" to
"Unpaid Item," or "UPI". While this language change may seem relatively
trivial, translating a new name into sixteen languages around the world and
updating thousands of help pages is no easy task. We felt the effort was
worth it because the language shaped the way buyers and sellers thought
about the process. "Deadbeat bidder" assigned blame for the issue in the
very name of the process, while at the same time insulting one of the parties.
"Unpaid Items" can be resolved-just get the item paid for, or cancel
the transaction by mutual agreement.
The UPI process was designed from inception to be technology only.
There are no human third parties that work with the buyer and seller to
address the issue. The only human involvement in the process is when a
buyer appeals a UPI strike. The buyer can complete a form explaining why
they think they received the strike in error. What the buyer doesn't know is
that all first appeals from buyers are granted automatically, so only the second
appeal is actually reviewed by a person. And because 98 percent of UPI
strike appeals are from buyers who are appealing their first strike, an automated
volume of millions of cases generates only a few thousand manual
appeals per year.2 0
Originally, any buyer who received three strikes was thrown off of the
system. However, some high-volume buyers (such as professional buyers)
said that this arbitrary number was unfair, because they engaged in so many
more transactions and it created jeopardy for them in their account. So
eventually the policy was changed so that buyers were thrown off of eBay if
they received too many UPI strikes in too short of a period of time.
There was a lot of discussion within the team about the use of the word
"dispute." I felt it was very important to use that word, for several reasons:
(a) it implied that the issue was resolvable, (b) it put the responsibility for
that resolution on the parties, and (c) it communicated a certain seriousness
19. See Criticism of eBay, WIKIPEDIA, https://en.wikipedia.org/wiki/Criticism of eBay (last
visited June 10, 2016).
20. Notes by Colin Rule on Internal Data Analysis Reports Conducted by eBay (2007-2010).
2017] 361
UNIVERSITY OF ST. THOMAS LAW JOURNAL
to the matter. Some argued that the word should be more neutral, like "report
a problem" or "send a communication." But the difficulty with those
terms is that they didn't imbue the users with any responsibility around the
resolution process. Users might be tempted to use a "send a communication"
process just to chat with each other about the item purchased, or to
discuss celebrity headlines.
We did a lot of testing of the platform with outsiders brought in to use
prototypes. We would all sit in a room behind a one-way mirror, and the
outsider would come in and sit down at a computer. The tester would then
walk them through a series of prototypes where they were asked to complete
certain tasks. They were asked to narrate their thinking at each step of
the process. We could also see a duplicate of their screen in our room,
watching where their mouse pointer wandered as they pondered each question
put to them. They probably knew they were being watched, but they
didn't know there were fifteen people behind the glass, eagerly rooting for
them to click on the right link. We even recorded tapes of the screens, the
narration, and the facial expressions of the users, so we could pore over
them later to identify every possible area of confusion.2 1
In these experiments, it was quite clear that there was resistance to
some of the language of the dispute resolution field. Any time we used a
word like "dispute," users would indicate that they felt it was overly negative.2
2
Also, there was great confusion about common terms in dispute resolution.
Users frequently confused "mediation" with "meditation," and when
asked what thought first came to mind when they saw "resolution," users
would often say New Year's Day.23
The Unpaid Item process had a fairly simple flow. The seller would
come to the Dispute Console and report an item as unpaid by entering in the
item number. The buyer would then be notified of the new case and was
given seven days to respond. When the buyer responded to the dispute, they
had several response options:
1. I have already paid for this item.
2. I would like to pay for this item now.
3. I do not want to pay for this item.2 4
The seller then had the ability to respond to the buyer. The seller had
the ability to end the discussion at any point and give the buyer an Unpaid
Item strike. If the buyer received too many Unpaid Item strikes in too short
of a period of time, then their account would be automatically suspended.2 5
21. Id.
22. Id.
23. Id.
24. What to do when a buyer doesn't pay, supra note 18.
25. Id.
362 [Vol. 13:2
LEARNING LESSONS FROM EBAY
Buyers and sellers could also cancel the transaction by mutual agreement.
If the seller then indicated that they were willing to release the buyer
from their obligation to purchase the item, then the purchase would be cancelled
in the eBay system and the seller would receive a refund of their
listing fees.
One issue we had to wrestle with constantly was gameability. Because
we had millions of users working through our process, there were plenty of
attempts from individuals to exploit the system. As soon as a system was
launched there were users who would test it out from all angles, looking to
see if there was a way to exploit it. One example was false transaction
cancellations. Merchants would successfully sell an item to a buyer and
make the profit, but then tell eBay that the transaction had been cancelled in
order to get their listing fee refunded. We had to find a way to determine if
the transaction had been actually cancelled, or if it was a seller trying to
game the system. We achieved this by requiring the buyer to confirm that
they actually had agreed to cancel the transaction. If the buyer did not confirm,
then we knew the seller might be trying to game the system to make
more money.
B. Buyer Complaints vs. Seller Complaints
One of the benefits of online communication is that eBay captures all
the information generated by the disputes coming through the process, creating
a very valuable warehouse of data that can be used to improve the
resolution process. One of the conclusions we drew from that data was that
the first message posted in a thread really set the tone for the conversation
from there on out. If the first message had a negative tone then the tone of
the overall thread was usually negative.
The challenge is that the buyer is the complainant in item-related disputes,
and they're usually the frustrated or angry party. They have little
incentive to be reasonable or measured because they don't have the same
stake as sellers in avoiding negative feedback. If they are given an open text
box during the filing process where they can express themselves, they are
more likely to post angry or negative sentiments, which can run the conversation
immediately off the rails.
As a result, we structured our processes so that the first post from the
buyer was intermediated by technology. The buyer filled out a series of
very comprehensive forms where they picked the reason for the dispute.
From these selections we could compose a first post for them that accurately
described the nature of their complaint, but avoided any threats or
insults. The buyer was not given an open text box to explain their situation
2017] 363
UNIVERSITY OF ST. THOMAS LAW JOURNAL
because of the risk that they would use that opportunity to attack the
seller.26
The seller, on the other hand, had great incentives to resolve the case
amicably. If a seller gets negative feedback, the willingness of future buyers
to make purchases from them might be reduced. The seller already has the
ideal outcome for the transaction: they have the money, and the buyer has
the item. If the payment is reversed then the seller either: (a) loses money,
or (b) wastes time. As such, we gave sellers the first opportunity to post an
open message, because they were much more likely to set a positive tone.
This increased the likelihood of an amicable and mutually-agreed-upon
resolution.
It was the rare message thread that started out negative and then had
the tone turn around in the middle to become constructive. Usually what
happened in the threads that turned around was that a seller or buyer specifically
requested a reset in the tone of the communication. Maybe the item
arrived in the midst of the conversation and the buyer admitted that they
had overreacted, or the seller discovered they had shipped the wrong item in
error, apologized, and took full responsibility. Creating an overall tone for
the resolution process that promoted this kind of accountability, empathy,
and reasonableness was also important in keeping communications productive
and solution-focused.
C. Independent Feedback Review
In 2005, the eBay Motors division approached the dispute resolution
team with a complaint from some of their highest volume sellers. These
sellers felt that they were overly vulnerable to feedback extortion from buyers.
The reason for this was the asymmetrical valuation of feedback between
buyers and sellers of cars.2 7
Vehicle sellers sold relatively few items on the site per month-if they
sold ten cars that was a lot. So if they got a negative review on their profile,
it would be visible on the front page of their feedback for a long time. And
sellers knew that a prominent negative feedback could make buyers look
elsewhere to buy a car, which might cost them thousands of dollars in lost
opportunities.
Buyers, on the other hand, probably only bought a car once every five
or six years. And while they were shopping for that car, they were also
buying lots of other types of items, many of which were low value. So if
they got one negative feedback it was no big deal, both because they
26. See Security & Resolution Center, FrequentlyAsked Questions- Item Not ReceivedProcess,
EBAY.COM, http://www.onlineresolution.com/inr/ (last visited June 10, 2016) (copy of
cached page on file with author).
27. See Independent Feedback review, EBAY.COM, http://pages.ebay.com/help/feedback/feed
back-review.html (last visited June 10, 2016).
364 [Vol. 13:2
LEARNING LESSONS FROM EBAY
weren't selling any items so there were no buyers to scare away, but also
because they could make a lot of low-value purchases to make the negative
feedback quickly move off of their front page of results.
Sellers understood this dynamic and it made them feel exposed. Occasionally
they would sell a car to a buyer and the buyer would admit the car
was as described, but the buyer would still threaten to give the seller a
negative feedback unless the seller provided some additional piece of value
that was not included in the original listing. Sellers then made a calculation:
they didn't want to give in to the extortion, but they knew the negative
feedback would cost them more than a thousand dollars in lost sales. So
more often than not they begrudgingly gave the buyer what they were asking
for.
Seller complaints about this inequity reached the eBay Motors team,
who wanted to find a solution. They came to the ODR team to ask if we
could build a dedicated resolution process for feedback in Motors. Motors
was unique, they argued, because of these high value transactions. They
wanted to build a resolution system that would enable sellers to get unfair
feedbacks removed, and combat the problem of feedback extortion.
This situation was different than the challenges we faced with Unpaid
Items and item-related disputes. In these new cases, the buyer had nothing
to gain in the dispute and everything to lose. What would be their incentive
to participate if there was no upside? Also, these decisions were more evaluative
than facilitative, and they required the third party neutral to have
substantive knowledge of both the details of the transaction and the specifics
of auto repair and maintenance.
In designing the system, we based the process on the Internet Corporation
for Assigned Names and Numbers's "Uniform Domain Name Dispute
Resolution Protocol" ("UDRP"). ICANN's design for the UDRP was
adopted in 1999. It had resolved domain name disputes very successfully
for several years, so it was a natural basis for our design. However, there
were some well-documented problems with the UDRP that we felt we
would need to address.28
One big problem with the UDRP was the issue of forum shopping.
Complainants picked which provider would hear the case, so all the case
volumes went to the UDRP provider who was perceived to be the most
complainant-favorable. Providers that had "fairer" (in quotes because fairness
is somewhat in the eye of the beholder) outcome percentages simply
28. See, e.g., Kieren McCarthy, Why ICANN's domain dispute rules areflawed: PartI, THE
REGISTER (July 11, 2001), http://www.theregister.co.uk/2001/07/11/why-icanns-domain-dispute
rules/.
2017] 365
UNIVERSITY OF ST. THOMAS LAW JOURNAL
didn't get the cases, because complainants wanted to maximize the chance
that they would win.29
The eBay Motors resolution process was titled Independent Feedback
Review ("IFR"). Instead of one provider, we expanded the process to have
three providers: the Better Business Bureau, NetNeutrals.com (a program of
the dispute resolution firm DeMars Associates), and SquareTrade.com. We
designed the flow so that the complainant (usually a seller, but both buyers
and sellers could make use of the system) would come to eBay to file the
case. eBay would then refer the case out to one of the providers on a roundrobin
basis. We envisioned that this round-robin assignment would eliminate
the problem of forum shopping, and it would keep all of the providers
honest because they wouldn't want to have resolution rates significantly out
of line with the other two providers.
We worked jointly with eBay Motors and the providers to craft a set of
decision guidelines to shape participant expectations and guide the decisions
of neutrals under the program. Each evaluator would use these criteria
to determine whether or not a specific piece of Feedback should be "descored."
After much discussion and wordsmithing, we came up with the
following guidelines:
1. The member leaving the Feedback didn't show a good faith effort
to complete the transaction. Failure to pay for or sell an item by
itself isn't sufficient to meet this guideline.
2. The member didn't submit the Feedback in a reasonable amount
of time.
3. There is clear and convincing evidence that the transaction-related
information contained in the Feedback comment is factually inaccurate,
even if this was the result of a misunderstanding or an honest
mistake.
4. The member leaving the Feedback made an attempt to extract excessive
value from the other party.30
The ICANN process was precedent-based, so the prior decisions
served to guide the decisions of evaluators in new cases. We didn't want to
deal with the overhead of a precedent-based system, with all the necessary
document searches and debates about particular cases. We used these very
specific decision guidelines to serve the same purpose. If the decision criteria
were detailed enough, and well understood across the evaluators, we
imagined that the decisions would have the same kind of consistency that
could come from precedent-based systems.
29. See Pamela Segal, Attempts to Solve the UDRP's Trademark Holder Bias: A Problem
that Remains Unsolved Despite the Introduction of New Top Level Domain Names, 3 CARDOZO
ONLINE J. CONFLICT RESOL. 1 (2001).
30. Independent Feedback review, supra note 27.
366 [Vol. 13:2
LEARNING LESSONS FROM EBAY
We also were worried about abuse of the system by bad sellers who
would file over and over again to get their records cleaned. So we put several
mechanisms in place to combat this gaming. First, feedbacks found not
to be justified under the IFR program were not removed from eBay. The
negative was "de-scored" so that it didn't count in a seller's overall Feedback
score, and the Feedback was annotated with a line after the comment
explaining that an independent evaluator determined that the Feedback in
question was not warranted. This meant that even if a seller succeeded in
their IFR appeal, the history of the exchange was still visible, so if they
abused the process a buyer could read between the lines and see how many
times the seller had used the process to cleanse their record. Second, we
charged a relatively high filing fee for IFR cases: $130 per filing.3 1
This
created a financial reason for sellers to pick and choose which cases they
would file against. This also enabled the process to be self-funding, so eBay
never had to pay a dime to support it.
Both the Independent Feedback Review process and the Unpaid Item
process are still running on the eBay platform, now ten years later, and
while they have received small updates over the years, the core systems
design of both flows have not been substantially altered.
D. LearningSystems
Because we had so much volume coming into the system, we were
rarely surprised by a new dispute. We had seen pretty much every kind of
dispute before. However, there were some circumstances where we noticed
patterns in the disputes being filed, and those patterns identified needed
changes in the upstream processes.
One example was return-related disputes. We had a large number of
cases where the buyer and seller disagreed over who should pay for shipping
if an item needed to be returned. Maybe the buyer said they wanted to
return the item after the seller's specified return window. Maybe the seller
charged a restocking fee, or refunded the buyer in store credit instead of
cash. Or perhaps (as was most often the case) the seller insisted that the
buyer pay for return shipping, and the buyer wasn't happy about it. When a
buyer received an item that they thought was inaccurately described in the
listing, they felt that they should be able to return it and get all their money
back. Sellers usually disagreed because they were out the shipping price on
the original item purchase, and they weren't excited about having to pay
shipping again.
We realized that the best way to tackle these disputes was not after
they arose, but upstream in the process. So we took our learnings and went
to the "Sell Your Item" team, who owned the form that sellers used to list
31. Frequently Asked Questions (FAQs), NETNEUTRALS, https://www.netneutrals.com/
FAQs.aspx (last visited June 10, 2016).
2017] 367
UNIVERSITY OF ST. THOMAS LAW JOURNAL
their items for sale on the site. Working with that team, we put in a listing
feature that captured return information in a much more detailed fashion.
Sellers could specify their return window, how they delivered returns, and
whether or not they would cover return shipping. We also raised the profile
of this information in the listing view, so buyers would be fully aware when
they were contemplating a purchase. As a result, we saw a big drop off in
the volume of dispute filings related to returns-almost a 30 percent reduction
in the quarter after the new feature went live.3 2
V. CONCLUSION / LESSONS LEARNED
Now that the systems we built at eBay have processed hundreds of
millions of disputes, we've learned a lot about how ODR systems can be
best designed to deliver fast and fair resolutions. The bottom line is the
users want the process to be simple to use, fair to all participants, and easy
to understand.
The main observation is that users just want their resolution processes
to be quick and easy. They want to be able to work out problems with
minimal effort. Redress systems need to be simple to find and intuitive to
complete. Users don't expect to get incentives or giveaways as part of their
resolution; they just want to get a fair outcome so they can put the issue to
bed and move on. And they want to know that the policies that guided their
outcome will be applied consistently and fairly across other disputes as
well.
We also learned that, just like in offline dispute systems design, it's
very important to pay attention to power differentials. Because sellers are
repeat players (and because they have more resources), they have a systematic
advantage over buyers, and it is important to design systems that compensate
for that asymmetry. Part of tackling this challenge is thinking hard
about how your system may be gamed by users trying to find shortcuts or
advantages, and finding ways to address each identified vulnerability.
Also, tone matters. Language shapes the way we see the world, and it
shapes the way we think about resolutions. It may seem like the name of
your process isn't as important as the steps within it, but that perspective is
short sighted. If your language promotes empathy and reason, then that's a
big step toward encouraging resolutions. If your language warns users to
beware because fraud and deception is running rampant, it will provide an
additional obstacle to helping users reach agreements.
Many times at eBay we observed that we were essentially building a
civil justice system for an online country (alongside, it must be noted, the
fraud investigations team, who were building a parallel criminal justice system
at the same time). If you counted users as citizens, in 2006 eBay would
32. Notes and Analysis by Colin Rule of eBay's Internal Data (2009-2010).
368 [Vol. 13:2
LEARNING LESSONS FROM EBAY
have been the fifth largest country in the world.33
As one of the first large
internet marketplaces, we were wrestling with new design questions around
how an online justice system might work. We didn't always get the right
answer, but it is easy to see the lessons we learned reflected in the policies
of newer companies like Airbnb34
(really, the eBay of travel) or Uber35
or
TaskRabbit.36
Our society is becoming more digital every day, and face-to-face civil
justice systems are now confronting some of the same challenges we faced
on eBay. The lessons we learned on eBay may offer guidance for how justice
systems can evolve over the coming decades. Citizens now live and
work online, and they bring expectations drawn from sites like eBay (and
Google and Facebook) to their interactions with government and the courts.
In fact, it may turn out that the justice systems of the future will resemble
the designs we crafted for eBay more than the geographically-bound systems
of today.
33. See Population2006 country rankings, PHOTIUS.coM, http://www.photius.com/rankings/
population/population_2006_0.html (last visited June 10, 2016).
34. AIRBNB, http://www.airbnb.com (last visited June 10, 2016).
35. UBER, http://www.uber.com (last visited June 10, 2016).
36. TASKRABBIT, http://www.taskrabbit.com (last visited June 10, 2016).
2017] 369