A question framework for global content moderation
I. Welcome back & follow up!
Good Morning and thank you again for inviting me into your inbox. I write about insights and provocations on the evolving nature of digital design. You can always subscribe here.
Welcome to a special Friday edition!
Follow up: As mentioned last week, I received so many comments on the previous newsletter on content moderation that this week will be dedicated entirely to the topic.
We’re still on lockdown here in The Netherlands, therefore this column will be a little longer than usual. I’m still determining my own stance in how each of these is solved. This is only a start and no doubt this will be a future newsletter.
Let’s get started!
II.A question framework for content moderation
Introduction
We’re no stranger that over the past several years the discussion of content moderation on social media platforms has grown exponentially. This has been escalated even quicker by the recent protests and riots at the US Capitol.
Like most, I do support freedom of speech, freedom of press, and embrace the “5th estate” - the idea that the individual voice can now have a platform. I support digital platforms for debate, discussion, and open dialogue. Also like most, I do not support hate speech, bullying, abuse, various illegal forms of content, or factually incorrect content (e.g. “fake news”). While these views are mostly universal, it’s the details that makes moderation wicked.
Let’s also clarify that the good and the bad from providing a platform for individual voices are mutually exclusive. While the ability to freely post content on platforms has led to horrible consequences, it has also provided many positive outcomes. These are systems, and like any system, proper and appropriate design must serve as an underpinning to preserve integrity and build the product we all want.
To solve this, we must separate, and clearly focus on, the areas we want to improve - not simply denounce the entire service. We can focus on how we stop illegal, abusive, violent, and false information in various forms, while also assisting in building positive communities, connections, and constructive debates.
Content moderation is a design problem we are only starting to tackle now because we’ve never had such situations in the past. Previously, Governments and countries provided direction on content policies - what individuals could say, write, publish, and speak. Now with the internet, those geographic borders are gone, yet we still pretend content moderation demands a homogeneous and consistent approach. This is the starting position and crux of the problem we are faced with.
I could be brash and express that these current conversations are “too late”, but I don’t believe that. Could they have happened sooner? Yes. But this is a new field we’re stepping into and the build up to this point could simply be seen as defining the field of play. Rushing to say “this should have happened sooner” shortchanges the implications in asking a private company to take control over global speech and content standards. I’d rather focus on how we design and think of the future than point fingers to the past.
To start solving this, as designers, we know a fundamental rule in problem solving relies on first finding the right questions.
Looking forward, I have a light framework of five (5) questions that can provide a foundation on which we can start to build a more constructive discussion. Each of these questions most likely would have their own frameworks and questions within them. These questions do not provide solutions (yet), but they aim to better define the field-of-play we are working within.
A question framework for content moderation
Whose rules do we follow?
When do we have a problem?
How do we manage scale?
How do we plan forward?
What might the end result look like?
These questions mostly sit at the top layer of the internet, meaning the open web, platforms, and apps. For further reading on the different levels at which content moderation can be applied, read Navigating the Tech Stack: When, Where and How Should We Moderate Content? By Joan Donovan or A Framework for Moderation by Ben Thompson.
. . . . .
1. Whose rules or principles do we follow?
As stated in the opening, the Internet has presented us with a unique challenge that has never been faced at this scale. Companies like Facebook, Twitter, Tiktok, Snapchat, and others operate at a global level.
Even as we often read a multitude of articles on Section 230, we should all be aware this is only a US law. With companies that span the entire world, there is a spectrum of moderation methods. On one side we have global policies, and on the other, individual policies per country. Thus far for moderation, we have kept on the global side. There have been exceptions, and over the past few years, more countries have been passing policies which make moderation move more towards the middle of the spectrum, where local policies are mixed with a global standard.
To find our place on this spectrum, the question we should ask, as we think of global policies, is “Whose rules and principles do we follow?”. Across the world countries such as Saudi Arabia, Philippines, Germany, and China, have different policies on what can be published, different principles their societies adhere to, and different cultural norms. For current global policies within social media companies, we have often leaned towards the US stance on freedom of speech and information.
As we look forward, we should plan for a two part system and solution. First, define the areas we can all agree on as common rules and principles of moderation. Second, on top of those areas, create the policies that are country specific based on cultural impact or governmental impact.
. . . . .
2. When do we have a problem?
As we break down questionable or problematic content, it’s worth invoking this classic quote from From Mike Campbell in The Sun Also Rises
“How Did You Go Bankrupt?”
“Two Ways. Gradually and Then Suddenly.”
As we continue to define the rules of the field, and the types of problematic content, we have to paradoxically define when problems become problems. Assuming we still have a desire and foundation of free speech, and we provide an outlet for dissenting opinions, we also need to define when such an opinion becomes a problem. This too comes with layers and levels.
As an example, a person may ask the following question:
“I don’t understand why we need gender neutral toilets. You have two genders, male and female - so why do we need gender neutral?”
This in itself isn’t specifically problematic - it’s a question. However, we can probably all understand how in 3-5 comments, this can quickly escalate en mass to transphobic behaviour, bullying, hate speech, and even violent actions.
But when does that happen?
To summarise Casey Newton on Platformer, how many grains of sand do you need to add to a pile before it becomes a heap? What are our thresholds as dissenting discussion may escalate to bullying or hate speech, and how do we handle that escalation? What is the cut-off point?
In many ways, this can be confusingly summarised by Karl Poppers Paradox of Tolerance. That is, a society that has no limits to what it tolerates will eventually be destroyed by intolerance. When grains become a heap, we have a problem. Therefore at what point is intolerance defined, and thus stopped or removed? When does the heap form?
The problem with heaps, like bankruptcy, is they will happen quickly and before we know it. Then, once an eventual heap gets too large, it begins to move to become a reality. Once a heap is reality, we now have a splintering or bifurcation of realities. Defining an event horizon in moderation of conversations is difficult, and creating a “line to cross” is impossible in standard terms. But without moderation, the growth of new heaps will become more normal, and as we know "A house divided against itself cannot stand.".
Handling escalations, like most eventual conclusions to moderation, is not a black and white, but more likely a system of filters and contexts that can help determine when a heap of intolerance is forming, and ensuring it’s stopped - which leads us to the next question.
. . . . .
3. How do we manage scale?
As heaps continually grow and form, it’s not simply the rules of moderation, but the scale of management. Content is growing at the scale of Moore's Law or faster. While tens of thousands of individuals work to police content across the world, the problem has already exceeded human scale and will need to be converted to AI and automation.
This means at some point, AI moderation will define our reality. The ability to make the subjective into objective policies is the biggest challenge AI will face in moderation. How does an engine understand context, culture, intent, and historic meaning? As opposed to a set of rules, I believe it will be a combination of contexts, principles, and rules to assist in filtering content.
As an example, if the statement “Khashoggi was murdered” is presented to an AI engine, will it be recognised as bad or good? We would need a type of filtered decision making to make an assessment. Something akin to the following:
What is the country of origin? (context)
Does that country support free speech? (principle)
Was it a quote from another individual than the author? (rule)
Does it contain profanity or blocked words? (rule)
Is it written as a statement of fact, or a question of debate? (context)
Has this statement or perception of this statement changed over time? (context)
Of course these questions are partially unique to this individual statement. and this type of general purpose moderation for AI difficult, teetering on impossible. How to distill historic and cultural contexts layered with opinions into an algorithm. A first step may be to break down a framework of different types of content, and the types of filters that go with it. From that point, a more feasible view of the problem may come into focus with a clearer idea of how to fix it.
As already seen years ago with Tay, leveraging crowdsourcing or using existing material to train AI can have disastrous results. Therefore the creation of such systems will likely become a strong company advantage. As we begin to ask private global companies to police speech instead of Governments, the approach, roll-out, and ability to adapt will become a defining advantage.
. . . . .
4. How do we plan forward?
We need to accept there is no fixed solution. Speech and moderation of content is not static. It is ever evolving and changing. Therefore we must develop systems that are flexible and adapt with each step we take.
In times of ambiguity, the best we can do is provide clarity where we can. While there may be unknown-unknowns, as we chip away the knowns, we need to be clear on what those are and what they mean for users.
Current content guidelines are broad and difficult to interpret. While we can all agree child pornography is clearly horrible, the image of a small naked child running down the street after her village was bombed in the Vietnam war is historic, not pornographic. This level of context and understanding is what future systems must understand.
As rules and guidelines are developed, examples can be used to assist in understanding. Much like many brand guidelines contain visual and textual presentations, clear language and examples will assist in clarifying for those who need more understanding. Even if you may not agree with something, at least you can understand it.
Finally, as this is a work in progress, continuing to improve customer support will play a critical role in creating a positive and clear experience. Being able to truly understand what guidelines were abused, and why content was removed should play an increasing role over time.
Similar to how developers leverage threat models and scenario planning for security issues in software, content should look to develop these systems for their needs for preparation, planning, and response.
. . . . .
5. What might the end result look like?
To close, platforms are fundamentally an outlet of human behaviour. That means we have to accept the bad and the good in behaviour, but it does not mean the bad needs to be published or seen. It doesn’t mean everyone is entitled to publish what they want. Like children on a playground, without moderation, we will allow bad actors or bullies to flourish and influence others. We must paradoxically say “there is a line”, when we know there is no such line.
We should no longer assume the good will police the bad en mass. We can not afford that optimism. Optimism is a hopeful stance. It’s a view that in the end, we will be ok. It is inaction, and thus a sin of omission. This is a time for pessimism - that we see and acknowledge the negative. Because it is only at that point we can truly confront and solve the problem. We should prepare for bifurcations of reality and assume that is the default, thereby preparing how to solve it.
This year, we need to go back to the start and ask what we want to build. What is the future we want to enable? Then we need to acknowledge the bad - those who have malcontent, take advantage, and abuse the system. Start with the bad and the good, accept them both, and understand they are mutually exclusive. Our goal was to create communities, not realities.
Build for the threat. It seems only then we will have the opportunities we want to create!
Want to talk? Catch me on Twitter.