Content moderation: b*obs, or no b*obs?

The challenge is not how to identify what good vs. bad look like, but how to operationalize and enforce opinions at scale

Disclaimer: personal opinions and views

The other day, I wrote a post about sex and thinking where I briefly touched upon some of the work I did while at TikTok and I received some questions about it. I thought I would write about content policy and why moderation is so hard. I will go into maybe 0.01% of the challenges TT is dealing with. I won’t go into too much detail but I will share some general learnings as I feel like there aren’t enough nuanced discussions about this topic out there :)

This post will cover:
1. The pros and cons of working at TikTok
2. What does good content look like?
3. Example: b*obs or no b*obs
4. Binary choices - Trump or No Trump

1. Working at TikTok

I recently left TikTok, where I worked with policy for 1,5 years. Many people ask me what it's like to work at TikTok, referencing concerning articles shining a light on stressful and toxic work culture. People always struggle to keep several truths in the same brain. The reality is, TikTok was both a fantastic place to work for, and very toxic at the same time. Some teams have awful work-life balance, moderators are treated bad, the Chinese working style and culture is very, very different to ours, and the company is not very transparent. I wouldn't say I liked some aspects. But it's also true that you get incredible scope at TikTok (hello writing global policies affecting one billion users from scratch in your third week!), and it's awesome working for a company with perrrfect product-market fit (!!!) with smart people who genuinely wanted to do the right thing. It was incredible seeing the growth from the inside. I had never experienced anything like it before.

I witnessed similar challenges that you would see in any big tech company with growing pains; inefficient processes, decision-making is flawed, you need to find the right people to speak to, the product teams want to do one thing while the legal teams say the opposite, regulators ask us to do xyz but their requirements and demands are completely out of touch with reality and how things work in practice. You know, the classic problems we read about in textbooks. Plus China dynamics.

2. What does good content look like?

Working in policy means deciding what good and bad looks like and where to draw the line between what is considered inappropriate or not. The various policy teams are dealing with “keeping users safe and removing harm from the platform”, on both the paid and organic side. The work boils down to questions like: where do they draw the line for what is too sexual? How should Russia/Ukraine be managed? How much responsibility should they take on as a platform to keep people from scams? Should they act on these accounts that look like fraud if they don't have concrete evidence? At what point would content promoting fitness and diets become toxic? Should the platform be liable for these borderline claims people are making around making money? Should the platform allow political discussions, and if yes, how can they do that without being biased?

In a way, TikTok is like a mini society (albeit a lot less complex lol) because they need to create rules and governance systems for:

• Who should have decision power? How do we choose these people and what training do they need?

• How do we enforce “laws”?

• When is the law negotiable? Under what circumstances do they make exceptions?

• Who goes to prison (which accounts do we remove)?

• How do we classify risk and harm on a scale of 1-10? Are drugs worse than guns? If users are allowed to post violence during the war and it leads to more misinformation and fraud, is it still worth it?

• How do we tackle information/content around biased news, climate change, human rights, hate speech, free speech, safety, etc

Of course, your decisions are based on a wide range of factors other than your opinion; input from key stakeholders, laws, regulations, various risks, etc. Every area requires different thinking and approaches, and no case is the same. At the end of the day, you're still the one realizing the vision by putting everything in a document and clicking publish.

While these conversations are exciting to discuss from an intellectual and philosophical point of view (is free speech still good if it's harming people?), in practice, the nature of the work is less..sexy. One of the biggest challenges is operationalizing opinions and morals at scale. We need simple rules that moderators can use to categorize content quickly. With content, I mean millions and millions of videos from all walks of life and an endless range of original formats. They need to write simple sentences that work across hundreds of markets and that every individual, regardless of culture, understands. These rules need to work on audio, visuals, objects, words, education, and satire, all of which make TikTok videos addictive and fun.

Also, every case/industry/scenario is different. Many other aspects, such as technical limitations, labels, laws, knowledge, humans, and computers, make the process incredibly complex, but let’s ignore most of them.

Sounds straightforward, but it's not. Let's look at an example and simplify it a bit.

3. Example: b*obs or no b*obs

Ok, I could perhaps have chosen a more sophisticated topic, like elections or political content. But my view count went through the roof when I wrote about this a few days ago, so…

Many people criticize platforms for censoring nipples and nudity, and it's been like that for a while now. I agree with the complaints, but I am not here to express my opinion but to shine some light on the operational challenges behind these scenarios. Yes, it's true TikTok (and other platforms, it seems) remove content with too much skin exposure, regardless of intent. Are they being lazy? Yeah. Is it more difficult than it seems? Yeah.

Sexual content is a personal topic of mine because it was one of the first areas I got to cover and draft policies for when I joined. Just before TikTok, I had completed a master's in tech policy where we spent hours discussing content moderation from a very theoretical point of view, and so coming to TikTok and having to say the full version of the abbreviation ‘WAP’ in front of 150 people in my first weeks, including leaders who used to work with former PMs, was a bit of a shock to the system. Enforcing content policies at scale boils down to one question:

Can we do this at scale? 

And, breaking down the problem further, we are left with the question

B*obs, or no b*obs?

The same idea can be applied to any area where people have subjective opinions on good vs. bad; politics, human rights, negative body image, controversial products, financial advice, the war, conflicts, minor safety, etc.

Let's look at this photo below.

Photo: @claramorberg 

If you go to the user’s feed, you would see it's a beautiful shot taken by a photographer and a visual expression of motherhood. Now imagine a similar photograph, but if you looked at the account, you would see it's a spam account with women looking underage. It's an entirely different vibe in their feed. So you can understand intuitively which is good, but both photos would be identical if you would describe them without subjective words. Essentially, they are both nude photos.

If you know nothing about the user or the context of this image, would you be able to provide a sentence and keywords/objects that capture which one should be forbidden and which is ok to keep? Could you do this without too much bias while ensuring anyone, regardless of culture, religion, and view on sexual content, would interpret in the same way? Something a computer could pick up? All this, while meeting the regulators ambiguous and absurd requirements and addressing the concerns they receive from NGOs and parents and civil society?

People say phrases like "if it's a creative expression, it's not harmful," or "if it's art/artsy it's ok" - these statements are not helpful because they don't work at scale. What the f is "artsy"? What is "feminist nudity"? (Yes, people have complained about TT removing “feminist nudity”). Moderators don't have access to the whole context; they don't care or know who you are and why you're posting what you're posting. Remember, this is one example of thousands of scenarios, so even if TT could find a way to manage this specific situation, there would be a backlog of millions of exceptions waiting for them.

This + inefficient processes + training errors + human/computer errors + lack of transparency + weird company dynamics + scale + unknowns + technical issues + a lot of other things = the output will inevitably be flawed.

4.Binary choices

If you want to allow content criticizing Donald Trump, because that’s what most people would consider reasonable, you also need to allow content encouraging and promoting his ideas. If you allow female nudity because we shouldn’t censor something natural, you will also see an increase in indirect or direct pornographic content. This is the hard truth. Users get upset if TikTok decides what is considered a good political debate. Still, if TikTok lets users choose, users get mad because TikTok enables the spread of wrong opinions. If we allow some, we need to allow everything, including those political opinions that are considered morally wrong (with obvious exceptions like explicit hate speech, since this would never be allowed). It’s all about binary choices, which is hard because content isn’t binary, so the system, processes, and structures the platform uses make little sense.

Moderating content is very hard, and I have deep admiration for the work moderators are doing for us to enjoy this amazing platform. Without them, you would not want to spend much time on the platform; I can tell you that. Policies and moderation exist to protect users against the worst-case scenarios. Unfortunately, this also means a cost of business to the platform in the form of unnecessary removals and bans.

I have more to say about this and social media, how it works, why it’s hard to fix “obvious” problems such as misinformation and scams, and the operational challenges behind it. There are also many black boxes I never managed to untap while working at the company.

Remember, this was a very simplified post about something complex with more nuances than you could imagine.

For new readers, check out my most popular posts below:

Body Image and Content Moderation

About knowing who you are

About love that hurts

xx