Beginning in 2015, AI trust and safety initiatives took on a new urgency. Prior to 2015 efforts focused on identifying problematic images and video. The world was beginning to realize that grave danger, such as grooming, and recruitment by extremists and terrorists were occurring at alarming rates within these communities. Better tools were needed to monitor posts, private messages, video messages, and live streaming events.
Natural Language Processing
Over the past few years, the world has witnessed atrocious acts of violence that were live streamed, prompting the public and government officials to demand social media platforms “do something” about the problem. But the traditional method of monitoring live stream events, creating transcripts of conversations and taking action against bad users after the fact, wasn’t up to the task. Transcripts are also extremely expensive to produce, which limited their use.
Transcripts are also highly inaccurate as they can’t capture the intent of a speaker who may be expressing sarcasm via his or her tone of voice or body language. Intent is a huge mitigating factor when determining if exchanges represent likely incidents of self-harm, cyberbullying, grooming or recruitment.
Enter: natural language processing or NLP. NLP is a field of science that sits at the intersection of linguistics, computer science and AI. Data scientists seek to develop models that are capable of processing and analyzing vast amounts of natural language data. NLP is particularly useful in monitoring live-streamed events, as it can pick up on non-verbal signals and context.
It’s difficult to overstate the importance of context. A 14-year-old boy may say to his best friend, “I tripped and fell right in front of Julia in math today. I should just kill myself.” In this situation, the writer probably won’t commit suicide. However, it’s another matter altogether if a recently divorced person who is concerned about financial issues and says, “I want to kill myself.”
A challenge for content moderators is that they must review large volumes of information quickly. NLP can serve as a first-line assessment tool, distinguishing real threats from hyperbole, so that the more likely threats are sent to a human moderator close to real time for faster review.
There are numerous other advantages of NLP. For instance, the more it’s used, the better its models will get at successfully identifying the content that moderators need to see ASAP. It also allows for greater scale, enabling more conversations to be analyzed for troubling content. And as the cost of super computing and AI development go down, more communities will be able to deploy it to keep their users safe.
Discover Content Moderation in Multiple Languages
