Often, the first step for individuals that create an account to interact on an online platform is selecting a username. This could be on a social platform, a dating app, a marketplace - anywhere that user-generated content is displayed, a customer-generated username is the norm.
In most cases, people will select a harmless and inoffensive username. However, sometimes an individual will use poor judgment when creating a username that is profane, offensive, or in some way toxic to the other users of the platform.
What is the problem with toxic usernames?
The problem with this is that a profane or offensive username can affect the tone of future interactions between that user and the rest of the community. First, an offensive username affects other users directly, making them feel discomfort and impacting their experience on the platform. It also erodes user confidence in how well the platform is managed and monitored - a strict policy against toxic behavior is only meaningful if it is enforced for all user-generated content (UGC).
Ideally, to ensure a safe, inclusive environment for your members, online toxicity should be prevented in all UGC – including usernames. This isn’t as simple as it seems on the surface, however.
Challenges to username moderation
Timeliness
Depending on how large the community is, and how quickly it is growing, it can be extremely difficult to manually review usernames as they are created. This can lead to a backlog of usernames that must be reviewed by an employee; or reliance on an automated solution.
Context
Whether or not a username is appropriate is largely dependent on context. Different standards apply to different types of platforms: what is okay for an adult dating app may be incredibly inappropriate on an educational website for children.
L33T Speak
Some individuals that are more technologically astute than the general population - often gamers, programmers, or self-identified hackers - are familiar with an alternative communication style known as L33T Speak, or LEET Speak. People use numbers in place of letters to create words, which serves a few different purposes. It works as a means of identifying other people that are a part of a subset of internet users: those with programming, hacking, or hard-core gaming experience. But L33T Speak also circumvents traditional content scanning measures, complicating the moderation process.
Multiple Languages
A word or phrase that is innocuous in one language or within a certain cultural context, can be extremely offensive to people speaking another language or with different cultural associations. Most automated solutions fail to scan for different languages and cultural contexts, which can also be challenging for human moderators.
Username moderation solutions
Employees
The content moderation standard for all types of UGC is human moderators: having a person review all user-generated content, including usernames, comments, and posts. However, this approach has a number of drawbacks; it is resource-intensive and inefficient; it can be detrimental to the mental health of employees, and it is difficult to manage in real time.
Filters
Many platforms have created or purchased filtering solutions to partially automate the content moderation process, and relieve employee moderators of some of their burden of work. However, filters have some significant drawbacks: they are standardized and completely overlook the importance of context; they are rigid and inflexible; and they can rarely be adjusted to accommodate multiple languages, cultures, or L33T Speak. This lack of accuracy means that offensive content may ‘pass’ the filter, and be seen by other users on the regular platform.
Ideal Solution
Ideally, an automated content moderation solution should work in real time, and be able to evaluate username moderation in context: with a near-human understanding of not only the text, but also the implications and associations of inappropriate user-generated content. Moreover, it should be sophisticated enough to understand L33T speak and have the capability to interpret multiple languages in both a language and cultural context.
The Meet Group Case Study
Effective username moderation is vital to safeguarding the user experience, especially on gaming, dating, and social platforms.
Spectrum Labs helped The Meet Group implement real-time validation of display names, accurately identify harmful usernames, and automate moderation, reducing incidents requiring human intervention by 50%.
Spectrum Labs AI was created to answer these questions, meet these challenges, and help platforms create and enforce appropriate UGC. Our Contextual AI solution evaluates user-generated content in real time, over multiple languages, deciphering context and adapting to a changing environment. Accurate, automated and reliable: Contextual AI relieves employees of the burden of content moderation, allowing you to channel those resources to higher-level, strategic objectives.
