Facebook has announced new tools to prevent sharing of images, videos and any other content which includes child sexual abuse material (CSAM) on its platform. For one, it will warn users when they are sharing images which could contain potential CSAM material. Second, it will prevent users from searching for such content on its platform with a new notification.
While the first one is aimed at those who might be sharing this content with non-malicious intent, and second is aimed at those who go searching for such content on Facebook with plans to consume this content or to use it for commercial purposes.
“We don’t allow instances of child sexual abuse or the use of our platform for inappropriate interactions with minors. We actually go the extra mile. Say when parents or grandparents sometimes share innocent pictures of their children or grandchildren in the bathtub, we don’t allow such content. We want to make sure that given the social nature of our platform we want to reduce the room for misuse as much as possible,” Karuna Nain, Director, Global Safety Policy at Facebook explained over a Zoom call with the media.
With the new tools, Facebook will show a pop-up to those searching for CSAM content offering them help from offender diversion organisations. The pop-up will also share information about the consequences of viewing illegal content.
The second is a safety alert that informs people when they are sharing any viral meme, which contains child exploitative content.
The notification from Facebook will warn the user that sharing such content can cause harm and that it is against the network’s policies, adding that there are legal consequences for sharing this material. This is aimed more towards those users who might not necessarily be sharing the content out of malicious reasons, but might share it to express shock or outrage.
Facebook study on CSAM content and why it is shared
The tools are a result of Facebook’s in-depth study of the illegal child exploitative content it reported to the US National Center for Missing and Exploited Children (NCMEC) for the months of October and November of 2020. It is required to report CSAM content by law.
Facebook’s own admission showed that it removed nearly 5.4 million pieces of content related to child sexual abuse in the fourth quarter of 2020. On Instagram, this number was at 800,000.
According to Facebook, “more than 90% of this content was the same as or visually similar to previously reported content,” which is not surprising given very often the same content gets shared repeatedly.
The study showed that “copies of just six videos were responsible for more than half of the child exploitative content” that was reported during the October-November 2020 period.
In order to understand the reason behind sharing of CSAM content better on the platform, Facebook says it has worked with experts on child exploitation, including NCMEC, to develop a research-backed taxonomy to categorise a person’s apparent intent behind this.
Based on this taxonomy, Facebook evaluated 150 accounts that were reported to NCMEC for uploading child exploitative content in July and August of 2020 and January 2021. It estimates more than 75% of these people did not exhibit malicious intent, that is they did not intend to harm a child or make commercial gains from sharing the content. Many were expressing outrage or poor humour at the image. But Facebook cautions that the study’s findings should not be considered a precise measure of the child safety ecosystem and work on this field is still on-going.
Explaining how the framework works, Nain said they have five broad buckets for categorising content when looking for potential CSAM. There is the obvious malicious category, there are two buckets which are non-malicious and one is a middle bucket, where the content has potential to become malicious but it was not 100 per cent clear.
“Once we created that intent framework, we had to dive in a little bit. For example in the malicious bucket there would be two broad categories. One was preferential where you preferred or you had a preference for this kind of content, and the other was commercial where you actually do it because you were gaining some kind of monetary gain out of it,” she explained adding that the framework is thorough and developed with the experts in this space. This framework is also used to equip human reviewers to be able to label potential CSAM content.
How is CSAM identified on Facebook?
In order to identify CSAM, the reported content is hashed or marked and added to a database. The ‘hashed’ data is used across all public space on Facebook and its products. However, in end-to-end (E2E) encrypted products like WhatsApp Messenger or secret chats in FB Messenger would be exempt because Facebook needs the content in order to match it against something they already have. This is not possible in E2E products, given the content cannot be read by anyone else but the parties involved.
The company claims when it comes to proactively monitoring child exploitation imagery, it has a rating of upwards of 98% on both Instagram and Facebook. This means the system flags such images on its own without requiring any reporting on behalf of the users.
“We want to make sure that we have very sophisticated detection technology in this space of Child Protection. The way that photo DNA works is that any, any photograph is uploaded onto our platform, it is scanned against a known databank of hashed images of child abuse, which is maintained by the NCMEC,” Nain explained.
She added that the company is also using “machine learning and artificial intelligence to detect accounts that potentially engage in inappropriate interactions with minors.” When asked what actions Facebook takes when someone is found to be a repeat offender on CSAM content, Nain said they are required to take down the person’s account.
Further, Facebook says it will remove profiles, pages, groups and Instagram accounts that are dedicated to sharing otherwise innocent images of children but use captions, hashtags or comments containing inappropriate signs of affection or commentary about the children in the image.
It admits that finding CSAM content which is not clearly “explicit and doesn’t depict child nudity” is hard and that it needs to rely on accompanying text to help better determine whether the content is sexualising children.
Facebook has also added the option to choose “involves a child” when reporting a picture under the “Nudity & Sexual Activity” category. It said these reports will be prioritised for review. It has also started using Google’s Content Safety API to help it better prioritise content that may contain child exploitation for our content reviewers to assess.
Regarding non-consensually shared intimate images or what in common parlance is known as ‘revenge porn’, Nain said Facebook’s policies not only prohibit sharing of both photos and videos, but making threats to share such content is also banned. She added Facebook would go so far as to deactivate the abuser’s account as well.
“We have started using photo matching technologies in this space as well. If you see an intimate image which is shared without someone’s consent on our platform and you report it to us, we’ll review that content and determine yes, this is a non-consensually shared intimate image, and then a hash will be added to the photo, which is a digital fingerprint. This will stop anyone from being able to reshare it on our platforms,” she explained.
Facebook also said it is using artificial intelligence and machine learning to be able to detect such content given victims complained that many times the content is sharing places which are not public such as private groups or someone else’s profile.