Deepfake-driven Scams Weaponize Pity Narrative to Defraud at Scale

A young woman with Down syndrome pleads for you to stop scrolling and take a look at the handmade crafts she is selling. An old lady invites you into her small workspace to show the humble art she makes from sea glass. The narrative in both short-form videos is the same – vulnerable humans pulling at the heart-strings to try and convince you to buy their ‘art’. But the people aren’t real, and the ‘art’ is mass-produced and drop-shipped from a factory in China. 

In a new deepfake-drive scam technique being tracked worldwide by our Threat Intelligence team, hyper-realistic synthetic protagonists with disabilities, old people, even people with terminal illnesses, are generated with the intent of convincing large numbers of consumers to spend their money on what they think are unique, hand-made items. 

Synthetic protagonists appeal to pity to sell deceptive goods

But the items in question are mass produced and drop-shipped from a factory, and the social media videos are mass produced by AI-generators to follow a templated and well-structured narrative that fraudsters know will appeal to a broad audience. 

Some of the social media profiles of these fictional people have hundreds of thousands of followers and some of the posts have millions of likes, demonstrating just how effective the deepfake-drive scam technique is at going viral. 

This deepfake-driven fraud clearly leverages the cognitive attack framework exposed in our report on the Narrative Kill Chain, but in a non-military context. Each short-form video is designed to deliver an emotional payload along with a fragment of the overall narrative. The target does not have to consume the entire narrative in one go, but over time and with exposure to enough narrative fragments, will be psychologically encouraged to buy a product, and ideally, share the video with their own audience. 

Weaponizing a sob story: The Narrative Kill Chain

Some of the narrative stages noted by our Threat Intelligence team include:

Manufactured identity and plea – The synthetic protagonist pleads with the viewer to stop scrolling for a few seconds and listen to their, typically tragic, story. Sometimes the protagonist or another synthetic character, such as a mother, is crying. The intention here is to appeal to the target audience with a manufactured identity you feel pity for, then engage with the viewer in a moment of distress. 

Victimization – In this stage the protagonist is typically being abused or ridiculed by a synthetically generated member of the public, or a malicious comment may be superimposed on a video of the protagonist looking sad and vulnerable. Sometimes the product is briefly introduced at this stage as an additional target of ridicule – “no one will buy your stupid [product]”. The intention here is to elicit feelings of pity to try and get the viewer to start emotionally supporting the protagonist as well as to manufacture positive feelings about the product itself, even though it hasn’t been revealed in detail.

Determination – This is the stage where we typically get a ‘behind the scenes’ view of how the art or crafts are made. Fully generated scenes will show the protagonist making the items by hand, often overlaid with quotes along the lines of “mama told me not to quit”, or “I won’t give up”. The intention here is to continue the emotional buy-in and strengthen support for the protagonist as they attempt to overcome adversity, while introducing the product for sale and establishing it as a hand-made item, which has an emotional investment of its own. 

Product showcase – Along with more fabricated insight into the hand-made manufacturing process, we now get AI-generated glimpses of the finished product available for purchase. The items often appear to be of very high quality and are typically accompanied by text or quotes such as: “Be honest guys, what do you think? Should I keep making [product]”. The intention here is to cement positive emotions about the product being sold and continue the emotional investment in the protagonist overcoming adversity. 

Validation – Now we begin to follow the success arc of the narrative, where the protagonist appears to overcome adversity and capitalises on the positive emotional sentiment already built up within the target audience. A typical segment is the “I’m so excited I just got my first order”, where the protagonist thanks the audience for support and shows themselves packing the product for a customer.  The intent here is to shift the emotional state from pity to redemption, and to appeal to confirmation bias that people are buying the product, so maybe you should too.

Sometimes a fully AI-generated scene will show supposed customers receiving and enjoying the purchased items. The intent here is to generate third-party validation that the product and the protagonist are legitimate. 

From real unboxing videos we’ve seen, the actual products received are much lower quality, mass factory-produced items that bear only a passing resemblance to the AI-generated items in the videos. 

Some related scams using a similar playbook, manipulate stolen videos of real children with terminal illnesses and incorporate the sick child as part of the narrative, pushing the story that the craft sales are to help cover medical expenses for a fictional son or daughter. 

Deepfake-driven scam dissemination strategy leverages social sharing

Many of these scam accounts have tens of thousands of followers and tens of millions of engagements on content. The same content is found posted across multiple profiles and the same video templates are used with different protagonists and products. 

As we describe in the attack framework accompanying the Narrative Kill Chain, deepfake-enabled scam narratives are typically introduced through high-velocity social platforms such as TikTok or YouTube, where short-form, emotionally charged content is structurally advantaged by algorithmic ranking systems, benefiting from recommendation engines optimized for engagement rather than verification.

The more ‘likes’, comments – even negative comments, and engagement a piece of media gets, the more it is exposed to other lookalike members of the same and adjacent audiences, eventually creating a snowball effect designed to appeal to the broadest audience possible.

The fraudsters also appear to make good use of bots to automate and inflate the number of comments and engagements on the posted videos. The same comment structure is seen time and again, either expressing support for the protagonist, approval for the product, “how do I buy this” followed by a reply with instructions, or “I just bought this!”. 

The Narrative Kill Chain is used in a non-military context

Since we exposed the Narrative Kill Chain matrix and dissemination playbook, we are seeing the same attack framework for cognitive weapons used in increasingly non-military contexts, involving everything from fraudsters begging for donations for synthetic terminally ill victims, to holistic health and wellbeing apps and supplements pitched by AI-generated sufferers of various mental and physical illnesses, to synthetic propaganda intended to distort public perception around political events.

The democratization of generative-AI, which can now be used to create highly realistic deepfakes at large scale and low cost, means that in the last 12 months, the explosive growth of synthetic media has lowered the barrier to entry for fraudsters and turned an already challenging situation for law enforcement into an almost insurmountable problem, requiring a restructuring of the investigative and forensic process.