While AI whispers, content creators make magic

Generative artificial intelligence (AI) can improve content consumption by filling missing metadata, but a study finds it can’t beat humans, for now

Featured faculty: Philip Zhang Renyu
Written by Putro Harnowo

Social media has become more than just a means to connect, but also a way to build communities and markets beyond borders. When TikTok was on the verge of a US ban due to national security concerns earlier this year, a mass exodus of American users to another Chinese social media platform, RedNote, ensued as a form of protest.

A few months after the ban was postponed, ByteDance, the company behind TikTok, aims for 20 per cent annual revenue growth to US$186 billion, just a smidge below Meta’s projection of US$187 billion, this year. Platforms are not the only ones living the good life. Social media users can earn money from sponsors as content creators, and if they have amassed a large following, they can earn celebrity-level income as influencers.

The secret to keeping users engaged is matching content with viewers accurately through organic recommendations, where algorithms scan a vast array of video titles to suggest content relevant to users based on their viewing history and preferences. Users can also find content through a search query, but this accounts for only a fraction of total viewership.

“That’s the reason why content creators are encouraged to add titles with hashtags or descriptions, which are the only metadata used by the recommender system to improve content relevance,” says Philip Zhang Renyu, Associate Professor of the Department of Decisions, Operations and Technology at the Chinese University of Hong Kong (CUHK) Business School. “This metadata provides structured and concise details about the video, enabling the system to better understand the content crucial for the platform’s recommender system.”

Unfortunately, most user-generated content platforms suffer from metadata sparsity, where much of the content lacks descriptive titles or hashtags. Many of them then leverage generative AI to create and fill in missing metadata, but the outcome leaves much to be desired, as Professor Zhang and his co-authors found in their latest work, The value of AI-generated metadata for UGC platforms: Evidence from a large-scale field experiment.

“Although human-generated titles tend to outperform AI-generated ones, AI-generated titles remain valuable as they reduce the effort of title creation and at least provide a usable starting point for humans,” he says. “Even if AI performs worse overall, its output can still offer guidance, particularly for low-skilled producers who may struggle to craft effective titles on their own.”

When AI becomes a game changer

In the study, Professor Zhang, along with Goh Khim Yong of the National University of Singapore and his PhD student Zhang Xinyi, as well as researcher Sun Chenshuo, partnered with a leading short-video platform in Asia to conduct a series of experiments. While the platform boasts more than 300 million daily active users, only 60.7 per cent of the videos have titles, similar to other user-generated content platforms.

The team analysed more than two million users who produced more than 10 million videos. The videos were categorised into utilitarian videos, whose primary purpose is practical, informational, and educational, such as news, reviews and tutorials; and hedonic videos like entertainment or lifestyle content, such as vlogs, comedy sketches and travel videos. Before the experiment, 61.3 per cent of utilitarian and 57.3 per cent of hedonic videos had titles.

“AI-generated metadata can enrich the input to recommender systems, thereby improving the accuracy of user-content matching and enhancing overall personalisation within the system.”

Professor Philip Zhang Renyu

The platform introduced generative AI tools in July 2023 to create metadata by capturing multiple frames from the video, then extracting visual elements and text to write metadata that reflects the content. The researchers assigned creators randomly to either the treatment group, where they were given access to AI-generated titles, or the control group, which could only make titles by themselves.

The AI tool was found to increase the likelihood of videos having titles by 41.4 per cent and hashtags by 72.4 per cent. Overall, videos with AI-generated titles had an average 1.6 per cent increase in viewership and a 0.9 per cent increase in watch duration.

For low-skilled creators or those with a total number of followers and videos below the median among all creators, AI-generated titles helped increase viewership by 1.6 per cent and watch duration by 1.3 per cent. Looking beyond viewership, a deeper analysis of an additional dataset of almost 94 million videos found that the number of likes, shares, and follows was significantly higher among the group with AI tools.

“AI-generated metadata can enrich the input to recommender systems, thereby improving the accuracy of user-content matching and enhancing overall personalisation within the system,” Professor Zhang says. “Videos with AI-generated titles are associated with higher viewership diversity, suggesting that improved metadata enables the system to surface content to a broader and more varied audience.”

Human brain vs. machine intelligence

While the results seem positive, AI may not be suitable for everyone. Utilitarian content saw a 3.1 per cent decrease in viewership and a 3.0 per cent decrease in watch duration with AI-generated titles. For content that already had titles written by humans, changing them with AI decreased viewership by 37.9 per cent and 32.6 per cent in watch duration.

However, when creators edited the AI-generated titles, each 10 per cent change would bring 9.8 per cent more views and 8.2 per cent more watch duration. When the similarity between the AI-generated and the revised titles drops to 20 per cent, the videos perform better than those without AI-generated titles. This suggests that content creators should significantly edit AI-generated titles to maximise their performance.

Further analyses revealed that AI produces a basic thematic summary but often misses context-specific descriptions helpful for clarity and engagement. For instance, AI would create a title that says, “Enjoy the beauty of nature #ScenicNature,” and the content creator could enhance it to “Lush mountains and flowing streams: embrace nature’s serenity”. More detailed descriptors improve lexical richness and enable the recommender system to match videos with users more accurately.

Collaborative forces across platforms

Generative AI has been praised for its ability to create content, but it also raises concerns about originality and copyright issues. The study offers another perspective on how AI can enhance metadata to boost recommendation systems and content discovery.

Given that AI-generated metadata is beneficial for low-skilled and hedonic content creators, platforms should consider focusing on these segments when deciding to scale up AI tools. Prioritising new content creators to access AI would have the most immediate and noticeable impacts. Additionally, rather than automatically integrating AI titles, platforms should consider providing an option to edit to enhance the generated metadata. This strategy is not only beneficial for user-generated content platforms but also for other platforms where recommendations drive content consumption, including e-commerce.

“On e-commerce platforms, products often require metadata such as material, colour, and size, which can be time-consuming for sellers to input due to the large number of items. AI-generated metadata can help streamline this process and is likely to produce similar improvements in recommendation accuracy and efficiency,” says Professor Zhang.

With advances in large language models and multimodal AI that can process and integrate information from multiple data types, Professor Zhang believes that AI-generated metadata is likely to evolve far beyond simple titles to include richer and more structured descriptions. Future AI would automatically generate summaries, hashtags, emotional tones, scene-level annotations, and even inferred narratives that integrate visual, audio, and textual cues.

“Such metadata would offer deeper semantic insights into content, enabling recommender systems to make more nuanced and context-aware matching decisions, especially valuable for addressing cold-start problems and enhancing personalisation across diverse content types.”