The Media Bias Analyzer
In this prototype, I created a Google Chrome extension that, when a user navigated to a news article, ran a LLM analysis over the text and returned a dashboard with an article summary, the political lean, the emotional charge of the language, the surrounding context, and inline highlighting on the specific passages driving each assessment. This was an especially interesting and applicable use case for LLMs since the core of the technology is based on pattern recognition and statistical approximation and thus I was interested in seeing how well it would perform when assessing different levels of media bias present in the news people consume.
Media Bias Analyzer demo
Article analysis
In designing the prototype, I wanted to shape it around those 3 aforementioned principles by providing transparency via showcasing the exact AI system prompt being used for the analysis, allowing user agency to modify and edit the AI prompt instructions accordingly, all the while affording privacy as it was entirely hosted and run locally.
Editable system prompt
In total, the extension worked well and the results were reasonably accurate. There was plenty of room for growth and expansion of the features and capabilities of this application, but after demoing the project for my class, I decided to end development and open-source it for anyone to use. It was an interesting and exciting idea that I felt could have a positive social impact, but after showcasing this demo to others and collecting feedback, I knew that this solution wouldn’t be the right shape to tackle such a challenge for a couple of reasons.
Why the Prototype Stopped
The first reason was trust in the core technology. Despite efforts to provide transparency in the AI system prompt being used and providing agency in modifying it, LLMs regardless are a non-deterministic technology, meaning that its output is not always determined and despite efforts to minimize error, the core technology is fundamentally prone to hallucination errors. It is a reality that was very apparent to me having previously worked for as an Implementation Manager for GameOn Technologies where a large responsibility of my job was leading AI hallucination and validation efforts for our chat bots that would serve thousands of users and their queries for high value clients like the New York Yankees, Las Vegas Raiders, among others. Despite attempting to leverage my experience in AI validation training, it was clear that there would always be a hurdle towards it as people would understandably always question its output. For an analysis tool whose goal is to help you trust the content you’re reading, how is a user expected to reliably use it if they fail to trust the tool itself?
The second reason was the specific user interaction patterns needed to use the tool. In doing some more market research, I found that roughly 57% of U.S. adults get their news through mobile devices versus 30% on desktop/laptop computers.Pew Research A Chrome extension made prototyping and experimenting easier, but it failed to meet at the right channel where most people go to search their news. I even experimented with creating an iOS version of this analyzer which still exposed a fundamental flaw with the whole design.
From Endpoint to System
With this, in mind I realized that the solution of analyzing one article at the endpoint was a last-resort behavior. More specifically, it catered to a very niche segment of skeptical news power users who are motivated enough to go through the hurdles of seeking out a second opinion, a hurdle that a vast majority of people wouldn’t bother with despite desiring a solution that would make the news media landscape more transparent. The Media Bias Analyzer was treating a symptom of media mistrust rather than providing a real solution to the core problem with the digital media landscape: that these platforms are not optimized for truth, but for engagement. The Media Bias Analyzer only sought to tell you if a single end-point article was biased, but as long as social media, news aggregation, and other media platforms continued to optimize for this engagement metric, a solution like this would be like playing wack-a–mole.
After retiring the project with these takeaways in mind, I kept thinking about what the right solution would be to tackle the core of the problem here. I thought about how if there was a single independent platform that made the whole news aggregation and distribution process transparent, provided users with the agency to control the levers behind this process and the tools to audit it, as well as preserved the complete privacy of their information and behavioral habits, perhaps it could be a role model for realizing City Three.
Chapter Summary
The “Media Bias Analyzer” was a simple experiment, if generative AI technology was being used to spread misinformation (and ultimately capture attention), how could that same technology be used to make us more aware of how that content worked on us? In this prototype, I created a Google Chrome extension that, when a user navigated to a news article, ran an LLM analysis over the text and returned a dashboard with an article summary, the political lean, the emotional charge of the language, the surrounding context, and inline highlighting on the specific passages driving each assessment. Although the extension worked well and the results were reasonably accurate, I decided to end development and open-source it for anyone to use. I realized that the solution of analyzing one article at the endpoint was a last-resort behavior. More specifically, it catered to a very niche segment of skeptical news power users who are motivated enough to go through the hurdles of seeking out a second opinion, a hurdle that a vast majority of people wouldn’t despite desiring a solution that would make the news media landscape more transparent. After retiring the project with these takeaways in mind, I kept thinking about what the right solution would be to tackle the core of the problem here. I thought about how if there was a single independent platform that made the whole news aggregation and distribution process transparent, provided users with the agency to control the levers behind this process and the tools to audit it, as well as preserved the complete privacy of their information and behavioral habits, perhaps it could be a role model for realizing City Three.