Three Experiments

Experiment 1: Could the Aggregation Foundation Exist?

The first experiment addressed a dependency beneath every other product requirement. Before I could organize reporting into Stories, explain why content appeared, or give readers control over their feed, I needed a reliable stream of news data that I could organize under my own rules.

I initially investigated whether an existing news API could provide that foundation. The available services presented an awkward trade-off for a self-funded product at such an early stage. Lower-cost and free plans were generally suitable for development but could impose delayed results, restrictive request limits, incomplete article data, or prohibitions on production use. More capable plans provided real-time access and richer metadata, but could cost hundreds of dollars per month before I had established whether the experiences built on top of that data would provide sufficient value. And the capabilities most relevant to the concept, including story clustering and custom classification, were also reserved for advanced or enterprise plans.NewsAPI pricing NewsData.io pricing NewsAPI.ai plans GNews pricing

Cost was only one concern. Depending on another aggregation provider would also constrain which publishers I could include, what metadata I could collect, how articles could be classified, and how quickly the system could adapt as later experiments revealed new requirements. Because the product hypothesis depended on making the organization and distribution of news transparent, outsourcing the underlying corpus to another largely fixed system would limit how completely I could fulfill that promise.

I therefore reframed the decision from “Which API should I purchase?” to “Could I assemble a sufficiently capable aggregation service from information that publishers already made publicly available?”

The initial prototype was deliberately small. I gathered RSS feeds from approximately 30 publishers and created a scheduled ingestion process that regularly checked them for new articles. Incoming entries were converted into a consistent internal structure containing fields such as the publisher, headline, publication time, article link, description, image, and available category information. Because RSS implementations varied considerably between publishers, I also experimented with fallback methods for recovering information that was missing, inconsistent, or formatted differently.

This prototype helped expose a broader requirement that had not been as visible at the outset. Aggregation was not simply the act of collecting links, for the resulting information to support clustering, comparison, and personalization, every article needed to pass through a normalization and classification pipeline capable of turning inconsistent publisher data into a dependable shared format.

Although far from being product ready, the experiment validated the technical feasibility of creating an independent news aggregation pipeline that could continuously collect and normalize reporting, and its modular structure provided a credible path for adding publishers and new classifications as the product evolved. Most importantly, I could control the fields, rules, and source coverage required by subsequent experiments without committing to a costly external dependency before validating their value.

That evidence was sufficient to give me the confidence to continue. I now had an initial corpus of consistently structured but disconnected articles. The next uncertainty was whether those individual pieces could be organized into something more useful: coverage of the same event, followed by the larger narratives connecting those events over time.

Experiment 2: Could Disconnected Articles Become Reliable Stories?

The aggregation experiment produced a growing collection of normalized articles, but each article remained an isolated piece of information. Before attempting to build the complete hierarchy I had envisioned, I narrowed the next experiment to its most fundamental assumption: could articles covering the same event be reliably identified and organized into a coherent Story?

I ended up testing several approaches for identifying relationships among articles and ultimately settled on text embeddings. Simpler NLP methods could recognize obvious overlap, but often struggled when publishers described the same event using different headlines, terminology, or framing. Embeddings allowed the system to compare the semantic meaning of the reporting rather than depending entirely on matching words.

Another critical element of the experiment was assessing the clustering quality. If the clustering rules were too permissive, unrelated events could be merged into the same Story. If they were too restrictive, coverage of the same event could be divided among several clusters. I repeatedly reviewed the resulting Stories, adjusted the clustering logic and weights, and reexamined whether each grouping remained coherent when read as coverage of a single event.

This process also sent me back to the aggregation foundation from the first experiment. Clustering could identify related reporting, but creating a useful Story behind it required more than grouping links. To show the composition of the coverage and compare how different parts of the media landscape framed the event, the system needed consistent information about the publishers behind every article.

I extended the backend to incorporate attributed third-party publisher assessments, including information from Media Bias/Fact Check. These assessments allowed publishers to be organized into broad coverage lanes and made it possible to produce left-, center-, and right-coverage distributions. Importantly, the output was based on third-party inputs which remained inspectable for readers to audit.

Once the articles, publishers, and coverage lanes were assembled into a coherent Story, I experimented with using AI to generate titles and framing summaries from that bounded collection of reporting. This was a deliberately narrower role for AI than the one I had explored in the Media Bias Analyzer. The earlier prototype asked a language model to independently judge the bias and reliability of an article. Here, the model was given a defined body of source material, relevant publisher context, and explicit instructions governing how that material should be summarized. For perspective comparisons, the reporting was separated according to the attributed coverage lanes before the corresponding summaries were generated. The objective was not to ask AI to invent what “the left” or “the right” believed, but to synthesize how the available reporting within each group framed the same event.

Of course, providing additional context could not eliminate the possibility of error or make a nondeterministic model infallible. It could, however, constrain the task and make the output more auditable. Readers would be able to move from a generated summary to the underlying coverage, while the prompts and generation methodology would eventually be published through the product’s public methodology. The goal was to alleviate any potential technology risk by making the process observable and auditable for readers to question it.

Through repeated testing and refinement, the article-to-Story clustering approach became reliable enough to serve as the foundational unit of the product’s information architecture. Although not perfect just yet, it established that disconnected reporting could be consistently organized into inspectable event-level coverage that provided a practical path for observation and continued improvement.

Feeling good with these results, I now prepared to tackle the final uncertainty which regarded how to deliver/distribute this data to users.

Experiment 3: Could Personalization Be Transparent and Controllable?

Having established that disconnected articles could be organized into coherent stories, the next question was how readers would navigate all of that information. Some form of ranking and personalization would be necessary, but adopting the conventional personalized algorithmic approach would risk recreating the same power imbalance the product was intended to address: a system that observed the reader, decided what they should see, and provided little visibility into how it reached that decision.

The prevailing assumption around personalization favored scale and complexity. The largest news aggregators had access to enormous content indexes, years of behavioral information, mature infrastructure, and specialized teams responsible for developing their ranking systems. For example, Google described its news algorithms as analyzing hundreds of factors, with some personalized surfaces drawing on activity from across Google products and YouTube.Google: How News Works Google News Help Apple Support

I wanted to test whether this level of complexity and behavioral information was actually necessary to create a useful personalized newsfeed. My hypothesis was that a much smaller set of understandable signals could achieve comparable informational utility where instead of inferring what would maximize a person’s engagement, we let them have control over these signals to set their priorities.

It was important to get this right as transparency could not compensate for a distribution system that lacked basic utility if the resulting feed consistently felt stale, incomplete, or disconnected from the reader’s interests. If this experiment failed, the Story hierarchy, perspective comparisons, Publisher Report Cards, and other features surrounding the feed would become considerably harder to discover and use.

I began by identifying ranking signals that could be calculated consistently and explained in plain language. These were split between “source signals” of an article such as source credibility, the freshness of the reporting, the speed at which a Story was developing, the breadth of publisher coverage, as well as “user preference” signals such as the reader’s topic interests, and publisher preferences. Each signal represented an observable and deterministic property that each article could be evaluated against.

Much of the information needed to calculate these signals already existed because of the previous experiments. MBFC publisher enrichment supported source-credibility scoring, story clusters made it possible to measure coverage breadth and the speed at which reporting was accumulating, and normalized publication times supported freshness. Other requirements only became visible once I began constructing the formula. Some ranking inputs were not yet captured consistently by the aggregation pipeline which required returning to the backend to extend its capability.

The resulting Feed Formula assigned each signal a level of importance and normalized those priorities into a shared weighting system. Each feed item could then receive a relevance score between zero and one hundred based on the reader’s configuration. Additionally, users could inspect how much each signal contributed and see the same calculation that determined the item’s position.

This created a visible connection between a user’s choices and the resulting feed. Increasing the importance of freshness should move newer reporting upward. Prioritizing coverage breadth should favor Stories reported by more independent publishers. Selecting topics or publishers should visibly affect both the ranking and the explanation attached to each result. The relevance-score badge therefore served as more than an abstract rating. It allowed readers to audit whether the system had responded to their instructions.

The experiment also required a different privacy architecture. The reader’s formula and preferences were stored on their iPhone rather than as part of a persistent server-side profile. When the app requested a feed, it temporarily sent the current weights and preferences to the backend which applied them to the shared news corpus and returned the ranked results with their score breakdowns. Those request-specific preferences did not need to be retained as an identity-bound history of the reader. This created a useful separation of responsibilities. The backend maintained an understanding of the news, including its publishers, topics, freshness, coverage, and Story activity and the reader’s device localized their preferences privately.

Testing the formula required evaluating more than whether it computed successfully, but also how the experience generally felt from a user perspective. This included rather arbitrarily how impactful and accurate each signal felt in testing and how that was reflected in the relevance badge feedback.

One particularly important distinction emerged between nominal control and effective control. Testing exposed some cases where preferences could technically influence the formula while producing such a small change that the reader would never recognize its effect. This created a dilemma where if increasing a signal’s importance did not noticeably improve the feed’s composition, the interface would provide the appearance of agency without its substance. This made calibration an ongoing product responsibility involving signal weights, scoring ranges, and defaults of each control.

Within the scope of my testing, the results challenged my assumption about how much complexity useful personalization required. The transparent formula produced a feed that was broadly comparable to existing aggregators in its timeliness, topical relevance, and coverage of major events. The overall experience also exposed greater breadth and depth than a conventional list of personalized headlines because the formula operated together with the infrastructure built during the previous experiments. A ranked item could open into the full Story, the surrounding publisher coverage, and the different framing around it thus creating a more integrated feeling product.

The results demonstrated that useful personalization did not need to operate as a black box and that a comparatively simple system could produce a relevant feed, showing readers exactly which factors shaped it. This allows them to change those factors directly, providing personalization without maintaining a centralized behavioral profile. Although the exact signals, weights, and presentation would continue to evolve through subsequent testing, the validated scoring architecture nevertheless created a foundation for features that could make its control more approachable and expressive.