Original Data Pipeline: How AI Collects Proprietary Evidence for Every Article

AI engines have been trained on the entire public internet. When ChatGPT or Claude encounters a page that restates what they already know from thousands of other sources, there is no reason to cite that specific page. The information is redundant. The page adds nothing.

Original data changes the equation. When your article contains a statistic from a government database published last month, a contradiction between two named industry experts sourced from different publications, or a trend analysis synthesized from 15 Reddit threads - the AI engine encounters information it cannot get elsewhere. That is what triggers a citation.

Here is the test we apply to every paragraph: remove the brand name and ask whether this content could appear on a competitor's site. If a home health care company publishes "the industry is growing rapidly and technology is improving outcomes" - that sentence could appear on any of 500 competing sites. It is not original. But if the same company publishes "CMS reduced the home health payment rate by 1.7% for 2026, while agencies using remote patient monitoring report 23% fewer 30-day readmissions according to JAMA Network" - that specific synthesis of current data points, with sources, is original analytical work.

The problem is that producing original data manually is expensive. Researching current statistics, finding expert contradictions, tracking sentiment shifts across Reddit and industry forums - this takes hours per article. Most content teams skip it and default to whatever the LLM generates from training data. The Original Data Pipeline automates the entire process, making every article rich with proprietary evidence by default.

The Artifact Generation System is block B-47 in the content pipeline - a Web Intelligence Collector that runs automatically during article creation as part of Tier 1 (before any writing blocks execute). It collects live data from up to 15 source types, condenses each piece into structured extractions, and assembles a corpus that feeds into the synthesis stage.

Source collection operates in two tiers. Tier A sources always run because they are free or very cheap with high signal: NewsAPI (10 current articles), HackerNews via Algolia (5 stories with comments), and Google Scholar via ScrapingDog SERP (5 academic papers and abstracts). Tier B sources are industry-selected - an AI model picks 3-5 source categories based on the article topic and client industry. Options include industry publications (3 articles from each of 3 publications), government and education sites (3 documents), Reddit threads (5 threads with comments), Substack and Medium articles, blogs and forums, and local or regional news.

Every scraped piece goes through LLM condensation using Haiku for cost efficiency. The condensation prompt extracts every meaningful fact, data point, statistic, expert opinion, prediction, entity mention, disagreement, and trend indicator. The output is structured into six categories: key facts with source attribution, opinions and predictions, entities with roles, notable quotes with attribution, trends and signals, and methodology references. This is comprehensive fact extraction - not summarization.

After condensation, deduplication removes redundant results using URL matching and title fuzzy matching. The deduplicated corpus - typically 30 to 50 condensed sources - gets stored in the execution metadata and passed to B-46, the Insight Synthesizer, which produces a 13-section structured artifact with themes, entity maps, predictions at three time horizons, contradictions between sources, trend analysis, interesting standalone facts, and contrarian takes.

Before the Original Data Bridge, three data systems existed in isolation. Knowledge items lived in their own table and only the Reddit drafter consumed them. Client insights were manually entered and fed into B-46 and writing blocks. Mentions and Reddit scan data powered the visibility UI but never reached the article pipeline. Three pools of proprietary data, none talking to each other.

The bridge unifies all three through the knowledge items table as the universal original data layer. Mentions scans and Reddit scans now automatically generate knowledge items after every run. When you scan a domain's brand mentions across the web, the system creates data point items with current citation counts, sentiment trends, and competitor comparison metrics. When a Reddit scan finds threads discussing the client's industry, it creates case study and expert items from the highest-signal discussions.

These knowledge items then flow into the article pipeline through the same channel as manually entered client insights. When B-46 runs, it receives the full set of knowledge items alongside the live corpus from B-47. Writing blocks like B-05 (body sections) and B-07 (data visualization) also receive them. A data point knowledge item becomes a UNIQUE_CLAIM with type proprietary_data. A case study item becomes an OWNED_INSIGHT. An expert item gets attributed by name.

The practical result: a client who has run a Mentions scan and a Reddit scan before creating their first article will see that article automatically incorporate real brand mention data, actual Reddit sentiment, and competitive positioning - all without anyone manually typing a single insight. The pipeline fetches up to 30 knowledge items per domain, grouped by type, and injects them with clear instructions to weave them naturally and never fabricate beyond what the items actually state.

Fabrication prevention is built into every stage of the pipeline, not bolted on as a final check. The system operates on a fundamental constraint: every claim must trace back to a verifiable source URL.

At the collection stage, every scraped piece retains its source URL, publish date, and domain. The condensation prompt explicitly instructs the model to extract facts with source attribution - if a fact cannot be attributed to the specific source being condensed, it does not enter the corpus. This prevents the LLM from injecting its own training data during the extraction phase.

Deduplication runs after condensation and before synthesis. It operates on two dimensions. URL-level deduplication catches the same page appearing across multiple search queries or source types. Title-level fuzzy matching catches the same story reported by different outlets - keeping the most detailed version and discarding near-duplicates. This prevents the synthesis stage from over-weighting a single data point that appeared in multiple sources.

During synthesis, the 13-section artifact structure enforces source grounding at every level. Each theme must list source URLs as key evidence. Each prediction must include a reasoning chain referencing specific sources. Each contradiction must name both sides with their respective source URLs. The interesting facts section requires a source URL and attribution for every entry. If the model cannot ground a claim in the collected corpus, the claim does not make it into the artifact.

The writing blocks that consume the artifact inherit this provenance chain. When B-05 writes a paragraph citing a statistic, that statistic traces back through the artifact to the condensed extraction to the original scraped source. The system does not generate claims and then look for sources to support them. It collects sources first, extracts their claims, and then synthesizes those specific claims into article content.

The pipeline executes five stages during every article creation, running automatically without manual intervention.

Stage 1 - Scan Sources. When article creation begins, B-47 triggers source selection. An AI model receives the article topic, client industry, and client region, then selects 5-7 source categories from the registry. It returns specific parameters for each: industry publication domain names, subreddit names, regional news qualifiers. The three Tier A sources (NewsAPI, HackerNews, Google Scholar) always run regardless of selection.

Stage 2 - Collect and Condense. The system fires parallel requests to all selected sources using the existing ScrapingDog infrastructure for SERP queries and page scraping. Rate limiting uses a token bucket pattern shared with the Mentions system. Each scraped result goes through Haiku-powered condensation in parallel, producing structured extractions. The typical yield is 30 to 50 condensed source documents.

Stage 3 - Deduplicate and Validate. URL-level and title-level fuzzy matching removes redundant results. Source URLs are validated as resolvable. The deduplicated corpus is stored in the execution metadata so it persists with the article and can be inspected later.

Stage 4 - Synthesize Artifact. B-46 receives the deduplicated corpus alongside any existing knowledge items from the Original Data Bridge. It produces the 13-section artifact: executive summary, themes with source counts and emerging/sustained/fading status, sentiment analysis by source type and entity, entity map with roles and positions, predictions at short/medium/long time horizons with confidence levels, defensible theses with counter-arguments, temporal trends, contradictions between sources, statistical summary with chart data, interesting standalone facts, image generation prompts, contrarian takes, and identified knowledge gaps.

Stage 5 - Inject into Blocks. The artifact and knowledge items flow into every writing block. B-04 (introduction) uses the executive summary and top theme. B-05 (body sections) weaves in specific facts, contradictions, and entity context. B-07 (data blocks) renders chart data and statistical summaries. B-09 (FAQ) draws from interesting facts and contrarian takes. The result is an article where every substantive claim traces back to a live source collected during that specific article's creation.

• Schema.org Article Type Reference -schema.org/Article
• Schema.org CreativeWork isBasedOn Property -schema.org/isBasedOn
• Google Article Structured Data -developers.google.com/search/docs/appearance/structured-data/article
• Anthropic Claude Models Overview -docs.anthropic.com/en/docs/about-claude/models
• OpenAI ChatGPT Web Search -platform.openai.com/docs/guides/tools-web-search