Scrape and Summarize News Posts Without RSS

Scheduled trigger → fetch list page, extract links and dates → filter recent posts → fetch each article → GPT-4 summary and keywords → save to NocoDB

Who

Content operators or market researchers who need to monitor news sites that lack RSS feeds

You get

Automatically grab latest news weekly, generate under-70-word summaries and 3 technical keywords, store structured results in a database

You need

n8n

Steps

1. Set up schedule trigger and target site

   Add a Schedule Trigger node in n8n, set to weekly (e.g. Wednesday 04:32). Add an HTTP Request node with the target news list URL (example: https://www.colt.net/resources/type/news/), response format set to text.

2. Extract article links and dates

   Use HTML nodes to extract from the list page: 1) article links (CSS selector targeting a tag href, returnArray=true); 2) publication dates (separate CSS selector, returnArray=true). Then use Item Lists nodes to split each array into individual JSON items (field names set to Link and Date respectively).

3. Merge and filter recent posts

   Merge link and date items by position with a Merge node. Add a Code node running JavaScript to filter: calculate current date minus lookback days (e.g. 7, must sync with cron), keep items where Date is within range.

   Execution prompt · Source text。Code node jsCode field

   Copy prompt

   const currentDate = new Date();
   const sevenDaysAgo = new Date(currentDate.setDate(currentDate.getDate() - 70)); // Change the number of days going back to your liking (e.g. from -7 to -1) -> BUT sync with the cron job (first node)
   
   const filteredItems = items.filter(item => {
       const postDate = new Date(item.json["Date"]); // Assuming "Date" is the field name in the extracted html
       return postDate >= sevenDaysAgo;
   });
   
   return filteredItems;

4. Fetch and extract individual articles

   For each filtered article, use an HTTP Request node with {{ $json["Link"] }} as URL to fetch detail HTML. Use an HTML node to extract title (e.g. h1.fl-heading > span:nth-child(1)) and content (e.g. .fl-node-5c7574ae7d5c6 > div:nth-child(1)), then merge with date and link via a Merge node.

5. Generate summary and keywords with GPT-4

   Call two OpenAI nodes in parallel (model gpt-4-1106-preview): Summary node takes content and generates under-70-word summary; Keywords node extracts 3 most important technical keywords (keywords only, no explanations). Use Set nodes to rename outputs to summary and keywords, then merge them.

   Execution prompt · Source text。OpenAI Summary node prompt.messages[0].content

   Copy prompt

   =Create a summary in less than 70 words {{ $json["content"] }}

   Execution prompt · Source text。OpenAI Keywords node prompt.messages[0].content

   Copy prompt

   =name the 3 most important technical keywords in {{ $json["content"] }} ? just name them without any explanations or other sentences

6. Merge data and write to NocoDB

   Merge article metadata (title, date, link) with GPT-4 outputs (summary, keywords) using a Merge node. Add a NocoDB node, configure projectId, table, authentication (nocoDbApiToken), map fields: News_Source=Colt, Title={{ $json["title"] }}, Date={{ $json["Date"] }}, Link={{ $json["Link"] }}, Summary={{ $json["summary"] }}, Keywords={{ $json["keywords"] }}, operation set to create.

Build an n8n workflow that: 1) periodically fetches a news list page; 2) extracts article links and dates with CSS selectors; 3) filters for recent posts; 4) fetches each article body; 5) calls GPT-4 for summary and keywords; 6) writes results to NocoDB. You need your own OpenAI API account and NocoDB credentials; CSS selectors must be adapted to the actual target site.