Web scraping with Powershell is very easy, this example will scrape all headlines from the Daily Telegraph website and print them to the terminal.
This is using the invoke-RestMethod cmdlet. Plus using a regex to match certain HTML entities.
([regex]::Matches((Invoke-RestMethod "https://www.dailytelegraph.com.au/"), '(?<=<p class="storyblock_standfirst g_font-body-s">).*?(?=</p>)') | ForEach-Object { $_.Value }) |
PS C:\Users\Intel i5> ([regex]::Matches((Invoke-RestMethod "https://www.dailytelegraph.com.au/news/nsw"), '(?<=data-tgev-container="feed-1">).*?(?=</a>)') | ForEach-Object { $_.Value }) Surprise attack: Hamas in brazen dawn raid on Israel WAGS Guide: The ladies who get Premiership Panthers purring South coast picks up pieces after first houses burn Why families are ditching playgrounds for wildlife Put your phones away: School crackdown to start Last days of three siblings and grandfather lost in plane crash The faces of Bathurst Bathurst 1000 fans show their true colours Winging it! Aussie who skipped uni to become SailGP star ‘It was terrible’: Slow motion agony for Aussie SailGP team First Nations voters divided on Voice too The Sell: Shark snares ‘sleek’ beachside entertainer The Sauce: NSW Liberals reveal how they will vote on the Voice Pullin: ‘My mini-me makes me smile every day’ How to make a Tim Tam Fit, fun and fabulous: Why turning 50 has never felt better ‘Not just for veterans’: The changing face of PTSD in Australia Where the magic is made: A trip inside the Tim Tam factory Teens in horror car crash, Bayview $200k bump: the value of homes in every Sydney suburb revealed Voice: Every place you can vote early to avoid queues Kean to Young Libs: Ignore your mum and dad on the Voice Inside Slave House Mum’s anger after public school hits capacity after three years Yes case fury at lack of ALP help on Voice |
This is another example that prints the subject line of each headline.
PS C:\Users\Intel i5> ([regex]::Matches((Invoke-RestMethod "https://www.dailytelegraph.com.au/"), '(?<=<p class="storyblock_standfirst g_font-body-s">).*?(?=</p>)') | ForEach-Object { $_.Value }) Israel is fighting to remove the Hamas militant group from inside its territory, as it races against time to declare it has restored “sovereignty and order”. Follow the updates. Hamas forces stormed through Israel’s defences just before 6am. Palestinian gunmen were soon walking through towns, battering down front doors — and taking people away. How did it unfold? And how did it escalate so quickly? Hundreds of people are dead after a surprise attack on Israel this weekend, but at home it’s the statements made by the PM and Foreign Minister that have drawn criticism. They’ve won three NRL Premierships in a row and don’t look like slowing down anytime soon, but who will they be celebrating with this off season? Meet the partners of the Panthers’ top 30 roster and find out who is in their bachelors’ club. Sharks superstar Nicho Hynes was just starting his pre-season holiday on the Sunshine Coast when he was forced to bail on his Airbnb – but he had a very good reason. Bulldog’s highlights, lowlights. Big names fell over in a typical brutal Bathurst 1000, and the chaos threatened to take over the Great Race – but a thrilling finish looms as it goes down to the wire. Follow live inside. The resignation of high profile Sydney columnist Joe Aston and his quick exit has raised questions around his departure with some of the view he has self-sabotaged and blown himself up with a column too vicious for bosses to ignore, writes Annette Sharp. Nathan Merritt’s sister Teanne has updated the South Sydney great’s condition after he was placed on life support following being hospitalised on Friday afternoon. Polling from inside the No campaign has cast doubt on a central claim of the Yes camp that the Voice is supported by an overwhelming majority of Indigenous Australians. Terry Biviano and Anthony Minichiello’s Vaucluse house renovations continue to drag on although there has been a commitment as to when it will be finished. The two hottest tickets in town for more than two decades — Bondi Icebergs and Catalina’s New Year’s Day soiree’s — are done and dusted. With just days until the first HSC exam, here’s some last-minute advice from experts including exam markers, top teachers, elite students and tutors. Read the the ultimate guide to success. In a wide-ranging interview, NRL CEO Andrew Abdo opens up on the league’s plans for expansion, the threat of rugby union and the Vegas dream that has been three years in the making. Fashion designer Wayne Cooper has found love again, with the British-born father of two proposing to his partner of seven months. A startling rise in the number of people reporting being a victim of slavery or human trafficking has police devising new tactics to counter the surge. The good news for those who think the government has been talking too much about Indigenous issues is that, after next Sunday, Canberra ‘needs to get back on to cost-of-living’, writes James Campbell. As the debate around the Voice gets more abusive, with Ray Martin joining the fray this week, what is even more evident is that PM Anthony Albanese is clueless, writes Peta Credlin. The meaning of turning 50 has changed dramatically. Turning 50 is quite the milestone – and no longer is it mid life, near retirement or ‘old’ in any sense. For some, it’s a time to start again and set new goals, writes Lisa Mayoh. |
PS C:\Users\Intel i5> ([regex]::Matches((Invoke-RestMethod "https://www.news.com.au/"), '(?<=data-tgev-container="tops">).*?(?=</a>)') | ForEach-Object { $_.Value }) Dashcam video reveals terrorists’ brutality Brave act saves woman from Hamas PM tears up: ‘There’s no alternative’ Heartbreaking update for WSJ reporter Cocaine Cassie dramatically quits TV show Craig McLachlan reveals secret fallout of trial AFL premiership skipper’s Voice vote reveal Scary sign ‘bloodiest’ war has ‘barely started’ ‘Seamless’ trade hits snag as Swans rejected Disturbing truth behind Hamas victims Unlikely celeb duo travel mates on new show Boyfriend’s ‘typo’ exposes what men want NRL great selling private beach, mum’s house |
It is also possible to do this on Linux. Use the htmlq program to web scrape easily from a website.
Firstly install Cargo.
[root@localhost Documents]# dnf in cargo |
Then install the htmlq utility.
(jcartwright@localhost) 192.168.1.5 Documents $ cargo install htmlq Updating crates.io index Ignored package `htmlq v0.4.0` is already installed, use --force to override warning: be sure to add `/home/jcartwright/.cargo/bin` to your PATH to be able to run the installed binaries |
Then you may begin parsing text from a website. I am choosing an A tag with the CSS class of ‘storyblock_title_link’. This will automatically get the value of the HTML entity.
(jcartwright@localhost) 192.168.1.5 Documents $ curl --silent -L https://www.news.com.au | htmlq a.storyblock_title_link --text ‘CANNOT IMAGINE HOW HORRIFIC’: Gory find after festival ‘massacre’ ‘It’s over’: New polls spell death for Voice Formula 1 tension explodes in ugly scenes One thing Aussie star did to lose 40kg Horror find inside suburban home Australia exposed in World Cup nightmare Aussie star caught up in Israel attacks Olivia Wilde slammed for Taylor dig Photos show Beckhams’ history is repeating Super change to benefit 8.9m Aussies Ray Martin turns around ‘d***head’ attack Hamas fighters parade captured woman Two European nations on brink of war $60m special place Musk can’t get rid of Tobacco store rammed, set ablaze Israel, Gaza reel as death toll soars above 1,000 in war with Hamas Horror find inside suburban home Can you score 90% on our all-Aussie travel challenge? Abbie’s restaurant blow-up over racy outfit Voice coach hits back: ‘False and hurtful’ ‘Never speak’: Kanye’s rules for Aussie wife King Charles’ shock new title for Meghan Star comes out, announces death of partner ‘Alarming’ statement from Yes campaign What Aussies need to know about the Voice Truth about ‘d***head’ No vote comment Map exposes suburbs voting No in Voice Truth behind ad exploding on social media Where you can vote in referendum today Voice winner‘s incredible final performance ATO warning as deadline looms |
Here is another example.
(jcartwright@localhost) 192.168.1.5 Documents $ curl --silent -L https://www.news.com.au/technology/online/internet | htmlq a.storyblock_title_link --text Win for tattooed, diamond-toothed influencer Optus to axe 100 Aussie call centre staff ‘Cancel. My. Service’: Wild Optus exchange Big crackdown on video game gambling Internet in meltdown over gaming ‘scam’ ‘Days’: Update after cop’s Croatia fall Aussies’ info leaked in Pizza Hut hack ‘No question’: Warning on rise of AI PM called out over porn passport move Aussies’ surprising take on 5G phones Telstra credit ‘outage’ fixed Child abuse fiends to face more scrutiny The device Kim K couldn’t do without Millions of Aussies to be hit in 3G shutdown Cyber warning for Aussie businesses Aussies paying for child abuse game: cops Satellite internet trial transforms rural farming Campsite detail reveals Aussie trend Thousands of Google accounts deleted Millions of Aussies to get free internet boost Albo’s bold plan for one group Man charged over alleged $176k cyber scam Man threatens to kill MP: court Government docs leaked onto dark web Death threats allegedly sent to MP Another sign Aussies are doing it tough TikTok star’s cocaine video revealed Major update for Aussie punters Woman’s wild offer to score pop-up pastry |
This is another way to get the value of a tag with a certain CSS class.
(jcartwright@localhost) 192.168.1.5 Documents $ curl --silent -L https://www.news.com.au/technology/online/internet | htmlq '.storyblock_title_link' --text Win for tattooed, diamond-toothed influencer Optus to axe 100 Aussie call centre staff ‘Cancel. My. Service’: Wild Optus exchange Big crackdown on video game gambling Internet in meltdown over gaming ‘scam’ ‘Days’: Update after cop’s Croatia fall Aussies’ info leaked in Pizza Hut hack ‘No question’: Warning on rise of AI Aussies’ surprising take on 5G phones Telstra credit ‘outage’ fixed Child abuse fiends to face more scrutiny Australia ‘caving to porn industry’ The device Kim K couldn’t do without Millions of Aussies to be hit in 3G shutdown Cyber warning for Aussie businesses Aussies paying for child abuse game: cops Satellite internet trial transforms rural farming Campsite detail reveals Aussie trend Thousands of Google accounts deleted Millions of Aussies to get free internet boost Albo’s bold plan for one group Man charged over alleged $176k cyber scam Man threatens to kill MP: court Government docs leaked onto dark web Death threats allegedly sent to MP Another sign Aussies are doing it tough TikTok star’s cocaine video revealed Major update for Aussie punters Woman’s wild offer to score pop-up pastry |
How to list all the most recent TikTok videos for a certain account.
(jcartwright@localhost) 192.168.1.5 Documents $ curl --silent -L https://www.tiktok.com/@howridiculous | htmlq span.tiktok-j2a19r-SpanText --text | awk NF Sports Racing is CRAZY! 😲Giant Axe Vs. Giant Rock This Took SO Long to Set Up 😲Blending our TV 😳That last throw 🤯Fridge vs Giant Blender😱Can't Believe He MISSED ONE! GIANT AXE VS ANVIL!! Consequence Racing is INTENSE! #1 took 2389 attempts...... Balloon Pop MARATHON! Vehicles You SHOULDN'T Race With! This Shot Took 13 Years... Axe Throw RACING is NEXT LEVEL... Wife Vs Husband House Chores Race?! Sports Racing is INTENSE!! |
Another example.
(jcartwright@localhost) 192.168.1.5 Documents $ curl --silent -L https://www.tiktok.com/@treyyingst | htmlq span.tiktok-j2a19r-SpanText --text | awk NF Anothe update from southern Israel Military equipment in southern Israel Another update from southern Israel Another update from southern Israel Another update from southern Israel Update from southern Israel Update from southern Israel We continue to report from the Gaza border Apocalyptic scenes right now in southern Israel. BREAKING: Rocket fire targeting Tel Aviv right now. Multiple interceptions as air raid sirens sound. Russia targets Ukrainian civilians, killing nearly 50 people. Ukraine looks for more support as the Russian invasion continues. What do you think happens next? An attack in Ankara left two police officers injured Exclusive reporting from inside an Israeli military raid Can’t put full report on TikTok or video will be removed. Full video on my Instagram. Exclusive access to a night time raid conducted by the Israeli military. Ukraine to recieve 31 American tanks |
This should be very useful indeed. This is how to get this information with the command line.