Sanrith Descartes
You have insufficient privileges to reply here.
- 43,968
- 117,662
Anyone have a good guide to set up scraping apps to hit news/social media sites?
- 1
Scraping data from news/social media sites?
- Thread starter Sanrith Descartes
- Start date
-
Guest, did you like the last podcast notification?? Well here's the second one, and for the third podcast! Yep. This Thursday 8-8-24 at 7pm Central
Go here to check out his podcast thread for info on topics and links!Go into the FoH discord and find the Happy Time Fun Hour channel and give it a listen!
ShakyJake
<Donor>
- 7,884
- 19,826
Sanrith Descartes
You have insufficient privileges to reply here.
- 43,968
- 117,662
I have zero experience with this. Looking to automatically aggregate news from specific news sites and personalities' posts from social media.
- 1
Sanrith Descartes
You have insufficient privileges to reply here.
- 43,968
- 117,662
I am a news junkie and am just too damn busy these days to keep up.
- 3
Edaw
Parody
- 13,175
- 87,201
RSS already does everything you want and How to Create RSS Feeds from Twitter [Updated 2021]
I'm sure free opensource versions exist but gives you some idea what can be done with RSS.
- 2
- 1
Mist
REEEEeyore
- 31,084
- 23,420
Python Web Scraping: Full Tutorial With Examples (2024)
Learn about web scraping in Python with this step-by-step tutorial. We will cover almost all of the tools Python offers to scrape the web. From Requests to BeautifulSoup, Scrapy, Selenium and more.
www.scrapingbee.com
A friend of mine wanted to scrape every biomedical device patent off the FDA website. Took me like 2 hours to teach myself how to do it.
- 3
ExPatriot
This is the end...
- 678
- 2,051
Gotchu...
I have a friend in the Web ad business and he talks about this stuff all the time so I have picked up a bit of understanding...
Will be interested to hear what you end with for a solution!
- 1
Bandwagon
Kolohe
- 24,134
- 64,566
Is there someone that can help me scrap all the tutorials/blog posts/whatever off of this website? Supposedly they're taking all of the terrasolid documents offline in about 2 weeks and I was hoping there was an easy way to have all of them stored?
support.geocue.com
Terrasolid Knowledge Base Center - GeoCue Support Knowledge Base
GeoCue Group's knowledge base filled with hints, tips, tutorials and additional support information for the Terrasolid product line.
support.geocue.com
ShakyJake
<Donor>
- 7,884
- 19,826
There used to be a way to make a website available offline. Could you do that instead? Is this all static content?Is there someone that can help me scrap all the tutorials/blog posts/whatever off of this website? Supposedly they're taking all of the terrasolid documents offline in about 2 weeks and I was hoping there was an easy way to have all of them stored?
Terrasolid Knowledge Base Center - GeoCue Support Knowledge Base
GeoCue Group's knowledge base filled with hints, tips, tutorials and additional support information for the Terrasolid product line.
ToeMissile
Pronouns: zie/zhem/zer
- 3,138
- 2,024
Looks like the site uses wordpress, maybe some easy options here:
Bandwagon
Kolohe
- 24,134
- 64,566
Thanks. I used one of the programs listed in those search results and fiddle fucked with it for about an hour this morning, but couldn't figure out how to get it to iterate and download an offline page for each. I'll try messing with it again when I get home tonight
- 1
- 12,578
- 48,907
Depending on how you go about analyzing and what use you're going to do with it, have you considered a tool that simply crawls the site and stores a local copy for "offline viewing" then go at it from that front? I'm looking at a similar idea because I want to keep an offline archive of certain sites to use with an LLM model I'm working on as it's "knowledge store".
(I chime in almost a year later...)
(I chime in almost a year later...)
- 1