Sanrith Descartes
Veteran of a thousand threadban wars
- 41,469
- 107,524
Anyone have a good guide to set up scraping apps to hit news/social media sites?
- 1
Scraping data from news/social media sites?
- Thread starter Sanrith Descartes
- Start date
ShakyJake
<Donor>
- 7,627
- 19,255
Sanrith Descartes
Veteran of a thousand threadban wars
- 41,469
- 107,524
I have zero experience with this. Looking to automatically aggregate news from specific news sites and personalities' posts from social media.
- 1
Sanrith Descartes
Veteran of a thousand threadban wars
- 41,469
- 107,524
I am a news junkie and am just too damn busy these days to keep up.
- 3
Edaw
Parody
- 12,290
- 77,813
RSS already does everything you want and How to Create RSS Feeds from Twitter [Updated 2021]
I'm sure free opensource versions exist but gives you some idea what can be done with RSS.
- 2
- 1
Mist
Eeyore Enthusiast
- 30,410
- 22,189
Web Scraping with Python: Everything you need to know (2022)
Learn about web scraping in Python with this step-by-step tutorial. We will cover almost all of the tools Python offers to scrape the web. From Requests to BeautifulSoup, Scrapy, Selenium and more.
www.scrapingbee.com
A friend of mine wanted to scrape every biomedical device patent off the FDA website. Took me like 2 hours to teach myself how to do it.
- 3
ExPatriot
This is the end...
- 442
- 1,268
Gotchu...
I have a friend in the Web ad business and he talks about this stuff all the time so I have picked up a bit of understanding...
Will be interested to hear what you end with for a solution!
- 1
Bandwagon
Kolohe
- 22,725
- 59,582
Is there someone that can help me scrap all the tutorials/blog posts/whatever off of this website? Supposedly they're taking all of the terrasolid documents offline in about 2 weeks and I was hoping there was an easy way to have all of them stored?
support.geocue.com
Terrasolid Knowledge Base Center - GeoCue
GeoCue Group's knowledge base filled with hints, tips, tutorials and additional support information for the Terrasolid product line.
support.geocue.com
ShakyJake
<Donor>
- 7,627
- 19,255
There used to be a way to make a website available offline. Could you do that instead? Is this all static content?Is there someone that can help me scrap all the tutorials/blog posts/whatever off of this website? Supposedly they're taking all of the terrasolid documents offline in about 2 weeks and I was hoping there was an easy way to have all of them stored?
Terrasolid Knowledge Base Center - GeoCue
GeoCue Group's knowledge base filled with hints, tips, tutorials and additional support information for the Terrasolid product line.
ToeMissile
Pronouns: zie/zhem/zer
- 2,707
- 1,652
Looks like the site uses wordpress, maybe some easy options here:
Bandwagon
Kolohe
- 22,725
- 59,582
Thanks. I used one of the programs listed in those search results and fiddle fucked with it for about an hour this morning, but couldn't figure out how to get it to iterate and download an offline page for each. I'll try messing with it again when I get home tonight
- 1
- 11,043
- 41,724
Depending on how you go about analyzing and what use you're going to do with it, have you considered a tool that simply crawls the site and stores a local copy for "offline viewing" then go at it from that front? I'm looking at a similar idea because I want to keep an offline archive of certain sites to use with an LLM model I'm working on as it's "knowledge store".
(I chime in almost a year later...)
(I chime in almost a year later...)
- 1