I’ll develop an automated ✅ web scraping ✅ tool to extract all posts from both forums, But first we need to do some tests to see if there's any rate limiting issue or unstructured data.
You can visit my profile where you can see my work, reviews & more details about the way I handle those tasks.
I can also add additional features to it (If needed):
✅ download the data in any format you may need it in (JSON, CSV, EXCEL, TEXT, XML, Google Sheets…).
✅ provide a dashboard to control and follow up with the process.
✅ connect the script to a Database or API (post it to your website).
Over the years, I managed to develop similar scripts for all sorts of websites & platform.
I use web development technologies to program the tools, In comparison to other solutions like Python, They have the apprehend, Simply because these were made for this particular purpose:
◘ NodeJs (js runtime for the server), ExpressJs (NodeJs framework), MongoDB (NoSQL database).
◘ Puppeteer (browser automation), Cheerio (Html loader) ...
◘ HTML, CSS, JavaScript
◘ VueJs (js framework), NuxtJs (VueJs helper).
◘ AdobeXD, Photoshop, Illustrator… (for design)
contact me with more information then I can give you more details about the process of making such scraping tools & answer your questions.