Hi,
do this through VBA is not a good choice, the best solution is do web automation.
What do I mean?
I mean write a program, it will open a browser and control the browser do navigation automatically. Selenium is a good solution to do such things.
You can think,
The program open browser itself, navigate to the page itself, login itself.
When the program reach the destination (the web page where the replies are located at), it will find proper elements through XPath.
When the program complete 1 page, it will sleep a long duration (for example, 4 minutes, 8 minutes... the value is random, so the website cannot find it's a bot, because it's like a human surfing on the website and view their pages)
Then the program go to next page and scrape again, then sleep again.
....
Until all pages complete, the biggest problem is: the program will take more than 1 hour to scrape only 20 pages...
Do you think is OK? If you feel it's safe to shorter the sleep time (such as only sleep 1 minute or 2 after visit every page), you can do so.
I write program with C#, run on Windows, need Chrome browser installed.
Looking to talk with you. Could you hire me with a low budget (such as 50 USD), then add more milestones into the contract after the initial work done?
Thank you.