Thread Starter
Mntz
(@mntz)
Yeah, but our scraper is a registered user that can log in 🙂
But it’s ok, we’ve modified the sphider to ignore meta and robots.txt
Thread Starter
Mntz
(@mntz)
Hi Russell Mann,
Did you have any luck automating it in the meantime?