I have a site that seems to need a referrer. So if you run the page as in domain.com/...page=[4-6].... it's not found but when clicking on the link on the page it is. Could also be that something is sent via post.
How to emulate this OR scrape the currently shown page?