Invalid results scraped

Bug Reports
#1

Web Scraper version: 0.3.8
Chrome version: Version 68.0.3440.106 (Official Build) (64-bit)
OS: Win 10

Sitemap:

{"_id":"vic-pestcontrol","startUrl":["http://122.252.13.117/environment/pest_licences.asp?z_FIRSTNAME=LIKE%2C%27%25%2C%25%27&z_SURNAME=LIKE%2C%27%25%2C%25%27&z_LICENCE_NUMBER=LIKE%2C%27%25%2C%25%27"],"selectors":[{"id":"Pagination","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#mainContent","multiple":true,"delay":"2000","clickElementSelector":"form#ewpagerform td a:contains(\"Next\")","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Licence Table","type":"SelectorTable","parentSelectors":["Pagination"],"selector":"table.stripeTable","multiple":true,"columns":[{"header":"Licence #","name":"Licence #","extract":true},{"header":"Status","name":"Status","extract":true},{"header":"Expiry","name":"Expiry","extract":true},{"header":"Name","name":"Name","extract":true},{"header":"Authorisation","name":"Authorisation","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr","tableHeaderRowSelector":"thead tr"}]}

Error Message:

Not applicable (there are no error messages)

The site (currently) contains 1562 results; only 159 items are scraped, and they are all duplicates of the final two results.

#2

Because each Next button press loads another page, you dont want to use pagination.

http://122.252.13.117/environment/pest_licences.asp?start=[1-1562:10] as the site to scrape

The easiest thing would be to rewrite it as normal, with a selector for each column

License # - tr.Odd:nth-of-type(n+2) td:nth-of-type(1)
Status - tr.Odd:nth-of-type(n+2) td:nth-of-type(2)
Expiry - tr.Odd:nth-of-type(n+2) td:nth-of-type(3)
Name - tr.Odd:nth-of-type(n+2) td:nth-of-type(4)
Auth - tr.Odd:nth-of-type(n+2) td:nth-of-type(5)

#3

Thanks,
Most of the sites I'm trying to scrape use a link that loads another page and many of these use js to do a callback with form data to get the next page. I guess if webscraper doesn't support this use case I'll have to look elsewhere.
Cheers,
Crewbs

#4

I literally told you how to scrape the whole thing super easily.

The reason you dont want to use pagination on a site like that is because it is going to be a lot less complicated and much faster without it.

But if you dont want to learn that fine

#5

I understand how your suggestion would work for the original case I raised and I thank you for that suggestion.

Unfortunately, that approach won't work for other sites I'm also having pagination issues with. I'd prefer to use a single tool and unfortunately this tool doesn't meet my needs

#6

How do you have pagination setup? whats the config look like and whats the problem that you are getting?

My first post was poorly worded, as in if you know how many pages you want it faster to specify then to push the button.

Pagination is really simple once you understand how to use it.

#7

I always set up pagination as the top level selector and identifying either a "next" type button, or the buttons for each page.

Then where it's just a list, I either use a table selector or for more complex structures, I use individual text selectors to get the data I want. These selectors are under the pagination selector (as shown in the tutorial videos).

The problems I'm getting is, that although it visits each page, I don't get all the data, just a subset repeated over and over. Another site I tried (which uses JS to do postbacks to get each new page), the site has 928 unique records and I could only reliably get 72 unique records out of 1000 records captured. I've tried recreating the sitemaps from scratch and triple checking that my selectors identify only the correct elements, but I get the same issues each time, across multiple sites.