Only scrapes first 3 pages then stops

mikeb · August 8, 2023, 8:58am

I've set up my first scrape which appears to yield data ok. But it stops scraping after 3 pages, even tho there are many more to scrape.
I've seen this problem reported on the forum and have tried various solutions, notably increase the delay on the ElementClick selector (upto 10,000). but no luck.
can someone advise me on possible solutions please? as you can see from the sitemap json its a simple table with an input to click for the next page (the post being managed by js).

Thanks!

Url: GME - Parts Lookup Database

Sitemap:
{"_id":"greg_marsh_parts3_simple","startUrl":["https://www.gregmarsh.com/MC/PartsList.aspx"],"selectors":[{"columns":[{"extract":false,"header":"Tools","name":"Tools"},{"extract":true,"header":"Manual PartNo","name":"Manual PartNo"},{"extract":true,"header":"Brand","name":"Brand"},{"extract":true,"header":"Model","name":"Model"},{"extract":true,"header":"PageNo","name":"PageNo"},{"extract":true,"header":"RefNo","name":"RefNo"},{"extract":true,"header":"PartNo","name":"PartNo"},{"extract":true,"header":"Description","name":"Description"},{"extract":true,"header":"Qty","name":"Qty"},{"extract":true,"header":"Remarks","name":"Remarks"},{"extract":true,"header":"Notes","name":"Notes"},{"extract":false,"header":"Changed By","name":"Changed By"},{"extract":true,"header":"Changed","name":"Changed"}],"id":"table-whole","multiple":true,"parentSelectors":["_root"],"selector":"table#MainContent_ctlList","tableDataRowSelector":"tr:nth-of-type(n+2)","tableHeaderRowSelector":"tr:nth-of-type(1)","type":"SelectorTable"},{"clickElementSelector":"input#MainContent_btnPagerNext","clickElementUniquenessType":"uniqueHTML","clickType":"clickOnce","delay":7000,"discardInitialElements":"do-not-discard","id":"click","multiple":false,"parentSelectors":["_root"],"selector":"#MainContent_btnPagerNext","type":"SelectorElementClick"}]}

leemeng · August 20, 2023, 7:53am

Type: Table would not work for this case and you'll need to build the data scraper for each column. Try this example below where I have built the scraper for 3 columns. You can continue adding the columns you require.

I fixed the paginator too. Tested this sitemap by skipping ahead to page 2335 and it did continue until the last page.

Sitemap:

{"_id":"gregmarsh-b","startUrl":["https://www.gregmarsh.com/MC/PartsList.aspx"],"selectors":[{"id":"Table selector","parentSelectors":["_root","Click next"],"type":"SelectorElement","selector":"tr:contains('Manual PartNo') > td.texttop > table[class^='GMETable']","multiple":false},{"id":"Click next","parentSelectors":["_root","Click next"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"div#MainContent_pnlPager input#MainContent_btnPagerNext:not([disabled])","clickElementUniquenessType":"uniqueText","clickType":"clickMore","delay":5500,"discardInitialElements":"discard-when-click-element-exists","multiple":false,"selector":"td div#pnlMenu + table[class^='GMETable'] "},{"id":"Result rows","parentSelectors":["Table selector"],"type":"SelectorElement","selector":"table#MainContent_ctlList tbody > tr:not([valign])","multiple":true},{"id":"Manual Part No","parentSelectors":["Result rows"],"type":"SelectorText","selector":"td:nth-of-type(2)","multiple":false,"regex":""},{"id":"Brand","parentSelectors":["Result rows"],"type":"SelectorText","selector":"td:nth-of-type(3)","multiple":false,"regex":""},{"id":"Page no","parentSelectors":["Result rows"],"type":"SelectorLink","selector":"td:nth-of-type(5) a","multiple":false,"linkType":"linkFromHref"},{"id":"Current page","parentSelectors":["Table selector"],"type":"SelectorElementAttribute","selector":"tr input#MainContent_txtPager","multiple":false,"extractAttribute":"value"}]}

Sample result: