Scrape UPC with a spec table clik

ricco · July 25, 2023, 11:15am

scraping UPC and IMGE URL from spec table click. currently running scrap but looks like it runs to last page and only scrapes that data?

https://humminbird.johnsonoutdoors.com/us/shop/accessories

Sitemap:

{"_id":"HUMMIN-JOHN-ACC","startUrl":["https://humminbird.johnsonoutdoors.com/us/shop/accessories"],"selectors":[{"id":"element","parentSelectors":["pages"],"type":"SelectorElement","selector":"div.views-row","multiple":true},{"id":"pages","parentSelectors":["_root","pages"],"paginationType":"auto","selector":".pager__item--next span[aria-hidden]","type":"SelectorPagination"},{"id":"link","parentSelectors":["element"],"type":"SelectorLink","selector":".product-plp__title a","multiple":false,"linkType":"linkFromHref"},{"id":"spec-click","parentSelectors":["link"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"a#tab-specs","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","multiple":true,"selector":"a#tab-specs"},{"id":"details","parentSelectors":["link"],"type":"SelectorText","selector":"tr:contains('Model') td:nth-of-type(2)","multiple":false,"regex":""},{"id":"sku-UPC","parentSelectors":["link"],"type":"SelectorText","selector":"tr:contains('UPC') td:nth-of-type(2)","multiple":false,"regex":""},{"id":"image-main-url","parentSelectors":["link"],"type":"SelectorImage","selector":".col-md-6 img","multiple":false},{"id":"features","parentSelectors":["link"],"type":"SelectorText","selector":".uncoltable .uncode_text_column ul","multiple":false,"regex":""},{"id":"warranty","parentSelectors":["link"],"type":"SelectorText","selector":"tr:contains('Warranty') td.value","multiple":false,"regex":""}]}

ricco · July 26, 2023, 8:54am

oh so cause its in a table on the page I don't need to actually click the spec tab, I can just scrape it, but my issue was it skipped the first two pages of results and scrapped the data from 3rd page on?

ricco · July 26, 2023, 9:53am

3HAT0K:

{"_id":"HUMMIN-JOHN-ACC","startUrl":["https://humminbird.johnsonoutdoors.com/us/shop/accessories?page=2"],"selectors":[{"id":"element","multiple":true,"parentSelectors":["_root"],"selector":"div.views-row","type":"SelectorElement"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["element"],"selector":".product-plp__title a","type":"SelectorLink"},{"id":"details","multiple":false,"parentSelectors":["link"],"regex":"","selector":"tr:contains('Model') td:nth-of-type(2)","type":"SelectorText"},{"id":"sku-UPC","multiple":false,"parentSelectors":["link"],"regex":"","selector":"tr:contains('UPC') td:nth-of-type(2)","type":"SelectorText"},{"id":"image-main-url","multiple":false,"parentSelectors":["link"],"selector":".col-md-6 img","type":"SelectorImage"},{"extractAttribute":"","id":"features","parentSelectors":["link"],"selector":"#collapse-features li","type":"SelectorGroup"},{"id":"warranty","multiple":false,"parentSelectors":["link"],"regex":"","selector":"tr:contains('Warranty') td:nth-of-type(2)","type":"SelectorText"}]}

do I add the pagination again at _root > element or _root > element > link

ricco · July 29, 2023, 3:52am

hey was working on this again where do I add the pagination again at _root > element or _root > element > link