Hello,
I am trying to scrape products on Amazon.com and I have two issues that I could use some assistance with.
First, when I try and scrub price data, I get null for a lot of listings where the price field is slightly in a different location, to work around this I was able to select a larger field that ended up fixing that issue, but now it duplicates the price. Ex. $9.99$9.99, is there a way I can eliminate this duplication?
The second issue is that the original URL shows the pagination to be 6 pages long, but once it reaches the final page, it adds more pages to scrub with similar products. These are not what I want in the scrub, so is there a way to stop the pagination after that 6th page?
Thank you for any assistance provided!
{"_id":"amazon","startUrl":["https://www.amazon.com/s?k=visiontek&rh=n%3A172282%2Cp_89%3AVisionTek&dc&ds=v1%3A16KshzXXXIHkC0ABDZzcDmsf%2BwKwOyKAtYmDcilo620&crid=29QHTXVEXJU5E&qid=1668446531&rnid=2528832011&sprefix=visiontek%2Caps%2C109&ref=sr_nr_p_89_1"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"auto","selector":"svg[xmlns]","type":"SelectorPagination"},{"id":"element-card","parentSelectors":["pagination"],"type":"SelectorElement","selector":"div.sg-col-4-of-24.s-result-item:nth-of-type(n+2) div.sg-col-inner","multiple":true},{"id":"link","parentSelectors":["element-card"],"type":"SelectorLink","selector":".a-size-mini a","multiple":false},{"id":"element","parentSelectors":["link"],"type":"SelectorElement","selector":"div.a-container","multiple":true},{"id":"description","parentSelectors":["element"],"type":"SelectorText","selector":"span.product-title-word-break","multiple":false,"regex":""},{"id":"price","parentSelectors":["element"],"type":"SelectorText","selector":"#corePrice_feature_div div","multiple":false,"regex":""},{"id":"ASIN","parentSelectors":["element"],"type":"SelectorText","selector":"tr:contains('ASIN') td","multiple":false,"regex":""},{"id":"review-stars","parentSelectors":["element"],"type":"SelectorText","selector":".a-size-base i.a-icon-star","multiple":false,"regex":""},{"id":"part number","parentSelectors":["element"],"type":"SelectorText","selector":"span.product-title-word-break","multiple":false,"regex":"[0-9][0-9][0-9][0-9][0-9][0-9]"}]}