Next Page pagination not scraping any results

How to?
#1

Hi guys, trying to scrape some data from this website that requires a click more button. Trying to run the sitemap from below does not return any results. I have tried it without the pagination selector and this works in returning results from the first page. I have also tried this sitemap with a element click component to help pause the webpage, but this results in an infinite loop where the scraper never finishes.

Url: https://cloudmarketplace.oracle.com/marketplace/en_US/homePage.jspx

Sitemap:
{"_id":"oracle-marketplace-links-test-3","startUrl":["https://cloudmarketplace.oracle.com/marketplace/en_US/productHomePage"],"selectors":[{"id":"wrapper","parentSelectors":["page"],"type":"SelectorElement","selector":"div.col-lg-12.col-xs-12","multiple":true},{"id":"link","parentSelectors":["wrapper"],"type":"SelectorLink","selector":"a","multiple":false,"linkType":"linkFromHref"},{"id":"blurb","parentSelectors":["wrapper"],"type":"SelectorText","selector":"span.x2dg","multiple":false,"regex":""},{"id":"review","parentSelectors":["wrapper"],"type":"SelectorText","selector":"span.reviewCount","multiple":false,"regex":""},{"id":"licence","parentSelectors":["wrapper"],"type":"SelectorText","selector":"span.x28y","multiple":false,"regex":""},{"id":"page","parentSelectors":["_root","page"],"paginationType":"auto","type":"SelectorPagination","selector":"img.flipRTLImage"}],"websiteStateSetup":{"enabled":true,"performWhenNotFoundSelector":"div.paginationTextPaddingLargescreenOnly","actions":[{"selector":"img#pt1\:listResults\:recentApps\:i2","type":"click"},{"selector":"img#pt1\:keyWordSearch\:cil4\:\:icon","type":"click"}]}}

#2

seems like all data is in iframe without any clue how to reach it...

here is a sitemap to click over all pages:

{"_id":"oracle-marketplace-links-test-3","startUrl":["https://cloudmarketplace.oracle.com/marketplace/oci?category=all"],"selectors":[{"id":"wrapper","multiple":true,"parentSelectors":["page"],"selector":"div.smallTileOuterContainerMob","type":"SelectorElement"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["wrapper"],"selector":"a.smallTileGoLinkStyle","type":"SelectorLink"},{"id":"blurb","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"span.x2dg","type":"SelectorText"},{"id":"review","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"span.reviewCount","type":"SelectorText"},{"id":"licence","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"span.smallTilePricingLabelStyle","type":"SelectorText"},{"id":"page","paginationType":"clickMore","parentSelectors":["_root","page"],"selector":"a:contains(\"Next Page\")","type":"SelectorPagination"}],"websiteStateSetup":{"enabled":true,"performWhenNotFoundSelector":"div.paginationTextPaddingLargescreenOnly","actions":[{"selector":"img","type":"click"},{"selector":"img","type":"click"}]}}
#3

Thanks for your response. Seems like this JSON still does not return scraped data, is the data inaccessible?

#4

I've put your pagination into my code, it scrapes the first page but stops after the second. Note I am setting up in row format instead of tiles.

{"_id":"oracle-marketplace-links-test-3","startUrl":["https://cloudmarketplace.oracle.com/marketplace/en_US/productHomePage"],"selectors":[{"id":"wrapper","multiple":true,"parentSelectors":["page"],"selector":"div.col-lg-12.col-xs-12","type":"SelectorElement"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["wrapper"],"selector":"a","type":"SelectorLink"},{"id":"blurb","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"span.x2dg","type":"SelectorText"},{"id":"review","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"span.reviewCount","type":"SelectorText"},{"id":"licence","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"span.x28y","type":"SelectorText"},{"id":"page","paginationType":"clickMore","parentSelectors":["_root","page"],"selector":"a:contains(\\\"Next Page\\\")","type":"SelectorPagination"}],"websiteStateSetup":{"enabled":true,"performWhenNotFoundSelector":"div.paginationTextPaddingLargescreenOnly","actions":[{"selector":"img#pt1\\:listResults\\:recentApps\\:i2","type":"click"},{"selector":"img#pt1\\:keyWordSearch\\:cil4\\:\\:icon","type":"click"}]}}
#5

I wrote you - that code is not working properly.... The pagination works, all selectors are located correctly, but I think the problem is an iframe and some blocked functionality inside sourcecode on Oracle website...