I am trying to scrap this website by retrieving all of the company urls after clicking into the company name. I want to scrap the entire webpage. The pagination has page numbers and a next page option. I am either able to scrap the first page but not able to paginate, or able to paginate but not able to scrap the details in each company. Please note that you need to click continue->supplier directory->search before the scraper runs.
Url: GeBIZ | Supplier Directory Listing
Sitemap for pagination only:
{"_id":"clickpagi","startUrl":["https://www.gebiz.gov.sg/ptn/supplier/directory/index.xhtml"],"selectors":[{"clickActionType":"real","clickElementSelector":"input.formRepeatPagination2_NAVIGATION-BUTTON-PAGE:nth-of-type(n+4)","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":2000,"discardInitialElements":"do-not-discard","id":"sede","multiple":true,"parentSelectors":["_root"],"selector":"#contentForm\:j_idt201\:j_id43\:j_idt203 div.formSectionHeader6_SERIAL-NUMBER","type":"SelectorElementClick"},{"id":"link to company","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":"a.commandLink_TITLE-BLUE","type":"SelectorLink"}]}
Sitemap for scraping but no pagination:
{"_id":"gebiz_test","startUrl":["https://www.gebiz.gov.sg/ptn/supplier/directory/index.xhtml"],"selectors":[{"id":"clickintocompany","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":"a.commandLink_TITLE-BLUE","type":"SelectorLink"},{"id":"companyinfo","multiple":true,"parentSelectors":["clickintocompany"],"selector":"div.design-02_rui-layout_default_BODY-960","type":"SelectorElement"},{"id":"urlofcompany","multiple":false,"parentSelectors":["companyinfo"],"regex":"","selector":"#contentForm\:j_idt159 div.formOutputText_VALUE-DIV","type":"SelectorText"}]}