Pagination in sites that have more than 2 pages to scrape.(WITH JAVA AND AJAX)

How to?
#1

Hi!
I have built a sitemap to scrape a website.It scrapes correctly the first two pages(page 1 and page 2 with all the data that I want).But it doesen´t go further than page 2,and I need to scrape all the links of pagination in that page in all the categories.

My current sitemap is the following:

{"_id":"lubricantesdscomponentes","startUrl":["http://www.dscomponentes.es/web/articulo/familia.php?id=83&selSubFamilia=222&selSubSubFamilia=544"],"selectors":[{"id":"Links","type":"SelectorLink","selector":"div.pr_imagen a","parentSelectors":["_root","Next"],"multiple":true,"delay":0},{"id":"Elemento","type":"SelectorElement","selector":"div.fichaart","parentSelectors":["Links"],"multiple":false,"delay":"3000"},{"id":"Referencia","type":"SelectorText","selector":"div.REFERENCIA","parentSelectors":["Elemento"],"multiple":false,"regex":"","delay":0},{"id":"Precio","type":"SelectorText","selector":"div.precio","parentSelectors":["Elemento"],"multiple":false,"regex":"","delay":0},{"id":"Titulo","type":"SelectorText","selector":"div#descPP","parentSelectors":["Elemento"],"multiple":false,"regex":"","delay":0},{"id":"Next","type":"SelectorElementClick","selector":"div.numerosProductos:nth-of-type(5)","parentSelectors":["_root"],"multiple":true,"delay":"10000","clickElementSelector":"div.numerosProductos a:nth-of-type(3)","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

The firs trouble that i found was to move from page 1 to page 2,I solved it increasing the delay time on the "element click".But,as I said before now I can´t go further than page 2 with this sitemap.

Any ideas to fix that and move on to the other pages?

Thaks from Spain!

#2

You won't be able to scrape this site because the page reloads when you are going through pagination. Element click selector works only if elements change in page after click not when the page reloads.

#3

Those are really bad news.
I can´t do it with the https://webscraper.io/ or it can´t be done anyways?

You know any sistem or progam to do it anyways?
I need it for creating my store.

Thanks for answering!

#4

There might be a software that can scrape that site.

#5

I won´t be able to scrape it page by page?

I have another problem with other site:In this site the url changes when pagination but my sitemap only scarp the 2 first pages then stops.
How can I fix this to make it scrape till the end?
My current sitemap is the following:

{"_id":"ferrestock","startUrl":["https://www.ferrestock.com/categoria-producto/electricidad-e-iluminacion/pilas-y-baterias-f/pilas-sf/alcalinas/"],"selectors":[{"id":"Links","type":"SelectorLink","selector":"a.woocommerce-LoopProduct-link","parentSelectors":["_root","pagination"],"multiple":true,"delay":0},{"id":"elementos","type":"SelectorElement","selector":"div.post-area","parentSelectors":["Links"],"multiple":true,"delay":0},{"id":"short Descripcion","type":"SelectorText","selector":"div.woocommerce-Tabs-panel p:nth-of-type(1)","parentSelectors":["elementos"],"multiple":true,"regex":"","delay":0},{"id":"Referencia","type":"SelectorText","selector":"h1.product_title","parentSelectors":["elementos"],"multiple":false,"regex":"","delay":0},{"id":"Titulo","type":"SelectorText","selector":"div.woocommerce-Tabs-panel h2","parentSelectors":["elementos"],"multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","selector":"p:nth-of-type(n+2)","parentSelectors":["elementos"],"multiple":false,"regex":"","delay":0},{"id":"Precio compra","type":"SelectorText","selector":"li:nth-of-type(1) p.price span","parentSelectors":["elementos"],"multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorElementClick","selector":"li.product.type-product","parentSelectors":["_root"],"multiple":false,"delay":"3000","clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

Thanks a lot!!

#6

Hello
test this:

{"_id":"test","startUrl":["https://www.ferrestock.com/categoria-producto/electricidad-e-iluminacion/pilas-y-baterias-f/pilas-sf/alcalinas/"],"selectors":[{"id":"link","type":"SelectorLink","selector":"a.woocommerce-LoopProduct-link","parentSelectors":["_root","pagination"],"multiple":true,"delay":0},{"id":"ref","type":"SelectorText","selector":"h1.product_title","parentSelectors":["link"],"multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","selector":"div.woocommerce-Tabs-panel p","parentSelectors":["link"],"multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","selector":"a.page-numbers","parentSelectors":["_root","pagination"],"multiple":true,"delay":0}]}

#7

You saved my ass dude!!! IT WORKS!!!!!

Thanks a lot!! I really appreciate it!!!!!