I'm try to scrap a site using "Next" pagination. I have +1000 pages. When I start scraping, it start going through pages but it doesn't extract data yet. It would seem as if it tries to index every page first, and then it would go through them again to extract data.
Question 1: Is it supposed to scrap the data as it goes through every page?
Question 2: If this behaviour isn't standard, can it be set up like that?
Url: http://example.com
Sitemap:
"selectors": [
{
"id": "cadaPersona",
"multiple": true,
"parentSelectors": ["_root", "pagination"],
"selector": ".vuetable-body tr",
"type": "SelectorElement"
},
{
"id": "nombre",
"multiple": false,
"parentSelectors": ["cadaPersona"],
"regex": "",
"selector": "td:nth-of-type(1)",
"type": "SelectorText"
},
{
"id": "apellido",
"multiple": false,
"parentSelectors": ["cadaPersona"],
"regex": "",
"selector": "td:nth-of-type(2)",
"type": "SelectorText"
},
{
"id": "documento",
"multiple": false,
"parentSelectors": ["cadaPersona"],
"regex": "",
"selector": "td:nth-of-type(3)",
"type": "SelectorText"
},
{
"id": "email",
"multiple": false,
"parentSelectors": ["cadaPersona"],
"regex": "",
"selector": "td:nth-of-type(4)",
"type": "SelectorText"
},
{
"id": "pagination",
"paginationType": "auto",
"parentSelectors": ["_root", "pagination"],
"selector": "a.page:nth-of-type(n+3)",
"type": "SelectorPagination"
}
]
}