Crawl infinite scroll page - WS does not fetch any data

How to?
#1

Hi fellas.

I have to crawl ralali[dot]com every each category (https://www.ralali.com/all-categories).

Some category can be crawl with this sitemap (example):
{"_id":"ralali_Agriculture n Food","startUrl":["https://www.ralali.com/c/air-mineral-hl1189","https://www.ralali.com/c/alat-pertanian-hl1626","https://www.ralali.com/c/bahan--bumbu-makanan-hl1176","https://www.ralali.com/c/buah-buahan-segar-hl1594","https://www.ralali.com/c/cemilan--sarapan-hl1201","https://www.ralali.com/c/coklat-teh-kopi--susu-hl1173","https://www.ralali.com/c/madu--selai-hl1522","https://www.ralali.com/c/makanan-instant-hl1196","https://www.ralali.com/c/makanan-kaleng-hl1332","https://www.ralali.com/c/minuman-lainnya-hl1538","https://www.ralali.com/c/peralatan--perlengkapan-makan-hl1258","https://www.ralali.com/c/sayuran-segar-hl1591"],"selectors":[{"id":"scroll article","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"article","multiple":true,"delay":"3000"},{"id":"name","type":"SelectorText","parentSelectors":["scroll article"],"selector":"a div","multiple":true,"regex":"","delay":0},{"id":"harga","type":"SelectorText","parentSelectors":["scroll article"],"selector":"div.item-price","multiple":true,"regex":"","delay":0},{"id":"lokasi","type":"SelectorText","parentSelectors":["scroll article"],"selector":"div.item-location","multiple":true,"regex":"","delay":0},{"id":"nama toko atau supplier","type":"SelectorText","parentSelectors":["scroll article"],"selector":"a.ng-binding","multiple":true,"regex":"","delay":0},{"id":"link","type":"SelectorLink","parentSelectors":["scroll article"],"selector":".item-name a","multiple":true,"delay":0}]}

but this sitemap cannot be applied to all category, including this one : https://www.ralali.com/c/otomotif-hl0014
It will only crawl visible cards, not the entire card that loaded after scrolling.

I tired to modify the sitemap :
{"_id":"ralali_recrawl_otomotif","startUrl":["https://www.ralali.com/c/otomotif-hl0014"],"selectors":[{"id":"scrolldown","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.row.ng-scope:nth-of-type(n) item:nth-of-type(n) article","multiple":true,"delay":"4000"},{"id":"nama","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"a div","multiple":true,"regex":"","delay":0},{"id":"harga","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"div.price","multiple":true,"regex":"","delay":0},{"id":"lokasi","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"div.item-location","multiple":true,"regex":"","delay":0},{"id":"nama toko atau supplier","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"a.ng-binding","multiple":true,"regex":"","delay":0}]}

But still no luck. instead, Webscraper does not fetch any data.

So, what's wrong? Could anybody guide me through this problem? Any tips or trick to crawl card on infinite scroll?

Any comment will be appreciate.
Thanks.

#2

Any comment will be appreciated.

#3

This will get all of the items:

{"_id":"ralali_recrawl_otomotif","startUrl":["https://www.ralali.com/c/otomotif-hl0014"],"selectors":[{"id":"scrolldown","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"item","multiple":true,"delay":"2000"},{"id":"nama","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"a div","multiple":false,"regex":"","delay":0},{"id":"harga","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"div.price","multiple":false,"regex":"","delay":0},{"id":"lokasi","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"div.item-location","multiple":false,"regex":"","delay":0},{"id":"nama toko atau supplier","type":"SelectorText","parentSelectors":["scrolldown"],"selector":"a.ng-binding","multiple":false,"regex":"","delay":0}]}

The only issue you might run into is that you can run out of memory if the scroll is too long. For this, you would need to segment the data into categories, so the scroll time is reduced.

#4

Thank you for your comment.

But unfortunately, Scraper still doesn't crawl till the last. Only scroll 2 - 3 times, and then "Scraper has finished".
I also tried to increase the scroll delay, but also no luck. Scraper still doesn't scrape all item.

Internet connection, Memory, and/or processor was not an issue for me since I am using a well-good rented server specs to run this job.

Do you have any idea?
Any tips or trick to crawl card on infinite scroll?
What selector have to be select for special case (inf scroll) like this?
Why you choose Item selector? Could you please elaborate more detail?

#5

The scroll seems to be working on my end here. Try maybe increasing the scroll delay, as it might just not have enough time to render the newly scrolled items.

#6

Its true.

I already tried with another category that cannot be scraped. They all are same. Only scrolled 2 - 3 times. Delay also increased 25 secs. That's nearly impossible if browser not have enough time to render.

Here some category sample :
https://www.ralali.com/c/perkakas-tangan-hl0043 https://www.ralali.com/c/power-tools-hl0045 https://www.ralali.com/c/material-handling-hl0041 https://www.ralali.com/c/office-equipment-hl0051

Any other suggestion or idea?

#7

I would suggest troubleshooting your local machine, browser, scraper version, etc., as all of the categories are being scrolled on my side.

#8

Webscraper 0.4.1
Chrome Version 75.0.3770.100 (Official Build) (64-bit)

KVM Processor 2.10 GHz (2 Processors) 8 CPU
RAM 18.1 GB
Windows Server 2008 R2 Enterprise 64Bit

Is there something wrong? Could please you give me another way to get the data?

#9

I have already scrape some site that the total amount is more than ralal1 . Just wondering what makes scraper cannot scrape the page...

Ia there any scraper log or something?

#10

Hi, I took a look at the site with the inspector and it seems they sub-group the items in blocks of 20 which you can reference with selector div.row.ng-scope:nth-of-type(x)

So I modified Webber's code to use that as the scroller selector. I also limited the scroller to stop at block 10 (it will not scroll infinitely). That will get you 200 items (10 x 20 = 200).

If you want more items, just change the number in the scroller selector, div.row.ng-scope:nth-of-type(-n+10) to a higher number. e.g. If you change it to 25, you'll get 500 items (25 x 20 = 500).

I suggest you don't scrape for infinite items 'cos you may run out of RAM. Experiment with the scroller number to get the best results without crashing the browser:

{"_id":"ralali_recrawl_otomotif_v2","startUrl":["https://www.ralali.com/c/otomotif-hl0014"],"selectors":[{"id":"scrolldown","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.row.ng-scope:nth-of-type(-n+10)","multiple":true,"delay":"1100"},{"id":"nama","type":"SelectorText","parentSelectors":["item _selector"],"selector":"a div","multiple":false,"regex":"","delay":0},{"id":"harga","type":"SelectorText","parentSelectors":["item _selector"],"selector":"div.price","multiple":false,"regex":"","delay":0},{"id":"lokasi","type":"SelectorText","parentSelectors":["item _selector"],"selector":"div.item-location","multiple":false,"regex":"","delay":0},{"id":"nama toko atau supplier","type":"SelectorText","parentSelectors":["item _selector"],"selector":"a.ng-binding","multiple":false,"regex":"","delay":0},{"id":"item _selector","type":"SelectorElement","parentSelectors":["scrolldown"],"selector":"item","multiple":true,"delay":0}]}