Hi Scrapers, I have success scraping from hotels sites like booking.com, trivago.com, etc; but I'm stuck at agoda.com. The frame element seems to be inconsistently named.
Would like to scrape out the hotel name, and star rating across multiple pages but it involves element scroll and "next page" pagination involved, which I'm not sure how to do it.
Url:
Appreciate any help, thanks!
May i see your sitemap?
Referring to this?
If you stuck, i want to see where you stuck, so i could give you a hint.
Dear Tian, here's my sitemap! Hope you can guide me on this thanks!
{"_id":"agoda","startUrl":["https://www.agoda.com/en-sg/pages/agoda/default/DestinationSearchResult.aspx?asq=Q6eYGMvEw0vwN5Y6vAd41M9bSWwrgVIp%2FkuPo%2BglZxgC6HiP12cD0Zy7SXGY5q2db3bIqL%2F1E0bTsAdkgQALYkUrfaaVUNUbHgA%2Ffwfl7yvIhYvfXwXBsbHj5QPv1iUEFdTQnUsLEztJFKpWHWppAcFnTSu8zadhS2b8gjfyQG4Fv5nnxevNNmi%2BuIzTLbWm6C4EEXce%2B8HNa9xUe5ibi6JZmOQ3f7RRTkRZuTVr3pU%3D&city=10787&cid=-218&tick=637113088077&languageId=1&userId=2080219e-3717-4c3e-ab84-ba401517c038&sessionId=hb0xyr4oco0xgiv3fuaa11at&pageTypeId=1&origin=SG&locale=en-SG&aid=130589¤cyCode=SGD&htmlLanguage=en-sg&cultureInfoName=en-SG&ckuid=2080219e-3717-4c3e-ab84-ba401517c038&prid=0&checkIn=2020-02-13&checkOut=2020-02-14&rooms=1&adults=2&children=0&priceCur=SGD&los=1&textToSearch=Udon%20Thani&travellerType=1&familyMode=off&productType=1&hotelStarRating=3,2,1&sort=priceLowToHigh"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"a[data-is-usp='false'][data-page-number='1'][data-review-providerid='332'], li[data-hotelid='535665']","multiple":true,"delay":"5000"},{"id":"next_page","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"a[data-is-usp='false'][data-review-providerid='332'][data-review-snippet-score='0']","multiple":false,"delay":0,"clickElementSelector":"button.pagination2__next","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"frame","type":"SelectorElement","parentSelectors":["_root"],"selector":"a[data-is-usp='false'][data-review-providerid='332'][data-review-snippet-score='0']","multiple":false,"delay":0},{"id":"hotel_name","type":"SelectorText","parentSelectors":["frame"],"selector":"h3","multiple":false,"regex":"","delay":0},{"id":"hotel_star","type":"SelectorText","parentSelectors":["frame"],"selector":"i.ficon-star-25","multiple":false,"regex":"","delay":0}]}
Yes, Agoda is a tricky site. You're on the right track with the separate scroller. Here's the sitemap i use. It has two separate scrollers and two custom delays. Note that I have changed the booking dates for testing; you'll need to generate a Url with the booking/filter options you want. I used Page load delay: 6600
Also, before you start scraping Agoda, load the Url you want, then zoom out (reduce) the browser view to 50%. You can do this by pressing Ctrl - (Ctrl-minus) or by using the browser settings. This will force the browser to load more rows and will also help the scroller to work better. Remember, you will need to perform this zoom out step before you start the scraper.
Sitemap:
{"_id":"forum-agoda-march","startUrl":["https://www.agoda.com/en-sg/search?city=10787&languageId=1&userId=891be9b9-36be-45b8-aeb2-baff73c58195&sessionId=tccvwfst53avsi0qnm0ae402&pageTypeId=103&origin=MY&locale=en-SG&cid=-218&aid=130589¤cyCode=SGD&htmlLanguage=en-sg&cultureInfoName=en-SG&ckuid=891be9b9-36be-45b8-aeb2-baff73c58195&prid=0&checkIn=2020-04-27&checkOut=2020-04-30&rooms=1&adults=2&children=0&priceCur=SGD&los=3&textToSearch=Udon%20Thani&travellerType=1&familyMode=off&productType=1&sort=agodaRecommended"],"selectors":[{"id":"scroller1","type":"SelectorElementScroll","parentSelectors":["_root","Paginator Next"],"selector":"ol.hotel-list-container li.PropertyCardItem.ssr-search-result:nth-of-type(-n+23)","multiple":true,"delay":"6000"},{"id":"Custom delay1","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"span[id='paginationPageCount']:contains('Page')","multiple":false,"delay":"1100","clickElementSelector":"span[id='paginationPageCount']:contains('Page')","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"scroller2","type":"SelectorElementScroll","parentSelectors":["_root","Paginator Next"],"selector":"ol.hotel-list-container li.PropertyCardItem.ssr-search-result:nth-of-type(n+23)","multiple":true,"delay":"6500"},{"id":"Custom delay2","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"span[id='paginationPageCount']:contains('Page')","multiple":false,"delay":"1100","clickElementSelector":"span[id='paginationPageCount']:contains('Page')","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Row wrappers","type":"SelectorElement","parentSelectors":["_root","Paginator Next"],"selector":"ol.hotel-list-container li.PropertyCardItem.ssr-search-result","multiple":true,"delay":0},{"id":"hotel_name","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"h3","multiple":false,"regex":"","delay":0},{"id":"hotel_star","type":"SelectorElementAttribute","parentSelectors":["Row wrappers"],"selector":"i[class*='ficon-star'][data-selenium*='star-rating']","multiple":false,"extractAttribute":"aria-label","delay":0},{"id":"Paginator Next","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#searchPageRightColumn","multiple":false,"delay":"6300","clickElementSelector":"button#paginationNext.btn.pagination2__next","clickType":"clickMore","discardInitialElements":"discard","clickElementUniquenessType":"uniqueHTML"},{"id":"area","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"span[class='areacity-name-text']","multiple":false,"regex":".+(?= - View on)","delay":0}]}
Thanks soo much sir, but i want looking for prices of the opponent. It's very important my datas. Help me add code about prices . Love sir long time.