Sitemap is providing data insufficiently/missing data

How to?
#1

Hi, I am trying to scrape data from the below website. But even after creating a proper sitemap I am getting insufficient data. It seems it is skipping some data or giving us interleaved data.

  1. I need suggestions on why this is happening ?
  2. Is there a way to resolve it?
  3. Do we have a list of websites or website structures where webscraper.io doesn't work properly?

Url: https://www.hioscar.com/faq#for-medicare

In this website, I need only data from medicare section.

Sitemap:
{"_id":"hioscar","startUrl":["https://www.hioscar.com/faq#for-medicare"],"selectors":[{"id":"link","parentSelectors":["_root"],"type":"SelectorLink","selector":"a.h-219IrFXyCMpob8eCiWjqAa","multiple":true,"linkType":"linkFromHref"},{"id":"data","parentSelectors":["link"],"type":"SelectorText","selector":"div.h-10C2yOzK-Pu-3erMTexkfi","multiple":true,"regex":""}]}

#3

image
image1670×835 68.2 KB

Hi, I need only data from medicare section that's why I selected only those link selectors for that.
Also the sitemap that you provided me is not giving the required data.

#5

image
image1044×137 10.6 KB

Oh I have written before the sitemap url.

#7

So how to create a sitemap for this use case?

#9

This is not known to me that whether they keep changing their content. For now I need the existing links in medicare section and their data. I think it has about 90 links.

#10

One more thing - is it possible to get the embedded links present in data ?i.e. when I export the data will the data (having embedded links) get replaced with actual links or a separate column will be provided for the links ?

#13

image
image1903×593 73.3 KB

This is my observation..we still got only 65 entries but the links were like 90. Do you know why this is happening ?Is it because of website structure?

#16

oh yes in the second url sitemap I am getting 90 links. Thanks.

I have a doubt - how have you chosen click selector?

selector - body
Click selector - div#mainContent:not(:has[footer]) li div:contains("For Medicare"):first-of-type

Also will selector will always be a body ?

I always used to select on the element to get its classname automatically but it seems you are not doing that.

#18

I see.
Still can you explain this -
Click selector - div#mainContent:not(:has[footer]) li div:contains("For Medicare"):first-of-type