Data Preview works but not the actual scrape

tesh · June 28, 2019, 4:24pm

Describe the problem.
Hi there,
I have come up with a problem with one particular website that I have a scrape build for. It used to work perfectly fine but now nothing is extracted from the website.

But I have noticed that when I click on the data preview on the element selector, I am getting the correct results but when i actually do the whole scrape, there is nothing extracted.

I have tried this both on chrome and firefox and they both are extracting no data.

Is anyone else having similar issues? Is there any fix or workaround for this?

Thanks

Url: https://www.dewalt.com

Sitemap is below

Sitemap:

Emed_Medina · July 9, 2019, 3:09am

Hi,

I have the same problem.

Any Solution?

tesh · July 11, 2019, 5:23pm

No I still don't have a solution. Which website are you trying to scrape? Maybe it is certain types of websites. Not really sure.

leemeng · September 17, 2019, 12:53pm

I generated a new sitemap and did a test scrape on a category. Seems to work OK and I got all the info. You can copy/modify the map below; you probably need to expand the "Info Selector". I used Page load delay (ms): 6000

{"_id":"dewalt_new","startUrl":["https://www.dewalt.com/products/power-tools/drills"],"selectors":[{"id":"Click Item Links","type":"SelectorLink","parentSelectors":["_root"],"selector":".product-card__product-name a","multiple":true,"delay":0},{"id":"Info Selector","type":"SelectorElement","parentSelectors":["Click Item Links"],"selector":"article","multiple":false,"delay":0},{"id":"model","type":"SelectorText","parentSelectors":["Info Selector"],"selector":"span.pdp-details__sku","multiple":false,"regex":"","delay":0},{"id":"name","type":"SelectorText","parentSelectors":["Info Selector"],"selector":"span[itemprop='name']","multiple":false,"regex":"","delay":0},{"id":"Desc","type":"SelectorText","parentSelectors":["Info Selector"],"selector":"p","multiple":false,"regex":"","delay":0}]}

Avinash · September 19, 2019, 1:50pm

Hi,

What does "Info Selector" mean? Can you please explain. I'm using Linkedin Sales Navigator for scraping and there is same problem.

Jeff_Jones · October 28, 2019, 11:57pm

I have the same problem on LinkedIn. Data Preview shows the correct data. But Scraping only returns the first 2 of 25 rows. Also, data preview is flaky. It works sometimes, and others not.

I have worked with WebScraper and LinkedIn a lot in the past. So I am lost to understand what the problem is.

I use the Chrome console and jquery selectors and commands to verify that it should be finding 25 rows. (e.g. >> $('artdeco-entity-lockup-title a').length correctly returns 25.

leemeng · October 29, 2019, 2:55am

Hi, "Info Selector" is merely the name of the wrapper element (container) for all the data. I meant, you would need to add more data scrapers (in addition to model, name, desc) to get all the data you want. Anyway I tested the dewalt scraper again today (Oct 29) and it still works.

Jeff_Jones · October 29, 2019, 2:17pm

I figured out what the problem is with LinkedIn's new web page. They initially load only those candidates that would be visible on the page. And then dynamically load additional candidates as you scroll down. This is why a Data Preview is inconsistent. The data it returns depends on whether you have scrolled down the entire page first before clicking the button. And that Scraping only returns those candidates visible in the initial page load of the smaller WebScraper window during scraping.

Now the question is, how to get WebScraper to scroll to the bottom of the page before scraping on each new page load? Anyone?

leemeng · October 29, 2019, 3:01pm

Here's a scraper for the LinkedIn jobs page, with scroller. It works for me. I don't have Sales Navigator, so modify this for your needs:

{"_id":"linkedin-jobs","startUrl":["https://www.linkedin.com/jobs/"],"selectors":[{"id":"Scroller","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"li.card-list__item.job-card","multiple":true,"delay":"3000"},{"id":"Position","type":"SelectorText","parentSelectors":["Scroller"],"selector":"h3","multiple":false,"regex":"","delay":0},{"id":"Company","type":"SelectorText","parentSelectors":["Scroller"],"selector":"h4","multiple":false,"regex":"","delay":0},{"id":"Location","type":"SelectorText","parentSelectors":["Scroller"],"selector":"h5","multiple":false,"regex":"","delay":0}]}

Jeff_Jones · October 29, 2019, 5:17pm

Thank you LeeMeng! I now have a SelectorElementScroll working for my needs. And it is per how you are using it above. However, how do I get pagination (SelectorElementClick I assume) to work in order to get me to the next pages? I am trying several things but I haven't figured out the correct way to do this. I assumed _root >> SelectorElementClick >> SelectorElementScroll >> ... hierarchy, but that isn't working.

Jeff_Jones · October 29, 2019, 7:05pm

This code works for one page.

{
  "_id": "one_page_works",
  "startUrl": [
    "https://www.linkedin.com/talent/..."
  ],
  "selectors": [
    {
      "id": "FullName",
      "type": "SelectorText",
      "parentSelectors": [
        "scroll_element"
      ],
      "selector": "artdeco-entity-lockup-title a",
      "multiple": true,
      "regex": "",
      "delay": 0
    },
    {
      "id": "scroll_element",
      "type": "SelectorElementScroll",
      "parentSelectors": [
        "_root"
      ],
      "selector": "article.profile-list-item",
      "multiple": true,
      "delay": "500"
    }
  ]
}

This code correctly pages through all 40 pages...

{
  "_id": "onepage3",
  "startUrl": [
    "https://www.linkedin.com/talent/hire/..."
  ],
  "selectors": [
    {
      "id": "pager",
      "type": "SelectorElementClick",
      "parentSelectors": [
        "_root"
      ],
      "selector": "ol.profile-list",
      "multiple": true,
      "delay": "2000",
      "clickElementSelector": ".mini-pagination__quick-link [type='chevron-right-icon']",
      "clickType": "clickMore",
      "discardInitialElements": "do-not-discard",
      "clickElementUniquenessType": "uniqueText"
    }
  ]
}

So, I made the Pager element in the second one the a child of root, and made the Scroll Element from the first a child of the Pager element.
It works down to about the 20th page, and then closes. No data, no error msg, nothing!

Any ideas?

xagg · May 13, 2022, 12:46pm

It could be because webscraper seems to scrape using a small window size, which may change the size of the elements displayed as compared to when you configure the selectors.

Try changing the window size of your browser on the site you want to scrape. Does it change the layout? Does it remove some elements that you want to select?

If yes, try configuring the selectors using a smaller browser window size.

In my case, some elements completely disappear on the smaller window, and I'm looking for settings to scrape using a larger window.

xagg · May 13, 2022, 12:53pm

w/r my own reply: found it, just change the scraper window.