Sitemap exits immediately after Website State Setup

How to?
#1

Web Scraper version: 1.107.22
Chrome version: 145.0.7632.160
OS: Windows 11

Link to the site you were scraping: lowes.com

Sitemap (Please make the sitemap as minimal as possible so it’s easier to replicate the bug. You can export the sitemap by opening it and choosing “Export Sitemap” in the dropdown menu):

{"_id":"Lowes_debug","startUrl":["https://www.lowes.com/pl/beverage-wine-chillers/kegerators/4294761676"],"selectors":[{"delay":2000,"elementLimit":5000,"id":"scroll","multiple":false,"parentSelectors":["_root"],"selector":"main","type":"SelectorElementScroll"},{"id":"ItemLink","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":".prd-brand-description-wrapper a","type":"SelectorLink"}],"websiteStateSetup":{"enabled":true,"performWhenNotFoundSelector":".store-zipcode-wrapper:not(:contains(\"02026\"))","actions":[{"type":"openUrl","url":"https://www.lowes.com/store/"},{"selector":"input[role='searchbox']","type":"click"},{"selector":"input[role='searchbox']","type":"textInput","value":"02026"},{"selector":"button.rightArrowBtn","type":"click"},{"selector":"div[class*=\"StoreInfoWrapper\"]:nth-of-type(1) button span[data-id=\"sc-set-as-my-store\"]","type":"click"}]}}

Pretty printed:

{
  "_id": "Lowes_debug",
  "startUrl": [
    "https://www.lowes.com/pl/beverage-wine-chillers/kegerators/4294761676"
  ],
  "selectors": [
    {
      "delay": 2000,
      "elementLimit": 5000,
      "id": "scroll",
      "multiple": false,
      "parentSelectors": [
        "_root"
      ],
      "selector": "main",
      "type": "SelectorElementScroll"
    },
    {
      "id": "ItemLink",
      "linkType": "linkFromHref",
      "multiple": true,
      "parentSelectors": [
        "_root"
      ],
      "selector": ".prd-brand-description-wrapper a",
      "type": "SelectorLink"
    }
  ],
  "websiteStateSetup": {
    "enabled": true,
    "performWhenNotFoundSelector": ".store-zipcode-wrapper:not(:contains(\"02026\"))",
    "actions": [
      {
        "type": "openUrl",
        "url": "https://www.lowes.com/store/"
      },
      {
        "selector": "input[role='searchbox']",
        "type": "click"
      },
      {
        "selector": "input[role='searchbox']",
        "type": "textInput",
        "value": "02026"
      },
      {
        "selector": "button.rightArrowBtn",
        "type": "click"
      },
      {
        "selector": "div[class*=\"StoreInfoWrapper\"]:nth-of-type(1) button span[data-id=\"sc-set-as-my-store\"]",
        "type": "click"
      }
    ]
  }
}

Running this it will perform the site setup, and then exit after returning to the start URL, yet it works fine without the web site setup steps, returning 24 items.

Log Information (Screen Record):
N/A

Tutorial on how to submit a video bug report: How to submit a video bug report

#2

Hi,

The Website State Setup logic does not look correct. Are you trying to set a store or a delivery location?

#3

Yes, and it seems to work. The issue is with the scrape itself not initiating after setup

#4

Are you trying to set a store or a delivery location?

#5

Either one/doesn't matter. They're actually linked (it sets estimated delivery ZIP after choosing a store), and this is setting store. I set both in my full code, this is just a simplified example. But again, the setup code runs (once one clears cookies before starting)... yet nothing is collected afterward: the scraper immediately exits after returning to start URL.

#6

Any word on what might be causing this?

#7

The logic of your setup is not quite correct. The Perform when not found selector has to be an element that is present on all pages after performing the WSS steps.

Here is a reference sitemap for setting a store location:

{"_id":"Lowes_set_location","startUrl":["https://www.lowes.com/pl/beverage-wine-chillers/kegerators/4294761676"],"selectors":[{"delay":2000,"elementLimit":5000,"id":"scroll","multiple":false,"parentSelectors":["_root"],"selector":"main","type":"SelectorElementScroll"},{"id":"ItemLink","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":".prd-brand-description-wrapper a","type":"SelectorLink"}],"websiteStateSetup":{"enabled":true,"performWhenNotFoundSelector":"#store-search-handler:contains(\"Dedham Lowe\")","actions":[{"type":"openUrl","url":"https://www.lowes.com/store/MA-Dedham/1914"},{"selector":"button.variant--filled","type":"click"}]}}
#8

I see. That performWhenNotFoundSelector works, but I daresay your explanation here contains a rather subtle distinction that is not clear from the documentation :upside_down_face: "Has to be present after WSS" versus "If not found WSS will trigger" are different, especially in cases like this where the structure of that portion of the page changes.

Hopefully this thread can help others who encounter a similarly confounding scenario.

Thanks!

#9

Thank you for your feedback. To clarify, the purpose of the Perform when not found selector is to check whether it is necessary to trigger the WSS steps or not.

Thus:

  1. If the element is present: scraping is executed without performing the WSS steps
  2. If the element is not present: WSS steps are executed, and the scraper checks again if the specified element is present.
  3. If the element is still not preset after WSS steps, scraping will fail.

I hope this helps to clarify the functionality of Website State Setup.

#10

Right. I understand the purpose. The issue was that the selector I chose was sufficient to enter WSS (#2) but that the script immediately exited afterward because the chosen element disappeared after the store was selected. That's the bit that bit me since the extra check in #3 is not obvious and as far as I can tell is not documented.