Load more button + tables

How to?
#1

I'm trying to extract the data from this website:

https://albiononline.com/en/killboard/matches/crystalgvg

but I can't get the webscraper to do both, click on the load more button repeatedly and copy the data of the table

(time / team 1 / team 2 / details link)

I have tried element click which opens the page and continuously clicks the load more, but I can't get it to take the data. Tutorials I watched aren't helpful either.

Can someone tell me how I can get it to output this data structure:

{"_id":"crystal","startUrl":["https://albiononline.com/en/killboard/matches/crystalgvg"],"selectors":[{"id":"table","type":"SelectorTable","parentSelectors":["_root"],"selector":"table","multiple":true,"columns":[{"header":"Time (UTC)","name":"Time (UTC)","extract":true},{"header":"Team 1","name":"Team 1","extract":true},{"header":"Team 2","name":"Team 2","extract":true}],"delay":0,"tableDataRowSelector":"tr.fixed-row-height","tableHeaderRowSelector":"tr.reactable-column-header"}]}

.. but in addition to that clicking the load more button repeatedly?

#2

Try this on for size

{"_id":"crystal","startUrl":["https://albiononline.com/en/killboard/matches/crystalgvg"],"selectors":[{"id":"table","type":"SelectorTable","parentSelectors":["Element Click"],"selector":"table","multiple":true,"columns":[{"header":"Time (UTC)","name":"Time (UTC)","extract":true},{"header":"Team 1","name":"Team 1","extract":true},{"header":"Team 2","name":"Team 2","extract":true}],"delay":0,"tableDataRowSelector":"tr.fixed-row-height","tableHeaderRowSelector":"tr.reactable-column-header"},{"id":"Element Click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.killboard-wrapper","multiple":false,"delay":0,"clickElementSelector":"a.load-more-text","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"}]}

I wrapped your table selector as the child of an element click selector.

#3

this infinitely clicks the load more button, but does not extract any data :frowning:

#4

There're a lot of rows in the table, which could cause WS to crash, so it is probably better to limit the Load More clicker. In this example, I've made it only click twice which would get you about 100 rows. To increase the number of Load More clicks, you'd need to duplicate the "Click Load More" section as needed. Notice there are currently two identical sections, which means only click twice. I've also separated the table scraper from "Click Load More", so it only scrapes after the clicks are done.

{
  "_id": "z_crystal-v2",
  "startUrl": [
    "https://albiononline.com/en/killboard/matches/crystalgvg"
  ],
  "selectors": [
    {
      "id": "Click Load More",
      "type": "SelectorElementClick",
      "parentSelectors": [
        "_root"
      ],
      "selector": "div.killboard-wrapper",
      "multiple": false,
      "delay": "1800",
      "clickElementSelector": "a.load-more-text",
      "clickType": "clickOnce",
      "discardInitialElements": "do-not-discard",
      "clickElementUniquenessType": "uniqueCSSSelector"
    },
    {
      "id": "Click Load More",
      "type": "SelectorElementClick",
      "parentSelectors": [
        "_root"
      ],
      "selector": "div.killboard-wrapper",
      "multiple": false,
      "delay": "1800",
      "clickElementSelector": "a.load-more-text",
      "clickType": "clickOnce",
      "discardInitialElements": "do-not-discard",
      "clickElementUniquenessType": "uniqueCSSSelector"
    },
    {
      "id": "get_table",
      "type": "SelectorTable",
      "parentSelectors": [
        "_root"
      ],
      "selector": "table",
      "multiple": true,
      "delay": 0,
      "columns": [
        {
          "header": "Time (UTC)",
          "name": "Time (UTC)",
          "extract": true
        },
        {
          "header": "Team 1",
          "name": "Team 1",
          "extract": true
        },
        {
          "header": "Team 2",
          "name": "Team 2",
          "extract": true
        }
      ],
      "tableDataRowSelector": "tr.fixed-row-height",
      "tableHeaderRowSelector": "tr.reactable-column-header"
    }
  ]
}
#5

[quote="bretfeig, post:2, topic:4484"]
{"_id":"crystal","startUrl":["https://albiononline.com/en/killboard/matches/crystalgvg"],"selectors":[{"id":"table","type":"SelectorTable","parentSelectors":["Element Click"],"selector":"table","multiple":true,"columns":[{"header":"Time (UTC)","name":"Time (UTC)","extract":true},{"header":"Team 1","name":"Team 1","extract":true},{"header":"Team 2","name":"Team 2","extract":true}],"delay":0,"tableDataRowSelector":"tr.fixed-row-height","tableHeaderRowSelector":"tr.reactable-column-header"},{"id":"Element Click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.killboard-wrapper","multiple":false,"delay":0,"clickElementSelector":"a.load-more-text","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"}]}
[/quot

It works. The problem is that the element-click needs to cycle through all the click-more(s) and then it grabbs the data. Now if there is so many pages that your browser crashes before it runs out of clicks.. yea, won't get data. Go with @leemeng's suggestion