Crawling DOM tree data with jQuery functionality

Ekliptor

Administrator
Staff member
Requirements:

Getting started:

WolfBot can easily crawl HTML from the web during live trading, parse the DOM tree and use this information for your strategies.
To do this you just create a strategy as a subclass of AbstractCrawlerStrategy.

Then your strategy has access to get and post functions that automatically:
  • manages cookies
  • bypasses possible CloudFlare restrictions
  • adds common headers such as User-Agent
  • can easily access the DOM tree using $ (jQuery-like functionality)

Functions:
JavaScript:
    /**
     * Perform a HTTP GET request
     * @param address
     * @param callback
     * @param options
     */
    protected get(address: string, callback: utils.UtilsHttpCallback, options: any = {}): request.Request

    /**
     * Performa HTTP POST request
     * @param address
     * @param data
     * @param callback
     * @param options
     */
    protected post(address: string, data: any, callback: utils.UtilsHttpCallback, options: any = {}): request.Request
The response of these functions is a HTML string. WolfBot can parse the DOM tree from that string to easily extract information.

Example function which counts the number of long and short trading ideas on the first page of TradingView's cryptocurrency ideas:
JavaScript:
this.get("https://www.tradingview.com/markets/cryptocurrencies/ideas/", (body, res) => {
    if (body === false) {
        this.warn("Error getting TradingView ideas", res);
        return;
    }
    let $ = this.jQueryLoad(body);
    let tradingViewLongCount = $(".tv-card-label--long").length;
    let tradingViewShortCount = $(".tv-card-label--short").length;
});
This code can be called anywhere inside your strategy inheriting AbstractCrawlerStrategy.
 
Last edited: