Golang : Get FX sentiment from website example

Just a quick and simple tutorial. Got a friend that asked me on how to get FX(Forex) sentiment data by "scraping" from a web page. As a trader, knowing the market sentiment helps us to plan ahead and make decision either to short(sell) or long(buy) an instrument.

Hang on....but why he wanted to scrape the sentiment data instead of looking at the website itself with his eyeballs? That is because he eyeballs are no longer in good condition and he needs the sentiment data to feed into his algorithm/trading bot.

Alright, below is a simple example that searched for the word Bearish,Bullish and Mixed from a HTML page.

NOTE : The example below is configured to get sentiment data from dailyfx.com and it shouldn't be hard to re-configure to get sentiment data from other websites.

Here you go!

 package main

 import (
  "fmt"
  "log"
  "os"
  "strings"

  "github.com/PuerkitoBio/goquery"
 )

 //<div class="row" data-sentiment-id="NZDUSD">
 // <div class="col-sm-15 col-xs-6 text-center ins-rate">
 // <div style="height: 2px;"></div>
 // <div style="font-size: 13px;">Last Week</div>
 // <div class="neu-color ins-data">2.78</div>
 // </div>
 // <div class="col-sm-15 col-xs-6 text-center ins-rate">
 // <div style="height: 2px;"></div>
 // <div style="font-size: 13px;">Current</div>
 // <div class="neu-color ins-data jsdfx-sentiment-present">2.1954</div>
 // </div>
 // <div class="col-sm-15 col-xs-6 text-center ins-rate">
 // <div class="hidden-xs hidden-sm" style="width: 110px;">
 // <div style="margin-top: -24px;">
 // <div style="display:inline;width:110px;height:200px;"><canvas width="137" height="250" style="width: 110px; height: 200px;"></canvas><input class="sentiment-knob jsdfx-short-knob" value="" readonly="readonly" style="display: none; width: 0px; visibility: hidden;"></div>
 // </div>
 // <div style="font-size: 13px; margin-top: -170px;">
 // Clients Short<div class="bearish-color ins-data jsdfx-sentiment-short" style="font-size: 24px;">31%</div>
 // </div>
 // </div>
 // <div class="hidden-lg hidden-md">
 // <div style="height: 2px;"></div>
 // <div style="font-size: 13px;" class="bearish-color">Clients Short</div>
 // <div class="bearish-color ins-data jsdfx-sentiment-short">31%</div>
 // </div>
 // </div>
 // <div class="col-sm-15 col-xs-6 text-center ins-rate">
 // <div style="height: 2px;"></div>
 // <div style="font-size: 13px;">Chg. Open Interest</div>
 // <div class="ins-data">-7.3%</div>
 // </div>
 // <div class="col-sm-15 col-xs-6 text-center" style="margin-top: 14px;">
 // <div style="height: 2px;"></div>
 // <div style="font-size: 13px;">Outlook</div>
 // <div class="bearish-color ins-data text-capitalize">Bearish</div>  <------- we are looking for this
 // </div>
 //</div>

 func GetDailyFXSentiment(url string) (sentiment string) {
  doc, err := goquery.NewDocument(url)

  if err != nil {
 panic(err)
  }

  doc.Find(".row .text-capitalize").Each(func(i int, s *goquery.Selection) {
 // For each item found, get the band and title
 data := s.Text()

 // we are only interested in the first item
 if i == 0 {
 log.Printf("%d - %s\n", i, data)

 if strings.ToLower(data) == "bearish" {
 sentiment = "BEARISH"
 }

 if strings.ToLower(data) == "mixed" {
 sentiment = "MIXED"
 }

 if strings.ToLower(data) == "bullish" {
 sentiment = "BULLISH"
 }

 }

 // the rest can be ignored
  })

  return sentiment

 }

 func main() {

  if len(os.Args) != 2 {
 fmt.Printf("Usage : %s url\n", os.Args[0])
 os.Exit(0)
  }

  url := os.Args[1]

  // perform a simple sanity check
  if url == "" {
 fmt.Println("URL address cannot be empty!")
 os.Exit(0)
  }

  fmt.Println("Querying market sentiment at : ", url)

  sentiment := GetDailyFXSentiment(url)

  if sentiment == "" {
 sentiment = "missing sentiment data"
  }
  fmt.Println("Sentiment : ", sentiment)
 }

Sample output:

$ ./getDailyFxSentiment https://www.dailyfx.com/usd-thb

Querying market sentiment at : https://www.dailyfx.com/usd-thb

Sentiment : missing sentiment data

$ ./getDailyFxSentiment https://www.dailyfx.com/eur-usd

Querying market sentiment at : https://www.dailyfx.com/eur-usd

2019/06/11 10:38:50 0 - Mixed

Sentiment : MIXED

$ ./getDailyFxSentiment https://www.dailyfx.com/nzd-usd

Querying market sentiment at : https://www.dailyfx.com/nzd-usd

2019/06/11 10:38:55 0 - Bearish

Sentiment : BEARISH

$ ./getDailyFxSentiment https://www.dailyfx.com/eur-usd

Querying market sentiment at : https://www.dailyfx.com/eur-usd

2019/06/11 10:13:20 0 - Mixed

Sentiment : MIXED

References:

https://www.socketloop.com/tutorials/golang-get-final-balance-from-bit-coin-address-example

https://socketloop.com/tutorials/golang-how-to-extract-links-from-web-page