Golang : How to extract links from web page ?
Golang's third party package goquery provides excellent tools to assist developers in building HTML parser/crawler. In this tutorial, we will learn how to use goquery to extract links from a web page. The code below will scan a part of the web page that is within the <div class=".nav-collapse ..." >
and <ul class=".nav..">
and extract the links.
package main
import (
"fmt"
"github.com/PuerkitoBio/goquery"
"strings"
)
func ScrapeLinks(url string) {
doc, err := goquery.NewDocument(url)
if err != nil {
panic(err)
}
// process this part :
// <div class="nav-collapse collapse navbar-responsive-collapse">
// <ul class="nav nav-pills">
// <li>
// <a href="/references">References</a>
// </li>
// <li>
// <a href="/tutorials">Tutorials</a>
// </li>
// </ul>
// </div>
doc.Find(".nav-collapse .nav").Each(func(i int, s *goquery.Selection) {
Title := strings.TrimSpace(s.Find("li").Text()) // https://www.socketloop.com/tutorials/trim-white-spaces-string-golang
// convert string to array
Fields := strings.Fields(Title)
// go deeper by 1 level to get the <a href=""></a>
doc.Find(".nav-collapse .nav a").Each(func(i int, s *goquery.Selection) {
Link, _ := s.Attr("href")
Link = url + Link
fmt.Printf("Title is [%s] and link is [%s]\n", Fields[i], Link)
})
})
}
func main() {
ScrapeLinks("https://socketloop.com")
}
Output :
Title is [References] and link is [https://socketloop.com/references]
Title is [Tutorials] and link is [https://socketloop.com/tutorials]
Bear in mind that goquery's doc.Find only look for first part of the <div class="nav-collapse collapse navbar-responsive-collapse">
.. i.e .. only
doc.Find(".nav-collapse") // will do.
in this tutorial case, because we want to dig 1 level deeper to <ul class="nav...">
we use this instead
doc.Find(".nav-collapse .nav")
Reference :
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+25.6k Mac/Linux and Golang : Fix bind: address already in use error
+16k Golang : Check if a string contains multiple sub-strings in []string?
+8.7k Golang : How to find out similarity between two strings with Jaro-Winkler Distance?
+32.2k Golang : Regular Expression for alphanumeric and underscore
+11.3k Get form post value in Go
+21.6k Golang : Match strings by wildcard patterns with filepath.Match() function
+7.7k Golang : Tell color name with OpenCV example
+28.8k Golang : missing Git command
+4.7k Golang : Check if a word is countable or not
+6.8k Golang : Gorrila mux.Vars() function example
+16.5k Golang : XML to JSON example
+5.8k Golang : Get missing location after unmarshal binary and gob decode time.