Link Scraper

Overview

Link Scraper works by parsing the HTML content of a web page to extract all the links. It returns the links in a structured format, including the link URL, title, and more.

Endpoint

One host, one path per API. The block below shows this call in four languages; every one of them is the same HTTP request. Making requests covers the timeouts, retries and parameter rules that apply to all of them. The SDKs wrap the same call in a typed client.

curl -X POST https://api.apiverve.com/v1/linkscraper \
  -H "x-api-key: your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
  "maxlinks": 20,
  "includequery": false
}'

const res = await fetch('https://api.apiverve.com/v1/linkscraper', {
  method: 'POST',
  headers: {
    'x-api-key': 'your_api_key_here',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
  "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
  "maxlinks": 20,
  "includequery": false
}),
});

if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);

const { data } = await res.json();
console.log(data);

import requests

res = requests.post(
    "https://api.apiverve.com/v1/linkscraper",
    headers={"x-api-key": "your_api_key_here"},
    json={
    "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
    "maxlinks": 20,
    "includequery": False
},
    timeout=15,
)
res.raise_for_status()

print(res.json()["data"])

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := strings.NewReader(`{
  "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
  "maxlinks": 20,
  "includequery": false
}`)
	req, _ := http.NewRequest("POST", "https://api.apiverve.com/v1/linkscraper", body)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "your_api_key_here")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, _ := io.ReadAll(res.Body)
	fmt.Println(string(out))
}

Replace your_api_key_here with the key from your dashboard. When the inputs arrive as a list rather than one at a time, batch requests run up to 200 of them through this same API in a single call.

Authentication

Send your key in the x-api-key header. That is the only auth step — there is no token exchange and no per-endpoint scope to configure. Authentication covers creating, rotating and revoking keys.

Parameters

Sent as JSON in the request body. Premium parameters are accepted on every plan but only take effect on plans that include them.

Response

Every API returns the same three top-level keys, so one response handler covers your whole integration: status, error and data. Only data changes shape. Response format covers the envelope, the other output formats and how premium fields are withheld.

{
  "status": "ok",
  "error": null,
  "data": {
    "url": "http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
    "linkCount": 16,
    "externalLinkCount": 13,
    "internalLinkCount": 3,
    "links": [
      {
        "text": "Documentation",
        "href": "http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html/index.html",
        "external": false
      },
      {
        "text": "Amazon EC2 Instance Types Guide",
        "href": "https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-types.html",
        "external": true
      },
      {
        "text": "Amazon EC2 Auto Scaling",
        "href": "https://docs.aws.amazon.com/autoscaling/",
        "external": true
      }
    ],
    "uniqueDomains": [
      "docs.aws.amazon.com",
      "aws.amazon.com"
    ],
    "maxLinksReached": false
  }
}

Response fields

Paths are relative to data. Premium fields are absent rather than zeroed on plans that do not include them, so check for presence instead of comparing to 0.

Errors

Read the HTTP status first, then error for the specific reason. The body names the parameter that has to change. Error handling covers the full status list and which of them are worth retrying.

Use cases

SEO Analysis

Use the Link Scraper API to scrape web page links for SEO analysis. Use the data to analyze backlinks, internal links, and anchor text

Content Aggregation

Aggregate content by using the Link Scraper API to scrape web page links. Use the data to collect articles, images, and videos from multiple sources

Web Indexing

Index web pages by using the Link Scraper API to scrape web page links. Use the data to build search engine indexes, improve search results, and enhance user experience

Link Building

Build links by using the Link Scraper API to scrape web page links. Use the data to find relevant websites, submit URLs, and increase website traffic

Competitor Analysis

Analyze competitors by using the Link Scraper API to scrape web page links. Use the data to study link profiles, identify link sources, and benchmark performance

Other ways to use Link Scraper

Set up Link Scraper on APIVerve, or reach the same source a different way. Your APIVerve account and credits work on all of them — one key, one balance.

Related

More in Data Scraping: