· 6 min read
To extract URLs from text, scan for strings that start with http://, https:// or www., stop at whitespace and quote characters, trim the punctuation that belongs to the sentence rather than the link, and remove duplicates. A URL extractor does this in one pass and returns a list with one link per line. The hard part is deciding where a link ends, since a full stop, a comma or a closing bracket can be part of the address or part of the sentence around it.
TL;DR: http, https and www links are found, trailing . , ; : ! ? are trimmed, a closing bracket is kept only when it matches an opening one inside the link, and repeats are removed. Bare domains like example.com, ftp: and mailto: links are not found, query parameters such as utm_source stay, and a link that differs only by a trailing slash counts as a different link.
The specification for URIs treats this as a known problem. RFC 3986 has an appendix on delimiting a URI in context, and it recommends wrapping a link in angle brackets and leaving trailing punctuation outside it, because a space-delimited scan cannot tell a final full stop from a final dot in a path. An extractor therefore works from a heuristic, and it is worth knowing which one.
Source: RFC 3986, Appendix C, delimiting a URI in context
We tested the AstralText extractor on short strings to see what it does at that boundary. A link followed by a full stop, comma, exclamation mark or question mark comes back without the punctuation. A link inside round brackets comes back without the closing one. A Wikipedia address such as https://en.wikipedia.org/wiki/Foo_(bar) keeps its bracket, because the opening one is inside the link. A link wrapped in quotes or angle brackets stops at the wrapper.
| In your text | Extracted? | Result |
|---|---|---|
| https://a.com/x?id=2, then | Yes | https://a.com/x?id=2, comma removed |
| (https://b.org/p). | Yes | https://b.org/p, bracket and full stop removed |
| www.c.com/a | Yes | www.c.com/a, kept without a scheme |
| [the study](https://example.org/s) | Yes | https://example.org/s, the Markdown link text is not kept |
| example.com on its own | No | A bare domain has no http or www marker to find |
| ftp://x.com, mailto:a@b.com | No | Only http, https and www are looked for |
| https://a.com/x twice | Once | Repeats are removed |
| https://a.com/x and https://a.com/x/ | Both | A trailing slash makes them different strings |
| https://A.com/x and https://a.com/x | Once | Compared without regard to case, the first form is kept |
The extractor returns each link as written, so ?utm_source=newsletter and similar parameters stay attached. Links copied from email and social platforms often carry them, and they are noise in a reference list and occasionally identify who shared the link. Removing them is a second step, and it is not always right: a parameter such as ?id=2 or a video timestamp may be what makes the link work, so decide link by link rather than stripping everything after the question mark.
Duplicates are removed by comparing the whole string, ignoring case. That is right for a clean list of what a document links to. It is too loose for the part of a URL after the domain, which can be case-sensitive on the server, so two paths that differ only by case are merged even though a site might treat them as different pages. It is too strict in the other direction, because a trailing slash, a different scheme or an extra parameter makes two links to the same page look different. If you are auditing links, scan the list once for near-duplicates.
No. The tool reads the text in your browser tab and requests nothing, so the document is not uploaded and no link is opened. The same goes for shortened links: they are listed as they appear, not resolved, which would mean requesting them and telling the shortener that someone looked.
Try it: extract URLs from text
Related: How to extract email addresses from a block of text
Related: How to remove HTML tags from text without mangling it
Paste the text into a URL extractor and copy the result. It finds http, https and www links, trims punctuation that belongs to the sentence, removes repeats, and lists one link per line.
Sentence punctuation sits directly against the link, and a simple scan cannot tell it apart from the address. This extractor trims a trailing full stop, comma, semicolon, colon, exclamation mark or question mark, and a closing bracket that has no matching opening bracket in the link.
No. It looks for http://, https:// or www. at the start, so a bare domain is not found. Accepting bare domains would also catch file names that look like domains.
No. Links come back exactly as written, apart from trimmed punctuation. Remove tracking parameters as a separate step, checking that the parameter is not required for the link to work.
No. It only reads the text in your browser and requests nothing, so it cannot tell you whether a link is alive.