Find Subdomains

I’ve spend half a day playing around with various tools that could get me a list of subdomains for a given domain. Here’s what I found.

I know of the following tools:

• Sublist3r
• https://crt.sh
• OWASP Amass
• https://dnsdumpster.com/
• https://securitytrails.com/
• assetfinder
• https://subdomainfinder.c99.nl
• SubDomainizer
• Anubis
• https://dnsdumpster.com
• https://github.com/Gr1mmie/sumrecon — I haven’t played with this one yet.
• https://tls.bufferover.run/ — I haven’t tried this one at all yet.

I must say that some of these tools are pain to work with, namely because of these reasons:

• you need API keys in various services in order for the tool to work, an example of this is OWASP Amass; and for the API keys, you often have to register, and that in turn might require some sort of verification beyond just using an email address (e.g. using a phone number as well)
• some of the tools are buggy; e.g. Sublist3e often returns some ugly exception
• some of the tools get you only so much in a free version
• some of the tools take forever to execute and then they come back with two subdomains

All in all, for my purposes I pretty much decided to stick with only crt.sh, which is a Certificate Transparency Log where you can find all certificates belonging to a domain.

crt.sh homepage

For example, I can check Tesla certificates on this link: https://crt.sh/?q=tesla.com

That’s nice but I also wanted to use this without the need to open my browser. I bet there are already various tools on GitHub for this (a brief search while writing this article returned this tool https://github.com/az7rb/crt.sh), but why not write my own that does exactly what I need?

So, the goal was to get a list of all subdomains that crt.sh returns, and then check which one of them are alive. For the second part, I used this tool https://github.com/tomnomnom/httprobe written in Go:

$ go install github.com/tomnomnom/httprobe@latest

which will add the executable into ~/go/bin.

Then I just scripted my way around these and other tools. You can see the result here: https://github.com/pavelsaman/scripts-backup/blob/master/subdomains/subdomains.sh The whole Bash script looks like this:

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
die() {
  echo "Fatal: ${1}" >&2
  exit 1
}
installed_and_executable() {
  cmd=$(command -v "${1}")
[[ -n "${cmd}" ]] && [[ -f "${cmd}" ]] && [[ -x "${cmd}" ]]
  return ${?}
}
print_help() {
  cat<<ENDHELP
Subdomains
Usage:
  subdomains.sh [-j|-s] -d <domain>
  subdomains.sh -h
Options:
  -d <domain>  Domain to search for, e.g. "example.com"
  -j           Print result to stdout in JSON format
  -s           Silent, no info output to stdout
  -h           Show this help  
ENDHELP
}
check_deps() {
  local deps=(curl jq httprobe)
for dep in "${deps[@]}"; do
    installed_and_executable "${dep}" || die "Missing '${dep}' dependency or not executable"
  done
}
main() {
  check_deps
local domain=
  local tojson=false
  local silent=false
while getopts ":d:jsh" opt; do
    case "${opt}" in
      d  ) domain="${OPTARG}"                                                     ;;
      j  ) tojson=true                                                            ;;
      s  ) silent=true                                                            ;;
      h  ) print_help; exit 0                                                     ;;
      \? ) echo "Unknown option: -${OPTARG}" >&2; exit 1                          ;;
      :  ) echo "Missing option argument for -${OPTARG}" >&2; exit 1              ;;
      *  ) echo "Unimplemented option: -${opt}. See help (-h option)" >&2; exit 1 ;;
    esac
  done
if [[ -z "${domain}" ]]; then
    echo "No domain, see help (-h option)" >&2
    exit 1
  fi
result_dir=~/.cache/subdomains/$(date "+%s")-"${domain}"
  mkdir --parents "${result_dir}"
[[ ${silent} = false ]] && echo " [+] Getting subdomains from crt.sh..."
  curl --silent "https://crt.sh/?q=${domain}&output=json" \
    | jq '.[].name_value' \
    | sed --regexp-extended 's/\\n/",\n"/g;s/"$/",/;s/[",]//g' > "${result_dir}/crt-all.txt"
[[ ${silent} = false ]] && echo " [+] Getting a unique list of domains..."
  sort --unique "${result_dir}/crt-all.txt" \
    | grep --invert-match '[*]' > "${result_dir}/crt-uniq.txt"
[[ ${silent} = false ]] && echo " [+] Getting a list of alive domains..."
  httprobe -t 5000 < "${result_dir}/crt-uniq.txt" \
    | sed --regexp-extended 's/^http(s)?:\/\///' \
    | sort --unique > "${result_dir}/alive.txt"
[[ ${silent} = false ]] && echo " [+] Results are in ${result_dir}: crt-all.txt, crt-uniq.txt, and alive.txt"
if [[ ${tojson} = true ]]; then
    [[ ${silent} = false ]] && echo " [+] alive.txt in JSON:"
    sed --regexp-extended 's/^/"/;s/$/",/;1s/^/[/;$s/,$/]/' "${result_dir}/alive.txt" | jq
  else
    [[ ${silent} = false ]] && echo " [+] alive.txt in plain text:"
    cat "${result_dir}/alive.txt"
  fi
}
main "${@}"

See the help:

$ subdomains.sh -h
Subdomains
Usage:
  subdomains.sh [-j|-s] -d <domain>
  subdomains.sh -h
Options:
  -d <domain>  Domain to search for, e.g. "example.com"
  -j           Print result to stdout in JSON format
  -s           Silent, no info output to stdout
  -h           Show this help

And a result can look like this:

$ subdomains.sh -d tesla.com
 [+] Getting subdomains from crt.sh...
 [+] Getting a unique list of domains...
 [+] Getting a list of alive domains...
 [+] Results are in /home/pavel/.cache/subdomains/1661093105-tesla.com: crt-all.txt, crt-uniq.txt, and alive.txt
 [+] alive.txt in plain text:
akamai-apigateway-vehicleextinfogw-prdsvc-st.tesla.com
apacvpn1.tesla.com
apacvpn.tesla.com
apf-api.eng.vn.cloud.tesla.com
apf-api.prd.vn.cloud.tesla.com
autobidder.powerhub.energy.tesla.com
autobidder-preprd.powerhub.energy.tesla.com
autodiscover.tesla.com
billing.tesla.com
cnvpn1.tesla.com
cnvpn.tesla.com
cx-apac.tesla.com
cx-api-apac.tesla.com
digitalassets-accounts.tesla.com
digitalassets-energy.tesla.com
digitalassets-learning.tesla.com
digitalassets-shop.tesla.com
digitalassets.tesla.com
employeefeedback.tesla.com
energysupport.tesla.com
engage.tesla.com
feedback.tesla.com
gridlogic.energy.tesla.com
image.emails.tesla.com
ir.tesla.com
kronos-dev.tesla.com
kronos.tesla.com
mobile-links.eng.vn.cloud.tesla.com
mobile-links.prd.vn.cloud.tesla.com
mobile-ops-links.prd.vn.cloud.tesla.com
powerhub.energy.tesla.com
referral.tesla.com
serviceapp.tesla.com
solarbonds.tesla.com
sso-dev.tesla.com
sso.tesla.com
teslacmgna01.tesla.com
tesla.com
teslatequila.tesla.com
toolbox.tesla.com
vehicle-files.eng.euw1.vn.cloud.tesla.com
vehicle-files.eng.usw2.vn.cloud.tesla.com
vehicle-files.prd.euw1.vn.cloud.tesla.com
vehicle-files.prd.usw2.vn.cloud.tesla.com
vpn1.tesla.com
vpn2.tesla.com
www.tesla.com
www-uat.tesla.com
xmail.tesla.com

I believe there are more such tools, it just takes more searching to find them. This one performs a quick search on crt.sh and returns results while also backing up results in ~/.cache/subdomains/{{timestamp}}-{{domain}} directory.