How to use Gobuster to find Interesting Directories & Files on a Website
How to use Gobuster to find Interesting Directories & Files on a WebsiteToday we will learn how to use gobuster to enumerate the files and directories on a web server.
This may frequently give vital information that makes it easier to conduct a precise assault, leaving less space for errors and lost effort. There are several tools available to accomplish this, but not all of them are made equal.
Gobuster, a directory scanner developed in Go, is definitely worth exploring.
Conventional directory brute-force scanners, such as DirBuster and DIRB, perform properly but are frequently sluggish and prone to mistakes. Gobuster is a Go implementation of these utilities that is available in a command-line manner.
The key benefit Gobuster offers over other directory scanners is its speed. Go is well-known for its speed as a programming language. It also provides great concurrency support, allowing Gobuster to make use of several threads for speedier processing.
The one limitation of Gobuster is the lack of recursive directory scanning. Unfortunately, another scan will be required for directories that are more than one level deep.
This isn’t always a significant concern, and other scanners can step in and fill in the gaps for Gobuster in this situation.
Gobuster has an easy-to-use command-line interface. It offers several helpful choices, but not so many that getting buried in the intricacies becomes difficult.
Overall, it’s an excellent tool that is both effective and quick.
In this article, we’ll look at it using DVWA as the target and Kali Linux as the attacker system. You may either follow along with them or use a comparable testing setup.
Step 1: Install Gobuster
# apt-get install gobuster
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: gobuster 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/1,532 kB of archives. After this operation, 4,963 kB of additional disk space will be used. Selecting previously unselected package gobuster. (Reading database ... 412624 files and directories currently installed.) Preparing to unpack .../gobuster_2.0.1-1_amd64.deb ... Unpacking gobuster (2.0.1-1) ... Setting up gobuster (2.0.1-1) ... Processing triggers for man-db (2.8.5-2) ...
Then, simply type gobuster in the terminal to run the tool.
~/gobuster# gobuster
2019/05/06 11:43:08 [!] 2 errors occurred:
* WordList (-w): Must be specified (use `-w -` for stdin)
* Url/Domain (-u): Must be specified
We can see it gives us a couple of errors. It needs at least two parameters (-u for the URL, and -w for the wordlist) to run properly. We can also display the help menu with the -h flag.
~/gobuster# gobuster -h
Usage of gobuster:
-P string
Password for Basic Auth (dir mode only)
-U string
Username for Basic Auth (dir mode only)
-a string
Set the User-Agent string (dir mode only)
-c string
Cookies to use for the requests (dir mode only)
-cn
Show CNAME records (dns mode only, cannot be used with '-i' option)
-e Expanded mode, print full URLs
-f Append a forward-slash to each directory request (dir mode only)
-fw
Force continued operation when wildcard found
-i Show IP addresses (dns mode only)
-k Skip SSL certificate verification
-l Include the length of the body in the output (dir mode only)
-m string
Directory/File mode (dir) or DNS mode (dns) (default "dir")
-n Don't print status codes
-np
Don't display progress
-o string
Output file to write results to (defaults to stdout)
-p string
Proxy to use for requests [http(s)://host:port] (dir mode only)
-q Don't print the banner and other noise
-r Follow redirects
-s string
Positive status codes (dir mode only) (default "200,204,301,302,307,403")
-t int
Number of concurrent threads (default 10)
-to duration
HTTP Timeout in seconds (dir mode only) (default 10s)
-u string
The target URL or Domain
-v Verbose output (errors)
-w string
Path to the wordlist
-x string
File extension(s) to search for (dir mode only)
Step 2: Install Some Extra Wordlists
Wordlists on Kali are located in the /usr/share/wordlists directory.
~/gobuster# ls /usr/share/wordlists/
dirb dirbuster dnsmap.txt fasttrack.txt fern-wifi metasploit nmap.lst rockyou.txt.gz sqlmap.txt wfuzz
The dirb and dirbuster ones are fine, but there is another wordlist I like to use for directory brute-forcing. There is a whole repository of useful wordlists on GitHub called SecLists.
The “common.txt” wordlist contains a good number of common directory names.
We can download the raw file into our current directory using the wget utility.
~/gobuster# wget https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt
--2019-05-06 11:46:40-- https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.148.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.148.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 35744 (35K) [text/plain] Saving to: ‘common.txt’common.txt 100%[======================================================================================================================>] 34.91K --.-KB/s in 0.03s2019-05-06 11:46:40 (1.24 MB/s) - ‘common.txt’ saved [35744/35744]
Alternatively, if we wanted to install the whole SecLists repository, we can do so with the package manager.
~/gobuster# apt-get install seclists
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
seclists
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/221 MB of archives.
After this operation, 795 MB of additional disk space will be used.
Selecting previously unselected package seclists.
(Reading database ... 412629 files and directories currently installed.)
Preparing to unpack .../seclists_2019.1-0kali1_all.deb ...
Unpacking seclists (2019.1-0kali1) ...
Setting up seclists (2019.1-0kali1) ...
That will install the entirety of SecLists in the /usr/share directory. It contains a ton of content, and if you’re looking for a go-to set of wordlists, SecLists will likely be the last thing you’ll ever need.
/usr/share/seclists# ls
Discovery Fuzzing IOCs Miscellaneous Passwords Pattern-Matching Payloads README.md Usernames Web-Shells
Step 3: Scan Directories & Files
Now that everything is set up and installed, we’re ready to use Gobuster. Let’s run it against our target with the default parameters.
~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt
=====================================================
Gobuster v2.0.1 OJ Reeves (@TheColonial)
=====================================================
[+] Mode : dir
[+] Url/Domain : http://10.10.0.50/dvwa/
[+] Threads : 10
[+] Wordlist : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Timeout : 10s
=====================================================
2019/05/06 11:50:25 Starting gobuster
=====================================================
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/.hta (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/about (Status: 302)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/index (Status: 302)
/index.php (Status: 302)
/php.ini (Status: 200)
/instructions (Status: 302)
/logout (Status: 302)
/robots (Status: 200)
/robots.txt (Status: 200)
/login (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/setup (Status: 200)
/security (Status: 302)
=====================================================
2019/05/06 11:50:38 Finished
=====================================================
We can see it gives us some information about the tool in the banner, and then it kicks off the directory discovery process. It returns the names of the directories and files as well as their status codes.
We can have it return the length of the body with the -l flag, which can be useful for enumeration.
~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -l
=====================================================
Gobuster v2.0.1 OJ Reeves (@TheColonial)
=====================================================
[+] Mode : dir
[+] Url/Domain : http://10.10.0.50/dvwa/
[+] Threads : 10
[+] Wordlist : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Show length : true
[+] Timeout : 10s
=====================================================
2019/05/06 11:52:41 Starting gobuster
=====================================================
/.hta (Status: 403) [Size: 292]
/.htaccess (Status: 403) [Size: 297]
/.htpasswd (Status: 403) [Size: 297]
/README (Status: 200) [Size: 4934]
/config (Status: 301) [Size: 319]
/docs (Status: 301) [Size: 317]
/external (Status: 301) [Size: 321]
/favicon.ico (Status: 200) [Size: 1406]
2019/05/06 11:52:52 [!] Get http://10.10.0.50/dvwa/about: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/php.ini (Status: 200) [Size: 148]
2019/05/06 11:52:54 [!] Get http://10.10.0.50/dvwa/cmd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/robots (Status: 200) [Size: 26]
/robots.txt (Status: 200) [Size: 26]
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index.php: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/instructions: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/login: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/logout: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/phpinfo.php (Status: 302) [Size: 0]
/phpinfo (Status: 302) [Size: 0]
/security (Status: 302) [Size: 0]
/setup (Status: 200) [Size: 3549]
=====================================================
2019/05/06 11:53:01 Finished
=====================================================
Usually, if something is zero bytes, it isn’t even worth looking into. It can save loads of time, especially when dealing with a large website or a large number of directories. If we only want specific status codes to be displayed, we can do that using the -s flag followed by the code we want.
~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200
=====================================================
Gobuster v2.0.1 OJ Reeves (@TheColonial)
=====================================================
[+] Mode : dir
[+] Url/Domain : http://10.10.0.50/dvwa/
[+] Threads : 10
[+] Wordlist : common.txt
[+] Status codes : 200
[+] Timeout : 10s
=====================================================
2019/05/06 11:54:16 Starting gobuster
=====================================================
/README (Status: 200)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:54:27 Finished
=====================================================
Use commas to specify multiple codes.
~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200,301
=====================================================
Gobuster v2.0.1 OJ Reeves (@TheColonial)
=====================================================
[+] Mode : dir
[+] Url/Domain : http://10.10.0.50/dvwa/
[+] Threads : 10
[+] Wordlist : common.txt
[+] Status codes : 200,301
[+] Timeout : 10s
=====================================================
2019/05/06 11:54:58 Starting gobuster
=====================================================
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:55:10 Finished
=====================================================
Let’s say we just wanted a quick way to view the directories, without the extra noise of the banner and status codes. Use the -q flag to hide the banner, and the -n flag to hide the status codes.
~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -q -n
/.hta
/.htpasswd
/.htaccess
/README
/config
/docs
/external
/favicon.ico
/about
/cmd
/php.ini
/index
/index.php
/instructions
/logout
/robots
/robots.txt
/login
/phpinfo.php
/phpinfo
/setup
/security
Another useful feature is the ability to save the results to a file. Use the -o flag to specify the output file.
~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -o results
Now the results are saved and can be viewed at a later time.
~/gobuster# cat results
/.hta (Status: 403)
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/about (Status: 302)
/cmd (Status: 200)
/php.ini (Status: 200)
/instructions (Status: 302)
/index (Status: 302)
/logout (Status: 302)
/index.php (Status: 302)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/security (Status: 302)
/setup (Status: 200)
And that’s it.
Today, have fun uncovering some secrets on hackthebox boxes or other bounty-targeted websites, and stay tuned for the next interesting article.
