Spiderfoot is an open source intelligence (OSINT) tool that automates the process of information gathering from OSINT sources. The tool can be used to gather information related to people, web applications, and networks. Spiderfoot makes use of more than 100 public data sources (OSINT) to collect the information. The tool captures information like domain names, IP addresses, emails, DNS record, contacts, integrated technologies, entities, networks, services, content, security tools etc.
Spiderfoot can be cloned from Github using the following command.
git clone https://github.com/smicallef/spiderfoot
After cloning the tool, navigate to the spiderfoot directory to install the tool dependencies
cd spiderfoot pip install –r requirements.txt
How Spiderfoot Works
To start Spiderfoot, run the following command in the terminal.
Follow the url path generated in the terminal to open Spiderfoot web interface. The Spiderfoot dashboard can be seen in the following screenshot. The main dashboard shows links to three other pages in the form of tabs (new scan, scans, settings). The setting page is used for general configuration of the tool. The new scan (reconnaissance) page is used to initiate the new scanning session. The record of the old scanning processes as well as the progress of the running scanning process can be seen on the scans page.
To search target, click on the new scan button. The scanning process requires campaign name and information about the target. The information can be an IP address, domain name, sub-domain name, subnet or an email address. The following screenshot shows the format required to fill the target field.
Apart from selecting target, search methodology is also required to initiate the scanning process. Spiderfoot offers more than 100 OSINT sources divided into three search methodologies namely (a) use case, (b) required data, and (c) modules. Each search methodology gathers different set of information. For example, the use case option works best in footprinting and passive scanning scenarios. The required data tab contains a large number of OSINT tools that gather information like archived data, web content, IP addresses, domain names, certificates, target location, contacts etc. Spiderfoot also tries to find relation between the target and the malicious IPs, networks etc.
The modules option gains information from different networks like Bing, Google, Github, Shodan, social media networks, TOR servers etc. The following screenshot gives a brief overview of OSINT sources integrated in the modules section.
Let’s start a new scan with 192.168.10.1/24 subnet as a seed target as shown in the following screenshot.
We have a number of OSINT sources to scan the networks. Let’s select DNS and port scanning options from the module section.
After selecting the scanning options, hit the Run button that appears at the bottom of the tool to immediately start the scanning process. Spiderfoot starts gathering information about the target based on the selected modules. The scanning progress appears on the screen soon after starting the scanning process. The progress page shows different set of information, such as scan status, data browsing, graphical view of the data, and log information.
The status tab shows the scan status (whether running or not), captured data, uniqueness of the gathered data, and error information caused during the scanning process (if any).
The browsing tab gives the choice of exploring the data being gathered during the scanning process.
Clicking on each data item expands the information in a tabular format. For instance, if we click on the Open TCP Port data, we get an expanded version of TCP ports information as shown in the following screenshot.
Spiderfoot also gives the option of tracing the data path in a graphical manner. The graphical view of the data helps in mapping the target for further analysis.
Once the scanning process is finished, the results are archived in the following format.
Spiderfoot is a handy penetration testing tool that can help in carrying out network and web applications reconnaissance tasks. The tool is loaded with a number of modules and OSINT sources to aid the information gathering process.