Guide to Open Source Intelligence Gathering (OSINT) and Its tools

Open Source Intelligence Gathering (OSINT) also called Passive Recon is carried out by Governments, Law Enforcments, Vulnerability testers and other organizations to gather information about the target from Public resources. There is a lot of information that is missed by Google Search Engine because it may not be present on surface web or Google’s policies don’t allow it to collect. OSINT Tools can gather this type of information which can’t be found using Search engine. There are a lot of service providers that provide different information gathering service like Shodan (shodan.io) is a Hacker’s Search Engine that gives a lot of information about vulnerable IoT devices. HaveIbeenPwned is an online service that tells you whether if your email has been hacked or not. Dorking is another excellent resource of information gathering, there are a lot of ways to use the power of Search Engine for information gathering called as dorks. Exploit Database maintains Google Hacking Database (GHDB) which contains many dorks that can be used for specific purposes.

OSINT plays a very important in Information Security by doing a great help in threat modeling and malicious activities that can harm an organization or business. Using the Search Engine for information gathering is common but collecting information from uncommon resources that aren’t regularly used is rare and sometimes can be more effective than traditional methods. For Penetration Testers and BlackHat Hackers, OSINT also helps in launching spear phishing campaigns against a target.

Doing OSINT manually from individual resources is a tedious and time-consuming process which can be automated using tools. OSINT Tools like Spider Foot and Maltego combine these services to make an effective Information gathering about a target. They use APIs of individual resources like Shodan, Virustotal, HaveIbeenPwned to get you the information that Google can’t. In this article, we’ll look at some famous and useful tools that are used for OSINT.

Maltego

Maltego is proprietary tool used for OSINT and forensics and it is developed and maintained by Paterva. Maltego uses transforms to automate the process of querying different data sources.

Maltego can be used to search information related to the following entities:

• People.
• Groups of people (social networks).
• Companies.
• Organizations.
• Web sites.
• Internet infrastructure
• Affiliations.
• Documents and files.

Installation

Maltego is a proprietary and paid software but it has a freeware version called Maltego-CE. To install Maltego-CE in Linux, Add Kali Linux repositories to your distro and install it using apt.

apt-key adv --keyserver pgp.mit.edu --recv-keys ED444FF07D8D0BF6
echo "deb https://http.kali.org/kali kali-rolling main non-free contrib" > /etc/apt/sources.list
sudo apt update
sudo apt install maltego -y

Be sure to remove Kali Linux repositories afterwards, it can crash your Kernal while upgrading

Usage

Maltego-CE is a limited but powerful edition. It has ability to perform link analysis on up to 10 000 entities on a single graph only, paid edition has more. For example, we need to gather information about a person and we only have his email and his name. Open maltego and from left side of the panel, drag and drop Email Address to the graph.

Click on the bubble and enter the email address.

Now right click on the bubble and run all transforms. Maltego will verify the email and will search for affiliated accounts. In this case, a Flickr account is found.

Similarly, to get information about a domain name, select domain from the left panel and drag to the graph. After running transforms, you’ll see graph like this. It’ll tell you about its name servers and from where it was registered.

If you want to gather information about a website, drag “website” from the panel and run a transform. It’ll gather information about the platform that is used to build this website and other related info.

These are just some examples, Maltego has ability to do a lot more. It has a lot of third party APIs and plugins that can be installed and used manually to enhance its functionality.

SpiderFoot

SpiderFoot is another free and open source tool that automates the OSINT Process. It has the ability to gather information about IPs, domains, Networks and People. SpiderFoot is written in Python and has simple easy-to-use User Interface. It uses more than 100 public information resources and analyzes the data collected from DNS Servers, emails, IPs etc. It also uses 3rd party plugins to collect and analyze information.

Installation

You can install SpiderFoot in Windows, Linux and Docker. You just have to download its source code and install dependencies. To install it in Linux, run

git clone https://github.com/smicallef/spiderfoot.git
sudo pip install cherrypy
sudo pip install mako
sudo pip install beautifulsoup4
sudo pip install lxml
sudo pip install netaddr
sudo pip install requests
sudo pip install ipwhois
sudo pip install ipaddr
cd spiderfoot
sudo python setup.py install

If you want to use a separate environment like Docker, you can use the following commands.

git clone https://github.com/smicallef/spiderfoot.git
docker build -t spiderfoot .
docker run -p 5009:5001 -d spiderfoot
docker ps
docker exec -it [container_addr] /bin/sh
python ./sfcli.py -s http://localhost:5009

Now open that link in your browser.

If you open settings, here you can change DNS and other configs. Also SpiderFoot uses APIs of a lot of other Engines and Sites like VirusTotal, HoneyPotCheck and Shodan. You can set their API keys, without them you can’t use their functionality.

Now, we’ll run a scan on an email to gather information about linked accounts and related data. Click on new scan and enter the email address, and then run scan.

It will show current status of scan.

After scan is finished you can see results in Browse tab. This will show you every detail about linked accounts, leaked information and etc.

The same way you can gather information about websites, IPs and much more. SpiderFoot is, indeed a Simple and Amazing tool which combines the power of other search engines and services to automate the lengthy process of OSINT.