Photon: Light and Fast Web Crawler

Photon is a lightning fast web crawler which extracts URLs, files, intel & endpoints from a target. 160 requests per second while extensive data extraction is just another day for Photon!

Main Features


Data Extraction

Photon extracts the following data while crawling by default:

  • URLs (in-scope & out-of-scope)
  • URLs with parameters (example.com/gallery.php?id=2)
  • Intel (emails, social media accounts, amazon buckets etc.)
  • Files (pdf, png, xml etc.)
  • JavaScript files & Endpoints present in them
  • Strings based on custom regex pattern

Intelligent Multithreading

Here’s a secret, most of the tools floating on the internet aren’t properly multi-threaded even if they are supposed to. They either supply a list of items to threads which results in multiple threads accessing the same item or they simply put a thread lock and end up rendering multi-threading useless. But Photon is different or should I say “genius”? Take a look at this and decide yourself.

Ninja Mode

In Ninja Mode, 3 online services are used to make requests to the target on your behalf. So basically, now you have 4 clients making requests to the same server simultaneously which gives you a speed boost if you have a slow connection, minimizes the risk of connection reset as well as delays requests from a single client.

Web Crawler Photon
Web Crawler Photon

Plugins

Photon’s capabilites can be further extended by using plugins.

Available plugins:

  • dnsdumpster: Generates an image containing the DNS data of the target domain.
  • Exporter: Plugin to export results in JSON, support for more formats is being worked on.

Plugins in active development:

  • Quark: A plugin to plot a graph making it easier to inspect relationships between different webpages using Quark.
  • XSStrike: Modular & targeted version of XSStrike to be used with Photon.
  • dnsdumpster: A new version of the plugin is in development which will save the DNS data in a nicely formatted HTML file.

Compatibility


Python Versions

Photon is fully compatible with python versions 2.x – 3.x at present but will most likely end up deprecating python2.x support in future as this project is under heavy development and may require features that aren’t available in python2.

Operating Systems

Photon has been tested on Linux (Arch, Debian, Ubnutu), Termux, Windows (7 & 10), Mac, and works as expected. Feel free to report any bugs you encounter.

Colors

Mac & Windows don’t support ANSI escape sequences so the output won’t be colored on Mac & Windows.

Dependencies

  • requests
  • urllib3
  • argparse

Rest of the python libraries used by Photon are standard libraries which come preinstalled with a python interpreter.

Usage

usage: photon.py [options]

  -u --url              root url
  -l --level            levels to crawl
  -t --threads          number of threads
  -d --delay            delay between requests
  -c --cookie           cookie
  -r --regex            regex pattern
  -s --seeds            additional seed urls
  -e --export           export formatted result
  -o --output           specify output directory
  --timeout             http requests timeout
  --ninja               ninja mode
  --update              update photon
  --dns                 dump dns data
  --only-urls           only extract urls
  --user-agent          specify user-agent(s)

Recent Articles

Hackers Install Secret Backdoor on Thousands of Microsoft SQL Servers

Cybersecurity researchers today uncovered a sustained malicious campaign dating back to May 2018 that targets Windows machines running MS-SQL servers to deploy...

COVID-19: SentinelOne Offers Free Platform Access

As the world battles COVID-19, enterprises are coping with immediate work-from-home needs and the challenges of protection beyond the network perimeter, says...

XSS vulnerability in the HTML Data Processor for CKEditor 4.0

A cross-site scripting (XSS) vulnerability in the HTML Data Processor for CKEditor 4.0 before 4.14 allows remote attackers to inject arbitrary web...

Unpatchable Flaw Affects All Intel CPUs Released in Last 5 Years

All Intel processors released in the past 5 years contain an unpatchable vulnerability that could allow hackers to compromise almost every hardware-enabled...

Photon: Light and Fast Web Crawler

Photon is a lightning fast web crawler which extracts URLs, files, intel & endpoints from a target. 160 requests per second while extensive...

Related Stories

Leave A Reply

Please enter your comment!
Please enter your name here

Stay on op - Ge the daily news in your inbox