This page describes Shelob, the distributed, learning network crawler for ARS.

Shelob is an RFC1493-compliant network crawler, that understands the Nortel AutoTopology protocol (and could understand others, with minimal effort). The main crawling engine is available across all versions, and written specifically to allow the application to take advantage of clusters. It's been used in-production on the Borg with great success, and should work fine on any SSI cluster with no additional programming.

Shelob comes in a few flavors

1. C-variant: Provide your core switch(es), and Shelob will crawl the network using its AutoTopology knowledge, discovering changes to the topology and essentially "remapping" the network as she mines connected addresses
2. E-variant: Uses the AMI] "Electronics" table as a list of edge switches to crawl, ignoring AutoTopology. This method allows much more control over the crawler and is great for static environments.
3. G-variant: Like C, uses autotopology to build a view of the electronics, and classifies the switches, but builds a cache file so subsequent runs will be faster because the point of G is to graph the network.
4. M-variant: Like C, uses autotopology to build a view of the electronics, and classifies the switches, but does nothing to glean non-topology MAC addresses. Can update the AMI "Electronics" table too.
5. O-variant: Takes a list of switches from the command-line and prints the current list of connected MAC-addresses/ports on those switches. This version does not dump the results into the AMI CRAWL cache.
6. S-variant: Searches the network for a particular MAC address in real-time, following the topology.