Monday, August 19, 2013

Refining Your Nmap Scan Strategy

By Tony Lee

Introduction

Disclaimers:  This is not a troll.  :)  Scans were run as root (or sudo) using Nmap version 6.25.

Few would dispute that the de facto standard scanning tool is Gordon Lyon’s (aka Fyodor) Nmap.  However, if we asked you to thoroughly and efficiently scan a remote class C (192.168.1.1/24) network, which of the following would you run?

  1. nmap 192.168.1.0/24
  2. nmap -sS 192.168.1.0/24
  3. nmap -sS 192.168.1.0-255
  4. nmap -sS -Pn -p 0-65535 192.168.1.0/24

The answer we hear most often is option a.  While this may work for small networks, it does not scale for larger networks or more thorough assessments.  The astute reader will notice that options a, b, and c, operate identically.  Option a provides the network range in CIDR notation and since -sS is the default scan type when no options are supplied--option b is identical to option a.  Examining option c, reveals that it is the same as options a and b, except that the target is supplied using a network range instead of CIDR notation.  The problem with options a, b, and c is that they will not thoroughly scan the remote class c network as they will only scan the top 1000 TCP ports.  Option d is close to what we are looking for since it scans all of the TCP ports; however, it lacks efficiency since we will be scanning all ports on all hosts, including dead IP space.

The most important thing to keep in mind is that in this simple example, we are only asking to scan a class C network.  However as security practitioners, we are often faced with multiple class C’s, class B’s or even in some cases, class A networks.  We will take the rest of this article to outline a scan strategy for networks of all sizes and some tips and tricks for carving the output.

Outline

  • Host Discovery
    • -sn “Ping Scan”
    • Throttling Pro-tips
    • Generate Live Hosts List
  • Port Discovery
    • Most Common Ports
    • Full Port Scans
  • Putting it all Together
  • Service and Operating System
  • Nmap Scripting Engine (NSE)
  • Manipulating Output
    • Carving Standard Output
    • Convert to HTML
    • Processing Script
  • Summary

Host Discovery

A methodical approach to scanning usually involves performing host discovery first, which eliminates the problem we saw with the above option d--scanning copious amounts of dead IP space.  Fortunately, Nmap is quite intelligent and flexible for performing host discovery scans.  The common host discovery scan options are listed below:


 -sn: Ping Scan - disable port scan
 -PS/PA/PU/PY[portlist]: TCP SYN/ACK, UDP or SCTP discovery to given ports
 -PE/PP/PM: ICMP echo, timestamp, and netmask request discovery probes
 -PO[protocol list]: IP Protocol Ping


The option we will focus on in this article is the most popular and the default for host discovery (-sn “Ping Scan”).

-sn Ping Scan

One thing to note about the -sn “Ping Scan” option (formerly known as -sP) is that it does much more than just a traditional ICMP echo request.  By default against a remote network the “Ping scan” consists of the following packets:
  • ICMP echo request
  • TCP SYN to 443
  • TCP ACK to 80
  • ICMP timestamp request



Figure 1:  Packet capture confirms behavior of a remote "Ping Scan"
When scanning a local network, Nmap automatically switches to using ARP requests.  This is desirable intelligence because fewer packets are sent and it is generally more accurate.  A system that blocks all ICMP traffic and filters TCP ports 80 and 443 would be invisible to the four remote discovery packets sent above.  However, hosts typically respond to ARP requests because layer 2 addresses are necessary to construct packet headers.


Figure 2:  Packet capture confirms behavior of a local "Ping Scan"
For the first packet capture, the scanning host was placed on a different subnet than that of the targets--this caused Nmap to use the 4 discovery packets shown in Figure 1.  For the second packet capture, the scanning host was placed in the same broadcast range and thus ARP was utilized for discovery as shown in Figure 2.  Both of the packet captures were created with the same discovery scan shown below.



root@DVORAK:~/scans# nmap -sn -T4 -oA Discovery 192.168.1.0/24

Starting Nmap 6.25 ( http://nmap.org ) at 2013-05-13 13:21 EDT
Nmap scan report for Wireless_Broadband_Router.home (192.168.1.1)
Host is up (0.0023s latency).
MAC Address: 00:26:62:XX:XX:XX (Actiontec Electronics)
Nmap scan report for 192.168.1.2
Host is up.
Nmap scan report for android.home (192.168.1.3)
Host is up (0.025s latency).
MAC Address: CC:3A:61:XX:XX:XX (Unknown)
Nmap scan report for computer.home (192.168.1.147)
Host is up (0.00035s latency).
MAC Address: 24:77:03:XX:XX:XX (Intel Corporate)
Nmap done: 256 IP addresses (4 hosts up) scanned in 4.45 seconds



Options explained:
-sn = “Ping scan”
-T4 = Throttle to aggressive
-oA <basename>= Output in all three formats (normal, XML, greppable)

Throttling Pro-tips:

Aggressive (-T4) throttling is substantially faster than the default Normal (-T3) throttle.  We have rarely (if ever) seen Aggressive scanning crash a host or flood a network.  This should be what you start with unless you know of particular hosts that are sensitive to scanning.  With that said, if hosts are known to crash on simple scans or become easily flooded, avoid scanning them with -T4 and possibly try Polite (-T2) throttling.  Please realize that -T2 may be up to 10 times slower than -T3, so be patient and only run it on your most sensitive hosts--not hundreds at a time.  More importantly, avoid any one-off scans such as version scanning as these are more likely to crash hosts than the speed of the scan.  Some older SCADA components are known to fall over from simple port scanning, not to mention version scanning.  In general, we do not recommend Insane mode (-T5) as this can negatively affect accuracy.  Lastly, we only recommend -T0 or -T1 when trying to be extra stealthy (IDS evasion) and only for scanning a few ports on a few hosts because it will likely be too slow for anything else.

Generate Live Hosts List

Nmap, like other tools, accepts a file containing a list of IP addresses as input.  Because the remaining scans are more intensive than the discovery scans, we will want to only feed live hosts to Nmap.  We can use the output files from our discovery scan to generate this list of live hosts.  Three files are generated as a result of the -oA option:



root@DVORAK:~/scans# ls
Discovery.gnmap  Discovery.nmap  Discovery.xml



Files explained:
.nmap = Normal output (what is printed to the screen)
.gnmap = Greppable output
.xml = XML output

We will extract the live hosts from the .gnmap (greppable) file using a command such as the following:



root@DVORAK:~/scans# grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt


root@DVORAK:~/scans# cat LiveHosts.txt
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.147


Cut options explained:
-f = field number (in this case, field 2)
-d = delimiter (in this case, a space)

The resulting LiveHosts.txt file is one IP address per line, which can be supplied as input to Nmap and other tools as well.

Port Discovery

Now that we have generated our LiveHosts.txt file, we can start port discovery on these hosts.  If you have many live hosts to scan and only a short amount of time, you may want to break it up into two sets of scans:
  1. Most common ports
  2. Full port scans

Most Common Ports

Fortunately, Fyodor performed a substantial amount of research for his Top Ports Project which he presented in 2008 at BlackHat and Defcon.  He performed scans on millions of Internet IPs and incorporated internal scans from large organizations to determine the most commonly open TCP and UDP ports.  The key take-away from this research is that Fyodor made Nmap default port scans more efficient by targeting the top 1000 TCP and top 1000 UDP ports when no port options are provided.  But just how effective is scanning only the most common ports?  Here is an eye-opening table that summarizes the results of Fyodor’s research:

Table 1:  Summary of Fyodor's research


TCP
UDP
­­top­ports 10: 48%
­­top­ports 10: 50%
­­top­ports 50: 65%
­­top­ports 50: 86%
­­top­ports 100: 73%
-­top­ports 100: 90%
­­top­ports 250: 83%
-­top­ports 250: 94%
­­top­ports 500: 89%
­­top­ports 500: 97%
­­top­ports 1000: 93%
­­top­ports 1017: ~100%
­­top­ports 2000: 96%

­­top­ports 3674: ~100%



As shown in the table above, according to Fyodor’s research, scanning the top 1000 TCP ports is 93% effective in identifying all open TCP ports, while scanning the top 3674 TCP ports is almost 100% effective.  Additionally, scanning the top 1017 UDP ports is nearly 100% effective at identifying all open UDP ports.  Since the default scan is the top 1000 TCP and UDP ports, no port options are needed below.  (If scanning the top 1000 ports is not desirable, use the --top-ports option and specify a new number of ports to scan.)

Most common port scans:


nmap -sS -T4 -Pn -oA TopTCP -iL LiveHosts.txt


nmap -sU -T4 -Pn -oA TopUDP -iL LiveHosts.txt


New Options explained:
-sS = SYN Scan (half-open scanning) - Does not complete the 3-way handshake so it is faster than -sT (default scan - specifying is optional)
-sU = UDP Scan
-Pn = Do not perform discovery scan (we already know the hosts are alive)
-iL = Input from a file

Full Port Scans

Each host can have a maximum of 131,072 ports (that is 2^16 = 65536, then multiply by 2 for TCP and UDP).  But why would anyone need to scan all of those ports?  Why isn’t the top 1000 good enough--especially after we highlighted how effective it was?  One surprising port that is not contained in the top 1000 is TCP/2433 (the alternative default MS-SQL port, which can be a misconfigured Achilles heel for many networks).  Thus, skipping the full port scan can miss critical services that may be the entry point for an attacker or a piece of malware.   

Assuming that full port scans are desired, these can be executed when the most common port scans finish.  Full port scans will take substantially longer to finish than the common port scans.  Thus, it is ideal to have the results of the common port scans as something to review while the full port scans are running.  

Note:  Full port UDP scanning is VERY slow, and for extremely large networks may never finish in a reasonable amount of time.

Full port scans:


nmap -sS -T4 -Pn -p 0-65535 -oA FullTCP -iL LiveHosts.txt


nmap -sU -T4 -Pn -p 0-65535 -oA FullUDP -iL LiveHosts.txt


New Options explained:
-p <ports> = port list

Putting it all together

So what does the scanning process look like when combined?



Discovery:
nmap -sn -T4 -oA Discovery 192.168.1.0/24


Generate Live Hosts List:
grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt


Common Ports:
nmap -sS -T4 -Pn -oA TopTCP -iL LiveHosts.txt
nmap -sU -T4 -Pn -oA TopUDP -iL LiveHosts.txt


Full Ports:
nmap -sS -T4 -Pn -p 0-65535 -oA FullTCP -iL LiveHosts.txt
nmap -sU -T4 -Pn -p 0-65535 -oA FullUDP -iL LiveHosts.txt



Following this process should perform efficient host discovery, live host file generation, and full port scans for both TCP and UDP (along with the top 1000 ports, in case the full port scan does not finish in time).

Service and Operating System

After reviewing the common port scans (or full port scans if available), a few hosts of concern may emerge.  If this is the case, a more intensive scan may be desired such as a service and/or operating system scan.  These more intensive scans should not be used in discovery on a large network because it may take too long to finish.  If you have many live hosts, only run these scans on select hosts where more information is desired.

Service (Version) Scan

The ports that were discovered in prior scans were quickly labeled with associated services based on the nmap-services database of about 2,200 well-known services.  The example scan below shows ports 22, 23, and 80 all labeled with the expected services ssh, telnet, and http respectively.



Example SYN scan:
nmap -sS -T4 -PN -n 192.168.1.2
PORT   STATE SERVICE
22/tcp open  ssh
23/tcp open  telnet
80/tcp open  http


However, since applications/services can run on any arbitrary port, additional probing may be desired to ensure that the service matches the associated port.  For this additional probing, version detection (-sV) is useful.



Example version scan:
nmap -sV -T4 -PN -n 192.168.1.2
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 5.3p1 Debian 3ubuntu7 (Ubuntu Linux; protocol 2.0)
23/tcp open  telnet  Linux telnetd
80/tcp open  ssh     OpenSSH 5.3p1 Debian 3ubuntu7 (Ubuntu Linux; protocol 2.0)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel



Notice that while the SYN scan (-sS) labeled port 80/tcp as http (per the nmap-services information), a deeper version scan (-sV) revealed that port 80 was really another instance of SSH.



Figure 3:  The screenshot above shows a SYN scan and a Version scan.  The version scan revealed that port 80 is actually SSH.  The version scan also correctly guessed the version of SSH.
Version scan syntax:


nmap -sV -T4 -Pn -oA <host>-service.txt <host>



New Options explained:
-sV = Service Version Scan

As you can see, version scanning is quite useful, but as mentioned earlier, it comes at a cost.  It requires more packets and thus takes longer to scan.  Notice in figure 3 that the SYN scan finished in 0.08 seconds, while it took the version scan 10.33 seconds.  Multiply the difference by thousands of machines and it adds up.

Operating System Scan

For very large networks, the operating system for each scanned host is not always known.  There are two quick checks that almost always work for high level operating system discovery (Windows vs. Linux)--especially when the two techniques are combined:
  1. TTL - Time To Live value (quickly obtained via ping, be sure to subtract the number of hops)
    1. Windows will generally have a TTL of 128
    2. Linux will generally have a TTL of 64
    3. Network devices will generally have a TTL of 255
  2. RPC (port mapper) port
    1. Windows hosts will have TCP 135 (msrpc) open
    2. Linux hosts will have TCP 111 (rpcbind) open

However, if you need more detail, such as the specific version of Windows or Linux, this is where Nmap’s operating system scan comes in handy.  Nmap uses TCP/IP stack fingerprinting and an nmap-os-db of more than 2,600 known OS fingerprints in order to accurately identify the operating system.  In most cases it is in the ballpark and if it does list a few operating systems, the correct OS is usually in the list.



nmap -O -T4 -Pn -oA <host>-service.txt <host>



New Options explained:
-O = Operating System Scan

Caution:  Add the necessary ports using the -p option if the ports you want to use for operating system scans are not in the top 1000 ports.  Also be sure to include at least one open and one closed port to achieve a more accurate operating system guess.




Figure 4:  Comparison between SYN scan and operating system scan.
Some may want to use both -sV and -O at the same time, which is fine.  There is even a shortcut option for this discussed in the next section; however, we will provide a word of warning there as well.

NSE

Prior to the Nmap Scripting Engine (NSE), Nmap had very limited vulnerability scanning capabilities.  However, with the addition of NSE scripts, it can now seek out (and in some cases exploit) vulnerabilities.  This increases the flexibility and extensibility of Nmap by allowing users to write their own scripts with just a little bit of LUA scripting knowledge.

At the time of this writing, there are 433 NSE scripts provided as part of the Nmap download.  To view the scripts that are available, list the directories that contain the NSE scripts.

On Windows hosts the scripts are typically in the following location:
c:\Program Files (x86)\Nmap\scripts\

On Linux hosts the scripts are typically in the following location:
/usr/local/share/nmap/scripts/

You can also check this reference for a listing and description:  http://nmap.org/nsedoc/

Since these are interpreted (and not compiled) scripts, it is possible to view the source to better understand what they do and even tweak them to behave differently.  It is also quite helpful to start from one of these functioning scripts if LUA is not your forte.

To activate Nmap NSE scripts, use the --script option and supply comma separated script names, categories, or directories:

--script filename|category|directory|expression[,...]

Since NSE was designed for a variety purposes, including extended version scanning, vulnerability detection and even exploitation, all scripts have one or more categories defined.  This not only adds organization, but it can simplify the choices when running multiple scripts by being able to select a family of scripts based on their categories.

Category options are:  auth, broadcast, brute, default, discovery, dos, exploit, external, fuzzer, intrusive, malware, safe, version, and vuln.

In addition to running scripts based on category, there are a couple really nice shortcuts, but they may not be the best choice in all circumstances:
  • -sC = Script Scanning:  Equivalent to --script=default and will execute scripts which have the default category defined
  • -A = Aggressive scanning:  Performs the following: Operating System (-O), Version (-sV), Script scanning (-sC) and traceroute (--traceroute).

The reason why caution should be applied when running either of these options is that default scripts can include scripts in the intrusive category, which have the potential to crash systems or make printers go crazy.

Instead, it may be desirable to exercise more control over which NSE scripts are executed by using the boolean AND with the “safe” category.  For example:


nmap -sS -T4 -Pn --script “default and safe” <host>


nmap -sS -T4 -Pn --script “http-* and safe” <host>


nmap -sS -T4 -Pn --script “smb-* and safe” <host>


nmap -sS -T4 -Pn --script “safe” <host>



Figure 5:  Results of an Nmap script scan against a Windows 8 host
For more information, a great resource is Fyodor’s NSE usage page:

Manipulating Output

As mentioned earlier, the output options for Nmap are normal (.nmap), greppable (.gnmap), and XML (.xml).  We will now explore how we can manipulate these standard output formats to achieve even greater insight--which is particularly useful when viewing larger output files.

Carving Standard Output

The standard output files can be quite useful for generating statistics or feeding other tools.  The nmap and gnmap files are good for pulling information such as:

  • Unique open ports
  • Number of unique open ports
  • Open ports to feed into other tools
  • Hosts that are listening on a particular port
  • Number of hosts listening on a particular port

Unique open ports:



grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu


22/tcp
23/tcp
80/tcp
135/tcp
--snip--


Number of unique open ports:



grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | wc -l


16


Feed open ports into another tool (space separated):



grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' | xargs


22 23 80 135 139 443 445 902 912 992 4445 4567 5357 8080 8443 49155


Feed open ports into another tool (comma separated):



grep " open " FullTCP.nmap | cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' | xargs | sed 's/ /,/g'


22,23,80,135,139,443,445,902,912,992,4445,4567,5357,8080,8443,49155


Hosts that are listening on a particular port (telnet used below as an example):



grep 23/open/tcp FullTCP.gnmap | cut -f 2 -d ' '


192.168.1.1
192.168.1.2


(Replace port 23 with whatever port number you wish to find other ports/hosts of interest)

How many hosts are listening on a particular port (telnet used below as an example):



grep 23/open/tcp FullTCP.gnmap | cut -f 2 -d ' ' | wc -l


2


(Replace port 23 with whatever port number you wish to find other ports/hosts of interest)

Convert to HTML

We just illustrated the usefulness of normal and greppable output files, however XML files are also useful.  Nmap XML files can be converted into HTML format for easy viewing within a web browser.  This is possible through using Extensible Stylesheet Language Transformations (XSLT).

Detailed information on various options can be found here:  http://nmap.org/book/output-formats-output-to-html.html

One very reliable method is using the xsltproc tool (from http://xmlsoft.org/XSLT/).  Using it is simple and the result is portable.  Use the Nmap XML file for input and use -o to specify the output file.

Syntax is below:


root@DVORAK:~/scans# xsltproc TopTCP.xml -o TopTCP.html
root@DVORAK:~/scans# firefox TopTCP.html


Result:


Figure 6:  HTML result of using xsltproc
Notice that the output is very organized and port information is even displayed in easy to read tables.  Navigation is made convenient through the use of the included hyperlinks and search functionality provided by the browser.

Summary

With little-to-no training, anyone can scan a network using Nmap by simply specifying the host or target range.  While Fyodor has done a great job making Nmap default options as efficient as possible, you should still be able to improve scanning efficiency by customizing your scans.  Hopefully, by sharing a proven scan methodology and tricks for manipulating the output, we have provided ideas for you to refine your Nmap scan strategy and get the most from your efforts.


For this article and others like it, check out the following magazine:

Special Thanks To

Bill Hau
Dan Dumond
Dennis Hanzlik
Jacob Martinson
Ron Nguyen
Rudolph Araujo

2 comments:

  1. Great article, I've seen many similar posts where people advocate for identifying live hosts via port scanning in preparation for later in depth scanning. I believe this would be a great idea if you never wanted to do additional in-depth scanning such as versions or OS fingerprinting. Nmap uses a phased approach to each scan and will not scan dead IP space as many have suggested, it will first go through a host discovery phase before launching the port scanning phase. See http://nmap.org/book/nmap-phases.html.

    ReplyDelete
    Replies
    1. Thanks for reading and posting the insight James. You are absolutely correct that nmap inherently performs intelligent phased port scanning. While the native tool's capabilities will work fine in most situations, it can sometimes be advantageous to manually split the scans into phases as shown in the article above to provide more control to the tester. Two such scenarios are below:
      1) Extremely large corporate networks
      2) Enhanced discovery

      In the case of extremely large networks, your scans may never finish if you have a tight time window. Having some results are better than no results. Thus we will often split the scans not only into regions but also phases to ensure that some of the scanning completes. Multiple class A's and class B's can be very time intensive.

      Enhanced discovery can be a bit tricky with the command line options--so it is nice to run it separately. One of my favorite discovery technique's that Fyodor posted a while back is the following:

      nmap -sP -PE -PP -PS21,22,23,25,80,113,31339 -PA80,113,443,10042 -oA EnhancedDiscovery -iL targetlist.txt --excludefile excludefile.txt

      As a real world example, this scan just found 11 more hosts than the typical -sP when run against 3 class C's this morning. Hope that helps explain why someone may want to manually split scans into phases. These are just a couple of examples. Thanks again.

      Delete