Avidity Software

It certainly has been a while since the last update, and to be honest, I can't remember all the stuff I've worked on since. :) I do know that I've played with probe timing a bit and added max/min timeout settings to the scan engine. I've also added some parallelism features to the engine so more than one probe gets sent at one time.

I'm facing a couple issues right now that I've been avoiding. One of which I've figured out and have test code working for. This is the ability to find the proper network interface to send probe packets from on machines with more than one. I just need to get off my butt and write the code in a way that's more formal than my test code. :)

The second issue seems to be a large one. The problem is that when sending probe packets really fast, I've noticed that I don't always get replies. However, if I wait a millisecond between sending each probe, I'll reliably get the replies, but this adds up when you scan many hosts along with many ports. So I need to build in much more sophisticated timing code, along with the ability to send retries for hosts we think we should be getting responses from. These kinds of hosts would be ones for which we have no state information on. If we can determine the host is definitely up, we know we should get a RST back. If we don't, it could be filtering those probes, so we'd need to send a few retries to determine this. It seems pretty complex, so I've been doing a lot of thinking about the problem, and little coding about it. But that'll change soon enough. :)

On another note, in the past few weeks I've started learning Lisp and Python. I'm learning Lisp mainly to get a new perspective on the code world and to find out what all the fuss is about that I keep hearing out of Paul Graham and other Lisp hackers. As far as Python, there's been a lot of fuss about it out of pretty much everybody lately, so I wanna see if it'll knock Perl out of first place as my favorite dynamic programming language. From what I can tell so far, Perl is quite secure in this position. :)

Speaking of Perl; having used it for a while now and being pretty proficient in it, it makes it tough to learn other languages because when I try to use the new language I inevitably get a case of, "Why am I doing this? I could do this SO easily in Perl.". Perl just makes everything so easy, IMHO of course. Hopefully I can successfully fend this attitude off until I get a good grasp of both Python and Lisp.

21-80

21,23,80

or even...

21-80,5900,5800

I had been wondering if the TCP connection throttling that I mentioned here would have an effect on my current scanning code. Since any probe packets that I'm sending are sneaking by the kernel without it's knowledge, I had a hunch that it would no longer be a concern because without the kernel knowing about the packets, it can't throttle them.

I tested this the other day and it turns out that my hunch was correct! What does this mean for future Yavar users? It means that you won't have to install the "patch" to be able to scan more than 10 hosts. On the other hand, you will need to have WinPCap, (or LibPCap for Linux and other superior OSs), installed to use this functionality, but come on, what hardcore administrator doesn't already have this installed? ;) Seriously though, this is a much better trade off since you don't have to "patch" your TCPIP.sys file with a utility that you may not trust. At this point you may be wondering how you can trust WinPCap or LibPCap. Well, they're open source. Download them, audit them, and compile them yourself if you're concerned about what they may be doing on your system. :)

A while back I'd noticed that if I had VMWare installed on my PC, VNCAdmin scans would take forever. Shortly after that, I realized that if I disabled the two network interfaces that VMWare installs, the scan speed went back to normal. This bothered me, but I assumed that it was just an isolated issue on my machine. However, I asked around and tested on other machines and found that it wasn't just an isolated issue.

Well, after a year or two of wondering, I've finally figured out what the issue is. It has to do with using the gethostbyaddr() function for retrieving hostnames when all you have is an IP address. This function will use several different methods of finding the hostname if it needs to before giving up, (I forget in which order it does these, or even what all the methods are :) ). At some point, this function will try sending NetBIOS name requests to the remote PC. The issue here is that it sends NetBIOS requests on each network interface that is defined on your host. So we're doing a lot more work, for the same result, (If you wonder how I know this, it's because I've sniffed the network traffic that the function generates). However, these VMWare interfaces are configured with 192.168.0.0/24 addresses, so they may not even be communicating with your network. I would assume that this would cause gethostbyaddr() to wait for responses on those interfaces or timeout when it doesn't get one, thus slowing it down even more.

However, now the question is, "How do I get around this?". Do I look for a newer reverse lookup function? Do I just write my own name resolution code by crafting and injecting DNS/NetBIOS packets? Do I ignore this as it's probably a pretty rare occurrence? We'll see I guess...

I can't help but wonder if the NetBIOS name requests being sent to each interface is a protocol spec, or if it's just the way Microsoft decided to implement the function? Perhaps neither I suppose. Either way, it's great to finally know what on earth was causing the slowdown. :)

I've added a link under the downloads section for the Yavar source code. For now, the interesting networking code is in VA_Utils.cpp. Enjoy!

Disclaimer: This is all alpha testing code, use at your own risk! :) I've not included any makefiles or build instructions as of yet since the project is so young.

It's been a while since my last update, but there's been a LOT of progress in the last month. I've revamped my scanning code to be much easier to use and more moduler, (however it's still not in it's final form :) ). This has made adding features and probe types fairly trivial.

At the moment I'm able to scan both internal and external subnets. What's the difference you ask? Well, when building a packet you start from the Ethernet header, then onto the IP header, then the TCP header, (assuming we're creating a TCP/IP packet). In order to write an Ethernet header you need the MAC address of the remote host. For local subnets, this is the MAC address of the remote machine itself, and this information can be retrieved by sending broadcast ARP requests and then sniffing for the replies that contain the remote hosts MAC address. However, on external subnets, the MAC address to the remote host that gets entered into the Ethernet header if the MAC address of the sending hosts default gateway!

So I needed to find a way to get the sending hosts default gateway IP address. Once I have that, I can send an ARP request for that IP address in order to get it's MAC address. This was easier said than done, (well, it's only currently working on Windows, I'll be working on Linux soon). However, stepping through the Nmap source code was a big help. Thanks Fyodor!

I've also added the ability to send a ping request to external subnets in order to find all the live hosts before actually sending the probe packets that test port status. This makes it so I'm not sending SYN, (or other probe types), packets to every host in the scan range. Why this is a good idea, I'm not quite sure yet. :) I've only done this for external subnets for now at least, since ARPing for the hosts on the local subnet seems to fill this function already.

So at this moment, this VNC scanner is enormously faster than the old VNCAdmin. However when comparing scan results to the old scanner and Nmap scans, I noticed I wasn't finding the same number of results. So I thought I'd throttle the rate at which I sent probe packets a little bit since I had been sending them to the network card as fast I could generate them. Adding a few milliseconds of sleep time between each probe gave me perfect results that now matched my old scanner and Nmap. w00t!

However using this kind of timing mechanism makes me a little uneasy for some reason, so I'm wondering if there's a better way to throttle sends...

I've recently decided to forgo writing all the low level details of packet creation and manipulation and instead take full advantage of the fantastic libnet library. This has many benefits, including not reinventing the wheel, using a library that's already been debugged and tested through use in many applications, and progress will just be much faster. :) The only reason I wanted to do the low level stuff myself was to learn from it, but I feel I've got a good handle on it already, and having looked through the libnet source code, it's VERY clean and readable. If I wanna know how they did something, I can easily find out. Ya gotta love open source. :)

So having decided this, I started using libnet's packet building functions instead of building the packet myself and just using libnet's libnet_write() function to put it on the wire. After getting by my ignorance, things fell into place. Note to self: when working with libnet, you need to build the packet top down instead of bottom up, (TCP header, then the IP header, then the ethernet header). So at the moment, I'm sending probe packets on ranges of IP addresses, however I still need to add the code to get the replies or lack thereof back.

I also found a link to a really good Powerpoint presentation about the latest incarnation of the libnet library that can be found here.

So I've recently made a small breakthrough on the scanning side of things. I've managed to send raw ARP requests and sniff the replies off the wire. The nice thing about this is that I no longer need to hard code MAC addresses for my target hosts into my code. W00t! This has larger implications as it verifies that the actual packet sniffing functions in libnet don't block, so I can start sniffing and call the packet processing functions that DO block at a convenient time, or maybe a different thread? Good food for thought! This also lays out the basic structure of the code I need to write to send a SYN packet and wait for the reply.

I've also been thinking of how to start integrating this network code into my wxWidgets code for the VNCAdmin rewrite. It feels awkward since the wxWidgets code is nice and tidy in their C++ classes, however the network code is some down and dirty C splattered all over the place. Oh well, we'll get there.

For the past few weeks I've been delving as deeply as I can into TCP/IP coding, especially dealing with raw sockets and raw packet data. It's terribly interesting. My first epiphany was that a packet coming off the network card is just a string of characters, (unsigned chars). To make working with this string easier, you can map it directly to C structures which allows you to fill in the specific fields of the packet and shoot it off to the wire. This also works in reverse for reading and getting data out of the packets. Excellent!

So now I'm reading and writing the raw packets I need to be able to send any kind of scan I want, (SYN, FIN, etc), but I need to be able to read the right ones! I can send a SYN packet to a machine, but I need to get the reply back, whether it be a SYN/ACK or a RST or any other possibilities. Using the libpcap library I can capture/filter packets easily, however it seems the library functions for capturing block program execution like a stop light, so I'm thinking that I need to look into spawning a packet filtering thread. Sounds reasonable to me.