Avidity Software

What the Haskell!?

Casey — 2007-01-16T21:29:54-06:00

I know I said my next directory recurser would be in Ruby, but Haskell has drawn my attention in a big way. Here's what I came up with...

module Main     where
import IO
import System
import Directory

main = do
  [recDir] <- getArgs
  printDirectory recDir

printDirectory path = do
  putStrLn path
  contents <- getDirectoryContents path
  mapM_ (\f -> do
      let newPath = (path ++ "/" ++ f)
      isaDir <- doesDirectoryExist newPath
      if isaDir
           then printDirectory newPath
           else putStrLn newPath
      ) (drop 2 contents)

Haskell seems like a pretty sweet language that I can learn a lot from. The pattern matching is very sweet and I'm looking forward to having a good understanding of Monads.

Giving XMMS the "what for" with Perl (AKA, C developers, append to enums, don't insert)

Casey — 2006-09-26T22:09:14-06:00

I've always wanted a way to be able to remotely control my MP3 player that runs on my linux box, through some sort of web interface or something. So I took some time off recently and thought it'd be fun to hack something together to let me do this. So the first thing I do is search the CPAN for any modules that let me control XMMS from Perl. What I found was the Xmms::Remote set of modules which just tied itself with a little XS code to a C library that is meant for sending commands to XMMS.

To make a long story short, this didn't work for me because when I used this module from my CGI script running through apache on my linux box, it was looking for the apache users instance of XMMS, not MY instance of XMMS. Ugh! So I wondered how this library was communicating with XMMS and guessed correctly that there was a Unix domain socket open. A little "netstat -a" action revealed my socket endpoint at /tmp/xmms_myusername.0. So what the library does is find your username and builds this path when it tries to connect to the socket in order to send XMMS a command. So, my CGI script, behind the scenes, was actually trying to connect to /tmp/xmms_apache.0, which of course, didn't exist.

This lead to the realization that I could use this socket to send my own command to XMMS! Sweeeeeeeeet! So I cracked open the XMMS source code and found its mechanisms for accepting commands on this socket and implemented my code for sending them to it. In this source code, I found an enumeration that went a bit like this...

enum
{
CMD_GET_VERSION, CMD_PLAYLIST_ADD, CMD_PLAY, CMD_PAUSE, CMD_STOP,
}

What this does is maps numbers starting at 0 to each of the names in the enum. So CMD_GET_VERSION is 0, CMD_PLAYLIST_ADD is 1 and so on. So this enum maps out the commands that can be sent to the socket.

Next I needed to figured out the protocol that is used to send commands and receive data back from XMMS. This was easily found with a little more digging. There are two C structs defined for sending and receiving....

typedef struct
{
     guint16 version;
     guint16 command;
     guint32 data_length;
} ClientPktHeader;

and

typedef struct
{
guint16 version;
guint32 data_length;
} ServerPktHeader;

So if you want to send a command to XMMS, you just need to create a ClientPktHeader, enter the values and send it to the socket and XMMS will see it and execute the command. In order to do things like this in Perl, you need to get cozy with the pack() and unpack() functions. With pack() you can pack values into memory, or in other words, create a C struct. In order to make a ClientPktHeader I just needed to do this...

my $data = pack( 'SSI', $xmms_protocol_version, $command, $data_length );

What this does is packs a C short (16 bits) or "S", followed by another one, followed by a C int (32 bits), for a total of 8 bytes, into memory. Once this is done, you can feed pack()s output to Perls send() function to send this data to the socket. Wonderful! Connecting to the socket is straightforward if you've done any socket work. The key is that it's a Unix domain socket, not a TCP socket.

Everything worked great, and quite honestly, I couldn't believe it did. :) However as I went on to implement functions to send XMMS commands, I noticed that some of them seemed to work and some didn't. One even crashed XMMS! I spent hours looking at my code and I was doing everything correctly. My solution? Watch TV for a couple days and forget about it. A few days later though, I came back to it and after a few hours realized that the enum that mapped out all of the XMMS commands must have been out of sync with what XMMS was actually expecting! Ugh! So when I was trying to call CMD_GET_VOLUME, it was actually calling CMD_SET_VOLUME and was crashing because it was expecting me to send it 8 bytes of data representing the new volume of the 2 stereo channels and didn't get it. The result? Segmentation Fault!

What had happened was that when my linux distro (Gentoo) built and installed XMMS it also applied patches to the source tree and made changes to that enum. So what read

enum
{
CMD_GET_VERSION, CMD_PLAYLIST_ADD, CMD_PLAY, CMD_PAUSE, CMD_STOP
}

was actually now

enum
{
CMD_GET_VERSION, CMD_PLAYLIST_ADD, CMD_SOMETHING_ELSE, CMD_DIFFERENT_CMD, CMD_PLAY, CMD_PAUSE, CMD_STOP
}

The result of this is that I was sending incorrect commands to XMMS without knowing it. I finally got the correct enum out of the patched XMMS tree and everything now works great.

The moral of this story? When you're writing enums that someone could possibly get their grubby hands on and start using, please, please, please append any changes to the end of the enum instead of inserting items in the middle of it! Of course you'd probably say, "Well, you shouldn't go and do stuff like you did, it's your own fault. You should know better than to use someone elses code that could easily change!" And you'd be right I suppose, but I don't care. :-P

At the very least, even if this project is doomed to failure by unpredictable XMMS enums, it was still a fantastic learning experience and a lot of fun to boot.

P.S. I know I could have just used C or C++ and used the XMMS library that is meant for this sort of thing, but I really wanted to implement a pure Perl solution instead of something that needed a binary to work.

P.P.S. I also know that there is a xmms-shell program that lets you control XMMS from a command line, so I can SSH into my box and control it remotely that way. This works great and I highly recommend it.

A filesystem recurser in Python too

Casey — 2006-09-26T22:00:00-06:00

So I'm learning Python here and there and thought I'd reimplement my directory recurser with it. Again, I'm only learning, so there may be a better way to do this in Python. Of course I could use os.walk(), but that would defeat the purpose of creating my own recurser. Why am I doing this simple task? Just cuz. :)

#!/usr/bin/env python
import os, sys

def printDir( path ):
     "Print a directory and it's files recursively"

     print 'Directory: ' + path

     for item in os.listdir( path ):
          newPath = path + '/' + item
          if os.path.isdir( newPath ):
               printDir( newPath )
          elif os.path.isfile( newPath ):
               print item

if len(sys.argv) > 1:
     printDir( sys.argv[1] )
else:
     printDir( '.' )

Next up...Ruby...

A shorter filesystem recurser in Perl

Casey — 2006-09-18T22:59:44-06:00

Shortened it up a bit...

#!/usr/bin/perl use strict; use warnings;
printdir( $ARGV[0] );

sub printdir
{
     my $dir          = $_[0] ? $_[0] : '.';
     my $handle      = ();

     print "Directory: $dir\n";

     opendir( $handle, $dir );
     my @items = sort( readdir( $handle ) );
     closedir( $handle );

     map
     {
          -d "$dir/$_"
          ? printdir( "$dir/$_" )
          : print "$dir/$_\n"
     } @items[2 .. $#items];
}

And people call Perl just a bunch of line noise?! Bah, sissies! :-P

Recursivly print directories in Perl...with no modules

Casey — 2006-08-26T19:24:31-06:00

Just for fun I came up with some Perl to recursively print out a directories files and child directories. Why am I posting it? It's not that it's all that difficult or obscure, I just feel like it, and when I did a web search for code on this that doesn't use the File::Find Perl module, I was surprised as to how few results there were. So in the interest of reinventing the wheel a bit, here's what I came up with.

#!/usr/bin/perl use strict; use warnings;
printdir( $ARGV[0] );

sub printdir
{
     my $dir          = $_[0] ? $_[0] : '.';
     my $handle      = ();
     my @dirs      = ();
     my $i           = ();

     print "Directory: $dir\n";

     opendir( $handle, $dir );
     my @items = readdir( $handle );
     closedir( $handle );

     shift @items; #Get rid of .
     shift @items; #Get rid of ..

     @items = sort( @items );

     while ( $i = shift( @items ) )
     {
          if ( -d "$dir/$i" )
          {
               push( @dirs, $i );
               next;
          }
          print "$dir/$i\n";
     }
     printdir( "$dir/$i" ) while ( $i = shift( @dirs ) );
}

Timing Is Everything

Casey — 2006-02-05T01:50:00-06:00

It certainly has been a while since the last update, and to be honest, I can't remember all the stuff I've worked on since. :) I do know that I've played with probe timing a bit and added max/min timeout settings to the scan engine. I've also added some parallelism features to the engine so more than one probe gets sent at one time.

I'm facing a couple issues right now that I've been avoiding. One of which I've figured out and have test code working for. This is the ability to find the proper network interface to send probe packets from on machines with more than one. I just need to get off my butt and write the code in a way that's more formal than my test code. :)

The second issue seems to be a large one. The problem is that when sending probe packets really fast, I've noticed that I don't always get replies. However, if I wait a millisecond between sending each probe, I'll reliably get the replies, but this adds up when you scan many hosts along with many ports. So I need to build in much more sophisticated timing code, along with the ability to send retries for hosts we think we should be getting responses from. These kinds of hosts would be ones for which we have no state information on. If we can determine the host is definitely up, we know we should get a RST back. If we don't, it could be filtering those probes, so we'd need to send a few retries to determine this. It seems pretty complex, so I've been doing a lot of thinking about the problem, and little coding about it. But that'll change soon enough. :)

On another note, in the past few weeks I've started learning Lisp and Python. I'm learning Lisp mainly to get a new perspective on the code world and to find out what all the fuss is about that I keep hearing out of Paul Graham and other Lisp hackers. As far as Python, there's been a lot of fuss about it out of pretty much everybody lately, so I wanna see if it'll knock Perl out of first place as my favorite dynamic programming language. From what I can tell so far, Perl is quite secure in this position. :)

Speaking of Perl; having used it for a while now and being pretty proficient in it, it makes it tough to learn other languages because when I try to use the new language I inevitably get a case of, "Why am I doing this? I could do this SO easily in Perl.". Perl just makes everything so easy, IMHO of course. Hopefully I can successfully fend this attitude off until I get a good grasp of both Python and Lisp.

Multi-port support

Casey — 2005-12-15T18:57:48-06:00

I've recently written a std::string tokenizer, (which was really fun), and extended it to be able to tokenize port lists that looks like these...

21-80
21,23,80
or even...
21-80,5900,5800

So now Yavar can scan as many ports and as many kinds of combinations as you might want. I don't know if I want to give Yavar the ability to take this kind of port argument since it seems kinda malicious, however I think it's good to have the ability to do it within the scanning engine, so whether or not this sticks around will remain to be seen.

Windows XP SP2 TCP connection throttling...No problem!

Casey — 2005-12-10T10:19:13-06:00

I had been wondering if the TCP connection throttling that I mentioned here would have an effect on my current scanning code. Since any probe packets that I'm sending are sneaking by the kernel without it's knowledge, I had a hunch that it would no longer be a concern because without the kernel knowing about the packets, it can't throttle them.

I tested this the other day and it turns out that my hunch was correct! What does this mean for future Yavar users? It means that you won't have to install the "patch" to be able to scan more than 10 hosts. On the other hand, you will need to have WinPCap, (or LibPCap for Linux and other superior OSs), installed to use this functionality, but come on, what hardcore administrator doesn't already have this installed? ;) Seriously though, this is a much better trade off since you don't have to "patch" your TCPIP.sys file with a utility that you may not trust. At this point you may be wondering how you can trust WinPCap or LibPCap. Well, they're open source. Download them, audit them, and compile them yourself if you're concerned about what they may be doing on your system. :)

Answers! Finally!

Casey — 2005-12-07T18:23:07-06:00

A while back I'd noticed that if I had VMWare installed on my PC, VNCAdmin scans would take forever. Shortly after that, I realized that if I disabled the two network interfaces that VMWare installs, the scan speed went back to normal. This bothered me, but I assumed that it was just an isolated issue on my machine. However, I asked around and tested on other machines and found that it wasn't just an isolated issue.

Well, after a year or two of wondering, I've finally figured out what the issue is. It has to do with using the gethostbyaddr() function for retrieving hostnames when all you have is an IP address. This function will use several different methods of finding the hostname if it needs to before giving up, (I forget in which order it does these, or even what all the methods are :) ). At some point, this function will try sending NetBIOS name requests to the remote PC. The issue here is that it sends NetBIOS requests on each network interface that is defined on your host. So we're doing a lot more work, for the same result, (If you wonder how I know this, it's because I've sniffed the network traffic that the function generates). However, these VMWare interfaces are configured with 192.168.0.0/24 addresses, so they may not even be communicating with your network. I would assume that this would cause gethostbyaddr() to wait for responses on those interfaces or timeout when it doesn't get one, thus slowing it down even more.

However, now the question is, "How do I get around this?". Do I look for a newer reverse lookup function? Do I just write my own name resolution code by crafting and injecting DNS/NetBIOS packets? Do I ignore this as it's probably a pretty rare occurrence? We'll see I guess...

I can't help but wonder if the NetBIOS name requests being sent to each interface is a protocol spec, or if it's just the way Microsoft decided to implement the function? Perhaps neither I suppose. Either way, it's great to finally know what on earth was causing the slowdown. :)

Yavar source code is now online!

Casey — 2005-12-04T17:14:50-06:00

I've added a link under the downloads section for the Yavar source code. For now, the interesting networking code is in VA_Utils.cpp. Enjoy!

Disclaimer: This is all alpha testing code, use at your own risk! :) I've not included any makefiles or build instructions as of yet since the project is so young.

while( updates == 0 ) { progress++; }

Casey — 2005-12-03T14:04:13-06:00

It's been a while since my last update, but there's been a LOT of progress in the last month. I've revamped my scanning code to be much easier to use and more moduler, (however it's still not in it's final form :) ). This has made adding features and probe types fairly trivial.

At the moment I'm able to scan both internal and external subnets. What's the difference you ask? Well, when building a packet you start from the Ethernet header, then onto the IP header, then the TCP header, (assuming we're creating a TCP/IP packet). In order to write an Ethernet header you need the MAC address of the remote host. For local subnets, this is the MAC address of the remote machine itself, and this information can be retrieved by sending broadcast ARP requests and then sniffing for the replies that contain the remote hosts MAC address. However, on external subnets, the MAC address to the remote host that gets entered into the Ethernet header if the MAC address of the sending hosts default gateway!

So I needed to find a way to get the sending hosts default gateway IP address. Once I have that, I can send an ARP request for that IP address in order to get it's MAC address. This was easier said than done, (well, it's only currently working on Windows, I'll be working on Linux soon). However, stepping through the Nmap source code was a big help. Thanks Fyodor!

I've also added the ability to send a ping request to external subnets in order to find all the live hosts before actually sending the probe packets that test port status. This makes it so I'm not sending SYN, (or other probe types), packets to every host in the scan range. Why this is a good idea, I'm not quite sure yet. :) I've only done this for external subnets for now at least, since ARPing for the hosts on the local subnet seems to fill this function already.

So at this moment, this VNC scanner is enormously faster than the old VNCAdmin. However when comparing scan results to the old scanner and Nmap scans, I noticed I wasn't finding the same number of results. So I thought I'd throttle the rate at which I sent probe packets a little bit since I had been sending them to the network card as fast I could generate them. Adding a few milliseconds of sleep time between each probe gave me perfect results that now matched my old scanner and Nmap. w00t!

However using this kind of timing mechanism makes me a little uneasy for some reason, so I'm wondering if there's a better way to throttle sends...

Latest adventures with libnet

Casey — 2005-10-27T00:13:09-06:00

I've recently decided to forgo writing all the low level details of packet creation and manipulation and instead take full advantage of the fantastic libnet library. This has many benefits, including not reinventing the wheel, using a library that's already been debugged and tested through use in many applications, and progress will just be much faster. :) The only reason I wanted to do the low level stuff myself was to learn from it, but I feel I've got a good handle on it already, and having looked through the libnet source code, it's VERY clean and readable. If I wanna know how they did something, I can easily find out. Ya gotta love open source. :)

So having decided this, I started using libnet's packet building functions instead of building the packet myself and just using libnet's libnet_write() function to put it on the wire. After getting by my ignorance, things fell into place. Note to self: when working with libnet, you need to build the packet top down instead of bottom up, (TCP header, then the IP header, then the ethernet header). So at the moment, I'm sending probe packets on ranges of IP addresses, however I still need to add the code to get the replies or lack thereof back.

I also found a link to a really good Powerpoint presentation about the latest incarnation of the libnet library that can be found here.

ARP this!

Casey — 2005-10-17T22:45:41-06:00

So I've recently made a small breakthrough on the scanning side of things. I've managed to send raw ARP requests and sniff the replies off the wire. The nice thing about this is that I no longer need to hard code MAC addresses for my target hosts into my code. W00t! This has larger implications as it verifies that the actual packet sniffing functions in libnet don't block, so I can start sniffing and call the packet processing functions that DO block at a convenient time, or maybe a different thread? Good food for thought! This also lays out the basic structure of the code I need to write to send a SYN packet and wait for the reply.

I've also been thinking of how to start integrating this network code into my wxWidgets code for the VNCAdmin rewrite. It feels awkward since the wxWidgets code is nice and tidy in their C++ classes, however the network code is some down and dirty C splattered all over the place. Oh well, we'll get there. :)

Why do now, what you can do later?

Casey — 2005-10-11T20:22:58-06:00

So after playing with libpcap a little more, it seems like I can start capturing packets before I actually start processing them with pcap_loop(). This is good news because the capturing doesn't block the application. So I can send my SYN packets and do whatever sending needs to be done, and then process the captured packets at a more convenient time to get any answers back from the target. This will allow me to avoid using threads a little longer, leaving my attention squarely on scanning related topics. w00t!

What's in your packet????

Casey — 2005-10-10T22:48:31-06:00

For the past few weeks I've been delving as deeply as I can into TCP/IP coding, especially dealing with raw sockets and raw packet data. It's terribly interesting. My first epiphany was that a packet coming off the network card is just a string of characters, (unsigned chars). To make working with this string easier, you can map it directly to C structures which allows you to fill in the specific fields of the packet and shoot it off to the wire. This also works in reverse for reading and getting data out of the packets. Excellent!

So now I'm reading and writing the raw packets I need to be able to send any kind of scan I want, (SYN, FIN, etc), but I need to be able to read the right ones! I can send a SYN packet to a machine, but I need to get the reply back, whether it be a SYN/ACK or a RST or any other possibilities. Using the libpcap library I can capture/filter packets easily, however it seems the library functions for capturing block program execution like a stop light, so I'm thinking that I need to look into spawning a packet filtering thread. Sounds reasonable to me.