gushi: (Default)

It's the holiday time, and the IT-driven life I lead has been a little slow, as nobody wants to make sweeping changes over the holidays. Instead, I'm once again playing with my personal code, and my vendetta against the people who abuse servers.

In my last post, I detailed how bad DNS queries can indicate that someone is querying nonstandard sources on your DNS server. For example, someone querying your servers for the A records of the google mail servers.

What I've discovered since then is that with a tiny bit of perl, I can run through a day's logs, and feed the data into a hash, to de-duplicate them. Then, with the magic of nsupdate, I can feed them into an RBL that my mail servers will query. I don't even need any intermediate database.

So What's an RBL?

I'm sure those of you who use SpamAssassin or whatnot may know what an RBL is: it's a Realtime BlockList. Your mail servers do a special dns lookup. For example, if the ip address 1.2.3.4 is connecting to you, your mail servers may query the rbl at "zen.spamhaus.org", by first reversing the ip address, and then appending it to the blacklist name.
So 1.2.3.4 becomes 4.3.2.1.zen.spamhaus.org. If a lookup of that returns an ip, it's listed. Typically, the ip returned is in the 127.0.0.x range. On some blocklists, certain return codes have certain meanings.

Generate your key.

Most people who do dynamic updates with BIND, use a security method called TSIG (Transaction SIGnatures). The key in these cases is a "shared secret", and needs to be chunked into named.conf. This is the "Old Way" of doing things.

In my instance, I am using something most people don't get, called sig(0). Instead of having to put my keys in my named.conf file, I simply list them right in my zone. Instead of being the standard HMAC-MD5 keys that one sees using TSIG queries, I can simply tell any given party "generate a key, send me the public component" and never worry about the secret key crossing the wire. (Yes, to be sure, I should tell them to pgp sign it to make sure it's not modified in transit). The real beauty of this is that with a properly-crafted update-policy, I can set things up so that future keys can be added with nsupdate, and I never have to touch named.conf to add "feeders" again.

The command I ran to generate my keys was:

dnssec-keygen -a RSASHA1 -b 512 -k -n HOST rbl.gushi.org

That will give me two files: a .private file that's a bunch of Field: Value statements, and a single .key file which contains my key, in a resource record. While the format of these filenames may look very similar to those used for DNSSEC, the records being generated are of the "KEY" type, whereas DNSSEC uses "DNSKEY" records. I copied these to my home directory.

Create the zonefile

After generating the key, I created a basic zone:

$TTL 360        ; 6 minutes
rbl.gushi.org.           IN SOA  prime.gushi.org. root.gushi.org. (
                                2009123678 ; serial
                                7200       ; refresh (2 hours)
                                7200       ; retry   (2 hours)
                                604800     ; expire (1 week)
                                360        ; minimum (6 minutes)
                                )
rbl.gushi.org.          NS      prime.gushi.org.

And then added my key statement:

rbl.gushi.org.          KEY     512 3 5 (
                                AwEAAbt55viC4mTSNbvlZlEM9QN/aDRAcBiItmmGylNV
                                GDw9eBLF71TBtzF/zVLUExsptCj3ez/wYstkQjfWGfjO
                                zl0=
                                ) ; key id = 65002

Note that the contents of the .key file were literally:rbl.gushi.org. IN KEY 512 3 5 AwEAAbt55viC4mTSNbvlZlEM9QN/aDRAcBiItmmGylNVGDw9eBLF71TB tzF/zVLUExsptCj3ez/wYstkQjfWGfjOzl0= on a single line, but the above pretty-wrapped line was generated after the zone got rewritten by named.

I made sure to put the zonefile in a directory where named could write to (it would be overwriting the zonefile, as well as creating a ".jnl" file in the same directory).

(I also added an NS record in my main zonefile, pointing to prime.gushi.org exclusively for this zone), as well as specifying an update-policy in my zone definition in named.conf:

zone "rbl.gushi.org" {
    type master;
    file "d/rbl.gushi.org.hosts";
    update-policy { grant rbl.gushi.org. subdomain rbl.gushi.org A TXT; };
};

That update policy basically tells named that the key labeled rbl.gushi.org can update any subdomain of rbl.gushi.org, but only the A or TXT records. While this policy is pretty granular, named currently lacks the ability to say things like "this key can only add but not delete TXT records" or "this key can update records, but cannot change the number of records (i.e. it may not add two A records).

I then issued an "rndc reconfig" to tell named to reload the config files (in the "old days" this would have been done with a SIGHUP or by stopping and restarting the process. Then I checked the logs to be sure the reconfig had run, and that the new zone had been loaded (rndc doesn't tell you these things).

Testing dynamic updates

After that, I ran the following:

   prime# nsupdate -k /home/danm/Krbl.gushi.org.+005+65002.private
   > update add test.rbl.gushi.org. 3600 A 127.0.0.1
   > send
   > update add test.rbl.gushi.org 3600 TXT "this is a test"
   > send
   > quit

Then, a dig to test it:

   prime# dig @prime test.rbl.gushi.org ANY
   ; <<>> DiG 9.6.1-P1 <<>> @prime test.rbl.gushi.org ANY
   ; (1 server found)
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50427
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 1

   ;; QUESTION SECTION:
   ;test.rbl.gushi.org.            IN      ANY

   ;; ANSWER SECTION:
   test.rbl.gushi.org.     3600    IN      TXT     "this is a test"
   test.rbl.gushi.org.     3600    IN      A       127.0.0.1

   ;; AUTHORITY SECTION:
   rbl.gushi.org.          3600    IN      NS      prime.gushi.org.

   ;; ADDITIONAL SECTION:
   prime.gushi.org.        360     IN      A       72.9.101.130

   ;; Query time: 0 msec
   ;; SERVER: 72.9.101.130#53(72.9.101.130)
   ;; WHEN: Sun Dec 27 01:56:55 2009
   ;; MSG SIZE  rcvd: 115

From there, if I were to do an rndc freeze rbl.gushi.org, I would see it show up in the zonefile. Until that point, it would live in a journal (.jnl) file maintained by named.

At this point, I had a zonefile that I could add things to, and remove them from, but getting the data out of my logs was still a problem.

Parsing my logs

Enter perl. While programmers may love python, and web programmers may love ruby, to the sysadmin, perl is still the "swiss army chainsaw" we reach for in such a situation.

A quick well-caffienated night (christmas night) of programming yields this script, which works for me (and in fact, will work from multiple systems, even across the internet). I just run it against the requisite logfiles, and I'm set.

After running it, I issue an "rndc freeze rbl.gushi.org" to tell named to save the data back to the master zonefile and look at the zonefile:

$ORIGIN 109.rbl.gushi.org.
$TTL 3600       ; 1 hour
0.88.110                A       127.0.0.2
                        TXT     "Last seen Dec 26 16:11:19 polling for ASPMX4.GOOGLEMAIL.COM/A/IN"
144.28.121              A       127.0.0.2
                        TXT     "Last seen Dec 26 17:50:46 polling for ALT1.ASPMX.L.GOOGLE.COM/A/IN"
96.136.184              A       127.0.0.2
(...)

And so on.

Note that the format of those files is a little annoying: named currently lacks a config knob to tell it to dump in any format but that one, although I've submitted a suggestion request. I personally feel the above would be more readable if every entry were fully-qualified, without the $ORIGIN statements.

I'm also able to do digs against the above. (And so can you: dig 109.110.88.0.rbl.gushi.org ANY: try it!) Once I'm satisfied, and have made any manual tweaks, I can do an rndc thaw rbl.gushi.org and updates are allowed again.

Another feature subtly absent from BIND that I've put in for, is that there's no way to tell it "just save the data to the master, but don't stop accepting updates". Time will tell there.

Configuring Sendmail

Once I'm happy that the data is going to an easily-pollable source, I simply need to tell my mailer to read it. While I won't make any assumptions about the installed-based of readers here, I run FreeBSD, which comes with sendmail by default, and that's what I use.

Sendmail contains built-in support for this type of RBL. In addition to the other ones I poll, I simply added the following file to my sendmail.mc:

FEATURE(enhdnsbl',rbl.gushi.org',Message from $&{client_addr} rejected - GushiSystems Blocklist')`

I chase that down with a quick make install-cf && make restart and I'm ready to go. At any point, I can grep my maillogs for "GushiSystems Blocklist".

Sadly, what I'm finding (although as is typical, I didn't find it when I discovered the problem), is that the CBL detects most (but not all) of these as the cutwail SpamBOT, a rather massive command-and-control botnet.

Does that make this effort wasted, though?

Not at all. After all, now that the updating code is written, it's trivially easy to take the same code and do other things with it.

For example, it's still relevant in a DNS context to show how I could turn this on my maillogs, where both sendmail and spamassassin log, and do neat things with a few lines of perl like "blacklist any ip that sends a spam that scores over 20 points AND sends to three distinct domains in an hour. The above is literally five lines of code to correlate the requisite log entries, another ten perhaps to act on them.

Right now, the perl code is written to builds its big hash of information in a single pass (when my logs are rotated), and then write to the zonefile all at once. A trivial enhancement would be to have it listen on a named pipe, or socket, and periodically flush its cache, so it didn't grow forever over time.

I could at the same time, use the code to feed a list of people trying to send Guestbook Spam, or people posting to dead/abandoned phpBB boards. Again, most webservers don't have the ability to poll this sort of a database, but there's no reason it can't be easily added.

A minor footnote

I'm not sure where it stands in the POSIX standards or whatnot, but I've discovered that the date format used by FreeBSD's syslogd both does not log the year, and is non-configurable. Considering as I write this we're four days away from the end of the year, I may have some tweaky date-logic to write (keeping in mind that this script currently runs on the previous day's logs), so assuming the current year isn't the best answer.

On that note, I'd like to wish everyone here a safe new year. Here's looking forward to more cool stuff in 2010.

gushi: (Default)

Hello there! Do you care more about the underlying protocols that drive the internet and that make hostnames in general possible? Would you like to know how to take debug output of the most prevalent DNS software in use anywhere, and help the internet to run better? Are you just an ouright NERD? Then perhaps you should read... A howto about DNS logging. )

gushi: (Default)

I think it's pretty safe to talk about this publicly.

I just discovered something obnoxious in the course of my day.

Most unix machines have a "hostname", and this "hostname" includes a "domain". When your hostname is stated with your domain, it it said to be your "fully qualified domain name". For example, "prime.gushi.org", or "bitsy.mit.edu".

This is all well and proper. This is the way it has been in the unix world since TCP was invented. A computer knows its first and last name, and it corresponds with the name that systems use to look you up with, in protocols like the DNS.

Now, with Microsoft OSes, machines normally get their "domain name" by joining an Active Directory Domain. For example, in company.com, they may designate "ad.company.com" to be the active directory domain. Note carefully that this also sets something in windows called your "primary dns suffix" which means "the domain part of your hostname".

Normally, the procedure involved in setting this thing manually involves digging rather deeply into the system control panel, going to the "Computer Name" tab, clicking the "More" button, and setting it.

Now, here's the annoying thing I recently discovered:

I recently decided to run a very-tight box to only serve one thing: DNS (running my job's software). Thus, knowing that I'd never share any files, never want to connect to any servers to grab files, I uninstalled "Client for Microsoft Networks" and "File and Printer Sharing for Microsoft Networks".

What I then discovered was this:

After uninstalling these components, the "More" button in the Computer Name field disappears!!! Unless you have "Client for Microsoft Networks", your machine CANNOT POSSIBLY be configured with a fully-qualified domain name. Worse still is that windows machine names cannot include dots.

Note carefully that some other programs use this value. Some mail servers even DEPEND on it being set to something real.

Does Microsoft POSSIBLY think that the only thing that requires an FQDN is their own SMB networking stack?

Configuring a DNS search path is also done per-connection, as opposed to globally. How does that work? If I'm at a command prompt and type "ping foo", it doesn't ask me which network interface I want to use (although ostensibly the one that has my default gateway would be the primary one). It's still a kludge.

Now, I'd be a lot more angry if this actually stopped me from doing anything, this is purely a semantic issue, but hey, I work in the DNS field, I'm allowed to be a pedant about this.

gushi: (Default)

I think it's pretty safe to talk about this publicly.

I just discovered something obnoxious in the course of my day.

Most unix machines have a "hostname", and this "hostname" includes a "domain". When your hostname is stated with your domain, it it said to be your "fully qualified domain name". For example, "prime.gushi.org", or "bitsy.mit.edu".

This is all well and proper. This is the way it has been in the unix world since TCP was invented. A computer knows its first and last name, and it corresponds with the name that systems use to look you up with, in protocols like the DNS.

Now, with Microsoft OSes, machines normally get their "domain name" by joining an Active Directory Domain. For example, in company.com, they may designate "ad.company.com" to be the active directory domain. Note carefully that this also sets something in windows called your "primary dns suffix" which means "the domain part of your hostname".

Normally, the procedure involved in setting this thing manually involves digging rather deeply into the system control panel, going to the "Computer Name" tab, clicking the "More" button, and setting it.

Now, here's the annoying thing I recently discovered:

I recently decided to run a very-tight box to only serve one thing: DNS (running my job's software). Thus, knowing that I'd never share any files, never want to connect to any servers to grab files, I uninstalled "Client for Microsoft Networks" and "File and Printer Sharing for Microsoft Networks".

What I then discovered was this:

After uninstalling these components, the "More" button in the Computer Name field disappears!!! Unless you have "Client for Microsoft Networks", your machine CANNOT POSSIBLY be configured with a fully-qualified domain name. Worse still is that windows machine names cannot include dots.

Note carefully that some other programs use this value. Some mail servers even DEPEND on it being set to something real.

Does Microsoft POSSIBLY think that the only thing that requires an FQDN is their own SMB networking stack?

Configuring a DNS search path is also done per-connection, as opposed to globally. How does that work? If I'm at a command prompt and type "ping foo", it doesn't ask me which network interface I want to use (although ostensibly the one that has my default gateway would be the primary one). It's still a kludge.

Now, I'd be a lot more angry if this actually stopped me from doing anything, this is purely a semantic issue, but hey, I work in the DNS field, I'm allowed to be a pedant about this.

gushi: (Default)

Okay,

So after fixing my little mail logging issue I remembered that I had logwatch set up on my cobalt raq3.

Logwatch is cool. It emails you everything in the logfiles, you define great regular expressions as to what's harmless noise, and keep going till it's only the critical stuff that you get.

I just got a mail FULL of the following:

client 123.17.150.226 query (cache) 'mail.peregrinehw.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ALT1.ASPMX.L.GOOGLE.com/A/IN' denied: 1 Time(s)
client 123.18.118.42 query (cache) 'ALT2.ASPMX.L.GOOGLE.com/A/IN' denied: 1 Time(s)
client 123.18.118.42 query (cache) 'ASPMX.L.GOOGLE.com/A/IN' denied: 1 Time(s)     
client 123.18.118.42 query (cache) 'ASPMX2.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ASPMX3.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ASPMX4.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ASPMX5.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ALT1.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.213.68 query (cache) 'ALT2.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.213.68 query (cache) 'ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)     
client 123.19.213.68 query (cache) 'ASPMX2.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ASPMX3.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ASPMX4.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ASPMX5.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.59.189 query (cache) 'mail.peregrinehw.com/A/IN' denied: 1 Time(s)   
client 123.19.99.134 query (cache) 'ALT1.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.99.134 query (cache) 'ALT2.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.99.134 query (cache) 'ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)     
client 123.19.99.134 query (cache) 'ASPMX2.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.99.134 query (cache) 'ASPMX3.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.99.134 query (cache) 'ASPMX4.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.99.134 query (cache) 'ASPMX5.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  

So after I dig around for a bit (no pun intended), I realize.

What I'm looking at is a whole bunch of terribly broken DNS implementations. DNS implementations that bypass a host's DNS entry, and directly query ME instead of looking something up directly.

All the domains above are A records (address records) that are pointed to by MX (mail exchanger) records. I host sites that use those MXes, but I don't host (obviously) googlemail.com.

Okay, so I know why this is happening. It's mostly harmless.

My options:

1) Tune logwatch so I don't get these.

2) Tune BIND so it doesn't log these hits.

3) Use this information to feed a real-time blacklist -- it's fairly easy to write the parser but from the looks of it, most of these IPs are already on RBL's I use (spamhaus PBL, CBL).

4) Find a way (as recursive as this sounds) to block queries to my DNS server, based on this blacklist. I don't think BIND supports such a feature.

-Dan

gushi: (Default)
So I was tapped a few minutes ago to head into the city to replace a piece of equipment at one of the NYC telecom hotels. Writing this on the train now, will post later I guess since this laptop can't talk to the phone.

Well, that's a misnomer actually -- the phone itself can make LJ posts, and the phone itself has an SSH client. And since both speak IR, I could beam this whole post over to the phone and then upload it.

But that's overkill, ne?

Anyway, I've looked into the SSH key thing I mentioned before. And I've decided it's absolutely stupid.

Basically, publishing the keys in NORMAL dns isn't enough. You have to be using the DNSSEC secured DNS extensions. What this means is every time I changed the gushi.org zone, I would have to generate a signature (nothing inherently hard about that, everything's scripted anyway). I would have to publish that signature in my DNS. And then, what's worse, is that everyone else, has to be using a DNS client that UNDERSTANDS the security enhancements, and passes on the "yes, this is secure" data to the end user. Worse still, each of those DNS servers has to accept, and TRUST my DNS public key. So if you're a user on a dynamic comcast IP (and presumably using the comcast DNS servers), Comcast would have to accept my key and include it into their system. AND, they think you should be running some encrypted protocol between yourself and comcast's DNS servers, like IPSEC.

Now, why the hell I can't just take my GeoTrust certificate that says "Yes, we've certified that this person runs gushi.org", and stuff THAT into my zonefile (this is how Sendmail, Apache, ProFTPd, Webmin, Usermin ALL work)...and then comcast would say "we believe in GeoTrust, and they say to trust you, therefore everything seems to be in order".

Of course, the system outlined above seems to be a replacement for caching the keys locally, which is even more stupid. All I'd like to see is something like this.

shell#ssh danm@prime.gushi.org
checking dns for prime.gushi.org...
key found in DNS...

the ssh key coming from prime.gushi.org, id aa.aa.aa.aa.aa.aa.aa is not known,
HOWEVER, it *does* match the key found in DNS, as retrieved from ns2.gushi.org

Would you like to continue connecting? (y/n)



From there, it would be business as usual. The key caches would still be used, instead of relying purely on DNS. SSH would still check the key cache, and would still bitch heavily if the connecting public key didn't match the one in the cache. Period. This would only serve as a method of distributing the key that makes more sense than "just type yes".

I suppose, optionally, that this kind of thing could be checked *every time*...but the cache would still be preferred.

Now, it's assumed that someone with enough brains to spoof a man-in-the-middle attack would also be able to spoof the DNS query that grabs my key (that's why the guys were talking about DNSSEC).

The other thing I'd love to see, as an optional "comment" field in the key, is a how-to-verify field.

The ssh key aa.aa.aa.aa.aa.aa.aa.aa is not known, however, the creator of this key has stated that this key may be verified in any or all of the following way(s):

NOTE: You should personally check as many of the following as you feel are necessary to verify that this id is authentic.

"see url: http://www.gushi.org/keyinfo.txt"
"if in doubt, call Gushi at 1-866-LI-GUSHI, dial option 12"
"Check http://www.livejournal.com/userinfo.bml?user=gushisystems"
"Key fingerprint should be sent out in the footer of signup e-mails"
"fingerprint is printed on the back of Gushi's business card"

Now, of course, those methods are easily compromisable too...but security is a layered thing, but it's assumed that if someone is running a man-in-the-middle attack against prime.gushi.org, that they won't be able to gafutz with ALL those methods.

Doing the first bit, the DNS lookup, could be tweaked with only patching to the SSH code...right now, the spec states:

2.4 Authentication

   A public key verified using this method MUST NOT be trusted if the
   SSHFP resource record (RR) used for verification was not
   authenticated by a trusted SIG RR.

   Clients that do validate the DNSSEC signatures themselves SHOULD use
   standard DNSSEC validation procedures.

   Clients that do not validate the DNSSEC signatures themselves MUST
   use a secure transport, e.g. TSIG [9], SIG(0) [10] or IPsec [8],
   between themselves and the entity performing the signature
   validation.


Of course, the spec (http://www.snailbook.com/docs/dns-fingerprints.txt) also states "Expires March 5, 2004" so I'm not sure how real this is. I think I could make a serious motion toward getting this made real.

The sourceforge SSH servers got whacked a while ago, and a lot of people wound up revealing their sourceforge ssh passwords to the thing. The hackers were then able to log into the sourceforge shell accounts, and use the STORED KEYS that people had there to jump to other places. People actually VERIFYING KEYS would help this a lot.

As for the second part, the key "extensions", those would probably lead to widespread breakage, and we'd probably have to wait for the widespread adoption of ssh3 (which I'm not even sure is a draft yet).

May 2017

S M T W T F S
  123456
78910111213
14151617181920
21222324252627
28293031   

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 22nd, 2017 04:47 pm
Powered by Dreamwidth Studios