gushi: (Default)

A lot of the people I host are not coders. They don't understand things like php, globally scoped variables, deprecation warnings, database authentication plugins, insecure hash types, or the like.

They know only that they have code that worked fine for a decade, and then Some Jerk Ferrit did something that made their site not work.

Most of this is because PHP, as a language, is a Shit Show. The only reason PHP scripts are not still majorly responsible for most of the botnet activity on the internet is because someone decided to make smart light bulbs with globally routable ipv6 addresses.

Coding in php is like trying to sculpt something in clay, except that people keep dumping ingredients in the clay that change its consistency: sand, water, cement, cheerios.

For an admin, php is a security nightmare: you have 300 users, whose code can all alter each others' files. Oh, and on most webservers? Users can't alter the files PHP created. They're owned by the "www" user.

Shit. Show.

So, because vague reasons, the people who make the PHP language decide that a particular function is not workable in the particular coding style that they feel people should be using at that time. So, somewhere in a README file that nobody actually reads, they say "hey, you should stop using this function, it may go away in the next version".

I hosted several hundred websites at one time -- nobody knew about that README file, which, as far as they knew, were on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.”

So, a long long time ago, I solved two birds with one stone. I installed a program called "suPHP". What suphp brilliantly does, is sacrifice some of the speed normally present in PHP, by running everyone's PHP scripts as them. It does this by decoupling PHP from the webserver, and winding up a tiny little PHP process to spawn your files.

The unexpected side effect here, is that it can run different versions of PHP for different users.

Now, as far as the operating system is concerned, you can only install packages for one version of PHP at a time, and right now, at the time of this writing, that's PHP56, with a bunch of removed functions and deprecation warnings.

I've been building PHP from scratch for years, tho, and I know how to install a tiny little shadow copy of an older version of PHP where the webserver can get at it.

So, if you were going to go look at: this page, you'll see a php info page that talks about php version 5.5. If you look at this default one, you'll see that it in turn is running php 5.6.

In fact, I even have a separate copy of apache running with the mod_php going on, for my webmail, where I can use the speed.

Best part? You can control it.

If you were to look at this htaccess file, you can see how easy it is to signal to the interperter that you want 5.5. (Normally, apache won't serve .htaccess files out to the world, this one is special). Basically three lines of code:

<FilesMatch "\.php$">
  SetHandler application/x-httpd-php55
</FilesMatch>

In a former life, I let people use this to switch between php4 and php5. RIght now the only handlers are php5 and php55. I could maybe add php54 as well.

That said -- if you possibly can, I advise using upgraded code that supports the latest thing. So if you're running something like Wordpress, please do update. If on the other hand, you have an old copy of Gallery, and it's not being hacked or hammered, and it suddenly broke, the above will fix it.

gushi: (Nevar Button)

Mitigating a Mail Server Attack

Analysis

A few days ago, I noticed an unusual number of bounceback messages from one specific user directed at yahoo.com email addresses. When I looked inside the mail queue, I noticed that each message had dozens of recipients.

The troubling thing is, looking at the mail logs, I saw the dreaded line

maillog.7.bz2:Aug 18 21:33:08 <mail.info> prime sm-mta[80096]: 
AUTH=server, relay=[84.32.158.42], authid=vivian@littlestdomain.com, 
mech=PLAIN, bits=0

The AUTH=server bit tells me that rather than a rogue script running here (a not-uncommon thing that happens when you let users run PHP scripts), this was an actual password that got leaked, and was being used to send mail just as a regular user would.

I quickly concluded "compromised account", changed the user's password, and contacted them out-of-band with a new password. Life seemed good.

...then I noticed a second account doing the same thing. Okay, that's weird. Maybe my users have started falling for a really effective phishing scam?

When it happened a third time, a few days later, I recalled the old Ian Fleming quote:

"Once is happenstance. Twice is coincidence. Three times, it's enemy action."

Combat Perl

So, faced with the task of seeing if there were any other users who were affected, I wrote a short little bit of perl code to analyze my mail logs, and spit out each login, as well as the number of times each IP had logged in.

The code looks like this. No, there's no fancy "use Strict" or anything like that. I used YAML as an output format because writing "Dump" is easier than writing a foreach loop to iterate over the hashes.

#!/usr/bin/perl

use YAML;
open FOO, "/usr/bin/bzgrep -i \"auth=server\" /var/log/maillog.0.bz2|";
my @lines = <FOO>;
my %thing;
foreach my $line (@lines) {
  chomp $line;
  if ($line =~ /\[(\d+\.\d+\.\d+\.\d+)\].*authid=(\S+),/) {
        print "ip address $1 authid $2 found in $line\n";
        $thing{$2}{$1}++;
  }
}

print Dump %thing;

Combat perl output

The code above produced output like what I have here. Note that I've altered all the logins and none of the below actually exist on my system. The IP addresses and counts, however, are real.

--- jim
---
209.85.219.47: 1
216.15.173.10: 4
--- bob
---
107.144.99.110: 1
--- moe
---
107.144.99.110: 10
--- curly@curly.net
---
163.23.34.45: 9
177.19.195.242: 10
177.55.38.11: 3
177.6.83.93: 1
179.182.193.136: 2
179.182.198.161: 3
186.222.71.36: 2
187.110.215.243: 6
187.35.157.21: 1
188.11.56.33: 4
189.75.25.147: 9
190.220.7.227: 2
193.152.213.171: 7
193.32.77.45: 4
200.63.165.85: 2
201.15.121.108: 3
201.216.221.245: 3
201.231.237.7: 4
201.41.166.52: 5
201.7.227.106: 1
201.77.115.15: 2
212.174.233.143: 4
212.40.242.27: 6
213.98.140.39: 4
213.98.78.213: 2
217.149.97.78: 5
217.216.67.202: 2
217.7.241.62: 1
219.92.29.90: 10
31.145.8.126: 3
37.159.194.66: 8
46.25.232.183: 1
50.79.128.153: 4
62.42.1.181: 1
66.17.46.53: 4
72.67.57.38: 3
77.229.13.35: 7
77.231.97.130: 7
78.186.71.8: 7
78.188.184.173: 5
78.188.19.231: 6
78.189.53.17: 2
80.35.0.235: 3
81.133.208.133: 4
81.142.195.149: 3
81.149.106.151: 2
81.214.85.206: 7
81.215.226.231: 13
81.218.104.229: 4
81.45.181.255: 4
82.108.126.242: 1
82.127.51.45: 2
82.153.165.168: 8
82.85.115.185: 5
82.91.75.35: 12
83.19.111.210: 3
83.56.26.36: 8
85.105.139.205: 7
85.105.85.72: 8
85.152.27.1: 2
85.251.7.78: 1
85.71.229.247: 3
87.139.118.153: 3
87.22.242.43: 2
87.23.197.245: 4
88.103.141.154: 6
88.12.31.52: 1
88.2.171.129: 4
88.247.171.72: 5
88.247.78.60: 5
88.248.23.202: 4
88.249.248.163: 4
88.250.240.101: 3
88.87.205.230: 3
88.9.119.148: 5
89.119.131.171: 4
92.56.76.179: 7
93.207.41.7: 1
95.236.181.166: 7
95.243.214.226: 1
95.60.124.88: 8
--- thing
---
67.68.160.180: 1
--- stuff
---
207.161.109.87: 2
--- steve
---
209.85.215.49: 1
209.85.217.180: 1
--- joe
---
70.117.105.120: 1

So, one of these things is not like the others. It's understandable that a person may have two or three ips in a given period. Their ips change, they're logging on from multiple computers.

Remember that these are ONLY the ip addresses pulled from the sendmail logs. Only on connections where a piece of mail is sent, using SMTP auth.

So, that curly@curly.net entry? Yeah, that is that what security researchers call a "snowshoe" attack -- not one server sending hundreds of mails (which would be easy to block), but instead, it's spread out, and even though I now have a list of ips I could block, what we're looking at here is a botnet of otherwise compromised machines on a dozen or more ISPs.

The other thing to note about the curly@curly.net entry is that it's a full user@domain entry. Put another way, it's an email address -- one where the LHS (left hand side) just so happens to match the user's actual login.

What was going on here is that the way I (and most people) do SMTP auth in sendmai, there can be a concept of multiple "realms" defined -- for example, to log in against different authentication databases. As I'm not using this feature, the realm and everything after is ignored (but still logged).

As I normally instruct my users to only use the barename to log in, any login using a full realm must be a compromised account.

Notifying the User

So, there's a problem here. While I can easily change the user's password and send them mail, this effectively locks them out of their account and keeps them from getting anything done until we touch base.

What I wanted to do was find a way to block the users who were using the "bad" format, while letting good users go on. I wanted a quick, guilt-free way to block the sending of mail, without breaking the communication link.

What I discovered was a ten year old post in the old usenet group comp.mail.sendmail, here.

With a little bit of tweaking, I had applied that same config to my own sendmail, and had configured a line in the access database to block a test user. The account still worked, but it wouldn't let them send mail. Perfect. I could block "curly@curly.com" without blocking "curly". (And yes, this relies on a little bit of obscurity -- but it's a botnet, not monkeys at typewriters, it's only going to try what it knows).

Identifying the source

So, three accounts with relatively secure passwords compromised in a week. What was the common thread? Could these people have all used the same insecure passwordless wifi networks? Is there some newfangled router exploit that mails your traffic all off to the highest bidder?

I spoke to all the users. None of them had fallen for any phishing emails. They were running different OSes, so a password-stealing virus was out. And then it hit me. Like a ton of bricks.

I've recently seen a surge of spam to addresses like macromedia-at-gushi.org, adobe-at-gushi.org, adobe2012-at-gushi.org.
The reason?

Well, it's all because adobe sucks at securing your data.
Sometime last year, people were able to download 150 million usernames and passwords from adobe's backend servers. And, as the article I just linked will tell you, those passwords were encrypted weakly, and in a way that all the users had the same encrypted password string, for a given password.

While I'm not 100 percent sure this was the attack vector -- there's been several other leaks that happened (last.fm, Linkedin, E-harmony), and about 90 percent sure that this is the likely cause, even if I don't know which site it was that ultimately spilled my users' beans.

gushi: (Default)

Let's face it. Tokens are just cool to geeks.

While early on, working for a pi firm in 1999, I used Cryptocards, I was hooked.

Here was one of those little things that gave you a challenge, and you had to put into your cryptocard, get an answer back, and type out the response. A correct response couldn't be used more than once. On top of everything, the cryptocard itself had a PIN when you turned it on. It was, in a word, pretty foolproof. There's always the option of "put a gun to my head and ask for the pin", but I think the things even have a "duress pin", which bricks the device.

Cryptocards aren't user-friendly, tho. It gets annoying.

The most popular maker of this stuff on the market is a little company called RSA. They have a product that they call "SecuriD". These are still considered "two-factor" authentication (something you have, plus something you know), even though they aren't technically "challenge-response". These little dogtag-sized chips just print out a cyclic number every 60 seconds. They adapt themselves to applications by "supplanting" the password mechanism. If your password is "cookie", you would type "cookieNNNNNN" into whatever password dialog box you liked. Their server knows to chop the last six digits off, try that against its crypto algorithm, and try the rest of it against the password db.

The thing I hate in general about this kind of token is that if you get logged out, you're screwed and have to wait for the next token to come up. Even worse, if you're logging in over a slow session (think cases of extreme processor slog) it's entirely possible that by the time you're prompted for your password, or by the time the system gets around to calling the authenticator, it will have expired.

Any discussion about this stuff cannot ignore S/Key or OPIE, which stands for "Onetime Passwords in Everything". This IS totally open-source, and I use it, especially in places where I don't trust the computer or the network. But it's more complex than it needs to be. In my setup, opie can be used WITHOUT ever typing your real password, so it's not quite two-factor. It's not so much about making the auth stronger, as it is about making eavesdropping useless.

Ten years ago, this was james-bond kind-of stuff, or at the very least, large-enterprise and government-contractor grade. Now, with phishing and social engineering on the rise, and people just Not Getting Any Smarter, you can get one of these things to protect your paypal account, or even your World of Warcraft Account.

I should note that it's getting increasinly common to see this done as a cell-phone app. Either one that runs over sms, and generates a key on the fly, or an app that emulates a hardware token. In general, I'm not a fan of this because SMS is forgeable, and people never, ever, lock their cellphones. Besides, having my server require a magic tamagotchi for a login is still cooler in my brain than requiring Another Piece of Software.

I've wanted to do token-based auth of some kind with my systems for a while. The two problems with this are: A) the software is expensive and B) the tokens are expensive. About a year ago, I discovered a company called MyPW, that makes a simple third-party auth answer. There's no software to buy, and they basically "loan" you the tokens, in exchange for charging a very small fee for each auth request. The tokens (and ostensibly, the software) they use are made by a company I hadn't previously heard of, called Vasco. While the algorithm the tokens use is still not open, and it appears thus, impossible to write your own implementation, at least, at the entry cost of ten dollars, Anyone Can Do This.

I've ordered some tokens, and in my excitement, have started looking at implementation details. To quote taylor swift..."And then I wasn't excited anymore." In reading up on this, I've discovered the following:

  • Near as I can tell, not every SSH client implementation has the ability to display these kinds of challenges to users. However, there's an RFC for it, so that's cool. And apparently, that RFC does specify that multiple challenges can be presented, at least in ssh version two (which everyone should be using).

  • Looking over a few things, it seems as though I can do multiple prompts for a user, with ssh (I'll have to testbed this somewhere), but it's not clear to me how the mypw Pam Module prompts a user. However, some other systems, like Webmin, are only configured to do straight HTTP auth. And I rather like that a user cannot see any part of webmin without answering the http query. (I should also note that my webmin instances don't use system passwords, as a failsafe in case something mangles the password file).

  • I can't find a pam-wise way of doing things the "RSA Way" (i.e. with your password and your token-key mishmashed).
    Surprising, really.

  • I also can't find a way that Pam Module A can modify the user's supplied password on the stack before passing it to Module B. This may totally be possible, if coded into the module itself, I just don't see a config-wise way of doing it, and don't understand the pam api well enough.

  • I think the way openssh makes its calls to pam, there could be an issue. What happens if I am prompted for my password, and get it right, then my tokenid, but get it wrong? I believe the pam stack starts over from the top, which means I have to re-type my password. Look at this the other way around. Let's say my tokenid is first, and I type it right, but I mangle my password. I have to then wait, since the system won't accept the same id twice. Allowing the pam module to maintain a cache, and permit on subsequent auths doesn't help either, since pam has (as far as I know) no concept of ip addresses or sessions. (In other words, I don't think pam can differentiate between subsequent auths on the same session, and a completely different auth session from a different machine).

Finally, in the "readme" that mypw supplies, they say to implement your pam setup as follows:

#%PAM-1.0
auth       required     pam_xmlrpc.so service=system-auth
#auth       required     pam_stack.so service=system-auth
auth       required     pam_nologin.so
account    required     pam_stack.so service=system-auth
password   required     pam_stack.so service=system-auth
session    required     pam_stack.so service=system-auth

I see at least a few things wrong here.

  1. The service=system-auth thing only applies to pam_stack, so it's probably a typo. It's a minor typo, but this is the documentation for how to let people securely log into your server, so I kinda expect a little polishing.

  2. To look at this, this basically appears to act ONLY on the token-based password, which is inherently LESS secure than anything else. After all, if you steal my car keys, there's my WHOLE password sitting in plain text, you just have to guess my username.

Other missing things include:

  1. A way to define from the pam.conf file where the config file lives.
  2. A list of what options the pam module supports.
  3. A manpage.
  4. A definition for how to determine what happens if the auth server can't be contacted.
  5. A neat way for users to manage their own tokens.
  6. A way to support user-tokens via home-directory file instead of one-big-system file.
  7. The one-big-system-file approach, since it's a text file, doesn't scale. It should probably get some kind of db-file support.
  8. A way to manage "drift", i.e. how far expired a token may be. (That is, for slow and laggy connections, I'd like to see the option to still make a password one-time-only, but allow its use up to five minutes past its prime).

I think I'm going to write to these guys and suggest that they try to push in a FreeBSD port, as well as bringing this blog post to them.

Thsi looks like a fairly basic project to work with C on, if I spoke it...but I'm not comfortable cutting my teeth on such a critical piece of software. I guess I'll see what they say.

gushi: (Default)

I'm not sure why, exactly, but sshit on oldprime never seemed to work right.

Oddly, I would get ssh authorization warnings on the CLIENT end, but things would never make it to syslog.

I finally solved it by telling opensshd to log at loglevel VERBOSE as opposed to the default. Apparently, logging at level INFO (the default) is not enough to tell you when someone is REPEATEDLY FAILING to log into your system.

And yet, on the CLIENT end, it was enough to tell you what was failing:

%ssh testcase@oldprime.gushi.org
Password:
pam_unix: pam_sm_authenticate: UNIX authentication refused

You get that? I'm getting CLIENT logs telling me what subsystem is blocking me, but not on the server.

I bumped logging up to "verbose" and all seems to be back to normal, but still, this is really stupid. And it's not the first we've seen of this.

I'm just wondering when openssh will include the auto-ignore/auto-block functionality on its own. If all clients did this, it would make bruteforce scanning pretty ineffective.

Of course, that would require everyone to upgrade. And we know that's not happening.

gushi: (Default)

So, I just got this amusing email...

From slackerng@gmail.com Thu Jul  2 02:08:05 2009
Date: Thu, 2 Jul 2009 01:07:54 -0500
From: Cody Grunenwald <slackerng@gmail.com>
To: "root@gushi.org" <root@gushi.org>
Subject: I really need help

I saw that you had a crash file that you can crash wc3 users only by whispering
them. Now im a noob with technology and stuff so i was wondering if you could
get on battle.net and go to Channel CLAN STN and crash anybody in that channel
with praetor in their name. Long story short they hacked themselves into OP and
were a new clan so we have no shamans or anything and hes holding our clan
hostage. please help us.

Note that they emailed root@gushi.org. Now, there's only one place I use that. root@prime.gushi.org is common, but root@gushi.org was ONLY used, for a while, as the ServerAdmin for the gushi.org domain (as in, ONLY my personal domain). Thing is, it also shows up as the serveradmin for people who use www.gushi.org/~username aliases...and a quick google revealed the problem.

A user I had kicked off a while ago, who was using prime as his location for starcraft hacking tools (I know because I heard from Blizzard about it).

Remember fun LJ entries like this?

So, obviously what's happening is people are finding some webpage that links to this, getting a 403, and then EMAILING ME.

Gee, how ever could I find out who this is? Oh wait, look, I have my webserver logs!

%tail -1000000 access_log|grep -i celeron
gushi.org 75.72.94.81 - - [01/Jul/2009:20:48:09 -0400] "GET /~celeron/hacks/Exended1.4.zip HTTP/1.1" 403 345 "http://gaminkings.tripod.com/id8.html" 
"Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5"
gushi.org 97.83.163.193 - - [02/Jul/2009:01:56:50 -0400] "GET /~celeron/hacks/SCCRASH.zip HTTP/1.1" 403 342 "http://gaminkings.tripod.com/id8.html" 
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"
gushi.org 97.83.163.193 - - [02/Jul/2009:02:05:37 -0400] "GET /~celeron/hacks/SCCRASH.zip HTTP/1.1" 403 342 "http://gaminkings.tripod.com/id8.html" 
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"
%

So I checked out the tripod page (I especially love that the banners on the webpage are offering a DEGREE IN HACKING), and there they are.

Now, the question is, what to do with them?

  • Tell them "it's a game, there really are more important things"
  • Tell them "STFU NOOB!"
  • Tell them I'll hack THEM for disturbing me! (Mess with the best, die with the rest!)
  • Tell them I'll do it for 1000 gold, sent to the WOW account of anyone I don't particularly like?
  • Forward the tripod hacks page on to Blizzard?
  • Compain to tripod myself, as it's causing me negative traffic, perhaps threatening that if they don't take the page down, I will POPULATE those links?
  • Since they seem so willing to download and run files, send them a nicely wrapped boot sector rewriter?
  • Two words: mod_rewrite.
  • Send them a link to this entry.
  • Do nothing but blog about it, sigh, and shake my head sadly.

Yeah, probably that last one, but you never know. *sigh* *shakes head sadly*

gushi: (Default)

Huh? What's the Bagle problem? No, it has nothing to do with not being able to find a decent bagel store in California.
(Although that's definitely also a problem.)

It's not a problem per se, except that my webserver logs have hundreds of hits like this:

prime.gushi.org 71.30.188.182 - - [06/Jun/2009:15:02:08 -0400] "GET /777.gif HTTP/1.1" 404 324 "-" "Mozilla/4.0 
  (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; GTB5)"
prime.gushi.org 98.175.208.182 - - [06/Jun/2009:15:02:35 -0400] "GET /777.gif HTTP/1.1" 404 324 "-" "Mozilla/4.0 
  (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
prime.gushi.org 71.242.102.171 - - [06/Jun/2009:15:02:41 -0400] "GET /777.gif HTTP/1.1" 404 324 "-" "Mozilla/4.0 
  (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

Know what that is? It's a virus. One that specifically tries to download an exe-disguised-as-a-gif from prime.gushi.org.
That file was never actually available for download, but (I guess because of my security activities) I was targeted.

In fact, http://prime.gushi.org has always just returned a "forbidden" page.

So here's how it will work.

1) This site now rotates its logs daily. 2) A tool goes through "yesterdays" logs and parses them (this bit is already written). 3) It all gets stuck in a database. The time-field (the bit in the []'s) gets rewritten as a unixtime to handle the annoyance of timezone conversion (since unixtimes are all utc). 4) A tool then goes through the database, looks at all the "seen today" hits, and runs abuseEmail on the ips, and complains in the right direction.
5) This also gets logged in a database, so that I can see every report I've submitted to a given abuse contact, and track historicals.

I vaugely wonder if these infected machines would accept cookies so I could track them that way. The user-agents are not unique enough in some cases.

The problem is with #4 right now: AbuseEmail seems more than a little broken. It looks for SOA records in places it should be trying WHOIS, and it seems rather uneducated about modern whois/rwhois servers. I've contacted the author. It hasn't been maintained since 2001. I might have to fork it. It might also work better as a module. I also haven't structured the DB yet, but that's trivial.

If any of you coders out there want to try and help out with this, I'm more than happy to share credit. I've suggested to my job they might be interested in this data, but there doesn't seem to be much excitement. It's one (much older) worm, and not nearly as prevalent as some of the others out there. Still, with the right amount of work, I could singlehandedly wipe a virus off the planet.

I've already considered the fact: these computers are going to download a file from me, and run it silently with administrator privileges. I could give them a virus-cleaning, or even a courtesy-reformat, but that's defintiely the wrong thing to do.

gushi: (Default)

Huh? What's the Bagle problem? No, it has nothing to do with not being able to find a decent bagel store in California.
(Although that's definitely also a problem.)

It's not a problem per se, except that my webserver logs have hundreds of hits like this:

prime.gushi.org 71.30.188.182 - - [06/Jun/2009:15:02:08 -0400] "GET /777.gif HTTP/1.1" 404 324 "-" "Mozilla/4.0 
  (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; GTB5)"
prime.gushi.org 98.175.208.182 - - [06/Jun/2009:15:02:35 -0400] "GET /777.gif HTTP/1.1" 404 324 "-" "Mozilla/4.0 
  (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
prime.gushi.org 71.242.102.171 - - [06/Jun/2009:15:02:41 -0400] "GET /777.gif HTTP/1.1" 404 324 "-" "Mozilla/4.0 
  (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

Know what that is? It's a virus. One that specifically tries to download an exe-disguised-as-a-gif from prime.gushi.org.
That file was never actually available for download, but (I guess because of my security activities) I was targeted.

In fact, http://prime.gushi.org has always just returned a "forbidden" page.

So here's how it will work.

1) This site now rotates its logs daily. 2) A tool goes through "yesterdays" logs and parses them (this bit is already written). 3) It all gets stuck in a database. The time-field (the bit in the []'s) gets rewritten as a unixtime to handle the annoyance of timezone conversion (since unixtimes are all utc). 4) A tool then goes through the database, looks at all the "seen today" hits, and runs abuseEmail on the ips, and complains in the right direction.
5) This also gets logged in a database, so that I can see every report I've submitted to a given abuse contact, and track historicals.

I vaugely wonder if these infected machines would accept cookies so I could track them that way. The user-agents are not unique enough in some cases.

The problem is with #4 right now: AbuseEmail seems more than a little broken. It looks for SOA records in places it should be trying WHOIS, and it seems rather uneducated about modern whois/rwhois servers. I've contacted the author. It hasn't been maintained since 2001. I might have to fork it. It might also work better as a module. I also haven't structured the DB yet, but that's trivial.

If any of you coders out there want to try and help out with this, I'm more than happy to share credit. I've suggested to my job they might be interested in this data, but there doesn't seem to be much excitement. It's one (much older) worm, and not nearly as prevalent as some of the others out there. Still, with the right amount of work, I could singlehandedly wipe a virus off the planet.

I've already considered the fact: these computers are going to download a file from me, and run it silently with administrator privileges. I could give them a virus-cleaning, or even a courtesy-reformat, but that's defintiely the wrong thing to do.

gushi: (Default)

Okay,

So after fixing my little mail logging issue I remembered that I had logwatch set up on my cobalt raq3.

Logwatch is cool. It emails you everything in the logfiles, you define great regular expressions as to what's harmless noise, and keep going till it's only the critical stuff that you get.

I just got a mail FULL of the following:

client 123.17.150.226 query (cache) 'mail.peregrinehw.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ALT1.ASPMX.L.GOOGLE.com/A/IN' denied: 1 Time(s)
client 123.18.118.42 query (cache) 'ALT2.ASPMX.L.GOOGLE.com/A/IN' denied: 1 Time(s)
client 123.18.118.42 query (cache) 'ASPMX.L.GOOGLE.com/A/IN' denied: 1 Time(s)     
client 123.18.118.42 query (cache) 'ASPMX2.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ASPMX3.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ASPMX4.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.18.118.42 query (cache) 'ASPMX5.GOOGLEMAIL.com/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ALT1.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.213.68 query (cache) 'ALT2.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.213.68 query (cache) 'ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)     
client 123.19.213.68 query (cache) 'ASPMX2.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ASPMX3.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ASPMX4.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.213.68 query (cache) 'ASPMX5.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.59.189 query (cache) 'mail.peregrinehw.com/A/IN' denied: 1 Time(s)   
client 123.19.99.134 query (cache) 'ALT1.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.99.134 query (cache) 'ALT2.ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)
client 123.19.99.134 query (cache) 'ASPMX.L.GOOGLE.COM/A/IN' denied: 1 Time(s)     
client 123.19.99.134 query (cache) 'ASPMX2.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.99.134 query (cache) 'ASPMX3.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.99.134 query (cache) 'ASPMX4.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  
client 123.19.99.134 query (cache) 'ASPMX5.GOOGLEMAIL.COM/A/IN' denied: 1 Time(s)  

So after I dig around for a bit (no pun intended), I realize.

What I'm looking at is a whole bunch of terribly broken DNS implementations. DNS implementations that bypass a host's DNS entry, and directly query ME instead of looking something up directly.

All the domains above are A records (address records) that are pointed to by MX (mail exchanger) records. I host sites that use those MXes, but I don't host (obviously) googlemail.com.

Okay, so I know why this is happening. It's mostly harmless.

My options:

1) Tune logwatch so I don't get these.

2) Tune BIND so it doesn't log these hits.

3) Use this information to feed a real-time blacklist -- it's fairly easy to write the parser but from the looks of it, most of these IPs are already on RBL's I use (spamhaus PBL, CBL).

4) Find a way (as recursive as this sounds) to block queries to my DNS server, based on this blacklist. I don't think BIND supports such a feature.

-Dan

gushi: (Nevar Button)

So apparently there was a real nasty worm out and about called Bagle. We all remember it, right?

So I was noticing in clearing out my error logs that I have a ton of hits to prime.gushi.org/777.gif (it's a 404 and to the best of my knowledge always has been).

So here's where it gets scary: I go to the reputable trend micro site seen here, looking for info on 777.gif:

http://www.trendmicro.com/vinfo/virusencyclo/default5.asp?VName=WORM_BAGLE.FN&VSect=T

And then I see it, slightly obscured...

This worm downloads possibly malicious files from the following URLs:

[...]

http://pr{BLOCKED}ushi.org/777.gif

Yup. Apparently prime's listed as one of the distribution sites for one of the worst virii out there.

Now, near as I can tell, that file's NEVER been in that webspace. In fact http://prime.gushi.org returns a 403 and points at a null directory.

This is not the first time I've been targeted like this, nor will it be the last. But Hrmmmmmm...How can I use this?

Well, I now have the means to VERY EASILY compile a list of every infected user out there and tie it into an email script.

In fact, I just realized, I could write up a very quick virus removal script, run it through bat2exe, and drop it right in place there -- except people who have done similar things have already been sued in this ridiculous world of ours.

I just emailed one antivirus vendor (and will email others) with the following:


Hello,

I am the owner of prime.gushi.org -- I recently discovered that I am rather popular with the bagle worm (I am listed 
as a download site for the malware) -- a file called 777.gif.  As far as I can tell, I have never hosted this file 
(that's my hostname, but leads to a "null" site).

I would like to do a bit more research to clean this up, as there's a file commonly distributed with movabletype 
that's also titled 777.gif.

I'd like to know if you could give me the file's info (the size, md5, sha1, strings output, etc).

I suppose I can email all the ISP's out there and have them fix their users -- or for that matter report those users to the various blacklists. Or I can just mod_rewrite this crap out of my logs.
gushi: (Default)
Guys,

phpBB 2.0.16 is out. Mass upgrades will be ensuing tonight. If you don't want this, upgrade on your own.

-Dan
gushi: (Default)
Some people ask what I do about system security on prime. I'm interested in sharing.

I've seen a lot of posts that say "don't give out ssh access". I think that's bullshit. Anyone who wants to can upload a CGI/PHP script that will allow them the equivalent of shell access almost instantly. Given, there are a class of users on the system who can do NOTHING but email, and they have no SSH/CGI/FTP access. Similarly, setting someone's shell to /bin/date (which will allow ftp, but not ftp won't stop them from uploading a script.

Security is a layered thing. I certainly don't know everything about it, and I don't believe anyone can. I know what I need to, and always try to learn.

I run Webmin. I run it behind SSL, and I run it on a non-standard port. In the event of a compromise, lockout, or fat-fingered root password, webmin is a convenient back door. Additionally, it's proven an invaluable tool for certain things, like MySQL. I exchange about one email a month with the author about possible improvements.

I run aide. Aide basically takes a checksum of important binaries on your system (in my case, anything in *bin (/bin, /sbin, /usr/bin, /usr/sbin, /usr/local/sbin, /usr/local/bin), and checks everything nightly. The checksum database resides on (get this), a write protected floppy sitting in the floppy drive. Good luck hacking that.

I have no qualms about running webmin, although there have been holes discovered in the past, because I run it someplace different from usual. How do I know people won't find it on a portscan? Simple. My open ports list is like a minefield. If you connect to any of 60 commonly-exploited ports, prime will defend itself and firewall itself against you. Permanently. You won't be able to connect to it at all. The ports list is scattered enough that it's hard to hit by accident.

I have a logfile parser that runs once an hour, that goes through all my logfiles and emails me if it finds anything unusual or out of the blue (failed logins, possible attacks, etc).

Additionally, there's also a system in place that keeps track of when people FIRST log in, as well as when they log in from an unusual suffix, cross-checked against a list of country codes. (i.e. if Joe logs in from Venezuela).

I run MRTG, which normally is used to graph traffic, but I use it to graph things like system load, the number of logged in (and unique) users, and the number of active processes.

This is all stuff to protect the server. Part 2 will be the stuff I do to protect the user.
gushi: (Default)
So I was tapped a few minutes ago to head into the city to replace a piece of equipment at one of the NYC telecom hotels. Writing this on the train now, will post later I guess since this laptop can't talk to the phone.

Well, that's a misnomer actually -- the phone itself can make LJ posts, and the phone itself has an SSH client. And since both speak IR, I could beam this whole post over to the phone and then upload it.

But that's overkill, ne?

Anyway, I've looked into the SSH key thing I mentioned before. And I've decided it's absolutely stupid.

Basically, publishing the keys in NORMAL dns isn't enough. You have to be using the DNSSEC secured DNS extensions. What this means is every time I changed the gushi.org zone, I would have to generate a signature (nothing inherently hard about that, everything's scripted anyway). I would have to publish that signature in my DNS. And then, what's worse, is that everyone else, has to be using a DNS client that UNDERSTANDS the security enhancements, and passes on the "yes, this is secure" data to the end user. Worse still, each of those DNS servers has to accept, and TRUST my DNS public key. So if you're a user on a dynamic comcast IP (and presumably using the comcast DNS servers), Comcast would have to accept my key and include it into their system. AND, they think you should be running some encrypted protocol between yourself and comcast's DNS servers, like IPSEC.

Now, why the hell I can't just take my GeoTrust certificate that says "Yes, we've certified that this person runs gushi.org", and stuff THAT into my zonefile (this is how Sendmail, Apache, ProFTPd, Webmin, Usermin ALL work)...and then comcast would say "we believe in GeoTrust, and they say to trust you, therefore everything seems to be in order".

Of course, the system outlined above seems to be a replacement for caching the keys locally, which is even more stupid. All I'd like to see is something like this.

shell#ssh danm@prime.gushi.org
checking dns for prime.gushi.org...
key found in DNS...

the ssh key coming from prime.gushi.org, id aa.aa.aa.aa.aa.aa.aa is not known,
HOWEVER, it *does* match the key found in DNS, as retrieved from ns2.gushi.org

Would you like to continue connecting? (y/n)



From there, it would be business as usual. The key caches would still be used, instead of relying purely on DNS. SSH would still check the key cache, and would still bitch heavily if the connecting public key didn't match the one in the cache. Period. This would only serve as a method of distributing the key that makes more sense than "just type yes".

I suppose, optionally, that this kind of thing could be checked *every time*...but the cache would still be preferred.

Now, it's assumed that someone with enough brains to spoof a man-in-the-middle attack would also be able to spoof the DNS query that grabs my key (that's why the guys were talking about DNSSEC).

The other thing I'd love to see, as an optional "comment" field in the key, is a how-to-verify field.

The ssh key aa.aa.aa.aa.aa.aa.aa.aa is not known, however, the creator of this key has stated that this key may be verified in any or all of the following way(s):

NOTE: You should personally check as many of the following as you feel are necessary to verify that this id is authentic.

"see url: http://www.gushi.org/keyinfo.txt"
"if in doubt, call Gushi at 1-866-LI-GUSHI, dial option 12"
"Check http://www.livejournal.com/userinfo.bml?user=gushisystems"
"Key fingerprint should be sent out in the footer of signup e-mails"
"fingerprint is printed on the back of Gushi's business card"

Now, of course, those methods are easily compromisable too...but security is a layered thing, but it's assumed that if someone is running a man-in-the-middle attack against prime.gushi.org, that they won't be able to gafutz with ALL those methods.

Doing the first bit, the DNS lookup, could be tweaked with only patching to the SSH code...right now, the spec states:

2.4 Authentication

   A public key verified using this method MUST NOT be trusted if the
   SSHFP resource record (RR) used for verification was not
   authenticated by a trusted SIG RR.

   Clients that do validate the DNSSEC signatures themselves SHOULD use
   standard DNSSEC validation procedures.

   Clients that do not validate the DNSSEC signatures themselves MUST
   use a secure transport, e.g. TSIG [9], SIG(0) [10] or IPsec [8],
   between themselves and the entity performing the signature
   validation.


Of course, the spec (http://www.snailbook.com/docs/dns-fingerprints.txt) also states "Expires March 5, 2004" so I'm not sure how real this is. I think I could make a serious motion toward getting this made real.

The sourceforge SSH servers got whacked a while ago, and a lot of people wound up revealing their sourceforge ssh passwords to the thing. The hackers were then able to log into the sourceforge shell accounts, and use the STORED KEYS that people had there to jump to other places. People actually VERIFYING KEYS would help this a lot.

As for the second part, the key "extensions", those would probably lead to widespread breakage, and we'd probably have to wait for the widespread adoption of ssh3 (which I'm not even sure is a draft yet).
gushi: (Default)
But I'm pleased to announce that imap and pop3 now support SSL and IPv6 Natively. Stunnel is gone.

Please report any issues ASAP, to the usual channels.

I've also made SpamAssassin's spamd module support SSL, for the hell of it.

Other geekly thoughts follow this pattern.

August 2017

S M T W T F S
  12345
678 9101112
13141516171819
20212223242526
27 28293031  

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 26th, 2017 07:16 am
Powered by Dreamwidth Studios