gushi: (Default)

It's the holiday time, and the IT-driven life I lead has been a little slow, as nobody wants to make sweeping changes over the holidays. Instead, I'm once again playing with my personal code, and my vendetta against the people who abuse servers.

In my last post, I detailed how bad DNS queries can indicate that someone is querying nonstandard sources on your DNS server. For example, someone querying your servers for the A records of the google mail servers.

What I've discovered since then is that with a tiny bit of perl, I can run through a day's logs, and feed the data into a hash, to de-duplicate them. Then, with the magic of nsupdate, I can feed them into an RBL that my mail servers will query. I don't even need any intermediate database.

So What's an RBL?

I'm sure those of you who use SpamAssassin or whatnot may know what an RBL is: it's a Realtime BlockList. Your mail servers do a special dns lookup. For example, if the ip address 1.2.3.4 is connecting to you, your mail servers may query the rbl at "zen.spamhaus.org", by first reversing the ip address, and then appending it to the blacklist name.
So 1.2.3.4 becomes 4.3.2.1.zen.spamhaus.org. If a lookup of that returns an ip, it's listed. Typically, the ip returned is in the 127.0.0.x range. On some blocklists, certain return codes have certain meanings.

Generate your key.

Most people who do dynamic updates with BIND, use a security method called TSIG (Transaction SIGnatures). The key in these cases is a "shared secret", and needs to be chunked into named.conf. This is the "Old Way" of doing things.

In my instance, I am using something most people don't get, called sig(0). Instead of having to put my keys in my named.conf file, I simply list them right in my zone. Instead of being the standard HMAC-MD5 keys that one sees using TSIG queries, I can simply tell any given party "generate a key, send me the public component" and never worry about the secret key crossing the wire. (Yes, to be sure, I should tell them to pgp sign it to make sure it's not modified in transit). The real beauty of this is that with a properly-crafted update-policy, I can set things up so that future keys can be added with nsupdate, and I never have to touch named.conf to add "feeders" again.

The command I ran to generate my keys was:

dnssec-keygen -a RSASHA1 -b 512 -k -n HOST rbl.gushi.org

That will give me two files: a .private file that's a bunch of Field: Value statements, and a single .key file which contains my key, in a resource record. While the format of these filenames may look very similar to those used for DNSSEC, the records being generated are of the "KEY" type, whereas DNSSEC uses "DNSKEY" records. I copied these to my home directory.

Create the zonefile

After generating the key, I created a basic zone:

$TTL 360        ; 6 minutes
rbl.gushi.org.           IN SOA  prime.gushi.org. root.gushi.org. (
                                2009123678 ; serial
                                7200       ; refresh (2 hours)
                                7200       ; retry   (2 hours)
                                604800     ; expire (1 week)
                                360        ; minimum (6 minutes)
                                )
rbl.gushi.org.          NS      prime.gushi.org.

And then added my key statement:

rbl.gushi.org.          KEY     512 3 5 (
                                AwEAAbt55viC4mTSNbvlZlEM9QN/aDRAcBiItmmGylNV
                                GDw9eBLF71TBtzF/zVLUExsptCj3ez/wYstkQjfWGfjO
                                zl0=
                                ) ; key id = 65002

Note that the contents of the .key file were literally:rbl.gushi.org. IN KEY 512 3 5 AwEAAbt55viC4mTSNbvlZlEM9QN/aDRAcBiItmmGylNVGDw9eBLF71TB tzF/zVLUExsptCj3ez/wYstkQjfWGfjOzl0= on a single line, but the above pretty-wrapped line was generated after the zone got rewritten by named.

I made sure to put the zonefile in a directory where named could write to (it would be overwriting the zonefile, as well as creating a ".jnl" file in the same directory).

(I also added an NS record in my main zonefile, pointing to prime.gushi.org exclusively for this zone), as well as specifying an update-policy in my zone definition in named.conf:

zone "rbl.gushi.org" {
    type master;
    file "d/rbl.gushi.org.hosts";
    update-policy { grant rbl.gushi.org. subdomain rbl.gushi.org A TXT; };
};

That update policy basically tells named that the key labeled rbl.gushi.org can update any subdomain of rbl.gushi.org, but only the A or TXT records. While this policy is pretty granular, named currently lacks the ability to say things like "this key can only add but not delete TXT records" or "this key can update records, but cannot change the number of records (i.e. it may not add two A records).

I then issued an "rndc reconfig" to tell named to reload the config files (in the "old days" this would have been done with a SIGHUP or by stopping and restarting the process. Then I checked the logs to be sure the reconfig had run, and that the new zone had been loaded (rndc doesn't tell you these things).

Testing dynamic updates

After that, I ran the following:

   prime# nsupdate -k /home/danm/Krbl.gushi.org.+005+65002.private
   > update add test.rbl.gushi.org. 3600 A 127.0.0.1
   > send
   > update add test.rbl.gushi.org 3600 TXT "this is a test"
   > send
   > quit

Then, a dig to test it:

   prime# dig @prime test.rbl.gushi.org ANY
   ; <<>> DiG 9.6.1-P1 <<>> @prime test.rbl.gushi.org ANY
   ; (1 server found)
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50427
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 1

   ;; QUESTION SECTION:
   ;test.rbl.gushi.org.            IN      ANY

   ;; ANSWER SECTION:
   test.rbl.gushi.org.     3600    IN      TXT     "this is a test"
   test.rbl.gushi.org.     3600    IN      A       127.0.0.1

   ;; AUTHORITY SECTION:
   rbl.gushi.org.          3600    IN      NS      prime.gushi.org.

   ;; ADDITIONAL SECTION:
   prime.gushi.org.        360     IN      A       72.9.101.130

   ;; Query time: 0 msec
   ;; SERVER: 72.9.101.130#53(72.9.101.130)
   ;; WHEN: Sun Dec 27 01:56:55 2009
   ;; MSG SIZE  rcvd: 115

From there, if I were to do an rndc freeze rbl.gushi.org, I would see it show up in the zonefile. Until that point, it would live in a journal (.jnl) file maintained by named.

At this point, I had a zonefile that I could add things to, and remove them from, but getting the data out of my logs was still a problem.

Parsing my logs

Enter perl. While programmers may love python, and web programmers may love ruby, to the sysadmin, perl is still the "swiss army chainsaw" we reach for in such a situation.

A quick well-caffienated night (christmas night) of programming yields this script, which works for me (and in fact, will work from multiple systems, even across the internet). I just run it against the requisite logfiles, and I'm set.

After running it, I issue an "rndc freeze rbl.gushi.org" to tell named to save the data back to the master zonefile and look at the zonefile:

$ORIGIN 109.rbl.gushi.org.
$TTL 3600       ; 1 hour
0.88.110                A       127.0.0.2
                        TXT     "Last seen Dec 26 16:11:19 polling for ASPMX4.GOOGLEMAIL.COM/A/IN"
144.28.121              A       127.0.0.2
                        TXT     "Last seen Dec 26 17:50:46 polling for ALT1.ASPMX.L.GOOGLE.COM/A/IN"
96.136.184              A       127.0.0.2
(...)

And so on.

Note that the format of those files is a little annoying: named currently lacks a config knob to tell it to dump in any format but that one, although I've submitted a suggestion request. I personally feel the above would be more readable if every entry were fully-qualified, without the $ORIGIN statements.

I'm also able to do digs against the above. (And so can you: dig 109.110.88.0.rbl.gushi.org ANY: try it!) Once I'm satisfied, and have made any manual tweaks, I can do an rndc thaw rbl.gushi.org and updates are allowed again.

Another feature subtly absent from BIND that I've put in for, is that there's no way to tell it "just save the data to the master, but don't stop accepting updates". Time will tell there.

Configuring Sendmail

Once I'm happy that the data is going to an easily-pollable source, I simply need to tell my mailer to read it. While I won't make any assumptions about the installed-based of readers here, I run FreeBSD, which comes with sendmail by default, and that's what I use.

Sendmail contains built-in support for this type of RBL. In addition to the other ones I poll, I simply added the following file to my sendmail.mc:

FEATURE(enhdnsbl',rbl.gushi.org',Message from $&{client_addr} rejected - GushiSystems Blocklist')`

I chase that down with a quick make install-cf && make restart and I'm ready to go. At any point, I can grep my maillogs for "GushiSystems Blocklist".

Sadly, what I'm finding (although as is typical, I didn't find it when I discovered the problem), is that the CBL detects most (but not all) of these as the cutwail SpamBOT, a rather massive command-and-control botnet.

Does that make this effort wasted, though?

Not at all. After all, now that the updating code is written, it's trivially easy to take the same code and do other things with it.

For example, it's still relevant in a DNS context to show how I could turn this on my maillogs, where both sendmail and spamassassin log, and do neat things with a few lines of perl like "blacklist any ip that sends a spam that scores over 20 points AND sends to three distinct domains in an hour. The above is literally five lines of code to correlate the requisite log entries, another ten perhaps to act on them.

Right now, the perl code is written to builds its big hash of information in a single pass (when my logs are rotated), and then write to the zonefile all at once. A trivial enhancement would be to have it listen on a named pipe, or socket, and periodically flush its cache, so it didn't grow forever over time.

I could at the same time, use the code to feed a list of people trying to send Guestbook Spam, or people posting to dead/abandoned phpBB boards. Again, most webservers don't have the ability to poll this sort of a database, but there's no reason it can't be easily added.

A minor footnote

I'm not sure where it stands in the POSIX standards or whatnot, but I've discovered that the date format used by FreeBSD's syslogd both does not log the year, and is non-configurable. Considering as I write this we're four days away from the end of the year, I may have some tweaky date-logic to write (keeping in mind that this script currently runs on the previous day's logs), so assuming the current year isn't the best answer.

On that note, I'd like to wish everyone here a safe new year. Here's looking forward to more cool stuff in 2010.

gushi: (Default)

Mysql, as everyone knows, is a database server with two different storage engines.

One engine, MyISAM, is the default, and is reasonably fast and well-optimized.

The other engine, InnoDB, is a more "professional grade" engine, supporting things like transactions, row-level locking, and the rest. This engine is supported by a commercial entiry, InnoDB, in the same way that PHP is supported by Zend: in that sort of "you can do more useful things if you're willing to pay for it" sort of way. The other big, big, annoyance, is that out-of-the-box, InnoDB keeps all data, for all tables in all databases, in one huge fucking blob, that can automatically grow at a rate you specify, but that never shrinks. Not even if you drop every database. Right now, my little innoDB blob is five gigs, and that's for a remarkably small number of applications using it.

And there we have the rub. There are a few applications that use the functions of InnoDB, and that default their table types to be InnoDB tables. Among these are MediaWiki, and gallery2.

Looking at that blob, there's also no easy way to tell what databases are using it. MySQL's "show table status" (which could tell you the type) doesn't work on a global base, you have to select a database first (of which I have many.

The solution to this is an option that should have been on-by-default the whole time, the innodb_file_per_table option. This tells mysql to treat innodb blobs much like it would treat MyISAM tables: they go into the database-specific directories, so you can easily (with tools like du) tell which databases are bloating (because, for example, some user installed phpBB and then forgot about it).

After turning that option on, there's still a problem: it doesn't cause the SQL server to migrate your data for you.
It only affects newly created databases. It's easy enough to dump-then-restore each database during a maintenance window, but wouldn't it be nice if there was some way to spot the databases which needed it? (Remember, InnoDB is not the default).

As it happens, the following short shell script can do this for you:

    
#!/bin/sh
# InnoDB finder $Id: findinnodb.sh,v 1.1.1.1 2009/12/27 12:48:33 danm Exp $
# Dan Mahoney, danm@prime.gushi.org
# ISC License applies

cd /var/db/mysql
# Change to your DB dir
for i in `find . -regex '.*.frm' | cut -d '.' -f 1-2`
  do
  #  echo "testing $i"
  #  file $i.MYI
  dbname=`echo $i | cut -d "/" -f 2`
  tablename=`echo $i |cut -d "/" -f 3`
  if [ -e "$i.MYI" ]
  then
    echo "$i is MyISAM"
  else
        if [ -e "$i.ibd" ]
        then
          echo "$i is InnoDB, but self-contained"
        else
          engtype=`mysql -E $dbname -e "show table status like '$tablename'"| grep -i engine | cut -d ":" -f 2`
          # add a --password=xxx option above if you don't have one in .my.cnf or whatnot
          echo "$i is$engtype (from MySQL DB)"
        fi
  fi
done

Note that it goes by file-system-wise clues, instead of eating time connecting to mysql. It only connects to mysql if it can't figure it out. For example, without asking mysql, a memory-only table looks identical to an InnoDB.

I was rather surprised to find that nobody on the MySQL pages had suggested this, after all, "find the piggy" is a big part of detecting abuse and resource-problems

gushi: (Default)

Note: since I started writing this entry, in that half hour or so, the problem I was writing about went away, although it hadn't been for at least the past few days. This is one example of a problem I see fairly regularly, a site that answers on www.domain.com, but not on domain.com. These are the details, but they're far from an isolated case.

Back in the day, when IMDB first was started, they had a partnership arrangement with a little online-video company called reel.com.

Since that point, reel.com has been bought by hollywood video and has discontinued their online business, and their one storefront store still stands in berkeley.

However, this entry isn't about this, it's about countless websites that do a stupid-but-annoying thing.

reel.com has an IP address. www.reel.com has an IP address. They happen to be the same, but they don't have to be.

If you go in a browser to http://www.reel.com, you get to a "thanks for your patronage" page that points traffic at the Hollywood Video page. If you go to just plain http://reel.com (no "www"), you get...nothing. You get a "Virtual Directory Denied".

Finally, if you go to the ip address of reel.com, which is http://72.5.61.11, you get the main page.

Why?

Well, for starters, lets make it real clear. The webserver that serves this site is running Windows. That "Virtual Directory" error is an IIS thing.

Sending along a hostname isn't part of the original HTTP specification, it's part of a little add-on known as HTTP1.1, when it was realized the proliferation of the web would quickly exhaust the number of ip addresses out there if there was only a 1:1 mapping.

IIS, the piece-of-garbage webserver built into windows, has a site config window that looks like this.

Note that there's an option to specify ONE name for a site, and only one.

So, if you connect to an ip, but don't send a hostname (or send a hostname containing only an IP address), you get a site. If you send EXACTLY what's in that box above, you get a site.

In order to add multiple host headers, to the same site, even if they are really just aliases of the same site, requires work, it requires clicking that "Advanced" button, and putting the hosts in, OR it requires having your site be the ONLY one on that IP address, and specifying no header.

So, let's make it clear, what could the administrators do better?

1) They could configure a different site in IIS for reel.com, and configure it to be a redirect to www.reel.com. They can even preserve the path so a request for reel.com/images/logo.gif just gets handled the right way, instead of redirecting everyone to the "front door".

2) They could remove the need for host headers entirely, since chances are, the reel.com site is the only one running on that ip.

3) Radical option: they could simply remove the A record for reel.com. What this actually means is that the user will get an error that the domain doesn't exist. You can't tell me this error is any better or worse than the "Virtual Directory Listing Denied" error.

4) They could use a real OS/Webserver. Seriously, you're working for a publicly traded company like Hollywood video. I'm sure they're paying you good money to click the little boxes to turn on IIS. Perhaps they could instead pay someone who knows what they're doing? And if the OS and webserver require a degree of clue to get running instead of "click I Agree, click next, click next, click Finish", then maybe that's not a bad thing.

Don't get me wrong, windows is good for a lot of things. I run it at home because I don't feel like fighting my hardware just to run a GUI, and because I like working video, sound, and USB drivers. I like my stress-relief games to work without having to deal with the stupidity of an emulator.
I like being able to buy anything at the store and know I can plug it in and have it work without installing a bleeding-edge kernel.

But for a server? None of those apply. People all the time cite "But I need something supported" as a reason to use Windows, but nobody pays the per-incident Microsoft support fees, everyone just calls someone with more clue, or Googles. You can afford better than that. Unix is darwinistic: people who figure out how to use it are the ones that do. The clueful people advance. You want one of those, Hollywood.

I mean, potentially, this is free advertising for them. Don't you want money? Can you imagine if say, netflix.com didn't work, but www.netflix.com did? Think that would be a problem?

gushi: (Default)

LJ Preface

I recently wrestled with something, learned quite a lot, and came up with a document that I'm really rather proud of, that shares knowledge that's not all out there in one place anywhere else. Along the way I've written some software that I'm releasing, that makes all of what I've learned a lot easier, and may help make the world a little more secure. I'd like to share it here.

This is going to be a technical post. For that I apologize. The target of this post is anyone who has a GPG key that they'd like to expand to a greater audience, and who controls DNS for any of the email domains they publish. Anyone that I host DNS or mail for is also welcome to do this, if you use PGP, as part of the goal of writing this is to encourage adoption and use of these methods

This will be long and technical )

gushi: (Default)

I'm not sure why, exactly, but sshit on oldprime never seemed to work right.

Oddly, I would get ssh authorization warnings on the CLIENT end, but things would never make it to syslog.

I finally solved it by telling opensshd to log at loglevel VERBOSE as opposed to the default. Apparently, logging at level INFO (the default) is not enough to tell you when someone is REPEATEDLY FAILING to log into your system.

And yet, on the CLIENT end, it was enough to tell you what was failing:

%ssh testcase@oldprime.gushi.org
Password:
pam_unix: pam_sm_authenticate: UNIX authentication refused

You get that? I'm getting CLIENT logs telling me what subsystem is blocking me, but not on the server.

I bumped logging up to "verbose" and all seems to be back to normal, but still, this is really stupid. And it's not the first we've seen of this.

I'm just wondering when openssh will include the auto-ignore/auto-block functionality on its own. If all clients did this, it would make bruteforce scanning pretty ineffective.

Of course, that would require everyone to upgrade. And we know that's not happening.

gushi: (Default)

It's amusing to me. There are several PGP key servers out there, that are web enabled. None of them have the ability to display the photo data in a key.

As I use GPG mainly in shell applications (cause, you know, it's a textmode app and doesn't HAVE an xwindows version), I found this mildly amusing.

I tried setting my GPG image viewer to "AsciiView" (which has been known to work for other things in the past...):

But for identifying the owner of a public key, it's...less than useful.

Now, if there were at least ONE pgp keyserver that knew how to handle that image data and show it to me in a webpage, it might be helpful to me, since most pgp keyservers sync data with most others.

What I believe the alternative is going to need to do, is write up a new way of handling this, a new standard, for shell providers, whereby shell-apps can pass browser-openable data. I've started writing it, and it could be a lot of fun.

gushi: (Default)

I recently saw a "Geek Code Block" in a friend's LiveJournal profile, along with a link to "decode" it.

In the past, I'd wanted to be a geek, but always felt like a latecomer to the game. Now, I know I'm a geek. I embrace it fully. I live in Silicon Valley, and work for a nonprofit that makes open-source software. We're doing more for DNSSEC than any other company I can think of. I still have learning to do (still haven't learned C, still feel more comfortable with nano than vi), but I'm feeling like I'm getting there.

This entry, however, is about my problems with the Code.

Outdated Items

The latest revision of the geek code is at this time well over ten years old. It references things that were largely relevant at the time, but now one can see those age a bit.

Specifically, references are made to "Star Trek", the next generation versus TOS. In 1996, Voyager had started but not yet finished. Enterprise hadn't been thought of, nor had the recent movies.

No reference is made to BSG, although the "old" series was somewhat relevant historically at the time. Same for revived series like Doctor Who, or other "cult" shows like The Twilight Zone, The Outer Limits, or other such things.

There's a whole category about Doom. Just doom. If one were to analog this in a more modern sense, it might be half-life. However, there's also no mention of MMORPGS.

Dilbert is referred to as "Simply the geekiest comic strip in existence." Sorry, not anymore.

References are made to Kibology, Usenet Oracle, VMS and OS2. The only programming languages mentioned are perl and (implied) c.

And then, there's the categories where I find myself excelling, that don't even touch this.

Things Geeks should know

For one, there's networking. Routers. TCP/IP. DNS. Low-level protocols. No mention at all.

No mention is made of geek games that span centuries, like Chess and Go.

No mention is made of socialization: how often you go out, or for that matter, number of R/L friends to Online, which I believe was relevant even in 1996.

There's no category for the classic geek movies, the ones that embrace our differences and what we as geeks relate to, ranging from Daryl to Sneakers to Wargames to Revenge of the Nerds. No mention is made of Rocky Horror,

While above I note there's no mention of MMORPGS's (thus making thus outdating the code), there's no mention of MUDDING, MUCKING, MOOING, or any other similar service (which is well within the timeframe of the FAQ). All of these things have modern equivalents. Your MUCK has evolved (ugh) into Second Life, your MUSH into Eve Online. Your classic MUD is now called Wow. And Everybody's doing it.

The Decoder

There's a geek code decoder here, but it wasn't written by the person who made the code, so I've spotted at least one shortcoming in it.

My geek code, decoded, is here, but I've found at least one glitch in it. My geek code lists "!L+++", which the decoder states is: I don't even know what GNU/Linux is! But if I did participate in this category, I would characterize myself as follows: "I use GNU/Linux exclusively on my system. I monitor comp.os.linux.* and even answer questions sometimes."

What I actually mean by that, and I think makes sense considering the connotations of the "!" is: I'm good at linux. I make money at it, but god do I hate it and all it stands for.

It doesn't fit to mention anywhere else, but there's a few other bits that should be in the code. A last-updated date, in either seconds-since-epoch or YYYYMMDD format, as well as a field for "birthyear:precision", for example, I could put BY:1975+5, indicating I was born in 1975, plus or minus five years (for people who didn't want to give an exact age). The problem with the code itself is that it has a static age field, which is subject to change.

Conclusion

Yes, I'm a geek. In more ways than most people know. Things that used to be geeky but are getting more mainstream, I no longer do. I still read my mail in (al)pine. Rather than trying to figure out some cryptic code to determine who I am, the nerd version of dogs sniffing butts, why don't you get to know me?

However, if you want to know where I stand here?

-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCM/IT dpu s+:++ a C++++$ UBL++++$ 
P+++$ !L+++ E---- W+++$ !N Y++  PGP++ 
t++ !5 X R tv b+ !DI D--- G-- e h* r++ z+++
------END GEEK CODE BLOCK------

...and rising.

gushi: (Default)

As an experiment, I stripped all my (tag:) tags out of the previous entry, and posted it here under a cut. For anyone really interested, feel free to re-read and see if things still make sense. (I noted a few specific examples in brackets where things made a little less sense, or where I left parens in because they would have been left in by a parser.)

It's under here! )
gushi: (Default)

There is a spammer that has been annoying me. They're doing things halfway legit, so they bypass a lot of filters. They're advertising a site called nextjob.us, mostly telling me about candidates who I'd want to hire who need H1B visas or green cards.

I've complained via SpamCop, and also directly to their ISP (Cogent).

I did a google search for them recently, and discovered that not only are they being blocked by google, but that they're asking on google's forums for help!

I quickly typed out my own reply, which has since been deleted )

And they emailed me back, again asking for help, and seeming somewhat apologetic. )

While one might think I'd don my BOFH hat to handle this, I'm somewhat touched, because I know the answer to this.

My response was long, and almost didn't get to them, because they set their "Reply-To" header to "no-reply@nextjob.us". This alone indicates a serious case of "you don't know how this works".

My reply is below the cut )

gushi: (Default)

So, I just got this amusing email...

From slackerng@gmail.com Thu Jul  2 02:08:05 2009
Date: Thu, 2 Jul 2009 01:07:54 -0500
From: Cody Grunenwald <slackerng@gmail.com>
To: "root@gushi.org" <root@gushi.org>
Subject: I really need help

I saw that you had a crash file that you can crash wc3 users only by whispering
them. Now im a noob with technology and stuff so i was wondering if you could
get on battle.net and go to Channel CLAN STN and crash anybody in that channel
with praetor in their name. Long story short they hacked themselves into OP and
were a new clan so we have no shamans or anything and hes holding our clan
hostage. please help us.

Note that they emailed root@gushi.org. Now, there's only one place I use that. root@prime.gushi.org is common, but root@gushi.org was ONLY used, for a while, as the ServerAdmin for the gushi.org domain (as in, ONLY my personal domain). Thing is, it also shows up as the serveradmin for people who use www.gushi.org/~username aliases...and a quick google revealed the problem.

A user I had kicked off a while ago, who was using prime as his location for starcraft hacking tools (I know because I heard from Blizzard about it).

Remember fun LJ entries like this?

So, obviously what's happening is people are finding some webpage that links to this, getting a 403, and then EMAILING ME.

Gee, how ever could I find out who this is? Oh wait, look, I have my webserver logs!

%tail -1000000 access_log|grep -i celeron
gushi.org 75.72.94.81 - - [01/Jul/2009:20:48:09 -0400] "GET /~celeron/hacks/Exended1.4.zip HTTP/1.1" 403 345 "http://gaminkings.tripod.com/id8.html" 
"Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5"
gushi.org 97.83.163.193 - - [02/Jul/2009:01:56:50 -0400] "GET /~celeron/hacks/SCCRASH.zip HTTP/1.1" 403 342 "http://gaminkings.tripod.com/id8.html" 
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"
gushi.org 97.83.163.193 - - [02/Jul/2009:02:05:37 -0400] "GET /~celeron/hacks/SCCRASH.zip HTTP/1.1" 403 342 "http://gaminkings.tripod.com/id8.html" 
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"
%

So I checked out the tripod page (I especially love that the banners on the webpage are offering a DEGREE IN HACKING), and there they are.

Now, the question is, what to do with them?

  • Tell them "it's a game, there really are more important things"
  • Tell them "STFU NOOB!"
  • Tell them I'll hack THEM for disturbing me! (Mess with the best, die with the rest!)
  • Tell them I'll do it for 1000 gold, sent to the WOW account of anyone I don't particularly like?
  • Forward the tripod hacks page on to Blizzard?
  • Compain to tripod myself, as it's causing me negative traffic, perhaps threatening that if they don't take the page down, I will POPULATE those links?
  • Since they seem so willing to download and run files, send them a nicely wrapped boot sector rewriter?
  • Two words: mod_rewrite.
  • Send them a link to this entry.
  • Do nothing but blog about it, sigh, and shake my head sadly.

Yeah, probably that last one, but you never know. *sigh* *shakes head sadly*

gushi: (Default)

So I discovered today that I was getting a lot of spam mail that slid right through my filters...most of it by a company called Diversion.

I looked at the headers and found a few interesting things:

1) All the recipients had "real names", and the spam was directly addressed to them, as opposed to being bcc'd or sent to "undisclosed recipients".

From: Diversion Media <diversion_media@hearstmdinfo.net>
To: Mark Scribbins <marks@gushi.org> <-- like that
Subject: Get to Know Diversion.com for Physicians - at Your Fingertips

2) The links on the site, while going through a "Redirector" all matched, and was a sane domain, which corresponded with the link text, and which in turn was the same as the email domain. It wasn't a long subdomain, nor was it loaded with random letters or characters.

3) The text was relevant to the subject line, which in turn was relevant to the content, which was readable instead of the markovian crap I'd expect.

I looked at one of the articles...this one, and it's reasonably well-written and informative. Sure, a bit fluffy, but a decent read.

This didn't smell like spam to me.

I looked over their site, and found a "contact us" link. I called the number for their "advertising" department, and a person answered. Okay, too wierd!

The conversation went like this:

"Hey, how's it going. I seem to be on your mailing list several times, and I wanted to let you know that the whole domain goes to me, and I'm getting several distinct copies of these emails from you. Normally I'd report this stuff to spamcop or whatnot, but it seems you guys are legit. Like, if I were a doctor, I'd probably be interested in this stuff, it's well written and informative. So what I'm guessing probably happened is that you guys bought a bogus list, and I'm just calling to let you know you may want to go back to whomever sold it to you and take it up with them."

I gave them my domain name, and was told "yeah, unfortunately this isn't the first call like this I've gotten", and "thanks a lot, not everyone would have done what you did." (Again, not things I'd expect a spammer to say.)

Now, over time, I've gotten several spams that claim "PHYSICIAN LISTING!!!" or "50000 US MD LISTINGS!!!1!". And chances are, Diversions either bought such a list (how accurate could such a list be?), or someone who seemed more legit bought such a list and re-sold it to them. Welcome to the ponzi-driven internets :)

What this also indicates to me is that there are a number of services out there that "discover" domains that accept all domain-bound email. I suppose, historically speaking, I should look for the first emails sent to those services. (As I keep lots of email, and lots of logs, this isn't hard).

What it also means is that in my quest for better filters, I can now track everyone else who uses those lists, since the list-generators have managed to create a unique fingerprint for their lists. While I don't expect anyone to share with me where they bought it from or whatnot, I suppose if I were in a different field, I could offer to help legitimize these folks -- adding better verp detection, better feedback loop awareness, and the rest. And quite frankly, if I wind up blocking an otherwise legit site like this, because they bought a shitty list...oh well.

Somehow this reminds me of when I was parked in Home Depot, and there was someone running around, putting flyers on everyone's windshields, saying "Advertise in the Pennysaver, call this number!" Huh? If the Pennysaver is such an effective means of advertisting and communications, why do you need to be paper-spamming cars?

I mean, let's face it, marketing data is an asset, and I suspect, as Diversions is discovering right now, you get what you pay for. Or better still, let the buyer beware!

gushi: (Default)

Okay, everyone, have a quick look at a message I just exported EXACTLY from my mailbox. It's the raw message, with the raw headers, so it might not make sense to all of you.

It's right here. Note: when rendered in a BROWSER, the email looks like THIS

After that, I want you to have a look at how it looks in my mail client, click this preview:

I've got several MAJOR problems with this.

Note: it may help to refer to the entry in Wikipedia about MIME and how it works.

Techie Ranting about email formats )

gushi: (Default)

I just got this one tweaked a bit.

This is a recipe for handling a mailing list, by shuffling it off somewhere, but also, to add a "subject" tag (such things are the subject of holy wars on many mailing lists).

This example is for the spamassassin-users group. I do this within a specialized block because many lists were being shuffled to a generalized "otherlists" folder (and thus subject tags still made them relevant).

:0
# Match anything with this list ID
* List-Id:.*users.spamassassin.apache.org

{
  :0 fwh
  # Iff it doesn't have a subject tag, add one using "formail"
  # The f flag tells procmail to use it as a filter rather than as a final destination
  # w tells it to check the exit code (lets the filter fail gracefully)
  # h tells it to only feed the header to the formail program
  * !Subject:.*\[sa-list\]
  | formail -I"Subject: [sa-list] ${SUBJ_}"

  # From there, we deliver normally to the folder.
  :0:
  spamassassin-list

}
gushi: (Default)

Okay,

I'm at my dad's house. He has a new laptop, which he needs me to "set up". Normally this task is accomplished by opening the laptop, and because I didn't rush to his aid when his old computer (the last of several computers I've built for him) died, he got a new laptop from a friend.

[Unknown site tag]

Yick, Vista.

It wasn't so many years ago, that windows XP was the new, Ugly face of windows...and it's easy to see where Microsoft has still done a few things wrong.

For example, my dad's been a Dialup guy for a number of years. The Microsoft Office trial that comes with the system "Free for 60 Days, it swears!" won't work without a trial. Windows defender (which currently has definition version 1.0.0.0 -- HOW?) flashes an unhappy exclamation point at you, because it cannot update without an internet connection (although, I'm not sure where one would get spyware without an internet connection.) Norton complains unhappily as well. The HP "product showcase" that's so cleverly placed in it's "please click me" location on the desktop won't work. The pre-installed "Yahoo Search" that's stealing Taskbar real estate is functionally useless.

Vista Home Premium includes a game called Chess Titans, which is chess played on a 3d board. However, in the default layout, at the beginning of the game it shows a sweeping 360 degree view of the board, then goes to a top-down view from a 45 degree angle (i.e. white's eye view).

Of course, because it's a 3d chess program, all the taller pieces obscure all the smaller ones...and when viewed top-down (the way I've been playing computer chess since 1986), all the pawns are just round circles, instead of showing iconic, pawn-like representations. And the best part is, if you try to spin the board to give you a better view...it reverts after about 3 seconds.

There's a game called Inkball that has not only the most pitifully incomplete helpfile I've ever seen (apparently, in the game, your whole raison'd'etre is to draw lines to lead a ball into a hole -- but there's an unexplained bonus you get where you can drop "blocks" with no restriction, even to the point where you can completely block off your goal.

When I connected his external monitor, a nice dialog popped up saying "hey, do you want to mirror this, expand your desktop across both, or what".
But it's not the same behavior as the one in the Windows Display control panel, nor is it the same behavior that comes up when hitting the display-swap fn-hotkey. It reeks of sloppy design. As an aside, it IS the same as what's in the Nvidia Control panel -- but why the hell hasn't that just been integrated into the "Display" control panel -- we've only been doing this since 1995 now, can't we find a plugin architecture that works cohesively by now? I.e. "If a third party provides function X, show theirs, otherwise show the default MS interface."

There's about a dozen useless things preinstalled, of course. There's a shortcut to Ebay on the desktop. As if, someone who has used ebay before can't figure out how to get online.

It includes Office 2007...which as far as I'm concerned, the fucking Ribbon Bar will just convince him he wants to use OpenOffice...as soon as he learns what it is.

And so on, and so on.

Of course, I'm only here for a night...and Fios will be installed on tuesday. Naturally.

Well, I've got windows up and talking to his huge flatscreen monitor, and tells me that his friend Steve, who got him the laptop "has a little box on the side that everything just plugs into". I can't tell if he means a USB hub, or a full-blown port replicator.

I've also taken a few moments to revert his copy of Windows back to the same way 95-and-everything previous looked before this. Sadly, that's something that Windows keeps making harder and harder to find, so I hope to god the Fios guy doesn't look at this and say "Hey, we only support XP or better".

As usual, the system has a 64 bit processor (a Turion, whatever the fuck that means) and HP has decided to only include a 32 bit OS.

There's some part of me that really wants to tell my father "here, this will run natively on your hardware and will actually use it to its full potential", except Linux Just Ain't There Yet...

I wish he'd have gotten a mac.

gushi: (Default)

Okay, so I have a redhat 9 box in new york, with a whopping ten gigs of hard drive space. It's a cobalt raq3, and it's basically a spare DNS server, and Nagios box for me.

I recently discovered something ugly. There was no "postmaster" user on this system. Which is fine. /etc/aliases usually redirects that to root.

Except...there was no aliases file.

Picture this for a moment. "Postmaster" is the user that gets a notification whenever there's a delivery failure. Even when there's a failure delivering to "postmaster". Which in turn causes a failure...which causes postmaster to get a notification that that failed.

Yeah, headache, that.

So I try to rebuild the aliases file, and I get newaliases: no database format defined

Apparently, this version of sendmail doesn't actually link against an installed database engine.

So I go, attempt to download a new clean version of BerkeleyDB...only to find that a) the oracle site is down and not responding. I trace down another location for the file, and then I get to have FUN trying to get sendmail to notice that these libraries exist.

Apparently, a good number of the howto's out there are dead fucking wrong about this.

Also, couple this with the fact that Sendmail doesn't use your standard configure, make, make install procedure. It has a shell script in there called "Build". With no clear documentation. And once you run it, it magically holds on to your settings that you've run it with. There's no "make clean" option. Every time it didn't work, I got to blow away and reinstall the fucking thing.

I'm kind of where I want to be now -- although I'm kind of spoiled by BSD, and the ability to have my .mc file in /etc/mail and a nice makefile that just does anything I need it to.

I'd probably like to get StartTLS working on this thing, just so I can say I can....but I'm just happy to get the shit working.

Next: Building a new BIND. Headdesk

gushi: (Nevar Button)

So apparently there was a real nasty worm out and about called Bagle. We all remember it, right?

So I was noticing in clearing out my error logs that I have a ton of hits to prime.gushi.org/777.gif (it's a 404 and to the best of my knowledge always has been).

So here's where it gets scary: I go to the reputable trend micro site seen here, looking for info on 777.gif:

http://www.trendmicro.com/vinfo/virusencyclo/default5.asp?VName=WORM_BAGLE.FN&VSect=T

And then I see it, slightly obscured...

This worm downloads possibly malicious files from the following URLs:

[...]

http://pr{BLOCKED}ushi.org/777.gif

Yup. Apparently prime's listed as one of the distribution sites for one of the worst virii out there.

Now, near as I can tell, that file's NEVER been in that webspace. In fact http://prime.gushi.org returns a 403 and points at a null directory.

This is not the first time I've been targeted like this, nor will it be the last. But Hrmmmmmm...How can I use this?

Well, I now have the means to VERY EASILY compile a list of every infected user out there and tie it into an email script.

In fact, I just realized, I could write up a very quick virus removal script, run it through bat2exe, and drop it right in place there -- except people who have done similar things have already been sued in this ridiculous world of ours.

I just emailed one antivirus vendor (and will email others) with the following:


Hello,

I am the owner of prime.gushi.org -- I recently discovered that I am rather popular with the bagle worm (I am listed 
as a download site for the malware) -- a file called 777.gif.  As far as I can tell, I have never hosted this file 
(that's my hostname, but leads to a "null" site).

I would like to do a bit more research to clean this up, as there's a file commonly distributed with movabletype 
that's also titled 777.gif.

I'd like to know if you could give me the file's info (the size, md5, sha1, strings output, etc).

I suppose I can email all the ISP's out there and have them fix their users -- or for that matter report those users to the various blacklists. Or I can just mod_rewrite this crap out of my logs.

Stupidity

Apr. 15th, 2008 11:53 am
gushi: (Default)

Here, let me know if anyone spots the problem with this email I recently got:

Date: Tue, 15 Apr 2008 11:37:26 -0400
From: Avis <avis@avis.ed10.net>                                                                                               
Reply-To: Avis <avis.bwqw49.78761@avis.ed10.net>
To: avis@gushi.org
Subject: Great Offers from Avis                                                                                               
Parts/Attachments:
   1   OK     118 lines  Text
   2 Shown    177 lines  Text
----------------------------------------                                                                                      
Ensure our messages always go straight to your inbox. Add Avis@rent.avis.com to your Address Book or Safe List.               
Click here [click.avis.com] to learn how. If this e-mail does not appear properly, please click here
gushi: (Default)

Dear PHP.

Please for the love of god stop being such a monlithic pig.

I'm watching ONE INSTANCE of ONE PHP SCRIPT for ONE USER (that's ONE page hit) eat 107 megs of ram right now. To serve ONE http request.

I mean, seriously, apache has been a multithreaded daemon since april, 2002. And because "PHP is Glue", (and yes, that's literally your excuse), you decided that you'll most likely never be thread safe.

Not that I'll ever run you threaded, of course -- since unix can't do privilege separation on threads. Because you foster the ability for people to write such ugly and insecure, godawful code, I can't even use the techniques most ISP's do to "trick" you into running faster, running you as a module in my webserver.

Of course, PHP regardless of how you run, you could define yourself as a slim application, and dynamically load the modules you need on the fly. Do a quick inventory of which modules are present, and then load when needed. If someone does something like phpinfo(), then sure, load all the modules and show all your stuff. Otherwise, unless someone calls mysql_connect, don't load it. Be cognizant of when the mysql lib is there, maybe via a trusted host cache of modules-to-functions -- but there's no reason for 90 percent of the code in you to be in ram 90 percent of the time. But you don't get it, PHP. Instead, support for everything from TrueType to SSL is compiled, statically, into you.

While we're at this dynamic bit, PHP, security could be handled the same goddamned way. I could run you as a module if you'd have thought AT ALL about the problems you'd cause ISPs everywhere. But you didn't.

This too, could probably be done securely, if you knew what you were doing, PHP. Force all file access through a stub program, like suexec does. For files that merely need to be read, simply check permissions, and if you could read them anyway, just do it. For files in a trusted directory (like modules), just load them. For files that need to be read only by the user (like mysql configs), call your stub program to pipe the data in to you. For files that need to be written, actively...do all your writes through the stub, just like suExec or SuPHP. Because god knows we haven't had enough zero-day postnuke (ever wonder why they called it that?) or phpBB security holes.

But being sensible or logical is beyond you, PHP. I can't stand you, PHP. You continue to anger and frustrate me, and drive prime's load skyward whenever someone decides to run a badly-written gallery app to basically run a for-loop for a directory of images.

But if I disable you, I lose users. And they ostensibly give me money.

So you can stay, but you're sleeping on the fucking couch.

gushi: (Default)
Preface:

I am writing this in the event that someone will pick this up on Technorati or something similar...and that maybe it will help them. If I ever get my technical personal site set up, this is one of the articles I hope to have there. My friends are WELCOME and ENCOURAGED to read this, and I hope they do, because I'm looking to see:

a) if I explain things well enough for the layperson to understand
b) if there's any glaring errors in my writing/formatting/etc
c) if any of you non-techs learn something from it. For example, I'm not a mechanic...may never do anything more advanced on my car than changing the wipers, but I like knowing how a car works.

Introduction

I am writing today about Duplex Mismatch. This is a problem anyone, from the smallest home LAN, to the largest corporate network, will undoubtedly face. And in most cases, it will drive the network admin (or owner) COMPLETELY INSANE trying to fix everything else before one discovers this issue. I've seen people cite viruses, bad cables, bad cable modems, bad routers, bad network cards, bad power supplies (are we getting the idea here?)...all when it was caused by this once simple problem.

Overview

Most of us have network wires, and see a little light on our network cards, or our routers, that reads "FULL DUPLEX" or "FDX". Most people don't know what it means. That little light is SO CRUCIAL to your speed that you may never know it

Basically, what that little light (and the underlying circuitry) means is "can your network card SEND and RECEIVE at the same time".

Let's liken this to the old CB radios...where only one person could occupy the airspace at a time. This is why people started saying things like "BREAKER", "ROGER", "OVER", "OVER AND OUT". A CB radio is a HALF DUPLEX device. More modernly, think of your nextel phones. Only one person can be pressing their PTT button at the same time.

Now, a telephone is a FULL DUPLEX device...with some interesting logical exceptions. First, is that a telephone will give you some "echoback". I.e. if you are speaking to a person, you will also hear your OWN voice through your earpiece...which is done by the telco as sort of an assurance that you're being heard (this sort of thing also is why cell phones often do NOT do this -- it requires double the bandwidth). Also, calling a telephone truly "full duplex" is a misnomor because the human brain is NOT full-duplex capable. Two people can not talk continuously and simultaneously with the expectation that the other person will understand and absorb their side of the conversation while continuing to transmit. For reference, people can speak, conversationally, at about 200 words per minute (and can comprehend the same or higher).
However, the "telephone" example above still applies...it's not bidirectional. Two people can't speak and receive 400 words in one minute. (We'll ignore the fact that I've seen people on phones try to. This seems to be more prevalent when they are driving large SUVs and on cell phones...but I digress.)

I mention this particular nuance of the brain, because it illustrates an important concept in networking: the collision. If you've ever tried to talk with someone, where you both speak at once...then pause, and figuring the other person is waiting for you to speak, one of you speaks first, and figures whomever speaks first gets to talk...you have experienced a collision, and retransmission. (We'll also ignore the fact for now that more often than not, both people will re-collide several times before one finally SAYS "okay, you go first").

Every basic networking person will be asked on a job interview "what's the difference between a hub and a switch?"

While the correct answer may involve the distinctions between the number of collision domains, and Broadcast Domains, (which is true -- hubs will give you collisions and properly configured switches will not) the layman's answer is "Nobody uses a hub anymore". All a hub does is connects the SEND pairs of one ethernet jack, to the RECEIVE pairs of every other jack, and provide enough minor brains to give a link light (and as far as I know, that latter part is the only reason such things need to be powered. You can think of such a thing as a "BUS" network if you like.

I won't get into how network cards try to "gracefully" handle collisions. I won't get into Carrier Sense Multiple Access, or Carrier Sense Multiple Access with Collision Detection because they for the most part do not apply to Ethernet. The protocols were in some cases logical and intelligent, but as far as I am concerned, they are obsolete, or at least deprecated. You can buy a device for $30 which speaks full duplex ethernet. There's no reason for them.

So all the above is background. We're done with it.

Now on to the critical pieces of information you will NEED to know.

Nobody uses hubs anymore, because they only spoke one speed. 10baseT. I think in my entire house I only have one device that can't speak 100M (and those are my commercial-grade cisco aironet radios, which are not part of my production home network, but stand at the ready for special cases). You can find 10baseT switches, but they're pretty rare. The thing that makes them different is they have the ability to auto-negotiate (or to hard-set, in some very special cases).

With the advent of 100Meg, there came a need for a device not only to configure its duplex accordingly, but but also its speed -- and the protocol states that in the event of an absence of auto-negotiation -- a device must fall back to the lowest protocol.

This single thing has been the biggest network killer I've ever seen.

Because any communication requires some back-communication to the host (be it via ACKs, or "Okay, send more" or what not. And when you have a server (and that's the context I'm working in, in real life), every time one of those packets comes back, it causes a collision, a "back off and retry". The bigger catch, of course is, the MORE data you are trying to send (or, the faster you are trying to send it -- since it will be sent at wire-speed), the more collisions you get back.

Some devices (for example, the fast ethernet cards on 7500 series cisco routers), can ONLY speak 100 megabits. They also do NOT do auto-duplex negotiation. Thus, both sides of the connection need to be set, BY A HUMAN, to be 100 Megabits, full duplex (otherwise, the other side, in the absence of that signal, will fail down to 10M/half)

Wikipedia has a decent article about this HERE. But it gets technical and obscure, and that's part of what I'm trying to avoid here. So, without any further ado...

Rules to live by

The basic rules here are:

1) BOTH DEVICES must support autonegotiation, and have it turned on. If both don't, neither will autonegotiate.

2) Some devices (and operating systems) are buggy as hell in the regard that even though the SPEC supports being able to specify speed and duplex differently, it just won't work. For example, many devices (Cisco, Extreme, and HP among them) can not do "speed auto duplex full", or "speed 10 duplex auto". The protocol allows this -- it's possible for them to only advertise the ten megabit speeds, but all duplex modes -- but it was never coded into the Operating Systems.

Although I would love to be able to set this feature to automatically give a person a ten meg link while requiring no extra config on their end.

3) Some OSes will, despite being set auto, NOT AUTONEGOTIATE unless a switch is present (for example, when connecting computers with a crossover cable, you may or may not see duplex issues, since many OSes wait to see the capabilities of what they assume to be the "switch". I have also run into serious autonegotiation issues when cross-connecting two SWITCHES with a crossover cable. Thus, I consider autonegotiation to ONLY be reliable between SWITCHES and DEVICES (not between devices and devices, or switches and switches). Even though I have found nothing (yet) in the protocols about this, this seems to be a "universal truth".

4) DO NOT BELIEVE THINGS UNLESS YOU CAN SEE (and/or force) BOTH ENDS OF THE CONNECTION. This is a serious problem, because:

a) not all switches and/or cards have an indicator light for duplex and...
b) there is no way in windows to VIEW the duplex of a device (although there MAY be a way to force it)...
c) If you force duplex on one end (say, in windows), the other end will fail to see the autonegotiation pulse, and default to half...
d) Unless it's a managed device where you can force it there too (most "home" switches, you cannot).

5) Do not think that by forcing ONE END of the connection to be a certain way, that it will fix the problem. When you FORCE a setting, you DISABLE AUTONEGOTIION. You do NOT limit the number of "supported modes" that your device sends to "the one you set" (although in reality this might make more sense).

6) For whatever reason, there has been trouble reported between Intel NICs and Cisco Switches. I should personally mention that I've seen almost-perfect counterfeits of both, and I'm sure that doesn't help.

7) Even if you think you are running at full speed, even if you've forced it, your card MAY report collisions, which can be an indicator that either the other side isn't running right, or that you've done your "force" wrong (for example, forced speed but not duplex).

8) BE ESPECIALLY SUSPICIOUS of any connection you have between two like devices. For example, two switches, two machines without a switch, your cable modem to your router versus your cable modem to a PC. A common scenario I've seen is "everyone on the first floor can copy files from each other fine, and everyone on the second floor can copy from each other fine...but between floors (or offices, or workgroups, it's slow.)

9) Remeber that some protocols have support for retransmission...but UDP based protocols (many games, for example) just drop the frames and do not retransmit, so "lag" or "ghosting" might by a symptom of this.

10) If you are in doubt as to how a windows network card is working, you can boot up a linux "Live CD" like Knoppix or FreeSBIE to see how they detect your network card is working. If they don't come up as auto...chances are neither will your windows machine.

Reference

Here are some useful commands (for operating systems other than windows)...

Linux:

ifconfig -a will show you error and collision counts. mii-tool and ethtool will show you your link speed (and allow you to set it). However, the syntax is a little wierd. You can also put (under the redhat variants, anyway) in /etc/sysconfig/network-scripts/ifcfg-eth0...

ETHTOOL_OPTS="speed 100 duplex full autoneg off"

Unfortunately with the variety of linux distros out there, you may be relegated to a google search for your own particular flavor.

FreeBSD:

You can view collision counts with netstat -i. You can view the network card config with ifconfig -a. You can use ifconfig to force the media to a specific speed. For example, from my /etc/rc.conf:

ifconfig_em1="inet 10.1.1.1 netmask 255.255.255.0 media 100baseTX mediaopt full-duplex"

You can view the man page for the network card driver (in this case, man em) for the various modes supported under each card.

Windows:

Here's where you can FORCE the duplex setting (again, I do not recommend this unless you can do it on BOTH ENDS, which is useful if you're "cross connecting" two computers (to transfer files, as a quick-and-dirty-internal-network, etc).

Go Start-->Settings-->Network Connections-->Your network card (example: "Local Aera Connection")

At the top you'll see "Connect using:" and the actual name of your network card (like "Intel Etherexpress Pro" or something like that.) There wll be a "Configure..." button next to that network card name. Click it.

You should be taken to another page with tabs that will include: General, Advanced, Driver, Resources, Power Management...

Under "Advanced" you will see settings that may include names like "Mode" or "Media Type" or "Parity" or something similar. Sadly, this bit is at the mercy of the person who writes the driver, so there's no consistently-used name. But the options should be self-explanatory. You can tweak these if you like, but you should remember what it was originally set for...and set it back if it does not alleviate the issue.

Conclusion I've experienced this issue dozens, not hundreds of times. If your network is running "not quite as fast as you feel it should", it's a point to investigate. Once you understand the underlying media, it takes only seconds to check.

Questions for the Reader

Is fiber a full-duplex medium?

What about wi-fi in "infrastructure mode"?

What about wi-fi in "ad-hoc" mode?

How many of the 8 wires in an average network cable are actually USED to communicate signals, at 100MBPS?

How about at gigabit speeds?

gushi: (Default)
Preface:

I am writing this in the event that someone will pick this up on Technorati or something similar...and that maybe it will help them. If I ever get my technical personal site set up, this is one of the articles I hope to have there. My friends are WELCOME and ENCOURAGED to read this, and I hope they do, because I'm looking to see:

a) if I explain things well enough for the layperson to understand
b) if there's any glaring errors in my writing/formatting/etc
c) if any of you non-techs learn something from it. For example, I'm not a mechanic...may never do anything more advanced on my car than changing the wipers, but I like knowing how a car works.

Introduction

I am writing today about Duplex Mismatch. This is a problem anyone, from the smallest home LAN, to the largest corporate network, will undoubtedly face. And in most cases, it will drive the network admin (or owner) COMPLETELY INSANE trying to fix everything else before one discovers this issue. I've seen people cite viruses, bad cables, bad cable modems, bad routers, bad network cards, bad power supplies (are we getting the idea here?)...all when it was caused by this once simple problem.

Overview

Most of us have network wires, and see a little light on our network cards, or our routers, that reads "FULL DUPLEX" or "FDX". Most people don't know what it means. That little light is SO CRUCIAL to your speed that you may never know it

Basically, what that little light (and the underlying circuitry) means is "can your network card SEND and RECEIVE at the same time".

Let's liken this to the old CB radios...where only one person could occupy the airspace at a time. This is why people started saying things like "BREAKER", "ROGER", "OVER", "OVER AND OUT". A CB radio is a HALF DUPLEX device. More modernly, think of your nextel phones. Only one person can be pressing their PTT button at the same time.

Now, a telephone is a FULL DUPLEX device...with some interesting logical exceptions. First, is that a telephone will give you some "echoback". I.e. if you are speaking to a person, you will also hear your OWN voice through your earpiece...which is done by the telco as sort of an assurance that you're being heard (this sort of thing also is why cell phones often do NOT do this -- it requires double the bandwidth). Also, calling a telephone truly "full duplex" is a misnomor because the human brain is NOT full-duplex capable. Two people can not talk continuously and simultaneously with the expectation that the other person will understand and absorb their side of the conversation while continuing to transmit. For reference, people can speak, conversationally, at about 200 words per minute (and can comprehend the same or higher).
However, the "telephone" example above still applies...it's not bidirectional. Two people can't speak and receive 400 words in one minute. (We'll ignore the fact that I've seen people on phones try to. This seems to be more prevalent when they are driving large SUVs and on cell phones...but I digress.)

I mention this particular nuance of the brain, because it illustrates an important concept in networking: the collision. If you've ever tried to talk with someone, where you both speak at once...then pause, and figuring the other person is waiting for you to speak, one of you speaks first, and figures whomever speaks first gets to talk...you have experienced a collision, and retransmission. (We'll also ignore the fact for now that more often than not, both people will re-collide several times before one finally SAYS "okay, you go first").

Every basic networking person will be asked on a job interview "what's the difference between a hub and a switch?"

While the correct answer may involve the distinctions between the number of collision domains, and Broadcast Domains, (which is true -- hubs will give you collisions and properly configured switches will not) the layman's answer is "Nobody uses a hub anymore". All a hub does is connects the SEND pairs of one ethernet jack, to the RECEIVE pairs of every other jack, and provide enough minor brains to give a link light (and as far as I know, that latter part is the only reason such things need to be powered. You can think of such a thing as a "BUS" network if you like.

I won't get into how network cards try to "gracefully" handle collisions. I won't get into Carrier Sense Multiple Access, or Carrier Sense Multiple Access with Collision Detection because they for the most part do not apply to Ethernet. The protocols were in some cases logical and intelligent, but as far as I am concerned, they are obsolete, or at least deprecated. You can buy a device for $30 which speaks full duplex ethernet. There's no reason for them.

So all the above is background. We're done with it.

Now on to the critical pieces of information you will NEED to know.

Nobody uses hubs anymore, because they only spoke one speed. 10baseT. I think in my entire house I only have one device that can't speak 100M (and those are my commercial-grade cisco aironet radios, which are not part of my production home network, but stand at the ready for special cases). You can find 10baseT switches, but they're pretty rare. The thing that makes them different is they have the ability to auto-negotiate (or to hard-set, in some very special cases).

With the advent of 100Meg, there came a need for a device not only to configure its duplex accordingly, but but also its speed -- and the protocol states that in the event of an absence of auto-negotiation -- a device must fall back to the lowest protocol.

This single thing has been the biggest network killer I've ever seen.

Because any communication requires some back-communication to the host (be it via ACKs, or "Okay, send more" or what not. And when you have a server (and that's the context I'm working in, in real life), every time one of those packets comes back, it causes a collision, a "back off and retry". The bigger catch, of course is, the MORE data you are trying to send (or, the faster you are trying to send it -- since it will be sent at wire-speed), the more collisions you get back.

Some devices (for example, the fast ethernet cards on 7500 series cisco routers), can ONLY speak 100 megabits. They also do NOT do auto-duplex negotiation. Thus, both sides of the connection need to be set, BY A HUMAN, to be 100 Megabits, full duplex (otherwise, the other side, in the absence of that signal, will fail down to 10M/half)

Wikipedia has a decent article about this HERE. But it gets technical and obscure, and that's part of what I'm trying to avoid here. So, without any further ado...

Rules to live by

The basic rules here are:

1) BOTH DEVICES must support autonegotiation, and have it turned on. If both don't, neither will autonegotiate.

2) Some devices (and operating systems) are buggy as hell in the regard that even though the SPEC supports being able to specify speed and duplex differently, it just won't work. For example, many devices (Cisco, Extreme, and HP among them) can not do "speed auto duplex full", or "speed 10 duplex auto". The protocol allows this -- it's possible for them to only advertise the ten megabit speeds, but all duplex modes -- but it was never coded into the Operating Systems.

Although I would love to be able to set this feature to automatically give a person a ten meg link while requiring no extra config on their end.

3) Some OSes will, despite being set auto, NOT AUTONEGOTIATE unless a switch is present (for example, when connecting computers with a crossover cable, you may or may not see duplex issues, since many OSes wait to see the capabilities of what they assume to be the "switch". I have also run into serious autonegotiation issues when cross-connecting two SWITCHES with a crossover cable. Thus, I consider autonegotiation to ONLY be reliable between SWITCHES and DEVICES (not between devices and devices, or switches and switches). Even though I have found nothing (yet) in the protocols about this, this seems to be a "universal truth".

4) DO NOT BELIEVE THINGS UNLESS YOU CAN SEE (and/or force) BOTH ENDS OF THE CONNECTION. This is a serious problem, because:

a) not all switches and/or cards have an indicator light for duplex and...
b) there is no way in windows to VIEW the duplex of a device (although there MAY be a way to force it)...
c) If you force duplex on one end (say, in windows), the other end will fail to see the autonegotiation pulse, and default to half...
d) Unless it's a managed device where you can force it there too (most "home" switches, you cannot).

5) Do not think that by forcing ONE END of the connection to be a certain way, that it will fix the problem. When you FORCE a setting, you DISABLE AUTONEGOTIION. You do NOT limit the number of "supported modes" that your device sends to "the one you set" (although in reality this might make more sense).

6) For whatever reason, there has been trouble reported between Intel NICs and Cisco Switches. I should personally mention that I've seen almost-perfect counterfeits of both, and I'm sure that doesn't help.

7) Even if you think you are running at full speed, even if you've forced it, your card MAY report collisions, which can be an indicator that either the other side isn't running right, or that you've done your "force" wrong (for example, forced speed but not duplex).

8) BE ESPECIALLY SUSPICIOUS of any connection you have between two like devices. For example, two switches, two machines without a switch, your cable modem to your router versus your cable modem to a PC. A common scenario I've seen is "everyone on the first floor can copy files from each other fine, and everyone on the second floor can copy from each other fine...but between floors (or offices, or workgroups, it's slow.)

9) Remeber that some protocols have support for retransmission...but UDP based protocols (many games, for example) just drop the frames and do not retransmit, so "lag" or "ghosting" might by a symptom of this.

10) If you are in doubt as to how a windows network card is working, you can boot up a linux "Live CD" like Knoppix or FreeSBIE to see how they detect your network card is working. If they don't come up as auto...chances are neither will your windows machine.

Reference

Here are some useful commands (for operating systems other than windows)...

Linux:

ifconfig -a will show you error and collision counts. mii-tool and ethtool will show you your link speed (and allow you to set it). However, the syntax is a little wierd. You can also put (under the redhat variants, anyway) in /etc/sysconfig/network-scripts/ifcfg-eth0...

ETHTOOL_OPTS="speed 100 duplex full autoneg off"

Unfortunately with the variety of linux distros out there, you may be relegated to a google search for your own particular flavor.

FreeBSD:

You can view collision counts with netstat -i. You can view the network card config with ifconfig -a. You can use ifconfig to force the media to a specific speed. For example, from my /etc/rc.conf:

ifconfig_em1="inet 10.1.1.1 netmask 255.255.255.0 media 100baseTX mediaopt full-duplex"

You can view the man page for the network card driver (in this case, man em) for the various modes supported under each card.

Windows:

Here's where you can FORCE the duplex setting (again, I do not recommend this unless you can do it on BOTH ENDS, which is useful if you're "cross connecting" two computers (to transfer files, as a quick-and-dirty-internal-network, etc).

Go Start-->Settings-->Network Connections-->Your network card (example: "Local Aera Connection")

At the top you'll see "Connect using:" and the actual name of your network card (like "Intel Etherexpress Pro" or something like that.) There wll be a "Configure..." button next to that network card name. Click it.

You should be taken to another page with tabs that will include: General, Advanced, Driver, Resources, Power Management...

Under "Advanced" you will see settings that may include names like "Mode" or "Media Type" or "Parity" or something similar. Sadly, this bit is at the mercy of the person who writes the driver, so there's no consistently-used name. But the options should be self-explanatory. You can tweak these if you like, but you should remember what it was originally set for...and set it back if it does not alleviate the issue.

Conclusion I've experienced this issue dozens, not hundreds of times. If your network is running "not quite as fast as you feel it should", it's a point to investigate. Once you understand the underlying media, it takes only seconds to check.

Questions for the Reader

Is fiber a full-duplex medium?

What about wi-fi in "infrastructure mode"?

What about wi-fi in "ad-hoc" mode?

How many of the 8 wires in an average network cable are actually USED to communicate signals, at 100MBPS?

How about at gigabit speeds?

May 2017

S M T W T F S
  123456
78910111213
14151617181920
21222324252627
28293031   

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 22nd, 2017 04:47 pm
Powered by Dreamwidth Studios