Day in the life of a Systems Administrator

Day in the life of a Unix Systems Administrator

Wow, been almost a year since I blogged anything. I’m getting lazy.

So what’s the daily life of a systems administrator like? Here was today:

The plan coming this morning: Begin quarterly “Vulnerability Audit Report”.

What did I do?
Windows server starts alerting on CPU at midnight, again. We fixed the problem on Tues. Why is it alerting again? Of course it corrects itself before I can get logged in and doesn’t go off again all day. Send an email to the person responsible for the application on that server to ask if the app was running any unusually CPU intensive jobs. Respond with a screenshot showing times CPU alerts went off. Get response of “nothing unusual”. As usual.

We updated the root password on all Unix servers last week. Get a list of 44 systems from a coworker that still have the old root password.
Check the list, confirm all still have the old root password.
Check the list against systems that were updated via Ansible. All on the Ansible list. No failures when running the Ansible playbook to update the root password. All spot-checks that the new root password was in effect at the time showed task was working as expected.
Begin investigating why these systems still have the old root password.
Speculation during team scrum that Puppet might be resetting the root password.
Begin testing a hypothesis that root password was, in fact, changed, but something else is re-setting it back to the old password.
Manually update root password on one host. Monitor /etc/shadow to see if it changes again after setting the password. (watch -d ls -l /etc/shadow)
Wait.
Wait.
Wait some more.
Wait 27 minutes, BOOM! /etc/shadow gets touched.
Investigate to see if Puppet is the culprit. I know nothing about Puppet. I’m an Ansible guy. The puppet guy (who knows just enough to have set up the server and built some manifests and get Puppet to update root the last time the root password was changed before I started working here.) is out today.
Look at log files in /var/log. Look at files in /etc/puppet on puppet server. Try to find anything that mentions “passw(or)?d&&root” (did I mention I’m not a puppet guy?). Find a manifest that says something about setting the root password, but it references a variable. Can’t find where the value of that variable is set.
Look some more at the target host. See in log files that it’s failing to talk to the Puppet server, so continuing to enforce the last set of configuration stuff it got. Great, fixing this on the Puppet server won’t necessarily fix all the clients that have been allowed to lose connectivity that no one noticed (entropy can be a bitch.)
Begin looking at what to change on the client (other than just “shut down the Puppet service” and “kill it with fire!”). Realize it’s much faster to surf all the files and directories involved with “mc”.
Midnight Commander not installed. Simple enough, “yum install mc”.
Yum: “What, you want to install something in the base RHEL repo? HAH! Entropy, baby! I have no idea what’s in the base repo.”.
Me: “Hold my beer.” (This is Texas, y’all.)
(No, not really. CTO frowns on drinking during work hours or drinking while logged into production systems. Or just drinking while logged in…)
OK, so more like:
Me:
“Hold my Diet Coke.”
Yum: “Red Hat repos? We don’t need no steeeenking Red Hat repos!”
Me:

Start updating Yum repo cache. Run out of space in /var. Discover when this server was built, it was built with much too small a /var. Start looking at what to clean up.
Fix logrotate to compress log files when it rotates them, manually compress old log files.
/var/lib/clamav is one of the larger directories. Oh, look, several failed DB updates that never got cleaned up.
Clean up the directory, run freshclam. Gee, clamav DB downloads sure are taking a long time given that it’s got a GigE connection to the local DatabaseMirror. Check Freshclam config. Yup, the local mirror is configured… external mirror ALSO configured. Dang it. Fix that. ClamAV DB updates no much faster.
Run yum repo cache update again. Run out of disk space again. Wait… why didn’t Nagios alert that /var was full?
Oh, look, when /var was made a separate partition, no on updated Nagios to monitor it.
Log into Nagios server to update config file for this host. Check changes into Git. Discover there have been a number of other Nagios changes lately that haven’t been checked into Git. Spend half an hour running git status / diff / add / delete / commit / push to get all changes checked into Git repo.
Restart Nagios server (it doesn’t like reloads. Every once in a while it goes bonkers and sends out “The sky is falling! ALL services on ALL servers are down! Run for your lives! The End is nigh!” if you try a simple reload.
Hmm… if Nagios is out of date for this host, is Cacti…
Update yum cache again. Run out of disk space again.
Good thing this is a VM, with LVM. Add another drive in vSphere, pvcreate, swing your partner, vgextend, lvresize -r, do-si-do!
yum repo cache update… FINALLY!
What was I doing again? Oh, right, install Midnight Commander…
Why? Oh yeah, searching for a Puppet file for….?
Right, root password override.

Every time I log into a server it seems like I find a half dozen things that need fixing. Makes you not want to log into anything, so you can actually get some work done. Oh, right, entropy…

last few days

I’m back.
OK, I’ve been back about 36 hours now.
Not that most of you noticed I was gone.

Next time I have to go to Houston I’ll just drive. Travel time by Southwest Airlines from DAL to HOU, including getting a ride to DAL[1], allowing for security, waiting for boarding, waiting for shuttle to hotel from HOU, crack-head shuttle driver, is about an hour longer than it would have taken to just drive. Return trip was the same, sans crack-head driver, since we just took a taxi, whose driver had a bit more clue where he was going. And big cajones[2].

The hotel[3] was not the nicest I’ve ever stayed, but it was very nice. It was easily the nicest bed I’ve slept in. I must acquire a set of bedding like theirs. Mattress pad, nice sheets, top sheet, pad, another top sheet, nice comforter.
No vent fan in the bathroom, so all the mirrors (and my glasses) got fogged up. Who ever heard of a hotel/motel that doesn’t vent the bathroom?

The conference was, over all, a waste of time. Their “beginner track” was too basic. “Installation”, “Configuration” and “SSL” scheduled for an hour each, were done in 10 minutes. The “advanced track” covered “Advanced troubleshooting”, mySQL, Anti-spam and php. “Advanced Troubleshooting” was simply “How to use strace”. Gee, how informative. mySQL covered “why you shouldn’t upgrade to 4.1 unless you REALLY mean it”. PHP was “don’t install 5.0. Really. Just don’t.” All of them were presented by a guy who started each presentation with a rundown of his resume (as if we were supposed to be impressed that he was a “senior technician” with one of the vendors at the conference before he came to work for cPanel.) His anti-spam presentation basically amounted to “make anyone who sends you mail prove their a real person by blocking their mail until the respond to your auto responder” and “RBLs suck. The people who run them are evil and clueless.”[4] Obviously he’s been using the wrong RBLs and doesn’t know how much the “prove that you love me” technique just pisses people off.

However, it was two days off work, with pay, some good meals and socializing with other industry folks.

Yesterday, I met up with for a while. Turns out the place he’s staying here in Dallas is just the next apartment complex over. Afterward I came home and got ready for a pool party at Amythest’s, with her sister, and other DFW Ufies. Shared that bottle of wine I bought a couple of weeks ago at the wine tasting and watched a silly movie.

So far last night / today I’ve made progress on Project X by getting Open-LDAP installed and successfully added an entry to the database. Next I get to configure Qmail to authenticate against it.

[1] Since ${poe} was too cheap to pay for a shuttle. REALLY cheap, since we were going to need a shuttle at the HOU end anyway.
[2] Got in the exit lane for the freeway interchange, which came to a complete stop. So he got out of the lane, slammed on the gas, passed everyone waiting to get on the interchange and cut right back in at the very last second.
[3] If I ever have to travel on business and the person arranging the travel forgets to PAY for the hotel again, I will hand them my two week notice. Going to check into a $300/night hotel and being asked for MY credit card was not fun. One call to the boss and he took care of it with his card, but he had to fax them both sides of his credit card and drivers license.
[4] With FUD like “All it takes is your competitor forging headers once to get you added to a whole bunch of RBLs” and “You have to pay each of them a ‘bribe’ to their pet charity to get off their list”. Guess he’s never heard of rfc-ignorant, ORDB, MAPS-RSS, MAPS-DUL, SORBS, DSBL

Don’t suppose anyone knows how to get Plesk 8.0.0 to install on FreeBSD 6.0?

Start packages installation
Install package psa
bsdtar 1.02.023, libarchive 1.02.026
Use gtar
/usr/local/bin/gtar
bsdtar 1.02.023, libarchive 1.02.026
Use gtar
/usr/local/bin/gtar
To continue installing, you should install Perl 5.008008 (you have Perl 5.008007 installed)
Execute cmd failed: sh /root/psa/PSA_8.0.0/dist-standard-FreeBSD-6.0-i386/psa_v8.0.0_build80060406.16_os_FreeBSD_6.0_i386.sh
ERROR: Error while install .sh package
ERROR: Installation failed