/dev/zero

Gotta love working with an “engineer” who can’t explain anything and thinks your an idiot for asking detailed questions.

We started a new “fault tolerance” project to make all of our services completely redundant across data centers. When trying to explain how it works, my co-engineer used DNS for an example. All he could tell me is that “all the data will be replicated across all data centers so if one goes away, the others will answer”. When I started to try to get more details about how this will work (”Are all servers just going to point to the same IP and the system will redirect that IP to a working server?”) he couldn’t tell me. Then got frustrated with me because I was focusing on DNS. He kept wanting to give me the Executive Summary of the project, while I was asking specifics. And he doesn’t know the specifics and doesn’t understand the relevance of my questions.

To put it another way: this guy is not an engineer. He wants to be an engineer, he got the job title, learned a few things about a few specific services (and specifically, the software we use to implement them) but get outside of that and he’s lost. Problem is, I know next to nothing about the software we use to implement these services. I AM familiar with the concepts however. He mistakes not being familiar with our specific software for ignorance (even though he doesn’t know the general concepts.)

Delivering stuff to DataBank.

Any recommendations for books on Ruby on Rails?

Not from a programmer’s perspective, but a sys admin’s. I have a contract coming up where I’ll have to set up all the sys admin stuff to interface with applications coded in Ruby.

I know nothing about Ruby.

So in the middle of the previous issue I get a jabber from another coworker asking if I’ve “Followed up on ${xxx}’s firewall issue” ($xxx being yet another coworker).
I ask if $xxx opened a ticket somewhere I would see it (I don’t monitor all queues, only those I’m responsible for).
Coworker: “No, he just emailed me and Bill”
Me: “He EMAILED You and Bill…. so you want to guess what my answer about if *I* have followed up on his firewall is?”¹
Coworker: “dude, I have and(sic) ${major_customer} vps server down, a ${other_major_customer} vps server down, ${CEO}’s unreachable best friend screaming at ${president} that his server isnt magically backing up, some guy who wants to know why his server isnt rebooting, and a client who was promised an impossible world by ${incompetent_manager} screaming at ${my_boss} and bill over not having the ability to mirror with RAID-5 or having only one host at a time allowed to connect to iscsi…. can you please look at ${xxx}s fw when you get a chance?”
Me: “Sure. Tell ${xxx} to open a support ticket.”
Coworker: “ill comment the ticket”
Oh, so there is a ticket? (Actually I knew this, but I was making a point.) That ticket is in a queue I don’t monitor. COMMENTING it isn’t going to change the fact that I don’t monitor it.

[1] - Against procedure: We don’t do ANYTHING in support without a ticket. “Non scriptum, non est.” If it isn’t written, it didn’t happen. So he sends an email to two other people and I’m expected to now about it?

I’m communicating with a coworker about transferring knowledge from him to me before he moves on to a new position with a sister company. I emailed him to congratulate him on his change of jobs and let him know when I would be available to get with him to transfer this knowledge. I specifically said I would not be available on Monday, and when I would be available.
I also asked him a specific question about our VPN.
He wrote back thanking me for my kind words. I again asked about the VPN. He wrote back telling me all kinds of information I already knew about the VPN that didn’t answer my question. I was about to write back again restating my question. Before I was done he jabbered me to ask if I would be available on Monday.
I counted to 10, then quoted the passage from my original email explaining I would not be available on Monday.
He replies back asking when I would be.
The funny thing is, he writes perfectly clear English. He just doesn’t seem to be able to READ it.

Got to work today and was informed that the one person in Support (and one of only 5 left in the NOC) who has been with LT longer than me, has turned in his two week notice.

We have enough trouble hiring people into Support positions, let alone keeping them. I know of at least two other senior Support persons who have confided in me that if they don’t get a certain amount on their raises, due at the end of this month, that they will walk. I’m fairly certain they won’t get it, as I am not making what they want, and I’m the highest paid person in the dept at our level. (and higher than some people at a higher level).

I’m about ready to walk if I don’t get my promotion to Sys Admin I was promised almost 5 months ago.

Nuff said.

Oh, by the way, anyone got an opening for a *nix geek who doesn’t write code in the D/FW area?
I could point you to my resume, but I figure people can raise hands first.

So the coworker who sits right next to me (smoker from previous post) turns to me and asks “So, are you going to quit with me?”

Wow, she’s been here 2 whole weeks and already hit the frustration point.

She works in Data Entry, where she has to suffer through all the various data bases and other data systems that they have to update. Of course none of these systems talk to eachother so they have to do tons of cut and paste to get data from one system into another. Which of course means lots of points for human error to mess everything up.

Glad I don’t have to do their job.

Faugh!

The assault on the nostrils when they come back from a smoke break!

Kill me. Kill me now.

All quiet on the western front…Yeah, haven’t posted in a while. Been so busy…

Really hating my job. I am SO ready to be rid of ${POE}. I’ve been waiting for a promotion I was promised since Sept. I got the pay raise in mid Oct, but am still doing the old job, which I hate, with no change on the horizon. They can’t seem to hire people fast enough and they need to hire like 3 people in order to replace me.

Meanwhile, I’ve started doing consulting with . I flew out to San Diego this last weekend and did a job for a small company involved in mobile messaging. Was very fun. Got to play with some new (to me) technologies (ethernet interface bonding, HA failover with heartbeat, load balancing (not new to me, but the piece of Cisco kit being used, 11501, is). Was on the go the whole time. I took off at 15:30 from DFW on Sat, landed 16:30 at SAN (two hour time difference), got the car from Avis and met the client outside the data center. Got a bit of orientation with the client, then we looked at their gear. I’m not too impressed with the data center they use. Very lax about security (they never bothered to ask for my ID when I first came in) but they have some strange security rules. Many other customers in that data center make the cabling at ${POE} look good, which is really saying something. The data center didn’t even have the second ethernet drop in the client’s cabinet activated, per contract, until after the client emailed them this weekend. And THEN they never responded to the email, just fixed it silently.

I was intending to take lots of pictures of beautiful San Diego to post to the blog, but ended up not having time to take pictures, or even SEE beautiful San Diego. Landed after dark, was on the go the whole time, took off after dark.

I did get ONE picture, which I haven’t offloaded from the camera yet.

Dear Contractor,

If I write in a ticket that something you did was done incorrectly, I wasn’t just banging randomly on a keyboard. I mean you actually did something incorrectly, which I had to fix. Luckily I caught it before the customer did and the customer ended up being happy with us for a change. Posting a comment in the ticket, in all caps, about how you are “not in the habit of not doing what the customer asked” impresses me not at all. You fucked up. I had to clean up your mess. Believe me, had I not caught that error and the customer saw it, we would have gotten our asses reamed.

Your unimpressed supervisor.

Dear Cow-worker,

My offer to come in on my day off to help you out with a major project != “I’ll do it for you”. I expected I would be in the office for 2 hours or so while I help you sort out what needs to be done. I ended up staying 8 hours while you took off while I was still working. Believe me, I’ll remember that next time you need help.

Your colleague.

So earlier this week I got a hit off my Monster (or Dice) resume. I’d never heard of the product that would be supported by the job (sys admin position) and told the recruiter such. Their response was “don’t worry, no one outside our industry has. We’ll train you for that part.” and that everything else on my resume looked perfect for the position, including the fact that I like to travel. They asked if I was interested, I said yes, they asked the classic “what are your salary requirements so we don’t waste each other’s time?” Now, this is where I get annoyed. That’s like asking someone on eBay “What’s your reserve, so we don’t waste each other’s time with my bidding?”

Anyway, I came back with “What are you willing to pay?”

They responded today with a quite acceptable figure.

Just got off the phone.

I aced the technical part of the interview. Part of it was a “break / fix” where they had me log into a server as root, reported various problems and I had to fix them. I got 2 out of the 3. (of which 1 actually had 2 things wrong.)

They wouldn’t tell me the answer to the one I didn’t get. “If we bring you on board, we’ll tell you how to fix that one. :)”
Should hear by the end of the day, or the end of the week at the latest.

Helped parents move to Austin this weekend.
Looked for new job in Austin.
Possible new job. In San Antonio.
2x my current pay. Great benefits. Relocation expenses covered.
Phone interview in 10 minutes.

Happy birthday, mizdandylynn!

I got a call from the weekend overnight team lead to ask if I knew the password for the Plesk licensing web site.
At 04:06 freaking AM.

Needless to say, at that hour of the morning I was not in any mental state to recite, from memory, the login and password for anything.
I was very not-pleased to be called at that hour, on my day off, to be asked for a password the caller should have had in his own records for months now. He should be using it about 5 times each and every shift, so why he suddenly doesn’t have it I don’t understand.

Later in the day I read in my email that another employee has accepted the position of weekend overnight team lead.
I can’t help but believe the two are connected.

I’m back.
OK, I’ve been back about 36 hours now.
Not that most of you noticed I was gone.

Next time I have to go to Houston I’ll just drive. Travel time by Southwest Airlines from DAL to HOU, including getting a ride to DAL[1], allowing for security, waiting for boarding, waiting for shuttle to hotel from HOU, crack-head shuttle driver, is about an hour longer than it would have taken to just drive. Return trip was the same, sans crack-head driver, since we just took a taxi, whose driver had a bit more clue where he was going. And big cajones[2].

The hotel[3] was not the nicest I’ve ever stayed, but it was very nice. It was easily the nicest bed I’ve slept in. I must acquire a set of bedding like theirs. Mattress pad, nice sheets, top sheet, pad, another top sheet, nice comforter.
No vent fan in the bathroom, so all the mirrors (and my glasses) got fogged up. Who ever heard of a hotel/motel that doesn’t vent the bathroom?

The conference was, over all, a waste of time. Their “beginner track” was too basic. “Installation”, “Configuration” and “SSL” scheduled for an hour each, were done in 10 minutes. The “advanced track” covered “Advanced troubleshooting”, mySQL, Anti-spam and php. “Advanced Troubleshooting” was simply “How to use strace”. Gee, how informative. mySQL covered “why you shouldn’t upgrade to 4.1 unless you REALLY mean it”. PHP was “don’t install 5.0. Really. Just don’t.” All of them were presented by a guy who started each presentation with a rundown of his resume (as if we were supposed to be impressed that he was a “senior technician” with one of the vendors at the conference before he came to work for cPanel.) His anti-spam presentation basically amounted to “make anyone who sends you mail prove their a real person by blocking their mail until the respond to your auto responder” and “RBLs suck. The people who run them are evil and clueless.”[4] Obviously he’s been using the wrong RBLs and doesn’t know how much the “prove that you love me” technique just pisses people off.

However, it was two days off work, with pay, some good meals and socializing with other industry folks.

Yesterday, I met up with for a while. Turns out the place he’s staying here in Dallas is just the next apartment complex over. Afterward I came home and got ready for a pool party at Amythest’s, with her sister, and other DFW Ufies. Shared that bottle of wine I bought a couple of weeks ago at the wine tasting and watched a silly movie.

So far last night / today I’ve made progress on Project X by getting Open-LDAP installed and successfully added an entry to the database. Next I get to configure Qmail to authenticate against it.

[1] Since ${poe} was too cheap to pay for a shuttle. REALLY cheap, since we were going to need a shuttle at the HOU end anyway.
[2] Got in the exit lane for the freeway interchange, which came to a complete stop. So he got out of the lane, slammed on the gas, passed everyone waiting to get on the interchange and cut right back in at the very last second.
[3] If I ever have to travel on business and the person arranging the travel forgets to PAY for the hotel again, I will hand them my two week notice. Going to check into a $300/night hotel and being asked for MY credit card was not fun. One call to the boss and he took care of it with his card, but he had to fax them both sides of his credit card and drivers license.
[4] With FUD like “All it takes is your competitor forging headers once to get you added to a whole bunch of RBLs” and “You have to pay each of them a ‘bribe’ to their pet charity to get off their list”. Guess he’s never heard of rfc-ignorant, ORDB, MAPS-RSS, MAPS-DUL, SORBS, DSBL…

I will be traveling down to Houston for a cPanel training seminar next week. I arrive Houston Hobby Wed evening, 19:30. Return Fri evening 20:30. I’ll be staying at the Westin Galleria.

Anyone down in Houston want to get together for dinner Thurs evening, or even late Wed?

Since the car wouldn’t be ready in time for work (they had to special order a 4th hose), and still isn’t, I elected to walk to work.

The good news is I can still do it. ~5 miles. At 35 and about 80lbs over weight I can still make that walk. I wasn’t even overly tired when I finished. I was, however, very sweaty. No one likes to see a sweaty fat man, least of all me.

I was also very red faced.

But I feel good. I didn’t walk fast enough to be releasing endorphins or anything, but it actually felt pretty good.

I may do that more often, at least while the weather is still cool enough to get away with it. Which means about 1 more week. Might work off some pounds that way. Even if I don’t work off any pounds, it will be good for the heart.

But I will have to remember to carry another t-shirt with me to change into something dry when I get here.
The problem is walking home at 0200.

Now, if only the weekend shift who had been so “not busy” all weekend (I read the shift reports. They claimed to be un-busy all weekend) had actually got any work done, there wouldn’t be a ticket that’s been waiting since Friday to be completed.

It took me all of 10 seconds to do what needed to be done.

It’s not every day the Chairman of the Board / CTO of your company walks up to you at your desk, puts a hand on your shoulder and tells you “You write a hell of a ticket.”

He complimented me on the way I explain what we’re doing, what has been done, what needs to be done and what we can do for a customer in my tickets. (Contrasting it with the way some other support techs document their tickets, though he didn’t use any names, I got a pretty good idea who he was talking about.)

Of course, it’s not every day the COB / CTO is down here at the data center either, so it was already an unusual day.

‘Cause, like, it’s always a good idea to log in and change your root password while NOC Support is still logged in and working on your server. Cause, like, they never log off and have to come back later.


(19:38:39) ${level 1 moron}: what is the command to look at whats mounted?
(19:39:40) ${level 1 moron}: its(sic) shows what the hard drives are mounted to
(19:40:09) Dilbert: uh... mount.
(19:40:23) ${level 1 moron}: no
(19:40:46) ${level 1 moron}: the one that tells me what hda1 is mounted to
(19:40:52) ${level 1 moron}: you know the one that list them
(19:40:54) Dilbert: uh... mount.
(19:41:12) ${level 1 moron}: uh... thanks for the sarcasim(sic)
(19:43:50) ${level 1 moron}: yopu(sic) it really doesnt make me want to learn anything
(19:44:20) Dilbert: You asked for the command, I gave it to you. It's still the same command.


I had this exact conversation (the other way around) with at least 2 other Level 2 techs and it went exactly the same way. Except they looked at me funny because I should already know the answer.

Yeesh, work has been hell lately.

Busy, busy, busy. Why I haven’t been posting much.

Not sure why it’s been so busy lately, other than we’ve added something over 500 customers in the last month. But we’ve always done about that per month.

Fridays, however, is the day that everyone is on duty. All shifts overlap on Friday. So right now we have both the full weekday and weekend 1st and 2nd shifts on duty, which means 4 team leads, 4 Support 2 techs, 6 Support 1 techs all on duty. And ticket load is lighter than usual to begin with.

So we’re actually caught up with work today.

I can play my online go game against <a href=”http://www.livejournal.com/users/_techie>Techie</a>, browse LJ, read my comics, read technical documentation, mess with Ubuntu on the laptop…

Today was, if anything, worse than yesterday.

I can sit at my desk in the NOC and do nothing but play traffic cop with tickets, in which case nothing will get done, or I can do tickets, in which case the Triage queue will fill up before I can blink.

The problem with being out on the floor doing tickets is that every Level 1 support guy can’t go 5 minutes without having to come ask me how to do something. I swear, just one ticket I was working on, every time I started to reach for the keyboard on my laptop to tell the customer what I did, someone would come to me with a problem. I’d have to spend 10 minutes helping them out, then I’d reach for my keyboard again… wash, rinse, repeat.

Not once today did I even look at the Support queue. I spent the whole shift just trying to keep up with the Support Triage queue. I only got that down to 0 after my shift was already over. I then spent another hour trying to work a ticket that hadn’t been touched in several hours and the customer was still waiting. This is two days in a row I didn’t have time to take a lunch break and was on my feet constantly for 6+ hours.

At the end of my shift emailed my boss:

I need people who know what they’re doing and don’t need to ask me what to do every 5 minutes.
I need people who come back from lunch on time.
I need people who do what I ask them to do, let me know what they’re up to, don’t give me attitude when I ask them what they’re up to.
I need people at Data Bank for the whole shift.
I need a CSR.[1]
I need people!

Did not go well.

Note to self: When they ask “From the OK prompt, how do you tell a Solaris system to ‘reconfigure’ it’s list of hardware devices”, the answer is “boot -r”. <bangs head into desk>

“You have to use rlogin for that. You can’t pipe anything through SSH. It’s designed to not allow it, for security reasons.”

“If you say so, James…”

“I’m talking as someone with 20 years experience as a Unix sys admin…”

Guess when I go back to the NOC I’ll have to take him to school.

>yawn<

Start packages installation
Install package psa
bsdtar 1.02.023, libarchive 1.02.026
Use gtar
/usr/local/bin/gtar
bsdtar 1.02.023, libarchive 1.02.026
Use gtar
/usr/local/bin/gtar
To continue installing, you should install Perl 5.008008 (you have Perl 5.008007 installed)
Execute cmd failed: sh /root/psa/PSA_8.0.0/dist-standard-FreeBSD-6.0-i386/psa_v8.0.0_build80060406.16_os_FreeBSD_6.0_i386.sh
ERROR: Error while install .sh package
ERROR: Installation failed

So…
My boss got fired today.
He was a good guy. I really liked working for him. He came up “through the ranks” as they say; he knew what it was like being a support tech, a team lead, knew what we do and how we do it. He was a good manager.
But I don’t disagree with the decision. He choose the wrong way to handle a grievance and it cost him his job. And it should have.
I’m still going to miss him. He was a good guy.

First I get sarcastic email for actually suggesting we could help another department get something done, then when I get to work today, my supervisor, “W”, has to “talk to me” at some point.
When we both get a free moment to step outside and talk, it seems someone has informed W, and the boss, “J”, of an incident where I “screamed” at someone yesterday about not doing their job and I tried to tell them what to do. Apparently I’ve been trying to boss people around, giving orders to people from other departments, etc. Funny, I must have amnesia. I have absolutely no recollection of ever having “screamed” at anybody at work, ever. And I have never attempted to give orders to anyone from any other department or otherwise tell them what to do, unless they specifically asked. The person at whom I am supposed to have screamed was even named. I haven’t had anything other than perfectly polite conversations with this person since… well since the day she was hired. Including today. We had perfectly normal, polite conversations today, with no tension or anything.
Luckily my supervisor and my boss, know me better than that and just wanted to get my side of it, which was that I had absolutely no clue what that was about.
It would appear someone is out to stab me in the back. I have no idea why.

So apparently I’ve been sucked back into the time-sink that is LiveJournal…

Wait’ll I get caught upating my journal at work.
Everybody will want my LJ name so they can read it… wait, what am I saying? The people I work with are into “mysapce.com”.
Yeah, bunch of hip-hop slackers. No worries there.

Oh yea. For those who keep score on such things, I’m employed now…