Friday, June 23, 2017

Progress Bar For dd(1)

The dd(1) command is frequently used to both make and write out image files, such as .iso files for Linux releases.  It's mostly intended for this and works fine, but a common complaint from users is the lack of a progress indicator.  It's true:  dd is silent until it's complete.  But you can get a progress bar with the pv(1) command.

pv is short for Pipe Viewer.  You can get it over here or install it from your distribution's packaging system.  For example, Fedora users can do dnf install pv.

Once you have pv installed, you can start getting progress output for console commands.  The man page has some good examples, including one for tar(1).  The man page also suggests using pv by itself as a replacement for dd entirely, but I have had limited success with that.  Here's what I do:
dd if=ISOFILE | pv -p -e -a -s BYTES | dd of=DEVICE bs=4096
OK, that's a lot, but it's not that bad.  dd reads from stdin when not told otherwise.  Likewise, it writes to stdout when not told otherwise.  In this example, we connect two invocations of dd using pv, which also has the same behavior.  By not specifying either on pv, it reads from stdin and writes to stdout.

The ISOFILE is the name of the image file I want to write, such as release-3.2.0-ppc64-20170623.iso.  BYTES is the size of the image file in bytes.  This is obtained any number of ways, such as the long listing from ls(1).  DEVICE is the device node you want to write the image to, such as /dev/sdh.

You could extend this and have the command determine the size of the ISOFILE directly, such as:
dd if=ISOFILE | pv -p -e -a -s $(stat -c %s ISOFILE) | dd of=DEVICE bs=4096
Now this is starting to look like something you could put in to a helper script in your own environment.  Just another tool for your command line adventures.

Sunday, June 18, 2017

Did Someone Say Trains?

No?  Well, I guess I did.  My new commute has me taking three different lines on our transit system to get to work.  It reminds me back to the early 2000s where I would constantly give friend after friend updates on the fact that MARTA is single tracking for the day.  Like anyone cares.

Last week's heat wave caused some hiccups on the MBTA.  The Internet fortunately gives us a venue for all of the experts to offer up comments on what is wrong with the system when anything out of the ordinary occurs.  So what happened that I noticed?
  • Green Line power surge near Kenmore Square.  Since it was a heat wave, my unscientific guess is that people were firing up air conditioners, which impacted the grid.  I only say this because it's happened before both here and in New York.  My intern was affected by this on his first day in the office, but it wasn't really a major thing.
  • Medical emergency on the Red Line.  Well, these tend to happen daily, but I think on the heat wave days there were more.
  • Trash fire on the Red Line.  This is the best one I heard about but still not better than the time I heard on the Green Line radio an operator calling dispatch to report a huge turtle on the tracks (different day, different year).
  • Slow commuter rail.  Likely a heat order from dispatch, but I am guessing.  In extremely hot weather, continuous welded rail can buckle or kink which can lead to a derailment.  This has happened on multiple occasions for both passenger and freight train.  A heat order [usually] requires passenger trains to go 10 or 20 mph below the maximum allowed speed.  The slower speed is meant to help engineers spot rail problems and report and avoid them.

I was only involved in one issue last week, not noted above.  I was going to catch the 5:50 pm outbound train from North Station.  We boarded and then at 5:55 pm, a conductor told everyone the train was out of service because the locomotive wouldn't move (breakdown!).  So we all had to detrain and retrain (crosstrain?).  That took about half an hour.  As we left the platform, we were not in time to beat the Amtrak Downeaster so we had to stop for it to pass.  We continued after it passed, but were traveling at medium speed.

People were very angry about the entire thing.  And yeah, I get it.  Karen was waiting for me at Bistro 5 and our plans changed up, but whatever.

Tuesday, May 30, 2017

Ineffective Meetings

I've tried posting this a few times from work but because we have Google Drive and Gmail and Google Calendar and all sorts of other googley bits all up in our network, it's impossible for it to believe that I may not want to use a particular Google service as my non-work self.  So here I am posting this from home.  Maybe I should return my blog to software on my own server rather than using Blogger.  Eh, a topic for another day.

Ineffective meetings have become a part of my daily work routine.  I'm not really sure when it happened but that doesn't really matter much now.  I've been with Red Hat now for almost 12 years.  In that time we have acquired many companies, released many versions of Red Hat Enterprise Linux, and undergone a huge growth in headcount that has ballooned all existing departments.  With this can come some institutional pain, but that's not strictly a bad thing.  In this post I want to concentrate on just meetings.

I recently started logging my weekly meeting attendance.  I log the date of the meeting, the starting time, the duration in minutes, how many minutes I spend contributing in a meaningful way (i.e., statements other than "yeah, sounds good"), whether or not the meeting has an agenda, and whether or not the meeting ends with any actionable things.  I should have also been logging the number of people in the meeting as well because that's a key indicator as to whether or not a meeting will be a giant waste of your time.

Why have I've been logging my meeting activity?  Well, I feel like I'm always catching up on my work.  Where has my time gone?  I feel like the majority of my useful working hours have been sucked up by ineffective meetings.  It's so bad that I generally dial in to a meeting from my desk at the office just so I can do other work and not be so rude in the meeting itself (also, I've been doing that for years anyway).

In the last week I spent 841 minutes in meetings.  That's 14 hours, or 35% of the scheduled work week.  And for what?  I couldn't tell you because it's just an information overload.  Many of the meetings repeat themselves, with the same people, and lack a clear goal.  Here are my observations:

  1. Duration.  Meetings at Red Hat, at least in engineering, are usually scheduled for 1 hour.  Big important meetings get 1.5 hours.  Seldom do you see 30 minute or 45 minute meetings.  One hour is a lot of time to fill, especially when you have no agenda.  My thought is that meetings should start with a short time limit and grow only when necessary.  Beyond 1 hour and you should really break it down in to multiple meetings -or- figure out why you need so much time.  Maybe what you are intending to cover doesn't need to happen in a meeting.

  2. A meeting should be 30 or 45 minutes, tops.  All hands meetings and other types of department wide meetings can be exceptions, likewise big quarterly or planning type meetings can be exceptions.  The point is, a regularly scheduled meeting in the day to day business of doing your work should not be long.

  3. Guests.  The majority of my meetings have too many people invited.  In one meeting there were 66 people on the call.  Sixty-six!  Only two people really talked.  Do we all need to be there?  If you have no reason for me to be there, DON'T INVITE ME!  If you think I might possibly maybe be tangentially interested in the work you are doing, send me the minutes.  If I am interested, you'll hear from me.

  4. The more people you invite to a meeting, the less likely you'll accomplish anything of value.  Have you ever tried ordering a pizza with a group of people?  It's difficult, right?  Everyone wants different types of toppings, or are allergic to some, or have food sensitivities to others.  But what happens when the 1 or 2 pizzas show up?  People eat other types that they didn't order, right?  Was it really necessary to involve everyone in the decision or could you have just ordered one large pepperoni and one large veggie and saved yourself some time?

  5. Agenda.  This one surprises me a lot.  People scheduling a meeting with no agenda.  Why are we here?  Even still, I go to meetings where the person who called the meeting immediately defers to someone else to get things going.  What?  Why are we doing this?  An agenda is short and states why we're holding a meeting.  It should also tell me why I'm being invited.  If I get an invite with no agenda, I will decline it.  Tell me why you are holding a meeting and what you expect from me.  Do you need my input?  Are you assigning work?

  6. Actions.  I really dislike the term action item, but it's relevant for meetings.  There should be a reason we are holding a meeting.  You are either asking me for information or asking me to do something.  Either way, we should come away from the meeting with new information related to the meeting topic or a list of things that need doing and who is doing them.

  7. End on Time.  Better yet, end early!  Many times we actually get through a meeting early, but people run out the clock with filler.  Why?!  End it early and give me back my time.  We can't come up with anything useful for 13 minutes of everyone's time, so let us all just go back to work.
I personally think a lot of people just like to fill up their schedule with meetings.  It certainly looks like you're working.  Crowded meetings also look really busy.  Sixty-six people on the call, wow that must be super important.

If you are going to schedule a meeting, ask yourself these questions:
  • Do I have an agenda?
  • Do I know who should be invited?
  • Does my agenda fit in 30 or 45 minutes?
  • Do I have a set of actions for the meeting?
If you answered no to any of them, take it to email.  So many of my meetings are just presentations.  Dump it on email and make me responsible for reading it in my own time.  I can also read that information and determine what, if anything, relates to my team and individuals on my team.

This only works when everyone is on board.  People feel guilty about declining a meeting.  Or if their boss goes to a meeting, they think they should go to.  While this may be true in some cases, it's not always the case.  Your time is worth something to the company and sitting in meetings doing nothing is probably not what they had in mind.  Make a case for your time being valuable.  Reclaim your work week so you don't feel like you are always behind and having to catch up.

Thursday, January 5, 2017

Web Browsers

After some recent annoyances using Firefox at work and seeing the memory in use by Firefox at home to almost 10GB, I have a renewed interest in looking at alternative web browsers.

Web browsers are not optional.  It accounts for nearly everything I do on the computer.  So it needs to be reliable, usable, and secure.  I have more or less always been a Netscape user and was excited when they open sourced the code in 1998.  Over time we saw Mozilla as a continuation of the (a?) Netscape code base and then eventually Firefox as a nice open source web browser.

But it's so slow.  Even on my dual Xeon, it's noticeably slow.  And the popularity of Firefox means it's a huge attack candidate, so we always see security updates for it.  This isn't Firefox's fault, it's a side effect of having popular software.

So anyways, back to browser alternatives.  Almost everyone has told me "Chrome!".  But that's going in the wrong direction for me, both in terms of the complexity and size of the browser as well as the walled garden of Googleworld.  Safari isn't an option for me because I don't use OS X.  Neither is Internet Explorer because I don't use Windows.  So I'm stuck looking for open source alternatives to Firefox.

I found that a lot of alternative browsers in the open source world are still going.  Dillo, for instance.  Even links, elinks, and lynx get mentioned from time to time.  But I have to have something that can work with modern web sites and not frustrate me.  For example,

WebKit really did make it possible for a lot of new browser projects to start up.  The rendering engine that began in the KDE project now supports a lot of big browser projects.  Looking through the alternatives available, I've narrowed it down to a handful that I want to try:
  • Midori
  • xombrero
  • surf
  • uzbl
Of these, I think that Midori or xombrero will likely be the ones that I actually find usable.  The last two are really out of curiosity.  The surf browser comes from the project.  That project aims to make alternatives to everyday software that, in their definition, does not suck.  Hit or miss, from what I can tell.  Sometimes the alternative offered is just too simple.  Sometimes it's great.  Where I think surf will irritate me is the lack of graphical controls, like being able to click a link.  It's all keyboard oriented.  uzbl appears to be similar, but I still want to look at both of these projects.

UPDATE:  This post has been sitting as a draft on my blog for maybe six months.  In that time I have tried to get WebKit going so I can build Midori or xombrero.  I have not been successful.  I am not running a mainstream Linux distribution so this is not a matter of just installing the distribution-provided packages.  Even if it were, I would likely struggle through building it from source because I prefer to understand at least a little bit about how the browser is put together.

It has been long enough, so maybe it's worth another try.  WebKit was extremely difficult to build, at least when I was passively working on it.  I could probably sit down and figure it out over a weekend but I don't really want to spend that much time on it.  However, recent releases of Firefox have left me thinking it might be worth the time.

Saturday, December 3, 2016

Windows Is User Hostile

I do not understand how people can tolerate Windows computers on a daily or semi-regular basis.  The frequent reasons usually boil down to either (a) well, it's just what I know or (b) I have to use Windows for the software I need.  Of these, option (b) is only partially believable to me.  For the vast majority of non-business users, Windows is no better or worse than a Linux or Mac desktop.

First, the familiarity.  People say they just know Windows.  Really?  It's changed a hell of a lot from release to release, much like Linux and Mac releases.  Is it really that familiar to you or do you just know realize you are adapting to a new interface?

Second, the software.  People say the programs they need are only on Windows.  I call bullshit on that.  Both Linux and Mac systems offer very good alternatives (and sometimes even better alternatives) to whatever shit you're wanting to run on Windows.  And then there's even things like Google Drive where you don't even need to install things like office software because it runs right in the damn web browser.  For nearly all home users, this is more than sufficient.

There is a Windows system currently in my office.  It belonged to my mother in law.  In an effort to preserve the data (e.g., photos), I offered to help extract those files from this system so we could then either sell or donate the computer.  What I thought would be a relatively simple task turned out to be basically impossible.

Let's Get The Files
Boot up and log in.  Seems mostly familiar.  The system is running Windows 7 Home Premium.  It does not look familiar to the Windows that I was last familiar with (Windows NT 4.0), but I am able to figure some things out.  But not everything.  How do I get a listing of what's on the disk?  Can I see a file manager?  How about a command prompt?  I find a thing on the menu called Computer that lets me navigate the C: drive, but I don't really know what I'm looking for.  I could really do this faster at a command prompt.  I remember being able to type cmd in the Run dialog box and it ran a DOS prompt.  That still works.  But turns out I don't really remember my DOS and am clumsily moving around the hard disk.  OK, this is a waste of time.

How About a Shell I Know
I download and install Cygwin and launch its bash shell.  I can now access C: via /cygdrive/c and things start looking familiar.  I think now I can just tar up some key directories, scp them to a real computer, and then unpack those archives and pull out photos and so on.  I install tar and get to it.  Oh, except I start getting permission denied errors from tar despite the fact that I am logged in via an Administrator account.  What is this shit?  Tar just won't dive in to these directories.

OK, What About a Drive Imaging Program
I know these exist in the Windows world.  How about I image the entire C: drive and we can replicate it somewhere else if we need to?  I'm now getting desperate and reducing the data extraction exercise to just moving bits over to another computer and not knowing if I will be able to do anything with them.  I find and install DriveImage.  It lets me select C: and then makes a ton of files which are image chunks.  I scp these to my real computer and then that's it.  Not really sure what to do now.  This is a non-starter.

We go on vacation.

Right, Back From Vacation
Still need to get these files off the system.  But you know, I'm not really sure what files are of interest.  It would be best to save everything just in case.  In fact, it would be best to be able to have Karen go back and look through this computer if need be.  Ah ha!  I can do a physical to virt conversion, or p2v, and set this system up in a virtual environment on her laptop.  Perfect.

Pick Your Poison
All of our computers run either Linux or Apple ProDOS.  Windows is alien.  So I need to find a virtual machine system that works on her laptop.  Oh wait, it's running CentOS and we include virt technology in that.  Perfect.  Except converting a physical system to something that runs in our KVM environment is sort of a bitch and half.  I'm feeling lazy, what else can I find?

VMware is an option, as is VirtualBox.  Microsoft offers it via Hyper-V, except Hyper-V only runs on Windows.  I look at VirtualBox but they are as complicated as our procedure for doing p2v.  VMware offers a nice method to convert a local system to virt.  So I download that tool and proceed to create the .vmx file.

Error: A General Error Occurred
VMware is terrible with error messages.  The VMware vSphere Converter tool that will let you create images from physical installations is ok, except that it's also terrible.  I launch it and it immediately tells me that a general error has occurred.  Um, ok?  Google takes me to the VMware kbase where it explains that the error is because I didn't run it in Administrator mode.  What?  I am the Administrator.  But in Windows, you can right click an icon and select Run or Run As Administrator.   Seems that even being logged is as the Administrator has a default mode of dropping privileges when running programs.  From a security standpoint, that's probably better for users.  However, it's not what I was expecting nor did the error message help me at all.

I select a place to store the image and tell it to image the boot drive and C:.  I then tell it to make C: as small as possible, which gets it down to 115G.  I set some other parameters and let it run.  It runs for hours and then fails.  Awesome.  It actually wrote out the image file, but still insists that it's a failure.  Why?

One Drive Per Image, Please
VMware can't easily handle more than one disk in the image.  Or this program can't.  Whatever, I don't care.  I reran it selecting just C: and it ran fine.  The other disk was the EFI boot partition, so I don't really care.  The VMware kbase tells me that the image won't be bootable and I will need to boot from the install media and repair the image.  Ugh, what the hell?  This is entirely too annoying.  But I now have an image.  Time to get it on the laptop and get it working there.

I Ransomwared Myself
Remember I said I was feeling lazy?  Well, that's a mistake I had to pay for.  VMware Workstation costs money.  Should have seen that coming.  I had to pay $200 for it.  I got that installed on the laptop and proceeded to copy over the image I made, except...

Low Disk Space Warning
That 115G disk image of the Windows computer sure is big.  I have the / volume and /home on the laptop.  The / volume has 43G free and /home has 31G free.  Hmmmm....this is bad.  Maybe I can shrink / and expand /home and also get the Windows image under 70G, right?  Totally doable.

Several problems.  I have no idea what I'm doing on Windows.  I start uninstalling programs that seem pointless.  I remove files from C:\WINDOWS\TEMP.  I empty the Recycle Bin.  I try to free up space, but I can only get it down to about 101G.  I go back to the laptop.  Resizing there is going to be a pain.  I took defaults when I set up CentOS on the laptop.  The volumes are within an LVM volume group and are XFS filesystems.  You can't shrink XFS.  You can only grow it.  DAMN IT!

But we do have xfsdump and xfsrestore to make the shrink process possible, but annoying.  Maybe I can run to Best Buy or Staples and pick up a new hard disk, migrate this laptop to a larger disk, then copy over the vmx image, get that set up and running.  Yeah, that should only take the entire weekend.

That's it.  No more.  I am going to have Karen and her brother comb through the Windows system, extract the files they want, then we will wipe and reinstall and get rid of it.  What I thought would be relatively straightforward turned out to be the most frustrating process ever.  I blame a lot of this on Windows because I really just can't figure out how to do things on Windows.  It's really no different than other systems in that regard.  If I knew what to go and find, this could have been done already.  The best option I could come up with was going p2v and setting it up to run in virt on a Linux system.  And even that turned out to be impossible with my resources.

If you need me I'll be on my Apple IIgs.

Tuesday, June 28, 2016

Upgrading to Fedora 24

Fedora 24 is now available and today I took time to upgrade my workstation to this release.  I have to admit that I was a little lazy and did not bother to upgrade to Fedora 23, so today's task was upgrading a Fedora 22 system to Fedora 24.

I have been working at Red Hat for a long time and have been part of the Fedora Project during the entire time.  A lot of work goes in to each Fedora release and I have to say that this release is very nice and was very easy to upgrade to.  Many non-Fedora users I talk to have the impression that Fedora changes way too quickly for easy upgrades to be possible.  Well, that's probably true when you compare it to more conservative distributions.  Rather than slow things down, Fedora has built a reliable mechanism in for upgrades.  It's taken a while to evolve and I am pleased that my team has contributed a large part of that work over the years.

So, going from Fedora 22 to Fedora 24, here's all I did:
  • Back up.  Seriously.  Everyone says this, most don't do it.
  • Log in on another system and get Firefox open and irc up.  The upgrade is going to take a while and I want to be online while that happens.  Red Hat is very nice and provides me with multiple computers at my desk.
  • Choose your upgrade method.  For me, I chose the dnf system upgrade option.  This is a continuation/evolution of what used to be called fedup in previous releases (and in a way the older yum distro-sync mechanism).  I also chose to upgrade first to Fedora 23, then to Fedora 24.  While an upgrade directly to Fedora 24 would probably have worked for me, I wanted to step it up through the releases because I know that path is actually tested by other Fedora Project members.
  • Execute the upgrade to Fedora 23:
  • sudo dnf update --refresh
    sudo dnf install dnf-plugin-system-upgrade
    sudo dnf system-upgrade download --refresh --releasever=23
    sudo dnf system-upgrade reboot
  • During the reboot the system will upgrade to Fedora 23 and then reboot again.
  • Execute the upgrade to Fedora 24:
  • sudo dnf update --refresh
    sudo dnf system-upgrade download --refresh --releasever=24
    sudo dnf system-upgrade reboot
  • And during this reboot, the system will upgrade to Fedora 24 and reboot again.
That's the process.  It involves more waiting than commands.  And I got my information from here:

As with all systems, there are a handful of adjustments I had to make or tips I collected during the upgrade.  Here's how I modified the steps above:
  • During the upgrade to Fedora 24, I had to add --allowerasing to the dnf command due to some problems with third party repositories.  This did not have a negative impact on the upgrade itself, though some third party packages I had installed were removed (e.g., vlc).
  • Once in Fedora 24, I had to disable the rpmfusion-nonfree-updates repository.  I guess as of now, it does not yet exist.
  • There are some optional commands explained on the wiki page above that are generally good for housekeeping purposes, but I did not find that any of them were required.
I am using GNOME 3 on my laptop and MATE on my workstation with Fedora 24.  Why?  Well, I like being aware of the different desktop options available in our current release and maybe next week I'll move to KDE or LXDE or something like that.  I do try to give each one a week or so of me using it so I can do a variety of everyday tasks in that environment.

And Fedora 25 development is just getting started now, so if you are interested in trying Fedora 24 you should do that now because it won't be current forever.

Sunday, May 22, 2016

Simple Sawhorse Plans

These are plans I put together for some simple sawhorses.  It's really a collection of different ideas I saw online and then slightly adapted.  The finished product:


Parts per sawhorse:
  • 246 inches of 2x4 lumber, which is approximately 2.56 eight foot boards, so buy three boards.  I bought Irving premium SPF at Home Depot.  Fewer knots, doesn't smell, cheap.
  • 24 deck screws of length 3".  I bought Deckmate #9 x 3" screws at Home Depot.
  • 8 deck screws of length 2".  I bought Deckmate #8 x 2" screws at Home Depot.
Cost per sawhorse:
  • $12 for lumber and screws
  • Compound mitre saw or your favorite way to make 15 degree cross cuts.
  • Cordless impact driver or a drill and lots of patience with a screwdriver.
  • Speed square
  • Pencil
  • Tape measure
Steps for the 1st sawhorse:
  1. The first thing to measure and cut are the three pieces that form the I-beam.  Measure 32", scribe, cross cut at 90°.  Use this board as a template to mark the other two you need.
  2. Assemble the three 32" boards in to an I-beam as you see in the picture.  This is simple.  There should be roughly 1" on either side of the center board.  Use the 2" deck screws to attach the top and bottom boards to the center.  I put 3 screws across the top and 5 screws across the bottom underside, all more or less evenly spaced.
  3. Now measure and cut the legs.  Legs are 29.5" long.  The foot is a 15° cut as you can see in the picture.  Make sure you measure the 29.5" on the longest side of the board.  Cut four legs all the same way.  You can also go ahead and cut four more for the second sawhorse now too.
  4. Attach each leg by pushing the 90° end up in to the I-beam corner as you see in the picture.  Use four 3" deck screws to attach each leg.  Be sure the angle cut foot is going in the correct direction before you attach each leg.
  5. Measure and cut the braces.  These are 16" long with a 15° cut on each end.  It's a trapezoid.  Measure the 16" across the longest side of this trapezoid.
  6. Attach the leg braces as shown in the picture.  Attach them to the OUTSIDE of the sawhorse legs.  Use four 3" deck screws for each brace.
Steps for the 2nd sawhorse:

It's the same as the steps for the first sawhorse with the following changes:
  • Make the I-beam 31.5" long (or 31....just shorter than 32).
  • Attach the leg braces on the INSIDE of the sawhorse legs.
When completed, you can stack the 1st sawhorse on top of the 2nd sawhorse.

I wrote all this from memory, plus what I had jotted down on an index card.  I may have left out a step.  If you do not use an impact drive to run the screws in, you'll need to predrill and take your time.