Linux: Read / dump contents of DMB file created with httxt2dbm

Berkley DB (DBM) files seem easy to create, but hard to read/manage. The most common task I can think of wanting to do on a database file is to dump or display it’s contents. And there’s no easy or straight forward way to do this (at least not that I found).

We were creating DBM files (from text files) for use by Apache/mod_rewrite using the httxt2dbm utility. If you create “file.dbm” it actually creates “file.dbm.pag” and “file.dbm.dir”.

To output the contents of the DBM file(s), Perl comes to the rescue. A colleague of mine wrote this after we searched the net for a working script and failed to find one. I’m sharing it in hopes someone will find it useful.

#!/usr/bin/perl
#- dumpdbm.pl
# /path/to/dumpdbm.pl /path/to/file.dbm
#
use SDBM_File;
use strict;

my ($name) = @ARGV;
die "Missing DBM name.\n" unless $name;
my %map;
tie( %map, 'SDBM_File', $name, 'O_RDONLY', 0644 ) || die "Can't open DBM file\n"; for my $k (sort(keys %map)) {
  # value is terminated by a null character, but the hash value contains cruft after that
  # so cut off the value at the first \0
  my @v = split("\0",$map{$k});
  print $k, "\t", $v[0], "\n";
}
untie %map;
exit;

FTP files from Unix to Windows and get double lines

Sometimes, but not always, you’ll download a file from a Unix server to your Windows workstation, only to find that all line breaks have been doubled! This consistently happens to me when downloading WordPress theme files that have been edited using the WordPress theme editor (thus modified by the server).

The FTP client can have its default transfer type set to ASCII, Binary, or Auto. The problem seems to be resolved by setting it to Binary. If that doesn’t work, try ASCII.

But maybe the damage has already been done. Maybe it’s been downloaded and uploaded multiple times, so now the extra lines are part of the real file. Many IDEs have a tool to apply source formatting. For example in Dreamweaver, Commands->Apply Source formatting.

There are many ways/tools to remove blank lines. A quick and easy online tool may be just what you need:
http://textmechanic.com/Remove-Empty-Lines.html

Big sites NOT using ‘www’ in their domain

This is a list of major websites I’ve encountered that don’t use ‘www’ in their primary URL. The criteria of the list is: the site must use the root domain, actively redirect all ‘www’ requests to the root domain, and have a Google page rank greater than 6.

http://twitter.com/
http://slashdot.org/
http://digg.com/
http://thesaurus.com/
http://nymag.com/
http://mashable.com/
http://allthingsd.com/
http://bigthink.com/
http://wordpress.org/
http://wordpress.com/
http://bbpress.org/
http://drupal.org/
http://jquery.com/
http://portableapps.com/
http://daringfireball.net/
http://stackoverflow.com/
http://meyerweb.com/
http://github.com/
http://js-kit.com/
http://clearleft.com/
http://tinyurl.com/
http://bit.ly/
http://tr.im/

And all the Gawker Media sites (running on the same platform):
http://gawker.com/
http://deadspin.com/
http://kotaku.com/
http://jezebel.com/
http://io9.com/
http://jalopnik.com/
http://gizmodo.com/
http://lifehacker.com/

CNAME

The only GOOD reason I’ve seen for being hesitant about not using ‘www’ is that you CANNOT create a CNAME record for the root domain. CNAME records are only valid when there is no other DNS for that domain, and the root domain always has other records.

As the root of a domain must have an SOA and NS records the rule above kicks in, preventing use of CNAMEs too.
http://serverfault.com/questions/170194/why-cant-a-domains-root-be-a-cname

Some Content Distribution Networks (CDN), like Akamai and Cotendo, use CNAME records as the method for one to hand over their website to the caching network. If you wanted to strictly use the root domain (no www), Akamai will tell you that you can’t due to the inability to create a CNAME record. CNAMEs are generally useful things.

There are a few ways around this:

  1. Work out a special deal with the CDN to not rely on CNAME records (If Digg is really a Cotendo customer, it’s proof it can be done)
  2. Use a different CDN that doesn’t require CNAME records (Anyone know any? Level 3 perhaps?)
  3. Use something else. Perhaps a high power/bandwidth super intelligent load balancer.

If anyone stumbles upon this article and has details on how any of these sites are able to scale while using their root domain, please share! My work-arounds list needs some work.

WordPress Plugin: Import Blogroll With Categories

I just wrote my first official WordPress plugin and have it listed in the WordPress plugin repository! Import Blogroll With Categories

By default, WordPress is lacking the functionality to handle categorized OPML feeds of links. When importing blogroll/links you are FORCED to select ONE category for ALL links to be inserted into.

My plugin adds an importer to WordPress, which is actually a modification of the standard OPML importer shipped with WP 2.7. The improved importer actually handles categories.

If the link categories already exist in your blog, links from the import within matching (case-insensitive) category names will be inserted into the correct category.

You can specify one of two options for the case where your blog doesn’t already have some categories that are found in the feed.

1) Allow it to create any new link categories needed (and insert the links into them)
2) Specify a default category that the links from unknown categories will be inserted into

I got jump start on some the coding changes a while ago here: http://www.dyers.org/blog/archives/2008/02/27/how-to-hack-the-wordpress-blogroll-importer-to-recognize-categories-from-your-feed-reader/

Until YESTERDAY there has been an item in the WordPress Trac that indicated that the milestone for its implementation in WordPress was version 2.8. I believe it would have nullified the reason for my plugin to exist. However, apparently, someone just changed the milestone from 2.8 to “Future Release (no date set)”.

So it looks like my plugin will still be useful for a long time =)

I’ve documented the steps of creating a plugin and getting it into the WordPress plugin repository here: Writing a WordPress Plugin

Writing a WordPress Plugin

I just wrote my first official WordPress plugin and have it listed in the WordPress plugin repository! Import Blogroll With Categories

I discuss the plugin in a separate post: WordPress Plugin: Import Blogroll With Categories

Since this was my first official WordPress plugin, I wanted to document the procedure. Writing a plugin and going all the way with it, is not for the faint-of-heart.

First, you actually have to come up with an idea that:
1) No one has written a plugin for yet (search for it!)
2) Is doable without donating years of your life
3) Functionality you really want!

Then you actually have to create the plugin. You’ll probably have to use multiple hooks called actions and filters. I wish I could tell you there was some easy way to learn exactly which you use and how to use them, but it seems to be something you mostly have to figure out on your own. Studying the code of other plugins you like will probably help.

In PHP you can’t override user defined functions, but you CAN override all the WordPress functions defined in: wp-includes/pluggable.php. WordPress only defines the default functions if plugin defined versions don’t exist.

Various sites provide a comprehensive list of WordPress functions, classes, and variables and indicates within which files they are used and defined. WordPress provides a decent one.

Many editors (including Dreamweaver) include a method to search over all files within a directory. The feature goes by many names and is not always easy to find, but is a life-saver for tracking down functions/code.

The general WordPress plugin creation guide: http://codex.wordpress.org/Writing_a_Plugin

After your plugin has been created, relatively well tested, and refined, a whole different journey begins (if you want your plugin in the WordPress plugin repository).

I chose to put my plugin on wordpress.org because I think it is best place for WP plugins to reside. It makes the most sense to users. WP is integrated with the automatic plugin installer/updater. It serves are one solid place to look for plugins that are all documented in the same fashion. If you are serious about your plugin, I recommend you follow the required steps to get it up there.

You need to submit a request to add your plugin. The request has to approved by a person. I had to wait 2 days to get my request approved. When you make the request, make sure you use the plugin name you want to use, and a solid description.

When approved, you will get access to a SVN (Subversion) directory just for your plugin. Like this: http://svn.wp-plugins.org/name-of-your-plugin/

What they don’t clearly explain, is how to actually upload your files. The only page they refer you to is: http://wordpress.org/extend/plugins/about/svn/

Which is SOMEWHAT useful, but it is all in command-line. They make no reference to Subversion clients. When it comes to file transfer, I would prefer a nice GUI. For comparison, FTP is also a command-line based tool but we all use FTP clients don’t we?

After studying this chart of Subversion clients, I decided to try out SmartSVN (free)

I’ll admit it’s a bit overkill for updating a single plugin, but maybe I’ll use it for other SVN connections in the future. Once I got the authentication figured out (wordpress.org login), I had no problems using the program. It worked pretty much just like I expected. Easy to see if a file has been changed locally or remotely, and update the files.

SmartSVN has a ton of features, and I only used the basic stuff, so if I start using it more, perhaps I’ll write a post reviewing its features.

After uploading files (use the general guidelines here and here), you can expect to wait up to 15 minutes to see your changes reflected on http://wordpress.org/extend/plugins/

BEFORE you upload anything you’ll want to do the most important thing. To create a VERY specifically formatted readme.txt file. This readme.txt file is used to CREATE your plugins page at: http://wordpress.org/extend/plugins/your-plugin-name/

You don’t have special admin access to update that page. The page is entirely controlled by the contents of your SVN directory (especially the readme.txt).

Because the readme.txt file is so important, WordPress provided a couple nice resources. An example readme.txt file containing notes about how to format your text. And a readme.txt validator. The validator is sweet. Not only does it validate your file, but it gives you a PREVIEW of the generated HTML formatting. This preview will help you work out formatting issues before you upload anything to your SVN folder.

Remember, your plugin will be live (in about 15 minutes) after you first upload your plugin file and valid readme.txt file. So try to already have a complete and properly formatted readme.txt file before you upload anything.