This summer I’m teaching a graduate class, Principles of UNIX, which is more or less a crash course in the Mother of All Operating Systems. One of our early topics is email on UNIX, in which I try to impart to the class just how transformative email was back in the day. For early Internet users (mostly UNIX users), this was an incredibly big deal.

Unfortunately, setting up email on a Linux or UNIX system is not quite as automatic as it once was. In our class we use mailx and sendmail as tools to send files from background processes or cron jobs – but mailx will typically not work out of the box. In this post I’ll discuss how to get it working on an Ubuntu 11 system.

Things Have changed

Back in the day if you wanted to send mail, you simply found a handy dedicated server that was accepting incoming SMTP connections. There were thousands, and they were undiscriminating, so this was no big deal.

The invention of spam ruined that.

Now any SMTP server you find is going to require you to go through a bit of a dance in order to authenticate and prove you are not a spammer. I started off this post with the intention of showing you how to use your gmail account to access Google’s SMTP servers. The process was fairly arduous, as it involved creating a certificate authority, your own certificates, and then setting up the mail server to use this authentication.

While working on this, my son Joey recommended that I just set up a free account on one of several email gateway providers, such as SendGrid or MailJet. Both services will let you access their servers and send up to 200 emails a day for free.

I took him up on it and found the process to be much simpler than using gmail, so I’ll pass along the setup procedure here.

Getting an Account

Obviously, SendGrid is in business to get you to purchase a commercial account so you can send thousands of emails a day from your web site. Accordingly, the don’t go out of their way to advertise their free plan. If you go to their pricing page, you will find a little tiny link to the free plan hidden at the bottom.

Setting up an account is easy, but SendGrid insists that you have a web site. For automatic verification they will need to find your email address on the site. I opted for an alternate provisioning plan in which I created a page on my site with the phrase “Sendgrid”.

Once you have an account, you have free access to the SendGrid SMTP servers for up to 200 outbound messages a day. So you are ready to configure your UNIX system to take advantage of it.

Ubuntu Configuration

Configuring Ubuntu 11 to send email is fairly painless. Using the Ubuntu Software Center, you can locate and install two packages: postfix and bsd-mailx. During the install of postfix, you will get dropped into a debconf window asking you some basic configuration questions:

Figure 1 – The initial configuration screen

• General configuration: Internet
• System mail name: dogma.net

That seemed to be all I needed for basic configuration.

Postfix Configuration

To configure postfix to use SendGrid was just a matter of adding a few lines to /etc/postfix/main.cf, using your SendGrid user name and password. Note that the file probably has an existing relayhost line, this one should replace it:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:username:password
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = may
relayhost = [smtp.sendgrid.net]:587

After making the changes you should restart postfix so it reads the new config options. I also start watching the mail log file so I can see if there are any problems on first use:

sudo /etc/init.d/postfix restart
sudo tail -f /var/log/mail.log

A test message sent to my cell phone arrived as a text message in just one or two seconds, with the following log messages:

Jun 26 17:02:08 ubuntu postfix/pickup[21145]: 51A5E5E1DA6: uid=1000 from=<mark>
Jun 26 17:02:08 ubuntu postfix/cleanup[21336]: 51A5E5E1DA6: message-id=<20110627000208.51A5E5E1DA6@ubuntu>
Jun 26 17:02:08 ubuntu postfix/qmgr[21146]: 51A5E5E1DA6: from=<mark@dogma.net>, size=273, nrcpt=1 (queue active)
Jun 26 17:02:08 ubuntu postfix/smtp[21338]: 51A5E5E1DA6: to=<xxxxxxxx@txt.att.net>, relay=smtp.sendgrid.net[174.36.32.204]:587, delay=0.33, delays=0.04/0.02/0.23/0.04, dsn=2.0.0, status=sent (250 Delivery in progress)
Jun 26 17:02:08 ubuntu postfix/qmgr[21146]: 51A5E5E1DA6: removed

Moving On to Better Things

Now that postfix is properly configured, I can really start taking advantage of the mail infrastructure on my system. The next obvious step is to create a .forward in my home directory, and give it my external gmail address. That external address will now be the recipient of output from cron jobs, or from at or batch. It’s nice to have the mail set up as in integral part of the O/S, and if you can just make it through a little bit of setup, it’s all yours.

With a limit of 200 messages a day you can still make extensive use of outbound email for system monitoring – whether it is via text to your phone or huge messages being sent to an account used for storing log files. Either way, integral email is still a great feature, almost forty years after it first showed up in UNIX.

For most of this year I’ve been working on a new product called Cisco OnPlus, a network management service for small business. In order to do its job effectively, OnPlus needs to know what devices are present on the network, and one of the key tools we use to accomplish this is DNS Service Discovery. In this article I will show you a little bit about how we use DNS-SD, and show you how you can put it to work effectively in your networks.

OnPlus

Cisco OnPlus is a cloud-based network management tool that helps resellers support their customers. The figure below shows you a typical view of a customer network from an OnPlus browser screen. (The customer in this case being my Dad.) OnPlus not only identifies the devices on the customer’s network, it also performs configuration backups, firmware updates, and monitors network performance.

The OnPlus Topology View

The OnPlus Network Agent

• DNS Service Discovery
• DHCP Packet Inspection
• DNS Packet Inspection
• Windows Management Instrumentation
• Cisco Discovery Protocol, or CDP
• NETBIOS/SMB
• UPnP
• SLP
• Traceroute
• ARP

When it comes to locating devices made by my business unit at Cisco, the most useful protocols are DNS Service Discovery and Cisco Discovery Protocol. DNS Service Discovery provides all the information the inventory process needs to fully identify a device: its specific Product ID (more or less a model number), the version number of both the hardware and the firmware, its MAC address, and its IP address. (CDP provides a nearly, but not identical bundle of data.) This information is readily available because devices made by Cisco’s Small Business Technology Group use DNS-SD to broadcast information using our proprietary service type: csco-sb.

A Quick Overview of DNS Service Discovery

So what exactly is DNS Service Discovery? If you’re like me, you became aware of DNS-SD because Apple uses it as part of Bonjour. Bonjour is a proprietary implementation of Zeroconf, a set of technologies marked by by three key network components:

• Address assignment
• Service discovery
• Name resolution

The history of Zeroconf is a somewhat quixotic story, based around the shared idea that setting up small networks ought to be a painless and simple process. The components of Zeroconf provide a nice, vendor-agnostic way to set up networks in such a way that no consumer would ever have to manually assign an IP address, set up a DHCP server, or manually enter the address of a printer.

Apple has embraced this idea, with their implementation of Zeroconf called Bonjour, an Apple trademark. If you run iTunes on your Apple or Windows PC, you may well see that there are other users out there running iTunes who would be happy to share their collections with you. This happens more or less with no work on your part, and can be a really nice feature in a big office:

Sharing iTunes Libraries

daap Digital Audio Access Protocol (iTunes)
     Amandeep Jawa 
     Defined TXT keys: txtvers, Version, iTSh Version, Machine ID,
                       Database ID, Machine Name, Password

This is a pretty simple definition – let’s see what it looks like on the network.

On my desktop Linux sytem, I have the avahi utilities installed. Avahi provides a nice suite of tools used to implement DNS-SD. I’ll use the avahi-browse command to see what these daap services actually look like:

mark@ubuntu:~$ avahi-browse _daap._tcp -t
+   eth0 IPv4 Itunes NAS Server on nas                      iTunes Audio Access  local
+   eth0 IPv4 Denise___s Library                            iTunes Audio Access  local
+   eth0 IPv4 Mark___s Library                              iTunes Audio Access  local
mark@ubuntu:~$

If I ask avahi-browse to resolve the services, it will query the service provider for the details in the advertisement. A partial output is shown below:

mark@ubuntu:~$ avahi-browse _daap._tcp -r -t
+   eth0 IPv4 Itunes NAS Server on nas                      iTunes Audio Access  local
+   eth0 IPv4 Denise___s Library                            iTunes Audio Access  local
+   eth0 IPv4 Mark___s Library                              iTunes Audio Access  local
=   eth0 IPv4 Itunes NAS Server on nas                      iTunes Audio Access  local
   hostname = [nas.local]
   address = [192.168.1.165]
   port = [3689]
   txt = ["ffid=075abcc4" "Password=false" "Version=196610" "iTSh Version=131073"
          "mtd-version=svn-1676" "Machine Name=Itunes NAS Server" "Machine ID=BE8926F6"
          "Database ID=BE8926F6" "txtvers=1"]

From this information, I know have everything I need in order to connect to to a server and start playing music. I have a hostname, IP address, and a port, all of which can be used to access the service. Finally I have a txt record that contains an aribtrary set of name/value pairs, as defined in the service definition. The us of these fields is up to the creator of the service, and in this case most of them are self-evident.

Browsing the Network

We use the avahi toolkit in OnPlus to browse the network for devices. It is worth doing a little exploring on my home network to see what kind of information we get out of this process.

To get a high-level view, I can ask avahi-browse to query for a special service: _services._dns-sd._udp. When this browse request goes out on the network, all the active nodes using DNS-SD issue records detailing the types of services they support. The result on my home network looks like this:

mark@ubuntu:~$ avahi-browse _services._dns-sd._udp -t
+   eth0 IPv4 _udisks-ssh                                   _tcp                 local
+   eth0 IPv4 _workstation                                  _tcp                 local
+   eth0 IPv4 _ir-hvac-021                                  _tcp                 local
+   eth0 IPv4 _ir-hvac-020                                  _tcp                 local
+   eth0 IPv4 _ir-hvac-000                                  _tcp                 local
+   eth0 IPv4 _pdl-datastream                               _tcp                 local
+   eth0 IPv4 _printer                                      _tcp                 local
+   eth0 IPv4 _tivo-videos                                  _tcp                 local
+   eth0 IPv4 _readynas                                     _tcp                 local
+   eth0 IPv4 _smb                                          _tcp                 local
+   eth0 IPv4 _afpovertcp                                   _tcp                 local
+   eth0 IPv4 _rsp                                          _tcp                 local
+   eth0 IPv4 _daap                                         _tcp                 local
+   eth0 IPv4 _http                                         _tcp                 local
+   eth0 IPv4 _csco-sb                                      _tcp                 local
mark@ubuntu:~$

As you can see, there are a surprising number of DNS-SD services present. On my network, an explanation for each of the services is:

This special command to show me the services available is not actually used in OnPlus. Instead, we call avahi-browse with the -r and -p commands, asking it to do a full resolution on all discovered services.

Cisco Devices

The place where we get the most interesting results from avahi-browse is when we tell it to look specifically for instances of the cisco-sb service. That command produces output like this:

mark@ubuntu:~$ avahi-browse -r -t _csco-sb._tcp
+   eth0 IPv4 switch32026a                                  _csco-sb._tcp        local
+   eth0 IPv4 onplus005229                                  _csco-sb._tcp        local
=   eth0 IPv4 switch32026a                                  _csco-sb._tcp        local
   hostname = [sg200-26p.local]
   address = [192.168.1.168]
   port = [80]
   txt = ["hostname=sg200-26p" "serialNo=DNI1515005U" "MACAddress=44E4D932026A"
          "PIDVID=SLM2024PT V01" "fmVersion=1.1.1.8"
          "deviceDescr=26-port Gigabit PoE Smart Switch" "deviceType=Switch"
          "model=SG 200-26P"]
=   eth0 IPv4 onplus005229                                  _csco-sb._tcp        local
   hostname = [PLG1000F0AD4E005229.local]
   address = [192.168.1.167]
   port = [80]
   txt = ["accessType=http" "MDFID=Unassigned" "hostname=onplus005229"
          "serialNo=PLGF0AD4E005229" "MACAddress=F0:AD:4E:00:52:29"
          "PIDVID=Unassigned" "fmVersion=6.2.2.007" "deviceDescr=Cisco OnPlus Network Agent"
          "deviceType=Service Appliance" "model=PLG1000" "version=1.0"]
mark@ubuntu:~$

If you look at the first ‘=’ records that is issued by avahi-browse, you can see that when it comes to discovery, I have really hit the jackpot. I’ve identified a device on my system that I can reach with a specific IP address. I have a MAC address that I can now use as a globally unique identifier. And I have the Cisco Product ID, the hardware version, and the firmware version, as well as the user-assigned host name and a friendly model name.

When I have information like this, it allows me to fill in the details in the topology map quite accurately. Better yet, since this is a Cisco device, the OnPlus appliance can now send it some queries to find out more network information. As an example, the switch’s CAM table provides me with a list of devices and the ports they are attached to, which helps me fill in some of the details of the topology picture.

Processing This Data

If you are a programmer, the natural question you might be asking is how you access these service advertisements from inside your program. In the case of Cisco OnPlus, most of the code running the inventory task consists of PHP scripts. As far as I know, there are no bindings in PHP to DNS-SD services, and we elected not to try to invent that wheel.

Instead, we use PHP’s popen() function to run instances of avahi-browse, collecting the output from the program and parsing it accordingly. We actually have three instances of the browser running at any time. Two are dedicated to Cisco-specific services, while the third looks at all other services. Even though other services might not give us as munch information as csco-sb, they still supply host names, MAC addresses, IP addresses, descriptions, and more, and we use whatever we can find.

These instances of avahi-browse run as independent discovery processes, collecting and storing data as it is seen on the network. The records that they collect are then used by the inventory process when it is periodically launched.

The DNS-SD discovery processes actually take an active role in the inventory scheme. When one of the avahi-based processes discovers a significant new device on the network, such as one advertising cisco-sb, it stores the data record and then triggers an early start to the inventory process. This allows OnPlus to respond to new devices on the network in something close to real-time.

Advertising

So let’s say you decide you want to use DNS-SD for some form of network discovery. I’ve shown you how you can you discover network nodes that are advertising a service, but how do you actually perform that advertisment?

In the OnPlus Network we use the avahi package to advertise services as well as to find them. The avahi-publish command does the job, typically started as a daemon to run for the lifetime of the system. When executing avahi-publish, you will normally want to specify:

• A service type
• The name of the service instance
• A port to access the service
• Text records containing whatever name/value pairs you want to publish

One thing to note is that you can use DNS-SD to advertise things that don’t necessarily map directly to a port. For example, if you just want to let everyone know of your existence, the port parameter might be irrelevant, but the name/value pairs could still be valuable.

As an example, I might create a photo sharing service that I advertise on the network using a service I’ll call PhotoMark. To let everyone know about this, I execute the following command when my system starts up:

avahi-publish -s mark _PhotoMark._tcp 9999 "payload=.gif" "folder=/pictures"

Anyone on the network using DNS-SD could then see that I was advertising photo sharing with photos of type GIF accessible on my pictures folder. Of course, the details of the protocol are not included in the advertisement – that’s outside the scope of DNS-SD.

Often a good choice for the service instance is to simply use the computer name:

avahi-publish -s `uname -n` _PhotoMark._tcp 9999 "payload=.gif" "folder=/pictures"

What’s Next

In this article I showed you how we use DNS-SD on the Cisco OnPlus Network Agent. Mostly it involves calling the avahi command line tools and parsing their output with PHP scripts.

In my next post, I’ll show you how to do the same thing on a Windows PC using the Apple Bonjour SDK. Unfortunately, Windows has not included DNS-SD support in the O/S, so instead of using the Win32 API to do service discovery, you will have to rely on some slightly less elegant methods. But when it comes to network discovery, functionality and interoperability trump elegance every day of the week.

In my last post I showed a particularly easy way to to set up an SMTP server so postfix could send mail from your Linux system. In this post I’m going to add a few tips that might help you get through some rough spots in the whole process.

Results From the Field

After writing a HOWTO page, there is no better test of its quality than to try it out with a group of students performing the task for the first time. That’s just what I did with my SMTP configuration post – I created a homework problem that asked my students to set up outbound email on their personal Linux systems.

Their resulting glitches highlighted a few rough spots in the process, most of which can be handled pretty easily.

Debian Package Configuration

When you install postfix on Ubuntu, you will normally get kicked into a simple package configuration dialog after the package has been downloaded and installed. This final step performs some very basic setup on postfix, getting into a roughly functional state.

If this configuration doesn’t run, postfix seems to be configured to only handle local mail – messages from one user to another. In some cases this may be just what you want, but normally it is not all that useful.

For unknown reasons, this configuration pass doesn’t always kick off properly. I’ve seen it fail to run, and some students have as well. Fortunately, you can execute the same process by hand from the command line:

sudo dpkg-reconfigure postfix

Running through this, I take the default settings for everything except Root and postmaster mail recipient, which I change to student, the name of my account on this system. After this is complete, following the instructions from my previous post will give you a proper Internet email setup.

Using Mailjet instead of SendGrid

After some glitches getting accounts on SendGrid, I had a couple of students decide to use Mailjet as an STMP provider. Registering with Mailjet is easier, as it an automated process. However, using Mailjet as your SMTP provider comes with a couple of interesting twists.

First, instead of using your Mailjet account credentials to authenticate with their SMTP server, Mailjet gives you a set of strong credentials specifically for SMTP authentication. You have to copy these from your Account Info page. No problem there, as long as you RTFM.

Slight more problematic is that Mailjet insists that the return address you use in your outbound emails must be an authenticated address. In other words, you have to be able to receive an email sent to that address and click on an authentication link.

When using postfix’s default settings, email sent by my students was going out with a return address of student@ubuntu, and there is obviously no way to authenticate that address. So we have to rewrite that address to something usable.

After taking that additional requirement into account, my changes to /etc/postfix/main.cf for Mailjet are shown below. The file had a default relayhost entry, which is replaced, and the remaining lines are all new. The username and password have to be replaced with the ones from your Mailjet account page:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:username:password
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = may
relayhost = [in.mailjet.com]:587
smtp_generic_maps=texthash:/etc/postfix/generic

The last line I added to the configuration tells postfix to look in a file called /etc/postfix/generic for a list of substitutions to make to email addresses found in headers being processed. I create a single line in this file:

student@ubuntu yourmail@yourprovider.tld

This entry in the map file changes the local student account to an authenticated email address. After making this change, emails sent from the student account will have their headers rewritten to give a sender identity of your personal email address, which means Mailjet will happily forward your messages to their final destination.

Once that’s done, I restart postfix and send a test message:

sudo /etc/init.d/postfix restart
echo hello | mailx -s "test mail" myemailaddress

The headers on my received email show that things worked more or less as expected:

One nice thing about using the generic substitution file is that now you can accommodate different sender email addresses for every account on your system. Since you have to enter account info by hand, this doesn’t scale to a large enterprise, but it works just fine for your personal system.

There’s no doubt that the complexities of managing mail on a Linux system is still a bit daunting, but at least the process of simply getting outbound mail working is manageable. Give it a shot!