Nessus is the security scanner. That’s not just a tagline, it’s the truth. Yes, other people make scanners. Yes, other companies make tons of money off of their scanners, but having used ALL of them (yes, every single purported ’security scanner’, that I have ever heard of, for the last 10 years or so. ALL), Nessus is the “best”.

Historically, one of its major values was its cost (free) as well as its source license (Free/Open). Cost is still, generally, free (restrictions and various non-free necessities apply) … but the source, and the product, are… not. This may upset some to the point of refusing to use it. I’m not one of them.

Nessus is very light-weight, its rule language (NASL) is very intuitive and powerful, and the sheer volume of support and flexibility provided is exceptional. It finds things other scanners only dream to. It can (not by default) work very stealthily, not setting off some IDSes. It can be configured not to destroy the systems it’s scanning. No one builds a better scanner. Truthfully, I wish someone would: Not because I don’t like or want to use Nessus, but because the ecosystem is very homogenous in this space: You either use Nessus, or you may as well find a dowser with a divining rod to point out your vulnerabilities.

Over the years, one of the things that has been failing with Nessus is the interface. I enjoy command-line interfaces as much as the next UNIX junkie, but when you need a tool you can put in the hands of Joe Average, it has to have a graphical interface… It just must. Historically, back when Nessus was free and Free, there was no shortage of Tk/GTK/Web/Qt/etc. etc. interfaces: A lot of those are still around, but are generally handicapped from the newer features. When your license prohibits reverse engineering, it makes it hard for people to maintain their interfaces to your product.

Recently, I re-acquired my main scanning system (after a stealth project became production, literally overnight (over a year ago)), and decided to upgrade everything… Including Nessus. The upgrade was flawless. It found my old install, upgraded it, re-registered my feed, etc. etc. Very slick. I then upgraded the client on my laptop. That was not so smart of me. Following a common trend, Tenable has outed the old interface in place of a “new” Qt-based interface. I have nothing against “new” or “Qt”, but the new interface is missing dozens of previously-exposed features and one… one very IMPORTANT feature has been removed from both the Qt version and the CLI version: html_graph. It appears there is no html_graph in Nessus 3.

What is html_graph? It was a report export format that presented an HTML report of the scanning results along with pie and bar graphs giving you a visual representation of what the report contained. Very very very important when presenting this information to people wearing ties.

I searched the Interlink: No vast outcry! Just one, lonely, unanswered post, reporting the problem and asking for help. I would guess that because he wrote in all lower-case, no one wanted to tell him the harsh truth: You can’t with Nessus 3. Nope. No more XML output. No more HTML+Graphs. No more. I’ll hypothesize this is because of the license changes: They can’t link against a whole slew of Free libraries that do this work for you, because of the shift from libre to locked. That’s just a guess. At first I thought it was just in the Qt-interface, which could have been simply a CBA or porting issue, still on the TODO list: But when I saw the neutered CLI – without XML or html_graph – well, that’s indicative of politics as opposed to growing pains.

So I don’t know, as I’ve said, why these are gone. But they are. Thankfully, as of right now, I can still convert the Nessus 3-generated scan files (NBE) into html_graph and XML using an older version of the Nessus CLI (version 2.something), but it’s a pain, it’s an extra-step (okay, okay, I automated it with 2 lines of Perl, but it’s a pain to someone!!), and it devalues an otherwise very very valuable tool… For if you cannot show pie and bar-graphs to those wearing ties, on-demand, the fact that you have found 4781 security problems is moot.

[NOTE: This was written a while ago]

[UPDATE: Levanta is toast].

Remember LinuxCare? That ambitious start-up back-in-the-day that wanted to revolutionize Linux support? Yeah, they died. But from the ashes rose Levanta. Unlike LinuxCare, Levanta has been studiously working on making proprietary products to help Linux environments self-support.

While unable to bake one personally (they’d love to ship one with a PO in hand and then take it back if I don’t like it), I did get a dog-and-pony show with a sales rep and an engineer that was quite impressive. Complete birth-to-death lifecycle management of server deployments (metal or virtual) using PXE and installing OS’s on systems automagically, managing update deployments, etc. etc. etc. Basically anything you can do with RPM (updates, installs, rollbacks, etc. etc.) and TripWire (change detection/management) in a box, on a larger-scale, with a nice GUI.

The product was impressive, but as usual, I wasn’t impressed with the required coinage.

Pretty quickly I developed some proof-of-concept soft systems that provided 75% of the non-GUI functionality using true NetBoots instead of the PXE-install-based methods the Intrepid uses. I also found a lot of other tools available including Cobbler that could help make a low-cost, high-value system of similar stature fairly easily. When thinking about virtualization, other tools such as oVirt, Virt-Manager, and Con-Virt could certainly be leveraged to provide and in-house solution to the TLM dilemma.

If you don’t have the ambition or skills to glue together your own, the Levanta Intrepid is by far a best-of-breed appliance for total-lifecycle-management of your RPM-based Linux server deployments.

I have a Roomba. Hal Arthur is, in fact, the most useful gadget I’ve ever acquired, and I have had a lot of gadgets over the years. Ironically, it’s the most useful gadget most anyone would have, if they were willing to give it a chance. Yes, Roombas install themselves as members of the family quite quickly, and as such near demand to be named. I was a skeptic until I saw one in action – an old 4th generation that you can’t even get this side of eBay.

The Roomba is not just a robotic vacuum that deftly navigates my house while eviscerating dust-bunnies, eradicating hundreds-of-miles of girl-hair, hoarding vast oceans of bird feathers and seed husks, scarfing up long-lost receipts under my bed, and pushing a long-lost fork out from under my couch…. DAILY… Nay, it is in fact a development platform openly accessible and encouraged by the manufacturer! How many hardware companies encourage you to take apart and build upon their technologies? Without voiding the warranty?

You can build a serial interface, bluetooth interface, gamepad interface, Wiimote interface, cellphone interface … and you can buy the vast majority of those via third-parties if you value time over money, or suck with a soldering iron. iRobot provides (free) documentation on the SCI and some sample code to get developers started, and from there the possibilities are really endless. I have some gumstix gear on order after successfully prototyping some onboard control systems over a serial cable, which will essentially add a WiFi-accessible Linux computer directly into the chassis, and fully able to control the Roomba (more to come on that project)… Possibilities include tying in GPS, cameras (ala motion sensing), infrared, scheduling, adaptive behavior, etc.

Friend: People are morons and anyone who buys a Roomba in specific.
Me: You know I own a Roomba, right?
Friend: I did not.

While I never resorted to name-calling, I was skeptical of their value or utility for quite a while until experience and research brushed it aside and I took iRobot up on a 30-day trial. I won’t be sending it back. I’m happily a moron who hasn’t had to vacuum- yet still has near-spotless, practically no-maintenance carpets, rugs, and hardwood floors- in a loooong time.

As someone with a history in filesystems, I have kept abreast of the EXT4 developments, most recently, an interview with Eric Sandeen on the topic, from an RH/Fedora angle. EXT4 will have lots of features new to EXT3 (as well as possibly pick some of the remnant EXT2 stalwarts up, shake them up a bit, and get them to convert). EXT4, unfortunately, has no features new to filesystems in general, nor even open source ones. After reading through the latest “new feature” list in Eric’s interview, it read like a list of XFS features from a decade ago… Then I got to the bottom, and Eric worked @ SGI on XFS for 5 years, almost a decade ago. Go figure.

This is divisive for me. Because I’m a Darwinist: I believe that it’s perfectly fine if there are multiple organisms that do exactly the same thing, as eventually one will win and one will die. At the same time, I see all the energy being put into EXT4, and the practical side of me is screaming “HEY, YOU’RE JUST GETTING EXT TO WHERE XFS WAS 10 YEARS AGO- HOW ABOUT PUTTING YOUR ENERGY INTO XFS AND GETTING IT TO WHERE IT WILL BE 10 YEARS FROM NOW, FASTER?! KTHXBAI”. The main source for this schism is that the environment is rigged. XFS has been open source and stably available on Linux since springish of 2000 (and BSDs, shortly thereafter), yet it still has to fight tooth and nail for any recognition or prominent placement within Linux distros.

XFS has been a workhorse for those in-the-know for a long, long time. I have clients that bought SGI products because they wanted XFS, yet is has consistently been relegated to the backseat, in favor of the old-guard UFS.. er I mean EXT family. XFS continues to amaze various Linux administrators I work with, who, after having XFS inflicted on them, can’t imagine life without it. The catastrophes you can survive- albeit painfully- with XFS, are totally unrecoverable with EXT. The extreme operating conditions you can endure with XFS, are outside the realm where EXT can safely exist.

So there it is: my internal Filing System Fight Club. It is the beauty of open source, while at the same time the ugliness of it. Reinventing the wheel. Again. Thankfully some people are doing new things in this space.

Introduction

Do you have some data you’d love to have backed-up, in real-time, somewhere else, but don’t trust the destination? RAID-E is a great solution for you. The concept is simple, every time you modify a file, RAID-E makes a copy, encrypts it using one of numerous methods described  below, encrypts the name of the file as well (by default), and then copies it off to where it should be.

RAID-E is a FUSE filesystem that can use any underlying, mountable filesystem (or folders within the filesystem) as its sources and targets. You can, for example, RAID-E your “Documents” folder on your laptop to some Windows-shared space on a file server or NetApp.  RAID-E may be ported to other FUSEless operating systems someday.

RAID-E supports a rainbow of encryption algorithms and works in one of three modes:

PGP/GPG Encryption

If you have a PGP/GPG key, and would like everything to be encrypted using that, RAID-E is happy to oblige, and will use your public key to encrypt your files before copy. If you want them signed as well, then you will need to provide your keyring passphrase in order to access your private and/or signing key.

Standard Encryption

RAID-E can generate an encryption key (or you can provide one) using one of a number of user-selectable algorithms which it will then encrypt (using one of a number, user-selectable algorithms) using a phrase of your choice. The encrypted key can be stored on a USB stick or other flash-based removable media. When RAID-E is mounted, the phrase is required to decrypt the key, allowing files to be encrypted.

Cornucopia Encryption

Using the same concept as ‘Standard Encryption’, Cornucopia uses a number of different encryption algorithms. Individual files are encrypted using a pseudo-randomly picked algorithm. For example, one file might use AES, while the next Two-Fish. While the security advantage of this is admittedly dubious (don’t Doghouse me, Bruce – I admit it!) , it won’t decrease your security and may protect fractions of your dataset against attacks directed at particular algorithms.

Bootstrapping/Offline Synchronization

To aid in the initial bootstrapping, and to make synchronizing after making off-line changes a snap, a tool called ‘mirror-e’ is also included. ‘mirror-e’ will use the same configuration and methodology as RAID-E to encrypt and copy any changed files or new files between the source and target.

Various Configurable Defaults

By default, RAID-E will never delete files from a target.

By default, RAID-E is only concerned with file names and contents, not metadata, attributes, etc.

Be default, RAID-E does not verify a copy operation.

By default, RAID-E will always overwrite a target file.

Status and Errata

RAID-E and its toolset is being developed independently, and will be released under  the GPLv2 license.

Billboarding used to be the way to share groups of read-heavy bit or small-byte data across numerous processes or systems. The concept was simple: Have a file with a bunch of zeros in it, and on occasion change one of the zeros to a one. Processes/systems reading that file would say “Hmmm, the 4th ’slot’ is now a ‘1′, and that means [thing]“. This was used for everything from node status, to process states, to primitive anticipatory scheduling. Then objects became popular. Why read out of a billboard, when you can just share flag data across objects?

Well they’re back, it seems. Billboards have made a small-yet-noticeable resurgence in a number of systemic regions, and more than a couple system architects have noticed clients requesting solutions that boil down to billboarding (although generally given a more sexy name like “shared state file” or “offline node graph”).

I’ve had two projects in the last 6 months that have required, in general, a billboard, and am very happy to see them come back into vogue. They’ve been “gone” long enough so that the cool kids think they’re brilliant and “out-of-the-box” when they propose the “radical concept” of the “shared state file”, and those of us who’ve been doing advanced system architecture for .. gasp .. almost 15 years now, just smile and jot “billboard” in our notepad.

Systemic Billboards

In environments where there is shared (and preferably clustered) storage, billboards make a ton of sense as a means of easily communicating with other nodes. More than just 0’s and 1’s, a node can communicate an array of information about its health – and also other nodes can ask other nodes to do something. Maybe setting a node state to ‘2′ means “please restart your user services” or ‘3′ means “please reboot”. Whenever a node reads its own entry it can go “Hey cool, I’m suppose to do something.” As a means of pre-failure fencing, this is exceptionally handy as one does not necessarily have a connection to a greater network, but still may have a connection to storage- Allowing a control system to tell others “Hey, something bad is happening, please shut down”, for example if one node detects a UPS failure or pending drainage.

I currently have a project that requires the assumption that if bad things happen, the only communication available between nodes will be a shared IEEE1394 drive array – A better place to use a billboard does not exist.

Non-Systemic, or Quasi-Systemic Billboards 

Given an application that may have any number of processes (for example, any web-based application), using a billboard as a light IPC for each process to communicate its state or intentions, can save a lot of otherwise tricksy IPC coding. Sure, you can have a pipe dangling out there- But what happens if a process needs to skip a pipe read for a given cycle, and in the mean time the status of the pipe changes? Yes, you could have one pipe per process, but then you’re looking at a mess that could be more elegantly solved with … a billboard. With 256 ASCII characters available (I’m not even going to get into the possibilities with Unicode), you can communicate up to 255 different things per process– all sitting happily waiting to be read whenever is convenient, or necessary, with very very little side-effect.

Yes, contention issues need to be addressed by your application.

Yes, if your application is poorly designed and process spawn rates are out-of-control, a billboard will destroy your performance.

Yes, if your storage disappears, there is a problem: A big problem that a web application cannot solve, so its inability to get to the billboard could definitely be a sign that maybe it should go into a maintenance state instead of processing transaction it can’t actually handle.

Yes, if the file becomes corrupted, Bad Things could happen: The application can/should detect such problems and Do The Right Thing.

There are certainly situations where billboards are not the answer. There are certainly situations where using IPC or nodal communication is a better solution. I’ve never advocated billboards as the end-all of inter-process or inter-nodal communication: Only that it shouldn’t be discounted in cases it DOES make sense.

Many moons ago, we decided to move away from the old standard POP -based e-mail, and offer/promote IMAP instead. The paradigms are drastically different.

With POP, users log in and [are supposed to] download their mail to their local computer. The transaction is quick, and the storage resources necessary are minimal, as we only need to store the mail between the time they receive it and next check it. Bandwidth was mildly important, system RAM was mildly important, processing power was fairly important.

With IMAP, users log in, check for new mail, download all of the headers from whichever folder they’re in to their local computer, and stay logged in all day long. Storage is consumed voraciously. Bandwidth is something that needs tending. System RAM is consumed with the hunger of a million starving suns, and processing power has never been adequate.

… Or so we thought.

Back before the Earth cooled, when we moved to IMAP, we adopted a popular IMAP product known as Courier-IMAP. I don’t remember the discussions that led us there, but I remember them focusing on the products that were venerable and supported the mailbox format we use (Maildir served by qmail). We went along for a bit with this and it was working well, until our usage grew beyond some point- Since then we’ve had chronic mail problems that we threw literally everything at: Faster disks.. didn’t matter. Quad-processor server… helped a little, but didn’t matter. Faster disk interconnect.. didn’t matter. More RAM… helped a little, but didn’t matter. Different filing systems… didn’t matter. More venerable disk architecture… Didn’t matter. Changed IMAP server… Instant Relief.

Wait… what? Changed IMAP server… Instant Relief. With relatively little advance development time (which is not normal for our group) we replaced Courier-IMAP with Dovecot, and immediately went from intolerably high-loads and abnormal process latencies to the way things are supposed to work. Immediately. Jeff had a development version of Dovecot up on a development box that he was seeing good results from, and cloned it over to the production mail server, running on a different port and firewalled off so only the invited few could use it – We saw mailbox access times that used to take minutes, be trimmed to single-digit seconds. We saw operations that chronically timed out, be completed in real-time as they were requested. On the same hardware. On the same production server. In parallel to the ongoing problems everyone else was experiencing using Courier.

After flipping the proverbial switch, and replacing Courier with Dovecot for all users, we saw immediate relief of previous performance problems. The screenshot on the right (or above) was of our monitoring system showing 2 days of resource usage: Day 1 was the last day everyone was on Courier, and Day 2 was the first day on Dovecot.

Today, is the 10th Day we’ve been using Dovecot, and the results are the same. I have nothing bad to say about Courier. If you’re using it and it’s working for you there is no reason to change. If you’re having scalability problems, the first thing I would try, today, is Dovecot.

I agree completely.

Now I guess it makes sense why Bram wanted to “buy all rights” from the other quasi-silent BT devs, in particular the protocol design. Shame, Bram. You should’ve been honest.

UPDATE 8/10: After a couple days to ponder this more, I’m even more disappointed than I was. While a large proponent of open source, I don’t much care what Bram does with it. The source code to BT isn’t where the value is, it’s the protocol. Closing a protocol, is like telling someone they’re not allowed to talk your language anymore. It would be the Bush Administration saying “Cubans are evil, so we’re not going to allow them to speak English anymore.” Given the historical ignorance of this White House, I’d have expected that BEFORE I expected Bram to close the BT protocol.

A protocol is a culture embedded in technology: It defines etiquette between systems; it defines grammar and speech; it defines nouns and verbs. You learn a lot about humanity when you read well-written protocols: It’s not like learning a new language, it’s like being immersed in a new culture. That culture has been taken to a private island, and no one is allowed to experience it unless you’re declared fit by the czar.

On the plus side, this will spur some new innovations. BT has some serious flaws: flaws that have been well known and accepted on the grounds of ease-of-coding and portability. The paradigm can change, the flaws fixed, and a new format can be created and in a Darwinistic dual, we’ll see which succeeds. Of course, this assumes that the czar doesn’t unleash the lawyer-hounds.

Protocols should never be exclusive. Never. They should be the most open parts of our technological society.

I read Jonathan Schwartz’s (CEO of Sun) blog regularly. Not only has Sun been my favorite UNIX, and the SPARC architecture been one of my favorites since I was in high-school, but he’s the shining example of good-guy-geek-turned-CEO. I enjoy reading writings from people who understand what they’re talking about, and Jonathan epitomizes that. He’s not some CEO that has a bevvy of “public relations” fauna that prep him before making public statements, model him after the corporate image, or vet his writings. He’s also done more for Sun in the last few years than anyone has done in the last 12, in my opinion.

In his latest post, he references a note yesterday from Linus about his opinions on Sun’s open source lovefest. Linus fired, Jonathan answered. They’re both right, in my opinion. Sun’s history is dotted with come-to-Jesus promises of true openness that haven’t materialized (point Torvalds), but since Schwartz has assumed the reigns they have come through with a number of the recent promises (point Schwartz) … Not the least of which was GPLing their Java bits. As Linus points out this could be because non-Sun Java’s (such as the GNU Compiler for Java). Could be.

My $0.02: Linus should take Jon up on his offer for dinner, and have a good chat.