Finding my copy of BASICball, re-reading my cute little grade-school comments (”Screen math is stupid”), remembering  anecdotes about what was going on at the time (”Our class gerbil is coming home with me for Thanksgiving break!!!!”), caught me up in an atypical wave of nostalgia. And in that wave, in looking back – really thinking about everything I have written over the years: millions upon millions of lines – that two things really stand out as being key to my success: Worse is Better, and Modularity is Ultimate.

I’ll write more about those things, I’m sure, but for now: Happy 20th Anniversary BASICball v6.0.

I just aged a bit, happily.

Endocrys has two primary modular components: Autocrys and Paracrys.

Autocrys is an extensible communication protocol atop XMPP. It governs the syntax of commands or queries sent to systems or groups, the responses of systems to those queries, how to manage their presence, and how to react to presence changes in others.

Paracrys is a database-driven deployment and configuration system. Paracrys allows module code and configuration data to be stored centrally and deployed to Endocrys nodes on-demand. Paracrys fully supports versioning, thus allowing changes to be rolled-back in the case of a major oopsie. How small can a Paracrys module be? Here’s an example that implements a command called ’shell’ that allows you to do, essentially, whatever you want on an Endocrys client:

BEGIN { $Endo::MODS{SHELL}++; $Endo::CMDS{SHELL} = \&shell; }
END { delete $Endo::MODS{SHELL}; delete $Endo::CMDS{SHELL}; }

sub shell {
 return `@_`;
}

Drop that puppy into the Paracrys MODULES table with some other data, issue a mass “fetch module SHELL; refresh;” command, and bingo, all of your systems now let you do very bad things. It’s that easy to create a command to do something… Hopefully something useful.

Of course you should note that there is no access control in the above code… How do we prevent Bad People from using our horrendously very bad shell command? That used to be managed by the Communication Masters using another database called EndoACL, but has been folded into Paracrys’ duties and drastically simplified. Each Endocrys client, when receiving the shell command, will now ask Paracrys if the user who sent it is authorized to issue that command. Previously, the clients never even received commands from users not authorized to send them, at great expense.

One of the major goals of the project originally was to have absolutely minimal dependencies on third-party code, so I reinvented the wheel in numerous places. Now that it’s mine again, those requirements are vapor and I’m ripping out large swaths of my code, and exchanging it for API calls into other code that is the de facto standard to do whatever. For example, I wrote a function that copies a file from one location to another. Ew. The File::Copy module is the Perl Way to do that, so that’s how we do it now. Less code I have to maintain, and less code you have to read to understand Endocrys.

Another major goal of the original project was absolute redundancy on all levels. With a requirement like that, I over-engineered what were called the Communication Masters (CMs) so that they heart-beated each other, transferred each other’s sessions, held elections to decide who was authoritative for which IP ranges, dealt with segmentation and partitioning, etc. All of this at the cost of highly-customized hybrid XMPP/SQL servers that weren’t readily upgradeable. Wednesday night I spent a lot of time diagramming, and tonight solidified the spec to separate the XMPP server from the SQL database, and rely on established high-availability tools like pen or an SLB appliance to ensure connectivity to a farm of XMPP servers if needed. Additionally, this separation has allowed me to use MySQL clusters for the Paracrys bits, which adds scary levels of redundancy to those very critical bits.

Lastly for this post, the entire ithread Endocrys implementation has been ripped out and replaced with EV and AnyEvent, and the Net::XMPP code has been replaced with AnyEvent::XMPP for one cohesive event loop that runs very very fast. Originally I envisioned an Endocrys client maintaining dozens of XMPP sessions while handling dozens of system events and receiving dozens of commands, so I stuck everything in threads, and allowed it to scream along on SMP boxes. While this works just fine, there is a LOT of extra complexity involved with sharing variables across threads, dealing with races, etc. and the benefits are dubious when compared against a good, strong, event-loop system. I’m not quite done yet, but the net loss should be about 30% of the main code modules, with reduced complexity for all sub-modules as well.

I don’t have an ETA as to when the code will be generally available, but I’ve had some pings from some bright people interested in hammering the retooled version in non-critical environments, so hopefully it will be this year.

This is one of my favorite products, and I’m more than a little excited to get it back and get it out. It’s been battle-tested for many years, and I’m very proud of it. I’m working on getting the code cleaned up and abstracted before releasing it under the GPLv2. Below are edited points from slides describing Endocrys and why you might be interested in it.

The Problems

Dozens… hundreds… of systems, physical and virtual, all going about their business. Then something happens: maybe a disk drive failed, maybe a process died, maybe someone ordered a ‘reboot’ or ‘halt’. Those systems don’t have a way of communicating that externally. There is no “Hey, I’m rebooting, BRB” in the server world.

Dozens… hundreds… of systems, physical and virtual, all going about their business. Then you have a question: How many of them have Western Digital harddrives listed in a recent recall? How many of them are running <2GB of RAM? How many of them are running a certain version of some software listed in a security advisory? There’s no way to ask that question to the farm. There is no “Dear Lazyweb, answer this question for me” in the server world.

The Purpose

At its most basic level, Endocrys is a conduit between all of the systems and you. Think of it like a gigantic Instant Messaging buddy list, where all of your buddies are systems. When they’re online, they are in the list and can set their status messages, send you messages, send each other messages, receive messages, etc. Endocrys leverages the eXtensible Messaging and Presence Protocol (XMPP) to tie this framework into existing clients, transports and APIs, enabling a near-infinite number of possible applications or functions you can deploy.

The Technology

Endocrys is built as a framework – an abstract set of rules that can be extended at any time by writing little modules. These modules can be applied across the Endrocrys network instantly, without any downtime.

By leveraging XMPP, the Endocrys network is highly-redundant with no single fail points. Any number of “Communication Masters” (XMPP servers) are online, but only one is needed to keep communication flowing. All network communication is encrypted and signed. Partitioning and segmentation is handled rationally.

Communication is very similar to Instant Messaging, there is relatively no latency, and XMPP assures delivery even to systems offline when the message was sent.

Monitoring and control systems can participate on Endocrys, automating the remediation of problems remotely and automatically.

The Protocol

XMPP sits atop TCP, and atop that sits the Endocrys Communication Protocol aka Autocrys. ECP is a fully-authenticated, fully-controlled, skeptical protocol that serves both for sending structured announcements as well as sending and processing commands. The entire ECP specification is listed in AUTOCRYS.TXT. Endocrys agents can be written in any programming language, attached to any other framework, at any OSI level, as long as they can speak XMPP and implement ECP appropriately.

RooCluster is a command-and-control application designed for the special needs of  multiple robots operating in the same space, or over large multi-room spaces. Each Roomba is being fitted with an RFID tag, which, in coordiation with some more wireless access points, allows me to triangulate where a Roomba is and its travel vector (sometimes, math is cool). This information can help RooCluster avoid nasty Roomba-on-Roomba collisions, and also presents the possibility of meta-virtual walls.

If you have a Roomba, you probably have a virtual wall – the little pylon that sends out an infrared beam that the Roombas treat just like a wall. With some work, RooCluster should be able to honor coordinate-based lines (which could, in turn, form other shapes) and effectively “wall-off” areas without needing a physical barrier, or a battery-sucking virtual wall. You can also overlay the position and vector data onto floorplans, and see exactly where the Roombas are, and where they’re going.

Of course, you can also use it to make your Roombas dance with each other.

Or joust.

“In the aftermath of Microsoft’s recent decision to contribute 20,000 lines of device driver code to the Linux community, Christopher Smart of Linux Magazine talked to Linus Torvalds and asked if the code was something he would be happy to include, even though it’s from Microsoft. ‘Oh, I’m a big believer in “technology over politics.” I don’t care who it comes from, as long as there are solid reasons for the code, and as long as we don’t have to worry about licensing etc. issues,’ says Torvalds. ‘I may make jokes about Microsoft at times, but at the same time, I think the Microsoft hatred is a disease. I believe in open development, and that very much involves not just making the source open, but also not shutting other people and companies out.’ Smart asked Torvalds if Microsoft was contributing the code to benefit the Linux community or Microsoft. ‘I agree that it’s driven by selfish reasons, but that’s how all open source code gets written! We all “scratch our own itches.” It’s why I started Linux, it’s why I started git, and it’s why I am still involved. It’s the reason for everybody to end up in open source, to some degree,’ says Torvalds. ‘So complaining about the fact that Microsoft picked a selfish area to work on is just silly. Of course they picked an area that helps them. That’s the point of open source — the ability to make the code better for your particular needs, whoever the “your” in question happens to be.’”

Users Are Un[trust]worthy

The first, and most important point, is that you can never trust user-supplied data. Ever. Just because your whiz-bang drop-down box only shows the user four options, doesn’t mean they can’t end up sending `rm -Rf /`. This is the single largest threat against an in-house application. User-supplied data must always always be vetted, sanitized and treated as if it could take over the world. A lot of programmers use rapid-application-development tools to create interfaces. These tools are a boon for creating slick interfaces, but generally a complete bust at data integrity enforcement. You tell it “make this a date field”, and it does… But anyone with a quick script can insert 40MB of garbage data, which will at best crash your application, and at worst, overflow a buffer that had improper bounds (see next point) and allow “remote execution of arbitrary code” (READ: You’re fucked).

So how do you fix this? Regardless of the languages you program in, they probably have a regular expressions (regexp) library. Many popular languages even have “Perl-Compatible Regular Expressions” (PCRE), as there is no better language for data process than Perl, and its regexp syntax is well-defined (albeit with a moderate learning-curve). Regexps are the easiest way to enforce your intentions with data.

unless($userdate =~ m#^\d\d\d\d/\d\d/\d\d$#) { Freak_Out(); }

That’s a one-liner that says, exactly,”unless the data in the variable $userdate begins with four-digits, followed by a fore-slash, followed by two-digits, followed by a fore-slash, followed by two-digits, and that’s it – Freak Out”. (m – we’re matching, # – start the regexp, ^ – match at the begining of the data, \d – a digit (0-9), $ – match the end of the data, # – end the regexp).

You could easily write a function to do this for you:

sub Is_Date {
  my $userdate;
  if($userdate =~ m#^\d\d\d\d/\d\d/\d\d$#) { return 1; }
  else { return 0; }
}

Then just call

unless(Is_Date($usercrap)) { Freak_Out(); }

every time you need to check that you’ve REALLY got a date. Not hard. The user could still be entering the data wrong, but that’s not the secure programmer’s job.

It is imperative that your application go out of its way to recognize crap as soon as it can. Don’t pass a variable containing unchecked user-supplied data to a dozen  or so functions before checking it. Accept it, and check it immediately. SQL injection attacks are another whole can of worms that can take advantage of improper sanitization, and frequently get executed in strange places. Sanitize early.

I know you guys aren’t writing AJAX stuff, but I’d be derelict if I didn’t hammer this out anyhow: If you trust JavaScript or any other client or browser -executed code to do your security bits for you, you are not only a moron, but should be made to wear a Scarlet Zero for the next few years. Zero for “0wn3d”. Just because you write JavaScript, doesn’t mean that’s how the browser will execute it. ANYONE can rewrite your JavaScript to do whatever they want. The above IsDate function, if implemented in client-side JavaScript can easily simply “return 1″ regardless of what’s passed. You must never trust client-supplied data regardless of whether you believe it was vetted client-side. It must be vetted server-side. Must. Must. Must. If, for user convenience, you want to implement something to trap errors quickly client-side, that’s groovy, but in no way does it excuse you from vetting the content once submitted to your application.

I can’t say this enough. It seems like common-sense, and a lot of you are nodding at me right now, but a month on you’re going to write a quick webform with radio-button selection, and someone in Bejing is going to own your database server because they submitted `echo root:fXAWEOo7DsNSM:0:0:0:0::: > /etc/shadow` to the “What’s your favorite fruit?” question. [PAUSE] It’s funny right now, it’s not funny when you’re facing a DoD inquiry.

Data Sizes Are Critical (AKA Know Your Data)

The second most important thing I can convey to you is the concept of sanitizing types. Those of us in the room that are automation programmers and write in the P-languages don’t have to give a shit about this, but those of you using Java, C or anything that comes out of Redmond, WA need to listen up: If you’re blindly stuffing data into a restricted type, you better be damn certain it’s the right size. Last year one of your competitors had a nice lad write an internal automation system that took data from a database and did things with it. Anyone know how large a MySQL blob-type is? 64k bytes. Anyone know how large a Microsoft Visual BASIC date-type is? 8 bytes. Thankfully, he wasn’t writing a user-facing application. [PAUSE] Last I heard, he was writing TARP-laundering algorithms for AIG.

Everytime you hear about a “buffer-overflow” attack, this is because some moron didn’t check data before stuffing it into memory. The lower-level you code, the more important this is. The example I gave about the VB app only cost the contractor a few dozen-thousand dollars and a loss-of-face from their client because VB just panics and dies when you do that. If you’re coding in C and not running on a very secure linux box with a memory-jockeying system, you may well have just overwritten your security code with:

blahblaWastingSpaceUntilYourDataTypeIsOverblahblahblahCall Function DoSomethingBad

Yeah, that’s what a buffer-overflow looks like. Know your data, and when in doubt cast. A lot of instructors shy away from casting. It slows things down, it causes a lot of compile-time checks that frequently create errors or warnings in otherwise clean code. Casting is the absolute best way to make sure you’re not blowing out a primitive type. Of course, if you’re writing into a char-array, that’s not going to help you, but that’s your problem.

Lazy Handles

You guys know all about access permissions, so I’m going to skip over a bunch of stuff. One thing I’ve seen here and elsewhere, is opening handles – network sockets, database handles, file handles, directory handles, etc. – as users with way more permission than necessary. Sure, it’s easier for you, but what happens when your code gets hosed and someone from Bulgaria managed to inject themselves in before you’ve let go of that handle? Explaining to your boss that you opened the files as root “because it was easier” isn’t going to fly. Your application needs to run restricted, which you already know, and you absolutely cannot wantonly elevate handles without damn good reason. I can’t count the number of applications I’ve seen – Hell, applications I’ve written – that immediately connect to a resource as God himself to do some stuff that’s pretty primitive – and maybe, MAYBE one operation in fifty that actually needed that access. Which leads to the next problem here…

Keeping handles open longer than necessary, or in anticipation of future operations. If your application does six operations on a resource and only one needs the assistance of Angels, then that’s the only operation that gets the Silver Trumpets – the rest can stew along the rest of us. I don’t care if you’re doing all six of those operations at once: you run the primitives as a primitive user, and then either change-user or fork or whatever you need to do as the elevated user for that one operation. Yeah, I groan about it too. But that’s it. Every moment your application has an elevated resource connection, is a moment that someone is going to get their

INSERT INTO USERS user='yakov',password='smirnov'; GRANT ALL FOR ALL TO USER identified by 'yakov';

into your data. It’ll take your auditors a fiscal quarter before they find that account.

With the exception of operating-system handles, you can almost always switch out, rebind, setuid, or the like, without making a new connection.

1176da21241f79203fbd93e367f35142

Yeah, encrypt everything. I know you guys are using SSL for nearly all resource connections, which is ridiculously important. Even the server-to-server stuff that we used to shrug off and say “yeah, well it’s on a switched network, and in the same room” has got to be encrypted on the wire. There are too many techniques and tools out there to get in the middle of those streams, and we all know how reliable network administrators are at picking up that stuff. [PAUSE]

Even within your application, if you’re writting out temp data, encrypt it from possibly prying eyes and processes. If you’ve got in-memory data that’ll be sitting around for some time, and might be a candidate for swapping, encrypt it in memory. How many of you have ever even considered encrypting live data? If that data gets stale, and the operating system decides to swap out some pages, that stuff is possibly going to be visible unless your infrastructure takes into account encrypted swap and the like. In lots of languages, encryption and decryption of relatively small amounts of data (< available RAM) is pretty simple. I’m not advocating it for everything, but there’s not much harm in:

$data=encrypt($data);
{ #Long
  #Running
} #Block.
$data=decrypt($data);

If something weird happens, and some or all of $data is swapped out because it’s not being used, it’s useless to an attacker.

This is divisive for me. Because I’m a Darwinist: I believe that it’s perfectly fine if there are multiple organisms that do exactly the same thing, as eventually one will win and one will die. At the same time, I see all the energy being put into EXT4, and the practical side of me is screaming “HEY, YOU’RE JUST GETTING EXT TO WHERE XFS WAS 10 YEARS AGO- HOW ABOUT PUTTING YOUR ENERGY INTO XFS AND GETTING IT TO WHERE IT WILL BE 10 YEARS FROM NOW, FASTER?! KTHXBAI”. The main source for this schism is that the environment is rigged. XFS has been open source and stably available on Linux since springish of 2000 (and BSDs, shortly thereafter), yet it still has to fight tooth and nail for any recognition or prominent placement within Linux distros.

XFS has been a workhorse for those in-the-know for a long, long time. I have clients that bought SGI products because they wanted XFS, yet is has consistently been relegated to the backseat, in favor of the old-guard UFS.. er I mean EXT family. XFS continues to amaze various Linux administrators I work with, who, after having XFS inflicted on them, can’t imagine life without it. The catastrophes you can survive- albeit painfully- with XFS, are totally unrecoverable with EXT. The extreme operating conditions you can endure with XFS, are outside the realm where EXT can safely exist.

So there it is: my internal Filing System Fight Club. It is the beauty of open source, while at the same time the ugliness of it. Reinventing the wheel. Again. Thankfully some people are doing new things in this space.

Do you have some data you’d love to have backed-up, in real-time, somewhere else, but don’t trust the destination? RAID-E is a great solution for you. The concept is simple, every time you modify a file, RAID-E makes a copy, encrypts it using one of numerous methods described  below, encrypts the name of the file as well (by default), and then copies it off to where it should be.

RAID-E is a FUSE filesystem that can use any underlying, mountable filesystem (or folders within the filesystem) as its sources and targets. You can, for example, RAID-E your “Documents” folder on your laptop to some Windows-shared space on a file server or NetApp.  RAID-E may be ported to other FUSEless operating systems someday.

RAID-E supports a rainbow of encryption algorithms and works in one of three modes:

PGP/GPG Encryption

If you have a PGP/GPG key, and would like everything to be encrypted using that, RAID-E is happy to oblige, and will use your public key to encrypt your files before copy. If you want them signed as well, then you will need to provide your keyring passphrase in order to access your private and/or signing key.

Standard Encryption

RAID-E can generate an encryption key (or you can provide one) using one of a number of user-selectable algorithms which it will then encrypt (using one of a number, user-selectable algorithms) using a phrase of your choice. The encrypted key can be stored on a USB stick or other flash-based removable media. When RAID-E is mounted, the phrase is required to decrypt the key, allowing files to be encrypted.

Cornucopia Encryption

Using the same concept as ‘Standard Encryption’, Cornucopia uses a number of different encryption algorithms. Individual files are encrypted using a pseudo-randomly picked algorithm. For example, one file might use AES, while the next Two-Fish. While the security advantage of this is admittedly dubious (don’t Doghouse me, Bruce – I admit it!) , it won’t decrease your security and may protect fractions of your dataset against attacks directed at particular algorithms.

Bootstrapping/Offline Synchronization

To aid in the initial bootstrapping, and to make synchronizing after making off-line changes a snap, a tool called ‘mirror-e’ is also included. ‘mirror-e’ will use the same configuration and methodology as RAID-E to encrypt and copy any changed files or new files between the source and target.

Various Configurable Defaults

By default, RAID-E will never delete files from a target.

By default, RAID-E is only concerned with file names and contents, not metadata, attributes, etc.

Be default, RAID-E does not verify a copy operation.

By default, RAID-E will always overwrite a target file.

Status and Errata

RAID-E and its toolset is being developed independently, and will be released under  the GPLv2 license.

Well they’re back, it seems. Billboards have made a small-yet-noticeable resurgence in a number of systemic regions, and more than a couple system architects have noticed clients requesting solutions that boil down to billboarding (although generally given a more sexy name like “shared state file” or “offline node graph”).

I’ve had two projects in the last 6 months that have required, in general, a billboard, and am very happy to see them come back into vogue. They’ve been “gone” long enough so that the cool kids think they’re brilliant and “out-of-the-box” when they propose the “radical concept” of the “shared state file”, and those of us who’ve been doing advanced system architecture for .. gasp .. almost 15 years now, just smile and jot “billboard” in our notepad.

Systemic Billboards

In environments where there is shared (and preferably clustered) storage, billboards make a ton of sense as a means of easily communicating with other nodes. More than just 0’s and 1’s, a node can communicate an array of information about its health – and also other nodes can ask other nodes to do something. Maybe setting a node state to ‘2′ means “please restart your user services” or ‘3′ means “please reboot”. Whenever a node reads its own entry it can go “Hey cool, I’m suppose to do something.” As a means of pre-failure fencing, this is exceptionally handy as one does not necessarily have a connection to a greater network, but still may have a connection to storage- Allowing a control system to tell others “Hey, something bad is happening, please shut down”, for example if one node detects a UPS failure or pending drainage.

I currently have a project that requires the assumption that if bad things happen, the only communication available between nodes will be a shared IEEE1394 drive array – A better place to use a billboard does not exist.

Non-Systemic, or Quasi-Systemic Billboards 

Given an application that may have any number of processes (for example, any web-based application), using a billboard as a light IPC for each process to communicate its state or intentions, can save a lot of otherwise tricksy IPC coding. Sure, you can have a pipe dangling out there- But what happens if a process needs to skip a pipe read for a given cycle, and in the mean time the status of the pipe changes? Yes, you could have one pipe per process, but then you’re looking at a mess that could be more elegantly solved with … a billboard. With 256 ASCII characters available (I’m not even going to get into the possibilities with Unicode), you can communicate up to 255 different things per process– all sitting happily waiting to be read whenever is convenient, or necessary, with very very little side-effect.

Yes, contention issues need to be addressed by your application.

Yes, if your application is poorly designed and process spawn rates are out-of-control, a billboard will destroy your performance.

Yes, if your storage disappears, there is a problem: A big problem that a web application cannot solve, so its inability to get to the billboard could definitely be a sign that maybe it should go into a maintenance state instead of processing transaction it can’t actually handle.

Yes, if the file becomes corrupted, Bad Things could happen: The application can/should detect such problems and Do The Right Thing.

There are certainly situations where billboards are not the answer. There are certainly situations where using IPC or nodal communication is a better solution. I’ve never advocated billboards as the end-all of inter-process or inter-nodal communication: Only that it shouldn’t be discounted in cases it DOES make sense.