A little later, I really expected the Palm Pre to be the nirvana of compact, hyper-connected mobile devices. It wasn’t pretentious. It had a f%$#!%& keyboard. It was ahead of the curve in a number of features. But it wasn’t substantially different. Palm made the concious decision to keep the interface very similar to the old Palm OS, and let’s face it – user interfaces have evolved since then. But I got one. I like it. It’s a nice toy. If you want a compact, full-features smart phone, it’s still your best bet.

Motorola unveiled the Sholes over the summer. Beefy processor. Best-of-breed screen. Ridiculous connectivity solutions. A f%$#!%& keyboard. And CDMA (like the pre) so I don’t have to use the Ancient Telegram & Trash network. 5MP camera w/ LED flash. Ran Android (1.6 at the time). Super cool. At that point, they were still shopping for a vendor. I was cautiously optimistic it wouldn’t be a metro-only network like T-Mobile or Sprint.

Then came the onslaught of Droid Does during the baseball post-season. Speculation ran wild as to which phone it was, beneath the hype. BGR scooped it and pointed it out as a Sholes running Android 2.0. Thursday last I received mine.

Metal. Everywhere.

Ridiculously clear screen with outstanding pixel density.

Incredibly fast processor.

A f%$#!%& keyboard.

Seamless integration with all of the stuff I use (e-mail, calendar, contacts, etc. etc.).

Transparent movement between WIFI and the Verizon 3G network.

Incredibly fast processor.

Deep interface built atop a fully-accessible Linux system.

Scads of customizability.

Surprisingly good camera with shockingly bright flash.

Oh, and an incredibly fast processor.

If the pricetag is $100 too high for you, Verizon is also offering an HTC-based version called the Eris with no keyboard, more plastic, and a mid-level processor, with the same interface and general feature-set.

Oh yeah, and when you send e-mail, it’s not tagged “Sent from my <BlackBerry|iPhone|Other Pretentious Device>”.

But, since most people seem to enjoy those things: This post authored using a WordPress app from my Droid.

Endocrys has two primary modular components: Autocrys and Paracrys.

Autocrys is an extensible communication protocol atop XMPP. It governs the syntax of commands or queries sent to systems or groups, the responses of systems to those queries, how to manage their presence, and how to react to presence changes in others.

Paracrys is a database-driven deployment and configuration system. Paracrys allows module code and configuration data to be stored centrally and deployed to Endocrys nodes on-demand. Paracrys fully supports versioning, thus allowing changes to be rolled-back in the case of a major oopsie. How small can a Paracrys module be? Here’s an example that implements a command called ’shell’ that allows you to do, essentially, whatever you want on an Endocrys client:

BEGIN { $Endo::MODS{SHELL}++; $Endo::CMDS{SHELL} = \&shell; }
END { delete $Endo::MODS{SHELL}; delete $Endo::CMDS{SHELL}; }

sub shell {
 return `@_`;
}

Drop that puppy into the Paracrys MODULES table with some other data, issue a mass “fetch module SHELL; refresh;” command, and bingo, all of your systems now let you do very bad things. It’s that easy to create a command to do something… Hopefully something useful.

Of course you should note that there is no access control in the above code… How do we prevent Bad People from using our horrendously very bad shell command? That used to be managed by the Communication Masters using another database called EndoACL, but has been folded into Paracrys’ duties and drastically simplified. Each Endocrys client, when receiving the shell command, will now ask Paracrys if the user who sent it is authorized to issue that command. Previously, the clients never even received commands from users not authorized to send them, at great expense.

One of the major goals of the project originally was to have absolutely minimal dependencies on third-party code, so I reinvented the wheel in numerous places. Now that it’s mine again, those requirements are vapor and I’m ripping out large swaths of my code, and exchanging it for API calls into other code that is the de facto standard to do whatever. For example, I wrote a function that copies a file from one location to another. Ew. The File::Copy module is the Perl Way to do that, so that’s how we do it now. Less code I have to maintain, and less code you have to read to understand Endocrys.

Another major goal of the original project was absolute redundancy on all levels. With a requirement like that, I over-engineered what were called the Communication Masters (CMs) so that they heart-beated each other, transferred each other’s sessions, held elections to decide who was authoritative for which IP ranges, dealt with segmentation and partitioning, etc. All of this at the cost of highly-customized hybrid XMPP/SQL servers that weren’t readily upgradeable. Wednesday night I spent a lot of time diagramming, and tonight solidified the spec to separate the XMPP server from the SQL database, and rely on established high-availability tools like pen or an SLB appliance to ensure connectivity to a farm of XMPP servers if needed. Additionally, this separation has allowed me to use MySQL clusters for the Paracrys bits, which adds scary levels of redundancy to those very critical bits.

Lastly for this post, the entire ithread Endocrys implementation has been ripped out and replaced with EV and AnyEvent, and the Net::XMPP code has been replaced with AnyEvent::XMPP for one cohesive event loop that runs very very fast. Originally I envisioned an Endocrys client maintaining dozens of XMPP sessions while handling dozens of system events and receiving dozens of commands, so I stuck everything in threads, and allowed it to scream along on SMP boxes. While this works just fine, there is a LOT of extra complexity involved with sharing variables across threads, dealing with races, etc. and the benefits are dubious when compared against a good, strong, event-loop system. I’m not quite done yet, but the net loss should be about 30% of the main code modules, with reduced complexity for all sub-modules as well.

I don’t have an ETA as to when the code will be generally available, but I’ve had some pings from some bright people interested in hammering the retooled version in non-critical environments, so hopefully it will be this year.

This is one of my favorite products, and I’m more than a little excited to get it back and get it out. It’s been battle-tested for many years, and I’m very proud of it. I’m working on getting the code cleaned up and abstracted before releasing it under the GPLv2. Below are edited points from slides describing Endocrys and why you might be interested in it.

The Problems

Dozens… hundreds… of systems, physical and virtual, all going about their business. Then something happens: maybe a disk drive failed, maybe a process died, maybe someone ordered a ‘reboot’ or ‘halt’. Those systems don’t have a way of communicating that externally. There is no “Hey, I’m rebooting, BRB” in the server world.

Dozens… hundreds… of systems, physical and virtual, all going about their business. Then you have a question: How many of them have Western Digital harddrives listed in a recent recall? How many of them are running <2GB of RAM? How many of them are running a certain version of some software listed in a security advisory? There’s no way to ask that question to the farm. There is no “Dear Lazyweb, answer this question for me” in the server world.

The Purpose

At its most basic level, Endocrys is a conduit between all of the systems and you. Think of it like a gigantic Instant Messaging buddy list, where all of your buddies are systems. When they’re online, they are in the list and can set their status messages, send you messages, send each other messages, receive messages, etc. Endocrys leverages the eXtensible Messaging and Presence Protocol (XMPP) to tie this framework into existing clients, transports and APIs, enabling a near-infinite number of possible applications or functions you can deploy.

The Technology

Endocrys is built as a framework – an abstract set of rules that can be extended at any time by writing little modules. These modules can be applied across the Endrocrys network instantly, without any downtime.

By leveraging XMPP, the Endocrys network is highly-redundant with no single fail points. Any number of “Communication Masters” (XMPP servers) are online, but only one is needed to keep communication flowing. All network communication is encrypted and signed. Partitioning and segmentation is handled rationally.

Communication is very similar to Instant Messaging, there is relatively no latency, and XMPP assures delivery even to systems offline when the message was sent.

Monitoring and control systems can participate on Endocrys, automating the remediation of problems remotely and automatically.

The Protocol

XMPP sits atop TCP, and atop that sits the Endocrys Communication Protocol aka Autocrys. ECP is a fully-authenticated, fully-controlled, skeptical protocol that serves both for sending structured announcements as well as sending and processing commands. The entire ECP specification is listed in AUTOCRYS.TXT. Endocrys agents can be written in any programming language, attached to any other framework, at any OSI level, as long as they can speak XMPP and implement ECP appropriately.

The first diagram below (left) shows the near-final design diagram, and I’ve confirmed that everything fits. Everything except the rear exhaust fan block. There will have to be a wee bit of … “furniture modification” as the slot for cable ingress is too small. A liquid cooling system was added mainly to keep the sound level down. The pump is quieter than most fans, and the large CPU heatsink and exhaust fan pictured were able to be replaced with a small copper block and some tubes.

The second diagram below (right) outlines the machining specification for the replacement front door panel. Originally it was high-end non-thermal glass (to allow remote control IR to pass through), but had to be replaced with something I could cut. After a bunch of trial and error, I settled on 1/8″ acrylic. I’m still working on this piece- I’ve gone through a bunch of scrap trying to get the cuts and breaks right, and that is holding up the final tweaking (and pictures!), mostly.

The liquid cooling system (pump and exhaust fans) has a total noise of 11dB, all of which are in the rear of the cabinet. The 120MM intake fan is almost completely silent ~5dB, but inward-facing so it doesn’t expel much noise at all (<1db immediately outside the cabinet). The noise measure from The Comfiest Couch in the world with the entire system runnng is 5 dB – quiet enough to hear the harddrives prattling about, and substantially quieter (5-6 times) than the air-cooled equivalent.

A while back, I got a new “TV”: An LG LH90. This TV, unlike my old TV (a Sony Wega that weighed 275lbs), could actually be placed atop furniture (as opposed to a steel-reinforced, purpose-built “Wega Stand”). Through the miracle of 4 HDMI ports on the TV, the cable box, Playstation 3 and my primary home PC can all be slaved  to a single gigantic screen: The TV also has a fiberoptic audio output which went smoothly into my old-yet-still-amazing surround-sound receiver, allowing me to use just the HDMI between devices for all video and audio. Consolidation nirvana!! But there was a problem:

My TV could be remote controlled from The Comfiest Couch In The World.

My PS3 could be remote controlled from The Comfiest Couch In The World.

My cable box could be remote controlled from The Comfiest Couch In The World.

My PC … couldn’t.

*twitch* *twitch* *sob* I had to get up and … *twitch* push a button.

So, I set upon the Intertubes with the goal of fixing that. Unfortunately, as forays into the morass of technology acquisition seem to go for me, I was sidetracked by a dangerous thing: potential.

Saga 2: Patiently Purveying Potential Possibilities… Primarily

While I should have been sufficiently pleased with acquiring a Thermaltake HTPC remote system, I kept listing restlessly at night, thinking of the other magical things I saw on the Interweb… Thinking of what I could do. After all, wouldn’t it be so cool if I built my PC into the furniture? That little cubby on the right was just begging to be modded- Begging to serve a higher purpose than banal storage.

But a mod of wood? Wood is a poor conductor of heat, and an amazing insulator: That would be a thermodynamic nightmare! And that point was precisely why I had to do it: I found a worthy challenge.

Specification

• Must not destroy, scratch or poke holes in the pricey (and pretty) furniture
• Must properly cool my 6-core system, and oversized graphics cards
• Must not be externally louder than current design (10-20dB)
• Must be completely controllable from The Comfiest Couch In The World
• Must be worthy

RooCluster is a command-and-control application designed for the special needs of  multiple robots operating in the same space, or over large multi-room spaces. Each Roomba is being fitted with an RFID tag, which, in coordiation with some more wireless access points, allows me to triangulate where a Roomba is and its travel vector (sometimes, math is cool). This information can help RooCluster avoid nasty Roomba-on-Roomba collisions, and also presents the possibility of meta-virtual walls.

If you have a Roomba, you probably have a virtual wall – the little pylon that sends out an infrared beam that the Roombas treat just like a wall. With some work, RooCluster should be able to honor coordinate-based lines (which could, in turn, form other shapes) and effectively “wall-off” areas without needing a physical barrier, or a battery-sucking virtual wall. You can also overlay the position and vector data onto floorplans, and see exactly where the Roombas are, and where they’re going.

Of course, you can also use it to make your Roombas dance with each other.

Or joust.

The hosted systems industry has turned another critical point. Several years ago we eschewed large mainframe systems in exchange for commodity servers that could divide load and work together to provide services without single-vendor lock-in and without a single piece of “iron” waiting to fail. The computing power of a $2,000,000 mainframe was dwarfed by the implementation of $80,000 in commodity hardware. With virtualization coming-of-age- with Intel and AMD putting hooks into their processors and chipsets to allow virtualization to be fully realized and not just a software-only hack- we’ve seen those same commodity systems hosting dozens of virtual systems reliably and at near-metal efficiency. The cost per virtual system is a number rapidly approaching zero.

New offerings from Sun, IBM and HP/Compaq are emphasizing something that “the server guys” haven’t needed to care much about: infrastructure. Historically, your network engineers and analysts worried about interconnection, route redundancy, and ensuring the bits could flow where they needed, reliably and sufficiently; and your system engineers worried about everything up to the point the bits hit “the network”. Moving forward, that is almost a debilitating dichotomy. Traditionally, in the post-mainframe era, a physical system did one or two things and its exclusion from the network or its under-performance on the network was a minor issue. With a physical system possibly hosting dozens of virtual systems- all with unique networking requirements, cross-talking requirements, and of course: networked storage requirements- your system engineers must be well-versed in network engineering. “The Network Is The Computer” is not just a Sun tag-line, or a lame cliche’. We’re now fully realizing the potency of that statement. Every system offering from the Big Three contains significant “infrastructure” features: Network features.

By pushing more and more network features into server systems- IBM servers with Cisco “swrouters” built-in, for example – the server itself has become more important and less relevant at the same time. Keeping it up and running well will require a new kind of system engineer because “the box” is now more complex: But at the same time, collections of “boxes” should be able to self-heal and adapt to the failures of others. Each system has now become disposable.

A large swath of the architectural literati are already deploying quantities of self-healing farms that take over the work – the very virtual machines – of failed or failing physical systems. Virtualization on its own wasn’t a game-changer. Virtualization with processor support and recognition sparked real potential. Virtualization on top of “infrastructure”-aware (e.g. heavily networked) physical systems has dramatically shifted the value of hybrid “networked systems engineers”, raised the bar for the “server guys” to get up to speed on the real internals of networking, and has provided the unprecedented opportunity to deploy redundantly resilient systems that can in-practice achieve five-to-seven “nines” of reliability.

“In the aftermath of Microsoft’s recent decision to contribute 20,000 lines of device driver code to the Linux community, Christopher Smart of Linux Magazine talked to Linus Torvalds and asked if the code was something he would be happy to include, even though it’s from Microsoft. ‘Oh, I’m a big believer in “technology over politics.” I don’t care who it comes from, as long as there are solid reasons for the code, and as long as we don’t have to worry about licensing etc. issues,’ says Torvalds. ‘I may make jokes about Microsoft at times, but at the same time, I think the Microsoft hatred is a disease. I believe in open development, and that very much involves not just making the source open, but also not shutting other people and companies out.’ Smart asked Torvalds if Microsoft was contributing the code to benefit the Linux community or Microsoft. ‘I agree that it’s driven by selfish reasons, but that’s how all open source code gets written! We all “scratch our own itches.” It’s why I started Linux, it’s why I started git, and it’s why I am still involved. It’s the reason for everybody to end up in open source, to some degree,’ says Torvalds. ‘So complaining about the fact that Microsoft picked a selfish area to work on is just silly. Of course they picked an area that helps them. That’s the point of open source — the ability to make the code better for your particular needs, whoever the “your” in question happens to be.’”

Users Are Un[trust]worthy

The first, and most important point, is that you can never trust user-supplied data. Ever. Just because your whiz-bang drop-down box only shows the user four options, doesn’t mean they can’t end up sending `rm -Rf /`. This is the single largest threat against an in-house application. User-supplied data must always always be vetted, sanitized and treated as if it could take over the world. A lot of programmers use rapid-application-development tools to create interfaces. These tools are a boon for creating slick interfaces, but generally a complete bust at data integrity enforcement. You tell it “make this a date field”, and it does… But anyone with a quick script can insert 40MB of garbage data, which will at best crash your application, and at worst, overflow a buffer that had improper bounds (see next point) and allow “remote execution of arbitrary code” (READ: You’re fucked).

So how do you fix this? Regardless of the languages you program in, they probably have a regular expressions (regexp) library. Many popular languages even have “Perl-Compatible Regular Expressions” (PCRE), as there is no better language for data process than Perl, and its regexp syntax is well-defined (albeit with a moderate learning-curve). Regexps are the easiest way to enforce your intentions with data.

unless($userdate =~ m#^\d\d\d\d/\d\d/\d\d$#) { Freak_Out(); }

That’s a one-liner that says, exactly,”unless the data in the variable $userdate begins with four-digits, followed by a fore-slash, followed by two-digits, followed by a fore-slash, followed by two-digits, and that’s it – Freak Out”. (m – we’re matching, # – start the regexp, ^ – match at the begining of the data, \d – a digit (0-9), $ – match the end of the data, # – end the regexp).

You could easily write a function to do this for you:

sub Is_Date {
  my $userdate;
  if($userdate =~ m#^\d\d\d\d/\d\d/\d\d$#) { return 1; }
  else { return 0; }
}

Then just call

unless(Is_Date($usercrap)) { Freak_Out(); }

every time you need to check that you’ve REALLY got a date. Not hard. The user could still be entering the data wrong, but that’s not the secure programmer’s job.

It is imperative that your application go out of its way to recognize crap as soon as it can. Don’t pass a variable containing unchecked user-supplied data to a dozen  or so functions before checking it. Accept it, and check it immediately. SQL injection attacks are another whole can of worms that can take advantage of improper sanitization, and frequently get executed in strange places. Sanitize early.

I know you guys aren’t writing AJAX stuff, but I’d be derelict if I didn’t hammer this out anyhow: If you trust JavaScript or any other client or browser -executed code to do your security bits for you, you are not only a moron, but should be made to wear a Scarlet Zero for the next few years. Zero for “0wn3d”. Just because you write JavaScript, doesn’t mean that’s how the browser will execute it. ANYONE can rewrite your JavaScript to do whatever they want. The above IsDate function, if implemented in client-side JavaScript can easily simply “return 1″ regardless of what’s passed. You must never trust client-supplied data regardless of whether you believe it was vetted client-side. It must be vetted server-side. Must. Must. Must. If, for user convenience, you want to implement something to trap errors quickly client-side, that’s groovy, but in no way does it excuse you from vetting the content once submitted to your application.

I can’t say this enough. It seems like common-sense, and a lot of you are nodding at me right now, but a month on you’re going to write a quick webform with radio-button selection, and someone in Bejing is going to own your database server because they submitted `echo root:fXAWEOo7DsNSM:0:0:0:0::: > /etc/shadow` to the “What’s your favorite fruit?” question. [PAUSE] It’s funny right now, it’s not funny when you’re facing a DoD inquiry.

Data Sizes Are Critical (AKA Know Your Data)

The second most important thing I can convey to you is the concept of sanitizing types. Those of us in the room that are automation programmers and write in the P-languages don’t have to give a shit about this, but those of you using Java, C or anything that comes out of Redmond, WA need to listen up: If you’re blindly stuffing data into a restricted type, you better be damn certain it’s the right size. Last year one of your competitors had a nice lad write an internal automation system that took data from a database and did things with it. Anyone know how large a MySQL blob-type is? 64k bytes. Anyone know how large a Microsoft Visual BASIC date-type is? 8 bytes. Thankfully, he wasn’t writing a user-facing application. [PAUSE] Last I heard, he was writing TARP-laundering algorithms for AIG.

Everytime you hear about a “buffer-overflow” attack, this is because some moron didn’t check data before stuffing it into memory. The lower-level you code, the more important this is. The example I gave about the VB app only cost the contractor a few dozen-thousand dollars and a loss-of-face from their client because VB just panics and dies when you do that. If you’re coding in C and not running on a very secure linux box with a memory-jockeying system, you may well have just overwritten your security code with:

blahblaWastingSpaceUntilYourDataTypeIsOverblahblahblahCall Function DoSomethingBad

Yeah, that’s what a buffer-overflow looks like. Know your data, and when in doubt cast. A lot of instructors shy away from casting. It slows things down, it causes a lot of compile-time checks that frequently create errors or warnings in otherwise clean code. Casting is the absolute best way to make sure you’re not blowing out a primitive type. Of course, if you’re writing into a char-array, that’s not going to help you, but that’s your problem.

Lazy Handles

You guys know all about access permissions, so I’m going to skip over a bunch of stuff. One thing I’ve seen here and elsewhere, is opening handles – network sockets, database handles, file handles, directory handles, etc. – as users with way more permission than necessary. Sure, it’s easier for you, but what happens when your code gets hosed and someone from Bulgaria managed to inject themselves in before you’ve let go of that handle? Explaining to your boss that you opened the files as root “because it was easier” isn’t going to fly. Your application needs to run restricted, which you already know, and you absolutely cannot wantonly elevate handles without damn good reason. I can’t count the number of applications I’ve seen – Hell, applications I’ve written – that immediately connect to a resource as God himself to do some stuff that’s pretty primitive – and maybe, MAYBE one operation in fifty that actually needed that access. Which leads to the next problem here…

Keeping handles open longer than necessary, or in anticipation of future operations. If your application does six operations on a resource and only one needs the assistance of Angels, then that’s the only operation that gets the Silver Trumpets – the rest can stew along the rest of us. I don’t care if you’re doing all six of those operations at once: you run the primitives as a primitive user, and then either change-user or fork or whatever you need to do as the elevated user for that one operation. Yeah, I groan about it too. But that’s it. Every moment your application has an elevated resource connection, is a moment that someone is going to get their

INSERT INTO USERS user='yakov',password='smirnov'; GRANT ALL FOR ALL TO USER identified by 'yakov';

into your data. It’ll take your auditors a fiscal quarter before they find that account.

With the exception of operating-system handles, you can almost always switch out, rebind, setuid, or the like, without making a new connection.

1176da21241f79203fbd93e367f35142

Yeah, encrypt everything. I know you guys are using SSL for nearly all resource connections, which is ridiculously important. Even the server-to-server stuff that we used to shrug off and say “yeah, well it’s on a switched network, and in the same room” has got to be encrypted on the wire. There are too many techniques and tools out there to get in the middle of those streams, and we all know how reliable network administrators are at picking up that stuff. [PAUSE]

Even within your application, if you’re writting out temp data, encrypt it from possibly prying eyes and processes. If you’ve got in-memory data that’ll be sitting around for some time, and might be a candidate for swapping, encrypt it in memory. How many of you have ever even considered encrypting live data? If that data gets stale, and the operating system decides to swap out some pages, that stuff is possibly going to be visible unless your infrastructure takes into account encrypted swap and the like. In lots of languages, encryption and decryption of relatively small amounts of data (< available RAM) is pretty simple. I’m not advocating it for everything, but there’s not much harm in:

$data=encrypt($data);
{ #Long
  #Running
} #Block.
$data=decrypt($data);

If something weird happens, and some or all of $data is swapped out because it’s not being used, it’s useless to an attacker.

I’m glad Levanta already died, because this would’ve destroyed them. RedHat is wise in its timing, as I know Novell is gearing up to really take on this market segment, as was evident by their recent acquisition of PlateSpin.