Endocrys has two primary modular components: Autocrys and Paracrys.

Autocrys is an extensible communication protocol atop XMPP. It governs the syntax of commands or queries sent to systems or groups, the responses of systems to those queries, how to manage their presence, and how to react to presence changes in others.

Paracrys is a database-driven deployment and configuration system. Paracrys allows module code and configuration data to be stored centrally and deployed to Endocrys nodes on-demand. Paracrys fully supports versioning, thus allowing changes to be rolled-back in the case of a major oopsie. How small can a Paracrys module be? Here’s an example that implements a command called ’shell’ that allows you to do, essentially, whatever you want on an Endocrys client:

BEGIN { $Endo::MODS{SHELL}++; $Endo::CMDS{SHELL} = \&shell; }
END { delete $Endo::MODS{SHELL}; delete $Endo::CMDS{SHELL}; }

sub shell {
 return `@_`;
}

Drop that puppy into the Paracrys MODULES table with some other data, issue a mass “fetch module SHELL; refresh;” command, and bingo, all of your systems now let you do very bad things. It’s that easy to create a command to do something… Hopefully something useful.

Of course you should note that there is no access control in the above code… How do we prevent Bad People from using our horrendously very bad shell command? That used to be managed by the Communication Masters using another database called EndoACL, but has been folded into Paracrys’ duties and drastically simplified. Each Endocrys client, when receiving the shell command, will now ask Paracrys if the user who sent it is authorized to issue that command. Previously, the clients never even received commands from users not authorized to send them, at great expense.

One of the major goals of the project originally was to have absolutely minimal dependencies on third-party code, so I reinvented the wheel in numerous places. Now that it’s mine again, those requirements are vapor and I’m ripping out large swaths of my code, and exchanging it for API calls into other code that is the de facto standard to do whatever. For example, I wrote a function that copies a file from one location to another. Ew. The File::Copy module is the Perl Way to do that, so that’s how we do it now. Less code I have to maintain, and less code you have to read to understand Endocrys.

Another major goal of the original project was absolute redundancy on all levels. With a requirement like that, I over-engineered what were called the Communication Masters (CMs) so that they heart-beated each other, transferred each other’s sessions, held elections to decide who was authoritative for which IP ranges, dealt with segmentation and partitioning, etc. All of this at the cost of highly-customized hybrid XMPP/SQL servers that weren’t readily upgradeable. Wednesday night I spent a lot of time diagramming, and tonight solidified the spec to separate the XMPP server from the SQL database, and rely on established high-availability tools like pen or an SLB appliance to ensure connectivity to a farm of XMPP servers if needed. Additionally, this separation has allowed me to use MySQL clusters for the Paracrys bits, which adds scary levels of redundancy to those very critical bits.

Lastly for this post, the entire ithread Endocrys implementation has been ripped out and replaced with EV and AnyEvent, and the Net::XMPP code has been replaced with AnyEvent::XMPP for one cohesive event loop that runs very very fast. Originally I envisioned an Endocrys client maintaining dozens of XMPP sessions while handling dozens of system events and receiving dozens of commands, so I stuck everything in threads, and allowed it to scream along on SMP boxes. While this works just fine, there is a LOT of extra complexity involved with sharing variables across threads, dealing with races, etc. and the benefits are dubious when compared against a good, strong, event-loop system. I’m not quite done yet, but the net loss should be about 30% of the main code modules, with reduced complexity for all sub-modules as well.

I don’t have an ETA as to when the code will be generally available, but I’ve had some pings from some bright people interested in hammering the retooled version in non-critical environments, so hopefully it will be this year.

This is one of my favorite products, and I’m more than a little excited to get it back and get it out. It’s been battle-tested for many years, and I’m very proud of it. I’m working on getting the code cleaned up and abstracted before releasing it under the GPLv2. Below are edited points from slides describing Endocrys and why you might be interested in it.

The Problems

Dozens… hundreds… of systems, physical and virtual, all going about their business. Then something happens: maybe a disk drive failed, maybe a process died, maybe someone ordered a ‘reboot’ or ‘halt’. Those systems don’t have a way of communicating that externally. There is no “Hey, I’m rebooting, BRB” in the server world.

Dozens… hundreds… of systems, physical and virtual, all going about their business. Then you have a question: How many of them have Western Digital harddrives listed in a recent recall? How many of them are running <2GB of RAM? How many of them are running a certain version of some software listed in a security advisory? There’s no way to ask that question to the farm. There is no “Dear Lazyweb, answer this question for me” in the server world.

The Purpose

At its most basic level, Endocrys is a conduit between all of the systems and you. Think of it like a gigantic Instant Messaging buddy list, where all of your buddies are systems. When they’re online, they are in the list and can set their status messages, send you messages, send each other messages, receive messages, etc. Endocrys leverages the eXtensible Messaging and Presence Protocol (XMPP) to tie this framework into existing clients, transports and APIs, enabling a near-infinite number of possible applications or functions you can deploy.

The Technology

Endocrys is built as a framework – an abstract set of rules that can be extended at any time by writing little modules. These modules can be applied across the Endrocrys network instantly, without any downtime.

By leveraging XMPP, the Endocrys network is highly-redundant with no single fail points. Any number of “Communication Masters” (XMPP servers) are online, but only one is needed to keep communication flowing. All network communication is encrypted and signed. Partitioning and segmentation is handled rationally.

Communication is very similar to Instant Messaging, there is relatively no latency, and XMPP assures delivery even to systems offline when the message was sent.

Monitoring and control systems can participate on Endocrys, automating the remediation of problems remotely and automatically.

The Protocol

XMPP sits atop TCP, and atop that sits the Endocrys Communication Protocol aka Autocrys. ECP is a fully-authenticated, fully-controlled, skeptical protocol that serves both for sending structured announcements as well as sending and processing commands. The entire ECP specification is listed in AUTOCRYS.TXT. Endocrys agents can be written in any programming language, attached to any other framework, at any OSI level, as long as they can speak XMPP and implement ECP appropriately.

I certainly empathize with your password management situation. Passwords are, actually, horrible security mechanisms and it is my opinion that they should be done away with altogether. Problem solved: No passwords means no password management headaches.

So, how to do prove you’re who you are? How do your systems trust who you say you are? A token. A “key”. A physical and logical item possessed by the user. Something they can lose or get stolen or drop in their coffee mug, but doesn’t matter because it’s useless without them leashed to it- and can be reproduced by authorized personnel in a jiffy.

The security industry likes calling it “two-factor authentication”: The two factors being something you have (the token) and something you know (the sentence uttered by your first girlfriend when she dumped you, song lyrics, the title of a book … whatever). Behind the scenes we shift from password management (gross and abhorrent) to key management (fun and exciting!)

Encrypted-key security is the only managed authentication scheme I have rolled out in client environments for the last 7…8 years. It can be “difficult” to wrench into an existing infrastructure, changing the culture, disrupting the status quo- but technologically is a vastly superior solution to identity management.

The defacto standard is PGP [1], although there are a lot of players in this market with varying quality of products, some aiming at various vertical markets. The link below gives a nice picture of how various systemic pieces tie together.

I know I didn’t answer your question- people tell me that a lot- but I can’t in good faith recommend password management. I haven’t been able to since 1999 or so, and certainly can’t as 2009 winds down. Sure, there are things you can do – the DoD uses the Mandylion [2], which you can buy on ThinkGeek [3] for $50 – but it doesn’t solve the actual problem of secure identity management: Please pardon the crudeness, but it’s like putting whipped-cream on dogshit.

[1] http://www.pgp.com/products/index.html
[2] http://www.mandylionlabs.com/
[3] http://www.thinkgeek.com/gadgets/security/91a2/

RooCluster is a command-and-control application designed for the special needs of  multiple robots operating in the same space, or over large multi-room spaces. Each Roomba is being fitted with an RFID tag, which, in coordiation with some more wireless access points, allows me to triangulate where a Roomba is and its travel vector (sometimes, math is cool). This information can help RooCluster avoid nasty Roomba-on-Roomba collisions, and also presents the possibility of meta-virtual walls.

If you have a Roomba, you probably have a virtual wall – the little pylon that sends out an infrared beam that the Roombas treat just like a wall. With some work, RooCluster should be able to honor coordinate-based lines (which could, in turn, form other shapes) and effectively “wall-off” areas without needing a physical barrier, or a battery-sucking virtual wall. You can also overlay the position and vector data onto floorplans, and see exactly where the Roombas are, and where they’re going.

Of course, you can also use it to make your Roombas dance with each other.

Or joust.

Over the next five years, gross storage needs will double every other year, sparked by industry trends that avoid deleting anything, ever; continued bloat in software programs; increased user demand for larger-file storage; increased user demand for indefinite storage; increased user, corporate, and industry expectation of system-side backups and frequent snapshots; and the enabling factor of meteoric-disk-size -to- paltry-disk-cost ratios.

Since the late 1990s, we have seen rapid acceleration of infinite data life. While storage vendors will use terms such as “information life-cycle management”, “information archiving” or “data warehousing” – they all converge onto the premise that corporate data life is no longer finite. The value of this is dubious, but irrelevant to argue: financial workers expect to be able to look at historical data for modelling purposes; draft and product workers expect to be able to look at long-dead projects that might now be of value with new knowledge; in the throes of bankruptcy, competent managers (and lawyers) will want to mine the archives for something… anything that may provide some value. Everything your organization has ever known is expected to be retained, indefinitely.

The average 10-page MS Word document in 1995 was 13K in size. The average 10-page MS Word document in 2006 is 1.4MB. While that size may still seem small, it’s indicative of a growing trend of software generating vastly wasteful content because they can. Software vendors don’t need to worry about their data fitting onto floppies anymore, so they don’t. Multiply this across dozens of applications, add in media, and you have truly huge data files with only a few pages of actual content.

Similarly, the users want ever-larger files. Gone are the days of compressing graphics, video and audio to the Nth degree: users want full-quality content. They don’t want a 120×120 “thumbnail” video, they want something that takes some real-estate on their oversized monitor. As bandwidth increases, so will the user-desire for better content faster. They then want to save that same content to their network volume. They want it backed up in case of catastrophe (or their own error). What was a 3MB MP3 file is now a 45MB FLAC or WAV file sitting in your database.

The increase in user-end space (desktop harddisks) has led users to demand not only more and more space from their storage providers, but also indefinite storage. Users no longer have to selectively delete their e-mails to stay in a predefined space, so they keep them all, forever. They expect the same from the rest of their digital attics: they expect every bad poem, doodle, patent-idea-on-a-napkin, picture of their grandkids, etc. to be immediately available, forever.

Forever. Even if your disks die. Even if they accidentally delete them. Even if a meteor pummels your datacenter. The old standard of weekly backups have long passed the borders of Being Prudent, travelled through the Fields of Marginally Acceptable, and have entered the Mountains of Irreparable Harm to Your Reputation. Users, customers, regulators, etc. are barely tolerant of losing a day of data, and this will get worse. In the next half-decade a truly monumental shift into multi-media backups, near-real-time data snapshots, and 100% protection of data assets will be fully realized, requiring several multiples more mixed-media backup storage than live data storage.

On the up-side, disk sizes are sky-rocketing, costs are plummeting and the reliability of the new serial ATA (SATA) architected drives have come up to a level that allows anyone to build in or expand networked disk with a trivial investment. A new generation of storage vendors are coming up and challenging the old way of thinking about networked storage, and adopting technologies with more agility than their behemoth competitors. We’re quickly on our way to 1TB disk drives, flash-based storage continues to be refined and is nearing enterprise-grade, holographic storage is being commercially realized for some applications, and all of these technologies are driving the cost per megabyte down.

Over the next five years, datacenter bandwidth will level off for a bit. With the 10GigE standard behind us we can finally pull our backbones up to a level where they’ll be able to breathe easier for a while. Storage speeds are still being gated by the storage devices themselves, and until either solid-state media becomes cost effective or disks rotate twice as fast as they are now, that isn’t going to change much. Aggregating virtual systems is actually causing an interesting bandwidth phenomena that I’ll address later. Regardless, a 10Gig, or Nx1Gig backbone should be able to breathe well for the next half-decade. Planned year-over-year demand increases of 5-7% should be expected.

Desktop network speeds have been about the same for the last five years, and will largely remain unchanged. A 32-bit computer system running a commercial desktop operating systems has too many architectural limitations, still, to be make use of more than 60-85Mb/s of bandwidth. While some vendors are running 64bit processors, they generally are using bus architectures that aren’t that wide, thus gating peripheral speeds back to 32bit. In the next five years that will clean up a bit, and 64bit “extensions” to the 32bit processors will become more common place, but still not impacting the network noticeably due largely to OS and bus architectural issues.

Environments consolidating onto virtualized systems are seeing an interesting gross decrease in datacenter network bandwidth use. Not surprisingly, they’re also seeing peak utilization well above what they had prior to consolidation. The latter is easily explained by virtualized systems generally “netbooting” their OS from the storage network or a bootserver, and now more than ever embracing networked storage completely. The gross decrease has been unexpected because of the higher demands on the network, but is explained by architectural constraints. We’re now seeing 10-15 virtual servers sharing one or two network connections, where previously each had one or two of their own. This has somewhat of a levelling effect on network use, but isn’t dramatically impacting service performance as one would expect. The network is more important in these environments, but as a whole not as taxed.

It was largely believed that mobile “broadband” availability and use would be much higher by now, but we have yet to see a real platform for use. The Palm Treo series is getting an overhaul “soon” and rumored platforms by Google and Apple may change that landscape. In general, even if fully realized, the network demands by these users will largely have no impact on the greater network, or on datacenter network needs. The next-generation, “4G”, will be changing that, but I don’t expect to see that kind of horsepower in a phone until late-2010-to-2012: the processors are still just too slow.

What will change dramatically will be the bandwidth access for remote users. While not directly impacting the datacenter we’re going to see dramatic growth in the cable/DSL/satellite “broadband” space. Internet-facing applications may see a 20-30% rise in client demands as users become less tolerant of waiting for application loads due to their expectations of “faster” service, on the order of 200-250% more bandwidth. It is expected that OSP asymmetrical provisioning will continue.

The hosted systems industry has turned another critical point. Several years ago we eschewed large mainframe systems in exchange for commodity servers that could divide load and work together to provide services without single-vendor lock-in and without a single piece of “iron” waiting to fail. The computing power of a $2,000,000 mainframe was dwarfed by the implementation of $80,000 in commodity hardware. With virtualization coming-of-age- with Intel and AMD putting hooks into their processors and chipsets to allow virtualization to be fully realized and not just a software-only hack- we’ve seen those same commodity systems hosting dozens of virtual systems reliably and at near-metal efficiency. The cost per virtual system is a number rapidly approaching zero.

New offerings from Sun, IBM and HP/Compaq are emphasizing something that “the server guys” haven’t needed to care much about: infrastructure. Historically, your network engineers and analysts worried about interconnection, route redundancy, and ensuring the bits could flow where they needed, reliably and sufficiently; and your system engineers worried about everything up to the point the bits hit “the network”. Moving forward, that is almost a debilitating dichotomy. Traditionally, in the post-mainframe era, a physical system did one or two things and its exclusion from the network or its under-performance on the network was a minor issue. With a physical system possibly hosting dozens of virtual systems- all with unique networking requirements, cross-talking requirements, and of course: networked storage requirements- your system engineers must be well-versed in network engineering. “The Network Is The Computer” is not just a Sun tag-line, or a lame cliche’. We’re now fully realizing the potency of that statement. Every system offering from the Big Three contains significant “infrastructure” features: Network features.

By pushing more and more network features into server systems- IBM servers with Cisco “swrouters” built-in, for example – the server itself has become more important and less relevant at the same time. Keeping it up and running well will require a new kind of system engineer because “the box” is now more complex: But at the same time, collections of “boxes” should be able to self-heal and adapt to the failures of others. Each system has now become disposable.

A large swath of the architectural literati are already deploying quantities of self-healing farms that take over the work – the very virtual machines – of failed or failing physical systems. Virtualization on its own wasn’t a game-changer. Virtualization with processor support and recognition sparked real potential. Virtualization on top of “infrastructure”-aware (e.g. heavily networked) physical systems has dramatically shifted the value of hybrid “networked systems engineers”, raised the bar for the “server guys” to get up to speed on the real internals of networking, and has provided the unprecedented opportunity to deploy redundantly resilient systems that can in-practice achieve five-to-seven “nines” of reliability.

Cloud computing fosters intense competition.

This is not just wrong (people are wrong all the time, and I don’t blog about it), it’s dangerously wrong. Nowhere in technology have we had less competition potential, and more monopoly potential than in the realm of cloud computing. Why? Cloud computing is all about size and marketing. I remember back in the day when I was happily using WebCrawler to scour the Intertubes for what I wanted to find, when some jerks from Stanford started some crappy search engine called Google. Pffft. “There’s tons of competition in the search engine market”, the analysts said. I was concerned then, I’m concerned now.

Any centralized Internet service is about size and marketing. Read up on my last post if you want to know how I feel about centralized services in general. Right now there are three major players in the “cloud computing” market. That will grow, as did search engines, to probably 20 major players and everyone will think I’m nuts, and then they will be slammed against the wall by One.

One that claims not to be evil.

One that believes (truly) they’re doing good.

One that rules them all.

This is not some psychic prediction, this is the recognition of a pattern that has played out too many times in the young life of the commercial Internet.

Yet humans are congregational, fashionable beasts. Humans, in general, want to be where there are other humans- even virtually. They want to be a “part of something”. They want to be on Facebook or MySpace. They want an iPhone. They want a Gmail account. They want to be LinkedIn. They love fads and the feeling of faux exclusivity that they bring, like Twitter. They fight against distribution.

The Fallout of Centralization

When Gmail is offline for a few hours, it makes international news. Millions of people across the globe are without e-mail! Crisis! Panic! When Facebook “gets a virus”, it’s international news! Millions of people are at risk! Crisis! Panic! When Twitter gets knocked offline, it’s international news! Millions of people can’t tell each other what they’re eating! Crisis! Panic!

Why won’t they get their e-mail from a more local provider? Why won’t they get other services from a more local provider? Why do people by Macs? Why do people like malls? They fight against distributed designs, and try to pull things together. “One-stop shopping” to a network neophyte is all giggles and rainbows- To a network architect it’s “bad design”.

Internet history is littered with fads and centralized solutions that inevitably fail to retain attention after the Next Big Thing arrives: But distributed solutions tend to last much much longer. Why?

The Argument for Centralization

There’s only one reason Google loses more money than my lunch cost every time someone watches a YouTube video: Control. Companies offer centralized services so they can control their users, their content, and their market. Facebook doesn’t care about user privacy or quality services, they care about making money (or, in their case, losing less money). They want you to be on their site as often and for as long as possible, and they’ve already sold your information in order to fund more gizmos and whirligigs (”apps”, they call them) to keep you coming back and sticking around. Control.

If Facebook was distributed – anyone could put up their own Facebook-compatible site, and allow other geographically-similar people to use it – how would they exercise control? How would they censor you, collect your personal data and sell it? How would they push you the latest version of your favorite whirligig? How would they profit off of you (or prevent someone else from profiting off of you)? Control.

While Gmail is a centralized interface to a distributed system (collectively known as “e-mail”), they’d just love it if everyone used Gmail and there was no distributed e-mail system. They mine your e-mail for keywords to target ads at you, making truckloads of money at the expense of your privacy- Ads that companies pay a premium for because Google can already “guarantee” you’re interested. Control.

The Argument for Distribution

Keeping components small, simple, and “close” is a doctrine of many disciplines, and is imperative for “critical infrastructure”. I have more than one client that intentionally buy equipment from multiple vendors, to reduce risk of single-vendor product failures. Several only update a fraction of systems and application software to new versions at a time, to reduce risk of debilitating bugs (one client, literally, has some 8 year-old operating systems running unpatched for this reason). More, and more institutions have multiple WAN (”Internet”) connections, from multiple providers, to reduce risk of single-provider failures.

Distributed infrastructure is more about “us” (the global network users)  than “you” (the individual). You don’t care whether Gmail (with a few million users) is down, or whether your ISP (with a few thousand users) is down: all you care about is that you can’t get to your e-mail. Because of this, it’s hard to convince “you” that distributed is better. You, the individual, don’t care about network survivability or how many other people are impacted- you only care about yourself, and your inability to access your mail.

Would you like to travel to your State capital to go to a centralized hospital? How about to the National Library to check out a book? When it comes to travel-related logistics, most people go “duh, local hospitals and libraries and schools and whatnot make tons of sense”: But once they go online – once that facade of geography is lifted – eYouSpaceFaceGoogleTwitterBookMailBay.com really is the best thing in the world, OMG?!!

HOWTO

So what can we do? What can you do? Appreciate and utilize local, distributed infrastructure. Don’t outsource/offshore your new whizbang application when your local hosting provider can handle it. Limit your dependence on centralized services, and find distributed substitutes where needed. Pretty much every infrastructure application has a distributed counterpart, many of them with much much less evil than their centralized cousins. I’m not saying “don’t go to website X” – I enjoy Wikipedia and CNN and all sorts of centralized information sources – but I don’t depend on them for communication, and I certainly know other places to go if they’re offline.

Distributed service architectures are about freedom and survivability. Centralized service architectures are about captivity and control. Happy Birthday, Internet. I look forward to another 40 years of architectural ingenuity.

Users Are Un[trust]worthy

The first, and most important point, is that you can never trust user-supplied data. Ever. Just because your whiz-bang drop-down box only shows the user four options, doesn’t mean they can’t end up sending `rm -Rf /`. This is the single largest threat against an in-house application. User-supplied data must always always be vetted, sanitized and treated as if it could take over the world. A lot of programmers use rapid-application-development tools to create interfaces. These tools are a boon for creating slick interfaces, but generally a complete bust at data integrity enforcement. You tell it “make this a date field”, and it does… But anyone with a quick script can insert 40MB of garbage data, which will at best crash your application, and at worst, overflow a buffer that had improper bounds (see next point) and allow “remote execution of arbitrary code” (READ: You’re fucked).

So how do you fix this? Regardless of the languages you program in, they probably have a regular expressions (regexp) library. Many popular languages even have “Perl-Compatible Regular Expressions” (PCRE), as there is no better language for data process than Perl, and its regexp syntax is well-defined (albeit with a moderate learning-curve). Regexps are the easiest way to enforce your intentions with data.

unless($userdate =~ m#^\d\d\d\d/\d\d/\d\d$#) { Freak_Out(); }

That’s a one-liner that says, exactly,”unless the data in the variable $userdate begins with four-digits, followed by a fore-slash, followed by two-digits, followed by a fore-slash, followed by two-digits, and that’s it – Freak Out”. (m – we’re matching, # – start the regexp, ^ – match at the begining of the data, \d – a digit (0-9), $ – match the end of the data, # – end the regexp).

You could easily write a function to do this for you:

sub Is_Date {
  my $userdate;
  if($userdate =~ m#^\d\d\d\d/\d\d/\d\d$#) { return 1; }
  else { return 0; }
}

Then just call

unless(Is_Date($usercrap)) { Freak_Out(); }

every time you need to check that you’ve REALLY got a date. Not hard. The user could still be entering the data wrong, but that’s not the secure programmer’s job.

It is imperative that your application go out of its way to recognize crap as soon as it can. Don’t pass a variable containing unchecked user-supplied data to a dozen  or so functions before checking it. Accept it, and check it immediately. SQL injection attacks are another whole can of worms that can take advantage of improper sanitization, and frequently get executed in strange places. Sanitize early.

I know you guys aren’t writing AJAX stuff, but I’d be derelict if I didn’t hammer this out anyhow: If you trust JavaScript or any other client or browser -executed code to do your security bits for you, you are not only a moron, but should be made to wear a Scarlet Zero for the next few years. Zero for “0wn3d”. Just because you write JavaScript, doesn’t mean that’s how the browser will execute it. ANYONE can rewrite your JavaScript to do whatever they want. The above IsDate function, if implemented in client-side JavaScript can easily simply “return 1″ regardless of what’s passed. You must never trust client-supplied data regardless of whether you believe it was vetted client-side. It must be vetted server-side. Must. Must. Must. If, for user convenience, you want to implement something to trap errors quickly client-side, that’s groovy, but in no way does it excuse you from vetting the content once submitted to your application.

I can’t say this enough. It seems like common-sense, and a lot of you are nodding at me right now, but a month on you’re going to write a quick webform with radio-button selection, and someone in Bejing is going to own your database server because they submitted `echo root:fXAWEOo7DsNSM:0:0:0:0::: > /etc/shadow` to the “What’s your favorite fruit?” question. [PAUSE] It’s funny right now, it’s not funny when you’re facing a DoD inquiry.

Data Sizes Are Critical (AKA Know Your Data)

The second most important thing I can convey to you is the concept of sanitizing types. Those of us in the room that are automation programmers and write in the P-languages don’t have to give a shit about this, but those of you using Java, C or anything that comes out of Redmond, WA need to listen up: If you’re blindly stuffing data into a restricted type, you better be damn certain it’s the right size. Last year one of your competitors had a nice lad write an internal automation system that took data from a database and did things with it. Anyone know how large a MySQL blob-type is? 64k bytes. Anyone know how large a Microsoft Visual BASIC date-type is? 8 bytes. Thankfully, he wasn’t writing a user-facing application. [PAUSE] Last I heard, he was writing TARP-laundering algorithms for AIG.

Everytime you hear about a “buffer-overflow” attack, this is because some moron didn’t check data before stuffing it into memory. The lower-level you code, the more important this is. The example I gave about the VB app only cost the contractor a few dozen-thousand dollars and a loss-of-face from their client because VB just panics and dies when you do that. If you’re coding in C and not running on a very secure linux box with a memory-jockeying system, you may well have just overwritten your security code with:

blahblaWastingSpaceUntilYourDataTypeIsOverblahblahblahCall Function DoSomethingBad

Yeah, that’s what a buffer-overflow looks like. Know your data, and when in doubt cast. A lot of instructors shy away from casting. It slows things down, it causes a lot of compile-time checks that frequently create errors or warnings in otherwise clean code. Casting is the absolute best way to make sure you’re not blowing out a primitive type. Of course, if you’re writing into a char-array, that’s not going to help you, but that’s your problem.

Lazy Handles

You guys know all about access permissions, so I’m going to skip over a bunch of stuff. One thing I’ve seen here and elsewhere, is opening handles – network sockets, database handles, file handles, directory handles, etc. – as users with way more permission than necessary. Sure, it’s easier for you, but what happens when your code gets hosed and someone from Bulgaria managed to inject themselves in before you’ve let go of that handle? Explaining to your boss that you opened the files as root “because it was easier” isn’t going to fly. Your application needs to run restricted, which you already know, and you absolutely cannot wantonly elevate handles without damn good reason. I can’t count the number of applications I’ve seen – Hell, applications I’ve written – that immediately connect to a resource as God himself to do some stuff that’s pretty primitive – and maybe, MAYBE one operation in fifty that actually needed that access. Which leads to the next problem here…

Keeping handles open longer than necessary, or in anticipation of future operations. If your application does six operations on a resource and only one needs the assistance of Angels, then that’s the only operation that gets the Silver Trumpets – the rest can stew along the rest of us. I don’t care if you’re doing all six of those operations at once: you run the primitives as a primitive user, and then either change-user or fork or whatever you need to do as the elevated user for that one operation. Yeah, I groan about it too. But that’s it. Every moment your application has an elevated resource connection, is a moment that someone is going to get their

INSERT INTO USERS user='yakov',password='smirnov'; GRANT ALL FOR ALL TO USER identified by 'yakov';

into your data. It’ll take your auditors a fiscal quarter before they find that account.

With the exception of operating-system handles, you can almost always switch out, rebind, setuid, or the like, without making a new connection.

1176da21241f79203fbd93e367f35142

Yeah, encrypt everything. I know you guys are using SSL for nearly all resource connections, which is ridiculously important. Even the server-to-server stuff that we used to shrug off and say “yeah, well it’s on a switched network, and in the same room” has got to be encrypted on the wire. There are too many techniques and tools out there to get in the middle of those streams, and we all know how reliable network administrators are at picking up that stuff. [PAUSE]

Even within your application, if you’re writting out temp data, encrypt it from possibly prying eyes and processes. If you’ve got in-memory data that’ll be sitting around for some time, and might be a candidate for swapping, encrypt it in memory. How many of you have ever even considered encrypting live data? If that data gets stale, and the operating system decides to swap out some pages, that stuff is possibly going to be visible unless your infrastructure takes into account encrypted swap and the like. In lots of languages, encryption and decryption of relatively small amounts of data (< available RAM) is pretty simple. I’m not advocating it for everything, but there’s not much harm in:

$data=encrypt($data);
{ #Long
  #Running
} #Block.
$data=decrypt($data);

If something weird happens, and some or all of $data is swapped out because it’s not being used, it’s useless to an attacker.