What is the plan?

First up, sorry I've been away. It's been difficult trying to balance things, but I'm committed to the long-term future of E2. I've never been super far away, and have managed the operations and business, but haven't spent a lot of time on dev. I've also needed to really get my head around some web concepts and I'm doing more managing than coding these days, but I'm back to having focus and that focus has helped me to see the path forward. I'll go through the places where I've tried and now we have a straight path to progress.

That behind me, I have a plan that's finally well-formed with actual items that I can chop into a project plan. It also is good because if other people want to help throw in on development we'll be able to articulate where those needs are. The high level plan looks like:

  • TLS (https://everything2.com)
  • Small COGS items
  • Real REST APIs
  • Web platform refresh (mod_perl 2.0, Jessie, NodeJS)
  • Single-page app w/javascript routing
  • Backend cleanup
  • New features!

TLS - As I said in this month's root log, TLS is incredibly important feature for the site. TLS stands for Transport Layer Security (the sucessor to SSL) and is being pushed by browsers and search engines as a validating piece of technology such that non-TLS sites get penalized. There's a TON of built in assumptions to the site that it would be http forever, so it's going to take some time to chase all of those down. In a week's time we're going to force everyone to use TLS forevermore and that should improve our search engine rankings over time.

Small COGS items - COGS stands for cost of goods and services and is something I constantly need to be worried about. We need to reduce our S3 bill some and take human-based backup manicuring out of the loop so I'll get that done. It is some small changes to the way we handle backups and is nothing major. As a fun fact, I have backups from this site going back to 2004 or so. This is going to take a day or so of solid development.

Real REST APIs - Here's the first real long pole in the future of E2. We need to expose the site to JavaScript in a more sane way, and the way to do it is to write real REST APIs. This will allow me to stitch a better experience together and do the real MVC separation. The site code is basically a ton of ultra-heavyweight controllers, and we need to create versioned, documented REST APIs so that we can create a front-end client for it that makes sense. This means having an API catalog that looks like:

  • /api/session/create
  • /api/session/destroy
  • /api/message/create
  • /api/message/651
  • /api/message/651/archive
  • /api/message/651/unarchive
  • /api/node/754
  • /api/cool
  • /api/vote
  • /api/nodenode
  • /api/trimsoftlink

As an example, session is pretty much done in my dev environment and I'll be putting it up once TLS is done. I'm undecided on a few of the technical pieces on the API, like how the client is ultimately going to handle superdocs, but the technicals are that we are going to throw away the opcode structure in favor of API creation. Once an API is created, the opcode use path will be swapped over to use that, and we'll eventually throw that away. That means that maintenances will eventually be gone as well, since they are essentially extensions of the nuke, update, and new pseudo-opcodes

At this point, Javascript is going to come out of the codebase, and be actually inside of git where it can be handled by the production push mechanism. Anything inside of the database is actually kind of hard to test, because it either needs the crazy-specific environment of the app or needs to be handled in this jigsaw update kind of way where I push a backwards-compatible version of a change, update the engine, and then remove the backwards compatibility. It's a time consuming process, and there's currently no good facilities for updating JavaScript other than editing a jscript node.

Web platform refresh - Currently, we're running on Debian Wheezy, and I'd like to move over to Debian Jessie or an Ubuntu variant. There are two primary reasons why: mod_perl 2.0 is out, which should put us in a better shape than the hacky version that we're on now (mod_perl with Bugzilla's changes), and the availability of NodeJS. This is a lot of work on the back-end for nearly no front-end development, but will keep us secure and reduce some complexity that's built-in right now. It also enables the platform we need for next-generation development, including a modern version of Apache 2.4. We'll also do a mysql version upgrade around this time.

Single-page Javascript app - A single-page JavaScript app is going to make the page much more responsive and allow us to really push front-end features in a way that we can't right now. The engine paradigm underneath E2 has shown its age for a very long time, and the volunteer coder crew has done a ton of incredibly novel things to polish up both the UI and the page routing. There's a lot to this, including an entirely new layout that we're going to phase in as the APIs get written and consumed. Eventually JavaScript will handle the page routing, and we can clean out a ton of the cruft inside of the engine.

I'm pretty settled on React.js as the web controller driver, as it maps incredibly well to the way E2 thinks about its object model. It's going to require Node.js as the compiler, and I need to get a bit smarter on it, but that will fit into the whole deployment model. This will end the alternate .css themes as we know them, but I only really test on Kernel Blue so it's reasonable to expect those to degrade over time.

Backend cleanup - Once Everything::HTML is no longer driving website navigation, and we are mostly clients that are consuming APIs, then I can do some code cleanliness exercises and fix up how we handle some of the backend stuff. This means that since there will be single entrypoints to node creation, we can fix up and make node hashes into real Perl Moose objects. This isn't strictly necessary and I am inclined to do new features first to inject some energy.

For far too long, I've toiled with ways to do the backend cleanup stuff first, and the real problem was how entirely entangled it is to Everything::HTML. The fact that the grand majority of the site code by LOC was in the database certainly didn't help, but I have been pulling code out of the database left and right, and only really containers and superdocs remain in any volume. This isn't a big deal and rather than attempt to flatten or compile them any more than I already have, I can keep working on the items that are hard and obnoxious. Containers are actually kind of a horrible performance killer, so once they are gone, the page will actually render much faster. Right now we are doing tons of regexes on the server to handle the controller nesting CONTAINED_STUFF. This of course is silly in the modern age. I had a solution worked up in a script which generated subroutines from the controller snippets, but I'd just have to throw that away after we went to APIs anyways, so I'm not going to bother refining that anymore.

New Features! - E2 has been on feature lockdown for far too long. I've long talked about some of the things that I want to do, but once we see how complex the API snarl gets, we'll have an idea for what the art of the possible is.