After reading e2 Lag Reduction Suggestions for Noders in my morning E2 daily report, a thought hit me:

Not everything needs a page generated.
The thing that caused me to think of this was my soft-linker script. It doesn't need that page generation to happen -- just a nice, simple softlink to be generated. (It's nice to be able to know whether or not a given link exists, but that's just topping)

Is there any way to cause this to happen? The XML interface helps some things, but it doesn't cover all areas. It also occured to me that for the few other areas where the HTML interface is unavoidable, it would be nice to pass a parameter to turn nodelets off, or somesuch.

nate sez: That's a good point. I've created the node null page to accomodate bots 'n' stuff which want to post information to the server, but don't need a page back. To use, include "displaytype=null" in your query string, and the server will return a "1" rather than a page.

Okay, here is how I see it: Everything does too much work, for unneeded things.

Think of this as a client - server model.

Everything2 (the web page) is a very, very dumb client. It loads everything, every time. Now I know other users is a cron job, and a there are a few other items that are cached, but one of the fundamental problems, IMO is that there is too much going on for each page generation.

And thus we need to build a smarter client. There are several ways:

  • JavaScript / hidden frames.. load partial data into hidden frames and then relocate the data with document.write. Not fun, not more than IE and Netscape, but workable. I'm working on a proof of concept, but i need several things.
  • Binary, platform dependant clients... A smart client that can marshall what it needs from the page, intelligently. Only load the writeup off of the DB. Only refresh the chatterbox. Only cast a vote. etc.


Benefits of binary clients: We already have a few... but for certain things. The javabox ticker N-Wing beat into working, cow of dooms massive sync operation... these are all examples of clients for certian things. We need one that does it all.

What a client writer needs from everything: Similar to "printable version" everything needs to be able to offer a stripped down version (with links) of each e2 item. Just that item. Like a clientable version of a WU, or a clientable version of the chatterbox. Client writers would need this so that we could actually parse the pages well enough. A small HTML change could whack out our more intelligent clients, if we had to parse all of it.

Smart clients would get a few benefits:
  • Idle detection. When it isn't being used, then don't update. That's pretty easy to look for.
  • Caching the homenode, and all of the usersearch items. People would not need to do a massive search on all their items.
  • Better editors, we could have a more feature-rich editor (although I am toying with one for e2s WU box)
  • Integrated, more kind node tracker. With these sorts of programs, we could queue up jobs when it would be least harmful to the server.
  • Only getting the node text for the majority of nodeviews.
  • Better messaging, etc.
  • Easier to implement "other" level powers. Things like vote queing (you're out of votes, but someone still deserves one), so they'll get one of tomorrows, when it comes around.


Admittedly, I am working on kind of a proof of concept for Win32 (my home platform, don't let the war start), but it is going to take some time, and unfortunately I don't have any short term solutions to this lag, as my perl is lacking right now, but getting a little bit better. Thoughts?

nate sez:You are correct -- Everything is an extremely dumb client which generates a lot of stuff for every pageload that hasn't changed (Other Users, Random Nodes, E Developer, etc). The problem is a this point we have to stick to the lowest-common-denominator -- putting out single pages with tables is a brute, but it's easier to code, more cross-browser compatable, and the output is much simpler than using layers/frames/JScript to finagle some smaller pageloads. I think the idea of a Win32 or GTK client that uses an XML interface is better than to get that high of a level of functionality from all the browsers. But, if someone produced a theme or a page which worked, I could be swayed. However, the effort involved in producing the client might not be worth the server-side savings: most of our server load is Guest Users -- how many of these people are going to bother D/L and installing a client when their browser works?

Installing a client would remove a lot of lag because those who DO NOT want to create a new user and would rather search around the site as a guest would be forced to download the client first in order to do so. And the "common" user wiil not want to download something just to look at a site, it would seriously cut down on all the guest users abilities to lag up the system. A user who is pissed at the site and its users can load the front page and then just hold down the shift key and click EVERYTHING that has a link to it. This would bog down his connection and ours yes, unless he had a good one. It's funny how hate takes much more energy to put forth than love does.

Log in or registerto write something here or to contact authors.