Hi, I'm kthejoker. You may know me from such E2 coding projects as the Wheel of Surprise, My Achievements, and that thing that's horrible and needs to be undone or the site will collapse any day now. You know the one.

I'm here today to help you learn the basics of what it takes to code for E2, and some of the basic tips and techniques to make your transition to E2 coder a smooth one. To help us out, we're going to be using the Epicenter nodelet as our chief document and reference guide. If I start talking about "the code", that's what I'm referring to.

The Basics: Perl

Before we get started diving into the code of Everything2, let's be clear: if you want to code in Everything2, you have to know some Perl. There are a lot of books and websites on Perl and its labyrinthine syntax and methods, but we're just going to cover the nitty gritty here. Basically, you need to know 4 things: variables, operators, functions, and loops.

Variables

The first thing you need to know about Perl variables in E2 is that we use strict mode, which means you must instantiate each variable in your code with the "my" syntax. This keeps everything in scope for the interpreter. We'll see how this used below.

There are three basic variable types in Perl: the scalar ($), the array (@), and the hash (%).

• The scalar is just a single valued variable.

  my $num = 1;

  initializes a scalar variable called num and sets its value to 1. Notice we used the "my" to initialize the variable, and we ended the statement with a semi-colon - another Perl requirement.
• The array is a variable that can hold multiple values in a numerically-indexed .. well, array. Generally, these values are scalar in nature, but you can also have an array of arrays, or an array of hashes.

  my @array = (1, 2, 3);

  initializes an array and sets its first three values to 1, 2, and 3, respectively.
• The hash is the last variable type, and is generally much more important than arrays for E2 coding. A hash is a lot like an array, except its keys can be associative. So, for example,

  my %book = {title => 'A Tale of Two Cities', author => 'Charles Dickens'};

  creates a hash with two keys, title and author, and sets their values. You can then retrieve these individual values using the "$$" syntax - $$book{title} - or the "->" syntax - $book->{title}. When an E2 node is retrieved from the database, it is returned as a hash - this is the basis for doing most of the dirty work.

Obviously, there's a lot more to all of this, but for now, this'll get you started. When looking through code, identify scalars, arrays, and hashes, and try to determine their purpose. Sometimes their name makes them obvious ($$NODE{title}) - other times, not so much ($isHappyPanda).

Operators

Lucky you, you've already learned one operator: the =. That, as you may have noticed, is the assignment operator - you use it to set variables to a particular value. There are also three other types of operators: math, concatentation, and comparison.

• Mathematical operators are pretty straightforward: +, -, *, and /. But there are also some shortcuts - for example, $num++; will increase the value of num by 1. $num += 100; will increase $num by 100.
• The chief concatenation operator is the period (.). This is used to combine strings together. So if you have

  my $word = 'Hello';
  $word = $word . ' World';

  , then at the end, $word is equal to 'Hello World'. This also has a shortcut, .= , which gets a lot of use here on the site.
• The comparison operators are a little trickier, because you have two different sets of them. The first set compares numbers and should be familiar: <, >, <=, >=, !=, and == (those last two are "not equal" and "equal to", respectively.) The second set compares strings and correlate with the numeric operators nicely: lt, gt, le, ge, neq, and eq. 

Again, not comprehensive, but should help you understand what if ($$NODE{title} eq 'Butterfinger McFlurry') means.

Functions

Functions and subprocedures are great tools in Perl, but here at E2 we don't generally write our own functions - at least not directly (more on htmlcodes later.) The E2 Perl modules come with a bunch of handy subs, though, that get a lot of use, so you should be familiar with the syntax of a function call:

functionName(parameter1, parameter2, etc);

Not every function has parameters, but if they are required, you'd better have 'em. Also, many functions return a value that can be used in your code. A simple example is

my $fearlessLeader = getNodeById(220);

getNodeById does exactly what you think it does, and in this case returns Nate to the variable. Then we do something like

return $$fearlessLeader{experience};

which will then print the returned value (Nate's XP) to the display.

Loops

Loops and conditionals are a fact of life in programming, but luckily they're also pretty straightforward to understand. There are three main loops and conditionals in use at E2:

• the if-else conditional. The syntax is

  if (someBoolean) {doSomething;}
  else {doSomethingElse;}

  The else statement is entirely optional - you can just say

  if (canVote) {castVote;}

  and be done with it. Also there is a shortcut for one line if statements - you can just do

  $num++ if $notDone

  and Perl will do as it's told. (You can also do $num++ unless $Done, but that's pretty rare.)
• the for/foreach loop. The for loop allows us to run the same code a certain preset number of times. The foreach loop goes through each item in an array or hash and runs code. So, same result, but different ways to implement a loop. This is useful when, for example, printing every writeup under an e2node, or printing the top 20 Staff Picks.
• the while loop. The while loop syntax is

  while (someBoolean) {doSomething;}

  and does what you would expect: while someBoolean is true, it keeps running the code. In this case, you can run into an infinite loop unless you include code that sets someBoolean to false somewhere within the loop.

I'm kind of glazing over these, because really reading an example is the best way to understand what's going on, which we'll cover in our next couple of sections.

Overview

So we've covered some of the basics of Perl. As we're going through code, you may find yourself asking what this or that bit of code is doing. Always try to break it down in terms of the variables, functions, loops, and operators being invoked. E2 isn't exactly a bastion of good documentation, so I often find myself backtracking from some error point until I get to the root variable or function that is really at issue. Being able to go from the big picture to the small detail is essential to understanding E2 code.

Next we'll talk about some of the basic syntax for E2-specific coding: the global variables, database calls, and htmlcode functions.

Tables and HTML validation

HTML in E2 writeups is filtered; only a subset of the available HTML tags are actually allowed. There are approximately two reasons for this:

1. Malice: to avoid allowing untrustworthy users to spoil the E2 experience for the rest of us by using scripts, images, and external links (to, eg. goatse.cx)
2. Functionality: Scripts or tables can quite easily break an E2 page, either by acccident or on purpose. For example, since layout is performed mostly with tables, a spare </tr> tag could wreak havoc with the layout of the rest of the page, particularly nodelets etc.

Tables would be fine if they could be validated to prove that they're well-formed and hence will not break tables outside the writeup. Tables in particular aren't allowed because:

• Table validation code was believed to be non-trivial to implement.
• It was believed that it'd be too expensive to implement in terms of CPU time on the web server

Also, another issue which was not discussed is that allowing tables in writeups might be "exploited" for forcing layout within writeups; eg. to split a writeup into columns, etc.

Having written some fairly simple code to validate table HTML, running some performance tests on it seems to show that the 'worst case' HTML (a nightmare of tables, taken from an entire pageload on Community2) indicates that the table validation takes about a fifth of the time the normal HTML validation takes.

Approach

The design of the table validation code emerges from a slightly different approach than is used for general HTML validation. The existing HTML validation code in ecore strips out disallowed tags to create valid HTML. Transforming the markup in such a way can be fairly expensive, and for tables could be quite complex to perform. Hence, a much simpler approach was adopted based on observations of writeup content.

1. Most writeups (currently all of them) do not contain any tables whatsoever.
2. Writeups which contain tables will mostly contain valid tables, and thus require no transformation to ensure valid and innocuous HTML is the result.

Thus the two cases for which performance should be optimised are the corresponding cases of no table tags at all, and tables which are valid and well-formed. Hence the main code merely scans the structure of the tables in the HTML and determines whether tables have valid structure or not. When a malformed table is discovered, the code adopts an entirely different approach (and one which may in the long run be more useful...) of rendering the table tags visible in the displayed HTML, and adding <div> tags with dashed outlines around the elements to aid debugging.

With this approach, the majority of the web server's activity will be in simply checking the validity of any table tags.

Here's the source to the htmlcode:

# Okay, in brief:
# fast 'cause it's optimised to the 'common' cases:
# Most writeups have no tables. Zoooom!
# Writeups that have tables will mostly have valid tables:
#   => Only a quick parse to validate.
# We 'enforce' the validity of tables by outputting debug info
#   for badly formed tables. This is UGLY so writeup authors will
#   fix 'em quick.
# In an HTMLcode, so compilation of this code is amortised.
# [screenHTML] should still be used, and can be used to control
#   attributes in the tags. Ideally this works on the output of
#   screenHTML, but only because the 'debug' output uses <div>s
#   with dashed outlines to help HTML writers find their oopsies.


# Should be reasonably fast: scans through the HTML using a m''g, which
# is about as fast as anything in perl can be. Stacks the tags (only
# looks at table tags) and checks the structural validity by 
# matching a two-level context descriptor (stack . tag) against
# an RE describing valid contexts. (again, perl and RE => faster than
# a bunch of ifs or whatever)
sub tableWellFormed ($) {
    my (@stack);
    for ($_[0] =~ m{<(/?table|/?tr|/?th|/?td)[\s>]}ig) {
        my $tag = lc $_;
        my $top = $stack[$#stack];

        if (substr($tag, 0, 1) eq '/') {
            # Closing tag. Pop from stack and check that they match.
            return (0, "$top closed with $tag")
              if pop @stack ne substr($tag, 1);
        } else {
            # Opening tag. Push, and check context is valid.
            push @stack, $tag;
            return (0, "$tag inside $top") 
                if (($top.$tag) !~ /^(table(tr)?|tr(td|th)|(td|th)(table))$/);
        }
    }
    return (0, "Unclosed table elements: " . join ", ", @stack)
        if ($#stack != -1);
    return 1;
}

sub debugTag ($) {
    my ($tag) = @_;
    my $htmltag = $tag;
    $htmltag =~ s/</amp;lt;/g; # should be encodeHTML, but of course
                            # I don't have that in my standalone testbench.
    $htmltag = "<strong><small>amp;lt;" . $htmltag . "amp;gt;</small></strong>";

    if (substr($tag, 0, 1) ne '/') {
        return $htmltag . "<div style=\"margin-left: 16px; border: dashed 1px grey\">";
    } else {
        return "</div>". $htmltag;
    }
}

sub debugTable ($$) {
    my ($error, $html) = @_;
    $html =~ s{<((/?)(table|tr|td|th)((\s[^>]*)|))>}{debugTag $1}ige;
    return "<p><strong>Table formatting error: $error</strong></p>".$html;
}

my ($text) = @_;
my ($valid, $error) = tableWellFormed($text);
$text = debugTable ($error, $text) if ! $valid;

$text;

A similar approach might yield speed improvements to htmlScreen too. It should be possible, for instance, to construct a single regular expression to determine if some HTML consists entirely of valid tags and attributes (and this regular expression can be constructed automatically, of course). However, since there's a large base of existing writeups that may have invalid tags and attributes that nobody has noticed (since the existing htmlScreen provides no real feedback), it's unlikely that this will be a particularly palatable idea.

As a last word, caching of computed results in ecore is something that could be done better, and more consistently; but we all know this by now, and it's late in the evening so I'm not going to tell you things you already know.

Y'all can try it out over at

http://community2.org?node=test+screenTable

or

http://kahani.org?node=test+screenTable

In particular, I want y'all to try and break it, defeat the validation to get unpleasant HTML through the validator. And remember that in the final version it will be used in conjunction with htmlScreen... ;)

Edevdocs are a problem.

Firstly, I've seen a lot of edevdocs around that seem to indicate that people aren't aware of the difference between edevdoc's and superdocs. So, you know, maybe we need a bit of documentation. Hey, it's just a thought.

Secondly... brackets. Brackets are, of course, a pain on Everythings in general, what with parselinks being used somewhat universally on content.

There is no simple, elegant way to denote a bracket that will survive a trip through parselinks. The standard way of passing a bracket through parselinks doesn't pass a bracket through at all; it just passes the HTML entity for a bracket, &#091;. This only happens to look a bit like a bracket when your browser renders HTML. As far as, say, JavaScript is concerned, though, it's not a bracket at all.

The 'solution' we currently have in place to this problem, is, in a word, wrong. The solution is that while normal documents are displayed with parselinks, edevdocs are displayed with devparselinks, an entirely different function.

This brings trouble when we have a javascript utility, such as E2 Bookmarklets or Writeup table formatting utility which we want to open up to the general E2 user population rather than just edev, the only real way to do this is to convert the edevdoc (viewable only by edev members) into a document, viewable by all. And here we can get bitten by the difference between documents and edevdocs.

E2 Bookmarklets uses its own scheme to get round the lack of brackets; if it had used the edevdoc scheme, it would be useless in a document.

There are ways to get round the lack of brackets, of course.

• construct any regular expressions using ASCII equivalents, and for array dereferencing, parselinks can be defeated by using, eg:

  array /*[*/ [i /*]*/ ] = f(i);

  This causes the HTML generated by parselinks to be encapsulated inside the javascript comment, while the following [ is thought to be inside the '[]' so isn't interpolated, and the closing ']' is thought to be an unmatched ']' so isn't interpolated.

  Please note, the technical term for this method is 'A Gross Hack', and if I find anyone attempting it, I shall kill them. Or... you know, laugh at them. Whichever I feel like at the time.

• encode the code (eg. with escape) and store in in a string, which can then be unescaped and evaluated. The edev JavaScript escaper will do this for you.

However, I would like to propose that we actually solve this problem, rather than simply bubble-pushing it. I see two solutions:

1. Create a publically accessible nodetype that uses devparselinks for its display page, which edevdocs can be transformed into to make them publically accessible. Naturally, for the same reasons that edevdocs can only be viewed by edev members, we would want to ensure that the new nodetype can't be created by non-deity edev members, but can easily be created from an edev member's edevdoc. Alternatively,..

2. Beef up parselinks so that it doesn't catch brackets which are inside HTML tags or HTML comments (a little tricky perhaps), so it ignores brackets inside properly commented-out scripts. Such a parseLinks might look a little like this:

       my ($field) = @_;
       my ($text) = $$NODE{$field};
       join "-->", map {
           my @a = split /<!--/, $_;
           $a[0] =~ s/\[(.*?)\]/linkNodeTitle ($1, $NODE)/egs;
           join '<!--', @a;
       } split /-->/, $text;
   

   Not quite as elegant as the current definition, but certainly does the job.

You may remember that, several months ago, we (edev) had a discussion regarding collapsible nodelets. A collapsible nodelet is a regular nodelet with a button that allows you to hide or display the nodelet contents. See a working example here. Collapsible nodelets would allow quick access to the nodelets you want, while keep your screen somewhat clean by giving you the ability to quickly hide or show the ones you refer to semi-frequently (without needing to go through User Settings).

If I recall correctly, most everyone liked the idea of collapsible nodelets, but there seemed to be no way to store info regarding which nodelets are collapsed without posting back to the server (not very fast at all). Since no solution appeared to this problem, the idea was pretty much abandoned. Today, however, it suddenly struck me to (duh!) use JavaScript to store data in cookies regarding which nodelets should be collapsed and which should not. I've got a working example running here.