edev

Purpose

To give the user an opportunity to confirm that they really want to do what the link they just clicked on/the form they just submitted is meant to do.

Implementation

1. For confirmation of opcodes, default javascript looks for links with class 'action' and parameter 'confirmop' in the href. When such a link is clicked on, the user is asked "Really <link title>?" If they confirm, the link is called (by AJAX where possible and desired) with the 'confirmop' parameter's name changed to 'op'.

2. For confirmation of actions carried out other than through opcodes, two parameters are required: confirmop=<value> and notanop=<name>. On confirmation they are removed from the query string and the parameter <name>=<value> is inserted before the link is called.

If the link is clicked and javascript is not available, page header calls confirmop to insert a callout into the page that is then loaded asking for confirmation of the action. It can ask informative questions about operations it knows about, so update it to let it know more as required.

Examples

<a href="/user/DonJaime/writeups/Viking+Toilet?cool_id=2009408&confirmop=cool" class="action ..." title="C! DonJaime's writeup">C?</a>

<a href="/title/Viking+Toilet?confirmop=hellyeah&notanop=uncoolme" title="uncool this node">uncool</a>

"CONTAINED_STUFF" is a bit of a special string within ecore. The container system that's used to generate the nodelets and things on every page, regardless of the page's content, replaces the first instance of the string 'CONTAINED_STUFF' in the container HTML with what's actually contained... normally, that's the writeup.

However, this process is applied recursively, beginning with the containing container for the htmlpage for the requested note, and following each container's 'parent container'. Usually, htmlpages specify the general container as their parent. This in turn is contained within the main container (or a variation as specified by a theme), which is in turn contained within the stdcontainer, which has no parent.

So, first stdcontainer is evaluated. Then main container is evaluated, and the text 'CONTAINED_STUFF' in the result of stdcontainer is replaced with the result of main container. This is repeated for main container and general container.

If the text 'CONTAINED_STUFF' appears more than once in the result of the container, the first one is usually the only one substituted. Usually 'CONTAINED_STUFF' is written into the source for the container. However, if it appears more than once, for example because it's a node title (which the main container will show), the wrong instance may get the contained text.

With hilarious consequences. Until someone fixes the bug.

---

Edit, 9^th May 2009: the fun is over. It is fixed. *sniff*.

This is intended to be a quick guide in order to allow edevites to work with the currently running ecore and submit patches via Mercurial, hereafter simply "hg". As of this noding, the patches can go to me, but hopefully someone else can undertake this task later on as they become more comfortable with hg. This is also not meant to be a definitive guide. There's a whole book for that. This is just a few quick pointers to get you started.

First, make sure you understand what hg is and does.

The next step is to get it installed on your system so that you can work with it. In an apt-based system like Debian or Ubuntu, this should be as easy as "aptitude install mercurial" (make sure your sources are up to date beforehand). For Windows and MacOS X, you can download pre-built binaries here. Since hg is a primarily command-line driven program, you might find some comfort in the available GUI tools. TortoiseHG might be worth checking out. If you feel like experimenting, hgview also looks nice.

Once you're setup, you should modify your .hgrc (Unix-like) or Mercurial.ini (Windows) to identify yourself. Create the file if it doesn't exist, and add two lines like this:

  ui
  username = Bluepill Name <email@domain.com>

It's my habit when working on free projects' source trees to not use pseudonyms, but if you prefer to use a pseudonym instead of your bluepill name, go ahead. That name will identify the patches you submit to ecore.

 

In what follows I will describe the concepts from a purely command line interface point of view. The concepts are what are important; if typing at a prompt makes you uncomfortable, I imagine that TortoiseHG must have graphical methods to accomplish the same, but the important thing is to first get a basic understanding of how source control works.

The first thing you should do is get a local copy of the ecore repository on which you can work on with your locally installed tools. For this you have to clone from the ecore repository like so

hg clone http://hg.everything2.com/ecore

This will create a local directory named "ecore" from wherever you run this command that will have a snapshot of the source tree and all of its history up until that point. I should remark that ecore isn't very big, so this cloning should be really fast. If using TortoiseHG or another graphical frontend, you can run it from this source tree to get the directed acyclic graph representation of the repository up until this time. From the command line

hg log

will give dump you with the entire changelog of the project since I first committed after removing hardcoded passwords from the code. From a Unix-like system, you might want to pipe that to less so that you can read it (hg log | less) or on any system, you might want to just see a range of revisions, say from the 20th to the 30th revision, with hg log -r 20:30. Incidentally, all hg commands are documented with "help" so hg help log and hg help clone will give you more information about these commands. On most any modern installation of bash (which I believe all of MacOS X, Debian, and Ubuntu have) typing hg and hitting <TAB> will give you the list of all hg commands available which you can peruse with hg help $command.

 

Now that you're setup, your basic workflow will be like this,

1. Update your working copy in case that that there have been changes since you last downloaded it with hg pull -u, which pulls from the remote repository and simultaneously updates your current copy with those changes it pulls.

2. Make your changes on your copy, using your IDE or editor of choice.

3. Review those changes either by the excellent integration that your editor has with hg, or by doing hg status to see which files you've modified and hg diff to get a thorough description of the changes you've made. In a Unix-like system you may want to pipe the result of these two operations to less so that you can actually read them.

4. Commit your changes locally with hg commit -m "your commit message".

5. If you have been given a user and password with which to push changes back into the main ecore repository, do hg push to do so, which will prompt you for your credentials. For now, I can give you a username and password with which you can push back changes. You can only commit changes, but not make them take effect; someone with shell access to the server still has to do that for you, so there's no real danger of giving widespread commit access.

   If you don't have or want a username and password for committing, you can export your patch with hg export -o some.patch tip assuming you only committed once, or hg export $r1 $r2 $r3 -o some.patch where $r1, $r2, $r3... are the revision numbers of your local changes (displayed by hg log or your GUI). Then email me or whomever's currently able to apply patches the file some.patch for inclusion into the main source tree.

That's basic usage and should be enough for starters. To summarise, clone first, then later (1) update, (2) modify, (3) review, (4) commit, (5) submit. I don't think for a quickstart you need to know too much about how to make branches and merge. You can read about that in the official book. For your own benefit and to ease your workflow, though, I am going to describe two other useful features, cloning and updating.

Cloning is something you do at the very beginning when working with a new repository, but it's something you can keep on doing once you have a local repository. The syntax is hg clone original_repo new_repo and all it does is create a new repository named new_repo that knows it was cloned from original_repo. This can be useful for a quick testing ground in case that you don't want to pollute your original repository tree with mistakes and invalid changes, and once you think you're done, you can move the changes back to the original one by running hg push from new_repo's root directory. If you have commit access to ecore, you can push your changes all the way back to e2.

Updating is being able to go back to any previous working revision of your code. You create revisions each time you run hg commit. You can see the revision number that you might want to go back to with hg log; in the changeset line, it's the number in front of the colon and before the gobbledygook (actually, a SHA-1 hash). When you do hg up (shorthand for hg update) this brings you to the latest revision, but you can go to any other revision in your changelog with hg up -r $revision_number. Note that if you're in an older revision, make changes, and commit those changes, you will have branched off at that revision, which in and of itself is ok. You'll still be able to export patches or push, but then that might involve a merge when committing those changes upstream. If you export the patch, then whoever applies the patch will be in charge of handling the merge; if you push yourself, then you'll be in charge of that merge. Try not to branch at first and keep the whole operation simple; think of updating to a previous revision just convenience for you so that you can see what the code looked like in a previous stage. If you forget at which revision you updated to, you can do hg id -n which will give you revision number at which you currently are.

That will do for now. I look forward to receiving your patches. Happy hacking!

First of all, you may want to begin by reading the previous two entries in this series by kthejoker: Coding for E2: A primer and Coding for E2: E2 Syntax. You might also be interested in first reading edev: Quickstart. In this entry, I'm going to try to guide you through some examples of first finding where the code that you may want to change is, so that at least you can get started with an idea of what to do.

Finding the code in e2 that affects whatever functionality you're interesting in changing can be difficult. Broadly, the code can be in one of five places:

1. Nodes
2. The server's current Everything Engine, hereafter abbreviated to "ecore"
3. Standard Perl libraries like CGI.pm
4. Standard Javascript libraries like Scriptaculous
5. Perl cron jobs running in the server

Thankfully, most of the code that you will want to see and modify is readily available to you in a node somewhere, but just finding which node can be a challenge. The running ecore is accessible from Gigantic Code Lister, along with other bits of code like htmlcodes (currently only available to admins; trying to change this ASAP so that edevites can participate too). External libraries like CGI.pm or Scriptaculous are found in external sites. Our global variable to access the CGI.pm object is $query; so you'll see that running around the code now and then for creating form elements. Lastly, the only way right now to look at the cron jobs is to have ssh access into the server. I would like to change this soon so that edevites really have read access to all the code they need in order to make e2 better. We'll see.

But like I said, most of the code is in nodes and accessible to edevites as long as the nodetype is accessible (e.g. restricted_superdoc is a nodetype that is only accessible to admins). You can see who has access to a particular nodetype by going to the nodetype node ("everything's a node", remember?). The codeful nodes can be further broken down into the following main categories:

• superdoc and slight variations of it (superdocnolinks, restricted_superdoc, oppressor_superdoc) — examples: Message Inbox, Everything User Search.

• htmlcode — examples: sendPrivateMessage, zenDisplayUserInfo

• nodelet – examples: Epicenter, Other Users.

• htmlpage — examples: e2node display page, debatecomment edit page

• container — examples: zen stdcontainer, general container

• opcode — examples: message, cool

• dbtable — examples: node, writeup

• maintenance — examples: writeup maintenance update, patch create maint.

• jscript — examples async voting, Everything2 ajax

• Smaller pieces of code like notification and achievement nodetypes.

You can go to List Nodes of Type in order to get the full list of each nodetype if you wish. Each of those nodetypes requires an explanation of what they are for, what kind of code it houses, and how do you find it.

superdocs and variations

The superdocs and their variations are nodes that display by themselves and often perform some kind of code. When you visit them, you'll see that they often render some kind of content. Let's look at a particularly simple example, the Fezisms Generator. Go to the superdoc node and hit viewcode in your edev nodelet.

As explained in the last article in this series, you'll see some raw HTML which isn't filtered in superdocs like it is in writeups and discussion posts. Immediately you'll see the magic "[%" signaling "Perl code begins now" and some simple Perl code that defines two arrays of things thefez might say. The Perl block of code ends with the symmetrical "%]" There are two blocks of Perl code here, and these blocks are essentially Perl functions: e.g. they can return values like any Perl function, or will return the last value in them when parsing through them. In this case, the Fezisms Generator returns two random chunks from each of the @wit arrays in each block of Perl code and that's all, creating enduring hilarity for the ages.

The other nodetypes of the superdoc family have differences in how they handle the filtering of the HTML (for example, superdocnolinks doesn't call the ecore function parseLinks , so links in brackets aren't rendered) or the permissions associated to them. Scratch Pads is a prominent example of a node that for whatever reason is of nodetype superdocnolinks and not superdoc, but for practical purposes in terms of code is pretty much the same. More interesting examples of superdocs could be user settings and nodelet settings or even the frontpage.

htmlcode nodetype

The htmlcode nodetype is an interesting and sometimes frustrating kind of code in e2. Its original purpose as its name indicates was clearly to generate boilerplate HTML within certain nodes. For example, for e2nodes (writeups containers), we have in e2node display page the firmlink, softlink, and addwriteup htmlcodes being called with the [{some_htmlcode}] syntax as described in a previous writeup. However, because htmlcodes can also be called a inside Perl blocks with ecore's htmlcode function, their original purpose has been perverted to be used as "general purpose functions" and using their return "HTML" as a pure string or scalar with which Perl can do other things. Examples of this usage of htmlcodes are rootbeer277's dragUsers or kthejoker's isinUsergroup.

Ecore's htmlcode function for calling htmlcodes deserves special attention, since the way it works is downright barbaric in our running 0.8ish ecore. Its non-intuitive functioning has been a source of frustration and has caused bugs in the past, even as recently as this year when trying to apply call's table screening code as an htmlcode (it was impossible as an htmlcode and we had to resort to jamming call's code into ecore in the server).

Our running ecore's htmlcode code function takes at most two and no more than two arguments: a Perl string with the name of the htmlcode to call and another string with its arguments. It then performs a very simple parsing in a Perlish way to split the second argument by commas, and the resulting array will get passed to the called htmlcode's argument array @_. From the called htmlcode, you can then do "my ($foo,$bar,$baz) = @_" to recover the three arguments that ecore passed to this htmlcode.

The problem is that the arguments have to be extremely simple, since ecore performs a very low-level split on commas alone. This means, for example, that you cannot pass a writeup text to an htmlcode as we wanted to do with the screenTable function, because writeups almost certainly contain commas in them, and ecore will interpret that as different arguments. Another example is that JSON objects are strings that contain commas in them, so you'll see for instance in debate maintenance create that the notification code has to escape the commas in the JSON which the addNotificaiton htmlcode has to unescape and split upon.

There is one further bit of complication with passing arguments to htmlcodes this way and that's that ecore essentially does string parsing and macro substitutions when passing arguments. What this means is that quotation marks are not escaped either, so it's difficult or impossible to pass any arguments to an htmlcode that contains quotes, because then Perl will be unable to parse the resulting string that ecore's htmlcode creates.

Despite these limitations, as long as one stays aware of them and is careful about the argument string that goes into an htmlcode, htmlcodes can be very useful to have certain functions within the database instead of implemented in the server.

nodelet nodetype

This is also pretty straightforward: because everything is a node, nodelets are also nodes, so they can be seen like ordinary nodes in addition to displaying in your sidebar. The kind of code inside nodelets is pretty much the same as the code for superdocs; you can also put unescaped HTML inside nodelets and Perl blocks are also signaled in the same way.

I do want to mention one peculiarity of nodelets and that is nodelet sections, which are sometimes a bit difficult to find. Nodelet sections are those parts of a nodelet that you can collapse with the little + and - buttons. Let's look at an example, the Statistics nodelet. When you look at this nodelet's source, you'll see that it's a rather terse bit of code that only calls the nodeletsection htmlcode with some parameters several times. After parsing N-Wing's code in nodeletsection, you can eventually figure out that what it's doing is concatenating the first two parameters passed to the htmlcode in order to invoke yet another htmlcode. So for example, the first section in the Statistics Nodelet, "stat,personal,Yours" becomes the htmlcode statsection_personal and the "stat,global,E2" the statsection_global htmlcode. The general formula for finding other nodelet sections applies to other nodelets.

htmlpage and container nodetypes

These are the nodetypes that are used to render nodes. The way it works is that ecore fetches a node, sees what htmlpage goes with this node, sees if that htmlpage has a container, if that container has another container, and all the way up to a parent container, and then starts rendering the page by calling the top container, replacing the infamous CONTAINED_STUFF in the top cotainer with the next container in the list, all the way down until it reaches the htmlpage and voilà, the node is rendered, along with the nodelets and all the goodness that is e2. This chain is also determined by the current theme you're using, Jukka, classic, or Zen.

You can see which htmlpage is being used to render the node you're looking at because it will be in your edev nodelet. If for example you want to add a new text field to the homenode edit page, you can go to your homenode, edit it, look at your edev nodelet and observe that classic user edit page is the htmlpage that it's using. This is what you would have to patch to add another textfield to the homenode edit screen.

Not all htmlpages have containers, though. For example, the fullpage display page htmlpage is very simple and has no container, the jscript nodetype is another example that has a jscript display page without a container. Chatterlight is an example of a node of type fullpage, and if you look at its code (put "&displaytype=viewcode" in the query string in the address bar of your browser when visiting chatterlight), you'll see that chatterlight does a lot of work of its own to render the page without relying on the default formatting that containers provide.

In particular, most of the site-wide CSS for the Zen Theme are provided in zen container and zen stdcontainer. This is where you can find a good chunk of the work that ascorbic and kthejoker did to bring e2 out of the dark ages of table-based layout and into a new era of CSS goodness.

opcode nodetype

An opcode is a bit of code for e2 that performs certain actions; it's called in conjuction with other bits of code. The way to call an opcode is to pass the "op=$opcode_name" CGI parameter to e2 somehow, either by a query string in a GET method or by POST. For example, if you look at chatterlight's rendered HTML source in your browser, you'll find that there is a hidden CGI form element

<input type="hidden" name="op" value="message" />

which means, "when this form is submitted, call the message opcode", and is exactly what will happen. Most things you think of as operations in e2, voting, cooling, sending /msgs, are opcodes. It's instructive to go to List Nodes of Type and browse through the opcodes that we have available.

dbtable nodetype

This is the nodetype that ecore uses for representing tables in the underlying MySQL table that houses all of our data. It isn't really a nodetype that contains code except in the sense that all the of e2's data is ultimately in dbtables, and if it's data in a node somewhere, like much of the code is, then it's in a dbtable.

You will often see ecore calls of type $DB->sql* which will invariably mention a dbtable in their first or second argument. If you ever need to refer to the structure of the underlying dbtable, go look at its node, which looks very much like any other MySQL table description, telling you what columns the table and their type. Unfortunately, unless you actually look at the data in the columns, it's not always easy to guess what's in each column. In case of doubt, consult an e2coder who will be able to make a quick query on the table and report back on its contents.

maintenance nodetype

Whenever a node, any node (and this includes writeups, users, nodeshells, because everything is a node) is created, updated, or deleted (nuked), ecore will call a corresponding createNode, updateNode, or deleteNode function. Each of these nodes will check in turn if this nodetype has an associated maintenance node that will perform those operations, and if it does (most nodes don't need that much maintenance), it will execute the code in this maintenance nodetypes.

When you look at a nodetype's node (e.g. the debatecomment nodetype), you'll see listed near the bottom after it lists the permissions any maintenance nodes associated to it, if any. For example, debatecomment has debatecomment maintenance create which amidst its maintenance code has the notification code for notifiying users when a new reply has been posted to usergroup discussions. Writeups have writeup maintenance delete which in turn contains code for adjusting the XP and writeup count of users for when their writeups get nuked. And so on.

jscript nodetype

This nodetype is quite simple: it's what contains javascript to be used by e2. It has no container, so if you want to patch jscript nodes, you'll have to manually manipulate your URL and attach "&displaytype=viewcode" to the query string, or you can click on their (patch) link from javascript repository. After that, zen stdcontainer makes a direct call by node_id to Javascript compiler to place the right javascript in the header. For now, async voting gets embedded directly by zen stdcontainer in a separate line without going through javascript compiler.

I should remark, this isn't the only bit of javascript in e2. We have lots of other bits of javascript in e2 embedded directly into the code without going through jscript nodetypes; TinyMCE is one prominent example of one. Usually it's easy by looking at the rendered page to guess which node has the embedded javascript when the javascript isn't being served by a jscript nodetype, though.

Other codeful nodetypes

These aren't the only nodetypes that can hold code, but they are the most important ones. We have minor nodetypes that only have a handful of nodes like the notification nodetype that handles what gets displayed in the Notifications nodelet or the achievement nodetype that has nodes with code that determine when an achievement in My Achievements has been... well, achieved. This should be plenty to get you started, though.

That's all for this node. Gigantic Code Lister, currently only available to admins, lists all of the code in the running ecore as well as optionally all the htmlcodes. Also in the server, under /usr/local/everything/bin, you can find some Perl scripts that cron calls to perform maintenance tasks like moving writeups from Node Row to Node Heaven. This should be enough to help you find your way around the code.

Happy hacking!

Hi. Welcome to edev. Or perhaps you've been in edev for a while, but you never really knew what to do with your %. Let me try to show you the ropes just to get you started.

E2 is mostly written in Perl, but familiarity with web technologies like HTML, SQL, CSS, XML, and javascript is of course desirable and often necessary.

The edev nodelet

If you haven't done so already, go to nodelet settings and enable the Everything Developer nodelet. It's got lots of nifty little things that you'll need for finding your way around the code, which I'll now describe briefly. The two most important things is that at the very top of the nodelet you'll see a "viewcode" option on any node that contains code. You'll also see a "using: $somepage" which refers to the code that e2 is using to render whatever node you're currently looking at. You'll be clicking on both of those buttons frequently in order to look at the code.

• Display types: This is at the very top of the nodelet. In addition to the aforementioned viewcode button, most nodes will also have an xmltrue button so that you can see what XML e2 will generate for the current node, useful for e2 clients or for Ajaxing.

• Node Information: The next bit you'll see is useful information about the node you're looking at, which in order is the node id, the nodetype, and its creation time. The mantra in e2 is everything is a node, writeups, e2nodes, superdocs, nodetypes, you, me, all nodes. The only big exceptions to things that are not nodes but just entries in an SQL table somewhere are scratch pads and messages.

• Page using to display: Next up is a brief little line that says "using: $somepage". That's the htmlpage that e2 is currently using to display the node you're looking at. For example, if you go to, say, Swap's Playground of type superdoc, and you click on "viewcode", you'll see this bit change from classic superdoc display page to superdoc viewcode page, and you'll see whatever code I happen to have in there at the time. If you ever want to change how e2 displays superdocs or renders them when viewing the code, this will tell you the htmlpage that you need to patch.

Those are the important bits. The rest of the nodelet has information that will be less frequently used, which is why all of it can be collapsed. Allow me to describe the next few bits of information that you'll see in the nodelet.

• CGIparam: Whenever you submit any kind of information to e2, and this includes just telling it which node you want to look at, you're submitting CGI variables, which will display here. For example, if you are talking in the chatterbox, you'll see the "message" CGI parameter showing up here with whatever you said, along with related variables that e2 needs in order to send your message. You will always be seeing at least the node_id CGI parameter here, and sometimes it's useful for other purposes to know exactly what information you're sending to e2.

• edev: Nothing terribly exciting, but whenever someone clicks on that edevify! button in your epicentre, that document will show up here.

• Documents: Old and mostly obsolete documents. Also contains as of this noding broken links to the Everything Bible. Should probably fix this.

• Util: Links to three mildly useful things. List Nodes of Type can be useful, for example, if you ever want to know what all the possible notifications are, or perhaps all the achievements. Everything Data Pages links to nodes that provide useful data that e2 clients can use. The Everything Document Directory will bring you to all the superdocs, or at least the ones you have permission to look at (yeah, admins have sooper sekrit restricted_superdocs that edevites cannot look at, not even their code).

• Globals: There are a few global variables that are always available. You can see their values here. Click on hashrefs to expand them. They're "global" in the sense that they're always available in every node, but most of them except HTMLVARS are really local only to you.

• Patches: Next you'll see a list vaguely reminiscent of New Writeups that shows you which patches have been recently provided and have not been applied. As soon as a patch is applied, it vanishes from this list. You can always go to patch manager to get the full list.

About that list bit, patches, let me go more into detail.

The patch system

Patching is one of the most important things that edevites can do. Whenever you are looking at a node in viewcode mode, you'll have a text box at the bottom of the node where you can see and modify the code. Once you made whatever modification you deem appropriate and you've written in the text field above it a brief description of what you think your code should do, you can submit the patch for review by a member of e2coders. Your patch will show up in this list in the edev nodelet and in patch manager, with status "unknown" and assigned to no one.

A remark about certain nodetypes whose display doesn't have nodelets, such as the fullpage nodetype or the jscript nodetype. If you stumble upon one of those nodes, you'll be faced with a display that doesn't have the edev nodelet for you to click on "viewcode". Worry not, you can still submit patches for those if you append in your browser's address bar "&displaytype=viewcode" to the query string that shows that node. You'll then be taken to the familiar display for every other node in which you can submit patches.

About how patches work, as of this noding, it's still a bit primitive, but hopefully it can be improved soon. A patch can be either be applied or unapplied, and there are several statuses for being unapplied, all of them currently woefully underused. When a member of e2coders looks over your patch and thinks it's worthy, it will be applied, and you'll be notified of this fact via /msg. Applying a patch exchanges the contents of the patch with whatever node it's patching, so once applied you'll see that the patch looks "reversed" in its display mode. Almost surely, there will be a mistake in the patch, and the same diligent e2coder will immediately unapply the patch again in order to review the problem. While the patch is not in an applied status, you can go and edit it with a little edit button that you'll see at the top right of the patch and try to fix the mistake, or perhaps the e2coder will do it for you. There will probably be some consultation, some going back and forth, a few more cycles of applying and unapplying the patch, and eventually if all goes well, your fine work will become an enduring part of this site. Well, that's the theory anyways.

That's it! That should be enough to get you started with the very basics of coding for e2. If you want to learn more, start reading the series by kthejoker and me of the same name.