A specific standard for the low-level coding of the parts of programs that interface external objects. Typically a compiler will follow an ABI in generating call sequences and return sequences (at least for extern functions), for performing system calls (an important special case of calling external functions), for the layout of pointers, and possibly for alignment and stack usage and layout of the stack frame.
The ABI is standard; you can only link together object files that follow the same ABI, or chaos will ensue (usually the linker will prevent such linking). Some machines (e.g. 64 bit IRIX boxes) can have more than one ABI; you must select the appropriate one yourself.
The PC world called this (or a similar subset) a "memory model" in the days of MS DOS.