The device of footnotes taking over the main body of a novel was used by Flann O'Brien in The Third Policeman, where at times the footnotes actually swamp the main body of the text, which is reduced to three or four lines at the very top of the page. It is never totally clear who is writing the footnotes, which gather together academic comments from several different sources, many of whom are as unhinged as the author they are critiquing.
Aside from the comic effect which both authors make use of, O'Brien uses this device as part of The Third Policeman's infinite spiral into the nothingness of meaning, whereas Nabokov was, I feel, primarily interested in the development of character through this and various other devices.