We have recourse to magic and belief in the supernatural when what exists isn’t what we want. Human desires being limitless, it is thus likely that a vestigial tropism for magic and the supernatural are likely to be with us always. Our twenty-first-century minds no longer grant credence to love charms or prophecies or spells, yet our hearts still thrill to fairy tales, ghost stories, and the wonders of the Arabian Nights entertainments. While science fiction, the literature of extrapolation, answers the question “if this goes on,” stories of the supernatural build on “what if,” or even the hushed unspoken wish “if only.” They are tales of transcendence, whether of incontrovertible facts like death or of the horrors of modern life or of the burden of our own personalities.

The fantastic pervades the world’s literatures of every time and place; our much vaunted realism is the sport, the mutant. The epic hero Gilgamesh pursues the secret of immortality, the wandering Odysseus encounters witches and monsters, Arthurian knights and Celtic bards return, transformed, to tell of the wonders of Faery. Are these interruptions in what the Greeks called the Heimarmene, the natural order of things, the accepted succession of cause and effect? Maybe, maybe not. As Hamlet reminds his friend, “There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”

Up through the seventeenth century, at least, the very air surrounding us buzzed with both angels and spirits from the vasty deep. While Marlowe’s Dr. Faustus sold his soul to the devil in return for knowledge and power and beautiful women, in “real life,” Dr. John Dee, possessor of the largest private library in Elizabethan England, spoke with astral beings through a scrying stone. Paracelsus and his fellow alchemists soberly researched the elixir of life as well as the means of turning base metals into gold. Even Issac Newton, the very icon of mathematical science, studied what one might call the dark arts. One recent biography calls him “The Last Sorcerer.” When asked if she believed in ghosts, the Marquise du Deffand—the beloved correspondent of Horace Walpole, who initiated the Gothic novel with The Castle of Otranto—answered, no, but that she was afraid of them nonetheless.

The romantic movement of the late eighteenth and early nineteenth centuries enshrined not just the natural, but the supernatural. As a boy, William Wordsworth felt himself pursued by chthonic monsters of the Lake District, Samuel Taylor Coleridge immortalized accursed mariners and maidens wailing for their demon lovers, John Keats yearned for fairy lands forlorn and belles dames sans merci. No less than Lord Byron himself inspired Polidori’s vampire Lord Ruthven and a still fashionable line of irresistible and sexy ladykillers.

Throughout the nineteenth century, virtually every major novelist or short-story writer, on the continent as well as in England and America, produced tales of the weird and supernatural. However, such stories weren’t simply campfire entertainments. By now they were “philosophical romances,” products of a world where science and revolution had overturned the ordered hierarchies and old verities. Above all, they explored the human personality and the galaxies of inner space. For instance, through his fantastic fables, E.T.A. Hoffmann examined mesmerism, automatons, “nervous” conditions, the psychology of artists and musicians, alienation, the nature of dreams, somnambulism, prenatal influence, magnetism—all hot topics of the day, and most still of interest to us. His most famous tale, “The Sandman,” eventually gave rise to one of Freud’s greatest and most influential essays, “The Uncanny.”

In France, even pillars of realism, such as Honoré de Balzac, Prosper Mérimée, and Guy de Maupassant, gravitated to the supernatural. Balzac’s earliest masterpiece, La Peau de chagrin—one of his etudes philosophiques—concerns a piece of wild ass’s skin that grants wishes, a charm that the novelist uses as a metaphor for personal and creative energy. In Mérimée’s “The Venus of Ille,” a mysterious statue of the ancient goddess provides a lesson in tough love and the proper valuation of women. As for Maupassant, this student of Flaubert and the human heart regularly uses the weird tale to explore psychological disorder. In his famous tale, “Le Horla,” a man detects an unseen presence in his daily life, then grows gradually convinced that this invisible entity has moved into his mind and may be controlling his actions. In “Who Knows?” the narrator imagines that his furniture is alive.

As should be clear, the supernatural is the habitual mode by which writers explore the irrational and the subconscious: As within, so without. Dickens’ Fat Boy used to whisper “I wants to make your flesh creep,” but most practitioners of the weird tale aim for a bit more than just that. It is, I think, significant that the great age of occult fiction—particularly in England—runs from the 1860s through the 1930s. The Oxford Movement and the Catholic revival after Newman, a disgust with the dehumanizing aspects of modern industrial society, the 1890s fascination with the decadent and Satanic, the scientific investigations of the Society for Psychical Research, the widespread belief in spiritualism and Theosophy, the pioneering modern psychology of William James and Sigmund Freud—all these fueled the period’s growing conviction that there were unacknowledged, unconscious or unseen forces at work all around us.

Little wonder, then, that the ghost story flourished. Sheridan Le Fanu’s “Green Tea” depicted a clergyman haunted by an invisible monkey: Is it a projection of unfulfilled desires? A guilty conscience? The return of the repressed? In “Carmilla”—still the best of all stories of the undead—Le Fanu transmuted vampirism into a metaphor for lesbian desire. When reading his work the reader is swept up by his narrative power, his artistry in the use of description, folktale and local color. But the nightmares touch archetypal nerves. In “Madam Crowl’s Ghost,” the servant Mrs. Jolliffe confronts the robot-like horror of Madam Crowl, who suddenly rises from her bed:

And in an instant she opens her eyes, and up she sits, and spins herself around, and down wi’ her, with a clack on her two tall heels on the floor, facin’ me, ogglin’ me in my face wi’ her two great glass eyes, and a wicked simper wi’ her old wrinkled lips, and long fause teeth…If I’d a thought an instant, I’d turned about and run. But I couldn’t take my eyes off her, and I backed from her as soon as I could; and she came clatterin’ after, like a thing on wires, with her fingers pointing to my throat, and she makin’ all the time a sound with her tongue like zizz-zizz-zizz.

Le Fanu’s greatest admirer was M.R. James, the Provost of Eton, whose Ghost Stories of an Antiquary (1904) scared little boys with accounts of revenants and mysterious mazes and accursed treasures and whistles that it would be best never to blow. In general, James’s tales all take the form of “a warning to the curious”—perhaps an appropriate message to Edwardian schoolboys—and they usually show the comeuppance or destruction of scholars and clergymen, which must always go down well with their pupils. In general, James’s protagonists simply go too far; they surrender to a passion, be it the solution of an arcane scholarly mystery or a spiteful desire for revenge on a colleague. In effect, though, James established the form of the classic English ghost story. He particularly specialized in ominous foreboding, capped usually by a single moment of revelation, a short, sharp shock. In one of his tales, the protagonist returns home after an unsettling day:

Then he dozed, and then he woke, and bethought himself that his brown spaniel, which ordinarily slept in his room, had not come upstairs with him. Then he thought he was mistaken; for happening to move his hand, which hung down over the arm of the chair within a few inches of the floor, he felt on the back of it just the slightest touch of a surface of hair, and stretching out in that direction he stroked and patted a rounded something. But the feel of it, and still more the fact that instead of a responsive movement, absolute stillness greeted his touch, make him look over his arm. What he had been touching rose to meet him.

For all the artistry of James and his acolytes (commonly referred to as The James Gang and including E.G. Swain, A.N.L. Munby, and H.R. Wakefield, among others), the very greatest tales of the supernatural achieve a sense of the cosmic. While James was a dweller on the threshold, Arthur Machen (1863-1947), Algernon Blackwood (1869-1951) and Walter de la Mare (1873-1956) were openers of the way. The first two, in particular, divide the honor of having produced the greatest weird tale of all time. In Machen’s “The White People” (1904), which critic Roger Dobson has called a Satanic Alice in Wonderland, an adolescent girl recalls her early childhood, chattering in run-on sentences of her nurse’s unsettling “fairy tales” and the private “games” they played, of an eerie place in the hills full of strange stones and circular patterns, of a certain little manikin made out of clay. Writing with a chilling innocence, the guileless girl recalls a series of increasingly disturbing encounters with a pagan Otherness:

I am going to write here many of the old secrets and some new ones, but there are some I shall not put down at all. I must not write down the real names of the days and months which I found out a year ago, nor the way to make the Aklo letters, or the Chian language, or the great bountiful Circles, nor the Mao Games, nor the chief songs. I may write something about all these things but not the way to do them, for peculiar reasons. And I must not say who the Nymphs are, or the Dols, or Jeelo, or what voolas mean. All these are most secret secrets…

In Algernon Blackwood’s “The Willows,” (1907) two men spend a night on a small island in the Danube and discover that they have “…intruded, have, in fact, come upon a gap in the curtain of the universe.” In the words of H.P. Lovecraft, who knew about these things, Blackwood possessed a preternatural ability to evoke the presence of “an unreal world constantly pressing upon ours.” It is worth noting that both Machen and Blackwood, like W.B. Yeats, were members of the theosophical society, the Order of the Golden Dawn. De la Mare, in his turn, was fascinated by dreams, illusions, childhood, and the uncanny throughout his life, producing ghostly poems like “The Listeners” and a score of unsettling stories, including that most subtle of vampire tales, “Seaton’s Aunt.”

Supernatural fiction typically works best in its short form. As Edgar Allan Poe famously argued, the unity of place, atmosphere and effect can then be sustained, unbroken. So it’s no surprise that Machen’s The Hill of Dreams (1907), De la Mare’s The Return (1910), and Blackwood’s The Centaur (1916) are somewhat neglected. These great novels of transcendence, all published within a ten-year span, are demanding works, written in lyrical prose, with Henry Jamesian subtlety, and marked by intense visionary power, each of them employing the conventions of the supernatural to reflect on questions of personal identity, the role of the artist in society, man’s relationship to nature, and the possibility of a more intense life, both in this world and the next. In these three books the supernatural merges with the mystical or metaphysical. They underscore the period’s fascination with unorthodox states of perception, as well as its obsession with what one might call astral porousness, the possibility of communication with an immanent realm of spirit.

All three novels are about dissatisfaction with the world as it is and a yearning for some kind of spiritual growth or enlargement of vision. In poetry and some mainstream authors (Alain Fournier, Proust), such yearnings might typically focus on the lost paradise of childhood. For a smaller group of writers, they might lead to a rhapsodic disordering of all the senses (Rimbaud) or to powerful records of schizophrenia, as in Paul Ableman’s I Hear Voices or the autobiographical fiction of Anna Kavan. But in the realm of the supernatural, the most common trope is possession, sometimes taking the form of an attack, sometimes of a seduction. Consider, as an example, the 1895 classic The Lost Stradivarius, by John Meade Falkner.

After discovering a mysterious violin, a virtuous and hard-working young baronet named John Maltravers undergoes a gradual change in personality. He withdraws from his friends and family, grows indifferent to his young wife, and spends more and more time in Naples, where he restores an ancient villa and, it is rumored, engages in pagan rituals. The violin, it turns out, once belonged to a decadent and lascivious nobleman who was drawn to esoteric learning and forbidden practices. Maltravers, in effect, emulates his predecessor. But is his personality corrupted or liberated? As the critic Mark Valentine has written,The Lost Stradivarius offers both a classic ghost story and “a finely poised spiritual drama counterpointing art, ecstasy and license against duty, decorum, and rectitude.”

That word ecstasy recurs throughout Arthur Machen’s fiction and nonfiction. While Walter Pater famously maintained that art should aspire to the condition of music, Machen believed it should go further, it should convey “rapture, beauty, adoration, wonder, awe, mystery, sense of the unknown, desire for the unknown.” Most realistic fiction, he felt, was nothing more than “dignified reporting.” Instead, Machen contended that “fine literature is simply the expression of the eternal things that are in man, that it is beauty clothed in words, that it is always ecstasy, that it always draws itself away, and goes apart into lonely places, far from the common course of life.” As one Machen character declares, “The whole universe, my friend, is a tremendous sacrament; a mystic, ineffable force and energy, veiled by an outward form of matter.” The aim of supernatural fiction should be to tear away the veil, to awaken us.

This esthetic informs The Hill of Dreams, Machen’s portrait of the artist as solitary and doomed. It begins: “There was a glow in the sky as if great furnace doors were opened.” Young Lucian grows up in the Welsh hills, an introspective, dreamy boy, given to meditating on the past, especially on early Britain, a time when Celtic magic brooded “on the wild hills and in the black depths of the forest.”

During his wanderings, in which he sometimes imagines himself a fairy-tale hero, Lucian repeatedly experiences moments of exhilaration and mystery, as if the forests were haunted, when rustling in the shadows might be more than just the wind’s passage. Aside from mooning about, Lucian spends most of his time reading old poets and esoteric volumes:

He dived deeper and deeper into his books; he had taken all obsolescence to be his province…The strange pomp and symbolism of the Cabala, with its hint of more terrible things; the Rosicrucian mysteries of Fludd, the enigmas of Vaughan, dreams of alchemists—all these were his delight. Such were his companions, with the hills and hanging woods, the brooks and lonely waterpools; books, the thoughts of books, the stirrings of imagination, all fused into one phantasy by the magic of the outland country.

In due course, Lucian makes “a pious attempt to translate into English prose the form and mystery of the domed hills, the magic of occult valleys, the sound of the red swollen brook swirling through leafless woods.” He suffers rejection, plagiarism and mockery. For a while he is saved by love. But his beloved Annie is compelled to visit a relative and leaves Lucian alone for weeks on end, then months. During her absence, he writes mystical poetry, illuminates his manuscript pages himself and then binds them into a golden book, a personal breviary.

Gradually, however, his imagination and curious learning lead him into a world of fantasy, one in which he sees not the present but the imagined past. Out of his vulgar, modern-day village, he reconstructs, quite literally in his mind’s eye, the ancient city of Siluria and the gardens of Avallaunus. It is a realm of wondrous villas, shady courts, tessellated pavements:

Lucian wandered all day through the shining streets, taking shelter sometimes in the gardens beneath the dense and gloomy ilex trees, and listening to the plash and trickle of the fountains. Sometimes he would look out of a window and watch the crowd and colour of the market-place, and now and again a ship came up the river bringing exquisite silks and the merchandise of unknown lands in the Far East.

In his dream-kingdom, he grows decadent and exotic in his tastes: “He made lovers come before him and confess their secrets; he pried into the inmost mysteries of innocence and shame, noting how passion and reluctance strive together for the mastery.” In his reveries he meets “women with grave sweet faces” who “told him . . . how they had played and watched by the vines and the fountains, and dallied with the nymphs, and gazed at images reflected in the water pools” or “had loved the satyrs for many years before they knew their race.”

But the longer Lucian spends his waking hours at Avallaunus, the more disconnected he grows from the real world. His neighbors begin to notice that his bones seem to be growing through his skin and that “he had all the appearance of an ascetic whose body has been reduced to misery by long and grievous penance. People who chanced to see him could not help saying to one another: ‘How ill and wretched that Lucian Taylor looks!’ They were of course quite unaware of the joy and luxury in which his real life was spent, and some of them began to pity him, and to speak to him kindly.”

Eventually, Lucian moves to London, where he attempts to write a masterpiece. But the noise, filth and vulgarity of the metropolis trouble him with their nightmarish intensity:Nothing fine, nothing rare, nothing exquisite, it seemed, could exist in the weltering suburban sea, in the habitations which had risen from the stench and slime of the brickfields.” When he goes out for food, “the rocky avenues became the camp and fortalice of some half-human, malignant race who swarmed in hiding, ready to bear him away into the heart of their horrible hills.” He stays indoors more and more, eats and drinks less and less, until the real and the unreal begin to blur, the outside world increasingly mirroring the inner one. “Truth and the dream were so mingled that now he could not divide one from the other.” Lucian hears the call of faeryland, or is it merely a final madness?

In The Hill of Dreams Machen’s hero is simultaneously alienated from commercial society by his poetic visions, swept away by the mystical chthonic forces emanating from the Welsh hills, and clearly growing insane. In what may be a deliberate echo of Balzac’s The Unknown Masterpiece—in which a painter creates a canvas that is nothing but swirls and ugly daubs of paint—Lucian’s great work turns out be a mass of meaningless scribbles. At least to our eyes. But Lucian himself follows the ecstasy within, abandoning this gimcrack world of ours forever. The book ends with a slight variation on its beginning: “There was a glow within as if great furnace doors were opened.”

If The Hill of Dreams might pass for a fin-de-siecle novel in the mode of Joris-Karl Huysmans’ A Rebours (Against the Grain), its theme of alienation is made almost literal in Walter de la Mare’s The Return. Arthur Lawford, “a rather fair, not unsubstantial, rather languid” middle-aged man, is recovering from illness when one day he wanders into the Widderstone graveyard. His thoughts, naturally enough, turn introspective: “ ‘What is the good of it all? he asked himself inconsequently—this monotonous restless, stupid life to which he was soon to be returning, and for good.” Growing a bit tired, he sits down on a bench next to the grave of a Frenchman named Nicholas Sabathier, said to be a suicide. When Lawford awakes from his nap, he feels full of unexpected energy, races home to be in time for dinner, and rushes up the stairs into his room to change. When he glances in the mirror, he discovers another man’s face gazing back at him.

His wife Sheila is appalled by this transformation—What will the servants say? What will the neighbors think if they see her living with this thin, distinctly wolfish and foreign-looking man? She’s not even sure that Lawford isn’t an impostor, demanding proof after proof of his identity. Is he, in fact, entirely the man he was? 

More and more, Lawford registers a heightening of his self-consciousness and what the ancients would call a psychomachia:

A consciousness had begun to stir in him that was neither that of the old, easy Lawford, whom he had never been fully aware of before, nor of this strange, ghostly intelligence that haunted the hawklike, restless face, and plucked so insistently at his distracted nerves. He had begun in a vague fashion to be aware of them both, could in a fashion discriminate between them, almost as if there really were two spirits in stubborn conflict within him. It would, of course, wear him down in time. There could be only one end to such a struggle—the end.

In an attempt to reverse his physical transformation, Lawford returns to the cemetery, where he converses with a rather enigmatical stranger named Herbert Herbert. When Lawford visits Herbert’s home he learns that his new face resembles that of the dead Sabathier. At the same time Lawford meets Herbert’s sister Grizel—and falls in love with her. But whom does Grizel love? There are hints that Sabathier committed suicide because of Grizel. Yet how can this be, since the Frenchman died a century earlier? And why has no one in the town ever heard of the Herberts and their house?

As with the protagonist of The Lost Stradivarius, Lawford’s “possession” initiates an expansion of personality, an introduction to a life far richer and more spiritually satisfying than that of his old petit bourgeois existence. On a simple level, this might be an account of a midlife crisis. But De la Mare makes nothing easy—he is as subtle an artist as Henry James, and the last chapters of the novel are roiled with ambiguity. Who returns and to what and why? Does the book end happily or not?

With the avowed pantheist Blackwood the presentation of alienation and rebirth grow positively mythic. In this classic of transcendental fiction, Terence O’Malley is deeply sensitive to Nature, susceptible to singular states of exaltation (that Celtic blood!), distinctly lonely. “Chief cause of his loneliness . . . seemed that he never found a sympathetic, truly understanding ear for those deeply primitive longings that fairly ravaged his heart.”

While sailing to the Caucasus as a kind-of freelance journalist, O’Malley encounters a mysterious, laconic Russian and his son. The man, in particular, is physically big, but also somehow suggests a greater bulk and size than meets the eye. It’s as though another invisible shape emanated from his body and extended it. When O’Malley finally speaks to him and explains that he is going to the Caucasus, the somewhat child-like stranger’s first words are:

“Some of us…of ours…” he spoke very slowly, very brokenly, quarrying out the words with real labour, “…still survive…out there…We…now go back. So very…few…remain…And you—come with us…”

O’Malley relates all this much later, to a London friend, who works in an insurance office. He stresses that there was nothing about this invitation that could be likened to a “ Call of the Wild,” a desire “to let off steam.” Instead he soon recognized in himself a hunger for something unavailable to him in this bustling, hustling century. This isn’t my time, he says, “it’s not even my world! And I loathe, loathe the spirit of today with its cheapjack inventions, and smother of sham universal culture, its murderous superfluities and sordid vulgarity.”

In the course of the ship’s voyage, O’Malley is warned about the big Russian. Stahl, the ship’s doctor, has known the man in the past and fears for O’Malley’s very soul. In fact, Stahl believes the Russian isn’t really quite human, that he is some kind of being from the primordial Urwelt of the earth. He explains that this gentle giant has been wandering the world, “companionless among men, seeking his own, finding no place where to lay his head.” That last phrase should set off echoes. As the Gospel of Luke says, “Foxes have holes and birds of the air have nests, but the Son of Man has nowhere to lay His head.”

Stahl contends that the Russian is actually a Cosmic Being, “a direct expression of cosmic life. A little bit, a fragment, of the Soul of the World, and in that sense a survival—a survival of her youth.” Following the teachings of the German philosopher and scientist Gustav Fechner, both Stahl and O’Malley agree that the Earth is “a conscious, sentient, living Being,” and that “the World-Soul or Cosmic Consciousness is something more than a picturesque dream of the ancients.”

As O’Malley and Stahl continue to observe the father and his son, they remark on the rapidity of their movement and a distinct sense of larger, shadowy forms hovering around their bodies. Despite Stahl’s cautionings, O’Malley continues to be drawn to the pair. For the Irishman, at least, “The gates were opening.” O’Malley yearns more and more for something wilder, simpler, something lost but perhaps not irretrievably so, “the true, pure, vital childhood of the Earth—the Golden Age—before men tasted of the Tree and knew themselves separate.” As he says when imagining that lost paradise:

I feel as if I’d fallen asleep in one world and awakened into another where life is trivial and insignificant, where men work like devils for things of no value in order to accumulate them in great ugly houses; always collecting and collecting, like mad children, possessions that they never really possess—things external to themselves, valueless and unreal.

Throughout The Centaur Blackwood mixes together a critique of industrial society, an early formulation of what is now called the Gaia thesis, his own pantheist philosophy, contemporary beliefs in astral bodies, and the new discoveries in psychology. His insistent, almost kerygmatic truth throughout is simply that we are much greater than we know.

In addition, O’Malley stresses the universal mystical belief “that a man must lose his life to find it, and that the personal self must be merged in a larger one to know peace—the incessant, burning nostalgia that dwells in the heart of every religion known to men: escape from the endless pain of futile personal ambitions and desires for external things that are unquenchable because never possible of satisfaction.”

Before the ship lands at Batoum, there will be a mysterious death, an apparent suicide. At Batoum itself, the Russian will disappear, but O’Malley is convinced that they will meet again. Making his way inland, toward the cradle of the human race, he hears tales of “true Pagans who worship trees, sacrifice blood, and offer bread and salt to the nature-deities.” He even picks up rumors of beings who come in the spring “and are very swift and roaring…You must always hide. To see them is to die. But they cannot die; they are of the mountains. They are older, older than the stones. And the dogs will warn you, or the horses, or sometimes a great sudden wind.”

I’ll say no more, though Blackwood goes on to describe O’Malley’s ecstasy in which he finally experiences “the Great At-one-ment.” Nonetheless he returns to this world for a while, though his ultimate destiny resembles that of Lucian in The Hill of Dreams rather more than that of Lawford in The Return. “He remembered dimly the Greek idea of worship in the Mysteries; that the worshipper knew actual temporary union with the deity in his ecstasy, and at death went permanently into his sphere of being.”

Far too long, I think, the realist novel has dominated our thinking about the course of English literature. Let us honor the marvelous as well as the matter of fact! It is time we paid more attention to metaphysical fiction, whether labeled fantasy, supernatural thriller or spiritual psychodrama. Some high spots of this lineage include Mary Shelley’s Frankenstein, James Hogg’s Memoirs and Confessions of a Justified Sinner, Emily Brontë’s Wuthering Heights, George MacDonald’s Phantastes, Robert Louis Stevenson’s Dr. Jekyll and Mr. Hyde, and Joseph Conrad’s Heart of Darkness and The Secret Sharer. These demanding and disturbing novels of Machen, De la Mare, and Blackwood belong in their company. But there are many more examples in the twentieth century, from the light-hearted to the tragic: Sylvia Townsend Warner’s Lolly Willowes, David Garnett’s Lady Into Fox, David Lindsay’s A Voyage to Arcturus, Franz Kafka’s Metamorphosis, H.P. Lovecraft’s The Shadow Out of Time, Charles Williams’ All Hallow’s Eve, John Crowley’s Little, Big, and Philip K. Dick’s The Three Stigmata of Palmer Eldritch, to mention just a few. Such books remind us that we are all strangers and pilgrims.