What is a mantra? This is not an easy question to answer. In reading about mantra you will typically come across a confusion of Buddhist, Hindu, and New-age ideas couched in terms that suggest that knowledge of mantras is so esoteric that nothing sensible can be said about it. Usually this means that no attempt is made to make the connection between mantra and Buddhist doctrine. Most serious teaching on the subject is purely pragmatic - "now, you chant this mantra in this way...". Often nothing at all is said about why you chant a mantra, or how a mantra achieves its goal. Compare this situation with meditation upon which dozens of books have been written over thousands of years spelling out in great detail what happens in the mind, why you do this and not that etc. Mantra begins to sound like rank superstition. I accept that in the long run that Awakening defies the intellect and words: but I do not accept that no rational account of Buddhist practice is possible.
After offering a very brief introduction to the hisotry of mantra, I introduce a threefold clsssification of mantras based on use and context.
Mantra is an old word. It occurs in the Ṛg Veda where it refers to the poetic hymns to the wild chthonic forces of nature which we think of as the Vedic gods. Sanskritists tell us that the word refers to the instrumentality (tra) of the mind (man). A later Tantric etymology suggests that it is something which protects the mind. Clearly in this case neither etymology tells us much. Frits Staal - an eminent Sanskritist - goes so far as to suggest that Vedic mantras, especially the ones that most resemble Buddhist mantra, are not language at all. The reasons that Buddhists started to assign significance (if not meaning) to syllables such as oṃ or hūṃ remains obscure.
The Pali Canon forbids monks from chanting mantras or from putting the Dharma into Vedic style verses for chanting; and the Buddha repeatedly mocks mantra chanting Brahmins. However at the same time there are a number of canonical chants (paritta) for warding off misfortune, and in the early post-canonical literature there are ceremonies which have formalised this. The paritta texts are frequently cited as precursors of Buddhist mantra, although they bear very little relationship (formally or doctrinally) to Tantric Buddhism.
Around the 2nd century strings of words, often with no grammatical relationship to each other, began to appear in sūtras - either as a main element or as an interpolation in the body of a larger sūtra. This was the dhāraṇī. Sometimes, as in the Golden Light Sūtra these dhāraṇī appear alongside apparently Hindu rituals which are not entirely assimilated into Buddhism. The term may have first been used in reference to the verses, the lines of which are in the order of the Gāndhārī alphabet (aka the Arapacana Alphabet), and relate aspects of the nature of dharmas. It means something like a memory aid, which it may have been originally, but actual dhāraṇī's seldom have this function and are more often simply prophylactics. The chanting of dhāraṇī became a very important practice in East Asian Buddhism as they functioned as magical protection for emperor and empire. Many scholars see dhāraṇī as a kind of proto-mantra. The chanting of Mahāyāna sūtras offers the chanter protection as well.
The final phase of development begins in the mid 7th century. Texts appear in which mantras are suddenly the main feature. These texts describe elaborate rituals of initiation and propitiation the goal of which is to transform the practitioner into a Buddha. These texts are not taught by Śakyamuni but come directly from the Dharmakāya Buddha. In the Tantric ritual each activity is marked, or even empowered, by a mantra. The fundamental ritual is the abhiṣeka or initiation. It was through the abhiṣeka that the Dharmakāya Buddha communicated his Awakened state - via the triple medium of mudra, mantra, and mandala - and it was this ritual which was passed down via Vajrasattva, Nagarjuna, Nagabodhi, Vajrabodhi, Amoghavajra, and Huiguo to Kūkai. The abhiṣeka ritual underlies all tantric sadhanas and each step - establishing the mandala, occupying the mandala, invoking and summoning the deity, etc - has an associated mantra, although one mantra in particular, the hṛdaya, (or heart) is used by the deity to transform the practitioner into a Buddha. This transformation is the point of tantric sadhana - as Kūkai was fond of saying "Awakening in this very life!". It is these hṛdaya mantras which feature on visiblemantra.org.
Tantra incorporates many ideas about sound, words, and spoken language which come from the Vedic tradition (ie from texts like the Ṛg and Atharva Vedas, and the Upaniṣads); from the more metaphysical speculations of the Sanskrit grammarians; and from Purāṇic Hinduism, especially the Śaiva branches. These are assimilated into the Buddhist context, and various attempts are made to weld them together. However the emphasis in this context is definitely on practice - one "understands" a mantra by chanting it 100,000 or a million times, not by thinking about it. The question of what a mantra means is far less important than what it does, and what it does can only be discovered by using it.
There are three main contexts in which mantras are used in Buddhism: Tantric Ritual, devotional rituals, and non-ritual personal use.
Mantra in Tantric Ritual
Fundamental to the Tantric mantra is the old Vedic idea of the interpenetration of phenomena incorporated via the Avataṃsaka Sūtra. As Michel Foucault has noted the ancients saw knowledge not in terms of identity and difference, but in terms of relationship and similarity. The Vedic priests could control wild nature (especially the monsoon and sun) by finding something similar on earth and manipulating it. Around this notion developed the elaborate ritual tradition of the Brahmins in which fire (personified as Agni) was the medium of communication between heaven and earth because it transformed things from one state into another: Agni played the role of Hermes, or of the Angels. The relationship became more and more abstract so that one could manipulate the world via imagination alone. In Buddhist terms simply bringing the Buddha to mind was to be in the presence of the Buddha in a literal sense. The idea of the ritual was implicit in the sadhana, and explicit in the fire rituals (Homa) imported directly from the Vedic tradition.
This is not a mystical explanation, it simply relies on an episteme (a way of knowing) which is not popular in the West. An exception would be something like Homeopathy which operates on the same principles, and provided only that you believe in it, it does actually work!
Kūkai used to say: "all sounds of the voice of the Dharmakāya preaching the Dharma". I take this to mean that the nature of all reality is explicit in all experiences. Experience is marked by impermanence and insubstantiality. Sounds are particularly well suited for contemplating the truth of this since they occupy a time frame which makes their arising and passing away more noticeable than thoughts or forms. Sound is naturally impermanent and insubstantial. Mantra here forms a focus for the contemplation of the central Buddhist insight into the nature of experience: pratītya-samutpāda or dependent arising.
The rules that pertain to this context are specific and localised. One must receive an initiation from a qualified master or you may chant the sounds and still not be chanting a mantra. One must chant the mantra at the appropriate place in a internalised ritual handed down from the Dharmakāya through generations of masters, accompanied by the appropriate mudra, else the mantra has no power. These rules are being relaxed to serve the needs of the Tibetan community in exile, and Western Disciples, however this causes some Buddhists a certain amount of disquiet. Part of the problem is that many mantra practitioners only have the Tantric paradigm with which to explain mantra, and yet there is a clear and traditional non-tantric explanation.
In devotional rituals mantras are used to evoke a Buddha, and to expression devotional feelings, and faith in the vow of the Buddha to save beings. The Buddha Amitābha is a frequent object of devotion, and features in Pureland Buddhism which emphasises this style of practice. In the Kārandavyūha Sūtra the mantra of Avalokiteśvara (oṃ maṇipadme hūṃ) is intended to be used in this way, even though later it does become a fully tantric mantra. Mantra use in this context is essentially a form of nāmanusmṛti - recalling [smṛti] of the name [nāma] of (usually) Amitābha, which activates his vow to ensure that any person who keeps in mind his name will be reborn in the pureland. There is also an element of an even older practice - buddhānusmṛti or bring the Buddha to mind. By imaginatively bring the Buddha to mind the practitioner can enhance feelings of inspiration to practice, and devotion.
An aspect of this style of practice is that it implies an "other power" which can help us. We rely on Amitābha or the Buddha to rescue us. This may seem to run counter to some Buddhist doctrines which say that we are responsible for the consequences of our actions, the fruits of our karma. Sangharakshita encourages us to see this type of practice in terms of Going for Refuge. It is the act of Going for Refuge which makes the Buddhist, and this kind of mantra practice can be seen as orthodox in that one is expressing one's Going for Refuge to the Buddha. However Going for Refuge must also manifest in behaviour and at the very least the Buddhist must practice ethics.
Rhythmic chanting or even singing is known to stimulate the release of endorphins which may account for the feeling of well being, and even ecstasy, which accompanies group chanting - a similar effect can be observed in secular choirs.
Finally many Buddhists use mantras in non-ritual contexts as expressions of devotion and faith, and also as a form of petitionary prayer to a Buddha. It is frequently noted that the words oṃ maṇipeme hūṃ are always on the lips of ordinary Tibetans. No rules apply to this kind of mantra use - what happens depends on the intent of the person chanting, and one often finds new age ideas of "powerful vibrations" at work. Some practitioners swear that chanting a mantra has warded off particular misfortunes, and many simply feel a sense of comfort from chanting. Such applications can also be seen in terms of Going for Refuge, as expressions of an individuals faith. Outside of the ritual context, however, the use of a mantra seems to me to verge on superstition: if I chant a mantra in order to ask for help from a supernatural being then it begins to look like theism and there is not much to distinguish it from prayer to say a saint or the Virgin Mary. The crucial difference is in the Buddhist placing of responsibility for the results of actions in the individual and not in the supernatural being. There is no substitute for the practice of ethics and meditation.
The explanation of mantra, then, is dependent on the context. Each context requires it's own explanation, and the rules of one may not apply in the other. In most accounts of mantra reference is only made to the Tantric context, but since this has specific requirements - such as abhiṣeka, accompanying mudras etc - it is seldom applicable outside the Tantric ritual. In this case we can consider mantra as a form of devotional practice, a recollection of the Buddha in order to bolster our determination to do what needs to be done.