Scarica Mapping Audiovisual Texts: Understanding the Role of Verbal and Non-verbal Communication e più Slide in PDF di Storia Inglese solo su Docsity!
The nature of the audiovisual text and its parameters
Patrick Zabalbeascoa, Universitat Pompeu Fabra, Barcelona, Spain
1. Introduction If we accept a text as a speech act or, more broadly, as any instance of communication, we will conclude that an audiovisual (AV) text is a communication act involving sounds and images. One of the main issues in the definition of AV text, and even ‘text’ plain and simple, is whether the presence of linguistic constituents is an absolute must. Can we still speak of text (and translation) if there are no words involved? And if words are an essential component, are they the only component? If they are not the only component, what is the minimum proportion of verbal elements that is required? Let us think for a minute of cartoons, on paper or film, where there is not a word to be seen or heard. Even when there are no words, these cartoons seem to fit well into what we intuitively think of as a text. Among other things, a story of some sort is told; there is an author, a reader or viewer (a text user), often a beginning, a middle, and an end; there are characters, there is action and description, often accompanied by food for thought and a moral. But if a series of pictures is a text, then what about a single picture? There are many newspaper cartoons made up of only one drawing. Now, let us think of a painting. A painting is also a single picture. But we do not normally think of a painting as a text – unless it is pointed out or we think carefully about the matter, as in cultural studies for instance – although most paintings have a story of some sort to tell (you can ask a child to tell you about their drawing or read bulky specialised literature on the semiotics of the visual arts throughout history and across the world). The issue of the picture as text (or not) raises the question as to whether the object world represented in many pictures should or should not also be considered as possessing text-like qualities. Indeed, an object (animal, vegetable, or mineral) is not necessarily a text, but under certain conditions its presence might be perceived as related to somebody trying to communicate something through that object, either as a symbol or part of a special code- system. On the other hand, our perception and understanding of the object world is greatly influenced by our cultural background, which includes the texts we have been exposed to (e.g. conversations, books, posters, lectures, mass media). This means that a book may be seen as a text (something to read) or an object (its physical properties), and also as a possession, a commodity with its personal, social or market value. A garment may be seen as an object (something to wear), a commodity (something to buy or otherwise own), or as a text (a statement of one’s cultural background, trendiness, mood, taste, daring, degree of
self-consciousness, social standing, identification with brand names, likeness of thinking, among many others). Moving pictures fit this scheme in a rather more complicated way. Firstly, there is the whole question of how to interpret objects (audio as well as visual) recorded onto the film. Secondly, the object nature of a film points to what it is recorded onto, celluloid or videotape, but we might also have to think of all of the projecting devices, including the screen, of course. The text is the projection of the film onto the screen for a given audience. The possession feature has several aspects: the owner of the rights to sell and hire, the owner of a licence to exhibit in public, the owner of a copy for personal use. The issue of rights unfolds to television rights, cinema screening rights and videotape/DVD renting or selling rights. If we examine many of the features that are supposed to define textuality, we see that they can be presented by nonverbal means as well as by words. Thus, there are nonverbal means of achieving cohesion, coherence, intentionality, informativity, acceptability, intertextuality, and complying with situationality conditions. 1 Many people do actually think that we can ‘read’ a painting or even a sculpture, that these things (object-texts) have a ‘statement’ to make on behalf of their authors. It has to be said, as a starting point, that there are verbal texts and non-verbal texts, and that there are texts that combine both verbal and nonverbal signs. If we accept this, the next step is to find out what kinds of relationships can be established between verbal and nonverbal signs within a text. Do they run along parallel lines, almost independently, or do they intertwine in a complex mesh that cannot be undone without destroying the essence of the message or without compromising intended textuality? In film translation, the former was often perceived to be the case; the words were meant to be translated as if they were one side of a coin, ultimately physically bound to the picture, but looked at separately. From this point of view, there has been an acceptance that certain words, even phonemes, have to appear at certain points in the film because they run parallel to certain pictures, hence the whole concept of synchronicity. But there has been little awareness of the possibility that verbal signs combine in various ways with other sounds and images to make up different patterns of cohesion, intertextuality and the other features of textual structure and meaning. This led to the proposal of the theoretical concept of ‘constrained’ translation (Mayoral, Kelly, Gallardo, 1988). The reasoning behind this proposal is that in translating the words of a film, one is up against the same situation as translating any other written form of communication plus the additional constraint of having to synchronise the words of the translation with the picture (and, presumably, the original sound effects), i.e. having to place the string of words alongside the parallel movement of the picture. The concept of constrained translation has had a rather negative effect since to some it seems to imply that such phenomena could not be regarded as translation proper, since translation proper must deal exclusively with words, whereas certain modes of translation had to account for other problems which translation theorists were not interested in considering yet. So, the concept of constrained translation has sometimes been used as a label to brand any variety of translation that forced the unwilling theorist to consider the important role of nonverbal elements, including the translation of songs and comics. It thus becomes necessary to beware of theorists who have a tendency to exclude certain types of text from their theory because they are complicating factors and mess up neat simplified accounts of what translation is or should be.
double axis (Figure 2). It is the kind of communication where the text users use their eyesight (to look, to watch and to read) and their ears (to listen to speech and other sounds) throughout the viewing of the AV text. In cases of interactive screen communication, users become interlocutors and use also speech, and possibly writing, and maybe even body language and other semiotic systems like in the case of phone-ins, emails and web-cams. This condition can sometimes be met by certain communicative situations where no screens are present (e.g. classroom presentations), and it is important to remember that some scholars (Merino, 2001) defend the idea that audiovisual should not necessarily entail the presence of a screen, and stage productions should be included in a general theory of AVT. (2) The various elements are meant to be essentially complementary, and as such may be regarded as inseparable for a fully satisfactory communication event. Furthermore, they have been produced specifically for the AV text at hand, or appear to be the only / best choice of performers / music / setting / words, etc. In other words, the music has been chosen to go with the pictures and words, for example, or any other combination of AV text items made to complement each other, regardless of whether they had been produced prior to their inclusion in the audiovisual production. The music in a film may be original or not, but what matters most, from a textual and communicative point of view, is the relationship established between the music, and the script, and the photography, and how they all add up and combine with each other, so that viewers can interpret them in certain ways. (3) There are three main stages of production: (a) pre-shooting (script writing, casting, rehearsing, etc.) and/or planning; (b) shooting (including directing, camera operating, make-up, and acting); (c) post-shooting (editing and cutting). We can see that this dynamics changes quite considerably for revoicing and subtitling, as perceived traditionally, in the sense that the picture and effects were untouchable.
3. Plotting text items and text types Let us see what might happen if we wished to plot texts, text-types or textual items onto the double axis plane illustrated in Figure 2, according to their ‘audiovisuality’ and the importance of verbal constituents. Strictly speaking, an AV text, such as a feature film, could not be without an audio component, or without the visual component, so, categories A and E (the far left and right of the horizontal axis) would fall out of the audiovisual domain, in principle. However, we may not want to be too strict, and it might be theoretically more advantageous to regard audiovisual as having the potential to include, at least at certain points, any combination of audio, visual, verbal and non-verbal elements. The whole range of television commercials, regarding their exploitation of code systems and channels, provide a handy case in point, since some of them have no sound, others use a blank screen, and so on.
Figure 2 The two axes of audiovisual communication A: only audio 1: basically verbal B: more audio than visual 2: more verbal than non-verbal C: audio and visual alike 3: both verbal and non-verbal alike D: less audio than visual 4: less verbal than non-verbal E: only visual 5: only non-verbal Below are a series of examples to illustrate what is meant by different combinations of different sign systems and channels and how they might be plotted on the ‘map’ in Figure 2. 1: Basically verbal 1A Oral speech with no visual contact or aid, no nonverbal sound effects. Voice only, over a public address system or on the radio. 1B Oral speech with the presence of paralinguistic facial expressions and body language, some written aids (e.g. posters, badges) may also be present. 1C As for 1B, plus important written material (writing on blackboard or transparencies, handouts). We might also include in this category those performative acts of reading out sentences or other legally binding documents that must be heard as well as seen. 1D Group reading led by someone (e.g. a teacher, a preacher) reading out loud from a book. 1E Silent reading of a message where layout and format cannot be altered, so they are not regarded as having any potential meaning, e.g. telegram, early email. A B C D E 1 + verbal
- non-verbal
- audio + visual 2 3 4 5
4B Televised rock music, with rather unimportant lyrics and pictures. 4C MTV video-clip music. Video music is by definition audio and visual, where both are equally important and balanced. This does not mean that they are inseparable. Words are not usually the most important feature although they can appear orally or graphemically. Karaoke combines music and captions and the non-verbal picture is separable and less important. 4D Candid camera, mostly visual, little presence of sound, except for one or two noises like bumps and screams, and a few words to introduce the trick or a small dialogue to start it off. 4E Silent film with intertitles, ‘train platform’ television with no synchronised sound. 5: Only non-verbal 5A The bottom left-hand corner of the ‘map’ in Figure 2 would include phenomena like symphony music (i.e. no lyrics) and whistle signals. Morse beeps constitute a hybrid case of non-oral, non-written signals that substitute verbal signs and thus cannot be considered non-verbal strictly speaking. Sign language for people with speech impediments is an analogous case for 5E, when each sign is a letter or word associated to a given language. But when the signs are more abstract or are not language-specific they shed their ‘verbal’ quality and become more closely related to international traffic signs. 5B Televised symphony orchestra, where the music is the main thing but it is also interesting and entertaining to watch the performers, and any noteworthy camera work. 5C Modern ‘silent films’, which tend to use sound effects even if there is no talk. 5D Fashion TV is a digital satellite channel where there is practically no oral communication, and very little writing on the screen, at least in its original format. English is used as a sort of lingua franca, and French reinforces the stereotype of fashion and glamour. The main attraction is the picture, to the extent that this is a popular channel to show in bars and shop windows with the sound off, so as not to interfere with other sounds like conversation or music. This is a practical example of separability of the sound from the picture, and, to a large extent, the picture is separable from all other semiotic sign systems. 5E Silent non-verbal films; fine arts (painting, sculpture); traffic signposts; mime; nonverbal cartoon strips; photography; formal languages (as understood for mathematics, logic and computer science) like ‘2+2=4’, ‘H 2 O’, and ‘e=mc 2 ’. In the case of mime it is important to distinguish between pure nonverbal mime, and the miming of syllables (as in some of the Marx Brothers’ films). Harpo Marx’s miming constitutes a particularly challenging problem for translation because it is obviously very difficult to translate each syllable as word, or verbal clue, and then make the sum of all the syllable-translations become an equivalent of the original word or phrase. For example, there is an instance where ‘beautiful’ is broken down, in mime, into bee + you + tea + full. It is bizarre to have a character, in dubbing, guess the right word from literal translations of each clue. In the case of formal languages, it is important to
remember that they are not necessarily universal, and that they are often only non- linguistic in writing, not when read out loud, of course.
4. The nature of the audiovisual text and its translation An AV text is a mode of communication that is distinct from the written and the oral modes, although it may not be easy to draw a clear borderline between the audiovisual and other modes. Even so, I am proposing that it is possible to map AV texts, types of AV texts and parts of them (Zabalbeascoa, 2001a: 119) on a plane defined by the following coordinates: a cline that indicates the presence (amount and importance) of verbal communication in proportion to other semiotic forms of expression; another cline for measuring the relative importance of sound in the audio channel weighed against visual signs. According to this scheme (Figure 3), the area that is closest to the centre of both clines (area X) is where one would expect to find prototypical instances of the AV text, in which intentions and meanings are conveyed (and effects produced) through both audio and visual channels and both verbal and non-verbal sign systems or codes (Figure 1) all acting together. To move away from this centre means that greater importance is being awarded to one channel over the other and/or the verbal sign system over non-verbal forms of expression (or vice versa). For example, a ‘writer’s’ film tends to favour the importance of the words heard (when there is no subtitling involved) over the musical score or the picture, so many of its scenes would appear nearer audio-verbal corner (area Y). The visual-non- verbal corner (area Z) indicates a kind of film, commercial or documentary, with pictures as its main constituents, giving little or no importance to verbal communication. Figure 3 The double axis of the audiovisual text +audio + visual + verbal
Z
X I further illustrate X, Y, and Z with examples from British television comedy. For X, The Black Adder (Martin Shardlow , 1983) is one of many programmes written specifically for television and home video. For instance, there is a scene where Lord Black Adder tells his servant to ‘Get the door, Baldrick’, after a while Baldrick comes back with the door under his arm. The obvious difficulty involved in translating this joke arises for those languages
lack of attention, or certain impairment in hearing or in eyesight. This might be the reason why many television commercials are so redundant because their authors do not know where people are looking, what else they might be listening to, or how loud the television is on. Thus, a number of commercials can be perfectly understood with the sound off or without looking at the picture. In this case we also talk of separability. Contradiction (or incongruity): defeated expectations, or some sort of surprising combination to create such effects as irony, paradox, parody, satire, humour, metaphor, symbolism. Mock incoherence, whereby incoherence is feigned; for example when a film presents the order of events in a seemingly incoherent manner, or when parody or symbolism involve an artful appearance of incoherence. Incoherence : inability to combine elements meaningfully, or as intended (in the source text or otherwise) because of failings in the script, the directing, the translation (of the script), the subtitling (techniques, norms, display), or the sound (i.e. revoicing, mixing, editing, special effects, music). Separability : a feature displayed by elements of a channel or sign system whereby they manage to function (better or worse) autonomously or independently from the AV text, as when the soundtrack is made into a successful audio recording. Free commentary is a type of AVT which, to large extent, depends on the separability of the original scripted words from the picture, i.e. the picture is such that it enables a new script to substitute the old one, in order to create a completely different sort of complementarity between word and picture. Aesthetic quality : text author’s intention to produce something of beauty by means of a certain combination of elements. If we regard filmmaking as an art form, then it involves combining the visual arts with literature, photography, and music. A translator might deal with certain parts of a text by giving priority to their aesthetic quality, rather than their semantic value. The relationship of separability (Zabalbeascoa, 2001a: 124) is proposed in order to relativise the degree of symbiosis and dependency between certain text items or sets of items. Complementarity is not necessarily unalterable, and constituent parts of a complementary relationship may, at times, be separated from the whole, generally providing a new meaning, and often a new lease of life for the newly independent parts. Of course, the semiotic and pragmatic value of an element or set will change depending on whether or not it appears in the company of other elements also intended for the same AV text. We must admit at the same time that there are cases where a soundtrack, for example, ends up living an independent life from the pictures and film it originated from, providing musical (‘pure’ audio) enjoyment, and most probably new contexts and meanings for the very same (recording) score, songs, voices, and musicians. Further examples of separability are: x ‘Decorative’ TV, such as MTV and Fashion TV , or certain sports channels, ideal for shop windows and bars, where the picture is often separable, i.e. seen without any sound or with a different sound source. x Television news, especially 24-hour channels, in which the newsreader and the captions are mutually redundant. Sometimes, these two are joined by the further redundancy of the picture, whereby the audience can understand a news item either by listening, by
watching or by reading, i.e. each one is separable. The fact that these combinations are redundant is illustrated by the “non-redundant” case of the ‘no comment’ stretches where the news programme continues only with pictures. x National anthems, and popular songs: a national anthem is a typical example of a song that can perform its function or be almost as meaningful with the words or without them, i.e. the music is separable from the words, and can stand alone on certain occasions. A trickier example, frequently found in films, and hence a translation problem, is when a song is played in a purely instrumental version because the lyrics, or at least their essence, are assumed to be known by the audience. We might have, for instance, a few bars of the Beatles’ Yesterday to underline the nostalgia of a certain scene or visual composition. x Films and cartoons: for some films, the musical score is separable from the rest of the film. Conversely, Tom and Jerry type cartoons provide examples of visual stories that hardly need the audio component.
5. The semiotics of AV The semiotic dimension of audiovisual communication requires: (i) a photographic analysis of stills, in search for a greater understanding of photographic composition, the relationship between film and photography and the visual arts, the use of colour, light and (visual) texture; (ii) a cinematic analysis of the relationships established (between stills) by moving pictures and sound, audiovisual narrative techniques, audiovisual cohesion, audiovisual rhetorical devices (such as repetition, ellipsis, metaphor, and metonymy) and the use of the camera (shifts of perspective, focus, light and colour, as well as camera movements like zooming and scanning). Each one these aspects may be intended to carry meaning or to help make the meaning of the words and script more explicit or dynamic. In the light of all that we have seen thus far, there seem to be sufficient grounds to propose an alternative (or at least a refinement) for the early concept of constrained translation, put forward to account for the translation of films, songs, and highly illustrated texts, such as comic strips (Figure 4):
Figure 5 An alternative model to constrained translation
The creation of a target text where everything fits into place (2)
accounts for the relative importance of the source text words (1).
AV source text AV target text
non-verbal possible change non-verbal
elements material or cognitive elements
source-text target-text
words words
3. cultural and knowledge “barrier” (differences)
From the point of view illustrated by Figure 5, the translator must be aware for each scene, for each frame almost, of which are the most important and relevant items (verbal or otherwise) in the meaning(s) and function(s) of the (AV) source text (Zabalbeascoa, 2000c:
- so as to make informed, context-sensitive, function-oriented, audiovisually-coherent decisions as to the words that will be the most appropriate for the task at hand. In other words, the best-case scenario is one where the translator manages to render the combined meaning(s) and function(s) of the words, icons and sounds, although most of the time solutions are restricted to manipulations on the verbal plane alone.
6. Parameters of the audiovisual factor Audiovisual communication requires careful thought into the nature of text modes and media as well as how texts are stored and distributed. The first thing that strikes the theoretician is that neat compartmentalisation (i.e. typologies and classifications with uncrossable, everlasting, unmovable dividing lines) is almost completely out of the question given the constant progress of technology and social dynamics. Below is a list (see also table 2) of five types of factors that are both a reflection of the nature of audiovisual communication and evidence of the difficulty in presenting a single straightforward classification of audiovisual types which could then lead on to making statements about each one to the exclusion of the others, since the items of each one of the five types would have to be crossed with the all the others (i.e. to see, for instance, how the complicating factors of text mode combine with and affect those of text medium or audience impact). Nor is it always possible to make sweeping statements to the effect that television is one thing, theatre is another and Internet is another, and then define each one independently. For example, film and stage productions used to be perceived as distinct from one another, but now many stage performances use screens as part of their props. Television and Internet used to be separate, but now they are converging. One can now watch a film on a cell
phone, or perform a play for which the audience will decide the ending. So, rather than searching for hard-and-fast classifications, it is probably more operative simply to understand which factors come into play for each case, and how one case might be relevant for others. Table 2 Examples of complicating factors vs ‘straightforward’ types for audiovisual translation Parameters ‘Straightforward’ types Complicating factors Text mode Written v. oral Audiovisual / multimodal Ephemeral v. recorded/ published Text medium Cinema v. television / video rental Internet television / multimedia Reader/ listener: present or distant Ownership / storage VHS / celluloid / DVD Internet access / computer files / clips Audience profile and impact Nationwide audience v. Special-interest groups SPAM, Internet pop-ups / breaking-news captions
- Text mode : the traditional binary division of textual mode, into written and oral, is problematic when we think in audiovisual terms. If there are only two language modes, oral and written, then any combination of oral speech and writing may be regarded as multimodal, including audiovisual texts. By this token, audiovisual texts are multimodal, since they potentially include oral and written language. If there are only two language modes, oral and written, then all non-verbal and paralinguistic constituents of communication acts that include either oral or written signs are simply incorporated into either one mode or the other (and forgotten or brought to the fore when convenient). ‘Oral mode’ would thus include (has included?) lots of cases of: film and television with no on-screen writing; stage performances (plays, recitals, etc). ‘Written mode’ would have to include things like comics, cartoons, and instruction booklets with diagrams. Alternatively, we might think of ‘audiovisual’ as a third mode, or even as an overarching category for all possible combinations of verbal and non-verbal sounds, graphemes and other visual semiotic signs. In either case, oral and written domains will probably ‘lose’ forms of communication like films and comics.
- Text medium/projection : how is the audiovisual text transmitted and received? The answer to this question provides types of media, such as: cinema theatre, family celluloid projector, television broadcast, video player (TV, ‘camcorder’, and portable video), Internet and computer screens, open-air large screens (stadiums, metropolitan squares), publicly-positioned screens (railway platforms, waiting rooms, bars, etc.), mobile phones. Writing can be done while the reader is present (e.g. blackboard), or at a distance, when the reader is absent, (e.g. correspondence). Oral can also be divided into ‘interlocutors present’ (e.g. breakfast table conversation, especially from behind the newspaper, thus making it more purely oral and less visual); ‘interlocutors at a distance’ (e.g. traditional telephone). Audiovisual communication might also consider this division of ‘viewers present’ (e.g. audience inside a theatre); or ‘real time’ (e.g.
regardless of whether more might appear in the future, since, again the point is to provide insight and pointers, not to be authoritative or prescriptive. Finally, there is an alternative to the whole concept of constrained translation, which appears almost as a logical conclusion to the previous proposals. Notes
- Textuality conditions as proposed by de Beaugrande and Dressler (1981), but we could also apply other models, such as Sperber and Wilson’s (1986) relevance theory, or Grice’s (1975) conversational maxims. References Chaume, F. (2000): «La traducción audiovisual: estudio descriptivo y modelo de análisis de los textos audiovisuales para su traducción», PhD manuscript submitted at Universitat Jaume I de Castellón. de Beaugrande, R. and W. U. Dressler. 1981. Introduction to Text Linguistics. London: Longman. Delabastita, D. (1989): «Translation and Mass-communication: Film and T.V. Translation as Evidence of Cultural Dynamics», Babel 35:4. p. 193 - 218. Fodor, I. 1976. Film Dubbing: Phonetic, Semiotic, Esthetic and Psycological Aspects. Hamburg: Buske Verlag. Grice, H. P. 1975. “Logic and conversation”. In P. Cole and J. Morgan (eds.) Syntax and Semantics 3: Speech Acts. New York: Academic Press, 41-58. Holmes, J. S. 1972 (1988). “The name and nature of Translation Studies”. In Translated! Papers on Literary Translation and Translation Studies. Amsterdam: Rodopi. Mayoral, R., D. Kelly and N. Gallardo. 1988. “Concept of constrained translation. Non- Linguistic perspectives of translation”. Meta 33(3): 356-367. www.erudit.org/revue/meta/1988/v33/n3/003608ar.pdf Merino, R. 2001. “Textos dramáticos traducidos y censurados en la España de Franco (años sesenta): el teatro y el cine, espectáculos controlados”. In J. D. Sanderson (ed.) ¡Doble o nada! Actas de las I y II Jornadas de doblaje y subtitulación. Alacant: Universitat d’Alacant, 85-97. Sokoli, S. (2000): «Research Issues in Audiovisual Translation: Aspects of Subtitling in Greece», Pre-doctoral research dissertation manuscript for Universitat Autònoma de Barcelona. Sperber, D. and D. Wilson. 1986 (1995). Relevance: Communication and Cognition. 2 nd edition. Oxford: Blackwell.
Zabalbeascoa, P. 2001a. “El texto audiovisual: factores semióticos y traducción”. In J. D. Sanderson (ed.) ¡Doble o Nada! Actas de las I y II Jornadas de doblaje y subtitulación. Alacant: Universitat d’Alacant, 113-126. Zabalbeascoa, P. 2001b. “La traducción del humor en textos audiovisuales”. In M. Duro (coord.) La traducción para el doblaje y la subtitulación. Madrid: Cátedra, 251-263. Zabalbeascoa, P. 2001c. “La ambición y la subjetividad de una traducción desde un modelo de prioridades y restricciones”. In E. Sánchez Trigo and O. Díaz Fauces (eds.) Traducción y Comunicación (II). Vigo: Universidade de Vigo, 129-150. Zabalbeascoa, P. 2003 “Translating audiovisual screen irony”. In L. Pérez González (ed.) Speaking in Tongues: Languages across Contexts and Users. Valencia: Universitat de València, 303-322.
