散文分镜语言：用于电影标注与导演的工具（中文全文）

The prose storyboard language is a formal language for describing movies shot by shot, where each shot is described with a unique sentence. The language uses a simple syntax and limited vocabulary borrowed from working practices in traditional movie-making and is intended to be readable both by machines and humans. The language has been designed over the last ten years to serve as a high-level user interface for intelligent cinematography and editing systems. In this new paper, we present the latest evolution of the language, and the results of an extensive annotation exercise showing the benefits of the language in the task of annotating the sophisticated cinematography and film editing of classic movies.

In movie production, directors often use a semi-formal idiom of natural language to convey the shots they want to their cinematographer. Similarly, film scholars use a semi-formal idiom of natural language to describe the visual composition of shots in produced movies to their readers. In order to build intelligent and expressive virtual cinematography and editing systems, we believe the same kind of high-level descriptions need to be agreed upon. In this paper, we propose a formal language that can serve that role. Our primary goal in proposing this language is to build software cinematography agents that can take such formal descriptions as input, and produce fully realized shots as an output. A secondary goal is to perform in-depth studies of film style by automatically analyzing real movies and their scripts in terms of the proposed language.

The prose storyboard language is a formal language for describing shots visually. We leave the description of the soundtrack for future work. The prose storyboard language separately describes the spatial structure of individual movie frames (compositions) and their temporal structure (shots). In film analysis, there is frequent confusion between shots and compositions. A medium shot describes a composition, not a shot. If the actor moves towards the camera in the same shot, the composition will change to a close shot and so on. Therefore, a general language for describing shots cannot be limited to describing compositions such as medium shot or close shot but should also describe screen events which change the composition during the shot. In the prose storyboard language, each shot is a complete sentence with at least one composition and an arbitrary number of screen events. This offers unprecedented expressive power for describing and directing movies.

Our language can be used indifferently to describe shots in pre-production (when the movie only exists in the screen-writer and director's minds), during production (when the camera records a continuous "shot" between the times when the director calls "camera" and "cut"), in post-production (when shots are cut and assembled by the film editor) or to describe existing movies. The description of an entire movie is an ordered list of sentences, one per shot. Exceptionally, a movie with a single shot, such as Rope by Alfred Hitchcock, can be described with a single, long sentence.

In this paper, we assume that all shot descriptions are manually created. We leave for future work the important issue of automatically generating prose storyboards from existing movies, where a number of existing techniques can be used [Bra97] [DMR05] [GCSS06] [GR13]. We also leave for future work the difficult problems of automatically generating movies from their prose storyboards, where existing techniques in virtual camera control can be used [HCS96] [CAwH*96] [JY06] [CO06] [GCR*13] [GRCS14] [GCLR15].

Our language is loosely based on existing practices in movie-making [TB09b] [TB09a] and previous research in the history of film style [Bor98] [Sal09]. Our language is also related to the common practice of graphic storyboards. In a graphic storyboard, each composition is illustrated with a single drawing. The blocking of the camera and actors can be depicted with a conventional system of arrows within each frame, or with a separate set of floor plan views, or with titles between frames.

In our case, the transitions between compositions use a small vocabulary of screen events including camera actions (pan, dolly, crane, hold, continue) and actor actions (speak, react, move, cross). Although the action vocabulary could easily be extended, we voluntarily keep it small because our focus in this paper is restricted to the blocking of actors and cameras, not the high-level semantics of the narrative.

We borrow the term prose storyboard from Proferes [Pro08] who used it as a technique for decomposing a films script into a sequence of shots, expressed in natural language. Other authors use the French term "decoupage" to describe this important step in film production. The name catches the intuition that the language should directly translate to images. In contrast to Proferes, our prose storyboard language is a formal language, with a well-defined syntax and semantics, suitable for future work in intelligent cinematography and editing.

Our proposal is complementary to the Movie Script Markup Language (MSML) [RKV*09], which encodes the structure of a movie script. In MSML, a script is decomposed into dialogue and action blocks but does not describe how each block is translated into shots. Our prose storyboard language can be used to describe the blocking of the shots in a movie in relation to an MSML-encoded movie script. For this purpose, MSML makes provision for a Manufacturing model and Animation model, which are only loosely described. The prose storyboard language can be seen as an alternative representation for both the Manufacturing and Animation models in MSML.

Our proposal is also related to the Declarative Camera Control Language (DCCL) which describes film idioms, not in terms of cameras in world coordinates but in terms of shots in screen coordinates [CAwH*96]. The DCCL is compiled into a film tree, which contains all the possible editings of the input actions, where actions are represented as subject-verb-object triples. Our prose storyboard language can be used in coordination with such constructs to guide a more extensive set of shot categories, including complex and developing shots.

Our approach is also related to the work of Jhala and Young who used the movie Rope by Alfred Hitchcock to demonstrate how the storyline and the director's goal should be represented to an automatic editing system [JY06]. They used Crossbow, a partial order causal link planner, to solve for the best editing, according to a variety of strategies, including maintaining tempo and depicting emotion. They demonstrated the capability of their solver to present the same sequence in different editing styles. But their approach does not attempt to describe the set of possible shots. Our prose storyboard language attempts to fill that gap.

Other previous work in virtual cinematography [SMAY03] [JY05] [FF06] [ORN09] [MJSB11] [GRCS14] [GCLR15] [LDTA17] has been limited to simple shots with either a static camera or a single uniform camera movement. Our prose storyboard language is complementary to such previous work and can be used to define higher-level cinematic strategies, including arbitrarily complex combinations of camera and actor movements, for most existing virtual cinematography systems.

Text-to-movie (or text-to-scene) authoring is a general class of methods that have been proposed for automatically generating 3D graphics and animation from natural language text. Good results have been obtained in limited domains, such as generating 3D scenes from natural language accident reports [ASSN03] or generating cartoon animation from scripted dialogue scenes [SY06]. Commercially available text-to-movie systems include NawmalMAKE and Plotagon Studio Such systems use marked-up dialogues as input and generate simple shots with minimal camera movements and editing. Other related work along the same lines includes Ye and Baldwin [YB08] who describe a method for generating storyboards from movie scripts; Marti et al. [MVW*18] who describe methods for generating previz animation, also from movie scripts.

Closer to our approach, Director Notation [Yan13] is a symbolic language intended to express the content of film (motion pictures), much as musical notation provides a language for the writing of music. But DN is a graphical notation whereas PSL is a pseudo-natural language, and DN describes the movie production process, whereas PSL describes the movie itself, as in a storyboard. TIMISTO [VHL*13] and SLAP [BS16] are pattern languages for creating animation from storyboards.

A previous version of the prose storyboard language was presented at the international workshop on intelligent cinematography and editing (WICED) in 2012. Since then, several variations of PSL have been used for generating cinematic replays in serious games [GRCS14], for generating synthetic complex shots from live video material [GRG14] [GR15], for staging complex scenes in 3D animation [LCL18], for directing cinematographic drones [GLC*18] and for learning film editing patterns from examples [WPRC18]. In this revised edition, we provide a definitive version of the language, illustrated with a large number of examples and a reference implementation, in the hope that it will stimulate future work in automatic annotation and generation of movies.

The prose storyboard language is designed to be expressive, i.e. it should describe arbitrarily complex shots, while at the same time being compact and intuitive. Our approach has been to keep the description of simple shots as simple as possible, and allowing for more complex descriptions when needed. Thus, for example, we describe actors in composition from left to right, which is an economical and intuitive way of specifying relative actor positions in most cases. As a result, our prose storyboard language is very close to natural language (see Fig.21 for examples).

It should be easy to parse the language into a non-ambiguous semantic representation that can be matched to video content, either for the purpose of describing existing content or for generating novel content that matches the description. It should therefore be possible (at least in theory) to translate any sentence in the language into a sketch storyboard, then to a fully animated sequence.

It should also be theoretically possible to translate existing video content into a prose storyboard. This puts another requirement on the language, that it should be possible to describe existing shots just by watching them. There should be no need for contextual information, except for place and character names. As a result, the prose storyboard language can also be used as a tool for annotating complete movies and for logging shots before post-production. Since the annotator has no access to the details of the shooting plan, even during post-production [Mur86] [Ond04], we must therefore make it possible to describe the shots in screen coordinates, without any reference to world coordinates.

The prose storyboard language is a context-free language, whose terminals include generic and specific terms. Generic terminals are used to describe the main categories of screen events including camera actions (pan, dolly, cut, dissolve, etc.) and actor actions (enter, exit, cross, move, speak, react, etc.). Specific terminals are the names of characters, places and objects that compose the image and play a part in the story. Non-terminals of the language include important categories of shots (simple, complex, developing), image compositions and developments. The complete grammar for the language is illustrated with the AND/OR graph in Fig. 14 and described in the PEG notation in Fig.15. A reference implementation using the Parsimonious toolkit for parsing PSL sentences using the Python language can be found at https://gitlab.inria.fr/vmurukut/psl.

The semantics of the prose storyboard language is best described in terms of a Timed Petri Net (TPN) where durative events such as compositions and camera actions are represented as places ; and instantaneous events (such as cuts and the start and end of other events) are represented as transitions. We use timed Petri nets, rather than finite state machines, as a semantic representation of shots in a prose storyboard, in order to adequately represent developing shots with an elaborate choreography of actor and camera movements taking place simultaneously, such as the opening shot in Orson Welles' "Touch of evil".

TPNs have been proposed for representing the temporal structure of movie scripts [RKV*09], character animation [MRR98] [BvKR01], game authoring [BBAG08], turn-taking in conversation [Cha12] and synchronisation and storage models for multimedia systems [LG90]. Our model differs from previous work by representing all durative events with places and using transitions to synchronize them. For lack of space, we differ the description of the Petri Net interpretation of the prose storyboard language to future work.

Image composition is the way to organise visual elements in the motion picture frame to deliver a specific message to the audience. In our work, we propose a formal way to describe image composition in terms of the actors and objects present on the screen and the spatial and temporal relations between them.

Following Thomson and Bowen [TB09b], we define a composition as the relative position and orientation of visual elements called Subject, in screen coordinates. In the simple case of flat staging, all subjects are more or less in the same plane with the same size, but in the case of deep staging, different subjects are seen at different sizes, in different planes. For the sake of generality, we therefore choose to indicate the size of each subject separately. As a result, each subject is defined by its size, profile angle, and screen position. See Fig. 3b and 3b for a visual explanation.

As a convention, we assume that the subjects are described from left to right. This means that the left-to-right ordering of actors and objects is part of the composition. Indeed, because the left-to-right ordering of subjects is so important in cinematography and film editing, we introduce a special keyword cross for the screen event of one subject crossing over or under another subject. Shot sizes are used to describe the relative sizes of subjects independently of the camera lens as illustrated in Fig.4a.

The Screen term describe the subject position in screen coordinate. It allows to slightly modify the generic framing by shifting the subject position to the left or right corner as shown in Fig.4b. Default Screen values are used to describe symetric compositions where subjects are evenly distributed from left to right. Non-default Screen values are used to describe asymetric compositions, e.g. taking into account head room and look room or the rules of thirds. We can also describe unconventional framing to create unbalanced artistic composition or to show other visual elements in the scene.

A shot is a sequence of frames over a continuous time period. For a cohesive and coherent narration the individual shots have to be joined in a manner so as to allow the spectator to mentally recreate the story [Cut16]. Transition describes the progression of a shot to the subsequent one or the starting and ending of a shot. In our model we include three of the most widely used transition techniques: cut, dissolve and fade. We use the simplest form of cut transition in which the two shots are played one after another. We also use the same notation to describe other types of cuts, such as cutaways in which shot A is followed by a intermittent shot with a different composition and then returns back to shot A. Dissolves and fades are used to describe the entry or exit of a shot in which the composition either slowly appears or disappears respectively.

As we enter the shot the initial composition can be of two types. It can either be a 'static' composition in which the actors are not performing any action or we can transition into action. In the former case, we describe the composition in the first frame of the shot. In the latter, we describe the composition in relation to the action that is being performed (figure 1). This is described under the non-terminal whileEvent which describes the first composition of the shot while the actors are performing an Action. As we transition into action, the composition can also be accompanied by a camera movement. In Fig. 11 we transition directly into an actor action with camera movement. Here the camera tracks with the actor thereby resulting in maintaining a single composition in the entire action shot with no developments.

The key feature of the language is that all shots are self contained entities. Starting from a transition to the final composition, including camera and script actions, each prose storyboard sentence is independent of the previous or the next shot. A shot description can always be written and read without requiring knowledge from the previous or next shot in a movie.

Based on the taxonomy of shots proposed by Thomson and Bowen [TB09b], the prose storyboard language allows to describe precisely three main categories of shots :

Some shots consist of a single composition from beginning to end. In many cases, however, the initial composition is followed by a number of developments. They are of two types, continuation or recomposition. A continuation is the PSL description of actions, either actor or camera, that do not lead to a change in composition from the previous one. It can be from an action such as speaking or looking that is important to the screenplay to mention but that does not cause a change in composition. These are described under the event non-terminal. Or the continuation can be due to a follow event in which the subjects start to perform an action in the cue and the camera moves with the subjects and tracks their movements so as to maintain the composition. This camera action is categorized under the camera with non-terminal.

The second type of development is the recomposition where there is, as the name suggests, a change in composition. This change can either be due to an action performed by the subject or by the camera or both but these two movements are not synchronized as in camera with. These camera actions are categorized under the camera to non-terminal. In the case of a change in composition only because of actor movements such as in simple shots (figure 9), the camera "holds" to the next composition.

After the initial composition, there can be any number of developments of either type. To the best of our knowledge, the prose storyboard language is the first description framework that correctly describes developing shots of arbitrary length and complexity.

For each of the three cases we propose a simplified model of a shot which consists of a sequence of compositions, cues and screen events. Cues are actor movement which trigger camera movements. They are an important construct in classical movie-making where camera movements are frequently motivated by the story. Cues are optional, which makes it possible to also describe unmotivated camera movements preceding the action, or descriptive camera movements not related to actor movements. Screen events can be actions of the camera relative to the actors, or actions of the actors relative to the camera. Screen events come in two main categories - those which change the composition (events-to-composition) and those which maintain the composition (events-with-composition).

In our model, we can be in only one of four different states:

In case (1), the shot can be described with a single composition. In case (2), we introduce the special verb hold to indicate that the camera remains static while the actors move, leading to a new composition. In case (3), we use the generic construction CameraWith (pan with, dolly with, crane with) to indicate how the camera moves to maintain the composition. In case (4), we use the generic construction CameraTo to indicate how the cameras moves to change the composition.

As a validation of the proposed language, we have manually annotated extended scenes from existing movies covering many different styles and periods.

We have presented a language for describing the spatial and temporal structure of movies with arbitrarily complex shots. The language can be extended in many ways, e.g. by taking into account lens choices, depth-of-field and lighting, and diegetic sound including speech. Future work will be devoted to the dual problems of automatically generating movies from prose storyboards in machinima environments, and automatically describing shots in existing movies. We are also planning to extend our framework for the case of stereoscopic movies, where image composition needs to be extended to include the depth and disparity of subjects in the composition. It would also be interesting to extend the language even further for the case of panoramic video and immersive virtual reality movies. At this stage, we believe that the proposed language can be useful to extend existing approaches in intelligent cinematography and editing towards more expressive strategies and idioms, and to bridge the gap between real and virtual movie-making.