Opening Drawers and Toolboxes

A Diversity of Approaches in the Salzburg Workshop on Multimodal Methods

Two days full of eye-opening presentations and engaging talk yielded colorful idea of where we are in (linguistic) multimodality research. Methods were seen as relating to: 1) suitable theories to grab concepts from and win categories for annotation; 2) annotated corpora and their quantificational resources; 3) the description of multimodal patterns; 4) visualizations of the results from corpus studies; 5) improvements on annotations and feedbacks into theory.

Helen CAPLE (UNSW, Sydney) showed dimensions of multimodal methods e.g., intra- vs. intermodal perspectives, and presented a nifty little visualization tool for annotation data called Kaleidographic.

Jana PFLAEGING (University of Salzburg) demonstrated how genre marking has developed, diversified and intensified over the last 100 years in National Geographic Magazine and visualized the results beautifully.

Dusan STAMENKOVIC (University of Nis) looked at the development of gaming screens over the last couple of decades paying attention to screen types and their compositional patterns and resources.

Martin KALTENBACHER (University of Salzburg) traced the changes affecting an originally medieval cultural tradition of the Austrian Alps (Krampus) and pointed out how medial transpositions have altered the symbolic attributes of the Krampus figure.

Hartmut STÖCKL (University of Salzburg) introduced a comprehensive annotation scheme to explore a collection of social/non-profit advertisements for typical multimodal patterns based on rhetorical figures and arguments linking text and image.

Assimakis TSERONIS (Örebro University) showed the significance of neatly defining and delineating rhetorical figures and their visual/verbal representation for studies in multimodal rhetoric and argumentation using magazine covers.

Charles FORCEVILLE (Amsterdam University) teased apart visual/filmic manifestation of two general metaphors conceptualizing depression (as a mental affliction) in animated cartoons arguing they are facilitated and based on the medial affordances of animation.

John BATEMAN (Bremen University) argued that semiotic modes and their resources are an empirical matter showing with the help of computational procedures on richly annotated data how page composition has developed in comics since the 1940s, undergoing significant pattern renewal and change.

Markus OPPOLZER (University of Salzburg) suggested blending and narrative construal by responsive readers as theories to explain the decompression and compression of actions/events in largely wordless comics/picture books.

Janina WILDFEUER (Bremen University) reviewed popular approaches to discourse semantics and argued that it is an integral part of any semiotic mode as it provides users with a mode-specific way of making sense of its material, medial and lexico-grammatical resources.

Tuomo HIIPPALA (Helsinki University) presented an annotation scheme used to professionalize and optimize machine-generated annotations of diagrams highlighting rhetorical structure theory to model and classify the various linkages between the diagram’s elements.

Ulrich HEID and Claudia ROSSKOPF (University of Hildesheim) demonstrated how reviewers of art exhibitions create multimodal links between the exhibits and other images/works of art that function to comment on the exhibition, highlighting this type of inter-pictoriality as a means to appropriate art.

Konstanze MARX (University of Greifswald) used multimodal interaction/conversation analysis to describe the structure of online gaming interactions pointing out typical media-(sub-)cultural behavior.

Rebekah WEGENER (University of Salzburg) and Jörg CASSENS (University of Hildesheim) outlined the design principles and ideas underlying an automated lecture-note taking system, using this as an empirical basis to introduce a novel conceptualization of multimodality rethinking the role of context types.

THANKS VERY MUCH TO ALL OF YOU!!!