Visual Information

Information & Interaction Design

Mick McQuaid

University of Texas at Austin

09 Nov 2026

Week TWELVE

Visual Information

Semiology of graphics

pioneering work in data visualization

Semiology of graphics

the name of the textbook for what I’d like to teach you today

Semiology of graphics

how to make graphics systematically

Semiology of graphics

you may think graphics are separate from information

Semiology of graphics

you may think graphics should be constructed by artists

Semiology of graphics

what about a graphics system to express information?

Semiology of graphics

it does not belong to the artist

Semiology of graphics

it does not belong to the artist

Salvador Dali is pictured at right

Semiology of graphics

it belongs to anyone communicating information

Information needs graphics

give me half an hour to convince you of this

First an exercise

please take 5 minutes to sketch a graphic

A table from the Census Bureau

a subset of manufacturing data

Industry in Oregon counties

36 counties: all mfg subsectors, total sales, hours worked, wages

How would you graphically portray some or all?

whole table available at: Canvas > Files > misc > ocounties.pdf

Pretend you’re going to give the table & sketch to an assistant to polish

whole table available at: Canvas > Files > misc > ocounties.pdf

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

While you do that I’ll show some images

I image googled statistics on Oregon counties

This textbook was unknown in English-speaking countries until …

… this very popular book made it okay for quant people to talk about graphics …

The basic problem you just tackled

The basic problem you just tackled

The basic problem you just tackled

You chose retinal variables matched to relationships

Let’s drill down into one retinal variable, color

Imagine some statistic for each county

We can easily perceive bins by value

We can try the color spectrum but what’s wrong?

We see the spectrum as a wheel not as a line

We can do better by ‘cutting’ the wheel in a different place

Another solution uses contours with the original color choice

What do you think of this color choice?

Pretty and without meaning

Exercise: repair a color scale

Form a pair.
Open this Google Doc.
Look again at the spectrum map of France, the one with the rainbow legend. Write one question about the departments it answers quickly and one it fails. (2 min)
Sketch three legends for the same seven bins: one varying value alone from light to dark, one cutting the color wheel somewhere other than red, and one using two hues that diverge from the zero bin. (5 min)
For each legend, write one question it answers faster than the other two. (3 min)
Report your best legend and its question to the class. (4 min)

Back to the exercise

please think about how you would do it now

A system for producing graphics

underlying these examples is a process for making graphics in which we analyze the information to determine visual aspects best suited to each kind of information … let’s look at that process

1. The invariant

begin to analyze the information by identifying the invariant, the thing that will stay the same across all relationships in the graphic, before considering the components, the things that change

1. The invariant

the invariant is the topic of the graphic, for instance sales by county, or labor by county … the invariant gives us the title for the graphic

Limits of the system

step 1 takes no account of persuasion or provocation because the system strives for the single goal of supporting immediate answers to simple questions about the information

2. The components

each component will be represented by a legend on the graph … the components are what varies, such as sales, type of timber, market rank

2. The components

each component has three properties to analyze to determine next steps: order, length, and level … three words given precise definitions in the system

2. The components

order ranges from components considered as raw values to fractions of components governed by some criteria … for instance workers and workers by county

2. The components

length of a component refers to the number of divisions we can identify in it … fewer than four will lead to special constructions while more than a dozen will lead to standard constructions

2. The components

level of a component refers to whether the concept is nominal, ordered, or quantitative

Back to the exercise

please think again about how you would do it now

Visual information example

Thanks to the two students who worked with me on this study of one student’s solution to the Bertin exercise. This study is more detailed than what I expected you to do in class but serves to illustrate the process.

bertin exercise example

The Invariant

The invariant is represented by the title

The invariant should be reflected in the title of the information graphic. If it is not included, or if it can be improved, it is appropriate for the critique to include a suggestion. In this case, the invariant seems to me to be Number of establishments by Size, County, and NAICS. Both the content and the order of this invariant matter, as will become clear in the following paragraphs. This invariant appropriately summarizes the information graphic.

Having critiqued the invariant and suggested a solution, I am in a position to answer the first of the three questions I ask you to consider for every exercise, whether the student has tackled a worthy problem. Certainly I believe that this is the case here because it is of interest to a great many people to summarize economic activity of a state by number of establishments, size of establishments, specific county of establishments, and kind of establishments. (The NAICS, which stands for North American Industry Classification System, is a standard for describing kinds of businesses). This is not a controversial claim because a great deal of policy and economic activity depends on such summaries.

The components

The components are the things that vary in the graphic

The components are represented in the information graphic and one way to identify them is to see what is displayed. The \(x\) and \(y\) axes are frequently used to explore the most important components. Here, the \(y\) axis displays the number of establishments, an integer ranging from 0 to 700. The \(x\) axis displays bars representing combinations of counties and NAICS codes.

I have mentioned that each component should have a legend. The above components don’t really have a complete legend, although Bertin would consider the axis labels to be part of a legend.

There is one legend. That legend depicts two categories of size, so size is the remaining component. We could make different claims about the characteristics of the retinal variables involved in this component and we will do so later.

For now, let us list the components along with their orders, lengths, and levels. Remember that order refers to raw numbers or percentages (usually based on population or area), length refers to the number of objects being represented, and level refers to nominal, ordinal, or quantitative measurements.

Number of establishments is an integer, evidently ranging from 1–700. Its order is raw, its length is either 700, the total number of establishments in a county/NAICS combination or the total number of establishments overall. Its level is quantitative because we can easily say that twenty establishments is twice as many as ten establishments. In other words, we can form a meaningful ratio between measures of the component.

Size has the order raw because it refers to number of employees rather than number of employees as a percentage of available working age population, for instance. Size has length 2 because we can only distinguish between establishments with fewer or more than twenty employees. Size is of ordinal level because we can say that an establishment in the 20 and over category is larger than an establishment in the under 20 category but we can’t say how much larger given this information.

County is of raw order, of length either 7 (represented) or 36 (available in the entire state), and of level nominal because one county is not countier than another. Note that an attribute of county like name is ordinal in an alphabetical sense and an attribute of county like area is quantitative. All this component does, though, is to identify counties.

NAICS is a little trickier. I will call its order raw because of the way we use it here, but each code represents a location in a hierarchy of specialization so another interpretation might be arguable. Its length could be construed as the number of different codes represented in the graphic or in the state or in the entire catalog. Its level is also tricky because of the hierarchical ordering of the codes. I will call it nominal for our purposes because I have seen no evidence that it is meant to be used in another way.

To critique the components alone is simply to say whether they make sense as components of this graphic. More important is to critique their representation by retinal variables and to see if they are well matched to their retinal variables and whether better use could be made of the retinal variables used up by each component.

Retinal Variables

The retinal variables are the things that form an image on our retinas

Bertin describes the retinal variables in a dense, 438 page book, originally published in French in 1967 but widely discussed in other languages and recently printed in English as Bertin (2011). There is a lot to say. Nevertheless, we will consider the retinal variables depicted in the slide show at a very basic level. The retinal variables include size, value, texture, color, orientation, and shape.

Three of the retinal variables here are described on the axes by bars. Bar height is very well matched to an integer ranging from 0 to 700, so I regard the use of the y axis to describe number of firms as successful. There is no other component as well suited to representation by bar height in the data.

County and NAICS are both described on the x axis by the bar names. A tractable number of combinations has been shown. One shortcoming of this graphic is that there are not instructions to the assistant mentioned in the assignment spec. Recall that you were contributing a sketch and instructions to someone polishing the graphic. There is not enough to go on. This, among other issues, has led me to rethink the picture / word ratio constraint on sketchbooks. Instead, I would like to use picture amount and let words appear as needed without penalty.

One question about the components County and NAICS, is whether they should be considered as one component or separately. That will remain an open question for now.

Finally, we come to the Size component. The representation of this component allows a number of interpretations. We could refer to the choice of solid black and diagonal hatching as values, since one can be interpreted as black and the other as gray. We could also regard them as colors gray and black. We could also regard them as having orientation in the case of the hatching versus no orientation in the case of the black. Finally, we could consider them has having texture since we could call the black smooth and the hatching ridged.

One point Bertin makes about using more than one retinal variable to depict a component is that retinal variables may reinforce each other or be in conflict with each other. In the latter case, cognitive dissonance may be produced in the mind of the viewer. From Bertin’s perspective, cognitive dissonance is always bad. Many people who produce information graphics seek to produce cognitive dissonance for reasons involving persuasion. For example, the information graphics produced by opponents of the health care plans of two recent US presidents have been designed explicitly to create cognitive dissonance in the minds of viewers. This usage stands outside Bertin’s system which seeks only to support immediate answers to simple questions about information.

In this case, my reading is that the various retinal variables work together, reinforcing a consistent reading. It is not surprising that diagrams of this style may be seen in many publications. They serve an informative purpose.

Exercise: invariant and components

Form a pair.
Open this Google Doc.
Choose one of the eight Oregon charts from the Bertin slideshow. The doc lists them by number. Write its invariant as a one-line title. (2 min)
List its components. For each, give its order (raw or fraction), its length (number of divisions), and its level (nominal, ordered, or quantitative). (5 min)
Name the retinal variable carrying each component. Pick the worst match and propose a swap. (4 min)
Report your worst match and your swap to the class. (4 min)

Bertin’s Theory

It should not be surprising that information design owes so much to a cartographer. This is one of two fields, the other being social network analysis, with the most influence on building information containers today. The remaining sections of this chapter explore Bertin’s theory of information graphics.

Theories

The word theory is used to mean many things. To obtain a useful definition, it might be helpful to consider what Stephen Hawking says about theory in A Brief History of Time on page 7.

A theory is a good theory if it satisfies two requirements. It must accurately describe a large class of observations on the basis of a model that contains only a few arbitrary elements, and it must make definite predictions about the results of future observations. For example, Aristotle believed Empedocles’s theory that everything was made out of four elements: earth, air, fire, and water. This was simple enough, but did not make any definite predictions. On the other hand, Newton’s theory of gravity was based on an even simpler model, in which bodies attracted each other with a force that was proportional to a quantity called their mass and inversely proportional to the square of the distance between them. Yet it predicts the motions of the sun, the moon, and the planets to a high degree of accuracy.

Efficiency

Bertin defines efficiency as a measure of how quickly the process of reading a graphic can be completed and defines the process in some detail under the influence of (then) contemporary philosophy. Let’s read some information graphics now to get an idea of what this might entail.

It is important to understand that the above definition and review of information graphics is meant as scaffolding on the way to understanding efficiency. The precise definition of efficiency, quoted from Bertin, is If, in order to obtain a correct and complete answer to a given question, all other things being equal, one construction requires a shorter observation time than another construction, we can say that it is more efficient for this question. This is a measurable, precise definition. Bertin characterizes it as implementing Zipf’s notion of mental cost.

Efficiency alone could be abused, so Bertin augments with some additional concepts. How do you think efficiency could be abused? What processes could be described to augment efficiency? (See Semiology of Graphics, p. 166 for an example.)

Categories of information graphics

Bertin describes four categories of graphics: diagrams, networks, maps, and symbols. It is the use of the last three categories, coupled with the detail in Bertin’s theory, that persuade me that here is a good foundation for describing information architecture.

Symbols

Symbols establish a correspondence between a single element on the graphic plane and something external to the graphic, through immediate recognition. Symbols are used extensively in information architecture and Unicode offers a flexible way to represent 113,000 currently defined symbols in type, along with a plan for annual additions with a limit of about 1.1 million symbols using the current international agreements.

Here I am using the word flexible to mean that typographic glyphs are represented by curves that can be completely described by a short series of numbers in a specified order. If you have used Bezier curves in Photoshop, you know how to determine very complex shapes by specifying the location of a few control points. This same principle allows for device independence, arbitrary scaling of size, advanced algorithms for micro-kerning and other processes that depend on the relative locations of glyphs. Thus, typographic symbols are ideal vehicles for multimodal information representation in handheld, wrist worn, or desktop display.

Diagrams

Networks

Bertin describes networks as the correspondence on the graphic plane between the divisions of one component. The underlying component is usually people in a social network but is usually represented as emails, texts, phone calls, links, or anything else that points from one object of a certain kind to another object of that kind.

Bertin’s advice about the representation of networks dates from the 1960s, before the dawn of social network analysis. Nevertheless, he specifies the process of portraying a network that highlights the problem that still vexes practitioners today. First, establish the correspondences and second, identify the representation of the correspondences with the fewest intersections. In Bertin’s book, he displays networks with only a dozen or a few dozen nodes. Most of the network representations of today suffer from the production, in the first step, of correspondences numbering in the billions between divisions (people or websites) numbering in the hundreds of millions.

Plenty of algorithms have been developed to accomplish step two of the above process. Many are inventoried at graph drawing.

Bertin claims that any network may be converted to a diagram by the addition of one or more components. He gives the example of speakers and listeners as two components and demonstrates their portrayal with a bipartite graph and an incidence matrix.

Maps

Maps are exactly like networks except that a geographic order is imposed. The process for creating a map requires the reproduction of the geographic order first, and recording the correspondences second.

Although the network may be easily converted to a diagram, the map introduces a geographic ordering to the graphic plane that is disturbed by the addition of another component. Thus, I suspect that Bertin would call the graphical constructions based on the famous NYC subway maps diagrams rather than maps. In most cases, these constructions only superficially resemble a map in the sense that no geographic ordering occurs. Nevertheless, the notion of a map and the rules of cartography are deeply appealing to people, perhaps because of our hunter-gatherer past.

The lack of flexibility in maps is a key feature. The fact that additional components disturb the geographic ordering renders maps useful for navigation but diminishes their navigational usefulness if they are transformed into diagrams. This characteristic explains both the advantage and limitation of map-like navigational aids in information artifacts. If you map your information architecture as a flow chart, for instance, do you impose a geographic ordering on the 2D plane? If you do, you drastically limit the flexibility of the flow chart to represent abstract concepts but may enhance its use in describing flow.

Do you regard bread-crumb trails as maps or networks? Every single bread-crumb trail I have ever seen on a website follows a horizontal left older right newer path. Every single one I have ever seen in a menu follows a vertical down older up newer path. Each crumb offers only a click to return to it or perhaps a tooltip to provide a more complete name. The bread crumb trail has defied HCI researchers trying to evaluate its effects. Might Bertin’s theory help us to understand what user studies have not?

Simplification

Recall my earlier question about adjuncts to efficiency. The main adjunct in my opinion is an exhaustive depiction of what it means to simplify information using retinal variables at the most detailed level possible.

A Picasso nude exemplifies simplicity

Example: Swim lane diagrams

Swim lane diagrams are used as a project management tool in IT. At a minimum, they portray roles, tasks, and handoffs. The name swim lane arises because they look like a swimming pool divided into lanes with a start and finish. Each role has a lane in the pool. Each task is rectangle inside a lane. Each handoff is an arrow from one lane to another.

Here is an example from processmodeling.info containing additional symbols specific to Unified Modeling Language.

This diagram shows three roles: accounting, hiring manager, and human resources, as well as five tasks: create job requisition, conduct interviews, setup payroll, and give first day briefing. The diagram has a beginning and an end and is read from left to right. A top to bottom version would work just as well but there must always be the sense that the diagram shows some process that has a definite start and stop and a sense of time precedence among the tasks.

The swim lane diagram is one of many project management tools. It’s so successful that you can image google it and find many versions of it. You may be subjected to refinements of it in the workplace, but here let’s just consider what they all have in common.

How can you produce a swim lane diagram?

The basic swim lane diagram should be easily understood by any worker in one of the roles displayed. The diagram may be drawn by asking someone in each role what they do and where their handoffs are. To construct a swim lane diagram, therefore, requires some skill in eliciting information from people about what they do. People find it notoriously difficult to describe what they do unless they are doing it. Interviewing someone about their job usually leaves critical gaps.

When visiting a worker to get help with a swim lane diagram, it’s a good idea to construct a diagram before you visit them. It is easier to correct a wrong diagram than to start from scratch. It also conveys to the person that you have been trying and need their help, rather than that you are trying to slough work off onto them.

The diagram you present to someone should not look polished. If possible, do it in ballpoint pen on cheap notebook paper. The reason to do this is to make it clear that you don’t believe the diagram is finished. If you produce a pretty diagram with fancy software and print it in color, the worker you interview may be more reluctant to change it than if it looks like something you did not intend as a finished product. Decorations are not helpful in eliciting information that would cause the decorations to have to be redone.

There should be a lot of room on the page that you present to a worker and you should hand them a pen or pencil of a different color than what you’ve already drawn so that afterward, you can distinguish between what they’ve added and what you’ve added.

It may be better to visit alone or with another team mate. If you’re interviewing a less experienced worker, they may be more intimidated by multiple visitors. On the other hand, a more experienced worker may have so much information to share that you need one team member to talk to them while another takes notes.

Recording an interview may reduce the candor of the worker being interviewed. If you must record, put the recorder within easy reach of the worker and invite them to turn it off if they want to say something off the record. Be aware that, if they turn it off at some point, they are unlikely to turn it back on for the rest of the interview. I take it as a signal that the recording makes them uncomfortable and I don’t ask to turn it back on later.

Be prepared with extra paper if the interviewee wants to start over from scratch and be sure to have a camera or time to copy in case they prefer to work on a whiteboard.

One skill that’s hard to teach is to know when to interrupt. If you don’t understand something, you should ask about it as soon as possible but, if you disturb the flow of the interviewee’s thinking, you risk never getting the info that would have flowed just after the interruption. Before you interrupt, you should have a good idea of how to proceed after the interruption is finished.

One luxury you may not have is to revisit the same worker after some other interviewee has modified what they’ve said. I have had this experience where two people involved in a single process have conflicting views of their roles. This presents an opportunity for process improvement that is beyond the scope of this discussion but a likely application of this tool in the workplace.

An exemplary swim lane diagram

Hanna Jung, a brilliant designer, shows an impressive example of a swim lane diagram at tracking civic theater volunteers where she also includes some additional symbols related to pain points in the process. Since I worked with Hanna Jung in the past, I should add something you can learn from her. Although she excels in many, many ways, one thing that you can see within a few minutes of watching her work is that she begins solving problems as soon as they present themselves and organizes her time and that of others to maximize impact. I’ve met many students who work hard, but can only think of two other students out of a thousand who can do as much in as little time as Hanna Jung. When others are still getting started, she is close to finishing. And when she’s finished, she reviews her work and that of others to decide whether to polish it or move on to the next problem. She never wastes time overthinking a problem.

I believe that she achieves this success through polishing her problem solving process. She determines how much time is available to solve a problem and divides the time into steps so that she won’t get so bogged down in one step and fail to make it to the end. She writes or draws her initial plan clearly and invites others to join her, thinking about what she knows of their skills and how they could contribute. Then she ruthlessly sticks to the plan, moving on to step two even if she’s not satisfied with step one. This way, there’s still time to go back to the beginning if there has been a false start. If she goes back to the beginning, she replans all the steps with the remaining available time. All her planning seems to take less than a couple of minutes for the length of assignments that are given in school, but that is because she refines the planning from course to course.

Visual Design of Information

Different perspectives drive different views

When we analyze what people see, we bring a perspective, in part based on where we are and what we do. Think about some bases for your perspective on the visible variables in presenting information, such as professional, cultural and biological.

An important perspective

When art critics get together they talk about Form and Structure and Meaning. When artists get together they talk about where you can buy cheap turpentine.

— Pablo Picasso

quoted by Steve McConnell, Code Complete (1993)

Another perspective

Modern philosophers have had little to say about the nature of aesthetic interest; … The concentration has been on the philosophy of art, and in particular on puzzles created by boring impostors like Duchamp: is this signed urinal a work of art? etc. This makes for an exceedingly dull literature, devoted to questions which can be answered in any way while leaving everything important exactly as it is.

—Roger Scruton, Modern Philosophy (1994), p. 589

Essence

Several famous pictures in the twentieth century illustrate the quest to depict the essence of things. The difficulty of doing so and the effort required to see the essence before depicting it can be found in a series of lithographs created by Picasso in a short period of time. There are essays about this series available online and many versions of pictures of them. This particular version is chronologically read first from left to right then top to bottom.

Picasso’s Bull Lithographs

More on Essence

There are many ways to interpret the word essence in the above series and I leave that to the essays mentioned above, that discuss the bull as the essence of masculinity or of Spain itself. Let me just add this proposed Olympics poster, undoubtedly influenced by Picasso in particular and Cubism in general.

Gestalt principles

proximity: proximate images form a group

similarity: similar images form a group

closure: perception completes images that appear to have missing parts

symmetry: perception links unconnected symmetric images

common fate: images perceived to have common fate (follow a path) form a group

continuity: continuous images form a group

prägnanz: images that seem to work well together form a group

Dieter Rams

Good design is innovative.

Good design makes a product useful.

Good design is aesthetic.

Good design helps us to understand a product.

Good design is unobtrusive.

Good design is honest.

Good design is durable.

Good design is consequent to the last detail.

Good design is concerned with the environment.

Good design is as little design as possible.

Edward Tufte

Comparisons–Show comparisons, contrasts, differences

Causality, Mechanism, Structure, Explanation

Multivariate Analysis—Show more than 1 or 2 variables

Integration of Evidence—Integrate words, numbers, images, diagrams

Documentation—label what data you have and measurements, etc.

Content Counts Most of All—depend on quality, relevance, and integrity

Formerly a public policy scholar, Tufte has carved out a career publishing coffee-table quality books, such as Tufte (2001), about quantitative graphics. For years, he promoted a principle called data ink ratio maximization. The principle sought to evaluate graphics by comparing the ink expended on portraying data and the ink expended on other purposes, such as meaningless decoration. This turned out to be a problematic principle. What do you suppose could possibly go wrong with it?

More recently, Tufte has promoted the above list of principles, found in the chapter “Fundamental Principles of Analytic Design” Beautiful Evidence, pp 123–155.

Tufte’s work is characterized in part on a reliance on high-resolution print media. His books are published by a fine art press and he advocates against powerlesspoint-type slide shows, claiming that their information density is too low to match the cognitive capabilities of humans. He suggests instead that speakers produce high-resolution pictures to be printed and provided to audiences to accompany talks instead of what he calls pee pee pfluff.

Tufte’s advocacy began when slide shows were being projected using lcd technology, capable of about \(640\times 480\) pixel resolution and a brightness of a few hundred lumens, displayed on makeshift screens (often no more than white walls) at sizes of perhaps a hundred inches diagonal. As I write in 2017, there are mainstream phones with a 3,820 \(\times\) 2,160 pixel resolution on a daylight-readable screen of about a 5.7 inch diagonal (specs of Sony’s Xperia Z5 Premium). Do you think this rapid advance in display technology affects or should affect Tufte’s advice?

Contemporary Perspectives on Design Principles

Get a GRIP and forget about CRAP

grouping: group like elements together; separate disparate elements

restatement: clarify meaning by repetition

implied lines: align objects to minimize visual distraction

popout: make some objects appear more prominently than others

Exercise: GRIP critique

Form a pair.
Open this Google Doc.
Open one UT page: the registrar’s course schedule, your department’s home page, or your Canvas dashboard. (1 min)
Find one instance each of grouping, restatement, implied lines, and popout. Paste a screenshot or describe each in a sentence. (5 min)
Find one violation of each of the four. (4 min)
Report your worst violation and how you would fix it to the class. (4 min)

Visual Information Elements

Color

Typography

Layout

Animation

We explored these when surveying prototyping in week 7.

END

References

Bertin, Jacques. 2011. Semiology of Graphics: Diagrams, Networks, Maps (English Translation). ESRI Press.

Tufte, Edward R. 2001. The Visual Display of Quantitative Information. Graphics Press.

Colophon

This slideshow was produced using quarto

Fonts are Fira Sans, Fira Sans Light, and Victor Mono Nerd Font

Math is set in Fira Math via MathJax 4