Download Direct Manipulation vs. Software Agents: User Control & Intelligent Interfaces in HCI/IUI and more Study notes Computer Science in PDF only on Docsity!
vs
Direct
Manipulation
B E N S H N E I D E R M A N
B
en Shneiderman is a long-time proponent of direct manipulation for user interfaces. Direct manipulation affords the user control and predictability in their interfaces. Pattie Maes believes direct manipulation will have to give way to some form of delegation— namely software agents. Should users give up complete control of their interaction with interfaces? Will users want to risk depending on “agents” that learn their likes and dislikes and act on a user’s behalf? Ben and Pattie debated these issues and more at both IUI 97 (Intelligent User Interfaces conference - January 6–9, 1997) and again at CHI 97 in Atlanta (March 22–27, 1997). Read on and decide for yourself where the future of interfaces should be headed—and why.
d e b a t e
P A T T I E M A E S
s
vs Interface
Agents
Excerpts from debates
at IUI 97 and CHI 97
“
Users have great control over the display and as they select items, the details appear in windows on the sides. FilmFinder video (see Figures 1a-c, CHI videotape or HCIL 1995 Video tape reports): This display shows thousands of films arranged on the x-axis from 1920 to 1993 and on the y-axis from low to high populari- ty. We can use the length slider to trim the set of films by the number of minutes in the film so we do not have to see films that are too
long, and then we can use the category button to show only drama or only action films. We can zoom in on more recent pictures and take only the more popular ones. And when there are fewer than 25, the titles will appear auto- matically. When we select one of them, we get a description of the film and information about the actress and actor. We can also hunt
d e b a t e
Figure 1(a): FilmFinder shows 1500 films in a starfield display where the location of each point is determined by the year of the film (x- axis) and its popularity in video store rentals (y-axis). The color encodes the film type (Ahlberg & Shneider- man, 1994). ftp://www.cs.umd.edu/pro jects/hcil/Screen- dumps/Film/film- alldots.gif
Figure 1(b): FilmFinder after zooming in on recent popular films. When fewer than 25 films remain, the titles appear automatically. ftp://www.cs.umd.edu/pro jects/hcil/Screen- dumps/Film/titles.gif
for films organized by actors. In this case, you might be interested in Sean Connery, and his films will appear on the screen.
Ben: Okay. I think you get an idea that the controls are visually apparent as you drag them. The updates occur in 100 milliseconds and users get a clear understanding of what the situation is. This work goes back to 1993, and the 1994 CHI conference has a couple of papers describing it [1, 2]. A general purpose version of that software was used for the Department of Juvenile Justice project, which you will hear about shortly. Here is a FilmFinder done in the UNIX version of the product called SpotFire (Figure 2). Chris Ahlberg has made a commercial product out of this and you can download the demo version off of the Web (http:// www.ivee.com).
It would be hard to see how to program any kind of agent tool to anticipate all of the possibilities that your eye would pick up. We show the age of the youthful offenders on the x-axis. There are 4,700 of them, from 10 to 19 years old. The number of days until a deci- sion was made on their treatment plan is shown on the y-axis. The rules of this organi- zation say that decisions must be made with- in 25 days, but you can see a lot more than they anticipated are well above the 25-day limit. Interesting things pop up whenever you try a visualization. I hope you will spot these yel- low lines—those were a surprise. We thought there was a bug in the program, but it turns out that if you start clicking on them to get the details-on-demand, you’ll find out they all occur in a Hartford County. They were all brought in on a certain day. They all have the
Figure 1(c): FilmFinder after selecting a single film. The info card pops up with details-on-demand. ftp://www.cs.umd.edu/projects/hcil/Screen-dumps/Film/film-sean.gif
vs
direct manipulation
interface agents
case and was requested by Brown. A click on Brown’s name gives the contact information. Here are the reviews which were written. A click on the code brings the text of the review. In the same way, I can get more details about each case and placement by clicking on the labels. Those screens are all for the old system, showing that LifeLines can be used as a front end and acts as a large menu for all of the screens. The top buttons can access general information, but any critical information will appear on the overview screen. For example, here’s a mention of suicide risk. Seeing the overview gives the user a better sense of how much information is available and what type of information it is. Of course, this implies that all of the information can be presented in one screen.
Ben: We think that LifeLines can also be applied for medical records, and we are now applying it in a project with IBM. We give an overview of patient histories that contains the consultations, conditions, documents, hospi- talizations, and medications. Users have great control over the display and as they select items, the details appear in windows on the sides. We think this strategy has great power in providing convenient access to large and complex databases in a way that gives the users an overview of what is happening and an appreciation of where the details fit into the overall context. The visual presentation gives users enormous bandwidth and there are potentially thousands of selectable items on the screen at once offering rapid access to the item that you are seeking. The third and final example, is a visual database of the human body called the Visible Human. Our role was to develop a browsing user interface to the 15 gigabytes of images at the National Library of Medicine (3; CHI videotape and HCIL 1996 Video Reports; free software available for Sun workstations from http://www.nlm.nih.gov and select the Visible Human links till you find our Visible Human Explorer ). This direct manipulation interface presents a thumbnail image of a coronal cross section of the body reconstructed directly from the axial cryosections (Figure 4). This coronal
image acts as an overview, giving a visual rep- resentation of the entire body. The axial cryosections are a local view showing a thumbnail corresponding to the slider posi- tion on the overview. We can explore the body by simply dragging the slider. It updates in real time, giving an experience of flying through the human body. Here we see the brain, the shoulders, the torso, the abdomen, the thighs, the knees, and all of the way down to the toes. We press the retrieve button and the corresponding full-size image is retrieved over the network from the NLM archive in a couple of minutes. We provide several useful alternative overviews and also the ability to generate any coronal section overview, for example, near the back of the body or near the front.
Ben: Other labs are working on related ideas of information visualization. From the Pacific Northwest National Labs, this textual data- base that has been presented in a two-dimen- sional mountains-and-clusters visualization to give users an idea of the volume and interrela- tionship of items. Steve Eick at AT&T Labs has these wonderful visual overviews of large textual documents. Here, the characters in a children’s book are color coded so that you can see the progress of the story as it moves on to different characters. Departmental e-mail networks and richer information, such as 3D network representa- tions, are part of the things he likes to show with a variety of user controls to filter the traf- fic and reveal patterns of usage that might be difficult to see with other data presentation strategies. The closing slide says that the overview is the most important. It gives users a sense of context, of what to look at—the big picture. Then they zoom in on what they want, filter out what they don’t want, and finally go for details-on-demand. My claim is that this gives users the feeling of being in control and there- fore they can be responsible for the decisions they make. Thank you.
Jim: Okay, thanks very much, Ben. That was perfect timing. We now hand it over to Pattie.
Ben: During the CHI97 debate, Pattie made a point about my use of autofocus cam- eras, suggesting that they were some sort of agent. As we were speaking Alan Wexelblat was taking pic- tures of the events using my camera. When I went through the color slides I found that most of the pictures he took were out of focus! As with most autofocus cameras, there is a narrow area in the center that is used for focusing and this must be placed on the intended subjects of the photo. Unfortunately Alan didn’t know this and simply pointed at the stage area but the focus point was on the background poster. Almost all the photos were unusable.
We can all find this amus- ing and leave it at that, but I think there is a seri- ous point which is that agents don’t always do what we expect them to do, and it takes some knowledge to make effective use of agents (or auto-focus cameras). If we were to assess responsi- bility — I would take part of it in failing to give ade- quate instruction to Alan, he might take part because he was the direct user, and maybe the manufacturer has another part for a poor design which fails to provide appropriate feedback.
Pattie: I’m not going to bribe the moderator with tea or T-shirts or anything. I hope that the work will speak for itself. The word “agent” is used in a lot of different ways. I want to start this presentation by explaining what I mean by the word “agent,” and in a particular, “software agent.” Basically, software agents are a new approach to user software, a new way of think- ing about software that end-users have to use. In particular, the way in which agents differ from the software that we use today is that a software agent is personalized. A software agent knows the individual user’s habits, preferences, and interests. Second, a software agent is proactive. It can take initia-
tive because it knows what your interests are. It can, for example, tell you about something that you may want to know about based on the fact that you have particular interests. Current software, again, is not at all proactive. It does- n’t take any initiative. All of the initiative has to come from the user. A third difference with current software is that software agents are more long-lived. They keep running, and they can run autonomously while the user goes about and does other things. Finally, software agents are adaptive in that they track the user’s interests as they change over time. So, you can ask, well, why do you call it an agent? Why call it an agent given that the term
d e b a t e
Figure 4: Visible Human Explorer user interface, showing a reconstructed coronal section overview (on the left) and an axial preview image of the upper abdominal region (on the upper right). Drag- ging the sliders animates the cross-sections through the body (North et al., 1996). ftp://www.cs.umd.edu/projects/hcil/Research/1995/visible-human.html
vs
direct manipulation
interface agents
“
We need to be able to delegate to what could be thought of as “extra eyes or extra ears” that are on the lookout for things that you may be interested in.
the ones we built in our lab. Letizia, built by Henry Lieberman, who is here at the confer- ence, is an agent which continuously watches you as you browse the Web, analyzing and memorizing all of the pages that you visit. It extracts from those pages the common key- words. Whenever you are using your Web brows- er, Letizia always looks ahead and checks whether within a certain depth of the current page, there happen to be any pages that you may be interested in. So, for example, if I am interested in scuba diving, my agent may have picked it up because I look at a lot of pages about scuba diving. If I go to a particular entertainment site, it may look ahead and say, hey did you realize that if you follow that link that there are some pages about scuba diving in the Florida area? The Remembrance Agent is another agent that continuously tracks the behavior of the user. It helps you remember certain things. It helps you remember who
sent e-mail or whether you already replied to a certain e-mail message. It may proactively remind you of information related to the information you are currently looking at. It works in EMACS. When I am, for example, looking at an e-mail message from a particular person, it proactively reminds me of the previ- ous e-mail messages from that same person, which is very useful because I may have for- gotten to reply to one of them. Firefly, some of you may have tried that agent, is basically a personal filterer for enter- tainment, not unlike the movie application that Ben talked about, except that this agent will again keep track of your interests, your preferences, and proactively tell you about new movies that you may be interested in which you even forgot to ask about in the first place. Yenta is another agent that we built which tracks what the user’s interests are by looking at your e-mail and files and extracting key- words. It talks to other Yenta agents belonging
d e b a t e
to other users, and if it notices that another user shares some of your interests, especially if those interests are very rare, then it introduces you to that other user. It may say “hey did you realize that at this IUI conference there is another person who is interested in going scuba diving in Florida” so that maybe then we can decide to go scuba diving together. Again, it’s suggesting something that you
wouldn’t have thought of yourself. Kasbah is another set of agents that buy and sell on behalf of users. We are currently setting up this experiment MIT-wide, meaning for 15,000 people. We have already done tests with 200 people. It’s basically a marketplace where you can create an agent who will buy or sell a second hand book or a second hand music CD for you. You just tell the agent, “I want to sell The Joshua Tree by U2. I want to ask $12 for it at first. You are allowed to go as low as $9. You have two months to sell this CD. You should be really tough and only change the price all the way at the end, near when the 2 months are over.” That agent will represent you in that marketplace, negotiating on your behalf with other people or other agents who may be interested in buying that CD from you. Again, it is sort of acting on your behalf. You don’t have to waste any time trying to make 10 bucks, but the agent will do this for you. I think it’s important to address some com- mon misconceptions about agents: First of all, sorry to say so, but agents are not an alterna-
vs
direct manipulation
interface agents
“
Opponents of agents typically argue that well- designed visual- ization interfaces are better. Like I said before, you still need a well- designed inter- face when incorporating agents in an application. However, some tasks I may just not want to do myself even if the interface was perfect. If my car had a perfect interface for fixing the engine, I still wouldn’t fix it. I just don’t want to bother with fixing cars. I want someone else to do it.
A second criticism is that agents make the user dumb. That’s actually more one of Jaron’s objections rather than Ben’s, I think. To some extent it’s true. If I don’t fix my car then I’m not going to learn about fixing cars. However, this does not constitute a problem. As long as there’s always an agent available or I can call one by a motor association like AAA, then that’s fine. It’s too bad that I will never learn about cars, but I want to learn about other things instead. A third criticism expressed is that using agents implies giving up all control. That’s incor- rect. I think you do give up some control when you deal with an agent. I tell the car mechanic to fix my car or to fix this or that part of the car. I don’t know how exactly he or she is going to do that. I don’t mind giving up some control, actu- ally, and giving up control over the details as long as the job is done in a more-or-less satisfactory way, and it saves me a lot of time. Okay, just very briefly, I want to say that I think where the true challenge lies is in designing the right user-agent interface. In particular, we need to take care of these two issues: understanding and control. Under- standing means that the agent-user collabora- tion can only be successful if the user can understand and trust the agent, and control means that users must be able to turn over control of tasks to agents but users must never feel out of control. I believe that this is a won- derful interface-design challenge, and we have come up with a lot of solutions to actually make sure that the agent’s user interface has these two properties that the user feels in con- trol or has control when he or she wants it, as well as that the user understands what the agent does and what its limitations are. Let me save that for later, maybe. Thanks.
Jim: Thanks very much. Okay, Ben’s going to go up to 5 minutes to say whatever he likes.
Ben: How interesting. We are debating, but part of me is drawn to the idea of celebrating Pattie Maes and encouraging you to follow her example. I want to draw the audience’s attention to her transformation during the months we’ve had these discussions. As I go back to Pattie Maes’s work and I read her ear- lier papers and her Web sites, she promotes
autonomous agents and presents an anthropo- morphic vision. Even in the current proceed- ings her article is titled “Intelligent Software,” so I was delighted with her opening remarks that rejected intelligent and anthropomorphic designs. The old Pattie Maes wrote “agents will appear as living entities on the screen, conveying their current state of behavior with animated facial expression or body language rather than windows text, graphics, and fig- ures.” So we’ve got two Pattie Maes. I will choose the newer one that demonstrates movement in my direction including her last slide which might have been written by me: “User understanding is central, and user con- trol is vital for people to be successful.” In fact, I have other ways of celebrating Pattie Maes. I encourage you to look at her Firefly Web site, which is an interesting appli- cation. Collaborative filtering, I think, will become an important approach for many domains. But as a user, I can’t find the agents on the Firefly Web site. In fact, as I searched to find the agents, all I came up with was that the company had previously been called Agents, Inc. and is now called Firefly. If you read the Firefly Web site, you will not find the word “agents” in the description of this sys- tem. In fact, the interface is a quite lovely, direct manipulation environment, allowing users to make choices by clicking in a very direct manipulation way. So, I think we’ve made progress in clarify- ing the issues in the past year of our ongoing discussions. For example I think we can sepa- rate out the issue of natural language interac- tion, which as far as I can see, has not been a success. The systems that were offered com- mercially even a few years ago, like Q&A from Symantec or Intellect from AI Corporation to do database query, and Lotus HAL for spread- sheets, are gone, and direct manipulation is the surviving technology. A second issue is anthropomorphic inter- faces such as chatty bank tellers and the Postal Buddy or the proposed Knowledge Navigator of Apple’s 1988 video. Microsoft’s playful attempt at a social interface in BOB is also a failed product. As far as I can see the anthro- pomorphic or social interface is not to be the future of computing.
vs
direct manipulation
interface agents
A third issue of adaptive interfaces is quite interesting. I would concede half a point and say that we now see two levels: the user inter- face level, which users want to be predictable and controllable, as Pattie has stated, and the level below the table, where there may be some interesting algorithms such as collabora- tive filtering. If those can be adaptive there may be benefits in the way that Pattie describes. This is related to other adaptations such as when I save a file to disk, I see it saved, and it is retrievable by me. Under the table, there’s a great deal of adaptation dealing with space allocation, disk fragmentation, and compression strategies, but from the user’s point of view, there’s no adaptation. It’s quite predictable. The same goes for engines in
d e b a t e
of proactive software is that the user does not necessarily always want to have all of that con- trol when searching for a movie. I believe that users sometimes want to be couch-potatoes and wait for an agent to suggest a movie to them to look at, rather than using 4,000 slid- ers, or however many it is, to come up with a movie that they may want to see.
Question: Okay, I have a remark to Ben, which then transitions into a question for both speakers. Ben, I was a little bit irritated, I think Pattie was too, with your lovely presentation on information visualization, which seems to be entirely beside the point. As Pattie said, we will take the best direct manipulation and visualization as we can possibly get. It seems to me the contribution to this discussion would be some negative examples of where agentlike things are bad. At some level, I take the thrust of your position, Ben, as being reactionary, to put it in simpler words, sort of fear-driven. I would like to test my theory by ask- ing the following question, which is, we have now a new medium on the interface of playing which is speech. Speech is now practical. IBM makes speech systems that are being used and so on. My question actually to both of you is to see what your reactions are to this new technology. I predict that if you are going to have speech with a computer, first of all, research at Stanford recently has shown that once you have a comput- er talking you cannot prevent people from anthropomorphizing the computer. I do not see how you are going to have a coherent speech interface without using human communication principles. So I predict that you will say just don’t do it. I also want to hear Pattie‘s comments about speech technology.
Ben: Do we have another half an hour here? I thought I said very positive things about Fire- fly and its agent and adaptation, and I cer- tainly like to see automaticity built into interfaces that amplify the user’s capabilities. I have trouble with the words like “agents,” and “expert,” and “smart,” and “intelligent” because, they mislead the designer, and designers wind-up leaving out important things. In fact, I love Pattie’s slide up here. If the agent-oriented community would adopt
d e b a t e
Ben Shneiderman is a Professor in the Department of Computer Science, Head of the Human- Computer Interac- tion Laboratory, and Member of the Institute for Systems Research, all at the University of Maryland at College Park. He has taught previously at the State University of New York and at Indiana University. He regularly teaches popu- lar short courses and organizes an annual satellite television presentation on User Interface Strategies seen by thousands of professionals since 1987. The third edition of his book Designing the User Inter- face:tion (1987) has recently been published by Addi- son-Wesley Longman. Ben is the author of Software Psychology: Human Factors in Computer and Information Systems (1980) Addison-Wesley Longman, Reading, MA and his 1989 book, co-authored with Greg Kearsley, Hypertext Hands-On!, contains a hypertext version on two disks. He is the originator of the Hyperties hypermedia sys- tem, now produced by Cognetics Corp., Princeton Junction, NJ. In addition he has co-authored two text- books, edited three technical books, published more than 190 technical papers and book chapters. His 1993 edited book Sparks of Innovation in Human-Computer Interaction collects 25 papers from the past 10 years of research at the University of Maryland. Ben has been on the Editorial Advisory Boards of nine journals including the ACM Transactions on Com- puter- Human Interaction and the ACM interactions. He edits the Ablex Publishing Co. book series on “Human-Computer Interaction.” He has consulted and lectured for many organizations including Apple, AT&T, Citicorp, GE, Honeywell, IBM, Intel, Library of Congress, NASA, NCR, and university research groups.
these principles it would go a long way in making me sympathetic. For example Pattie writes “Make the user model available to the user.” I don’t see that being done in most of the work about agents. Explanations should be available and methods of operation should be understandable to the user. So much of the work in agentry goes against the principles. I like the new Pattie. I am ready to be partners and collaborate with the new Pattie. Now, to focus on speech. We have heard for 25 years the great hope and dream that speech is going to solve our user interface problems. Dreamers prophecy that the Star Trek scenario is going to take over and we will talk to our computers. I do not believe that speech will be a generally usable tool. It has important niches: opportunities for disabled users, for certain hands-busy, eyes-busy, and mobility-required applications. In preparing the third edition of my book I worked hard to find speech applications that do recognition effectively. I am quite happy with speech store-and-forward applications by telephone, but the recognition paradigm is not being widely accepted, even for minor tasks such as voice dialing. Speech output, except by tele- phone, is also a problem because speech is very slow and disruptive of cognitive process- ing. I think what annoys me the most about the devotees of speech, is their failure to take in the scientific evidence that speaking com- mands is cognitively more demanding than pointing. Speech uses your short-term memo- ry and working memory. By contrast, hand- eye coordination can be conducted in parallel with problem solving by another part of your brain and therefore does not degrade your per- formance as much as speaking.
Question: You can do both at the same time.
Ben: Yes, you can do hand-eye tasks in paral- lel with problem solving, more easily that you can speak while problem solving. This fact is not a barrier to use of speech, but it is a hur- dle that designers of speech systems must rec- ognize if they are to find ways to overcome it. Pattie: I must admit, I actually agree with a lot of what Ben says. I haven’t used speech at all
in my research, the main reason being com- pletely personal—that these systems often don’t understand my accent, but apart from that, I do agree that it is not a very high band- width kind of connection. There is also a lot of ambiguity. So I personally would like to see speech being used just for situations where the hands are not available, like in your car or as an addi- tional channel, actually, for example, giving an agent some additional advice while you are also pointing at something. For example, Henry Lieberman, sitting here in the audi- ence, did some interesting work where he taught an agent a particular procedure, and while he was performing actions with the mouse, he would give speech inputs to tell the agent what it had to pay attention to. For example, pay attention to this corner here that I am dragging or to this side of the rectangle. So in that situation it is very useful because your hands are already doing something else, and you need that additional channel to con- vey some more information in parallel.
Question: This question is for both of you. Both of you seem to be concerned about protecting the user’s control of the environment, but the one things studies have shown time and time again is that users are very good at making mistakes. So how do your positions relate to time-critical deci- sion-support environments, such as medical sys- tems or cockpit systems?
Pattie: I have actually been focusing on a completely different kind of application, a type of application that is not as critical. For example, if your World Wide Web agent gives you a wrong Web page to look at—it assumes that you are interested in a Web page and you are not—that is not at all critical. It is not a big deal. I have been focusing on that kind of situation and those kinds of problems, the ones where if there is an error it is not very costly. I’ve been doing that because I believe that it will be very hard to make agents that always come up with the right answer, always do the right thing. I believe that there is a very large set of these kind of applications where things don’t have to be completely precise or
“
User interfaces should be predictable, so that users trust them. User interfaces should be thor- oughly tested, and users should be thoroughly trained for all emergencies. In emergency situations, people cannot solve problems. They can only do what’s rehearsed and predictable.
matured in interesting ways from where we were a year ago. I think we are all at the edge of looking at new interfaces, and so, as we push that envelope back, we are getting better understanding of the territory, of the strengths and weakness of direct manipulation, of the strengths and weaknesses of agents, and where they are appropriate. I am pleased by the progress in the discussion.
Pattie: I think we both have changed. Would you agree to that or not?
Jim: I think before they start kissing let’s move on. Last question.
Question: This starts out at least as a clarifica- tion question for Dr. Shneiderman, but it may go places from there. To what extent does direct manipulation, in your definition and in your view, admit autonomous system behavior? Because it seems to me that as soon as you admit anything unexpected, uncontrolled, potentially anthropomorphizable out of the system that you are interacting with, you have opened the door, you have taken a step down the slippery slope toward agentness.
Ben: Yeah. I am in favor of increased automa- tion that amplifies the productivity of users and gives them increased capabilities in carry- ing out their tasks, while preserving their sense of control and their responsibility, responsibil- ity, responsibility. I am sort of not answering your question because I don’t want to work in the language you are dealing with. We should be thinking about productivity improvement tools for users, whether they are graphical macros, dynamic queries, starfield displays or other things.
Question: Then what is it about agents that you dislike?
Ben: Can I go to my closing slide? I want to reassert the importance of scientific evalua- tions. We must get past the argumentation about my system being more friendly than yours or more natural or intuitive, and talk
about user performance. We can deal with sat- isfaction also, but please focus on user perfor- mance and realistic tasks. Please, please, please do your studies—whether they are controlled scientific experiments, usability studies, or simply observations, and get past the wishful thinking and be a scientist and report on real users doing real tasks with these systems. That is my number one take-away message. I am here to promote direct manipulation with comprehensible, predictable, and con- trollable actions. Direct manipulation designs promote rapid learning. It supports rapid per- formance and low error rates while supporting exploratory usage in positive ways. Direct manipulation is a youthful concept which is still emerging in wonderful ways. Our current work leans to information visual- ization with dynamic queries, but there are people doing fascinating things with enriched control panels style sheets, and end-user pro- gramming. Graphical macros would be my favorite project to advance the design of gen- eral computing tools. It is embarrassing that after 15 years of graphic user interface being widely available, we have no graphical macros tools. What is going on? This is the greatest opportunity for visual programming. Third—and I am answering your question here—I think the intelligent agent notion limits the imagination of the designer, and it avoids dealing with interface issues. That’s my view of the agent literature—there is insuffi- cient attention to the interface. Maybe the way agents will mature is as Pattie is suggest- ing; that the agents take care of the processes below the table, and there is a nice direct manipulation interface that the user sees. A leading AI researcher commented to me that the 30 years of planning work in AI is essentially down the tubes because of lack of attention to the user interface. The designers deliver a system and the first thing that the users say is, “This is great but what we really want to do is change these parameters.” The designers say, “Well, you know, we didn’t put that in the interface.” They just haven’t thought adequately about the interface, nor done testing early enough. I believe that this language of “intelligent, autonomous agents” undermines human
“
I am here to promote direct manipulation with compre- hensible, pre- dictable, and controllable actions. Direct manipulation designs promote rapid learning. It supports rapid performance and low error rates while supporting exploratory usage in positive ways.
vs
direct manipulation
interface agents
“
responsibility. I can show you numerous arti- cles in the popular press which suggest the computer is the active and responsible party. We need to clarify that either programmers or operators are the cause of computer failures. Agent promoters might shift some attention to showing users what is happening so that they can monitor and supervise the perfor- mance of agents. I was disturbed that in the Autonomous Agents conference that Pattie is participating in, the organizers refused to include the topics of supervision of agents and user interfaces for programming agents. By contrast, I like Pattie’s summary slide—I think her list is quite wonderful. My closing comment is that I think there are exciting opportunities in these visual inter- faces that give users greater control and there- fore greater responsibility in the operation of computers.
Jim: Thanks. Okay, Pattie?
Pattie: I want to conclude by saying that I believe that there are real limits to what we can do with visualization and direct manipu- lation because our computer environments are becoming more and more complex. We can- not just add more and more sliders and but- tons. Also, there are limitations because the users are not computer-trained. So, I believe that we will have to, to some extent, delegate certain tasks or certain parts of tasks to agents that can act on our behalf or that can at least make suggestions to us. However, this is completely a complemen- tary technique to well-designed interfaces— visualization, and direct manipulation—not a replacement. Users still need to be able to bypass the agent if they want to do that. Also I should say that what we have learned the hard way really is that we have to, when designing an agent, pay attention to user- interface issues, such as understanding and control. These are really very, very important if you want to build agents that really work and that users can trust. The user has to be able to understand what the agent does. The user has to be able to control things if they desire or to the extent that they want to control things. I agree with Ben that the agent
field definitely has grown a lot in the past 10 years or so. In fact, one of the ways I think in which a lot of this agent work distinguishes itself from traditional AI is that agent research focuses on building complete systems, systems that are tested, systems that really have to work, and those same principles and method- ologies can be seen in all of the agents work, whether it be robots, synthetic characters, or software agents. The field is maturing and paying more attention to building things that really work and paying attention to important UI issues. As to people taking responsibility for their agents, I think they indeed should. It is soft- ware that is running on your behalf and that you have delegated certain tasks to. So, per- sonally, I don’t see why that problem is specif- ic to agents as opposed to software in general. Thanks.
Jim: Okay, I would like to thank the Shnei- derman–Maes team for coming here today and talking to us, and thank you all for par- ticipating.
References [1] Ahlberg, C. and Shneiderman, B., Visual Informa- tion Seeking: Tight coupling of dynamic query filters with starfield displays, Proceedings Of ACM CHI Conference (April 1994), 313-317 + color plates. [2] Ahlberg, C. and Shneiderman, B., AlphaSlider: A compact and rapid selector, Proceedings of ACM CHI Conference , (April 1994), 365-371. [3] North, C., Shneiderman, B., and Plaisant, C., User controlled overviews of an image library: A case study of the Visible Human, Proceedings 1st ACM Internation- al Conference on Digital Libraries, (March 1996), 74-82. [4] Plaisant, C., Rose, A., Milash, B., Widoff, S., and Shneiderman, B., LifeLines: Visualizing personal histo- ries, Proceedings of ACM CHI96 Conference (April 1996), 221-227, 518.
PERMISSION TO COPY WITHOUT FEE, ALL OR PART OF THIS MATERIAL IS GRANT- ED PROVIDED THAT THE COPIES ARE NOT MADE OR DISTRIBUTED FOR DIRECT COMMERCIAL ADVANTAGE, THE ACM COPYRIGHT NOTICE AND THE TITLE OF THE PUBLICATION AND ITS DATE APPEAR, AND NOTICE IS GIVEN THAT COPYING IS BY PERMISSION OF THE A SSOCIATION FOR COMPUTING MACHINERY. T O COPY OTHERWISE, OR PUBLISH, REQUIRES A FEE AND /OR SPECIFIC PERMISSION. ©ACM 1072-5520/97/1100 $3.
d e b a t e
I believe that there are real limits to what we can do with visualization and direct manipula- tion because our computer envi- ronments are becoming more and more com- plex. We cannot just add more and more sliders and buttons. Also, there are limitations because the users are not computer- trained. So, I believe that we will have to, to some extent, delegate certain tasks or certain parts of tasks to agents that can act on our behalf or that can at least make sugges- tions to us.
