Command and Natural Languages

In the philosophy of language, a natural language (or ordinary language) is a language that is spoken or written in phonemic-alphabetic or phonemically-related iconographic form by humans for general-purpose communication, as distinguished from formal languages (such as computer-programming languages or the “languages” used in the study of formal logic, especially mathematical logic) and from constructed languages.

- Languages are a natural way to communicate
- Communication with systems

• Initially, programming languages
• Scripting languages
• Database query
• Command languages

- With menus and DM, why have languages? For some tasks,

• Natural
• Faster
• For tasks with many options, most effective
• Small footprint (screen, power, size)
• Logistics: Generating help, verification, etc.

Read more

Menu Selection, Forms, and Dialog Boxes

Menus

- Offer cues, users can categorize actions easier (no
syntax recall required)
- Especially effective when users have little training, use the UI intermittently, are unfamiliar with the
terminology or need help structuring their decision making process
- Can also be appealing to expert frequent users
- Goal: Create a sensible, comprehensible, memorable, and convenient organization relevant to users’ tasks
- Some issues/challenges:
Space (often the number of choices is quite large), complexity, organization (e.g., alphabetical vs. frequency of use), categorization, phrasing of items, sequence of items, graphic layout and design, shortcuts, online help, error correction and selection mechanisms

Classification of Menus

- Single Menus
Choice between two or more items, may allow multiple actions, may pop up on the current work area or remain permanently available
- Linear Menu Sequences
Series of interdependent menus that guide users through a series of choices; effective for novice users (guidance, one decision at a time,
e.g., “Wizard,” shopping cart, installation routine)
- Simultaneous Menus
Present multiple active menus at the same time, allow users to enter choices in any order, effective for experienced users, require more display space
- Tree-Structured Menus
Categorize similar items and create tree structure, mutual exclusive groups with distinct identifiers; Can be extremely large without cluttering the display (finding meaningful categories/grouping is crucial)
- Acyclic and Cyclic Menus (Network Structures)
Provide access from a menu item from different menu branches, can be faster, potential of getting lost higher (WWW is a good example)

- Pull Down Menus
Always visible, usually on a top-menu bar
- Toolbars, Iconic Menus, Palettes
Display all the actions, user can apply them to displayed objects
- Pop-Up Menus
Appear on the display as a response to a click, usually context depending (i.e., the content of the menu depends on the location of the cursor/mouse pointer), can be organized in a circle (pie menu)
- Embedded Menus and Hotlinks
Menu is part of the content (e.g., text, maps with embedded links), not explicit, no enumeration, very popular in Hypertext environments, permit items to be viewed in context, eliminate the need for a distracting and screen-space wasting explicit menu, help to keep the user focused on the task and objects of interest

Forms

- Meaningful title: Identify the topic and avoid computer terminology
- Comprehensible instructions (e.g., “Type Address” vs. “You/user should type X”, “Type” vs. “Enter”)
- Logical grouping and sequencing of fields: Arrange related fields together, use sequence of fields that is
consistent with common pattern (e.g, city, state, zip)
- Ensure a visually appealing layout of the form (alignment, margins, … )
- Use consistent terminology and abbreviations
- Align visible space with actual boundaries of data entry fields
- Enable convenient cursor movement (e.g., tab-key)

- Error correction for individual characters and entire fields (easy repair possible)
- Error prevention (e.g., Integer field does not allow characters)
- Error messages for unacceptable values should indicate how to fix the problem
- Provide immediate feedback (e.g., real-time feedback vs. HTML “submit”)
- Clearly mark optional fields (e.g., *)
- Provide explanatory messages for fields (e.g., pop-up textbox)
- Use a completion signal (e.g., submit button), avoid automatic submission, hence allowing for review

Dialog Box

- Purpose: Interrupt user task to request user input
(e.g., selection option, limited data entry, confirmation)

- Consistency is key! Prepare special dialog box guidelines. Examples: – Title, alignment, size, aspect ration, standard margins within boxes, visual orientation (e.g., “cancel” button always on the bottom right)
- Other guidelines:
- Make size as small as reasonable
- Popup box should appear next to the click, not on top of it, never hide the context!
- Enable easy distinction between usual work area and dialog box
- Use multiple boxes or tabbed areas if a lot of content is to display

here’s a good example.. download

Situational awareness

Situation awareness, or SA, is the perception of environmental elements within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future. It is also a field of study concerned with perception of the environment critical to decision-makers in complex, dynamic areas from aviation, air traffic control, power plant operations, military command and control – to more ordinary but nevertheless complex tasks such as driving an automobile or motorcycle.

Situation awareness (SA) involves being aware of what is happening around you to understand how information, events, and your own actions will impact your goals and objectives, both now and in the near future. Lacking SA or having inadequate SA has been identified as one of the primary factors in accidents attributed to human error (e.g., Hartel, Smith, & Prince, 1991; Merket, Bergondy, & Cuevas-Mesa, 1997; Nullmeyer, Stella, Montijo, & Harden, 2005). Thus, SA is especially important in work domains where the information flow can be quite high and poor decisions may lead to serious consequences (e.g., piloting an airplane, functioning as a soldier, or treating critically ill or injured patients).

Having complete, accurate and up-to-the-minute SA is essential where technological and situational complexity on the human decision-maker are a concern. SA has been recognized as a critical, yet often elusive, foundation for successful decision-making across a broad range of complex and dynamic systems, including aviation and air traffic control (e.g., Nullmeyer, Stella, Montijo, & Harden 2005), emergency response and military command and control operations (e.g., Blandford & Wong 2004; Gorman, Cooke, & Winner 2006), and offshore oil and nuclear power plant management (e.g., Flin & O’Connor, 2001).

Enhancing Driver Safety And Situational Awareness

Augmented Reality

Augmented reality (AR) is a field of computer research which deals with the combination of real-world and computer-generated data. At present, most AR research is concerned with the use of live video imagery which is digitally processed and “augmented” by the addition of computer-generated graphics. Advanced research includes the use of motion-tracking data, fiducial marker recognition using machine vision, and the construction of controlled environments containing any number of sensors and actuators.

Mobile Augmented Reality

Mobile Augmented Reality, or “mobile AR”, is a combination of AR and mobile computing technology on mobile phones. Mobile phone’s applications can use both fiduciary marker and markerless video tracking for image registration and insertion of 3d or 2d virtual objects into camera frame. Phone on-line connection in concert with a GPS unit, accelerometer and/or compass could also be used in combination with the camera for image registration.

Some of the earliest applications emerging in this field are projects such as Enkin and Wikitude on the Google Android platform, Tonchidot’s Sekai Camera on the iPhone platform, augmented reality marker-based games on the Nokia S60 platform.

Virtual Reality

Virtual reality (VR) is a technology which allows a user to interact with a computer-simulated environment, be it a real or imagined one. Most current virtual reality environments are primarily visual experiences, displayed either on a computer screen or through special or stereoscopic displays, but some simulations include additional sensory information, such as sound through speakers or headphones. Some advanced, haptic systems now include tactile information, generally known as force feedback, in medical and gaming applications. Users can interact with a virtual environment or a virtual artifact (VA) either through the use of standard input devices such as a keyboard and mouse, or through multimodal devices such as a wired glove, the Polhemus boom arm, and omnidirectional treadmill. The simulated environment can be similar to the real world, for example, simulations for pilot or combat training, or it can differ significantly from reality, as in VR games. In practice, it is currently very difficult to create a high-fidelity virtual reality experience, due largely to technical limitations on processing power, image resolution and communication bandwidth. However, those limitations are expected to eventually be overcome as processor, imaging and data communication technologies become more powerful and cost-effective over time.

Virtual Reality is often used to describe a wide variety of applications, commonly associated with its immersive, highly visual, 3D environments. The development of CAD software, graphics hardware acceleration, head mounted displays, database gloves and miniaturization have helped popularize the notion. In the book The Metaphysics of Virtual Reality, Michael Heim identifies seven different concepts of Virtual Reality: simulation, interaction, artificiality, immersion, telepresence, full-body immersion, and network communication. The definition still has a certain futuristic romanticism attached. People often identify VR with Head Mounted Displays and Data Suits.

And guess what? Virtual Dating?!

Future Direct Manipulation and Virtual World

Direct manipulation and its descendants are thriving. Visual overviews accompanied by user interfaces that permit zooming, filtering, extraction, viewing relations, history keeping, and details-on-demand can provide users with appealing and powerful environments to accomplish their tasks. I believe that most users want comprehensible, predictable and controllable interfaces that give them the feeling of accomplishment and responsibility.

See this Microsoft Surface Computer

And this

A virtual world is a computer-based simulated environment intended for its users to inhabit and interact via avatars. These avatars are usually depicted as textual, two-dimensional, or three-dimensional graphical representations, although other forms are possible (auditory and touch sensations for example). Some, but not all, virtual worlds allow for multiple users.

The computer accesses a computer-simulated world and presents perceptual stimuli to the user, who in turn can manipulate elements of the modeled world and thus experiences telepresence to a certain degree.[ such modeled worlds may appear similar to the real world or instead depict fantasy worlds. The model world may simulate rules based on the real world or some hybrid fantasy world. Example rules are gravity, topography, locomotion, real-time actions, and communication. Communication between users has ranged from text, graphical icons, visual gesture, sound, and rarely, forms using touch and balance senses.

Massively multiplayer online games commonly depict a world very similar to the real world, with real world rules and real-time actions, and communication. Communication is usually textual, with real-time voice communication using VOIP also possible.

Virtual worlds are not limited to games but, depending on the degree of immediacy presented, can encompass computer conferencing and text based chatrooms. Sometimes, emoticons or 'smilies' are available, to show feeling or facial expression. Emoticons often have a keyboard shortcut.

Direct Manipulation cont…

Direct manipulation vs WIMP/GUI interfaces

Direct manipulation is closely associated with interfaces that use windows, icons, menus, and a pointing device (WIMP GUI) as these almost always incorporate direct manipulation to at least some degree. However, direct manipulation should not be confused with these other terms, as it does not imply the use of windows or even graphical output. For example, direct manipulation concepts can be applied to interfaces for blind or vision-impaired users, using a combination of tactile and sonic devices and software.

It is also possible to design a WIMP interface that intentionally does not make use of direct manipulation. For example, most versions of windowing interfaces (e.g. Microsoft Windows) allowed users to reposition a window by dragging it with the mouse, but would not continually redraw the complete window at intermediate positions during the drag. Instead, for example, a rectangular outline of the window might be drawn during the drag, with the complete window contents being redrawn only once the user had released the mouse button. This was necessary on older computers that lacked the memory and/or CPU power to quickly redraw data behind a window that was being dragged.

Direct manipulation in point of sale graphic interfaces

The ViewTouch graphic touchscreen POS (point of sale) GUI developed by Gene Mosher on the Atari ST computer and first installed in restaurants in 1986 is an early example of an application specific GUI that manifests all of the characteristics of direct manipulation. In 1995 the ViewTouch GUI was developed into an X Window System window manager, extending the usefulness of the direct manipulation interface to users equipped with no other equipment than networked displays relying on the X network display protocol. This application is a practical and useful example of the benefit of the direct manipulation interface. Users are freed from the requirement of making use of keyboards, mice and even local computers themselves while they are simultaneously empowered to work in collaborative fashion with each other in world wide virtual workgroups by merely interacting with the framework of graphical symbols on the networked touchscreen.

Direct manipulation in computer graphics

Because of the difficulty of visualizing and manipulating various aspects of computer graphics, including geometry creation and editing, animation, layout of objects and cameras, light placement, and other effects, direct manipulation is an extremely important part of 3D computer graphics. There are standard direct manipulation widgets as well as many unique widgets that are developed either as a better solution to an old problem or as a solution for a new and/or unique problem. The widgets attempt to allow the user to modify an object in any possible direction while also providing easy guides or constraints to allow the user to easily modify an object in the most common directions, while also attempting to be as intuitive as to the function of the widget as possible. The three most ubiquitous transformation widgets are mostly standardized and are:

* the Translation widget, which usually consists of three arrows aligned with the orthogonal axes centered on the object to be translated. Dragging the center of the widget translates the object directly underneath the mouse pointer in the plane parallel to the camera plane, while dragging any of the three arrows translates the object along the appropriate axis. The axes may be aligned with the world-space axes, the object-space axes, or some other space.
* the Rotation widget, which usually consists of three circles aligned with the three orthogonal axes, and one circle aligned with the camera plane. Dragging any of the circles rotates the object around the appropriate axis, while dragging elsewhere will freely rotate the object (virtual trackball rotation).
* the scale widget, which usually consists of three short lines aligned with the orthogonal axes terminating in boxes, and one box in the center of the widget. Dragging any of the three axis-aligned boxes effects a non-uniform scale along solely that axis, while dragging the center box effects a uniform scale on all three axes at once.

Depending on the specific common uses of an object, different kinds of widgets may be used. For example, a light in computer graphics is, like any other object, also defined by a transformation (translation and rotation), but it is sometimes positioned and directed simply with its endpoint positions because it may be more intuitive to define the position of the light source and then define the light’s target, rather than rotating it around the coordinate axes in order to point it at a known position.

Other widgets may be unique for a particular tool, such as edge controls to change the cone of a spotlight, points and handles to define the position and tangent vector for a spline control point, circles of variable size to define a blur filter width or paintbrush size, IK targets for hands and feet, or color wheels and swatches for quickly choosing colors. Complex widgets may even incorporate some techniques from scientific visualization to efficiently present relevant data (such as vector fields for particle effects or false color images to display vertex maps).

Direct manipulation, as well as user interface design in general, for 3D computer graphics tasks, is still an active area of invention and innovation, as the process of generating CG images is generally not considered to be intuitive or easy in comparison to the difficulty of what the user wants to do, especially for complex tasks. The user interface for word processing, for example, is easy to learn for new users and is sufficient for most word processing tasks, so it is a mostly solved and standardized UI, while the user interfaces for 3D computer graphics are usually either difficult to learn and use and not sufficiently powerful for complex tasks, or sufficiently powerful but extremely difficult to learn and use, so direct manipulation and user interfaces will vary wildly from application to application.

Direct Manipulation

The theory of direct manipulation describes interactive systems where the user physically interacts with their operating system. The fundamental feature of such a system is user control. Instead of typing commands and allowing the operating system to act as a strange intermediary, a direct manipulation system allows the user to feel like she is in control, by allowing her to physically interact with files and directories, and presenting a visual representation of the progress and end point.

The term direct manipulation was coined by Shneiderman (1974, 1982, 1983) to refer to systems having the following properties:
1. Continuous representation of the object of interest.
2. Physical actions or labeled button presses instead of complex syntax.
3. Rapid incremental reversible operations whose impact on the object of interest is immediately visible. (Shneiderman, 1982, p. 251)
Direct manipulation interfaces seem remarkably powerful. Shneiderman (1982) has suggested that direct manipulation systems have the following virtues:
1. Novices can learn basic functionality quickly, usually through a demon-
2. Experts can work extremely rapidly to carry out a wide range of tasks,
3. Knowledgeable intermittent users can retain operational concepts.
4. Error messages are rarely needed.
5. Users can see immediately if their actions are furthering their goals, and stration by a more experienced user. even defining new functions and features. if not, they can simply change the direction of their activity.

Early Examples of Direct Manipulation
Hints of direct manipulation programming environments have been around for quite some time. The first major landmark is Sutherland’s Sketchpad, a graphical design program (Sutherland, 1963). Sutherland’s goal was to devise a program that would make it possible for a person and a computer “to converse rapidly through the medium of line drawings.” Sutherland’s work is a land-mark not only because of historical priority but because of the ideas that he helped develop: He was one of the first to discuss the power of graphical interfaces, the conception of a display as “sheets of paper,” the use of pointing devices, the virtues of constraint representations, and the importance of depicting abstractions graphically.

Read More about past Direct Manipulation

Scope, Application, and Limitations

Direct manipulation prescribes a general set of rules that have applications across the field of human computer interaction. Because direct manipulation systems present the user with an easy to use, familiar method of interaction, novices particularly have an easier time with these systems[Margono 87]. For experts, well designed systems can be sufficiently fast, though often using keyboard inputs will allow an expert to work faster [Morgan 91]. Thankfully, dual implementation of these systems is generally not that difficult.

Because the system is familiar and easy to use, novices are also likely to learn more quickly. Since direct manipulation shows the progress of steps, errors occur much less often than they do in command type systems. This fact alone gives users confidence to explore and learn more features of the software more quickly. Because of these benefits, direct manipulation is present in designs from word processing to video games.

Example

The “trash” on the desktop is an excellent example of direct manipulation within computer systems. Users can see both the trash and the files or folders they want to move to the trash. They then physically select the files and drag them to the trash can. While the user is doing this, all of the selected files move as well, illustrating which items the user has selected. When the mouse is placed over the trash, there is a shading indication that it has been selected. The user then has to release the mouse button to move the files to the trash. If there are many files, a dialogue box will show up illustrating the progress of moving files to the trash. Once the action is completed, the files are no longer visible in their original location. The user can change her mind at any time while dragging the files to the trash. Even after placing them there, they can as easily be taken out of the trash and put back in place.

Figure 1: Visual progress of moving a folder to the trash

Applicability to HCI

Direct manipulation has become the method by which most computer users interact with their machines. Since the introduction of the Mac GUI in List and the early Macintosh machines, and slightly later the Windows interface, users have come to expect a mouse with visual and physical interaction with their operating system and software. Direct manipulation is evident in many other areas as well.

Video games are a prime example. With only a small amount of training, users pick up controllers with buttons and knobs that generally map very well to the way the character is moving in physical space.

The extension into virtual reality is an easy one to make. In these situations, users are surrounded by an environment. In environments like the CAVE at Argonne National Laboratories [Cruz 93], they can literally move around in that physical space, and visualize with normal movements. Since objects are oriented in 3-dimensional space, actions easily map to natural movements. Users can physically reach out, and using the provided hardware, “grab” an object and move it somewhere else.

Other virtual reality systems detect the 3-dimensional movements of the user and adjust accordingly. Remote surgery provides a good example. A surgeon may wear a head set to detect his position and adjust a camera view in the remote site. Gloves detect his exact position and movements, allowing a robotic counterpart on the remote end to replicate his exact movements. In this case, the surgeon is using direct manipulation as a method for controlling the remote system.

User Interface Tools Cont…

There are many tools you can use to help you building the Interface.

The commonly use to draw diagram is UML.

The Unified Modeling Language^TM – UML – is OMG’s(Object Management Group) most-used specification, and the way the world models not only application structure, behavior, and architecture, but also business process and data structure.

We only discuss a bit of UML here.

I only can show you a couple of examples

You can learn more

at here or  here

There are also a lot of tools to build the visual interface, such as Adobe Flash, Adobe Director, Microsoft Visual Studio, Visual Fox Pro, Eclipse and many more.

User Interface Tools Cont…

Adaptation to the available screen size is a recurrent problem in graphical user interface design. On the one hand it is not trivial for a designer to specify how a Graphical User Interface (GUI) should adapt; on the other hand it is very hard to construct a predictable adaptation algorithm that never leads to undesired user interface presentations. In this paper, we present a technique to design by example user interfaces that automatically adapt to the available screen size while the designer and eventually the end user still have full control over the adaptation during runtime. This design by example technique could lead to novel tools that lower the threshold for designers as well as developers to design adaptive user interfaces. (more)