Topic 8 - Data Gathering, Analysis, & Design Evaluation

What are the six key issues to consider when planning a data gathering session?

1. Setting goals, 2. Identifying participants, 3. Relationship with participants, 4. Ethical considerations, 5. Triangulation, and 6. Pilot studies.

What is triangulation in the context of data gathering?

Investigating a phenomenon from more than one perspective by collecting data from different sources, investigators, theoretical frameworks, or techniques.

What is the purpose of a pilot study in interaction design research?

To run a small trial of the main study to test procedures and ensure the setup works as intended before full deployment.

What is the core difference between unstructured, structured, and semi-structured interviews?

Unstructured is not directed by a script (rich but not replicable); Structured is tightly scripted (replicable but lacks richness); Semi-structured balances both using a guide while exploring depths.

What types of questions should an interaction designer explicitly avoid during user interviews?

Long questions, compound sentences, technical jargon, leading questions that make assumptions, and questions introducing unconscious biases.

What are the five essential chronological phases of running a successful user interview?

1. Introduction, 2. Warm-up (easy questions), 3. Main body (logical order), 4. Cooling-off period (defuse tension), and 5. Closure.

What role do "props" play in enriching the interview process?

Using concrete physical or digital artifacts—such as personas, prototypes, or scenarios—to serve as prompt devices for the interviewee.

What sampling problem frequently occurs during online questionnaire evaluations?

Sampling becomes highly problematic when the true absolute size of the target user population is unknown.

What are Likert scales and where are they most beneficial to employ?

Versatile questionnaire tools used to gather valuable semantic agreement insights for customer feedback, employee engagement, and market research.

What is the primary difference between direct observation in the wild vs. a controlled environment?

Observation in the wild happens in natural user settings (using frameworks or ethnography); observation in controlled settings uses structured methods like the Think-aloud technique.

What are the three simple components of the quick observation framework, and what is its detailed alternative?

Simple: The person (Who?), place (Where?), and thing (What?). Detailed (Robson & McCarten): Space, Actors, Activities, Objects, Acts, Events, Time, Goals, and Feelings.

What is Ethnography as an evaluation philosophy?

An immersive research method where the investigator embeds themselves directly into the culture they are studying to collect comments, incidents, and artifacts.

What is indirect observation in interaction design, and what tools track user activities this way?

Tracking user actions passively without direct presence using user diaries, interaction logs, web analytics, data scraping, and wearable sensors.

What is the difference between quantitative and qualitative data and their respective analyses?

Quantitative is expressed as numbers (numerical size/magnitude analysis); Qualitative is expressed as words/images (theme, pattern, and story analysis).

What are the three levels of coding utilized when performing a Grounded Theory data analysis?

1. Open coding (identifying categories), 2. Axial coding (linking to subcategories), and 3. Selective coding (forming a theoretical scheme).

What are the core focuses of Conversation Analysis vs. Discourse Analysis?

Conversation Analysis examines semantic details of dialogue line-by-line; Discourse Analysis focuses on how words convey implicit, subtle meanings assuming no objective scientific truth.

What software applications explicitly support Computer Assisted Qualitative Data Analysis (CAQDAS)?

Nvivo and Dedoose are major quantitative text-analysis and categorization tools supported by the CAQDAS project.

What are the four baseline dimensions for planning an interaction design evaluation?

Why to evaluate, what to evaluate, where to evaluate, and when to evaluate.

What can be evaluated at different stages of an iterative interaction design lifecycle?

A conceptual model, early or subsequent prototypes, more complete prototypes, and finished products compared against competitors.

What are the three primary types of evaluation environmental settings?

1. Controlled settings involving users, 2. Natural settings involving users, and 3. Any setting that does not directly involve users.

What is a Living Lab and what is a classic early example of one?

A realistic space used to evaluate technology within people's everyday routines over time (e.g., the sensor-embedded Aware Home by Abowd, 2000).

What did the DeepTake automated vehicle study measure during its evaluation session?

Driver takeover behavior via hour-long task simulations, post-task questionnaires, and eye tracking to log exactly where the driver was looking.

What was the purpose and mechanism of the automated "Ethnobot" case study?

An app used at the Royal Highland Show that directed participants to places and asked them to submit prewritten experience responses to questions.

Which specific evaluation methods require controlled settings, natural settings, or no users?

Observing and Asking Users (controlled/natural); Asking Experts (natural/without users); Testing (controlled only); Modeling (without users only).

What is the function of an informed consent form in user studies and evaluations?

It explains why an evaluation is being done, outlines user tasks, protects user rights, and acts as a binding contract.

How does Reliability differ from Validity when interpreting evaluation data findings?

Reliability checks if a method produces identical results on separate occasions; Validity checks if a method measures what it intends to measure.

What is Ecological Validity in user interface evaluation studies?

A measure of whether the physical environment of an evaluation setup distorts or biases the final data results.

What is the difference between Formative Evaluation and Summative Evaluation?

Formative evaluations occur during design to shape and improve prototypes; Summative evaluations occur at the end to assess a finished product.