It is more than just a word to add force to a statement or opinion.
Research requires evidence that meets a standard of credibility.
Three definitions of research:
Careful or diligent search.
Collecting information about a particular subject.
Investigation or experimentation aimed at the discovery and interpretation of facts and revision of accepted theories or laws in light of new facts.
Key elements of research:
Experimentation: Conducting experiments is a central activity in HCI research. An experiment is sometimes called a user study.
Facts: Facts are the building blocks of evidence.
Theories and Laws:
Theory: In the sense of Darwin’s theory of evolution or Einstein’s theory of relativity, the term theory is synonymous with hypothesis.
When confirmed through research, a theory becomes a scientifically accepted body of principles that explain phenomena.
Law: A law is more specific, more constraining, more formal, more binding. A law is a relationship or phenomenon that is “invariable under given conditions.”
Fitts’ Law: Fitts’ law refers to a body of work, originally in human motor behavior (Fitts, 1954), but now widely used in HCI. It includes equations and such for predicting the time to do point-select tasks.
Research involves discovery, interpretation, and revision.
Characteristics of Research
Research Must Be Published
Publication is the final and essential step in research.
Researchers must publish to avoid disappointment when applying for research funds or tenure-track professorships.
Publication allows the knowledge gained through research to extend, refine, or revise the existing body of knowledge in the field.
Publication requires a high standard of scrutiny in archived peer-reviewed journals or conference proceedings.
Research results are reviewed for their integrity, relevance, and contribution by peers.
Patenting an invention is a form of publication, meeting the must-publish criterion for research.
Citations, References, Impact
Citations connect research papers to other research papers.
Citations support intellectual honesty by acknowledging previous work.
Citations back up assertions and compare current results with earlier research.
The number of citations to a research paper measures the paper’s impact.
The H-index is a measure of a researcher’s publication record, quantifying research productivity and overall impact.
H−index=n has n publications each with n or more citations.
Research Must Be Reproducible
Research that cannot be replicated is useless.
Reproducibility is crucial, which is why standardized methodologies are important.
A consistent methodology ensures sufficient detail for replication.
Reviewers often critique work on replicability.
A highly cited paper is Lowry et al.’s, 1951 paper “Protein Measurement With the Folin Phenol Reagent,” which describes a method for measuring proteins in fluids and is easy to reproduce.
Research versus Engineering versus Design
Engineers and designers build things, striving for the best in form (design) and function (engineering).
Research is narrowly focused, incremental, and builds on previous ideas.
Researchers’ prototypes are early mock-ups, while engineers and designers use prototypes to assess alternatives at a late stage.
Tim Brown’s characterization of prototypes: Prototypes should command only as much time, effort, and investment as are needed to generate useful feedback an evolve an idea.
Research precedes engineering and design and progresses at a slower pace.
Engineers and designers develop products within the corporate world, while researchers provide the raw materials.
Example: The computer mouse took nearly 20 years to be engineered and designed into a successful product after Engelbart’s invention in the 1960s.
Example: The iPhone brought together decades of research, including multi-touch gestures and tilt as an interaction technique.
What is Empirical Research?
Empirical means originating in or based on observation or experience.
Empirical means relying on experience or observation alone, often without due regard for system and theory.
Researchers should be guided by direct observations and experiences without preconceptions.
Example: Nicolas Copernicus’ heliocentric cosmology was based on observation without bias toward existing theory.
Empirical means capable of being verified or disproved by observation or experiment.
HCI research is framed by hypotheses that can be verified or disproved by gathering and testing evidence.
Assertions should speak directly to empirical, observable, quantifiable aspects of interaction.
Research Methods
Three common approaches for conducting research in HCI:
Observational method
Experimental method
Correlational method
All three are empirical, based on observation or experience.
Observational Method
Observation is the starting point.
Encompasses techniques like interviews, field investigations, contextual inquiries, case studies, field studies, focus groups, think-aloud protocols, storytelling, walkthroughs, and cultural probes.
Tends to be qualitative rather than quantitative.
Achieves relevance while sacrificing precision.
Studies behaviors in a natural setting.
Concerned with discovering and explaining the reasons underlying human behavior (the why or how).
Focuses on human thought, feeling, attitude, emotion, etc., which are difficult to measure.
Observations involve note-taking, photographs, videos, or audio recordings rather than measurement.
Measurements, if gathered, use categorical data or simple counts of phenomena.
Examines and records the quality of interaction rather than quantifiable human performance.
Experimental Method
Knowledge acquired through controlled experiments in laboratory settings.
May involve gathering new knowledge or studying existing knowledge for verification, refutation, correction, integration, or extension.
Involves relevance-precision dichotomy, with diminished relevance but increased precision.
Requires at least two variables: a manipulated variable and a response variable.
Manipulated variable (independent variable or factor): A property of an interface or interaction technique presented in different configurations.
At least two configurations are required for comparison.
A “usability evaluation” assesses a single user interface, while a “user study” conducts a controlled experiment with different configurations.
Response variable (dependent variable): A property of human behavior that is observable, quantifiable, and measurable.
Common response variable: Time (task completion time).
Methodology borrowed from experimental psychology.
Full user study involves more than just measuring and analyzing human performance; it includes soliciting comments, thoughts, and opinions from participants.
Controlled experiments, if designed and conducted properly, allow for cause-and-effect conclusions.
Correlational Method
Involves looking for relationships between variables.
Example: Examining the relationship between users’ privacy settings and their personality, IQ, education level, etc.
Characterized by quantification and the use of categories for nominal-scale variables.
Data collected through observation, interviews, online surveys, questionnaires, or measurement.
Often accompanies experimental methods, using questionnaires.
Provides a balance between relevance and precision.
Data are circumstantial, not causal.
This book primarily focuses on the experimental method but often incorporates observational and correlational methods.
Observe and Measure
Observation
The starting point for empirical research in HCI is observing humans interacting with computers.
Observations can be made by a human observer or an apparatus (computer).
Human observers collect measurements manually using log sheets or notebooks.
Manual observation can involve timing activities, but it is difficult and inaccurate.
The apparatus (computer) is often used for observation, but it can be challenging in some situations.
Custom software can record events such as key presses, mouse movements, selections, finger touches, and associated timestamps.
Measurement Scales
Observation alone is of limited value. Measurement is essential for science.
Four scales of measurement: nominal, ordinal, interval, and ratio.
Nominal
Involves arbitrarily assigning a code to an attribute or category.
Examples: Automobile license plate numbers, postal codes, job classifications, military ranks.
Mathematical manipulations are meaningless.
Nominal data identify mutually exclusive categories.
Also called categorical data.
Often used with frequencies or counts.
Ordinal Data
Provides an order or ranking to an attribute.
Example: Ranking GPS systems by preference or ordering mobile phone features by personal importance.
Limitation: Intervals are not intrinsically equal between successive points on the scale.
Comparisons of greater than or less than are possible.
It is not valid to compute the mean of ordinal data.
Interval Data
Equal distances between adjacent values but no absolute zero.
Example: Temperature measured on the Fahrenheit or Celsius scale.
Meaningful to compute the mean of interval data.
Linear scales are commonly used in questionnaires. Example: Likert Scale
Rations of interval data are not meaningful. For example, 20°C is twice as warm as 10°C.
Ratio Data
Most sophisticated of the four scales.
Has an absolute zero and supports a myriad of calculations.
Can be added, subtracted, multiplied, divided; means, standard deviations, and variances can be computed.
Most common ratio-scale measurement in HCI: Time (task completion time).
Physical measurements are also ratio-scale, such as the distance or velocity of a cursor.
Social variables like a user’s age or years of computer experience.
Count (number of occurrences of certain human activities).
Expressive nature of a count is improved through normalization.
Research Questions
In HCI, experimental research aims to answer questions about a new or existing user interface or interaction technique.
Often pertains to the relationship between two variables: a manipulated interface property and an observed behavioral response.
Humans exhibit variability in their actions, affecting the confidence with which we can answer research questions.
Statistical techniques are used to gauge the confidence of our answers.
Research questions emerge from an inquisitive process.
Initial thoughts should be testable.
The expressions like “any good” or “better than,” although well intentioned, are problematic for research.
Qualities that are more easily observed and measured should be included (comparisons are possible).
Internal Validity and External Validity
Internal validity: The extent to which an effect observed is due to the test conditions.
External validity: The extent to which experimental results are generalizable to other people and other situations.
High internal validity means the effect observed really exists.
External validity implies that the participants were representative of a larger intended population.
Generalizable to “other situations” means the experimental environment and procedures were representative of real-world situations.
Experiment design is an exercise in compromise.
Posing multiple narrow (testable) questions that cover the range of outcomes influencing the broader (untestable) questions will increase both internal and external validity.
Ecological validity refers to the methodology (using materials, tasks, and situations typical of the real world), whereas external validity refers to the outcome (obtaining results that generalize to a broad range of people and situations).
Comparative Evaluations
Evaluations in HCI sometimes focus on a single idea or interface, making the research component questionable.
More meaningful and insightful results are obtained if a comparative evaluation is performed comparing a new user interface or interaction technique with one or more alternative designs.
Testable research questions are crafted as comparisons.
A controlled experiment must include at least one independent variable with at least two levels or test conditions.
Including an established design as a baseline condition serves as a check on the methodology and allows results to be compared with other studies.
A comparative evaluation yields more valuable and insightful results than a single-interface evaluation.
Relationships: Circumstantial and Causal
Looking for and explaining interesting relationships is part of what we do in HCI research.
A cause-and-effect relationship, or simply a causal relationship, is possible if a controlled experiment is designed and conducted properly.
Finding a causal relationship in an HCI experiment yields a powerful conclusion.
Finding a relationship does not necessarily mean a causal relationship exists. Many relationships are circumstantial.
Causal relationships emerge from controlled experiments with random assignment.
Research Topics
Most HCI research is not about designing products, but nipping away at the edges.
Most new research ideas tend to build on existing ideas and do so in modest ways.
*Finding a research topic is often the most challenging step for graduate students in HCI (and other fields).
Tip #1: Think small!
Forget about the big idea.
If you have a small idea, it’s probably worth pursuing as a research project.
Tip #2: Replicate!
An effective way to get started on research is to replicate an existing experiment from the HCI literature.
Tip #3: Know the literature!
The process of reviewing research papers on a topic of interest is an excellent way to develop ideas for research projects.
Tip #4: Think inside the box!
Thinking inside the box challenges one’s experiences—the experiences inside the box.\