meta-research and the watcher

what is meta-research

• research of how people do research

• not interested in findings, but as how we as a species produce findings

• understand the cog and social processes that lead to scientific discoveries and innovation

theories and observations

• usually exposed to theories and they have data to support, but that isnt how theories come about in the first place - usually come from observation

• so want to re-focus on observation, origin point of most psych knowledge and breakthroughs

• observe then build patterns, infer and generalise

questions about research

• How can we go about taking a snapshot of what is happening?

• How to make sense of how human endeavor is moving (or not) forward?

• Can we predict new discoveries? By whom? When? Where? Who is driving innovation 

• Who will publish the next breakthrough?

• Who will get grants? Who will get tenure? - knock on effects for careers 

questions we are interested in when conducting meta-research

• How many authors?

• How many institutions?

• How many junior/senior researchers?

• How many female/male authors?

• What is the main field of research of each author?

• How much funding supported a paper?

• Where are the authors’ institutions?

• What are the references cited?

• Which and how many papers cite in turn the paper?

• What is the title of the article?

• Is the paper open-access?

• How much does an institution publish?

• How well cited is an institution?

• What was the contribution of each author to a study?

types of findings

• expected and unexpected

• known unknowns - gap in knowledge in a particular field so focus research on that. eg higgs boson, expected to find it since the 70s etc

• unknown unknowns - things we dont know, cant even fathom how we would find it, eg penicilin

• middle - theory of evolution, years of research and observation, cross check theories etc

how does this link to the type of research

• unexpected = observation derived

• expected = theoretical derived

proportion of expected vs unexpected

• need a healthy mix

• But, if unexpected findings cannot be predicted, should we simply focus on theoretical-derivations? How do we go about finding those. should we focus on the right side as we can actually pursue those 

when are researchers most productive

• early years of research

• so if more research is happening earlier on, maybe more unpredictable research is happening there

when are researchers most disruptive

• order scientists papers from first to last

• identify most disruptive work

• equally likely to do disruptive work at any point in career

• quantity of papers does not predict quality or disruptiveness

• eg physics nobel proze winner got it early in career but chemistry got his later on

implications

• There seems to be little probability of accurately predicting when someone will produce its best work…

• So hard to balance sheets of who is producing research across that spectrum

• Q: Are there mechanisms adding “noise”? Biasing variables?

are there mechanisms adding noise and biasing variables?

• yes - sleeping beauties in science

what are sleeping beauties

• studies that are ahead of their time, not recognised so stay dormant, finding has been made, but not a lot of people know about it  

• The impact and value of a discovery is not necessarily readily/immediately appreciated.

• Research is “ahead-of-its-time”.

• If the importance of a paper can remain latent for many years, this hampers prediction

• It's not that the discovery wasn't predictable, it just was lost somewhere

time lag for studies to be recognised

• a lot longer

• before waiting less than 10 years for it to be recognised

• now it may be around 30 years

• for some, may never be recognised - lost under stack of new research

is the the size of a team linked to the type of discovery made?

• yes

• large teams tend to produce known unknowns - because if working in a lab with lots of people towards the same goal, have to be agreeing with each other, have a consensus. Already have a consensus so know should work towards goal

• small teams tend to discover more unknown unknowns - work off the gate, dont need consensus

other beh of small vs large teams

• Small teams cite earlier and less popular studies/references. - know more sleeping beauties

• Citation delay to small and disrupting teams (longer citation lag). Have a higher likelihood of becoming sleeping beauties themselves. Further away from consensus - delay for work to be recognised 

• Proportion of small teams has been decreasing over recent decades.

• It takes decades-long for large teams to start citing small teams’ work. Wild position becomes more mainstream and therefore is adopted by teams

team size and disruption potential for prize vs non-prize winners

• as team increases, disruption potential decreases drastically

• prize winners - dynamics of the group dif, can extend the team size a bit more for producing disruptive work

• team of 10 = incredibly low disruptive potenital - already know the kind of discoveries that will be made

funding for small vs large teams

• small receive a lot les

• large recive a lot more

awards for large vs small

• small get more awards - more potential for impact

• large teams produce less than would be predicted

team size over time

• increasing

• larger teams producing more developmental work

career prospects and team size

• For every person you add to a team, the phd student has 25% less chance of reaching tenure and becoming an established scientist 

• For every person you add, the higher the chance of the phd student working outside the field 

• So may want to go to a large team as they are getting more funding, however there are effects on the student 

team diversity examples and effects

• promotes abstraction & cognitive flexibility in collective problem solving

• Sex/age group

• Ethnic/religious/cultural background

• Socio-economic level

• Personality

• More angles over a subject/problem/puzzle

• Search compatibility + alignment of interests but avoid recruiting/teaming up with “clones”, or “team inbreeding”

• Dividends to those who take the high “moral” road

• Seeking a variety of perspectives and ideas, focusing on same project but from different angles 

  
  

How has scientific output and impact changed over the years/decades?

• teams doing larger work so becoming less disruptive

• disruptive potential slowly decreasing

slowed progress in large fields of science

• Number of papers published per year has increased over time, so we are producing more science, larger field, more people are doing it. But larger teams and fields may incur some costs 

• Policy measures (by funding agencies & universities) aim to increase paper output

• Paper output determines how career trajectories, departments’, unis’, nations’ evaluations

what does this increase in papers result in

• stagnation

• deprive reviewers and readers from fully recognizing and understand novel ideas

• Less time, less headspace to process what is being produced 

• When a scientific field grows large, citations flow disproportionately to already well-cited papers; the list of most cited papers “fossilizes”; anchor effect among a sea of incoming new waves of papers; (“comfort in the known”), famous studies just become more famous

• New papers are unlikely to ever become highly cited, slower turnover of knowledge 

• The size of scientific fields may impede the rise of new ideas

• Research quantity is not a substitute for research quality

has digitalisation and interconnectedness impacted knowledge

• yes

• remote collaborations lead to fewer breakthrough ideas

• increase distance between authors, disruptiveness of papers and patents decreases

• if in dif time zones, even less chance of disruptiveness

on site vs remote teams

• If working on site, every person has equal likelihood in doing each part of the work

• In remote teams, only have same likelihood of carrying gout the work, conception and writing up not happening remotely 

• Citations that studies harvest over time - when work comes from an onsite time, have a high likelihood of research citations, whereas fewer remote teams make a dent in the field, become less known 

• So in person is important 
  
  

impact of ai

• Engage with ai - produce 3 times as many papers 

• Also receive 4.8 times more citations  - make a bigger dent, but larger fields also tend to be less disruptive 

• However shrinks the volume of topics studied by 4.6% and decreases scientist’s engagement with one another by 22%

• Results in an expansion of individual scientists’ impact but a contraction in collective science’s reach

• works on known unknowns - theoretical derivations

so characteristics for disruptive discoveries

• Want to be on the left side of the spectrum otherwise ai and large teams will smash balance the other way round 

• Build/be part of a small team, “rogue teams”

• Build/be part of a diverse team, “no-clone teams” different perspective 

• Build/be part of a free team, “opiniated-and-open-to-critique team”

• Build/be part of personal team, “under-the-same-roof teams” 

• But more determinately, across these forms… we want improve our observation skills,

inattentional blindness

• Inattentional blindness is the failure to notice a fully-visible, but unexpected object because attention was engaged on another task, event, or object.

• So unpredictable objects etc are difficult to observe as we are not expecting them

• eg gorilla study

explanations for inattentional blindness

• Perception is a limited process (i.e., cognitive resources are limited). So don't have power to attend to other things. Other say paid attention 

• The unnoticed stimuli in a visual scene are attended to and consciously perceived, but are rapidly forgotten rendering them impossible to report.

• Lack of expectation for the stimuli

• (Un)Expectation-based self-fulfilling prophecy, if you lack expectation you wont pay attention, but you want to pay attention to search for the unexpected 

illusions and perceptions

• Our thinking goes together with our senses.

• They are extensions on one another.

• Perception “illusions” can trick us about reality.

• But how we think about reality can affect how we perceive reality.

to make sense of reality

• Clean observation

• Unbiased by-standing, let go of filters, just observe things as they are

• Detective’s eye, what does this thing being there imply

• Knowledge of context

black swan theory

• Universal truths are difficult to prove due to extremely rare events

• Rare events, by virtue of being unexpected, carry unproportionable consequences

• The power of N=1 vs. Large samples (i.e., what the majority does is by average is predictable, but the average is not disruptive).

• Thick data point 

• All swans observed by explorers were white, so concluded all swans were white

• When landed in new zealand, saw a black swan, so One point can disprove universal truth - can completely rewrite expectations

outliers in stats

• we learn to “exclude” outliers; data that fall beyond the theoretical predictions of where data should be. This helps to run statistical tests/models, but it comes at a price, because we are discarding our “mis-fit” data -> they are rare and disruptive!

• Only doing it for convenience of the tool

• Averages often bend reality (the average of two socks). As we average things we let go of important data

• How we “sample” the world determines our conclusions.

• Beware of grand-sweeping statements/theories

semiotics

• “Study of signs”

• Sign: any activity, conduct or process that communicates something

• Intentional (word uttered with a specific meaning) or not (a symptom of a particular medical condition, psychological/emotional state)

• Linguistic or not; anything can be meaningful IF the observer can extract information (a piece of clothing can be informative).

• Absence of a sign is also a sign - meaningful - by missing something you also extract information, if dont care about how you dress, can tell something about a person

• So it is An activity, conduct or process becomes a sign(al) when the observer can extract information.

• An observer needs to “diagnose” a situation or event (Reading “signs” was first a sub-discipline of Medicine) reading of the signs that allowed the doctor to complete the diagnosis 

• To find “give-aways” one data point can determine conclusion 

• In the present tech-age of big data,