Title of the research experiment: Visualizing confidence

Eligibility

You may only take part in this study if:

You understand English,

You are at least 18 years old,

Have at least a basic understanding of hypothesis testing and confidence intervals,

You use statistical tools in your research projects,

You are not using a handheld device such as tablet or phone to fill out the survey.

If you believe you may not be eligible, you can withdraw from the study without having to justify why.
Alternatively, you can ask clarifications about the eligibility criteria above before you make your decision by contacting any of the two investigators.

Should you wish to take part in this research experiment, please read the following information carefully.

Aim of Research

The goal of this experiment is to study the impact of visualization on statistical inferences.

Procedure, Data Collection, and Use of Gathered Data

The study is only run online through this webpage. Based on pilot-studies, we expect it to last less than 15 minutes. You can take a break at any moment,
but for technical reasons you cannot close the browser window containing the survey without terminating the study. In each page of the main study
you will be presented a single question.
It is not possible to go back to the previous questions or skip questions.

You will be asked, at the beginning of the experiment, to fill in some demographic information but you will not be asked to provide any identifiable information.
We will not save any information about your location, software, or hardware.
Therefore no one, including the researcher, will be able to identify you from your responses.
Your responses will be used and analyzed by the researchers as part of a dataset generated by other participants who complete the study.
The findings of the study may be presented in a written publication and/or presented at conferences in future, and the study dataset may be shared online to promote open science.

Your information will be dealt with in accordance with The Freedom of the Press Act and The Public Access to Information and Secrecy Act (2009: 400, OSL).
According to Linköping University your information will be subject to secrecy provisions in accordance with Chapter 24
Section 8 OSL and Section 7 of the Public Access to Information and Secrecy Ordinance (2009: 641).

Benefits, Compensation, and Risks

There will be no direct benefits for you taking part in the research, though you may gain some insight into how the different visualizations you will see could
be beneficial for your work. Your responses will be a very helpful contribution to research through furthering our understanding of people's perception of statistical results.
You can, after the study is over, contact the principal investigators to learn about the outcome of this experiment.

There will be no monetary compensation for participating in this study.

As the experiment does not involve any action other than the ones involved in the daily use of a computer, we expect that this study does not
present any other risk than the minimal risk of using a computer.

Contact

Should you have any question, please contact the main investigators at this address: jouni[_dot_]helske[-at-]liu.se

Assessing the effects of representation styles for statistical inference

Goal of the study

The goal of this experiment is to study whether various
representation styles of the results from a simple statistical analysis have effect on interpretion of these results.

Background story

A random sample of adults from Sweden were prescribed a new medication for one week.
Based on this sample, you are trying to figure out whether the medication resulted in a
change in the participants' body weight.
Note that this is just an illustrative story, you can think a similar example from your own research area.

Your task

In this experiment you will be presented results from multiple independent trials
like the one described above, with various representation styles of the results.
Your goal is to assess whether the medication has an effect on the body weight.

Explanation on the task

In each of the following pages, you will be presented with a figure or a text
describing the results of a simple trial.
Your task is to answer the question
"How confident you are that the medication has a positive effect on body weight? (increase of body weight)".

Answer is given on a continuous scale using a slider (from 1 to 100).
In order to avoid accidental skipping of the trials,
the cursor on the slider is hidden until you click the slider.

There will be four sets of 8 trials, followed by a few questions about the overall
impressions of the representation styles and the survey in general.

Note: It is not possible to go back to the previous questions or skip questions.
Refreshing the page will also interrupt the survey.

Representation using text

What is shown?

In these reprentations, you are given a text with a value of the sample mean,
limits of the 95% confidence interval, and a p-value from a two-tailed t-test
with null hypothesis that the population mean is zero (i.e. the medication has no effect on the body weight).

What is a p-value?

In the context of this experiment, the p-value is the probability of
observing a sample mean value as extreme or more extreme than the actual observed mean,
assuming that the true population mean is zero.

What is a confidence interval (CI)?

A 95% confidence interval of a mean refers to an interval estimate which,
if computed for infinitely many independent samples of equal size from the same population,
would contain the true population mean 95% of time.

The CI can be also interpreted in terms of
hypothesis testing as follows:
The confidence interval presents those values of the true population effect
which would not lead to the rejection of the null hypothesis when testing against our
observed mean using t-test with significance level of 0.05.

How is the CI computed?

The confidence interval for the mean is computed using a sample mean, sample standard deviation,
sample size and t-distribution. For example assuming sample size of 100, the upper limit of 95% confidence interval is computed as
x = sample_mean + t * standard_error, where t=1.98 is the critical value from t-distribution with 99 degrees of freedom.

Representation using confidence intervals

What is shown?

In these representations, you are given a graphical representation of a
95% confidence interval of the mean (lower and upper whiskers),
and the sample mean (black dot).

What is a confidence interval (CI)?

A 95% confidence interval of a mean refers to an interval estimate which,
if computed for infinitely many independent samples of equal size from the same population,
would contain the true population mean 95% of time.

The CI can be also interpreted in terms of
hypothesis testing as follows:
The confidence interval presents those values of the true population effect
which would not lead to the rejection of the null hypothesis when testing against our
observed mean using t-test with significance level of 0.05.

How is the CI computed?

The confidence interval for the mean is computed using a sample mean, sample standard deviation,
sample size and t-distribution. For example assuming sample size of 100, the upper limit of 95% confidence interval is computed as
x = sample_mean + t * standard_error, where t=1.98 is the critical value from t-distribution with 99 degrees of freedom.

Representation using t-violin plots

What is shown?

In these representations, the figure shows 95%, 95.1%, ..., 99.9% confidence intervals (0.1 percentage point increments)
as colored areas. Until 95% CI the color is constant, and it then gradually changes until 99.9% level.
The black line is a sample mean.
The width of the area is determined by the value of the corresponding density function of the t-distribution
used in constructing the CIs. For example, width of the figure at point x is defined as f((x - sample_mean) / standard_error) / standard_error,
where f is the density function of t-distribution with 99 degrees of freedom.

What is a confidence interval (CI)?

A 95% confidence interval of a mean refers to an interval estimate which,
if computed for infinitely many independent samples of equal size from the same population,
would contain the true population mean 95% of time.

The CI can be also interpreted in terms of
hypothesis testing as follows:
The confidence interval presents those values of the true population effect
which would not lead to the rejection of the null hypothesis when testing against our
observed mean using t-test with significance level of 0.05.

How is the CI computed?

The confidence interval for the mean is computed using a sample mean, sample standard deviation,
sample size and t-distribution. For example assuming sample size of 100, the upper limit of 95% confidence interval is computed as
x = sample_mean + t * standard_error, where t=1.98 is the critical value from t-distribution with 99 degrees of freedom.

Difference to common violin plots

Note that violin plot typically describes the estimated probability density function of the data, whereas in this case we visualize the uncertainty regarding
the estimation of mean. These figures are also closely related to so called cat's eye plot.

Representation using gradient plot

What is shown?

In these representations, the figure shows 95%, 95.1%, ..., 99.9% confidence intervals (0.1 percentage point increments)
as colored areas. Until 95% CI the color is constant, and it then gradually changes until 99.9% level.
The black line is a sample mean.

What is a confidence interval (CI)?

A 95% confidence interval of a mean refers to an interval estimate which,
if computed for infinitely many independent samples of equal size from the same population,
would contain the true population mean 95% of time.

The CI can be also interpreted in terms of
hypothesis testing as follows:
The confidence interval presents those values of the true population effect
which would not lead to the rejection of the null hypothesis when testing against our
observed mean using t-test with significance level of 0.05.

How is the CI computed?

The confidence interval for the mean is computed using a sample mean, sample standard deviation,
sample size and t-distribution. For example assuming sample size of 100, the upper limit of 95% confidence interval is computed as
x = sample_mean + t * standard_error, where t=1.98 is the critical value from t-distribution with 99 degrees of freedom.

A random sample of adults from Sweden were prescribed a new medication for one week.
Based on the information on the screen, how confident are you that the medication has a
positive effect on body weight (increase in body weight)?

Leftmost position of the slider corresponds to the case "I have zero confidence in claiming a positive effect",
whereas the rightmost position of the slider corresponds to the case
"I am fully confident that there is a positive effect."

Zero confidenceFull confidence

The p-value is from a two-tailed t-test
with null hypothesis that the population mean is zero (i.e. the medication has no effect on the body weight).

The figure shows 95% confidence interval of the mean (lower and upper whiskers),
and the sample mean (black dot).

The figure shows 95%, 95.1%, ..., 99.9% confidence intervals (0.1 percentage point increments)
as colored areas. Until 95% CI the color is constant, and it then gradually changes until 99.9% level.
The black line is a sample mean.
The width of the area is determined by the value of the corresponding density function of the t-distribution
used in constructing the CIs.

The figure shows 95%, 95.1%, ..., 99.9% confidence intervals (0.1 percentage point increments)
as colored areas. Until 95% CI the color is constant, and it then gradually changes until 99.9% level.
The black line is a sample mean.

Subjective Assesments

Thank you for your participation so far. Before the end of the experiment,
we would like to ask you to give us some
feedback of the visualization you have seen in this experiment.