Sleep experiment initial results and notes

Co-authors: niplav and No Magic Pill

Over the last few months we have been doing a sleep experiment inspired by our suspicion that orexin is an exciting target for sleep need reduction.

We mildly deprived ourselves of sleep (5-5.5 hours, relative to 7-7.5 hours normally) and took either a placebo or orexin intranasally. We tracked our sleep the night before and after taking a dose in the morning and completed various tests of mental acuity during the day.

The results from our initial experiment are exclusively null results that don’t cross standard thresholds for statistical significance. Not that this was particularly surprising, we expected a ~60% chance of this happening. We’re considering next steps, and need your feedback!

For now, there are a few things to cover in the results.

Trial Design

We performed a self-blinded randomized controlled trial with blocking, each participant took either the placebo (2.5 mL of sterile water) or the orexin (100 μg of orexin-A dissolved in 2.5 mL of sterile water). Here’s the procedure, repeated for every block:

1. Prepare two nasal atomizers, one with saline solution and one with orexin+saline solution

2. Night to the first day: Sleep 5-5.5 hours.

3. First day:

   1. At a consistent time of day, randomly select one and administer.

   2. Take mental acuity tests. Once mid-day, once in the evening.

   3. Track sleep, heartrate, etc. using a Fitbit Inspire 3.

4. Night to the second day: Sleep normally.

5. Night to the third day: Sleep 5-5.5 hours.

6. Third day: Repeat 3.1-3.3, using the remaining dose/placebo you prepared.

7. Night to the fourth day and fifth day: Sleep normally.

8. Fifth day: Record baseline sleep measurements and mental tests

Each person had a substantial amount of leeway in how they structured their day. On sleep deprived days, Sam preferred to get up early while No Magic Pill and niplav preferred to stay up late and get up at the usual time. We each took doses at a consistent time, but the time differed between people.

We didn’t standardize things because we thought it was more important to have ecological validity, i.e. that we were using orexin the way we would actually use it in everyday life. This is a higher variance, but lower bias approach.

The Results

In our initial proposal, we pointed out that the main thing we wanted to see was orexin causing less rebound sleep the following night. A simple stimulant effect isn’t enough for us, we wanted to use orexin to sleep less and get away with it.

So here’s the average sleep time for the night after taking orexin vs the night after taking placebo:

Unfortunately, the difference wasn’t significant and the effect size is small. This could be for a couple reasons that we want to address in the next trial.

Did orexin have any sort of stimulant effect during the day? Nope, none of the mental acuity tests are significantly different.

In setting up this trial we had a sneaky second hypothesis: Does sleep deprivation actually make you dumber?

One caveat before we look at the data. Typically our “baseline” days would come after our sleep-deprivation days. So that means baseline days enjoy more cumulative practice compared to sleep-deprivation days. That should bias the results by making baseline days look better. On the other hand, if sleep deprivation has long term cumulative effects, then perhaps baseline days are at a disadvantage. But that doesn’t match our experience of feeling significantly better on baseline days.

So, does sleep deprivation make you dumber? Not really!

Depending on how you correct for multiple comparisons, the psychomotor vigilance task (PVT) differences might be significant. And I’d expect the PVT differences to become significant with more data points. From what I (Sam) remember from doing PVT on sleep-deprivation days, I felt just as fast, but I would slip-up more from inattention or distractions. This is consistent with the large gap on the slowest 10% days.

But overall, this is a nice example of how our intuitions around sleep can lead us astray. It sure feels like sleep deprivation should make you dumber. But we don’t see that here. It’s important to actually check what changes our productivity because our intuitions around this are pretty fuzzy.

The Next Trial

There’s a few reasons why we might be getting a null result. We might have too few data points, or the dose might be too low, or more concerningly, we might be storing the orexin improperly.

So the first next step is a slightly bigger trial where we see if a higher dose of orexin changes our results. From anecdotes online, some have felt effects while others haven’t. But even if orexin doesn’t have obvious effects, it might still reduce sleep need. We need to try higher doses and collect more data to find out.

That said, sleep deprivation is uncomfortable for Sam and No Magic Pill and extremely uncomfortable for niplav. We’ve decided to try a different design: sleep ad libitum on all nights of the week, but observe whether orexin reduces the amount we sleep the night after. This should make it sustainable to collect a lot more data.

Appendix A: Details about the Data Analysis

We collected two separate datasets:

1. Data collected automatically from the Fitbit Inspire 3

   1. Measures of sleep

      1. Sleep duration

      2. Sleep efficiency

      3. Time spent in deep sleep

      4. Time spent in light sleep

      5. Time spent in REM sleep

      6. Time spent in nocturnal awakenings

   2. Additional measures

      1. HRV Daily RMSSD (ms)

      2. HRV Deep RMSSD (ms)

      3. SpO2 Avg (%)

      4. SpO2 Min (%)

      5. Breathing rate (breaths/min)

      6. Skin temperature Δ (°C)

      7. Steps

2. Data from the mental acuity testing

   1. Psychomotor vigilance task

   2. Digit symbol substitution task

   3. Digit span

   4. Stanford Sleepiness Scale

   5. Description of subjective state (free text)

We aggregated mental acuity tests per-test to avoid pseudoreplication (so two data points per day), and aggregated Fitbit data per-day. We analyzed the data via matched controls (with days in a participant-block being matched as to analyze within-pair differences) and ran two separate analyses on the data; one frequentist and one Bayesian. The code for the analysis, written in Julia by Claude Opus 4.6, is available here. Our mental acuity test data is available here, aggregated full data is available here.

Frequentist Analysis and Additional Results

In our frequentist analysis we ran the paired t-test on the paired data with cardinal measurements, and the Wilcoxon signed rank test on paired data with ordinal measurements, we also report Cohen’s d for the measurements. We Bonferroni-corrected the p-values, not that that was necessary…

[The table was too unwieldy to include, see here for more.]

Bayesian Analysis and Additional Results

We fit a hierarchical Bayesian linear model with participant random intercepts, using NUTS (4 chains × 2000 samples per metric). The primary estimand is δ, a standardized treatment effect (Cohen’s d-like), with a weakly informative N(0,1) prior.

Formally, the likelihood is yᵢ ~ N(μ + δσ·treatmentᵢ + α[pᵢ], σ), where treatmentᵢ ∈ {0,1} encodes placebo/orexin. The raw treatment effect on the outcome scale is δσ; δ alone is dimensionless. Priors: μ ~ N(0,10) (vague grand mean), σ ~ half-N(0,10) (residual SD), τ ~ half-N(0,5) (between-participant SD), α[j] ~ N(0,τ) iid for each participant j.

Priors and posteriors for cognitive acuity tests and sleep measurements:

Priors and posteriors for additional Fitbit data:

Learning Effects on Mental Acuity Tests

Circles for the first test of the day, diamonds for the second test of the day.

Appendix B: Threats to Validity

Our method seems simple on its face, but there were a lot of annoyances along the way.

1. Orexin was delivered at room temperature, and while the vendor claims the orexin was lyophilized, we are uncertain if the lyophilization was sufficient to prevent damage.

   1. In niplav’s case the orexin sat uncooled in customs for over a week during the delivery in July.

2. In order to distribute the orexin into vials, we had to dissolve it in water. This meant that we had to both store the orexin dissolved in water for almost a week, and freeze the rest. We are uncertain if any of those damaged the peptide structure.

3. We are uncertain if our route of administration can cross the blood-brain barrier.

4. One participant wasn’t aware that Fitbit data needs to be regularly synced, so we only have sleep data for two individuals. Additionally, Fitbit syncing and data collection is unreliable, leaving us with only 17 datapoints for nights of sleep following orexin.

5. Another headache was making sure sleep deprivation nights were scheduled between nights where we could sleep ad libitum. We also tried to keep a consistent schedule on trial days so that variations in exercise, nootropic consumption or other activity didn’t change our results.

Appendix C: Personal Experiences

Niplav:

• So much stuff can go wrong

• 2.5ml is way too much, let’s do 1ml next time

• Filling the syringes was fun! It felt very scientist-y.

• Plausibly we should’ve started with a higher dose but also safety concerns so whatever

• 5½ hours of sleep feels horrible

• I was ~completely unproductive on sleep deprivation days, and will put a high premium on this

• Caffeine was really helpful

• Two nights of normal sleep & then one more sleep deprivation would’ve been better

• Inform yourself about the reliability of the data collection tools

• Didn’t nap at all except in week two, when I just couldn’t stay awake

• I’m happy we did a short trial first so we discovered the data collection issue early

• Beat-by-beat experiment log here

Sam:

• Feel so much better and alive from my rebound sleep, better than a normal days sleep even.

• Consistently napped on placebo days, but orexin does seem to have stimulant effect.

• Couldn’t really tell which was which in general

• Felt mentally the same on trial days. But doing tests felt a little slower than on baseline day. In general I didn’t have a good sense of how productive/smart I was.

• Extra hours in the morning did get used, for somewhat intellectual tasks like reading papers and writing.

• Much easier to be up early when sun was up

No Magic Pill:

• Tests:

  • PVT: I don’t think my reaction time improved at all over the course of the testing (75% confident). I did not mind this test.

  • DSST: I don’t think my skills improved much over the course of the testing. I disliked this test the most.

  • Digit span: I am 75% confident that I got better over the course of testing. I was consistently able to get 9 both forward and reverse towards the end. I disliked this test the second most (behind DSST).

  • Sleepiness: I never scored that high and probably erred on the side of scoring higher because of a feeling that I needed to utilize most of the scale. I did not mind this test.

  • Feelings: I could have been more verbose here.

• It was wayyyyyyy easier to stay up late and get up at my normal time than it was to go to bed at my normal time and wake up early.

  • I was fairly productive when staying up late

  • I was NOT productive when getting up early

  • Most of the time I had full-body “tingles” when awakening after a sleep-deprived night. I’ve experienced this phenomenon for years: anything less than ~6 hours, or normal duration after a hard exercise session, leaves me “tingley” in the morning.

• I was a normal level of irritable and quick to anger after a sleep-deprived night. This has been consistent for years (like the tingles).

• I did not nap on any day (pre-test or day-of test) because I thought that would throw off the testing data.

• My motivation on sleep-deprived days often waned faster than non-sleep deprived days. This has been consistent for the past few years and is what I expected.

• I did not feel anything physically or mentally immediately following the orexin administration.

• I should have done a better job of isolating myself during testing. Sometimes it was a bit noisy or visually distracting, especially if I was at work. I should have noted down if I was distracted during the test.

  • Maybe a feature can be added to add comments at the end of each test?

• I agree with Niplav that 2.5 mL was too much water. 1.25 mL was good, if not a tad much as well. I think 1 mL is probably good for the future?