For the last two years or so I’ve been struggling with insomnia. I typically go to bed around midnight and get up around 8, but in between I frequently wake up around 3 or 4 and feel like I’m awake for at least an hour on good nights, longer on bad nights. I seldom feel well-rested; when I do sleep well it’s a cause for celebration.
My doctor told me I need to establish good sleep habits. Unfortunately (or fortunately) I already have pretty good habits—to bed around the same time every day, up around the same time every day, no naps, no caffeine after lunch, etc—so it’s difficult to see where I can improve. What I need is data. My doctor is not in favor of doing a sleep study though.
As luck would have it, I was browsing through Apple’s App Store and came across a pretty neat-looking iPhone app: Sleep Cycle, billed as an alarm clock that wakes you up when most natural, but which actually collects data about your movements during sleep. I was only moderately interested in the alarm clock feature, as I tend to wake on time without alarms anyway, but I was very very interested in the data-gathering function.
The way it works is you put the iPhone on your mattress face down, and as you move around in your sleep, the iPhone’s accelerometer registers the time and intensity of your movements. Over the course of the night, it collects this data; in the morning you can look at a time-series plot showing when you were moving the most and when you were most still. (The app puts sleep state on the y axis. It’s not clear to me how valid this is, which I’ll get to below.)
Hacking the App
This is already very cool, but I want the raw data. So I went digging into the iPhone’s file system and discovered that the app stores the data in a SQLite database. I transferred this over to my laptop and opened it in R using the RSQLite package. I could then get all the data from the ZSLEEPEVENT table. Each row has a sleep session number, an event type (an integer that seems to be between 1 and 4; I’m not sure what the numbers mean), an event intensity, and a timestamp. The figure below shows the raw data that went into the final plot the app shows me.
Obviously the app is doing some smoothing to get its figure. I spent some time trying to replicate it, and came up with something fairly close using moving averages. I finally decided to use the wapply() function in the gregmisc library. This figure shows the smoothed data.
In the locations of peaks and valleys it’s fairly close to what the app shows, though in peak/valley height it’s pretty different. Over the course of a couple days I’ve found that these wapply() plots correspond better than the app plots to how I feel about my quality of sleep, when I remember being awake or semi-awake, when I remember making big movements, etc.
What I really want to do is figure out what’s going on when I feel well-rested versus when I don’t, and what I might be able to do differently to feel more well-rested. Getting this sleep data is the first step. I also want to keep track of some other data about each night, such as whether I used any sleep aids, how late I last ate, whether I exercised, and so on. Then I want to look at all that with how well-rested I feel.
So I’m collecting all this data manually, rating my well-restedness on a 5-point Likert scale, and joining it with some features computed from the raw data. Right now I’m using area under the sleep curve, with greater area suggesting worse sleep. Another feature might be percent of total sleep time spent below some intensity threshold.
Once I have enough data, I’ll start looking for patterns. Hopefully I’ll be able to identify some things that I have control over and that consistently predict a good night’s sleep. This post is the setup; the next one will have some in-depth analysis.
- How well does movement intensity correlate to sleep state? My understanding is that in REM sleep, muscle paralysis sets in, so there’d be no movement. If that’s true, there must be some relationship. How strong it is, I don’t know.
- What other information can I compute from the raw movement data that I can hack on?
- Should I be comparing my sleep to a baseline? If so, how would I obtain that baseline?
- Will a future version of Sleep Cycle make it harder to access the raw data? Note to developers: I’d pay extra to be able to keep doing this.
Below the fold, some R code I used to extract the data and make plots.
Once the SQLite database has been transferred off the iPhone, getting the raw data is easy:
con <- dbConnect("SQLite", "eventlog.sqlite")
rs <- dbSendQuery(con, "select * from ZSLEEPEVENT")
d <- fetch(rs)
The data looks like this:
Z_PK Z_ENT Z_OPT ZTYPE ZSLEEPSESSION ZINTENSITY ZTIME
1 1 2 1 4 1 0.000000000 315884113
2 2 2 1 3 2 0.016427328 315899014
3 3 2 1 1 2 0.148254603 315899014
4 4 2 1 3 2 0.017570622 315899075
5 5 2 1 1 2 0.142615467 315899075
6 6 2 1 3 2 0.013057620 315899915
7 7 2 1 1 2 0.078949176 315899915
8 8 2 1 3 2 0.013900047 315900216
9 9 2 1 1 2 0.094113655 315900216
10 10 2 1 3 2 0.005114736 315902078
I select a ZSLEEPSESSION and smooth ZSLEEPINTENSITY along ZTIME with wapply():
d2 <- subset(d, ZSLEEPSESSION==2)
sm <- wapply(d2$ZTIME, d2$ZSLEEPINTENSITY, mean, n=100)
plot(sm$x, sm$y, type="l", xlab="time", ylab="intensity", xaxt="n")
The times above are in seconds since Jan 1st, 2001, at 7pm (as far as I can tell). To convert them to HH:MM format for the plot, I do this:
axp <- c(signif(min(sm$x)), signif(max(sm$x)), 10)
axt <- axTicks(1, axp)
time <- strftime(as.POSIXct(axt, origin="2001-01-01 19:00:00"), format="%H:%M")
axis(1, at=axt, labels=time, las=3)