Abstract

Objectives

To determine whether assessment-to-assessment fluctuations in episodic memory (EM) reflect fluctuations in olfaction over time.

Methods

Within-person coupled variation in EM and the Brief Smell Identification Test (BSIT) was examined in 565 participants aged 58–106 with autopsy data from the Rush Memory and Aging Project. A growth model for up to 15 years of EM data, with BSIT as time-varying covariate, was estimated accounting for main effects of sex, education, ε4 allele, and Alzheimer’s disease (AD) pathology, BSIT and time-varying BSIT, as well as the interaction between AD pathology and time-varying BSIT.

Results

Individuals with higher BSIT scores (b = .01, standard error [SE] = .004, p = .009) had slower declines in EM. High AD pathology (b = −.06, SE = .02, p = .001) was associated with more rapid declines in EM. The association between time-specific fluctuations in EM and BSIT differed by level of AD pathology (b = .08, SE = .034, p = .028), with a higher EM–BSIT association at higher levels of pathology.

Discussion

BSIT and EM fluctuate together over measurement occasions, particularly for individuals with AD pathology. Repeated intraindividual measurements provide information that could lead to early detection and inexpensive monitoring of accumulating AD pathology.

Odor identification deficits are associated with cognitive decline (Hedner, Larsson, Arnold, Zucco, & Hummel, 2010; Roalf et al., 2017; Schubert et al., 2008), Alzheimer’s disease (AD) pathology (Lafaille-Magnan et al., 2017; Wilson, Arnold, Schneider, Tang, & Bennett, 2007), neurodegenerative diseases (Albers, Tabert, & Devanand, 2006; Attems, Walker, & Jellinger, 2014; Mesholam, Moberg, Mahr, & Doty, 1998), motor function (Tian, Resnick, & Studenski, 2016), and death (Pinto, Wroblewski, Kern, Schumm, & McClintock, 2014; Schubert et al., 2017; Wilson, Yu, & Bennett, 2011). Olfactory tests are simple and quick to administer and the link between smell and cognition has proven useful for detecting neurodegenerative diseases. Individuals with lower cognitive test scores tend to have lower olfactory performance scores (Hedner et al., 2010) and vice versa (Albers et al., 2006; Schubert et al., 2008; Wilson & Schneider, et al., 2007). For example, those who have been diagnosed with AD have exceptionally poor olfaction (Doty, Reyes, & Gregor, 1987; Masurkar & Devanand, 2014; Mesholam et al., 1998) and declining olfactory identification performance may signal the early stages of developing pathology (Albers et al., 2006; Devanand et al., 2015; Wilson et al., 2009; Wilson & Arnold, et al., 2007), making olfactory tests useful as a preclinical screening tool. The underlying theory for the concomitant decline of olfaction that accompanies AD is that the earliest stages of AD pathology damage the olfactory pathway and associated olfactory processing brain regions (Jucker & Walker, 2011; Kovács, 2013; Kovács, Cairns, & Lantos, 2001; Masurkar & Devanand, 2014; Wilson et al., 2007).

Neurofibrillary tangles are thought to progress in a specific pattern triggered by cell-to-cell transmission (Braak & Braak, 1991; Thal, Rüb, Orantes, & Braak, 2002). The earliest stages of AD, Braak Stages I and II, are defined by the presence of neurofibrillary tangles in the locus coeruleus and transentorhinal region of the brain (Braak & Braak, 1991; Braak et al., 2003). Both of these brain areas are involved in olfaction. The locus coeruleus facilitates olfactory learning (Aston-Jones & Cohen, 2005; Benarroch, 2009), and the entorhinal cortex, part of the primary olfactory cortex, receives direct input from the olfactory bulb (Jucker & Walker, 2011; Shepherd, 2007). The progression of neuritic plaques is less predictable than that of neurofibrillary tangles (Serrano-Pozo, Frosch, Masliah, & Hyman, 2011), but also affects main and secondary olfactory brain areas early on (Jucker & Walker, 2011; Seubert, Freiherr, Djordjevic, & Lundström, 2013).

Other neurodegenerative diseases may have a shared pathogenic etiology in that they also damage the olfactory pathway, thus also causing olfactory decline. For example, Parkinson’s disease (PD) progresses from the brainstem upward to the cerebral cortex with the earliest lesions occurring in the olfactory bulb and the medulla oblongata (Braak et al., 2003; Jucker & Walker, 2011). Current odor identification tests are not specific to identifying preclinical AD, as olfaction can be abnormal in individuals with PD (Doty, 2012; Haehner et al., 2007; Ross et al., 2008) as well as other neurodegenerative diseases with pathology that damages olfactory areas. However, AD tends to impact the higher order olfactory functions more strongly, whereas PD tends to impact olfactory tasks independently of their cognitive loads (Rahayel, Frasnelli, & Joubert, 2012).

Previous research has found that low olfactory identification scores are associated with increased risk of mild cognitive impairment (MCI; Devanand et al., 2000; Wilson & Schneider, et al., 2007), predict conversion from MCI to AD (Devanand et al., 2000), and are related to an increased level of AD pathology after autopsy (Lafaille-Magnan et al., 2017; Wilson & Arnold, et al., 2007). This study uses Rush Memory and Aging Project data to examine two specific aims. The first aim is to examine the relationship between olfactory identification scores, measured using the Brief Smell Identification Test (BSIT), AD pathology, and episodic memory baseline performance and rate of change. In addition, a growth model for episodic memory with BSIT as time-varying covariate (TVC) was estimated to examine whether olfaction and cognition vary together over measurement occasions. It is expected that low average BSIT scores and high AD pathology will be related to low episodic memory (EM) scores and that at each measurement occasion olfactory identification scores will fluctuate in the same direction as episodic memory scores.

Methods

Database Description and Study Sample

Data used in this study were from the Rush Memory and Aging Project (MAP), a longitudinal study of common chronic conditions in old age (Bennett et al., 2005). Beginning in 1997, participants have been recruited from retirement and senior housing communities, as well as through Church groups and social service in the Chicago area. They participated in annual clinical evaluations, including episodic memory testing, for up to 18 years, with brain donation at death. Olfaction was first measured in 2000, and then annually beginning in 2011. While 1,853 individuals have participated in the study, the focus of this analysis was on the 803 with autopsy data. Of these, 238 were excluded for missing BSIT at baseline, leaving 565 participants in the analysis sample with up to 15 years of follow-up. Participants had an average of 6 years of cognitive follow-up and 3 years of BSIT follow-up. At baseline, the mean age of participants was 83 years, 67% were women, 25% had one or more apolipoprotein ε4 allele, mean BSIT score was 8.08 (standard deviation, SD = 2.5), and 2% were members of a racial or ethnic minority group (Table 1). At death, 16% had had a stroke and 37% of participants were diagnosed with dementia (34% had no dementia and 29% had MCI). Of those both who were anosmic (defined as a score of 5 or less on the BSIT) and who had one or more ε4 allele, 70% had a dementia diagnosis at death (62% for those with just an ε4 allele). Of those who had both normal smell and no ε4 allele, 22% had a dementia diagnosis at death (33% for those with just one ε4 allele).

Table 1.

Participant Characteristics by BSIT Group at Baseline

CharacteristicAll
BSIT 1–12
(N = 565)
Normal
BSIT 11–12
(n = 78)
Hyposmic
BSIT 6–10
(n = 370)
Anosmic
BSIT 0–5
(n = 81)
Sex, female %67%76%71%53%
Age, years, mean (SD)83 (6)82 (5.8)83 (6.1)85 (5.8)
Age at death, years, mean (SD)89.4 (6.2)88.8 (6.7)89 (6.5)89.7 (5.6)
APOE ε4 allele, N (%)190 (25%)14 (18%)81 (22%)31 (38%)
Memory domain episodic, mean z-score (SD)−0.25 (0.9)0.12 (0.6)−0.13 (0.7)−0.87 (0.9)
Education, years, mean (SD)14.34 (3)13.68 (2.5)14.2 (2.9)14.89 (3.7)
AD pathology, N (%)422 (64%)35 (52%)191 (62%)48 (74%)
Dementia at death (no dementia %)37% (34%)19% (51%)34% (38%)68% (11%)
CharacteristicAll
BSIT 1–12
(N = 565)
Normal
BSIT 11–12
(n = 78)
Hyposmic
BSIT 6–10
(n = 370)
Anosmic
BSIT 0–5
(n = 81)
Sex, female %67%76%71%53%
Age, years, mean (SD)83 (6)82 (5.8)83 (6.1)85 (5.8)
Age at death, years, mean (SD)89.4 (6.2)88.8 (6.7)89 (6.5)89.7 (5.6)
APOE ε4 allele, N (%)190 (25%)14 (18%)81 (22%)31 (38%)
Memory domain episodic, mean z-score (SD)−0.25 (0.9)0.12 (0.6)−0.13 (0.7)−0.87 (0.9)
Education, years, mean (SD)14.34 (3)13.68 (2.5)14.2 (2.9)14.89 (3.7)
AD pathology, N (%)422 (64%)35 (52%)191 (62%)48 (74%)
Dementia at death (no dementia %)37% (34%)19% (51%)34% (38%)68% (11%)

Note. BSIT = Brief Smell Identification Test; APOE ε4 = apolipoprotein ε4; AD = Alzheimer’s disease; SD = standard deviation.

Table 1.

Participant Characteristics by BSIT Group at Baseline

CharacteristicAll
BSIT 1–12
(N = 565)
Normal
BSIT 11–12
(n = 78)
Hyposmic
BSIT 6–10
(n = 370)
Anosmic
BSIT 0–5
(n = 81)
Sex, female %67%76%71%53%
Age, years, mean (SD)83 (6)82 (5.8)83 (6.1)85 (5.8)
Age at death, years, mean (SD)89.4 (6.2)88.8 (6.7)89 (6.5)89.7 (5.6)
APOE ε4 allele, N (%)190 (25%)14 (18%)81 (22%)31 (38%)
Memory domain episodic, mean z-score (SD)−0.25 (0.9)0.12 (0.6)−0.13 (0.7)−0.87 (0.9)
Education, years, mean (SD)14.34 (3)13.68 (2.5)14.2 (2.9)14.89 (3.7)
AD pathology, N (%)422 (64%)35 (52%)191 (62%)48 (74%)
Dementia at death (no dementia %)37% (34%)19% (51%)34% (38%)68% (11%)
CharacteristicAll
BSIT 1–12
(N = 565)
Normal
BSIT 11–12
(n = 78)
Hyposmic
BSIT 6–10
(n = 370)
Anosmic
BSIT 0–5
(n = 81)
Sex, female %67%76%71%53%
Age, years, mean (SD)83 (6)82 (5.8)83 (6.1)85 (5.8)
Age at death, years, mean (SD)89.4 (6.2)88.8 (6.7)89 (6.5)89.7 (5.6)
APOE ε4 allele, N (%)190 (25%)14 (18%)81 (22%)31 (38%)
Memory domain episodic, mean z-score (SD)−0.25 (0.9)0.12 (0.6)−0.13 (0.7)−0.87 (0.9)
Education, years, mean (SD)14.34 (3)13.68 (2.5)14.2 (2.9)14.89 (3.7)
AD pathology, N (%)422 (64%)35 (52%)191 (62%)48 (74%)
Dementia at death (no dementia %)37% (34%)19% (51%)34% (38%)68% (11%)

Note. BSIT = Brief Smell Identification Test; APOE ε4 = apolipoprotein ε4; AD = Alzheimer’s disease; SD = standard deviation.

Measures

East Boston delayed and immediate recall

A three-sentence story is read and participants are asked to recall it immediately (East Boston immediate recall) and after a 3-minute distraction (East Boston delayed recall). The number of units recalled is scored out of 12.

Logical Memory I (immediate recall) and II (delayed recall)

Logical Memory I and II are measures from the Wechsler Memory Scale. After a brief story is read, participants are asked to retell it immediately (I) and after a 30-minute delay (II). Both tests are scored as the number of story units recalled out of 25.

CERAD word list immediate (three-trial), delayed, and recognition

Word list immediate (three trial) is a measure from the Consortium to Establish a Registry for AD CERAD neuropsychological assessment battery. A 10-word list study-recall sequence is administered 3 times for a total of 30 words. The total score is out of 30, the number of words recalled over all three trials.

In word list delayed, participants are asked to recall the same 10 words that were presented in the previous immediate recall measure after a several minute delay. The number of words recalled out of 10 is scored.

In word list recognition, 10 sets of four words are presented. Participants select the word from each set that they recognize from the previous trials (immediate and delayed) among the distractor words. The test is scored out of 10.

Episodic memory composite

Raw scores from seven tests (East Boston immediate recall, East Boston delayed recall, Logical memory I (immediate recall), Logical memory II (delayed recall), CERAD word list immediate (three trials), delayed, and recognition) were converted to z-scores and averaged to yield an episodic memory composite score (for further detail, see Bennett et al. 2005) (Bennett et al., 2005).

Brief smell identification test

BSIT was used to assess odor identification (Doty, Marcus, & Lee, 1996). The BSIT is the 12-item version of the 40-item University of Pennsylvania Smell Identification Test (UPSIT; Doty, Shaman, Kimmelman, & Dann, 1984). The BSIT test is in booklet format with one encapsulated scent and four answer choices per page. Each scent microcapsule contains a culturally familiar odor, which is scratched with a pencil and placed under the nose of the participant. The participant then chooses one of the four options provided in the booklet and the score is the number of odors correctly recognized. Keeping with standard practice, missing responses are assigned a score of 0.25, to a maximum of two missing answers. If more than two responses are missing, the entire test is treated as missing (see Wilson & Schneider, et al., 2007, for further details). Performance on the 12-item BSIT has been shown to correspond to the 40-item UPSIT from which it was derived, and has a test–retest value of r = 0.71 (Doty et al., 1996).

Person-mean BSIT.

The person-mean BSIT variable (BSIT_PM) is a person-mean-centered variable (also known as group-mean-centering) representing each individual’s own personal mean for BSIT score over all occasions on which the test was administered to them. It is modeled as a between-person predictor in the growth model for episodic memory in order to account for between-person differences in olfaction. In the model, the parameter estimate for regression of episodic memory on this variable describes whether people with better BSIT (overall) score better on episodic memory tests (i.e., the episodic memory composite). Its interaction with time is also estimated in the model, to account for whether people with lower average BSIT scores decline more rapidly over time.

BSIT as a TVC

BSIT as a TVC (BSIT_TVC) is coded as BSIT score at each occasion minus each person’s own individual person-mean score (BSIT_PM). BSIT_TVC is a fixed, within-person variable and, along with time, acts as an index of how a person’s episodic memory score changes; in this case, how it changes with respect to changes in BSIT, while accounting for changes associated with time.

AD pathology

Using National Institute on Aging (NIA) Reagan Pathology criteria (Bennett et al., 2006), a neuropathological evaluation determined the level of AD pathology on a 4-point scale based on both neurofibrillary tangles and neuritic plaques (Wilson & Arnold, et al., 2007). Individuals with intermediate or high scores were coded as high AD pathology and those with low and no pathology were coded as no AD pathology, creating a binary variable for use in all analyses to account for pathologic diagnosis of AD.

Apolipoprotein ε4 allele

Individuals with one or more apolipoprotein (APOE) ε4 allele were considered to have an ε4 allele and those with none were not considered to have an ε4 allele (Yu et al., 2017).

Statistical Analysis

Statistical analyses were performed using version 3.3.1 of the R Statistical Software Package (Rdevelopment, 2013) and version 1.0.136 of R Studio (R Studio Team, 2015). Model estimation was conducted with R package lme4 (Bates, Mächler, Bolker, & Walker, 2014), and graphs and tables were produced using ggplot2 (Wickham, 2009) and sjPlot (Lüdecke, 2017), respectively. Marginal R2 (Rm2) describes the proportion of variance explained by the fixed factors alone, whereas conditional R2 (Rc2) describes the variance of both the fixed and random factors in the model; marginal and conditional effect sizes were produced using piecewiseSEM (Lefcheck, 2015).

Growth model for episodic memory with BSIT as TVC

In order to examine within-person coupled variation in episodic memory and olfaction, a growth model for episodic memory with BSIT as TVC was estimated with terms for sex, education, ε4 allele, and AD pathology. To model BSIT as a time-varying predictor the original variable is split into two: the between-person effect, represented by an individual’s observed mean across the study (person-mean BSIT), and the within-person effect, represented by an individual’s deviation from their person-mean across the study (raw BSIT score − person-mean BSIT), which we then call time-varying BSIT (Hoffman & Stawski, 2009). From this, we determined whether assessment-to-assessment variation in episodic memory at the level of the individual is correlated with individual level variation in olfaction over the same assessment occasions (Hoffman, 2015; Muniz-Terrera et al., 2016; Sliwinski & Mogle, 2008). In Model 1, the individual person-mean of BSIT (BSIT_PM) represents overall between-person differences in olfaction, and this average BSIT was subtracted from the raw score at each wave to provide a within-person change effect (BSIT_TVC). Terms for sex and education did not predict rate of change so were dropped from the final model.

(Model 1)

Here, Level 1 equation depicts change in episodic memory performance for a given individual (i) on a given measurement occasion (j), and specifies an intercept (β0i), effects of linear time (β1i) and linear within-person change in BSIT (β2i; BSIT_TVC), and the within-person residual (eij).

Level 2 specifies fixed effects for the common intercept (γ00), the between-person component of BSIT (γ01, person-mean BSIT values), fixed effect slopes for linear time (γ10) and coupled within-person variation between BSIT and episodic memory (γ20; BSIT_TVC). Person-specific deviations (random effects) from the fixed intercept (U0i) and fixed linear effect of time (U1i) are also included. Covariates for age at baseline (centered at the mean of 78), biological sex (female as reference), APOE status (ε4 allele or no ε4 allele), and education (centered at the mean of 14) were included.

TVC with interaction

Given that less BSIT and memory fluctuation was expected for individuals not diagnosed with dementia, Model 1.1 included an interaction between TVC for BSIT (BSIT_TVC) and AD pathology to evaluate whether TVC for BSIT differed by AD pathology group.

Results

Correlations

Poorer BSIT scores were associated with older age (r = −.16, p < .001), and lower episodic memory composite scores (r = .46, p < .001), but not sex or education.

Growth Model for Episodic Memory With BSIT as TVC

Between-person variation in odor identification was significantly associated with episodic memory (b = .13, SE = 0.014, p < .001; Model 1, Table 2). For every unit more in person-mean BSIT (BSIT_PM), episodic memory at baseline was also higher by 0.13. High AD pathology was related to lower episodic memory at baseline (b = −.24, SE = 0.07, p < .001).

Table 2.

Time-Varying Covariate Model Results

Model 1Model 1.1
B (CI)pB (CI)p
Fixed effects
 Intercept−0.80 (−1.06 to −0.53)<.001−0.79 (−1.05 to −0.52)<.001
 Time in study−0.08 (−0.15 to −0.01).033−0.08 (−0.16 to −0.01).024
 Age−0.02 (−0.03 to −0.01)<.001−0.02 (−0.03 to −0.01)<.001
 Sex−0.20 (−0.34 to −0.07).003−0.20 (−0.34 to −0.07).003
 Education level0.05 (0.03–0.07)<.0010.05 (0.03–0.07)<.001
 ε4 allele−0.28 (−0.43 to −0.13)<.001−0.28 (−0.43 to −0.13)<.001
 AD pathology−0.24 (−0.37 to −0.10)<.001−0.25 (−0.39 to −0.12)<.001
 BSIT_PM0.13 (0.10–0.16)<.0010.13 (0.10–0.16)<.001
 BSIT_TVC0.07 (0.04–0.10)<.0010.02 (−0.03–0.08).464
 Time:age0.00 (−0.00–0.00).9100.00 (−0.00–0.00).842
 Time:ε4−0.04 (−0.08 to −0.00).030−0.04 (−0.08 to −0.00).039
 Time:AD pathology−0.06 (−0.09 to −0.02).001−0.05 (−0.08 to −0.01).006
 Time:BSIT_PM0.01 (0.00–0.02).0090.01 (0.00–0.02).009
 AD pathology:BSIT_TVC0.08 (0.01–0.14).028
Random effects
σ20.1660.165
τ00, intercept0.3640.362
τ1, slope0.0040.004
N565565
 ICC0.6820.682
 Observations892892
Rm2 / Rc2.39 / .81.39 / .81
 AIC1864.0951861.229
 Deviance1832.0951827.229
Model 1Model 1.1
B (CI)pB (CI)p
Fixed effects
 Intercept−0.80 (−1.06 to −0.53)<.001−0.79 (−1.05 to −0.52)<.001
 Time in study−0.08 (−0.15 to −0.01).033−0.08 (−0.16 to −0.01).024
 Age−0.02 (−0.03 to −0.01)<.001−0.02 (−0.03 to −0.01)<.001
 Sex−0.20 (−0.34 to −0.07).003−0.20 (−0.34 to −0.07).003
 Education level0.05 (0.03–0.07)<.0010.05 (0.03–0.07)<.001
 ε4 allele−0.28 (−0.43 to −0.13)<.001−0.28 (−0.43 to −0.13)<.001
 AD pathology−0.24 (−0.37 to −0.10)<.001−0.25 (−0.39 to −0.12)<.001
 BSIT_PM0.13 (0.10–0.16)<.0010.13 (0.10–0.16)<.001
 BSIT_TVC0.07 (0.04–0.10)<.0010.02 (−0.03–0.08).464
 Time:age0.00 (−0.00–0.00).9100.00 (−0.00–0.00).842
 Time:ε4−0.04 (−0.08 to −0.00).030−0.04 (−0.08 to −0.00).039
 Time:AD pathology−0.06 (−0.09 to −0.02).001−0.05 (−0.08 to −0.01).006
 Time:BSIT_PM0.01 (0.00–0.02).0090.01 (0.00–0.02).009
 AD pathology:BSIT_TVC0.08 (0.01–0.14).028
Random effects
σ20.1660.165
τ00, intercept0.3640.362
τ1, slope0.0040.004
N565565
 ICC0.6820.682
 Observations892892
Rm2 / Rc2.39 / .81.39 / .81
 AIC1864.0951861.229
 Deviance1832.0951827.229

Notes. ε4 = apolipoprotein ε4; BSIT = Brief Smell Identification Test; AD = Alzheimer’s disease; BSIT_PM = BSIT person mean (each individuals own personal mean); BSIT_TVC = BSIT time-varying covariate; σ2 = the residual variance, τ00 = the variance for the intercept, τ1 = the variance term for slope; N = participants in the model; ICC = intraclass correlation; Rm2 = marginal R2; Rc2 = conditional R2; AIC = Akaike information criterion. The bolded values are the significance values for the preceding column “B (CI)”.

Table 2.

Time-Varying Covariate Model Results

Model 1Model 1.1
B (CI)pB (CI)p
Fixed effects
 Intercept−0.80 (−1.06 to −0.53)<.001−0.79 (−1.05 to −0.52)<.001
 Time in study−0.08 (−0.15 to −0.01).033−0.08 (−0.16 to −0.01).024
 Age−0.02 (−0.03 to −0.01)<.001−0.02 (−0.03 to −0.01)<.001
 Sex−0.20 (−0.34 to −0.07).003−0.20 (−0.34 to −0.07).003
 Education level0.05 (0.03–0.07)<.0010.05 (0.03–0.07)<.001
 ε4 allele−0.28 (−0.43 to −0.13)<.001−0.28 (−0.43 to −0.13)<.001
 AD pathology−0.24 (−0.37 to −0.10)<.001−0.25 (−0.39 to −0.12)<.001
 BSIT_PM0.13 (0.10–0.16)<.0010.13 (0.10–0.16)<.001
 BSIT_TVC0.07 (0.04–0.10)<.0010.02 (−0.03–0.08).464
 Time:age0.00 (−0.00–0.00).9100.00 (−0.00–0.00).842
 Time:ε4−0.04 (−0.08 to −0.00).030−0.04 (−0.08 to −0.00).039
 Time:AD pathology−0.06 (−0.09 to −0.02).001−0.05 (−0.08 to −0.01).006
 Time:BSIT_PM0.01 (0.00–0.02).0090.01 (0.00–0.02).009
 AD pathology:BSIT_TVC0.08 (0.01–0.14).028
Random effects
σ20.1660.165
τ00, intercept0.3640.362
τ1, slope0.0040.004
N565565
 ICC0.6820.682
 Observations892892
Rm2 / Rc2.39 / .81.39 / .81
 AIC1864.0951861.229
 Deviance1832.0951827.229
Model 1Model 1.1
B (CI)pB (CI)p
Fixed effects
 Intercept−0.80 (−1.06 to −0.53)<.001−0.79 (−1.05 to −0.52)<.001
 Time in study−0.08 (−0.15 to −0.01).033−0.08 (−0.16 to −0.01).024
 Age−0.02 (−0.03 to −0.01)<.001−0.02 (−0.03 to −0.01)<.001
 Sex−0.20 (−0.34 to −0.07).003−0.20 (−0.34 to −0.07).003
 Education level0.05 (0.03–0.07)<.0010.05 (0.03–0.07)<.001
 ε4 allele−0.28 (−0.43 to −0.13)<.001−0.28 (−0.43 to −0.13)<.001
 AD pathology−0.24 (−0.37 to −0.10)<.001−0.25 (−0.39 to −0.12)<.001
 BSIT_PM0.13 (0.10–0.16)<.0010.13 (0.10–0.16)<.001
 BSIT_TVC0.07 (0.04–0.10)<.0010.02 (−0.03–0.08).464
 Time:age0.00 (−0.00–0.00).9100.00 (−0.00–0.00).842
 Time:ε4−0.04 (−0.08 to −0.00).030−0.04 (−0.08 to −0.00).039
 Time:AD pathology−0.06 (−0.09 to −0.02).001−0.05 (−0.08 to −0.01).006
 Time:BSIT_PM0.01 (0.00–0.02).0090.01 (0.00–0.02).009
 AD pathology:BSIT_TVC0.08 (0.01–0.14).028
Random effects
σ20.1660.165
τ00, intercept0.3640.362
τ1, slope0.0040.004
N565565
 ICC0.6820.682
 Observations892892
Rm2 / Rc2.39 / .81.39 / .81
 AIC1864.0951861.229
 Deviance1832.0951827.229

Notes. ε4 = apolipoprotein ε4; BSIT = Brief Smell Identification Test; AD = Alzheimer’s disease; BSIT_PM = BSIT person mean (each individuals own personal mean); BSIT_TVC = BSIT time-varying covariate; σ2 = the residual variance, τ00 = the variance for the intercept, τ1 = the variance term for slope; N = participants in the model; ICC = intraclass correlation; Rm2 = marginal R2; Rc2 = conditional R2; AIC = Akaike information criterion. The bolded values are the significance values for the preceding column “B (CI)”.

Higher person-mean BSIT scores (b = .01, SE = 0.004, p = .009) also predicted slower declines in episodic memory. High AD pathology (b = −.06, SE = 0.02, p = .001) was associated with more rapid declines in episodic memory.

The within-person coupling effect represents the relative within-person effect of odor identification variation on variation in episodic memory. There was a robust positive association between the TVC of BSIT (BSIT_TVC = raw BSIT − BSIT_PM) and fluctuations in episodic memory (b = .07, SE = 0.02, p < .001). For every unit increase in BSIT relative to each person’s own mean BSIT score, episodic memory, for that same occasion, is expected to increase by 0.07 units.

TVC With Interaction

A graph of the association between time-specific fluctuations for episodic memory and BSIT_TVC, stratified by AD pathology group (Figure 1), shows the extent to which the two variables fluctuate relative to one another at the within-person level, separately for low and high AD pathology. At each measurement occasion, when an individual has a BSIT score that is lower (or higher) than their own personal average score, their episodic memory score is also correspondingly lower (or higher) than their own average episodic memory score. The figure also reveals an interaction between BSIT_TVC and AD pathology, indicating that the association is stronger for those with intermediate to high AD pathology. Specifically, occasion-to-occasion fluctuations between memory and BSIT are more closely associated in the higher AD pathology group. Model 1.1, identical to Model 1, but with the addition of an interaction between BSIT_TVC and AD pathology, was significant (b = .08, SE = 0.034, p = .028; Table 2) and supports the association seen in the graph.

Association between time-specific fluctuations in episodic memory and time-specific fluctuations in BSIT_TVC stratified by Alzheimer’s pathology group. BSIT = Brief Smell Identification Test; TVC = time-varying covariate.
Figure 1.

Association between time-specific fluctuations in episodic memory and time-specific fluctuations in BSIT_TVC stratified by Alzheimer’s pathology group. BSIT = Brief Smell Identification Test; TVC = time-varying covariate.

Discussion

Current theory suggests that odor identification losses reflect accumulating brain pathology that affect regions supporting both olfaction and episodic memory (Attems et al., 2014; Jucker & Walker, 2011; Kovács et al., 2001; Kovács, 2013; Wilson et al., 2007). This study found that individuals with more AD pathology at death had lower episodic memory scores at baseline, faster memory declines, lower baseline olfactory scores, and faster olfactory declines.

To the best of our knowledge, this is the first time a growth model for episodic memory with BSIT as TVC has been explored with AD pathology and it has provided two valuable insights: (a) episodic memory scores and BSIT scores fluctuate together over measurement occasion (Model 1, BSIT_TVC), and (b) there is a stronger positive relationship between BSIT and memory fluctuations for individuals with intermediate to high AD pathology (Figure 1; Model 1.1). The weaker association in the low AD pathology group (Figure 1) could be due to low variability in episodic memory scores, BSIT scores, or both, as the BSIT suffers from significant ceiling effects in individuals with normal smell (Freiherr et al., 2012). However, the greater association in those with higher AD pathology could also be related to the buildup of AD pathology in the brain. These results further support the idea that olfactory tests could act as an inexpensive and easy to administer bellwether indicating potential pathological brain disorders and cognitive decline as fluctuations in BSIT performance may provide an indication of developing AD pathology. Previous longitudinal studies have indicated that education, although related to level of cognition, is not related to rate of cognitive decline (Piccinin et al., 2012; Wilson et al., 2009). This study supports this, and found that education was a significant predictor of episodic memory at baseline but not rate of change, which could mean that although education is a risk factor for AD and associated with long-term differences in level of functioning, it may not protect against declines in old age (Piccinin et al., 2012; Wilson et al., 2009).

There are limitations when using a TVC model. Person-mean BSIT collapses across all occasions to provide each person’s own mean score (i.e., individuals who are declining in their BSIT scores will have lower person-mean), which may contribute to the significant association of the between-person BSIT_PM predictor with baseline memory. Existence of a trend in TVC, as in the BSIT here, can be problematic (Muniz-Terrera et al., 2016). Since BSIT itself is changing over time, BSIT_PM scores could end up being the same for someone with average but unchanging BSIT over time and someone with initially high BSIT who declines a lot over time. A follow-up study using a bivariate model is planned in order to clarify some of these questions.

AD pathology is measured at death (up to 1 year after the final occasion of measurement, and 1–15 years after the baseline visit) and there are no variables in this dataset to account for accumulating pathology over lifespan (e.g., MRI) nor measurements of brain volume, which have been shown to moderate cognitive decline (Bennett, Stark, & Stark, 2018; Erten-Lyons et al., 2009). The population from which the sample was selected has several homogeneous features and therefore generalizability may be affected: recruitment was from retirement facilities, participants agreed to brain donation at death, and there was a high level of education (M = 14.34).

Conclusions

High AD pathology and low person-mean BSIT scores are associated with more rapid declines in episodic memory. Episodic memory scores and BSIT scores fluctuate together over measurement occasions, and the association between time-specific fluctuations in episodic memory and BSIT differ by level of AD pathology. These findings suggest that olfactory identification and episodic memory may provide functional assessments of the brain areas that are at risk of degeneration in AD. This raises the possibility that repeated intraindividual measurements of olfactory identification could aid in the early detection of AD and provide an inexpensive method of monitoring accumulating AD pathology.

Funding

This work was supported by the National Institute on Aging of the National Institutes of Health (under award numbers P01AG043362 to A.M.P.; RF1AG15819 and R01AG17917 to D.A.B.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Acknowledgments

The authors are grateful to Gregory Klein for assistance with data information and to all of the participants in the project.

Conflicts of Interest

None.

References

Albers
,
M. W.
,
Tabert
,
M. H.
,
&
Devanand
,
D. P
. (
2006
).
Olfactory dysfunction as a predictor of neurodegenerative disease
.
Current Neurology and Neuroscience Reports
,
6
,
379
386
. doi:10.1007/s11910-996-0018-7

Aston-Jones
,
G.
, &
Cohen
,
J. D
. (
2005
).
An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance
.
Annual Review of Neuroscience
,
28
,
403
450
. doi:10.1146/annurev.neuro.28.061604.135709

Attems
,
J.
,
Walker
,
L.
, &
Jellinger
,
K. A
. (
2015
).
Olfactory bulb involvement in neurodegenerative diseases
.
Acta Neuropathologica
.
127
(
4
),
459
475
. doi:10.1007/s00401-014-1261-7

Bates
,
D.
,
Mächler
,
M.
,
Bolker
,
B.
, &
Walker
,
S
. (
2015
).
Fitting linear mixed-effects models using lme4
.
Journal of Statistical Software
.
67
(
1
),
1
48
. doi:http://dx.doi.org./10.18637/jss.v067.i01

Benarroch
,
E. E
. (
2009
).
The locus ceruleus norepinephrine system: Functional organization and potential clinical significance
.
Neurology
,
73
,
1699
1704
. doi:10.1212/WNL.0b013e3181c2937c

Bennett
,
D. A.
,
Schneider
,
J. A.
,
Arvanitakis
,
Z.
,
Kelly
,
J. F.
,
Aggarwal
,
N. T.
,
Shah
,
R. C.
,
&
Wilson
,
R. S
. (
2006
).
Neuropathology of older persons without cognitive impairment from two community-based studies
.
Neurology
,
66
,
1837
1844
. doi:10.1212/01.wnl.0000219668.47116.e6

Bennett
,
D. A.
,
Schneider
,
J. A.
,
Buchman
,
A. S.
,
Mendes de Leon
,
C.
,
Bienias
,
J. L.
,
&
Wilson
,
R. S
. (
2005
).
The rush memory and aging project: Study design and baseline characteristics of the study cohort
.
Neuroepidemiology
,
25
,
163
175
. doi:10.1159/000087446

Bennett
,
I. J.
,
Stark
,
S. M.
, &
Stark
,
C. E. L
. (
2018
).
Recognition memory dysfunction relates to hippocampal subfield volume: A study of cognitively normal and mildly impaired older adults
.
The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences
. doi:10.1093/geronb/gbx181

Braak
,
H.
, &
Braak
,
E
. (
1991
).
Neuropathological stageing of Alzheimer-related changes
.
Acta Neuropathologica
,
82
,
239
259
. doi:10.1007/BF00308809

Braak
,
H.
,
Del Tredici
,
K.
,
Rüb
,
U.
,
de Vos
,
R. A.
,
Jansen Steur
,
E. N.
,
&
Braak
,
E
. (
2003
).
Staging of brain pathology related to sporadic Parkinson’s disease
.
Neurobiology of Aging
,
24
,
197
211
. doi:10.1016/S0197-4580(02)00065-9

Devanand
,
D. P.
,
Lee
,
S.
,
Manly
,
J.
,
Andrews
,
H.
,
Schupf
,
N.
,
Doty
,
R. L.
,…
Mayeux
,
R
. (
2015
).
Olfactory deficits predict cognitive decline and Alzheimer dementia in an urban community
.
Neurology
,
84
,
182
189
. doi:10.1212/WNL. 0000000000001132

Devanand
,
D. P.
,
Michaels-Marston
,
K. S.
,
Liu
,
X.
,
Pelton
,
G. H.
,
Padilla
,
M.
,
Marder
,
K.
,…
Mayeux
,
R
. (
2000
).
Olfactory deficits in patients with mild cognitive impairment predict Alzheimer’s disease at follow-up
.
The American Journal of Psychiatry
,
157
,
1399
1405
. doi:10.1176/appi.ajp.157.9.1399

Doty
,
R. L
. (
2012
).
Olfaction in Parkinson’s disease and related disorders
.
Neurobiology of Disease
,
46
,
527
552
. doi:10.1016/j.nbd.2011.10.026

Doty
,
R. L.
,
Marcus
,
A.
,
&
Lee
,
W. W
. (
1996
).
Development of the 12-item Cross-Cultural Smell Identification Test (CC-SIT)
.
The Laryngoscope
,
106
,
353
356
. doi:10.1097/00005537- 199603000-00021

Doty
,
R. L.
,
Reyes
,
P. F.
,
&
Gregor
,
T
. (
1987
).
Presence of both odor identification and detection deficits in Alzheimer’s disease
.
Brain Research Bulletin
,
18
,
597
600
. doi:10.1016/0361-9230(87)90129-8

Doty
,
R. L.
,
Shaman
,
P.
,
Kimmelman
,
C. P.
,
&
Dann
,
M. S
. (
1984
).
University of Pennsylvania smell identification test: A rapid quantitative olfactory function test for the clinic
.
The Laryngoscope
,
94
(
2 Pt 1
),
176
178
.
doi:10.1288/00005537-198402000-00004

Erten-Lyons
,
D.
,
Woltjer
,
R. L.
,
Dodge
,
H.
,
Nixon
,
R.
,
Vorobik
,
R.
,
Calvert
,
J. F.
,…
Kaye
,
J
. (
2009
).
Factors associated with resistance to dementia despite high Alzheimer disease pathology
.
Neurology
,
72
,
354
360
. doi:10.1212/01.wnl.0000341273.18141.64

Freiherr
,
J.
,
Gordon
,
A. R.
,
Alden
,
E. C.
,
Ponting
,
A. L.
,
Hernandez
,
M. F.
,
Boesveldt
,
S.
,
&
Lundström
,
J. N
. (
2012
).
The 40-item Monell Extended Sniffin’ Sticks Identification Test (MONEX-40)
.
Journal of Neuroscience Methods
,
205
,
10
16
. doi:10.1016/j.jneumeth.2011.12.004

Haehner
,
A.
,
Hummel
,
T.
,
Hummel
,
C.
,
Sommer
,
U.
,
Junghanns
,
S.
,
&
Reichmann
,
H
. (
2007
).
Olfactory loss may be a first sign of idiopathic Parkinson’s disease
.
Movement Disorders: Official Journal of the Movement Disorder Society
,
22
,
839
842
. doi:10.1002/mds.21413

Hedner
,
M.
,
Larsson
,
M.
,
Arnold
,
N.
,
Zucco
,
G. M.
,
&
Hummel
,
T
. (
2010
).
Cognitive factors in odor detection, odor discrimination, and odor identification tasks
.
Journal of Clinical and Experimental Neuropsychology
,
32
,
1062
1067
. doi:10.1080/13803391003683070

Hoffman
,
L
. (
2015
).
Longitudinal analysis: Modeling within-person fluctuation and change
.
New York, NY:
Routledge Academic
.

Hoffman
,
L.
, &
Stawski
,
R. S
. (
2009
).
Persons as contexts: Evaluation between-person and within-person effects in longitudinal analysis
.
Research in Human Development
,
6
,
97
120
.
doi:10.1080/15427600902911189

Jucker
,
M.
, &
Walker
,
L. C
. (
2011
).
Pathogenic protein seeding in Alzheimer disease and other neurodegenerative disorders
.
Annals of Neurology
,
70
,
532
540
. doi:10.1002/ana.22615

Kovács
,
T
. (
2013
).
The olfactory system in Alzheimer’s disease: Pathology, pathophysiology and pathway for therapy
.
Translational Neuroscience
,
4
(
1
),
34
45
. doi:10.2478/s13380-013-0108-3

Kovács
,
T.
,
Cairns
,
N. J.
,
&
Lantos
,
P. L
. (
2001
).
Olfactory centres in Alzheimer’s disease: Olfactory bulb is involved in early Braak’s stages
.
Neuroreport
,
12
,
285
288
. doi:10.1097/00001756-200102120-00021

Lafaille-Magnan
,
M. E.
,
Poirier
,
J.
,
Etienne
,
P.
,
Tremblay-Mercier
,
J.
,
Frenette
,
J.
,
Rosa-Neto
,
P.
,…
Breitner
,
J. C. S
.;
PREVENT-AD Research Group
. (
2017
).
Odor identification as a biomarker of preclinical AD in older adults at risk
.
Neurology
,
89
,
327
335
. doi:10.1212/WNL.0000000000004159

Lefcheck
,
J. S
. (
2015
).
piecewiseSEM : Piecewise structural equation modeling in R for ecology, evolution, and systematics
.
Methods in Ecology and Evolution
,
7
(
5
),
573
579
. doi:10.1111/2041-210X.12512

Lüdecke
,
D
. (
2017
).
sjPlot: Data visualization for statistics in social science
R package version
,
2
(
0
).

Masurkar
,
A. V.
, &
Devanand
,
D. P
. (
2014
).
Olfactory dysfunction in the elderly: Basic circuitry and alterations with normal aging and Alzheimer’s disease
.
Current Geriatrics Reports
,
3
,
91
100
. doi:10.1007/s13670-014-0080-y

Mesholam
,
R. I.
,
Moberg
,
P. J.
,
Mahr
,
R. N.
,
&
Doty
,
R. L
. (
1998
).
Olfaction in neurodegenerative disease: A meta-analysis of olfactory functioning in Alzheimer’s and Parkinson’s diseases
.
Archives of Neurology
,
55
,
84
90
. doi:10.1159/000093759

Muniz-Terrera
,
G.
,
Robitaille
,
A.
,
Kelly
,
A.
,
Johansson
,
B.
,
Hofer
,
S.
,
&
Piccinin
,
A
. (
2016
).
Latent growth models matched to research questions to answer questions about dynamics of change in multiple processes
.
Journal of Clinical Epidemiology
,
82
,
158
166
. doi:10.1016/j.jclinepi.2016.09.001

Piccinin
,
A. M.
,
Muniz-Terrera
,
G.
,
Clouston
,
S.
,
Reynolds
,
C. A.
,
Thorvaldsson
,
V.
,
Deary
,
I. J.
,…
Hofer
,
S. M
. (
2012
).
Coordinated analysis of age, sex, and education effects on change in MMSE scores
.
The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences
,
68
,
374
390
. doi:10.1093/geronb/gbs077

Pinto
,
J. M.
,
Wroblewski
,
K. E.
,
Kern
,
D. W.
,
Schumm
,
L. P.
,
&
McClintock
,
M. K
. (
2014
).
Olfactory dysfunction predicts 5-year mortality in older adults
.
PLoS ONE
,
9
,
e107541
. doi:10.1371/journal.pone.0107541

Rahayel
,
S.
,
Frasnelli
,
J.
,
&
Joubert
,
S
. (
2012
).
The effect of Alzheimer’s disease and Parkinson’s disease on olfaction: A meta-analysis
.
Behavioural Brain Research
,
231
,
60
74
. doi:10.1016/j.bbr.2012.02.047

R Core Team
. (
2013
).
R: A language and environment for statistical computing
.
Vienna
:
R Foundation for Statistical Computing
. .

Roalf
,
D. R.
,
Moberg
,
M. J.
,
Turetsky
,
B. I.
,
Brennan
,
L.
,
Kabadi
,
S.
,
Wolk
,
D. A.
,
&
Moberg
,
P. J
. (
2017
).
A quantitative meta-analysis of olfactory dysfunction in mild cognitive impairment
.
Journal of Neurology, Neurosurgery, and Psychiatry
,
88
,
226
232
. doi:10.1136/jnnp-2016-314638

Ross
,
G. W.
,
Petrovitch
,
H.
,
Abbott
,
R. D.
,
Tanner
,
C. M.
,
Popper
,
J.
,
Masaki
,
K.
,…
White
,
L. R
. (
2008
).
Association of olfactory dysfunction with risk for future Parkinson’s disease
.
Annals of Neurology
,
63
,
167
173
. doi:10.1002/ana.21291

Schubert
,
C. R.
,
Carmichael
,
L. L.
,
Murphy
,
C.
,
Klein
,
B. E.
,
Klein
,
R.
,
&
Cruickshanks
,
K. J
. (
2008
).
Olfaction and the 5-year incidence of cognitive impairment in an epidemiological study of older adults
.
Journal of the American Geriatrics Society
,
56
,
1517
1521
. doi:10.1111/j.1532-5415.2008.01826.x

Schubert
,
C. R.
,
Fischer
,
M. E.
,
Pinto
,
A. A.
,
Klein
,
B. E. K.
,
Klein
,
R.
,
Tweed
,
T. S.
, &
Cruickshanks
,
K. J
. (
2016
).
Sensory impairments and risk of mortality in older adults
.
The Journals of Gerontology, Series A: Biological Sciences and Medical Sciences
,
72
(
5
),
710
715
. doi:10.1093/gerona/glw036

Serrano-Pozo
,
A.
,
Frosch
,
M. P.
,
Masliah
,
E.
,
&
Hyman
,
B. T
. (
2011
).
Neuropathological alterations in Alzheimer disease
.
Cold Spring Harbor Perspectives in Medicine
,
1
,
a006189
. doi:10.1101/cshperspect.a006189

Seubert
,
J.
,
Freiherr
,
J.
,
Djordjevic
,
J.
,
&
Lundström
,
J. N
. (
2013
).
Statistical localization of human olfactory cortex
.
NeuroImage
,
66
,
333
342
. doi:10.1016/j.neuroimage.2012.10.030

Shepherd
,
G. M
. (
2007
).
Perspectives on olfactory processing, conscious perception, and orbitofrontal cortex
.
Annals of the New York Academy of Sciences
,
1121
,
87
101
. doi:10.1196/annals.1401.032

Sliwinski
,
M.
, &
Mogle
,
J
. (
2008
).
Time-based and process-based approaches to analysis of longitudinal data
. In
S. M.
 
Hofer
&
D. F.
 
Alwin
(Eds.),
Handbook of cognitive aging: Interdisciplinary perspectives
(pp.
477
491
).
Thousand Oaks, CA
:
Sage Publications, Inc
.
doi:10.4135/9781412976589.n28

Team
,
R. S
. (
2015
).
RStudio: Integrated development for R
.
Boston, MA
:
RStudio, Inc
. Retrieved from http://www.Rstudio.com.

Thal
,
D. R.
,
Rüb
,
U.
,
Orantes
,
M.
,
&
Braak
,
H
. (
2002
).
Phases of A beta-deposition in the human brain and its relevance for the development of AD
.
Neurology
,
58
,
1791
1800
.
doi:10.1212/WNL.58.12.1791

Tian
,
Q.
,
Resnick
,
S. M.
, &
Studenski
,
S. A
. (
2016
).
Olfaction is related to motor function in older adults
.
The Journals of Gerontology, Series A: Biological Sciences and Medical Sciences
,
72
(
8
),
1067
1071
. doi:10.1093/gerona/glw222

Wickham
,
H
. (
2009
).
ggplot2: Elegant graphics for data analysis
.
New York
:
Springer
.

Wilson
,
R. S.
,
Arnold
,
S. E.
,
Schneider
,
J. A.
,
Boyle
,
P. A.
,
Buchman
,
A. S.
,
&
Bennett
,
D. A
. (
2009
).
Olfactory impairment in presymptomatic Alzheimer’s disease
.
Annals of the New York Academy of Sciences
,
1170
,
730
735
. doi:10.1111/j.1749-6632.2009.04013.x

Wilson
,
R. S.
,
Arnold
,
S. E.
,
Schneider
,
J. A.
,
Tang
,
Y.
,
&
Bennett
,
D. A
. (
2007
).
The relationship between cerebral Alzheimer’s disease pathology and odour identification in old age
.
Journal of Neurology, Neurosurgery, and Psychiatry
,
78
,
30
35
. doi:10.1136/jnnp.2006.099721

Wilson
,
R. S.
,
Hebert
,
L. E.
,
Scherr
,
P. A.
,
Barnes
,
L. L.
,
Mendes de Leon
,
C. F.
,
&
Evans
,
D. A
. (
2009
).
Educational attainment and cognitive decline in old age
.
Neurology
,
72
,
460
465
. doi:10.1212/01.wnl.0000341782.71418.6c

Wilson
,
R. S.
,
Schneider
,
J. A.
,
Arnold
,
S. E.
,
Tang
,
Y.
,
Boyle
,
P. A.
, &
Bennett
,
D. A
. (
2007
).
Olfactory identification and incidence of mild cognitive impairment in older age
.
Archives of General Psychiatry
,
64
(
7
),
802
808
. doi:10.1001/archpsyc.64.7.802

Wilson
,
R. S.
,
Yu
,
L.
, &
Bennett
,
D. A
. (
2011
).
Odor identification and mortality in old age
.
Chemical Senses
,
36
,
63
67
. doi:10.1093/chemse/bjq098

Yu
,
L.
,
Lutz
,
M. W.
,
Wilson
,
R. S.
,
Burns
,
D. K.
,
Roses
,
A. D.
,
Saunders
,
A. M.
,…
Bennett
,
D. A
. (
2017
).
APOE ε4-TOMM40 ‘523 haplotypes and the risk of Alzheimer’s disease in older Caucasian and African Americans
.
Plos One
,
12
,
e0180356
. doi:10.1371/journal.pone.0180356

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://dbpia.nl.go.kr/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
Decision Editor: Nicole D Anderson, PhD, CPsych
Nicole D Anderson, PhD, CPsych
Decision Editor
Search for other works by this author on: