Abstract

Background

To investigate the relative contributions of prenatal complications, perinatal characteristics, neonatal morbidities and socio-economic conditions on the occurrence of motor, sensory, cognitive, language and psychological disorders in a large longitudinal preterm infant population during the first 7 years after birth.

Methods

The study population comprised 4122 infants born at <35 weeks of gestation who were followed for an average of 74.0 months after birth. Developmental disorders, including motor, sensory, cognitive, language and psychological, were assessed at each follow-up visit from 18 months to 7 years of age. The investigated determinants included prenatal complications (prolonged rupture of membranes >24 hours, intrauterine growth restriction, preterm labour and maternal hypertension), perinatal characteristics (gender, multiple pregnancies, gestational age, birth weight, APGAR score and intubation or ventilation in the delivery room), neonatal complications (low weight gain during hospitalization, respiratory assistance, severe neurological anomalies, nosocomial infections) and socio-economic characteristics (socio-economic level, parental separation, urbanicity). Based on hazard ratios determined using a propensity score matching approach, population-attributable fractions (PAF) were calculated for each of the four types of determinants and for each developmental disorder.

Results

The percentages of motor, sensory, cognitive, language and psychological disorders were 17.0, 13.4, 29.1, 25.9 and 26.1%, respectively. The PAF for the perinatal characteristics were the highest and they were similar for the different developmental disorders considered (around 60%). For the neonatal and socio-economic determinants, the PAF varied according to the disorder, with contributions of up to 17% for motor and 27% for language disorders, respectively. Finally, prenatal complications had the lowest contributions (between 6 and 13%).

Conclusions

This study illustrates the heterogeneity of risk factors on the risk of developmental disorder in preterm infants. These results suggest the importance of considering both medical and psycho-social follow-ups of preterm infants and their families.

Key Messages

  • This study, conducted as part of a large longitudinal population-based cohort study, illustrates the heterogeneity of various developmental disorder risk factors in preterm infants.

  • The population-attributable fractions (PAF) for perinatal characteristics amounted to 60%, irrespective of the developmental disorder. By contrast, for neonatal and socio-economic determinants, the PAF varied according to the developmental disorder considered, with the highest contributions for motor and language, respectively. Finally, prenatal complications had the lowest contributions on developmental disorders.

  • Prevention of neonatal comorbidities should allow the incidence of motor and sensory disorders to be reduced by around 15%. Similarly, a favourable socio-economic environment should allow the incidence of language, cognitive and psychological disorders to be reduced by around 25%.

  • These results suggest the importance of considering both medical and psycho-social follow-ups of infants and families.

Introduction

Advances in neonatal care have led to an increase in the survival of very preterm children.1–3 However, developmental sequelae are still a major problem, with a high risk for a wide range of neurodevelopmental impairments, including neuromotor, neurosensory, cognitive and behavioural disorders.3–15 Whereas the rate of cerebral palsy has recently decreased to an overall rate of around 5–9%,3,16,17 global and fine-motor-skill impairments are frequent, with rates ranging from 14 to 50% for infants between 2 and 8 years of age.6,9,10,18 The frequencies of developmental disorders vary according to the criteria used for the assessment and the age of the infants. However, overall, the rates of cognitive disorders range from 12 to 32% for infants aged between 2 and 8 years.4,6,11,16,18,19 The rate of behavioural disorders, such as attention-deficit/hyperactivity disorder, emotional symptoms or peer problems, can reach 25% in preterm children between 5 and 13 years of age.15,20,21 Finally, language ability is reduced in school-aged preterm infants compared with term infants,22 with frequencies ranging from 22 to 38%.23–25

The determinants of developmental disorders in preterm infants are complex and highly multifactorial. These factors include prenatal complications, perinatal characteristics, neonatal morbidities and the socio-economic environment. In regard to perinatal characteristics, the lower the gestational age, the higher the risk of neurodevelopmental impairments,3,4,18,26 cognitive deficiency21 and behavioural disorders.27 Being male is a major risk factor for cerebral palsy.17 Furthermore, a low birth weight is associated with an increased risk of cognitive deficiency.19,21 Among neonatal morbidities, cerebral lesions are major risk factors for motor disabilities such as cerebral palsy, with a very high predictive value,17 as well as severe cognitive disorders.19 Bronchopulmonary dysplasias are associated with an increased risk of cognitive impairment.28 Low weight gain during hospitalization is associated with a higher risk of neurodevelopment disorders.29 Finally, neonatal infections (both early- and late-onset) are associated with a higher risk of cerebral palsy.16 In addition to perinatal and neonatal characteristics, the socio-economic environment also influences the risk of developmental disorders. A low socio-economic status increases the risk of cognitive disorders at 4 years of age19,26,30 and early language disorders (at around 2 years of age).31 Furthermore, living in an urban area is associated with lower cognitive neurodevelopment at 5 years of age.32

The relative contributions of prenatal, perinatal, neonatal and socio-economic determinants are likely to differ for each developmental disorder. Understanding the contributions of each of these determinants allows better prevention and/or targeting through early interventions to reduce the occurrence of developmental disorders. The objective of this study was to investigate the relative contributions of prenatal complications, perinatal characteristics, neonatal morbidities and socio-economic conditions on the occurrence of motor, sensory, cognitive, language and psychological disorders in a large longitudinal preterm infant population during the first 7 years after birth.

Methods

Study population

The study population comprised surviving preterm infants enrolled in the Loire Infant Follow-up Team (LIFT),33 born at less than 35 weeks of gestation between January 2003 and December 2008, and who had at least one follow-up visit between 1 and 7 years of age (Figure 1). The LIFT network includes 24 maternity clinics in the Pays de la Loire (PDL) region with the objective of screening for early clinical anomalies associated with preterm births and providing specifically tailored care. The follow-up consisted of standardized visits by trained physicians at 3, 6, 9, 18 and 24 months as well as at 3, 4, 5 and 7 years after the birth of the infant.

Flowchart of the inclusion.
Figure 1.

Flowchart of the inclusion.

Assessment of developmental disorders

Developmental disorders, including motor, sensory, cognitive, language and psychological impairments, were assessed at each follow-up visit from 18 months to 7 years of age. At the end of each visit, a medical summary was generated by gathering specific information from the referring paediatricians, psychologists, parents and teachers. A detailed description of the clinical diagnostic criteria and validated questionnaires that were used to assess developmental disorders according to the age of the infant is presented Supplementary Table 1, available as Supplementary data at IJE online.

Evaluations by the referring paediatrician

At each visit, the referring paediatricians performed a neurodevelopmental assessment, including physical examinations and a psychomotor sensory evaluation. Specific data regarding the diagnosis of cerebral palsy with an inability to walk independently, or milder signs, were recorded. Neuromotor assessments were based on Claudine Amiel-Tison’s recommendations and they comprised six sections including 35 items covering passive and active muscle tones, spontaneous motor activity and primary reflexes in particular.34 Visual and auditory disorders were also examined during the evaluation. When the children reached 4 years of age, the paediatricians completed the ERTL4 questionnaire (‘Epreuves de Repérage des Troubles du Language’), which is a validated questionnaire designed to screen and monitor language difficulties in French-speaking countries.35–37 The ERTL4 test is divided into four sections (E1 to E4) that assess the repetition of words, the repetition of sentences, adequate use of prepositions and the construction of sentences. Furthermore, at 18 and 24 months of age, the CHAT questionnaire was used to screen for autism disorders38,39 by assessment of whether the child had reached developmental milestones. Items include a report of behaviours such as taking an interest in other children, pointing and pretend play. Finally, at 5 and 7 years of age, the BREV questionnaire (‘Batterie Rapide d’Evaluation des Fonctions Cognitives’) was used to detect cognitive and language disorders.40 The BREV questionnaire was designed to explore oral language in both its expressive and receptive forms, and the main learning acquired in regard to reading, spelling and numeracy achievement.

Evaluations by specialized psychologists

Trained psychologists systematically performed a psychomotor evaluation at 2 years of age using the revised Brunet-Lézine (RBL) scale. The RBL scale evaluates four developmental areas: fine motor skills, social skills, language skills, posture and gross motor adaptation,41 thereby allowing four separate scores plus an overall developmental quotient (DQ) to be calculated. The RBL scale is routinely used in French-speaking countries.42

Parent-completed questionnaires

Two questionnaires completed by the parents were used between 18 months and 7 years of age. First, the Ages and Stages Questionnaire (ASQ) was completed at 18, 24, 36 and 48 months of age.43 This questionnaire comprises age-specific questions divided into five areas of child capacities: communication abilities, gross motor skills, fine motor skills, problem-solving abilities and personal-social skills. Second, the standardized Strengths and Difficulties Questionnaire (SDQ) was used at 4, 5 and 7 years of age to evaluate infant behaviour.44 Five subscales evaluated pro-social behaviour, hyperactivity, emotional symptoms, conduct and peer problems. An overall score was computed to evaluate whether an infant exhibits a significant problem.

Teacher-completed questionnaires

A last questionnaire, the Global School Adaptation (GSA) questionnaire, completed by the kindergarten teachers for children 5 and 7 years of age, was used to assess the children’s abilities and behaviour in the classroom.45,46 This questionnaire has been validated as a screening tool for adverse neurodevelopmental outcomes in preterm children.47 It comprises 20 items that examine five areas: language, transverse skills, socialization, motor skills and number processing.

Prenatal complications

Four prenatal complications were considered: (i) prolonged rupture of membranes >24 hours (yes vs no), (ii) intrauterine growth restriction (yes vs no), (iii) preterm labour (yes vs no) and (iv) maternal hypertension (yes vs no).

Perinatal characteristics

The perinatal data that were considered comprised (i) gender (male vs female), (ii) multiple pregnancies (yes vs no), (iii) the gestational age (GA) (24–31 vs 32–34 weeks of gestation), (iv) the birth weight (Z-scores <–1 vs ≥–1), (v) intubation or ventilation in the delivery room (yes vs no) and (vi) APGAR score (<7 vs ≥7). The birthweight Z-score was computed according to the Olsen standards.48

Neonatal morbidities

Four morbidities between birth and discharge were considered: (i) a low weight gain expressed as the difference between the Z-score of the weight at discharge and the Z-score of the weight at birth (<–1 vs ≥–1); (ii) respiratory assistance, including ventilation and continuous positive airway pressure (yes vs no); (iii) severe neurological anomalies assessed by ultrasound or magnetic resonance imaging, including a stage 3 or a stage 4 intraventricular haemorrhage, cerebral ventriculomegaly, periventricular leukomalacia or lesion of the basal ganglia (yes vs no); and (iv) nosocomial infection (i.e. administration of an antibiotherapy that lasted for more than 3 days) (yes vs no).

Socio-economic conditions

Three different dimensions of the socio-economic conditions were investigated: the socio-economic level, the parents’ relationship and the urbanicity of the residential municipality. These variables were considered at the first visit after birth (3-month visit). The socio-economic level was assessed based on the nature of the parents’ employment and the eligibility for social-security benefits for those with low incomes (yes vs no). Each parent’s employment was rated according to the official classification developed by the INSEE institute. If either parent’s job was highly rated, their socio-economic level was considered ‘high’; otherwise, it was considered ‘intermediate’. This categorization has been used in previous studies.49,50 Information regarding the parental relationship status was binary and rated either as the parents living together or the parents living separately. Finally, the residential municipality was considered to be either urban or rural, based on definitions developed by the INSEE institute that take into account the distance between buildings and the number of inhabitants.

Statistical analyses

The statistical analyses were performed in three steps. First, associations between prenatal complications, perinatal characteristics, neonatal morbidities, socio-economic conditions and the risk of developmental disorders were estimated separately for each type of disorder. Second, the corresponding population-attributable fractions (PAF) were calculated. Finally, the PAF of the four perinatal characteristics were added to yield an overall PAF of the perinatal characteristics. This was similarly done for the prenatal complications, neonatal morbidities and the socio-economic conditions. For the calculation of the PAF, continuous risk factors such as Z-score of birth weight were dichotomized. The choice of the cut-off values was based on visual inspection of smoothed functions estimated using generalized additive models. The PAF provide a useful way to express the burden of risk factors associated with the health outcome(s), taking into account both the prevalence of the risk factors and the strength of the associations between these risk factors and the outcome(s).

Associations between the prenatal complications, perinatal characteristics, neonatal morbidities and socio-economic conditions with the risk of developmental disorders were determined using survival analyses. A separate analysis was performed for each developmental disorder. The first occurrence of each developmental disorder was considered. To account for possible confusion, infants were matched by a propensity score. The factors considered for the estimations of the propensity scores are detailed in Supplementary Table 2, available as Supplementary data at IJE online. The factors to be considered in the propensity score were included based on the temporality between the different determinants. For example, to estimate the effect of the gestational age on the risk of motor disorders, the propensity score included prenatal complications, other perinatal characteristics and socio-economic conditions. In this case, the neonatal morbidities were not included, as they were posterior to birth and cannot therefore be considered as possible confounders. Then infants born at between 24 and 31 weeks were matched with infants born at between 32 and 34 weeks with a 1:1 ratio within a 0.2 caliper of standard deviation of the logit of the propensity score. A caliper width equal to 0.2 has been shown to have optimal properties in terms of bias reduction and resultant confidence intervals estimation.51 Covariate distributions between those exposed and those unexposed after matching were checked using standardized differences (SDs). Estimation of the effects of the different determinants on the risk of developmental disorders, expressed as hazard ratios (HR), was carried out using Cox models with a frailty variable to account for the matching between infants. To account for possible residual confounding after matching, the covariates used to estimate the propensity scores were added in the final models as adjustment variables.

The PAF were then calculated using the following formula:
(1)
where Pce is the proportion of cases exposed to risk factors and HR the adjusted hazard ratios from the Cox models.52 This formula produces an internally valid estimate when confounding exists. To calculate the overall PAF for prenatal complications, perinatal characteristic, neonatal morbidities and socio-economic conditions, only PAF corresponding to significant HR were added together within each type of determinant.

Ethics approval

Written consent was obtained for each patient before inclusion in the study and the cohort was registered with the French Data Protection Authority for clinical research (CNIL, No. 851117).

Results

Between January 2003 and December 2008, 4502 infants born at less than 35 weeks of gestation in the PDL region of France were enrolled in the LIFT cohort (Figure 1). The following infants were excluded from the study population: infants who had no follow-up visit between 12 and 84 months of age (n = 292), infants who did not reside in the PDL region (n = 78) and infants for whom there was no record of their birth weight (n = 10). Due to these exclusions, the study population comprised 4122 preterm infants. The overall median length of the follow-up period was 74.0 months (mean = 70 months, SD = 19 months).

The percentages of motor, sensory, cognitive, language and psychological disorders were 17.0, 13.4, 29.1, 25.9 and 26.1%, respectively, with median ages at first occurrences of 28.8, 37.8, 60.0, 38.1 and 60.5 months, respectively (Figure 2). In terms of the perinatal characteristics, 54.2% of the infants were male (n = 2235) and 35.7% were born at <32 weeks of gestation (n = 1472) (Table 1).

Table 1.

Description of the prenatal complications, perinatal characteristics, neonatal morbidities and socio-economic conditions of included and non-included infants

Infants included in the analysesInfants not included in the analysesP-value
4122380
Prenatal complications
 Premature rupture of membranes0.961
  No3600 (87.3)331 (87.1)
  Yes522 (12.7)49 (12.9)
 Intrauterine growth restriction0.015
  No3603 (87.4)349 (91.8)
  Yes519 (12.6)31 (8.2)
 Preterm labour0.322
  No2553 (61.9)225 (59.2)
  Yes1569 (38.1)155 (40.8)
 Maternal hypertension0.008
  No3543 (86)307 (80.8)
  Yes579 (14)73 (19.2)
Perinatal characteristics
 Gender0.258
  Female1887 (45.8)186 (48.9)
  Male2235 (54.2)194 (51.1)
 Multiple pregnancies<0.001
  No2574 (62.4)272 (71.6)
  Yes1548 (37.6)108 (28.4)
 Gestational age<0.001
  32–352650 (64.3)288 (75.8)
  24–311472 (35.7)92 (24.2)
 Z-score of birthweight0.062
  ≥–13101 (75.2)295 (79.7)
  <–11021 (24.8)75 (20.3)
 Intubation or ventilation0.001
  No2770 (67.2)287 (75.5)
  Yes1352 (32.8)93 (24.5)
 APGAR score0.387
  ≥73930 (95.3)358 (94.2)
  <7192 (4.7)22 (5.8)
Neonatal morbidities
 Low weight growth0.082
  No2488 (60.4)235 (61.8)
  Yes1019 (24.7)77 (20.3)
  Missing615 (14.9)68 (17.9)
 Respiratory assistance<0.001
  No2641 (64.1)285 (75)
  Yes1481 (35.9)95 (25)
 Severe neurological disorders0.527
  No3950 (95.8)361 (95)
  Yes172 (4.2)19 (5)
 Nosocomial infection0.031
  No3564 (86.5)344 (90.5)
  Yes558 (13.5)36 (9.5)
Socio-economic conditions
 Eligibility for social-security benefits0.776
  No3681 (89.3)337 (88.7)
  Yes441 (10.7)43 (11.3)
 Socio-economic level<0.001
  High1019 (24.7)36 (9.5)
  Intermediate3103 (75.3)344 (90.5)
 Parental separation0.07
  No3605 (87.5)345 (90.8)
  Yes517 (12.5)35 (9.2)
 Urbanicity0.5
  Urban1313 (31.9)128 (33.7)
  Rural2809 (68.1)252 (66.3)
Infants included in the analysesInfants not included in the analysesP-value
4122380
Prenatal complications
 Premature rupture of membranes0.961
  No3600 (87.3)331 (87.1)
  Yes522 (12.7)49 (12.9)
 Intrauterine growth restriction0.015
  No3603 (87.4)349 (91.8)
  Yes519 (12.6)31 (8.2)
 Preterm labour0.322
  No2553 (61.9)225 (59.2)
  Yes1569 (38.1)155 (40.8)
 Maternal hypertension0.008
  No3543 (86)307 (80.8)
  Yes579 (14)73 (19.2)
Perinatal characteristics
 Gender0.258
  Female1887 (45.8)186 (48.9)
  Male2235 (54.2)194 (51.1)
 Multiple pregnancies<0.001
  No2574 (62.4)272 (71.6)
  Yes1548 (37.6)108 (28.4)
 Gestational age<0.001
  32–352650 (64.3)288 (75.8)
  24–311472 (35.7)92 (24.2)
 Z-score of birthweight0.062
  ≥–13101 (75.2)295 (79.7)
  <–11021 (24.8)75 (20.3)
 Intubation or ventilation0.001
  No2770 (67.2)287 (75.5)
  Yes1352 (32.8)93 (24.5)
 APGAR score0.387
  ≥73930 (95.3)358 (94.2)
  <7192 (4.7)22 (5.8)
Neonatal morbidities
 Low weight growth0.082
  No2488 (60.4)235 (61.8)
  Yes1019 (24.7)77 (20.3)
  Missing615 (14.9)68 (17.9)
 Respiratory assistance<0.001
  No2641 (64.1)285 (75)
  Yes1481 (35.9)95 (25)
 Severe neurological disorders0.527
  No3950 (95.8)361 (95)
  Yes172 (4.2)19 (5)
 Nosocomial infection0.031
  No3564 (86.5)344 (90.5)
  Yes558 (13.5)36 (9.5)
Socio-economic conditions
 Eligibility for social-security benefits0.776
  No3681 (89.3)337 (88.7)
  Yes441 (10.7)43 (11.3)
 Socio-economic level<0.001
  High1019 (24.7)36 (9.5)
  Intermediate3103 (75.3)344 (90.5)
 Parental separation0.07
  No3605 (87.5)345 (90.8)
  Yes517 (12.5)35 (9.2)
 Urbanicity0.5
  Urban1313 (31.9)128 (33.7)
  Rural2809 (68.1)252 (66.3)
Table 1.

Description of the prenatal complications, perinatal characteristics, neonatal morbidities and socio-economic conditions of included and non-included infants

Infants included in the analysesInfants not included in the analysesP-value
4122380
Prenatal complications
 Premature rupture of membranes0.961
  No3600 (87.3)331 (87.1)
  Yes522 (12.7)49 (12.9)
 Intrauterine growth restriction0.015
  No3603 (87.4)349 (91.8)
  Yes519 (12.6)31 (8.2)
 Preterm labour0.322
  No2553 (61.9)225 (59.2)
  Yes1569 (38.1)155 (40.8)
 Maternal hypertension0.008
  No3543 (86)307 (80.8)
  Yes579 (14)73 (19.2)
Perinatal characteristics
 Gender0.258
  Female1887 (45.8)186 (48.9)
  Male2235 (54.2)194 (51.1)
 Multiple pregnancies<0.001
  No2574 (62.4)272 (71.6)
  Yes1548 (37.6)108 (28.4)
 Gestational age<0.001
  32–352650 (64.3)288 (75.8)
  24–311472 (35.7)92 (24.2)
 Z-score of birthweight0.062
  ≥–13101 (75.2)295 (79.7)
  <–11021 (24.8)75 (20.3)
 Intubation or ventilation0.001
  No2770 (67.2)287 (75.5)
  Yes1352 (32.8)93 (24.5)
 APGAR score0.387
  ≥73930 (95.3)358 (94.2)
  <7192 (4.7)22 (5.8)
Neonatal morbidities
 Low weight growth0.082
  No2488 (60.4)235 (61.8)
  Yes1019 (24.7)77 (20.3)
  Missing615 (14.9)68 (17.9)
 Respiratory assistance<0.001
  No2641 (64.1)285 (75)
  Yes1481 (35.9)95 (25)
 Severe neurological disorders0.527
  No3950 (95.8)361 (95)
  Yes172 (4.2)19 (5)
 Nosocomial infection0.031
  No3564 (86.5)344 (90.5)
  Yes558 (13.5)36 (9.5)
Socio-economic conditions
 Eligibility for social-security benefits0.776
  No3681 (89.3)337 (88.7)
  Yes441 (10.7)43 (11.3)
 Socio-economic level<0.001
  High1019 (24.7)36 (9.5)
  Intermediate3103 (75.3)344 (90.5)
 Parental separation0.07
  No3605 (87.5)345 (90.8)
  Yes517 (12.5)35 (9.2)
 Urbanicity0.5
  Urban1313 (31.9)128 (33.7)
  Rural2809 (68.1)252 (66.3)
Infants included in the analysesInfants not included in the analysesP-value
4122380
Prenatal complications
 Premature rupture of membranes0.961
  No3600 (87.3)331 (87.1)
  Yes522 (12.7)49 (12.9)
 Intrauterine growth restriction0.015
  No3603 (87.4)349 (91.8)
  Yes519 (12.6)31 (8.2)
 Preterm labour0.322
  No2553 (61.9)225 (59.2)
  Yes1569 (38.1)155 (40.8)
 Maternal hypertension0.008
  No3543 (86)307 (80.8)
  Yes579 (14)73 (19.2)
Perinatal characteristics
 Gender0.258
  Female1887 (45.8)186 (48.9)
  Male2235 (54.2)194 (51.1)
 Multiple pregnancies<0.001
  No2574 (62.4)272 (71.6)
  Yes1548 (37.6)108 (28.4)
 Gestational age<0.001
  32–352650 (64.3)288 (75.8)
  24–311472 (35.7)92 (24.2)
 Z-score of birthweight0.062
  ≥–13101 (75.2)295 (79.7)
  <–11021 (24.8)75 (20.3)
 Intubation or ventilation0.001
  No2770 (67.2)287 (75.5)
  Yes1352 (32.8)93 (24.5)
 APGAR score0.387
  ≥73930 (95.3)358 (94.2)
  <7192 (4.7)22 (5.8)
Neonatal morbidities
 Low weight growth0.082
  No2488 (60.4)235 (61.8)
  Yes1019 (24.7)77 (20.3)
  Missing615 (14.9)68 (17.9)
 Respiratory assistance<0.001
  No2641 (64.1)285 (75)
  Yes1481 (35.9)95 (25)
 Severe neurological disorders0.527
  No3950 (95.8)361 (95)
  Yes172 (4.2)19 (5)
 Nosocomial infection0.031
  No3564 (86.5)344 (90.5)
  Yes558 (13.5)36 (9.5)
Socio-economic conditions
 Eligibility for social-security benefits0.776
  No3681 (89.3)337 (88.7)
  Yes441 (10.7)43 (11.3)
 Socio-economic level<0.001
  High1019 (24.7)36 (9.5)
  Intermediate3103 (75.3)344 (90.5)
 Parental separation0.07
  No3605 (87.5)345 (90.8)
  Yes517 (12.5)35 (9.2)
 Urbanicity0.5
  Urban1313 (31.9)128 (33.7)
  Rural2809 (68.1)252 (66.3)

The first occurrence of developmental disorders up to 7 years of age.
Figure 2.

The first occurrence of developmental disorders up to 7 years of age.

Overall, having an intrauterine growth restriction, a preterm labour, being male, having a low gestational age or a low birth weight, being intubated or ventilated in the delivery room, having severe neurological anomalies or a nosocomial infection during the hospitalization, and being raised in a family with a low socio-economic level were associated with an increased risk of developmental disorders (Figure 3 and Supplementary Figure 1, available as Supplementary data at IJE online). SD were calculated to check for the covariate distributions before and after matching. For each developmental disorder, 176 SD were calculated (880 for the five developmental disorders) corresponding to all the covariates considered in the matching procedure. Out of these SD, 56 (6.4%) were higher than 0.1 and only 2 higher than 0.25 (both equal to 0.26).

The increased risk of developmental disorders, expressed as hazard ratios (HR) with their 95% confidence intervals, according to (A) the prenatal complications, (B) the perinatal characteristics, (C) the neonatal morbidities and (D) the socio-economic conditions. M, motor disorders; S, sensory disorders; C, cognitive disorders; L, language disorders; P, psychological disorders. HR were estimated using survival analyses (Cox model). To account for possible confusion, infants were matched with a 1:1 ratio within a 0.2 caliper of standard deviation of the logit of the propensity score (the variables included in the propensity scores are presented in Supplementary Table 2, available as Supplementary data at IJE online). The risk factors that were considered were as follows: prolonged rupture of membranes >24 hours (yes vs no), intrauterine growth restriction (yes vs no), preterm labour (yes vs no), maternal hypertension (yes vs no), gender (male vs female), multiple pregnancies (yes vs no), gestational age (GA) (24–31 vs 32–34 weeks of gestation), birth weight (Z-score <–1 vs ≥–1), intubation or ventilation in the delivery room (yes vs no), APGAR score (<7 vs ≥7), low weight gain (difference between the Z-score of the weight at discharge and the Z-score of the weight at birth <–1 vs ≥–1), respiratory assistance (yes vs no), severe neurological anomalies (yes vs no), nosocomial infection (yes vs no), socio-economic level (intermediate level vs high), eligibility for social-security benefits for those with low incomes (yes vs no), parental separation (parents living together or parents living separately) and residential municipality (urban vs rural).
Figure 3.

The increased risk of developmental disorders, expressed as hazard ratios (HR) with their 95% confidence intervals, according to (A) the prenatal complications, (B) the perinatal characteristics, (C) the neonatal morbidities and (D) the socio-economic conditions. M, motor disorders; S, sensory disorders; C, cognitive disorders; L, language disorders; P, psychological disorders. HR were estimated using survival analyses (Cox model). To account for possible confusion, infants were matched with a 1:1 ratio within a 0.2 caliper of standard deviation of the logit of the propensity score (the variables included in the propensity scores are presented in Supplementary Table 2, available as Supplementary data at IJE online). The risk factors that were considered were as follows: prolonged rupture of membranes >24 hours (yes vs no), intrauterine growth restriction (yes vs no), preterm labour (yes vs no), maternal hypertension (yes vs no), gender (male vs female), multiple pregnancies (yes vs no), gestational age (GA) (24–31 vs 32–34 weeks of gestation), birth weight (Z-score <–1 vs ≥–1), intubation or ventilation in the delivery room (yes vs no), APGAR score (<7 vs ≥7), low weight gain (difference between the Z-score of the weight at discharge and the Z-score of the weight at birth <–1 vs ≥–1), respiratory assistance (yes vs no), severe neurological anomalies (yes vs no), nosocomial infection (yes vs no), socio-economic level (intermediate level vs high), eligibility for social-security benefits for those with low incomes (yes vs no), parental separation (parents living together or parents living separately) and residential municipality (urban vs rural).

The PAF for the perinatal characteristics were the highest among the four types of determinants for all of the developmental disorders, with values around 60% (Figure 4). When based only on potentially modifiable perinatal factors, i.e. gestational age, birth weight, intubation or ventilation in the delivery room and APGAR score, these PAF were equal to 54% for motor disorders, 33% for sensory disorders, 36% for cognitive disorders, 29% for language disorders and 32% for psychological disorders (dotted lines in Figure 4). On the other hand, the prenatal complications had the lowest contributions on motor, sensory and cognitive disorders (PAF around 10%). For the motor and sensory disorders, the second highest PAF corresponded to the neonatal morbidities (PAF around 15%) whereas, for the cognitive, language and psychological disorders, the second highest PAF corresponded to the socio-economic conditions (PAF of around 25%). The detailed PAF of each separate risk factor are presented Supplementary Table 3, available as Supplementary data at IJE online.

Population-attributable fractions (PAF) estimates for the prenatal complications (prolonged rupture of membranes >24 hours, intrauterine growth restriction, preterm labour and maternal hypertension), the perinatal characteristics (gender, multiple pregnancies, gestational age, birth weight, intubation or ventilation in the delivery room and APGAR score), the neonatal morbidities (low weight gain, respiratory assistance, severe neurological anomalies and nosocomial infection) and the socio-economic conditions (socio-economic level, social-security benefits, parental separation and residential municipality) according to the developmental disorders. The PAF were calculated for all of the significant hazard ratios (detailed in Figure 3) and added together within each category of determinants, i.e. prenatal, perinatal, neonatal and socio-economic. For the perinatal characteristics, two percentages are provided: the percentage corresponding to the six characteristics (on top of the vertical bars) and the percentage corresponding to the perinatal complications potentially amenable for intervention, i.e. gestational age, birth weight, intubation or ventilation in the delivery room and APGAR score (dotted lines).
Figure 4.

Population-attributable fractions (PAF) estimates for the prenatal complications (prolonged rupture of membranes >24 hours, intrauterine growth restriction, preterm labour and maternal hypertension), the perinatal characteristics (gender, multiple pregnancies, gestational age, birth weight, intubation or ventilation in the delivery room and APGAR score), the neonatal morbidities (low weight gain, respiratory assistance, severe neurological anomalies and nosocomial infection) and the socio-economic conditions (socio-economic level, social-security benefits, parental separation and residential municipality) according to the developmental disorders. The PAF were calculated for all of the significant hazard ratios (detailed in Figure 3) and added together within each category of determinants, i.e. prenatal, perinatal, neonatal and socio-economic. For the perinatal characteristics, two percentages are provided: the percentage corresponding to the six characteristics (on top of the vertical bars) and the percentage corresponding to the perinatal complications potentially amenable for intervention, i.e. gestational age, birth weight, intubation or ventilation in the delivery room and APGAR score (dotted lines).

Discussion

This study, which was part of a large longitudinal population-based cohort study, illustrated the heterogeneity of risk factors for developmental disorders. To our knowledge, this is the first study to investigate the effects of these various risk factors on different developmental dimensions in a large longitudinal population-based cohort of preterm infants. We found that the PAF for perinatal characteristics amounted to around 60%, irrespective of the developmental disorder. By contrast, for neonatal and socio-economic determinants, the PAF varied according to the developmental disorder considered, with values comprised between 2 and 27%. Finally, the prenatal complications had the lowest contributions around 10%. The strength of our study lies largely with the quantification of these various types of determinants within the same population that allow comparison of their relative contributions.

PAF can be interpreted as the proportions of disease cases that could be prevented by elimination of the exposures. Among perinatal characteristics, the gender and multiple pregnancies are not amenable to interventions, whereas the other factors could be targeted for intervention. Based on the PAF interpretation, elimination of the adverse perinatal characteristics should allow between 30 and 50% of the developmental disorders to be prevented. It has been shown that interventions such as cervical cerclage or smoking cessation could prevent between 2 and 8% of preterm births in women at risk (older age or with lower socio-economic level).53 Moreover, specific nutrition interventions during pregnancy, especially multiple micronutrients supplementation, had a positive effect in reducing low birth weight by 10–14%.54 The contributions of neonatal morbidities and socio-economic conditions varied according to the developmental disorder. Elimination of all of the neonatal morbidities should allow around 15% of the motor disorders to be prevented. Similarly, a favourable socio-economic environment should be able to reduce the language, cognitive and psychological disorders by around a quarter.

PAF are valid based on the assumption of a causal relationship between exposures and diseases. Potential violation of the causality could be due to measured and unmeasured confounders. Regarding measured confounders, the propensity score matching allows confounding biases to be limited. The balances of measured covariates between the exposed and unexposed groups showed that measured confounders were appropriately accounted for. Indeed, all the SD were ≤0.26 and, in the final models, the covariates used to estimate the propensity score were further used as adjustment variables. There is no widely accepted absolute criterion for assessing when balance occurs; however, some studies have suggested that the 0.25 value provides some guideline.55–57 Although the absence of unmeasured confounders cannot be formally established here, the number and diversity of risk factors for developmental disorders considered from prenatal complications to the socio-economic environment may have a priori reduced the risk of bias. However, some risk factors known to influence the developmental disorders were not taken into account, such as the age of the mother. The assumptions underlying valid PAF estimation also include the independence of the considered risk factors. To check for the validity of this assumption, the variance inflation factors (VIF) were calculated for all the risk factors considered. One model was run for each developmental disorder. The higher the value, the higher the collinearity. The definition of ‘high value’ is somewhat arbitrary but values between 5 and 10 are commonly used.58 All the VIF were <2.5 (Supplementary Table 4, available as Supplementary data at IJE online), indicating that the independence of the risk factors can reasonably be assumed, and therefore suggesting the validity of the PAF estimated.

The prevalences of motor,18 cognitive,4,11,18 language24,25 and psychological disorders20 found in this study are consistent with the prevalences reported for other studies conducted in preterm infant populations. In regard to the risk factors for developmental disorders, our results are in accordance with other studies conducted in preterm infant populations, with the exception of the effects of low weight gain during the hospitalization and the urbanicity (non-significant results). In regard to the effect of the low weight gain, the reason for this finding lies with the construction of the variable. We have considered a binary variable, corresponding to the difference between the Z-scores of the weight at birth and the weight at discharge <–1 vs ≥–1. When considering a categorical variable that allows comparison of infants with a difference between Z-scores <–1 and ≥0, we found an increased risk of motor disorders. This exposure was considered binary for the calculation of the PAF (i.e. exposed vs not exposed). In regard to the effect of urbanicity, we did not find association with cognitive disorders. Although it has been reported that there is an association between urbanicity and cognitive development at 5 years of age in preterm infants, this finding may be due to the fact that the study population in question only comprised preterm infants with optimal neurodevelopment at 2 years of age.32

A major strength of this population-based study was the longitudinal design that allowed developmental disorders of infants from 18 months to 7 years of age to be considered. Of the 4502 infants enrolled in the LIFT network, the longitudinal design meant that 91% of the infants (n = 4122) had at least one follow-up visit between 12 and 84 months of age and they could hence be included in the analyses. Moreover, investigation of various and diverse risk factors allowed part of the complexity of the associations between the prenatal complications, the infant characteristics at birth and during hospitalization, their socio-economic environment and their possible developmental disorders to be captured. The choice of the factors to include in each type of determinant was based on clinical expertise and literature. Each developmental disorder was assessed using appropriate diagnostic criteria and/or validated questionnaires according to the age of the infants. The fact that we found developmental disorder prevalences that are close to those reported in other studies underscores the appropriateness of the tools used to assess the developmental disorders. A further strength of our study is the large number of infants included, which allowed a high statistical power to be attained.

This study has limitations. First, the characteristics of infants who were excluded from the study population were not comparable to the characteristics of the children who were included (Table 1). Late-preterm infants with fewer neonatal complications born to families with a lower socio-economic level were overrepresented in the category that was not included for the analysis. However, the absolute differences in the characteristics were quite small, thus indicating that the inclusion criteria did not result in an obvious selection bias and that the findings of this study should be generalizable to the preterm birth infants in this part of France. Second, some infants were lost during the follow-up. However, for the infants enrolled in the LIFT network, 78.1% were still being followed at 5 years.33 Finally, some factors known to influence the developmental disorders were not available in this study, such as the parental age or their mental health.

Conclusion

This study illustrates the heterogeneity of risk factors on the risk of developmental disorder in preterm infants. These results suggest the importance of considering both medical and psycho-social follow-ups of infants and their families.

Funding

This work was supported by grants from the Regional Health Agency of the Pays de la Loire. The sponsor had no role in regard to this manuscript.

Conflict of interest: The authors have declared that they have no conflict of interest.

References

1

Schwartz
RM
,
Luby
AM
,
Scanlon
JW
,
Kellogg
RJ
.
Effect of surfactant on morbidity, mortality, and resource use in newborn infants weighing 500 to 1500 g
.
N Engl J Med
1994
;
330
:
1476
80
.

2

Wilson-Costello
D
,
Friedman
H
,
Minich
N
,
Fanaroff
AA
,
Hack
M
.
Improved survival rates with increased neurodevelopmental disability for extremely low birth weight infants in the 1990s
.
Pediatrics
2005
;
115
:
997
1003
.

3

Pierrat
V
,
Marchand-Martin
L
,
Arnaud
C
et al. .
Neurodevelopmental outcome at 2 years for preterm children born at 22 to 34 weeks’ gestation in France in 2011: EPIPAGE-2 cohort study
.
BMJ
2017
;
358
:
j3448
.

4

Larroque
B
,
Ancel
P-Y
,
Marret
S
et al. .
Neurodevelopmental disabilities and special care of 5-year-old children born before 33 weeks of gestation (the EPIPAGE study): a longitudinal cohort study
.
Lancet
2008
;
371
:
813
20
.

5

Marlow
N
.
Neurocognitive outcome after very preterm birth
.
Arch Dis Child Fetal Neonatal Ed
2004
;
89
:
F224
28
.

6

Marlow
N
,
Wolke
D
,
Bracewell
MA
,
Samara
M
; EPICure Study Group.
Neurologic and developmental disability at six years of age after extremely preterm birth
.
N Engl J Med
2005
;
352
:
9
19
.

7

Saigal
S
,
Doyle
LW
.
An overview of mortality and sequelae of preterm birth from infancy to adulthood
.
Lancet
2008
;
371
:
261
69
.

8

Potharst
ES
,
van Wassenaer
AG
,
Houtzager
BA
,
van Hus
JWP
,
Last
BF
,
Kok
JH
.
High incidence of multi-domain disabilities in very preterm children at five years of age
.
J Pediatr
2011
;
159
:
79
85
.

9

Foulder-Hughes
LA
,
Cooke
RWI
.
Motor, cognitive, and behavioural disorders in children born very preterm
.
Dev Med Child Neurol
2003
;
45
:
97
103
.

10

Goyen
TA
,
Lui
K
.
Longitudinal motor development of ‘apparently normal’ high-risk infants at 18 months, 3 and 5 years
.
Early Hum Dev
2002
;
70
:
103
15
.

11

Breeman
LD
,
Jaekel
J
,
Baumann
N
,
Bartmann
P
,
Wolke
D
.
Preterm cognitive function into adulthood
.
Pediatrics
2015
;
136
:
415
23
.

12

Saigal
S
.
Follow-up of very low birthweight babies to adolescence
.
Semin Neonatol
2000
;
5
:
107
18
.

13

Anderson
P
,
Doyle
LW
,
Victorian
ICSG
.
Neurobehavioral outcomes of school-age children born extremely low birth weight or very preterm in the 1990s
.
JAMA
2003
;
289
:
3264
72
.

14

Anderson
PJ
,
Doyle
LW
; Victorian Infant Collaborative Study Group.
Executive functioning in school-aged children who were born very preterm or with extremely low birth weight in the 1990s
.
Pediatrics
2004
;
114
:
50
57
.

15

van Baar
AL
,
Vermaas
J
,
Knots
E
,
de Kleine
MJK
,
Soons
P
.
Functioning at school age of moderately preterm children born at 32 to 36 weeks’ gestational age
.
Pediatrics
2009
;
124
:
251
57
.

16

Mitha
A
,
Foix-L’Hélias
L
,
Arnaud
C
et al. .
Neonatal infection and 5-year neurodevelopmental outcome of very preterm infants
.
Pediatrics
2013
;
132
:
e372
80
.

17

Beaino
G
,
Khoshnood
B
,
Kaminski
M
et al. .
Predictors of cerebral palsy in very preterm infants: the EPIPAGE prospective population-based cohort study
.
Dev Med Child Neurol
2010
;
52
:
e119
25
.

18

Marret
S
,
Marchand-Martin
L
,
Picaud
J-C
et al. .
Brain injury in very preterm children and neurosensory and cognitive disabilities during childhood: the EPIPAGE cohort study
.
PLoS One
2013
;
8
:
e62683
.

19

Beaino
G
,
Khoshnood
B
,
Kaminski
M
et al. .
Predictors of the risk of cognitive deficiency in very preterm infants: the EPIPAGE prospective cohort
.
Acta Paediatr
2011
;
100
:
370
78
.

20

Delobel-Ayoub
M
,
Arnaud
C
,
White-Koning
M
et al. .
Behavioral problems and cognitive performance at 5 years of age after very preterm birth: the EPIPAGE Study
.
Pediatrics
2009
;
123
:
1485
92
.

21

Bhutta
AT
,
Cleves
MA
,
Casey
PH
,
Cradock
MM
,
Anand
KJS
.
Cognitive and behavioral outcomes of school-aged children who were born preterm: a meta-analysis
.
JAMA
2002
;
288
:
728
37
.

22

Barre
N
,
Morgan
A
,
Doyle
LW
,
Anderson
PJ
.
Language abilities in children who were very preterm and/or very low birth weight: a meta-analysis
.
J Pediatr
2011
;
158
:
766
74.e1
.

23

Boyer
J
,
Flamant
C
,
Boussicault
G
et al. .
Characterizing early detection of language difficulties in children born preterm
.
Early Hum Dev
2014
;
90
:
281
86
.

24

Flamant
C
,
Branger
B
,
Nguyen The Tich
S
et al. .
Parent-completed developmental screening in premature children: a valid tool for follow-up programs
.
PLoS One
2011
;
6
:
e20004
.

25

Fernandes
LV
,
Goulart
AL
,
Santos
AMN
,
dos
et al. .
Neurodevelopmental assessment of very low birth weight preterm infants at corrected age of 18–24 months by Bayley III scales
.
J Pediatr (Rio J)
2012
;
88
:
471
78
.

26

Potijk
MR
,
Kerstjens
JM
,
Bos
AF
,
Reijneveld
SA
,
de Winter
AF
.
Developmental delay in moderately preterm-born children with low socioeconomic status: risks multiply
.
J Pediatr
2013
;
163
:
1289
95
.

27

Sucksdorff
M
,
Lehtonen
L
,
Chudal
R
et al. .
Preterm birth and poor fetal growth as risk factors of attention-deficit/ hyperactivity disorder
.
Pediatrics
2015
;
136
:
e599
608
.

28

Short
EJ
,
Klein
NK
,
Lewis
BA
et al. .
Cognitive and academic consequences of bronchopulmonary dysplasia and very low birth weight: 8-year-old outcomes
.
Pediatrics
2003
;
112
:
e359
.

29

Ehrenkranz
RA
,
Dusick
AM
,
Vohr
BR
,
Wright
LL
,
Wrage
LA
,
Poole
WK
.
Growth in the neonatal intensive care unit influences neurodevelopmental and growth outcomes of extremely low birth weight infants
.
Pediatrics
2006
;
117
:
1253
61
.

30

Wong
HS
,
Edwards
P
.
Nature or nurture: a systematic review of the effect of socio-economic status on the developmental and cognitive outcomes of children born preterm
.
Matern Child Health J
2013
;
17
:
1689
700
.

31

Wild
KT
,
Betancourt
LM
,
Brodsky
NL
,
Hurt
H
.
The effect of socioeconomic status on the language outcome of preterm infants at toddler age
.
Early Hum Dev
2013
;
89
:
743
46
.

32

Gouin
M
,
Flamant
C
,
Gascoin
G
et al. .
The association of urbanicity with cognitive development at five years of age in preterm children. Carpenter DO, editor
.
PLoS One
2015
;
10
:
e0131749
.

33

Hanf
M
,
Nusinovici
S
,
Rouger
V
et al. .
Cohort profile: longitudinal study of preterm infants in the Pays de la Loire region of France (LIFT cohort)
.
Int J Epidemiol
2017
;
46
:
1396
7
.

34

Amiel-Tison
C
.
Fetal and Neonatal Neurology and Neurosurgery
, 2nd edn.
London
:
Churchill Livingstone
,
1995
.

35

Alla
F
,
Guillemin
F
,
Colombo
MC
,
Roy
B
,
Maeder
C
.
[Diagnostic value of ERTL4: a screening test of language disorders in 4-year-old children]
.
Arch Pediatr
1998
;
5
:
1082
88
.

36

Lacert
P
.
[Screening of language disorders using the ERTL 4 test]
.
Arch Pediatr
1999
;
6
:
340
41
.

37

Petit-Carrié
S
,
Verret
C
,
Cossard
A
,
Maurice-Tison
S
.
[Access to early speech therapy in Gironde: evaluation of a language-screening campaign for 4-year-old children (1999–2001)]
.
Arch Pediatr
2003
;
10
:
869
75
.

38

Robins
DL
,
Fein
D
,
Barton
ML
,
Green
JA
.
The modified checklist for autism in toddlers: an initial study investigating the early detection of autism and pervasive developmental disorders
.
J Autism Dev Disord
2001
;
31
:
131
44
.

39

Baron-Cohen
S
,
Cox
A
,
Baird
G
et al. .
Psychological markers in the detection of autism in infancy in a large population
.
Br J Psychiatry
1996
;
168
:
158
63
.

40

Billard
C
,
Livet
MO
,
Motte
J
,
Vallée
L
,
Gillet
P
.
The BREV neuropsychological test: Part I. Results from 500 normally developing children
.
Dev Med Child Neurol
2002
;
44
:
391
97
.

41

Josse
D
.
Revised Brunet–Lezine Scale of Psychomotor Development of First Childhood
[in French]. Paris: Etablissement d’Applications Psychotechniques, 1997
.

42

Fily
A
,
Pierrat
V
,
Delporte
V
,
Breart
G
,
Truffert
P
,
Epipage Nord-Pas-de-Calais
SG
.
Factors associated with neurodevelopmental outcome at 2 years after very preterm birth: the population-based Nord-Pas-de-Calais EPIPAGE cohort
.
Pediatrics
2006
;
117
:
357
66
.

43

Squires
J
,
Bricker
D
,
Potter
L
.
Revision of a parent-completed development screening tool: ages and stages questionnaires
.
J Pediatr Psychol
1997
;
22
:
313
28
.

44

Goodman
A
,
Goodman
R
.
Strengths and difficulties questionnaire as a dimensional measure of child mental health
.
J Am Acad Child Adolesc Psychiatry
2009
;
48
:
400
03
.

45

Guimard
P
,
Cosnefroy
O
,
Florin
A
.
Évaluation des comportements et des compétences scolaires par les enseignants et prédiction des performances et des parcours à l’école élémentaire et au collège
.
L’orientation scolaire et professionnelle
2007
;
36
:
179
202
.

46

Guimard
P
,
Florin
A
.
School behaviors during the second year of kindergarten: prediction of success at elementary school
.
Psychol Psychom
2001
;
22
:
75
100
.

47

Boussicault
G
,
Nguyen The Tich
S
,
Branger
B
et al. .
The global school adaptation score: a new neurodevelopmental assessment tool for very preterm children at five years of age
.
J Pediatr
2013
;
163
:
460
64
.

48

Olsen
IE
,
Groveman
SA
,
Lawson
ML
,
Clark
RH
,
Zemel
B.S
.
New intrauterine growth curves based on United States data
.
Pediatrics
2010
;
125
:
e214
24
.

49

Simon
L
,
Nusinovici
S
,
Flamant
C
et al. .
Post-term growth and cognitive development at 5 years of age in preterm children: evidence from a prospective population-based cohort
.
PLoS One
2017
;
12
:
e0174645
.

50

Nusinovici
S
,
Olliac
B
,
Flamant
C
et al. .
Impact of preterm birth on parental separation: a French population-based longitudinal study
.
BMJ Open
2017
;
7
:
e017845
.

51

Austin
P.C
.
Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies
.
Pharm Stat
2011
;
10
:
150
61
.

52

Rockhill
B
,
Newman
B
,
Weinberg
C
.
Use and misuse of population attributable fractions
.
Am J Public Health
1998
;
88
:
15
19
.

53

Chang
HH
,
Larson
J
,
Blencowe
H
et al. .
Preventing preterm births: analysis of trends and potential reductions with interventions in 39 countries with very high human development index
.
Lancet
2013
;
381
:
223
34
.

54

da Silva Lopes
K
,
Ota
E
,
Shakya
P
et al. .
Effects of nutrition interventions during pregnancy on low birth weight: an overview of systematic reviews
.
BMJ Glob Health
2017
;
2
:
e000389
.

55

Stuart
E.A
.
Matching methods for causal inference: a review and a look forward
.
Stat Sci
2010
;
25
:
1
21
.

56

Rubin
DB
.
Using propensity scores to help design observational studies: application to the tobacco litigation
.
Health Serv Outcomes Res Methodol
2001
. https://link.springer.com/content/pdf/10.1023%2FA%3A1020363010465.pdf (24 October 2018, date last accessed).

57

Harder
VS
,
Stuart
EA
,
Anthony
J.C
.
Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research
.
Psychol Methods
2010
;
15
:
234
49
.

58

James
G
,
Witten
D
,
Hastie
T
,
Tibshirani
R
.
An Introduction to Statistical Learning: With Applications in R
.
New York
:
Springer
,
2013
.

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://dbpia.nl.go.kr/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Supplementary data