The Perceived Impacts of a Bilingual Learner's Dictionary

Abstract

This paper reports on research into teachers' perceptions of the impact of using a bilingual learner's dictionary. The research, a perceptions of impact study conducted in South Africa from March 2016 to February 2019, investigates the perceptions of teachers on the impact of the dictionary on themselves as teachers, and their perceptions of the dictionary's impact on their pupils. The findings show that teachers perceived dictionary use to have positive impacts on both the language production and language reception skills of pupils in their L2, in line with other studies. However, they also show unexpected teacher perceptions of impacts on content subjects, L1, teaching itself, and attitudes and behaviours (here called the socio-emotional), especially with regard to learning or teaching. The paper situates this research in the literature on dictionary use for production and reception in language learning, and of impact evaluation, as well as highlighting relevant aspects of the South African education system, particularly the use of English as a language of learning and teaching from an early stage. The dictionary in this study was the Oxford Bilingual School Dictionary: isiXhosa and English (De Schryver and Reynolds 2014).

1. Introduction

Bilingual dictionaries are popular with language learners, perhaps especially those who are beginners. Many studies have been undertaken to determine how bilingual, monolingual and bilingualised dictionaries compare in their support of writing and reading in a second or additional language (L2). In the last decade or more, electronic dictionaries, in their many forms, have grabbed the attention of lexicographic researchers. In South Africa, however, print dictionaries, whether bilingual or monolingual, continue to dominate school use, sometimes because of teachers' reluctance to accommodate connected devices (and the risks associated with them) in their classrooms, and sometimes because internet connectivity or electricity are unavailable or unreliable. For this reason, this research focuses on the use of a print dictionary.

South Africa participates in international educational evaluation studies such as PIRLS (Progress in International Reading Literacy Study) and TIMSS (Trends in International Mathematics and Science Study), evaluating literacy, and mathematics and science performance, respectively. Both these and more local studies such as SACMEQ (the Southern and Eastern Africa Consortium for Monitoring Educational Quality) show that, while the South African education system is improving, delivering better results than previously, the results themselves continue to be distressingly poor (Van der Berg and Gustafsson 2019). The most recent PIRLS results, from 2016, showed that 78% of grade 4 pupils cannot read for meaning (Mullis, et al. 2017). There are many contributing factors to this situation, for example, weak institutional functionality and wasted learning time (Van der Berg, Spaull, et al. 2016) and differences in test difficulty by language (Mtsatse and Van Staden 2021), all of which are beyond the scope of this paper, other than to suggest that literacy, both learning to read and reading to learn, are issues of urgent and critical importance in South Africa.

This study, conducted in the South African province of the Eastern Cape from 2016 to 2019, focuses on teacher's perceptions of impact of the use of a bilingual learner's dictionary in the classroom. The research instrument collected data on teachers' perceptions of impact on themselves and their perceptions of impact on their pupils, whether impact was positive, negative or absent, and perceptions of the counterfactual (i.e. what teachers perceived would be the result of no longer having access to the dictionary). It also collected contextual data including:

• the purposes for which dictionaries were used,

• how often dictionaries were used,

• which languages they were used to support,

• challenges experienced during use, and

• how many copies were available.

Section 2 of this paper deals with the literature on the impact of bilingual dictionaries in particular, Section 3 details the research question, Section 4 highlights relevant aspects of the educational environment, Section 5 touches on pertinent features of the dictionary for those unfamiliar with it, and Section 6 describes the research methodology. Section 7 presents the findings and Section 8 concludes with a discussion of the findings.

2. Literature

Bilingual dictionaries are considered to be ‘absolutely essential’ (Christianson 1997, 23) by most foreign language learners. Compared to both bilingualised and monolingual dictionaries, bilingual dictionaries have the best results for ‘unskilled dictionary users’ on production tasks (Laufer and Hadar 1997, 193). More specifically, Lew (2016) finds that pupils writing an argumentative essay with the help of a dictionary had better results than those without access to one, following on from Tall and Hurman (2002) who also found gains in those using a bilingual dictionary. Atkins and Varantola found that the bilingual dictionary in their study had more successful and more numerous look-ups than the monolingual dictionary they included (Atkins and Varantola 1997), while Lew and Adamska-Sałaciak note the key role of dictionaries as a whole in language learning and teaching, part of which is due to their ‘promot[ing] learner autonomy’ ((Lew and Adamska-Sałaciak, 2015), 47). They are also more widely used than monolingual dictionaries ((Atkins and Knowles 1990) in (Rundell 1999)), especially by less-skilled learners. Studies have also shown bilingual dictionaries benefiting learners on reception tasks, such as Dziapa (2001) (in Lew (2004)) who found bilingual dictionaries gave beginner learners an advantage in acquiring L2 vocabulary through reading.

This suggests a considerable consensus on the preference of language learners for bilingual dictionaries, and of the beneficial usefulness of bilingual dictionaries to these learners. Yet, some researchers (and teachers) disagree, believing that bilingual dictionaries either induce errors (Ard 1982) or are not as beneficial to learners as monolingual dictionaries (Ng 2016). In fact, some dictionary user studies have yielded contrasting rather than consensus results, sometimes due to methodological issues (Lew 2004, 30). However, regardless of academic consensus or the lack of it, learner preference and utility will have little effect if trends in language teaching or trends in teacher practice lean in a different direction, or if bilingual dictionaries are not available to learners. Both of these situations are relevant to South Africa.

As Hall and Cook (2012) note, academic trends take time to reach teachers and curriculum developers, with the result that actual classroom practice may be as informed by a teacher cohort's training at the beginning of their career as it is by current academic consensus or research evidence. This is certainly the case in South Africa, where the use of the mother tongue (or L1) in learning an additional language (or L2, usually English) has long been frowned upon, but is widely practised (for example, Probyn (2009) or Ngubane, Ntombela and Govender (2020)).

In South Africa, up-to-date bilingual dictionaries in language pairs with English are also not uniformly available for all official languages, nor are some of those dictionaries that are widely used specifically designed to support learners, for example, by including features that support L2 text production and L2 text reception, as suggested by Tarp (2004).

Furthermore, even where there is a modern, learner-focused bilingual dictionary for a particular official South African language, the dictionary is unlikely to be available to the majority of school pupils with the relevant L1. This could be as a result of educational resources being inequitably distributed during the apartheid era, the beliefs of teachers and curriculum developers about the suitability of bilingual or monolingual dictionaries, or inadequate funds on the part of government or parents for textbooks and other learning and teaching support materials (LTSM). The result is that those pupils with access to a dictionary are much more likely to have access to a monolingual English dictionary, but not to a bilingual dictionary including their L1. (There is little publicly available data on the proportion of South African pupils with dictionaries, or which type of dictionary is more common. The author therefore makes these statements based on some publishers' sales figures, Nielsen data on the sales of books through selected outlets, and resource-buying trends in the trade and by provincial education departments from about 2003 to 2018.)

The measurement and evaluation of impact is familiar to the world of social and development policy and economics (Gugerty and Karlan 2018); in the period since 2000 (White and Masset 2018), it has come to be widely applied in the field of education, for example, by the World Bank (Bruns, Filmer and Patrinos 2011). In general, these studies understand impact to have its general meaning of ‘the effect or influence of something on someone (or something)’. Studies commonly aim to identify both positive and negative impacts, as well as intended and unintended consequences, and to consider the counterfactual (usually, what would have happened if the intervention or treatment had not taken place). In practice, education studies have often been used to investigate which interventions, in particular, can be shown to have demonstrable positive impacts, in order to make policy more evidence-based and investments yield greater benefits.

Issues have, of course, been raised both with the use of certain impact methodologies, especially randomized control trials (Ravallion 2018), and with individual studies, by those who might be considered proponents (White and Masset 2018), as well as by those suggesting adaptations (Shah, et al. 2015) or alternatives. The concerns include the questionable relevance of research topics to practitioners, the difficulty in reaching conclusions that are applicable across a wide variety of settings, the cost and time that some studies absorb (Goldstein 2016), difficulties in implementing policies based on such studies, and low policy uptake or dissemination of studies (White and Masset 2018).

Not all impact studies use methodologies such as randomised control trials, or testing before and after an intervention (often called pre- and post-testing). Studying participants' perceptions of impact is another available methodology. This approach is undertaken where constraints such as a lack of publicly available and suitable data pertain, as well as where the perceptions of participants may yield information of interest not readily available through quantitative methods (McKay 2020, 5). This study, which attempts to evaluate the impact of using a particular dictionary, adopts the perceptions of impact methodology (Section 6 explores the reasons for this in more detail).

3. The research question

The purpose of this research was to ascertain whether teachers perceived that using the Oxford Bilingual School Dictionary: isiXhosa and English (De Schryver and Reynolds 2014) had an impact on themselves as teachers and on their pupils. If teachers perceived that using the dictionary had an impact, the research aimed to ascertain whether teachers perceived the impact to be positive or negative.

4. Context

The South African educational environment presents certain features that are worth noting as they shape the needs of those learning English differently to how they may be shaped in countries such as Poland or Japan, where English is a foreign language.

South Africa is a multilingual country, with eleven official languages, nine of which are African languages; the others are English and Afrikaans. All school pupils are required to study two languages throughout the twelve years of school-based education, one at home language level (HL¹, or L1), and the other at first additional language level (FAL, or L2) (Department of Basic Education 2011, 6–7). A pupil may choose to study a third language at second additional language level (SAL), which is much more limited than first additional language level and is mainly taken by those studying languages, such as French, that are not South African official languages (Ferreira-Meyers and Horne 2017). The overwhelming majority of school pupils study English as their L2 (Department of Basic Education 2021 (a), 11) (Department of Basic Education 2021 (b), 7), with an African language as their L1. Almost all high-school pupils study English, whether as L2 or as L1.

More importantly for its impact on dictionary user needs, English is the language of learning and teaching (LOLT, or medium of instruction) for almost all pupils from grade 4 onwards (Howie, Venter and Van Staden 2008). Although many African-language-speakers have their mother tongue as LOLT in the first three years of school, almost all schools transition to either English or Afrikaans as LOLT from grade 4 onwards, since these are ‘the only languages in which it is possible to write the secondary school leaving examinations’ (Taylor and Von Fintel 2016, 76). The result is that ‘more than 80% of South African pupils’ from grade 4 upwards learn in an additional language (Howie, Venter and Van Staden 2008, 554); for most of these children, their LOLT is English (Taylor and Von Fintel 2016). (This situation or similar has been in place since the late 1970s (Heugh 2008) even though Macdonald (1990) (in Heugh (2008)) showed that four or even five years of L1 as LOLT could not yield pupils sufficiently competent in English to switch to English as LOLT at the end of it.) Proficiency (or at least competence) in English is therefore crucial to a child’s success in almost all subjects, affecting their educational trajectory and life-path in a material way.

Most South African school pupils are therefore faced with learning English in order to learn other subjects. This makes their dictionary user needs quite different to that of a school pupil who studies Mathematics, Geography, Biology and other so-called content subjects through their L1. The South African pupil needs, at an early primary school level, to be able both to understand oral and written English and to produce oral and written texts in their L2, across a range of different subject disciplines. The dictionary they use therefore needs to do more than just use a controlled vocabulary and meta-language, and have an appropriately simple entry structure and layout, as should be the case if the dictionary user is a child (Tarp 2004, 303), although it must do these things too.

In terms of reception and production, the dictionary ideally needs to support not just the development of basic interpersonal communication skills (BICS) – often the starting point for beginner foreign language courses—but also the beginning stages of cognitive academic language proficiency (CALP), where learners acquire academic competency in their L2 (Cummins 2008). It therefore needs, for example, to help learners understand and distinguish between key CALP verbs, such as discuss, compare, and analyse, and help them use these words by providing examples. Some of these examples should arguably also feature CALP- rather than BICS-oriented contexts because English in CALP contexts is what pupils need to master to succeed at school.

In terms of the lemmas that the dictionary covers, the selection on the English to African language side will ideally need to include content subject vocabulary (e.g. pyramid) as well as language terminology (e.g. metaphor), to ensure learners can look up unfamiliar words they find in their textbooks. These words can be called curriculum terms; they may be relatively infrequent in general corpora, but earn their place in a dictionary for these users because of their appearance in corpora that include school textbooks (De Schryver and Reynolds 2014, xi). Conversely, the African language to English side will need fewer content subject terms. This is because learners are likely to encounter these terms and concepts for the first time in English, without knowing them first in their L1. Again, this makes the needs of these South African pupils different to those of, for example, older immigrant pupils in the United States, who may already have acquired at least some subject terminology in their L1. However, some space in the headword list on the African language to English side should be allocated for commonly accepted curriculum terms in the African language, to further spread their use, and encourage their standardisation (e.g. itshathi ‘a chart’, ixesha langoku ‘the present tense’, ukufaqa isangqa ‘to circle’).

5. The dictionary

The introduction to the Oxford Bilingual School Dictionary: isiXhosa and English (BDX) gives the following as the aims guiding its development (De Schryver and Reynolds 2014, x):

• enabling learners² to read, write, listen and speak competently in their additional language,

• making this dictionary exceptionally easy to use,

• ensuring that this dictionary reflects current usage and concerns,

• compiling it so that it is equally relevant and useful for learners of English and of isiXhosa, and

• offering learners valuable extras in the Study pages, including activities to develop dictionary skills, and clear grammar guides for both languages.

It is clear from the first and fourth items in this list that the dictionary is intended to be a language learner's dictionary, i.e. to support the reception and production of both English and isiXhosa (alternatively Xhosa). The introduction briefly sets out what is new and different about the dictionary. This includes a new approach to the lemmatisation of certain isiXhosa words, the use of modern class numbers, cross-references from plurals to singular forms and from extended verbs to base forms, and the use of an isiXhosa corpus (including textbooks) to support headword selection and to supply examples (De Schryver and Reynolds 2014, x-xi). For the purposes of this paper, a few example entries will help readers who are unfamiliar with the BDX to note some unusual features not mentioned in the introduction, which are relevant to the findings of the research.

Figure 1 shows the entry for longitude on the English-isiXhosa side of the dictionary. The guide to dictionary features in the front matter of the BDX explains that the Information symbol (directly after the translation equivalent iside) marks an entry that is ‘specially important in schoolwork’ (De Schryver and Reynolds 2014, vi-vii), in other words, a curriculum term. These entries include a subject label (here, Geography) and definitions in both languages, as well as the usual text types for a modern bilingual dictionary. Some, as here, include cross-references to related terms (here, latitude) and to illustrations.

Figure 2 shows the entry for pulley on the English-isiXhosa side of the dictionary. Although it is not marked as a curriculum term like the entry in Figure 1, and therefore does not have definitions in both languages, it is relevant to the study of content subjects. The illustration supports the understanding of an unfamiliar term without a definition, partly because labels, as well as a caption, are included, and both are presented bilingually.

On the isiXhosa-English side of the dictionary, Figure 3 shows the entry for -jonga, a highly frequent isiXhosa verb (frequency is shown by the stars following the lemma (De Schryver and Reynolds 2014, vi)). Of the two senses in the entry, the first is arguably a BICS-related sense, while the second, though brief, is more oriented to CALP, at a level appropriate to primary school children and using phrasing that may be familiar from pupils' textbooks. It draws isiXhosa-speaking users' attention to the fact that a familiar word may have a slightly different meaning in the learning environment.

Finally, Figure 4 shows the entry for the highly frequent isiXhosa noun intliziyo. The first sense gives the literal meaning (the bodily organ) while the second sense gives a figurative meaning.

The dictionary is the fourth in a series of bilingual school dictionaries published by Oxford University Press Southern Africa starting in 2007. By number of native-speakers, isiXhosa is the second largest official language in South Africa, with 8.1 million speakers out of a population of 51.8 million—about 15.6% (Statistics South Africa 2012, 23).

6. Methodology

The first part of this section notes overarching issues bearing on the methodology of the study. Sub-sections that follow give more detail about the study participants, how data were collected, and how data were processed and analysed.

In attempting to investigate the impact of the BDX in a credible way, two important factors were considered that directed decisions about methodology and approach.

It was important, firstly, not to give free copies of the dictionary to schools or teachers, but rather to contact those schools, teachers or classes who had already used the dictionary, having independently gained access to it. This was to avoid, as far as possible, social desirability, or the tendency of study participants to give answers that they perceive will be well-received. (Further steps to avoid social desirability are discussed in section 7.3.)

About a year before the trial stage of the research was set up, one of the nine South African provinces, the Eastern Cape, ordered resource materials for their schools, including the BDX. Since that province is home to the greatest number of isiXhosa-speakers (about 5 million out of a total 8.1 million), this province was well-suited to be the research location. As importantly, this province places central orders for LTSM unlike the Western Cape, an otherwise suitable province with the second largest number of isiXhosa-speakers. Since orders in the Eastern Cape were centralised, the provincial education department (PED) would be able to provide information to the researchers about which schools had ordered the dictionary.

Secondly, the funding available for the research was small, making it impossible to administer in-person tests to pupils (digital tests were not an option because most participating schools did not have working landline telephones nor computer laboratories to allow pupils access to the internet, and the Eastern Cape is more than 750 kms from where the authors are based). As a result, it was decided to focus on teachers' perceptions, rather than pre- and post-testing of pupils, for example, and to use remote research techniques, rather than, for example, the ‘think-aloud’ protocol (Krings (1986) (in Atkins and Varantola (1997)) or even a ‘paper equivalent’ to the think-aloud protocol (Atkins and Varantola 1997, 3). The chosen methodology was therefore a questionnaire administered to teachers by telephone.

Concerns about the use of questionnaires have been raised, such as those of Hatherall (1984) (in Lew (2002)). However, despite the imperfections of, and difficulties with, user research, practical lexicographers who fail to conduct and pay attention to it are likely to make products that fail to meet user needs. In this environment, Lew's ‘quick checklist’ on ‘how to make better questionnaires’ (Lew 2002, 270) suggests the following points relevant to the methodology of this study:

Do:

• write your questionnaire in the subjects' native language […]

• consider in each case whether multiple choice or open-ended or mixed question format is most appropriate

• decide before the design is complete how the results will be coded and processed

• screen your questions and multiple-choice answers for possible bias […]

• pilot your questionnaire […]

Don't:

• use technical language that subjects might not understand

• use complex syntax

• use negatives in questions […]

• give away your own position or preference in any way

The research instrument used is therefore in line with these guidelines insofar as they are relevant to the approach in this study.

6.1. Population and sample

The population for this research was all teachers of a language (either isiXhosa L1 or English L2) at grade 7 (the final year of primary school) in all educational districts of the Eastern Cape province. The sample was limited to teachers who taught at schools that:

• offered isiXhosa as a Home Language (L1), and

• offered English as a First Additional Language (L2), and

• had ordered a minimum of 10 copies of the BDX through the provincial education department (PED) in 2014.

This last limitation was necessary because the PED reduced the initial orders received from schools, but only made available the initial orders to the researchers. The limit was therefore used as a proxy for the revised order, with the aim of ensuring that all schools in the sample had received at least one copy of the dictionary despite the reduction in order quantities. The last limitation also meant that private (or independent) schools were not included in the sample—only public or government schools—since private schools do not order through the PED.

Copies of the dictionary were not given gratis, nor was any associated dictionary training offered to participating schools.

A preliminary study was conducted in the then-educational district of Port Elizabeth (later part of the Nelson Mandela district)³ in the Eastern Cape in March 2016. Following this trial, the study was extended in early 2018 to include all eleven remaining educational districts in the Eastern Cape, as well as those portions of the new Nelson Mandela district that had not previously been included; this part of the research was completed in February 2019.

Due to the timing of the research and the timing of the delivery of the 2014 resource order to schools (either late in 2014 or early in 2015, before the start of the school year in January), all candidate participating schools in the first stage (March 2016) would have had the dictionary available for use for more than a school year at the time of interview, with those in the second stage (2018 to early 2019) having had dictionaries available for use for a minimum of three years (2015–2018 and possibly into 2019), depending on the timing of the interview.

Table 1 shows the number of public schools in the districts, the number that met the three criteria above, the number of schools at which interviews were completed, and the number of interviews completed. Although the number of schools that completed interviews as a percentage of schools in the district is low (under 1%), the number of schools that completed interviews as a percentage of schools that met the criteria is more than a quarter (26.1%).

6.2. Data collection method

The author sought and received the permission of the PED for both stages of the research. The permission of all school principals to contact their teachers was also received.

Interviewers then set up telephone interviews with relevant teachers at each school. The research instrument (see the Supplementary material) was drawn up in English and translated into isiXhosa (the L1 of almost all the study participants) by one of the interviewers. It was administered in isiXhosa by the interviewers, who then translated and transcribed the recordings (or their notes, if a recording was not made) into English. Almost all interviews were recorded, and the recording kept (a small number of the total were either not recorded or the recording was deleted in error).

6.3. Data processing and analysis

The research data was extracted from interview transcriptions in Word to Excel spreadsheets to enable analysis both by question and thematically. Although the questionnaire had been set up in a structured way, with answer prompts (such as ‘a great extent/some extent/not at all’ and categories such as ‘Don’t know; Yes, they are required; No, they are not required'), responses were often wide-ranging, rather than confined to the prompts. The result is that it is possible to hear teachers' voices in a qualitatively satisfying way; the same fact, however, made analysis challenging.

The analysis, originally by question, was therefore adapted to allow themes and trends in participants' comments to be reflected more comprehensively in the results, by considering the translated interview transcript as a whole, rather than as discrete responses to specific questions. Thematic analysis of this sort is appropriate for and often used with interview transcripts, and other qualitative data. For this data, a general inductive approach (Thomas 2006) was followed, which allowed themes and patterns in the data to emerge through repeated readings.

The main analysis perspective is that of language reception (reading, listening, viewing) vs language production (writing, speaking, signing). This perspective aligns well with the structure of the South African language curriculum, the pillars of which, according to the NCS Curriculum Assessment and Policy Statement (CAPS) (Department of Basic Education 2011, 9), are:

• listening and speaking,

• reading and viewing,

• writing and presenting,

• language structure and conventions.

7. Findings

The findings of this study show that the scope and nature of the perceived impacts were broad, and included language production and language reception, L1 and L2, content subjects, attitudes and behaviours (here called the socio-emotional) especially with regard to learning or teaching, other grades, pupils and teachers, and both classwork and homework. In total, 35 valid interviews were completed at 28 schools, with more than one teacher taking part at a few schools (for example, the teacher of grade 7 isiXhosa L1 as well as the teacher of grade 7 English L2). The sections that follow focus on the nature of the schools that participated, the scope and nature of the perceived impact, data on how the dictionaries were used, challenges experienced by teachers using the dictionary, and the limitations of the study.

7.1. The nature of participating schools

In South Africa, public schools are categorized into quintiles on the basis of a school's infrastructure as well as the nature of the surrounding community. The quintile system is effectively a poverty ranking and is used to differentially allocate government funding. Quintile 1 schools are the poorest, while quintile 5 schools are the richest. All schools in quintiles 1 to 3 are no-fee schools; at the rest, school fees are payable. Of the 28 schools where interviews were completed, all 28 were in quintiles 1 to 3⁴; all were therefore no-fee schools. Of the 28, 12 are classified by the PED as rural, with the remainder urban; however, most schools that are classified as urban are in small towns, rather than cities. The participating schools can therefore be described as including the poorest, in the South African context; they also include ‘deep rural’ schools that often struggle to take advantage of educational interventions that pay dividends in better-resourced schools (Taylor 2019, 327). Due to their poverty and low learning outcomes, these schools are likely to be least efficient at converting inputs (such as extra textbooks or teachers) to better learning outcomes (Burger 2011).

Teacher participants show an awareness of their schools' remoteness and its potential impact on their pupils, with eight describing their schools as remote or rural⁵, and two describing English as a foreign language⁶, in their interviews.

7.2. The scope and nature of the perceived impact

Of the 35 teachers interviewed, 34 perceived that the BDX had an impact on their pupils, with all those stating that the impact was positive. The teacher who said that there had not been an impact⁷ attributed this to the BDX's limited coverage (approximately 5000 entries on each side), to which she ascribed a lack of growth in pupils' English vocabulary. In terms of impact on themselves as teachers, 33 participants perceived an impact, with the same number perceiving that it had been positive. Of the two teachers who did not report the dictionary to have had an impact on teachers,⁸ both evinced discomfort at the phrasing of the question (Question 4 in the Supplementary material) and said they could not comment on other teachers' experiences⁹. The question could have been better phrased to ensure teachers were asked about their own experience, rather than commenting on that of others.

The most interesting findings of this research, however, are those relating to the scope and the nature of the impact that teachers perceived, rather than solely the number of teachers with those perceptions.

Participants' perceptions of the impact of the BDX were broad. They include impacts on:

• language production and reception, and all four pillars of the language curriculum,

• L1 and L2,

• content subjects,

• the socio-emotional (attitudes and behaviours especially with regard to learning or teaching),

• grades other than grade 7 (the target grade),

• pupils as well as teachers, and

• classwork and homework.

Figure 5 shows the relative frequency of teacher comments about the perceived impact of the BDX, including those noting challenges. In line with what one might expect from a bilingual learner's dictionary, perceived impacts on language production and language reception are the most common. However, the frequency of other areas of perceived impact shown in Figure 5 is interesting and somewhat surprising: teaching, the socio-emotional, L1 and content subjects. Dictionary user studies seldom appear to seek to measure impact on teaching, nor on the feelings and attitudes of pupils or teachers in the classroom, making these comments unusual (though there have been studies on the impact of dictionaries on exams that indicate a reduction in stress (Tall and Hurman 2002)). Section 8 discusses this further.

Also unexpected are frequent comments about a perceived impact on the L1 and on content subjects: a monolingual, not a bilingual, dictionary is expected to be used to support a pupil's L1. But for African languages in South Africa, monolingual dictionaries are very seldom available at schools, very rarely exist in versions designed for school use but only for general or adult use, and for some languages may not exist at all or may no longer be in print. Zgusta noted that a bilingual dictionary might take the place of an unavailable monolingual (1971, 213), especially for a dead language such as Latin; the situation in South Africa is clearly different, but teachers may still find a bilingual dictionary helpful where no monolingual dictionary is available.

The presence of several isiXhosa varieties may influence teacher perceptions of impact on the L1: several teachers¹⁰ describe the BDX supporting their pupils' acquisition of ‘correct’ or standard isiXhosa, or say that they struggled with the isiXhosa definitions because they use ‘unfamiliar’ words.

Table 2 relates the components of the teacher comments about language production and reception to three of the four pillars of the English FAL (L2) curriculum, and also shows the curriculum range of teacher comments about perceived impacts.

It is interesting to note the presence of literature genres in teacher comments, both as something that pupils themselves are producing (e.g. poetry) and that they are studying (e.g. novel, poems). In one interview, a teacher notes that she manages to teach literature, something she could not do before she used the BDX¹¹. As can be seen from the frequency of comments about perceived impacts on ‘grammar’, in Figure 5, the BDX was also perceived to impact the fourth pillar of the curriculum, namely language structures and conventions.

Figures 6 and 7 show in greater detail the components of the first two of the unexpected perceptions of impact — teaching and the socio-emotional. Figure 6 shows the relative frequency of teacher comments on the impact of the BDX on teaching. These comments typically describe the dictionary making teaching easier or faster because teachers can spend less time explaining or translating in class — instead, they refer pupils to the dictionary. One teacher¹² memorably described her experience as follows:

‘We are no longer walking dictionaries for our learners. During reading of comprehensions, I just ask them to underline all difficult words. They check all difficult words from their dictionaries. […] It is not easy to be a walking dictionary for your learners. Most of your time is wasted doing translations. Also, though you are a teacher, there are some words which are not easy to explain but the dictionary gives clear easy explanations.’

The description of pupils being able to answer their own information needs independently, through using the dictionary, aligns with Lew and Adamska-Sałaciak's (2015) description of dictionaries enhancing learners' autonomy.

Figure 7 shows the relative frequency of teacher comments on components of perceived socio-emotional impact, with confidence and independence the most frequent. Most, but not all, comments relate to socio-emotional impacts on pupils, rather than on teachers.

The two sets of comments (on perceived impact on teaching and perceived socio-emotional impact) appear to make sense together: teachers will find their own workload reduced if pupils are able to resolve their information needs by using a dictionary independently.

Of the 61 comments about challenges using the dictionary (in Figure 5), almost half related to a perceived shortage of copies, with a further 31% (19) noting that it was time-consuming for pupils to use the dictionary or that their dictionary skills were lacking.

Bilingual dictionaries are not traditionally expected to support content subjects; rather monolingual Language for Special Purposes (LSP) or subject dictionaries are the conventional choice, focusing on the terminology of a particular subject area. Where pupils use their L1 as the LOLT for content subjects, this makes sense. However, the data from this study show that, where pupils use an L2 as the LOLT for content subjects and where a bilingual dictionary offers support for content subjects, as is the case with the BDX, teachers perceive positive impacts on content subject learning outcomes from using a bilingual learner's dictionary. This contrasts with Taljard, Prinsloo and Fricke (2011) who tested the ability of target users to benefit from a subject dictionary that included bilingual definitions; however, the dictionaries in that study were ‘of relatively low lexicographic achievement’ (2011, 93), which seems likely to have influenced the results. Figure 4 above shows that comments about a perceived impact on content subjects were made slightly less than half as often as those about language production or reception (33 vs 76 or 74 respectively), but that they are still amongst the most common comments. The subjects mentioned by teachers as being impacted range across the curriculum as follows: Economics and Management Sciences, Technology, Social Sciences, Life Orientation, Maths, Natural Sciences, and Creative Arts.

Figure 8 shows that most teachers in the study taught either other subjects at grade 7 or other grades, in addition to their teaching of a language at grade 7: in total, 24 teachers taught other grades or other subjects as well, while only 3 teachers reported teaching only a grade 7 language (data for 8 teachers were unavailable). This range of activity in the school, across subjects and grades, means that teachers who started using the dictionary with their grade 7 language class may also use it with other grades (higher or lower), or with other subjects they were teaching. The latter is a situation more likely to occur at primary schools, where teachers are commonly less specialised (e.g. teaching both language and content subjects), than at secondary or senior secondary schools (where a teacher might teach only one subject to several grades).

In addition to participant teachers who themselves perceive an impact on content subjects that they teach, seven participants reported that teachers from other grades used the dictionary, almost all (six) in the intermediate phase (grades 4 to 6).

The perceived impact of the BDX on content subjects may align with teachers perceiving that the dictionary helped them explain concepts (9 comments, Figure 6 above). Teacher comments appear to differentiate between the dictionary helping to explain concepts and the dictionary supporting vocabulary development, with the former suggesting a content subject orientation and the latter a language orientation, whether L1 or L2.

Finally, some teachers named specific results of the BDX's perceived impact on learning outcomes; these included:

- […] we have even improved in the activities of the Department of Education, like spelling competitions¹³

- The difference it has made is that learners have improved in English and even the schools we feed [i.e. high schools] have confirmed this and are praising us.¹⁴

- my learners can do summaries very easily. They can understand what they are reading about¹⁵

- We were doing analyses of June [mid-year exam] results and we have realised that most of the classes which are using this dictionary are doing very well in isiXhosa [i.e. L1]. English [exam results are] not very good, though their performance in assignments [during the term] is good. It shows that the dictionary was not there to support them on difficult words [in English in the June exams]¹⁶

- we are giving homework to our learners without being worried about language barriers. In the rural areas some parents cannot understand English to support their children. The dictionary is supporting them through the language they can understand [i.e. L1].¹⁷

This last comment links the dictionary to homework, although it is not clear whether the pupils are taking dictionaries home to support their homework or not. In better-resourced schools (such as in quintiles 4 and 5), it has been the norm for many years for parents to be asked to buy a dictionary for their child, often at the start of grade 4, for use at home or at school, since schools in these quintiles commonly do not provide class sets. In contrast, several teachers in the study report that they do not have enough dictionaries¹⁸ to allow pupils to take them home, with negative consequences for their ability to do homework. For example, one teacher commented: ‘when you give them homework on the meanings of words, they don’t cope because they don’t have dictionaries at home. As a result they don’t do the homework. In class, they cope and even find words on their own because of the availability of the dictionaries.’¹⁹

This raises the question of what influences teachers to give the amount of homework that they do. Teachers in low quintile schools may decide not to set homework or set undemanding homework (Hobden and Hobden 2019). This is important, since time spent on activities like homework has an impact on learner achievement (Reddy, et al. 2019). The frequency of homework, especially marked homework, may even be seen as a proxy for teacher effort (Burger 2011, 160), which in turn correlates with better learning outcomes. Participant comments suggest that it may not be that teachers lack the willingness to set and mark homework, but rather that teachers' expectations (for example, that pupils will not be helped with their homework by adults, and will not have access to tools (such as dictionaries) to tackle their homework successfully on their own) may lead to teachers' practices around homework. The description of pupils as ‘complicit in that they accept and encourage the very slow pace of instruction by doing minimal homework’ (Hobden and Hobden 2019) could therefore also be misplaced, at least in some schools.

7.3. How dictionaries were perceived to be used

Data was also gathered on how often the dictionaries were used (Question 10, Supplementary material), who used them, and when (whether in class or at home).

More than two thirds of participants (24 out of 33) reported that the BDX was used at least a few times a week, with 14 (40%) reporting daily or more frequent use. The dictionary was used by pupils (29 out of 33; 88%) as well as teachers (20; 61%), and was most frequently used in class. Only seven teachers reported that their pupils used the dictionary at home or for homework, with several²⁰ noting that the small number of dictionaries available to them made it impossible to allow pupils to take copies home (though some²¹ reported making photocopies, presumably of selected pages, for use at home). The results on these two issues—of who was using the dictionaries and where dictionaries were used—may be less reliable, because data on these issues was collected from the translated interview transcripts as a whole, rather than from direct questions.

7.4. Challenges in using the BDX

Figure 5 shows that 61 teachers experienced challenges using the BDX. Figure 9 shows that most of the challenges (28 out of 61) related to not having enough copies of the dictionary, while a further 19 participants found the BDX was time-consuming to use (12) or that their pupils lacked dictionary skills (7). The latter two types are arguably two sides of the same issue; therefore one might say about a third of teachers (19 out of 61) thought that pupils' lack of skills made their use of the dictionary slow. A further nine teachers thought either that the dictionary did not develop pupils' English skills (6) or that the dictionary's coverage of English was insufficient (3). Some of these expressed a preference for a monolingual English dictionary, e.g. ‘As English teacher, it is not really helping to me. I support it by using Cambridge monolingual dictionary. Only the content teachers are benefiting from it.’²²

It is perhaps unsurprising that a shortage of copies was the most frequent complaint considering that the median number of dictionaries was 10 (see endnote 18), that several teachers²³ reported that their pupils were sharing copies, and that nine teachers had fewer than two copies. Similar comments might be made about pupils needing time to acquire basic dictionary skills— for many, their use of the BDX seems to be the first time they are encountering a dictionary, while many teachers do not appear to have given pupils any overt dictionary training. Some teachers may themselves have weak dictionary skills, or may be struggling with the particular issues that isiXhosa lemmatisation practices (whether traditional or new²⁴) pose to dictionary users.

7.5. Limitations of the study and potential conflicts of interest

Due to the first and second stages of data collection being conducted at different times (2016 and 2018–19, respectively), the data from the two stages may not be directly comparable. However, the schools participating in both stages were drawn from the same group of schools that had ordered resources in late 2014, they met the same criteria, and the same interview instrument was used for both. The data have therefore been analysed together, not separately.

Quantitative data (such as tests from the period before the dictionary started to be used compared to tests from after the period of use, or tests of a control group compared to a treatment group) were not available to complement the data on perceptions of impact from interviews. This was mainly due to the testing of grade 7 only being undertaken at school level, not being moderated provincially or nationally, and therefore not being readily accessible. Because of the lack of moderation at a non-school level, quantitative data would also not be directly comparable across schools, even if it were available.

Candidate schools were not randomly selected to be candidates for participation in the study, but were rather selected due to their having ordered dictionaries — it is therefore possible that they may have been predisposed to consider dictionaries to have value, or be functioning more efficiently than schools with similar external characteristics. Participating schools were also not randomly selected; rather, their participation was largely determined by the ability of the research team to reach the school by telephone. Although in most cases this was achieved via a teacher or principal's cellphone, not a school landline, suggesting that participating schools were not well-resourced, it is possible that participating schools may function more efficiently than other schools with similar external characteristics, or may have more able, engaged or effective teachers.

Schools from nine out of 12 educational districts in the Eastern Cape took part in the research; unfortunately, three educational districts were not represented. This was an unintended consequence of the methodology chosen, and was not identified early enough to be addressed.

The interview instrument was designed to elicit both positive and negative comments, having been reviewed for bias by the National Foundation for Educational Research as part of their support for the development of OUP's Impact Framework at the time of the trial stage. However, social desirability may have influenced teacher responses, despite the danger of this being discussed with interviewers at briefing. Further steps taken to avoid social desirability were: informing participants that their answers would be treated confidentially; not rewarding teachers or schools for participating in the research, e.g. with vouchers, training, food or small gifts, as was sometimes done with research undertaken by publishers; and situating the research in an academic, rather than a marketing context, which teachers might otherwise have expected.

8. Discussion and conclusion

The wide-ranging nature of the perceived impact of using the BDX was a surprising aspect of the findings. The findings on the impact on L2 language production and reception are in line with the literature on the impact of bilingual and bilingual learner dictionaries, but the perceived impact on L1 was unusual. Reasons for this have already been suggested in Section 7.2, for example, the lack of access to suitable monolingual dictionaries. This is an important finding since the South African education department has a policy of additive bilingualism (Department of Basic Education 1997); reference materials and teaching practices that can support the maintenance or development of pupils' L1 are therefore in line with this policy and desirable. (Despite this official ‘additive’ policy, there is some evidence that, in practice, a form of subtractive bilingualism takes place, possibly due to the dominance of English and the lack of strong support of the L1 at school (Cummins 1984, 107). The result appears to be a loss of domains for the L1, exemplified in a participant teacher reflecting on how the BDX had helped her to reclaim the use of her L1: ‘I am not using English when I am trying to explain or to address our community. This is because I am confident of my isiXhosa vocabulary.’)

The impact on content subjects would also have been unusual in a more well-resourced and less unequal educational system, where a bilingual learner's dictionary would be unlikely to include a wide range of words related to content subjects. In South Africa, the very large majority of pupils studying content subjects through L2 English makes the inclusion of content subject terms a desirable feature; it appears also to be a feature that teachers perceive to be effective.

The perceived socio-emotional impact of the dictionary is also interesting. Although some studies have looked at the impact of allowing pupils to use dictionaries in exams (Tall and Hurman 2002) or have described teachers' frustration in a research setting at being required to use a monolingual dictionary instead of their preferred bilingual (Mandalios 2013), there appear to be few studies on the socio-emotional impact of dictionary use on language learners outside of exams, or on how bilingual dictionary use can impact teachers' attitudes towards, and feelings about, their work. Participant interviews here offer teachers a voice to express their experience of the classroom, for example, in comments such as this: ‘Teaching is a process; when you have proper teaching tools, it makes teaching easier and more interesting [emphasis added].’²⁵ Teachers further show themselves able to reflect on and have insight into pupils' experiences of the classroom, saying, for example: ‘Sometimes you may think that learners are not interested in reading but the challenge is they don’t understand what they are reading'²⁶ and ‘We know that our learners can think but not in English’²⁷, showing clearly the obstacle that this educational environment poses to those with a poor understanding of English, especially at a CALP rather than a BICS level.

Although this research has limitations, as explored in Section 7.5, it suggests that a bilingual learner's dictionary may have a wider range of impact in South Africa than previously expected, including in schools in lower quintiles, and without training of either teachers or pupils. Further research, especially if at least partly quantitative or mixed method, would be worth conducting.

It is easy, when faced with data from PIRLS, for example, showing the profound limitations to South African primary school pupils' understanding of what they are being taught, to forget that a lack of understanding is not just a sign of a poorly performing (if improving (Van der Berg and Gustafsson 2019)) educational system, but of a multitude of experiences of classrooms—classrooms where pupils ‘keep quiet in class’²⁸, and where teachers think that their time is ‘wasted’²⁹ because they do not have the tools they need to be effective. This research suggests that it is possible, even in poorly resourced schools and without special training, to experience instead a situation where ‘The learners are no longer memorising but they understand what is being taught by teachers. Both teachers and learners are enjoying the classroom.’³⁰

Footnotes

Acknowledgements

This research depended on a small team of freelance interviewers, chosen for their educational experience and expertise as isiXhosa speakers. They are Nontsikelelo Ntusikazi, Nomfundiso Mbali and Nomalungisa Ngondo. Also critical were the permission and help of the Eastern Cape provincial education department, especially Babalwa Pamla and Mark Hensberg; guidance and advice on impact from Penelope Woolf at OUP; and administrative assistance from Patricia Hofmeyer at OUP Southern Africa.

Supplementary material

The research instrument, as used.

References

A. Dictionaries

B. Other literature