Review of Appraising Research in Second Language Learning
|Date: Fri, 09 Jan 2004 16:38:59 -0500
From: Louise Manga
Subject: Appraising Research in Second Language Learning
Porte, Graeme Keith (2002) Appraising Research in Second
Language Learning: A Practical Approach to Critical Analysis of
Quantitative Research, John Benjamins Publishing Company, Language
Learning and Language Teaching 3.
Louise Manga, Canadian Forces Language School, CFB Borden.
This book is designed for students of applied linguistics and teaching
English as a foreign language (TEFL) and for practising language
teachers. It is intended to be the main textbook or a supplement to a
"research-techniques" book for undergraduate and graduate students so
they can critically read and evaluate research. (The book assumes the
reader knows the basic principles of research and common statistical
The book examines the different components of a research paper and
demonstrates how to critically analyze a research paper using examples
of research in learning a second language. Readers are then helped to
do their own critical analyses by the guided samples in the workbook
section. Porte hopes that "the experience of appraising in this way
will help the reader better to present their own work for publication
and peer evaluation."
The book consists of a Textbook (pp. 1-149) with four chapters:
Introduction, Method and procedures, Results, Discussion and
conclusions; and a Workbook (pp. 151-229) with worked out sample
appraisals and samples for guided appraisal. The book begins with a
Preface which introduces the aims of the book, presents the methodology
used in the book, and outlines the organization of the book. It ends
with a Glossary of key quantitative research terms and three Appendices.
The division of the Text into four chapters parallels the four basic
parts of a research paper. To help the reader evaluate the logic and
consistency in a research paper, Porte uses an "awareness-raising
methodology" whereby he asks readers to question, reflect on, predict
and summarize what they have read. Porte often summarizes what he has
said and relates it to material already covered or to be covered.
Throughout he introduces a topic by asking questions to get the reader
to think about the important issues and then he answers the questions.
Porte also provides a variety of examples from second language learning.
Chapter 1 Introduction lays the foundation for the reader to better
evaluate the completed research. It covers: what to look for in an
abstract; how to find and evaluate the background to the problem, the
problem statement and the variables chosen; whether the review of the
literature is adequate theoretically and empirically for the hypotheses
and/or research questions formed; and what to look for in research
questions and hypotheses, the appropriateness of the variables and if
operational definitions are given.
Chapter 2 Method and Procedures deals with "the nuts-and-bolts of the
research design", i.e., the appropriateness of the methodology and what
kind of confidence it gives to the results. This chapter considers
subjects and materials, procedures, and research design and data
analysis in order to appraise the reliability and validity of the
results, and the replicability of the study. Porte discusses internal
and external validity, and reliability of the subjects and materials.
Issues of subject selection, group assignment, observation of subjects
and instructions given are discussed. Different types of research
design are described and as the researcher does not usually mention what
type was chosen, the reader must be able to determine the type and its
suitability. Porte also discusses the assumptions behind the different
types of statistical analyses.
Chapter 3 Results looks at how to assess the results of the research.
The chapter covers how the data might be presented, what the reader
should look for in the presentation of results to determine how much
confidence to place in the results of the study and the statistical
measures used. Porte describes when to use and how to interpret
commonly used tests in second language research: correlation,
regression, t-tests, analyses of variance and chi-squared.
Chapter 4 Discussion and Conclusions covers the quality of the
discussion and conclusions in the research paper. It shows how to check
that any conclusions are consistent with the results, and how to
evaluate any interpretations, generalizations or stated implications.
The Workbook uses two fictitious sample research papers based on
quasi-experimental studies which would be representative of the type of
research in second language learning. Both papers are separated into
sections which correspond to the sections in the Text. The section from
sample 1 research paper has a full analysis by the author and is
followed by the corresponding section from sample 2 research paper for
the reader to analyze.
1.1 Abstract Ii Abstract 1, 2
1.2 Problem Iii Problem 1, 2
1.3 Literature review Iiii Literature review 1, 2
1.4 Research questions Iiv Research questions 1, 2
2. Method and procedures
2.1 Subjects, materials IIi Subjects,materials 1, 2
2.2 Procedures IIii Procedures/Design
2.3 Design and analysis and analysis 1, 2
Nature of findings III Nature of findings 1, 2
Analyses of variance
4. Discussion and conclusions IV Discussion, concl. 1, 2
Beside each sample text there is a column on the left for the reader to
summarize the gist of each paragraph and a column on the right for the
reader to record any thoughts or questions that arise during the reading
of the text. (Note that only I and IV require the reader to summarize
the gist of the text.) The reader should then re-read the text and
answer a series of leading questions which are closely related to the
material in the corresponding textbook section. Certain words or
sentences in the sample text have been numbered and there are
observation questions related to these for the reader to answer.
Sample 1 in each part has been fully analysed: paragraphs are summarized,
thoughts recorded, questions answered and observations answered.
Sample 2 in each section is for the reader to complete. Porte provides
guidance by asking relevant questions with prompts for helping to answer
them, and with observation questions about words or sentences in the
text that he has numbered.
Appendix I is a flow chart comparing what the research is trying to
discover, the related tests used and how to interpret the results. This
flow chart is "reprinted from Hatch, E., and Lazaraton, A. 1991, The
Research Manual. New York: Newbury House Publishers, pp. 544-545."
Appendix II is a table comparing assumptions in statistical tests. It
is "adapted from Brown, J.D. 1992. Statistics as a foreign language:
Part 2. Tesol Quarterly, 26, 4, pp. 629-664." Appendix III contains
the statistical tables referred to in the text.
I think this book fulfils its two fold aim of developing critical
readers and helping researchers do better research. It would be a
useful reference for classroom teachers who would like to do research on
teaching methods or materials.
The Text has been organized to correspond to the major sections of a
research paper, which is an excellent way to present Porte's methodology
and to develop critical thinking about reading research papers. The
only problems I had with the Text were some typo errors and the choice
of grey for highlighting.
There are two serious typos. On page 51 line 17, "Content validity" is
described as "a more subjective and formal evaluation" than "Face
validity". It should read "Content validity is a more objective and
formal evaluation". On page 91 line 24, "negative correlation" is
described as "(i.e., no linearity)". It should be described as "(i.e.,
no positive relationship)" or "(i.e., one variable decreases as the
other increases)". (Note that this error is not in the Glossary.) Other
typos are on page 48 line 10 "Materials - Internal and External
Validity" should be "Materials - Reliability"; on page 50 after line 21
and just before the questions in bold, "Materials - Validity" should be
inserted; and on page 41 line 17 "be" should be added after "this kind
I question the use of grey highlighting for terms to be found in the
Glossary as grey is lighter than the rest of the text and doesn't stand
out. Grey highlighting was perhaps seen as an option as sometimes words
in the text are in bold (e.g., p. 110) and sometimes underlined (e.g.,
pp. 16, 204). One glaring omission from highlighting and the Glossary
is in the section about the validity of materials pp. 50-54. Four types
of validity are described but only three are in grey and in the
Glossary. "Predictive validity" on p. 52 line 1 should have been in
grey and in the Glossary. There were a few other errors with the use of
grey highlighting. "Kendall's tau" p. 109 was the only test mentioned
that was not in grey nor in the Glossary. There was one term, "t crit"
(table on p. 115) that was in grey but was not in the Glossary.
This section is very thorough with good directions at the beginning.
The questions and answers reinforce the discussion in the Text and help
you to approach the research topic analytically. Each section of the
Workbook would best be done after reading the corresponding section in
the Text. It would be useful to have had the contents of the Workbook
listed in the Table of Contents so readers could see the relationship of
the Workbook to the Text before beginning. For those interested in
accessing more guided appraisals, Porte provides some on his website
(see Preface for his website). There are two typos: p. 195, last line,
first word should be "due" not "down"; p. 196, line 21, insert "be"
after "subjects are to".
This is a very useful part, however there is one glaring omission -
predictive validity. "Predictive validity" was not highlighted in grey
in the Text (see above) so was not included as a key term. "Predictive
validity" should also be mentioned in the brackets at the end of the
terms "Construct validity", "Content validity", "Face validity",
"External validity" and "Internal validity". Also the key term
"Continuous data measurement" should show "(non-continuous)" in the last
sentence after "A variable that is not continuous" since
"non-continuous" is a key term.
These are very appropriately included and will be very handy references
both when reading research papers and conducting one's own research.
| ABOUT THE REVIEWER:
ABOUT THE REVIEWER
Louise Manga holds a Ph.D. in linguistics from the University of Ottawa,
Canada and is currently teaching EFL to adult foreign nationals. Her
interests include second language acquisition, phonology, syntax,
semantics and corpus linguistics. She has done linguistic fieldwork on
Inuktitut, including first language acquisition. She has co-authored
bilingual English-Mongolian books on North American English
pronunciation and on English grammar. She has also prepared learning
kits to help students with individual problems.