Qualitative evaluation
Source evaluation involves evaluating a source based on its type, authenticity, credibility and authority. The quality of a source may be judged by questions like who it is written for, how it refers to other sources, who wrote it and when it was published. Evaluate the source you have in front of you based on:
Form a view on the basis of your assessment as to whether the source is useful for your project. In what way will you use the source? How and why do you need to use it? Assessing relevance.
Author
All sources must be evaluated in the light of the authors’ scholarly background. Are they specialists in the area? Or do they have another background? These are questions you need to answer before using the source in your thesis. If you already have a relevant reference to an author, you can investigate whether:
- They have written other material on the same topic – search for the author’s name in Oria or other databases.
- Search for articles and books that are referred to in the source you are evaluating.
Look at the list of references at the end of the text. Could these articles and books also be relevant for you? - Do a citation search to find out who has used the source in more recent research. Citation searches can be carried out using the ISI Web of Science database.
- If you want to find out who has cited a particular work by an author you should search for a combination of the author’s name and the work (abbreviated article or book title) and the year when the work in question was published.
The sources of the text
Have a look at the reference list of the text you are reading. It will show what sources the author has used in the text, i.e. what kind of information the author uses to support their arguments and conclusions on. Evaluate to what degree the author supports their argument through empirical data, theory and method.
- To what extent does the author refer to other sources?
- Who is referred to?
- Are the results verifiable?
Also read: Argumentation in text.
Audience and genre
Evaluate each source with regard to the intended audience. All texts are written with a reader in mind. This means that the imagined reader participates in the writing of the text. One way of identifying the intended audience of a text is to analyse its genre. In the following we discuss examples of different genres.
Textbooks
A textbook is a type of text written to introduce students to an academic discourse or to a new subject. A textbook will describe theories and historical developments within the field, with the aim of including students within the particular academic community. A textbook will not say what is “most important” in the field, but the author will often devote most attention to that which lies closest to their own interests.
Theses
A thesis is a substantial scholarly work limited to the discussion of a particular topic in the field. Theses are subject to stringent requirements concerning theory and method and are expected to describe new research results. Examples of these types of texts include doctoral theses and, to some extent, master’s theses. The author writes the thesis to be accepted into an academic community. A thesis is therefore primarily written for other researchers in the field.
Scholarly research papers
A scholarly research paper is a text in which the author investigates a particular topic or problem in depth. The author will be evaluating a problem or arguing for or against a point of view. A scholarly research paper must present new insights and should be in a form that allows the results to be verified and applied in subsequent research. Researchers use scholarly research papers to disseminate research results both within their own field and to researchers in other academic fields.
Literature reviews
A literature review is a research paper that summarises a topic in a particular subject area by discussing a selection of the published literature. The paper presents a summarising analysis of the topic, and is intended to make it easier for other researchers to get an overview of the most important developments in their field.
Popular science articles
Like scholarly research papers, popular science articles will contain new insights and research-based knowledge. Unlike a scholarly research paper, however, a popular article will aim to disseminate new knowledge to a wider audience beyond the scholarly research community. Popular articles often have a journalistic style. You will find examples in op-eds in newspapers, and in magazines such as New Scientist. Such articles are subject to less stringent requirements concerning verification. Requirements concerning empirical data and the substantiation of results are also less stringent and, as in the case of essays, there will be frequent use of metaphor.
Encyclopaedias
Encyclopaedias contain scholarly, quality-controlled articles and background information about a wide range of topics. Articles are collated by editors specialising in specific subject areas and are accompanied by excellent lists of references. Encyclopaedias exist both in print and online. Wikipedia is a free online encyclopaedia to which anyone may contribute. It can provide a useful starting point when looking for general information about a topic. You should always check what you find on Wikipedia against quality-controlled sources.
Tips!
Are there any specialist encyclopaedias in your subject area? If so, use them!
Op-Ed
An op-ed (from «opposite the editorial page») is a relatively short newspaper article that may be informative or argumentative in nature. Researchers often use op-eds to disseminate their research to the public. See more on op-eds on Wikipedia
Tips!
Use newspaper archives such as that maintained by the National Library of Norway or use the A-tekst digital archive to find op-eds in Norwegian newspapers.
Essays
An essay is an article written from the author’s personal point of view. The text will clearly bear the authors’ imprint and will express their opinions. Authors may recount personal experiences, tell anecdotes and use metaphors by way of illustration. Essays are often intended to entertain the reader and may use associative expressions. The essay is a genre that lies somewhere between scholarly writing and literary prose.
Blogs
Blogs can be useful for keeping track of discussions in different fora. You can subscribe to blogs as RSS feeds. Make sure you know who is responsible for the blog. Blogs often take the form of a daily journal and reflect personal experiences and opinions. Claims made in blogs will not necessarily be verifiable or properly documented.
Tip!
Are there any scholarly blogs in your subject area? See for example Language Log and the Nature News Blog.
Publisher
The publisher may have influenced the content of the material you have found.
Maybe the publisher wished to advance its own interests or simply make the publication more attractive to potential readers. In many cases it is easy to identify the publisher. The name of the publisher of a book is usually printed on the front or back of the title page. The publisher may more or less well-known. Sometimes it may be difficult to establish the identity of the publisher. If you have never heard of the publisher, finding out information is even more important.
- Check the name of the publisher
- Search in library catalogues or bibliographical databases to obtain a representative impression of the types of text the publisher publishes
- Look for comments about the publisher
- Search for the publisher’s name using a search engine in combination with the following search terms: homepage, about us, contact, official
- Look for a list of editors who reviewed the text prior to publication
- Look for descriptions of the types of materials that they are interested in and the topics they usually cover
- If the source is an academic journal, find out whether its contents are peer reviewed
Publication date
The date of publication will often be relevant when evaluating a source. Often you will be looking for up-to-date information about the latest research in a field. Make sure that you have the latest edition of a book if it has been published in several editions. At other times you may be looking specifically for older information. For example, you may be looking for information published during a specific historical period or during a specific year.
When did the source originate?
- Look for a book’s date of publication by consulting the colophon (usually on the back of the title page).
- Establish which edition you have. Is it a newly revised edition or a reprint of an earlier edition?
- Look at the dates of the references used in the source.
- If you are evaluating an online source, find out when the information was published and when it was most recently updated.
- Make a decision about the significance of the date of the source for the purposes of your dissertation.
Peer review
A peer reviewed article is reviewed by at least two scholars within the same field of research before it can be accepted for publication. The author of the article will receive comments about the text as well as specific suggestions regarding improvements that must be made before the article can be published. This process is meant to ensure that articles are of acceptable quality so that the article complies with prevailing academic and scholarly/scientific standards within the field. When searching for literature in journal databases it may be a good idea to limit your searches to peer-reviewed articles.
Non-peer-reviewed articles can also be relevant and of high standard. You should always evaluate your sources, regardless of whether they have been through peer-review or not.
Impact factor
A journal’s impact factor reflects the average number of citations to articles published in the journal during a particular year. The greater the number of citations, the higher the journal’s impact factor.
Accordingly a journal’s impact factor is a quantitative ranking tool. The factor is based on a formula developed by the Institute for Scientific Information (ISI) – now Thomson Reuters. Rankings are published annually in the Journal Citation Report.
The impact factor for 2004 for the journal GLOBAL BIOGEOCHEMICAL CYCLES is calculated as follows:
$$\text{Number of citations in 2004 to articles from 2003 + Number of citations in 2004 to articles from 2002} \over \text{Number of articles published in 2003 + Number of articles published in 2002}$$
If we insert data from ISI Journal Citation Reports into this formula we get:
$${288+468 \over 121+143}={765 \over 264}=2.864$$
The journal’s impact factor for 2004 is thus 2.864. In other words, articles in this journal were cited on average two to three times during 2004. In practice, however, some articles will have been cited far more often, while others may not have been cited at all.
You can find more information about the impact factor and other ways of assessing a journal’s impact (such as the ‘Immediacy Index’, ‘Cited Half-Life’, and ‘Citing Half-Life’) in the ISI – Journal Citation Report.
Use of the impact factor:
The originators of the impact factor created it as a tool for comparing different journals. In today’s competitive environment, however, it plays a role in
- research funding
- recruitment to academic positions
- university rankings
- journal evaluations
- helping researchers decide which journals to publish in
Such uses of the factor have a self-reinforcing effect and a journal’s impact factor may seem artificial.
Moreover, a journal’s impact factor is a marker of its quality in general – it says nothing about the quality of an individual article published therein. Using the factor as a basis for evaluating a research paper may therefore give an incorrect impression.
Other criticisms:
- The ISI is solely responsible for deciding which journals to include in their index. In practice, far from all journals are represented.
- 97% of the journals are in English and are grounded in Western research traditions.
- Journals can achieve a higher impact factor by encouraging self-citation (i.e., authors cite other articles published in the same journal).
- A journal’s impact factor will be affected by prevailing citation practices in its particular field. One should not compare journals in different fields on the basis of their impact factors.
- Newer areas of research are cited less than established fields.
- Using impact factors as a basis for evaluation allows these factors to influence the direction of research and cause important but unfashionable areas to lose out.
- Journals that publish review articles are likely to be cited more frequently and thereby achieve a higher impact factor than journals that only publish original research.
