Gathering data in research - an evaluation of reliability and validity
Primary and secondary data sources
A research instrument can gather data from either a primary or secondary source
Primary sources
Primary sources are original materials on which other research is based. They are not interpreted or evaluated. Examples of primary sources are data gathered by a questionnaire, statistics of population from the Government census or a first report of a research experiment and its finding
Secondary sources
Secondary sources are less easily defined than primary sources. Secondary sources interpret and analyse primary sources examples might be text books, or journals which review others work. Sometimes is it always easy to distinguish primary from secondary sources. A newspaper article is a primary source if it reports events, but a secondary source if it analyses and comments on those events.
Evaluating Reliability and Validity
In evaluating sources there are two elements reliability and validity. For a data source to be accurate and credible high levels of reliability and validity is the aim. Both elements are equal in importance in judging the accuracy and credibility of a source.
Reliability – Can the source provide the data?
For a source to be reliable we must evaluate the ability of the source to provide the information. We are looking at the question “Is it likely that this source can provide this data?”
The issue is therefore authority. To evaluate authority we can look at several aspects of the data source.
Author - Is the author an expert in the field? What qualifications do they have? For example an article on a website about HIV+ written by medical doctor might have more authority than one written by some one without qualification.
Professional standards. – Does the author have certain professional standards? The example of a doctor immediately comes to mind. Similarly academic writers who are published in academic journals or books have to conform to standards and have their work checked by other academics. Journalists mostly operate within a professional approach especially large international newspapers such as the New York Times or the Guardian (UK)Authority can mean expertise.
Publisher – Is the publisher reputable? Academic publishers need to maintain their reputation for accurate factual information so they also have editors to ensure a high standard. Other publishers such as newspapers, magazines etc need to avoid legal action for libel (telling lies about someone) so also should be careful to print the truth.
Organisation or Institution – If the data is from an organisation, for example the United Nations, we need to evaluate their reputation and their role or responsibilities. For example statistics on the economy from the East Caribbean Central Bank would come from a highly reliable source as the bank use the statistics to conduct the very important business of issuing bank notes and controlling the money supply in the region.
Research method – Could the research method chosen generate the data necessary? For example in researching teen pregnancy would carrying out an interview of an expert generate the data needed or would questionnaires of teens be a better choice.
Validity – Is the data true?
A source could have high levels of reliability. For example, academic research published in an academic journal by the leading expert in the field however the data may have a low level of validity in that it might be very out of date. Equally it may be possible that a source might not be considered highly reliable for example an intenet site which does not have the name of the author, organisation who maintains the site etc however the data is still true or valid.
In evaluating validity we need to look at accuracy and bias.
To evaluate accuracy we can look at several aspects of the data
Currency – When was the data published or gathered? Could the information be out of date? For example statistics on rates of HIV+ infection will need to be up to date to be accurate.
Relevance – Does the information relate to the circumstances you are applying it to? For example, will research carried out in the United States apply to the Caribbean?
Data collection – Was the data collected by reliable methods? Was it accurately recorded?
Sample size – Was the sample size large enough for generalisation to be accurate? For example if a newspaper article has only interviewed one person in a large crowd can we assume that all the points of view are represented? Similarly with social research the sample size is vital to judging whether the data is representative of the population as a whole.
Replicable – Do other sources have similar information? Would another similar piece of research have the same result? This is particularly relevant to sources such as the internet which lack references.
To evaluate bias we can look at:
Representation – Does the sample include all the variables within the population such as age, gender, social class, religion, education level which might affect response? Even with a large sample if the sample is not representative then bias in the data will occur.
Cultural bias – Has the data been collected by someone of the same or a different culture. For example, an Western researcher may misinterpret a non-Western culture and be biased due to racism or other factors. Similarly when researching within ones own culture, being subject to the same values and beliefs as the subjects may cause one not to question certain responses. For example when evaluating religious or other beliefs.
Political bias – Is the data being presented from either a right wing or a left wing perspective. The conservative agenda (e.g. free market economics, personal liberty above all other rights and fundamental religious views) will differ from the liberal agenda (e.g. some control of the market for social gain, social control for the good of society, religious tolerance for different views).
Social bias – Aspects such as gender, race, age and social class may affect the presentation of data. For example a women’s perspective on sexual equality may differ from a man’s views.
Faulty research methods – Even the best academic researchers can make mistakes and inexperienced researchers such as a student may have issues with poorly designed and executed questionnaires and interviews. Mistakes within the research method inadvertently cause bias. This is why academic research is reviewed by several other academics to evaluate the methodology and avoid bias in the conclusions or faulty conclusions.
Aim of the source in presenting the data – The reason for the data being presented will have an effect on bias. For example a Government might present certain statistics on economic performance if they are favourable and might avoid others. Whilst the data is valid, there might still be bias in that other relevant information is not present. If the source’s aim is persuasive again there may be bias. For example commercial sites wishing to sell products.
A research instrument can gather data from either a primary or secondary source
Primary sources
Primary sources are original materials on which other research is based. They are not interpreted or evaluated. Examples of primary sources are data gathered by a questionnaire, statistics of population from the Government census or a first report of a research experiment and its finding
Secondary sources
Secondary sources are less easily defined than primary sources. Secondary sources interpret and analyse primary sources examples might be text books, or journals which review others work. Sometimes is it always easy to distinguish primary from secondary sources. A newspaper article is a primary source if it reports events, but a secondary source if it analyses and comments on those events.
Evaluating Reliability and Validity
In evaluating sources there are two elements reliability and validity. For a data source to be accurate and credible high levels of reliability and validity is the aim. Both elements are equal in importance in judging the accuracy and credibility of a source.
Reliability – Can the source provide the data?
For a source to be reliable we must evaluate the ability of the source to provide the information. We are looking at the question “Is it likely that this source can provide this data?”
The issue is therefore authority. To evaluate authority we can look at several aspects of the data source.
Author - Is the author an expert in the field? What qualifications do they have? For example an article on a website about HIV+ written by medical doctor might have more authority than one written by some one without qualification.
Professional standards. – Does the author have certain professional standards? The example of a doctor immediately comes to mind. Similarly academic writers who are published in academic journals or books have to conform to standards and have their work checked by other academics. Journalists mostly operate within a professional approach especially large international newspapers such as the New York Times or the Guardian (UK)Authority can mean expertise.
Publisher – Is the publisher reputable? Academic publishers need to maintain their reputation for accurate factual information so they also have editors to ensure a high standard. Other publishers such as newspapers, magazines etc need to avoid legal action for libel (telling lies about someone) so also should be careful to print the truth.
Organisation or Institution – If the data is from an organisation, for example the United Nations, we need to evaluate their reputation and their role or responsibilities. For example statistics on the economy from the East Caribbean Central Bank would come from a highly reliable source as the bank use the statistics to conduct the very important business of issuing bank notes and controlling the money supply in the region.
Research method – Could the research method chosen generate the data necessary? For example in researching teen pregnancy would carrying out an interview of an expert generate the data needed or would questionnaires of teens be a better choice.
Validity – Is the data true?
A source could have high levels of reliability. For example, academic research published in an academic journal by the leading expert in the field however the data may have a low level of validity in that it might be very out of date. Equally it may be possible that a source might not be considered highly reliable for example an intenet site which does not have the name of the author, organisation who maintains the site etc however the data is still true or valid.
In evaluating validity we need to look at accuracy and bias.
To evaluate accuracy we can look at several aspects of the data
Currency – When was the data published or gathered? Could the information be out of date? For example statistics on rates of HIV+ infection will need to be up to date to be accurate.
Relevance – Does the information relate to the circumstances you are applying it to? For example, will research carried out in the United States apply to the Caribbean?
Data collection – Was the data collected by reliable methods? Was it accurately recorded?
Sample size – Was the sample size large enough for generalisation to be accurate? For example if a newspaper article has only interviewed one person in a large crowd can we assume that all the points of view are represented? Similarly with social research the sample size is vital to judging whether the data is representative of the population as a whole.
Replicable – Do other sources have similar information? Would another similar piece of research have the same result? This is particularly relevant to sources such as the internet which lack references.
To evaluate bias we can look at:
Representation – Does the sample include all the variables within the population such as age, gender, social class, religion, education level which might affect response? Even with a large sample if the sample is not representative then bias in the data will occur.
Cultural bias – Has the data been collected by someone of the same or a different culture. For example, an Western researcher may misinterpret a non-Western culture and be biased due to racism or other factors. Similarly when researching within ones own culture, being subject to the same values and beliefs as the subjects may cause one not to question certain responses. For example when evaluating religious or other beliefs.
Political bias – Is the data being presented from either a right wing or a left wing perspective. The conservative agenda (e.g. free market economics, personal liberty above all other rights and fundamental religious views) will differ from the liberal agenda (e.g. some control of the market for social gain, social control for the good of society, religious tolerance for different views).
Social bias – Aspects such as gender, race, age and social class may affect the presentation of data. For example a women’s perspective on sexual equality may differ from a man’s views.
Faulty research methods – Even the best academic researchers can make mistakes and inexperienced researchers such as a student may have issues with poorly designed and executed questionnaires and interviews. Mistakes within the research method inadvertently cause bias. This is why academic research is reviewed by several other academics to evaluate the methodology and avoid bias in the conclusions or faulty conclusions.
Aim of the source in presenting the data – The reason for the data being presented will have an effect on bias. For example a Government might present certain statistics on economic performance if they are favourable and might avoid others. Whilst the data is valid, there might still be bias in that other relevant information is not present. If the source’s aim is persuasive again there may be bias. For example commercial sites wishing to sell products.
Labels: module 1 research, presentation, reliability, SBA, validity
posted by Miz John at 7:20 AM
<< Home