Embedding plagiarism education in the assessment process

Lessons on paraphrasing and citing sources can only be partially effective if they are not perceived as immediately relevant to the individual student. We used electronic plagiarism detection tools to help students understand correct academic practice in using source material. In order to produce an essay on a specified topic, students were required to summarise a number of research papers. The 182 students who took part in this exercise were studying one-year Masters programmes in Computer Science, Automotive Engineering, and Electronics, mainly from China, India and Pakistan and new to the University. These students should have been building on previous study both in subject matter and study skills, but before they tackled the assignment, a series of lectures gave guidance on finding and summarising sources, and reminded students about what constitutes plagiarism. The students' essays were submitted to Turnitin and Ferret -- a straightforward, but resource intensive process -- and the resulting reports used to give individual feedback to students on how original their words appeared to be. This was effective in helping the students to understand plagiarism, because the reports identified plagiarised passages in their own work. Using a threshold of 15% of matching text, we found 41% of students had submitted work identified by Turnitin as possible plagiarism but this reduced to 26% on inspection by academics. After a second submission, incidence of plagiarism dropped to 3% overall. We found that the degree of matching text found correlated with a student's programme of study, but not with nationality.


Introduction
One of the strategies that a university can use as part of its approach to the problem of plagiarism is to deploy electronic plagiarism detection software that can identify identical passages of text in one or more documents.This software can be used to verify a lecturer's suspicions that a particular piece of work is plagiarised, or it can be used to process whole batches of coursework to enable the screening of submissions from entire groups of students.A demonstration of one of these software tools can also be effective in deterring students from plagiarising.
We have investigated another way to use such software, namely to directly link the use of the reports produced by the detection tool to the education of the students in the correct way to use source material.Electronic detection can also be very effective in detecting collusion, but that is not the focus of this study.

Perceptions of plagiarism detection
In a trial at the University of Auckland (Gulik & Tippin, 2003), 60% of instructors estimated that Turnitin would be effective in improving student referencing or writing practices.On the other hand, Savage (2004) found that students were worried that the use of electronic detection would not differentiate between "inept citation" and "intentional plagiarism", though they also felt that its use would be an incentive to improve citation skills.Students can fear that they may unwittingly use ideas that they have adopted as their own (Ashworth, Bannister & Thorne, 1997) and be unclear about what is common knowledge (Park, 2003), and in these situations accidental or unintentional plagiarism may be judged to be deliberate plagiarism.Unintentional plagiarism can also be through poor citation practice, inadequate paraphrasing or summarising, and vague referencing.This may be more of a problem for international students than home students because of cultural differences and the requirement to write in a second language.In a survey of 42 international students Errey (2002) found that most had a theoretical understanding of plagiarism, yet 69% of these felt that cutting and pasting passages of text was acceptable if the source was correctly cited.Lake ( 2004) describes cultural differences in Chinese writing, and found that more than half of the Chinese students he surveyed had no previous experience of acknowledging an author in academic writing.Cases of unintentional plagiarism, even with first year undergraduates, should be brought to the student's attention (Errey, 2002), because, as Caroll (2002) states, "delaying action on plagiarism is probably ineffective as well as misleading".
In contrast, for a number of students, cheating is an acceptable option (Underwood & Szabo, 2003) with fear of failure and time management problems given as reasons that a student would cheat.For other students, plagiarism is "no big deal" (Park, 2003).In a scale of offences, copying without reference comes out as not very serious (Dordoy, 2002;Franklyn-Stokes & Newstead, 1995); for these students, the risk of detection would be a deterrent (Savage, 2004).

Method
Our study arose from a belief that 'telling' students about plagiarism and collusion is not enough.The students do not relate to these issues until they need to be applied to an assessment, and even then the advice should be personalised.This piece of action research was carried out by submitting a complete batch of students' work to electronic plagiarism detectors and showing each student the detection report produced for their work.The students whose work was found to be plagiarised were given a second chance at the assessment.The incidence of plagiarism found at the first and second submission was measured.Some analysis of the students' previous educational background was undertaken and related to the plagiarism found.
The limitation of the study is that it is not possible to give a quantitative measure of the effectiveness in terms of subsequent attempts to avoid plagiarism; for example by comparing this cohort of students with previous years, because there are many variables that can affect the detection rate.The plagiarism detectors can also find evidence of collusion between students, but in the event, no collusion was found in the assessment undertaken as part of the study.We focus on plagiarism from Internet sources.This exercise is similar to that carried out by Culwin (2003) with first year undergraduates, which found that the primary purpose of involving the students in a practical activity to educate them about academic misconduct appeared to have been satisfied.

Participants
The 182 students who took part in this assessment exercise were studying in a one-year Masters programme, either in Computer Science, Automotive Engineering, or Electronics.They were from a variety of backgrounds, both in their previous degree and country of origin, so we could not assume a common understanding of plagiarism issues.The students were predominantly international students, mainly from China, India and Pakistan, with small numbers from other countries including the UK.The experience from previous cohorts on this programme is that there is a wide variation in ability to write an academic dissertation.

Design of the assessment
These students took a compulsory "Research Methods" course, in which one of the first assessments was to write a short summary on one of a number of given topics, based on journal papers and conference proceedings.Marks were to be awarded for the selection of literature to be reviewed, and for the production of a clear, well-structured report that demonstrated the student could critically review, summarise and use references appropriately.Ideally this should have been a totally formative assessment, but we chose to allocate marks worth 10% of the overall assessment to ensure all students took part and could benefit from the experience, especially in terms of the opportunity to understand what constitutes plagiarism.

Procedure
Before students tackled the assignment, a series of lectures introduced the techniques of information searching and retrieval and gave guidance on summarising information from sources.The expectation was that these students were building on their previous degree in both subject matter and study skills, and that they had carried out some report writing on their previous degree.Students were also reminded about what constitutes plagiarism but were not told that their work would be submitted to a plagiarism detector.It is worth noting that most staff would consider that this was more than adequate warning, and that any plagiarism would be penalised.
Once the students had submitted the assignment, these were processed through the electronic plagiarism detectors Turnitin (1998), and Ferret (Lyon, Barrett & Malcolm, 2004) to produce a report for each student on how original their words appeared to be.Our plan was to combine these tools in order to get a more fine-grained measure of how much text re-use was taking place.The Turnitin product was good at relatively large-scale verbatim quotations from the Web and other sources.We were using the U.K. JISC Plagiarism Advisory Service's detection facility (JISC 2004) which, as well as the Web, has a substantial database of material contributed by staff and students at U.K. institutions.In contrast, Ferret will detect a quotation as short as three words, but has no database: it compares files submitted in one detection run and is good at detecting collusion.We therefore used Turnitin to identify the sources that students had used, downloaded these, and then used Ferret to identify overlaps between a student's submission and any of the sources.The aim was to detect even moderately reasonable paraphrasing, and particularly to avoid revealing to the students just how little needs to be changed to avoid detection by Turnitin.During a subsequent tutorial, students were given feedback on their work, both on the academic content and their ability to use sources correctly.The visual reports, with sections of plagiarised text highlighted in a different colour, were effective in helping the students to understand plagiarism because the reports identified passages in their own work.Students who had plagiarised were given the chance to resubmit and the detection process was repeated.The maximum mark that these students could achieve was capped at a bare pass so that they would not be advantaged by the resubmission opportunity.

The extent of plagiarism found
A threshold of more than 15% of the work matching Web sources (or other students' work) was used.This percentage was determined by inspection of reports to identify where the borderline cases occur.Any threshold is somewhat arbitrary: the CAVAL study (quoted in Carroll, 2004) used a higher figure, 25%, as a threshold, and screened 1770 pieces of work from five Australian institutions finding just 8.8% of unattributed Web-based material.Turnitin grades work by the percentage of text that matches another source, colour-coding the matching text.We found that 74 out of the 182 submissions (41%) were above the 15% threshold.On inspection of these 74 Turnitin reports, it was found that a number could not be considered to be plagiarised for various reasons: perhaps the text was properly in quotation marks, references were a good proportion of the overall word count, or students had included parts of the assignment specification.The final count of plagiarised work which was passed to the tutors was 26% of total submissions.
In addition, five students who didn't carry out the assignment the first time were allowed to submit with the second submission, with the interesting result that three of those five students submitted plagiarised work.Ignoring these five, the results are shown in Table 1 below.This shows the percentage of students who submitted work that was identified as unacceptable, either because of unattributed work or because of inadequate citation, in each of the three subject areas.These percentages are expressed as a fraction of the total number of submissions in a subject area.The final column shows the overall percentage of students found plagiarising on the first and second submissions.The experience The process was straightforward, but fairly resource intensive.One member of staff downloaded the files from the University's Managed Learning Environment, checked submissions against class lists, removed duplicate submissions, processed late submissions, submitted the files to Turnitin, and colour printed all reports.Two members of staff made the initial decisions on whether the similarity found by the detection tools represented a problem or not.This was done to minimise the workload on individual tutors, but it also provided consistency in the treatment of plagiarism, whether intentional or not on the student's part.
One thing to note is that Turnitin can only give a measure of possible plagiarism; on inspection there were many false positives.In many cases it was found that in students' bibliographies, lists of references partially matched those in other documents on the web.We also found properly quoted and cited extracts from journal and conference papers that are available as Web documents, and many students had included parts of the assignment brief, and this too was flagged by the software.This 'downside' to Turnitin has been identified elsewhere (Evans, 2004;Gulik & Tippin, 2004) and suggests that the use of automatic detection could result in even more work rather than less for lecturers.
The three parallel subject areas used different methods to return the students' marked work and the detection reports.In one cohort all the work was returned in a classroom situation, but unfortunately the full advantage of the opportunity could not be taken because of lecturers' reticence to identify 'bad' students and these students had to be followed up after the tutorial.For the second cohort the work and the reports were given back outside the tutorial, but this had the disadvantage that not all students then attended the subsequent tutorial; further evidence of the general problem that some students are only interested in their mark rather than feedback (see, for example, Hughes, 2004).For the third cohort, the students who had plagiarised were asked to an interview with the tutor before the tutorial.Some students who may have been prepared to deny that there was a problem capitulated when they saw the evidence.The disadvantage of this method is that, without care, the emphasis will be on plagiarism detection rather than student education.Despite these difficulties, the tutorial in which 'cutand-paste' writing and plagiarism detection were examined was successful, with students being able to relate the discussion to their own work rather than more detached exhortations about plagiarism.
Structured interviews were held with a number of the tutors to ascertain their views on the value of the exercise.The overwhelming response was enthusiasm for tackling the students' difficulties, against a backdrop of some despair, both at the extent of poor writing skills and at the plagiarism that results either from this inadequacy or from a deliberate intention to cheat.The exercise provided a united front in tackling potential plagiarism; we note that peer-support for lecturers has also been found helpful in programming assessments (Barrett & Rainer, 2004).

An analysis of students' backgrounds
As can be seen from Table 1 above, it seemed that the Computer Science students were more likely to use others' words in their work than those from Automotive Engineering or Electronics.Some of the possible reasons for this finding are that the Computer Science group might be academically weaker students, that they are more adept at using the Internet, that a greater proportion of academically relevant Computer Science material is available on-line, or that less Automotive Engineering and Electronics material is in the Turnitin database.
We also investigated the students' previous backgrounds, looking at previous educational institutions.
We checked the previous education of the Computer Science students, and grouped them by nationality.We observed that the Indian, Pakistani, and Chinese students had been found to have plagiarised in roughly equal proportions, but we did note that the others (which included home students) showed a lower incidence of unoriginality.
Although the number of home students in our sample is rather low, it did seem from our results that overseas students were much more likely to use unoriginal text than home students.With this in mind, we wanted to see whether previous study in U.K. higher education reduced the degree to which students used unoriginal text in their answers.Most of the Indian and Pakistani students had come direct from bachelor's degree study in their home countries, but it turned out that amongst the Chinese students there were two reasonably large sub-groups who had previous educational experience in the UK.Of the 22 Chinese nationals, 12 had come direct from China, but nine had done one or more years of undergraduate study in the UK and 11 had done a bridging course of between three and 12 months duration.The results are shown in Table 2: These numbers are quite small, but one can observe that the students who came direct from undergraduate study in China are far less likely to plagiarise than those who did their undergraduate study in the UK.
There are many possible explanations for this unexpected and somewhat unwelcome finding: it could be that those students who do their undergraduate education in this country, and then stay on for a M.Sc.are less strong academically than those who come to this country to do their masters degree, having successfully completed their undergraduate education in China.But it certainly puts a question mark over cultural explanations of plagiarism amongst overseas students.It would appear that the experiences that this particular group of students had in their U.K. undergraduate courses did not had an effect on their ability to correctly acknowledge their sources.We suspect this confirms the observation we quoted in our introduction, that "delaying action on plagiarism is probably ineffective as well as misleading" (Carroll, 2002).

Discussion
This exercise was an addition to the teaching plan of the course.The lectures, tutorials and type of assessment were largely unchanged, including the lectures and the plagiarism prevention resources available to students.In previous years there had been a high incidence of plagiarism detected amongst students in this programme, including some in the final dissertation.It was intended that the emphasis on plagiarism education rather than detection would allay students' fears of accusations of culpable plagiarism through their inability or ignorance.As one colleague put it, being a weak student is not a disciplinary offence.In the event, we had very positive informal feedback from the students.
There is a rich source of plagiarism help for students on the Internet, but perhaps many of the exercises on paraphrasing and citation do not seem as relevant to technical students as this exercise appears to have been.We also suspect that there was a strong element of deterrence in seeing the effectiveness of electronic detection at first hand, shown by the improved reports on the second submission.The final overall results for this cohort, including those for the final dissertation, are not yet available, but despite our optimism we regret to say that some students who successfully negotiated this exercise were found to have plagiarised in a later assignment.
A high percentage of students had submitted work that was unacceptable, but most of these students were new to U.K. higher education, and most were from overseas.We believe that for the majority of these students this was due to difficulties in writing in their own words or in quoting sources correctly.The students were post-graduates, but in technical disciplines in which reports and essays may not have been the predominant way of assessment.A review of the work showed a variety of inadequate approaches to writing.The best known is 'patch-writing' -taking a little bit from here and a little bit from there to make an often incoherent whole.We also identified what we call 'omissionparaphrasing' where the student uses a single source for a large section of their work, and selectively changes just a few words, and cuts out whole sentences or paragraphs to make the work fit the assignment specification.It is likely that in many cases students feel that they have done an adequate job of paraphrasing, even though over 90% of the words they present in a particular section may be from the same source.More likely to be culpable is the 'bibliography-catch-all' where chunks of text are copied, and a reference is included, but the student fails to make clear how much is quoted and how the exact source material can be found.
After some of the students were required to repeat the exercise, producing a second report which was analysed in the same way, far fewer were found to have a substantial amount of copied text.This could be interpreted as an indication that students can work out how to apply referencing techniques from the teaching and lecture notes (together with the example from their first submission).Just 3% of the group was unable to write a sufficiently original report even after a second opportunity.For these students, the assessment exercise served as a 'wake-up' call, and for all students it reinforced the ideas of academic integrity in assessment.

Conclusion
The use of electronic plagiarism detection in a formative assessment can help in two ways.Firstly, plagiarism education is made relevant to the individual student by using their own work as an example; students who have plagiarised can be given a chance to redo the work.It may also be that a demonstration of electronic detection of plagiarism acts as a deterrent.Secondly, staff do not need to fear the possibly draconian consequences if they identify that students have submitted text which was not written by them.By operating in a team, a consistency of approach is ensured, and no one is tempted to minimise the seriousness of the problem.
As a by-product of this exercise, we were able to investigate whether any particular group of students was more prone to text re-use and poor citation practices than any other.Work needs to be done to more completely analyse this data, but clearly some surprising findings have been uncovered.At this stage the main observations are that there were quite large differences between subjects, but little difference from one nationality to another.We also conclude that the ranking of reports by amount of plagiarised work through an electronic plagiarism detector needs to be treated with caution, and manual inspection is essential.

Table 1 :
Percentages of plagiarism found in each subject area, and overall, after the first and second submissions

Table 2 :
Percentages of plagiarism found in the submissions of Chinese nationals, grouped by previous study