Collaborative Knowledge Building in Web-based Learning: Assessing the Quality of Dialogue

Elsebeth K. Sorensen (eks@hum.auc.dk)
Dept. of Communication; Aalborg University; Kroghstraede 3;
DK-9220 Aalborg Oest; Denmark;.Tel. (+45) 9635 9077; Fax. (+45) 9815 9434

Eugene S. Takle (gstakle@iastate.edu)
International Institute of Theoretical and Applied Physics; Iowa State University; Agronomy Hall;
Ames Iowa 50011 USA; Tel. (+1) 515-294-9871; Fax. (+1) 515-294-2619

Abstract:
This paper addresses the question of how to qualify a knowledge building dialogue in web-based collaborative learning. It investigates the idea of qualifying the dialogue through grading requirements and through providing students with meta-awareness around their dialogue. We investigate and analyze the changes in quality of student interaction in three different deliveries of the course that emerge in response to three different types of instruction on expected behavior. From this we draw preliminary conclusions on the extent to which descriptions of expected behavior influence the interactive process and the quality of the interaction.

Introduction
Web-based learning is assuming an increasing role in education at college and pre-college levels (Bates 1999; Collis 1996; Fjuk et al. 1999). It is not only situations of continuing education and lifelong learning which call for flexible organization of the learning process, e.g. with the purpose of bridging distances. The obvious advantages in terms of a more flexible, time and space independent organization of the learning process, also within higher education, are clear (Harasim 1999). The flexible organizational design of web-based learning, by itself, does not make this method superior to other methods. Of more importance is the quality and design of the learning process. Many flexible designs offer no possibilities for interaction and collaboration among students, but are rather sterile and lack a framework to stimulate collaborative knowledge building. Others do so, but have problems - even when interaction occurs — in qualifying a knowledge building dialogue.

This paper deals with the question of how to qualify a knowledge building dialogue through grading requirements and as an effect from providing the students with meta-awareness around the functions of the requested comments in the dialogue. The paper suggests that grading requirements as well as providing awareness around the requirements related to the function of comments in the dialogue contribute to qualifying a knowledge building process in collaborative learning.

In section 2 we provide some brief information of the web-based course producing the data of our analysis. Section 3 provides an account of our knowledge building perspective and the criteria of quality used in our analysis of web-based dialogues. Section 4 addresses the communicative conditions of web-based environments, forming part of the rational behind our hypothesis. In section 5 we give a more detailed account of our research design and of the method used in our analysis, while section 6 provides the basic analysis. We reflect on the results of the analysis in section 7 and discuss future research perspectives.

Development of Dialogue Requirements in "Global Change"
Global Change (GC) is a conventional science course for senior undergraduates or beginning graduate students at a US university. It gradually has been migrated to a web base over the last 6 years, with new features being added as ancillary software has become available. Learner-centered activities in place of or supplemental to conventional lectures have been introduced.

The course consists of a sequence of learning modules on different global-change topics, each having evolved from a conventional university class time period. Each unit has a set of objectives, summary information on the topic, student-submitted collaborative (2-3 students) summary of class time discussion, "problems to ponder" as discussion starters for the electronic dialogue, and extensive lists of web and other information on the learning module topic. Each unit has its own electronic dialogue for student discussion among themselves and with outside experts or representatives of selected groups. Electronic dialogue on individual learning unit topics is graded. The course is viewed by the designers as a laboratory for experimenting with a variety of pedagogical techniques and initiatives (Taber et al. 1997).

The course went on the web in Spring 1995. Affiliated with the course was an opportunity to engage in electronic dialogue. The use of dialogue and associated instructor evaluation of student engagement in dialogue changed over the years as follows:

1995:
Electronic dialogue was available separately for each lecture topic, and 5% of the evaluation of student performance was based on whether or not a student participated. There were no minimum requirements on amount of discussion and no attempt was made to judge quality as part of the evaluation of students.

1996:
Same as 1995 except 15% of the grade depended on the electronic dialogue. Some weak attempt was made to judge quality but primary criterion was participation and not quantity or content.

1997:
Requirements for participation in electronic dialogue were substantially enhanced. For full credit, the student was required to post at least 15 entries, which had to include responses to at least 6 other students’ comments. Furthermore, these entries had to elicit comments from 3 other students. They also were required to respond to 3 questions issued by the instructor and to give their views on 3 additional ethical questions relating to global change. Thirty percent of evaluation of students was based on these numbers but no evaluation of quality was attempted. A virtual portfolio was introduced for each student to manage discussion.

1998 and 1999:
Same as 1997.

2000:
Same requirements as 1998 with addition of the knowledge-building process (Stahl 1999) that required students to meet specific quality criteria in their discussion. Also, students were required to assess their own writing and demonstrate how they had met the criteria. Forty-two percent of the evaluation of student performance was based on discussion.

Criteria of Collaborative Knowledge Building in Web-Based Learning
To put our analysis of electronic dialogues in perspective, we briefly outline the set of criteria which we assert are signs of quality in a web-based collaborative knowledge building process.

In the principles of collaborative learning, the process of learning is viewed to be a fundamentally social phenomenon, regardless of the varying theoretical emphasis in each single approach (Dillenbourg et al. 1995). Several other learning theories confirm this view, e.g. Etienne Wenger in his latest book, "Communities of Practice" (Wenger 2000).

The process of knowledge building (KB) in collaborative learning, first explored by Harasim (1989), involves mutual exploration of issues, mutual examination of arguments, agreements and disagreements, mutual questioning of positions, dynamic interaction and weaving of ideas (Harasim 1989; Kaye 1992; Sorensen 1997). Mason (1993) finds this view to be in agreement with the communicative potential of the online environment, although she also points out the weaknesses of the online dialogue being that it quite often never reaches synthesis or closure (Mason 1993).

In view of the generally recognized difficulties in fostering online student dialogue that converges (e.g., synthesizing) rather than diverges (noted by Mason), Stahl (1999) suggests a set of factors that characterize quality in the KB process:

Brainstorming is the introducing of new ideas that relate to the topic or task and offer a perspective not previously considered;

Articulating includes explaining complex or difficult concepts;

Reacting provides an alternative or amplified perspective on a concept previously introduced by a student;

Organizing refers to assembling existing thoughts or perspectives in such a way that a new perspective emerges;

Analysis includes comparing or contrasting previously articulated views or puts new understanding on existing data;

Generalization takes comments or data already presented and extracts new information or knowledge that applies to a broader set of conditions.

Implementing these learning quality criteria of collaborative KB requires a corresponding meta-functional pedagogy or instruction that facilitates and motivates such collaborative dialogue (see section 4).

Reflective Dialogue Primary to Involved Dialogue
Promoting knowledge-building dialogue within the context of collaborative learning appears to be a complex challenge. In searching for reasons for this, some research has focused on the design and nature of the collaborative activities implemented in the learning process (Collis 1997), and others on the quality and nature of the virtual environment and the evolvement of electronic inter-human dialogue (Scardamalia & Bereiter 1996; Sorensen 1993). Alternative studies have concluded the social aspect of a group process to be an essential motivator for collaboration in online learning (Harasim 1993; Cornell & Martin 1997). It seems to carry a rather high proportion of a group member’s inclination to engage in any interaction with the group at all (Moore & Kearsley 1996). This seems in total agreement with the widely acknowledged insight, that inclination to interact online is sensitive to the perception of interaction (Gunawardena 1995).

Considerable research has been directed towards the role and meta-communicative behavior of the instructor (Feenberg 1989). Recent studies along this line of thinking have described the online universe as a meta-communicative world. Contrary to the physical world in which involvement is viewed to be primary to reflection (Heidegger 1986) - the virtual universe provides a context and an "ontology" in which reflection may be said to be primary to involvement (Sorensen 1999). Assuming this new dialogical paradigm, and this primary position of reflection and meta-communication, it is very conceivable, from the perspective of collaborative interaction and dialogue in Web-based learning, that also the task of scaffolding learning processes that aim at supporting both interaction and (self)reflection, must move at a meta-level in terms of creating awareness of the function in a dialogue of a contributed comment.

Research Design
In our analysis we wanted to investigate the role and nature of the instructions in the requirements given to the students, stimulating them to interact. We wanted to assess whether meta-communication in terms of providing meta-awareness of "the function of a comment" in the KB process, will improve the quality of the KB dialogue, according to the definition of quality as suggested by Stahl (1999). In our research design and analysis we make the following assumptions:

1. Characteristics of discussion and comments that contribute to a KB process are (Stahl 1999): brainstorming, articulation, reaction, organization (including synthesis), analysis, and generalization.

2. Use by students of characteristics of the KB process in written dialogue contributes to student learning (as characterized in section 3).

Our hypothesis for the analysis is then, that by explaining the characteristics of a KB-process, and by grading (Sorensen & Takle 1999) student discussion on the basis of their reflected use of these characteristics, students will measurably increase their use of these characteristics. The rational behind the hypothesis is an acknowledgement of the need in primarily reflective virtual environments (as described in section 4) for communicatively providing meta-awareness in relation to expected communicative actions (as described in section 3).

We have evaluated characteristics of student dialogue and its relationship to course requirements by assembling 10 comments from 1995, 10 from 1997, and 10 from 2000. These were drawn at random (although not strictly with statistical rigor) and represent comments made under quite different criteria for evaluation of student performance. In drawing the comments from the respective databases, care was taken to draw comments from the same (or very closely related) topics in each of the three years.

The data sets represent (a) comments with no obligations attached as far as evaluation of student performance was concerned (1995), (b) comments when numbers of responses and numbers of interactions among students were used for evaluation (1997), and (c) comments when the KB requirements were imposed. For 2000, we also have, for some database entries, the student self-assessment of their own writing. We also have assigned a numerical value from 1 to 10 representing the quality of the comment.

Analysis of Dialogues
For the purpose of our analysis we have categorized the analyzed comments in the online dialogues according to the knowledge building quality requirements presented by Stahl (1999) in section 3:

• comment with no KB characteristics
  
• question (a request for clarifying information)
  
• analysis (comparing or contrasting previously articulated views)
  
• articulation (the explanation of complex or difficult concepts)
  
• reaction (presenting an alternative or amplified perspective on a concept previously introduced)
  
• brainstorming (the introduction of new ideas that relate to the topic or task and offer a new perspective)
  
• organization (the organization of existing thoughts or perspectives)
  
• generalization (extracting new information or knowledge from comments already presented, that applies to a broader set of conditions)

A level of quality (1-10) was subjectively assigned to each comment. The mean quality went from 4.4 to 3.2 to 5.3 from 1995 to 1997 to 2000, respectively. In 1995 a preponderance entries were questions, whereas in 1997 there were more reactions to other students. In 2000, more student comments were characterized as articulation.
From these results, we draw the following tentative conclusions. In 1995, with no requirements on volume or quality of writing, students used the electronic dialogue for asking questions. The questions did not seem "forced" since their postings had essentially no bearing on how they were evaluated in the course. For 1997, the overall quality of the postings went down. There seemed to be more "forced" responses. More questions were of a rhetorical nature, rather than seeming to originate out of curiosity about the subject matter. We attribute this to a sense of urgency on the part of students to meet the volume obligations of the evaluation scheme. The increase in quality from 1997 to 2000 suggests that students responded to the demands for higher quality of writing by meeting the requirements of the KB process. A notable difference in the 2000 discussion as compared to previous years was the reduction in the "social" or "chat" nature of the comments. But at the same time, the number of comments increased. Comments were much more like individual essays and less spontaneous.

Perhaps the largest difference over the change in dialogue requirements was in the length of each entry. In 1995 the average length of the 10 entries was 88 words and in 1997 the average was 93 words. In 2000, however, the average soared to over 2,500 words. And the quality of the grammar of each posting increased markedly as well. This likely is attributable to the fact that students wrote their postings as essays in word processors on their own computers, subjected them to spell checking, and reviewed the grammar before cutting and pasting them into the course web-site. This increased quality and quantity came at the expense of spontaneity and "social" content.

Discussion and Future Perspectives
How do we evaluate the results of our analysis? Assuming that meta-awareness (Gutwin et al. 1995) of KB characteristics among students as well as grading of comments increase the process of collaborative learning, and accepting the value of the KB characteristics used in our analysis, we think it is fair to say that our experiment has proved itself to be relevant. Over the three trials the quality of student comments has increased, both as a result of grading requirements and as an effect from the meta-awareness around the functions of the requested comments.
Our sample size was quite small and therefore the results are subject to large uncertainty. Therefore, the extent to which the results can be generalized may be questioned. The selected comments, however, were drawn at random and taken out of the social context of the specific dialogue in which it appeared. Perhaps this could be considered a problem of the data set, since the KB requirements are concerned with the sequential development of online interaction.

As pointed our, the largest difference over the change in dialogue requirements was in the length of each comment, and the increased quality of the grammar used. It seems that the more carefully composed essay-like style of comments as well as the increased quality and quantity came at the expense of spontaneity and "social" elements in the interaction. It is conceivable that dealing with electronic dialogues and an electronic comment in terms of a request for reflection on its function, is inhibiting the evolvement and practice of a spontaneous and dynamic dialogue.

As such, the question could be posed, whether the strong meta-reflective requirements enforced on a dynamic phenomenon as human interaction (even though it is not in terms of formalization) appears problematic. Or perhaps whether the lack of recognition of social elements in the interaction works against an incitement to engage in collaborative interaction or knowledge building with others. Moreover, it may be pertinent as a basis for a potential conclusion around this aspect to specify, whether one is using data from an off-campus course (where the students have no complementary social interaction) or an on-campus course (where social interaction among students is part of daily life) as in the present study.

In the data from our experiment there is some indication, that the "forced requirements" on the collaborative dialogue actually functioned like a "kick-off" to engage in dialogue, and that it actually caused more interactivity than required to appear. But we find the data set from this study too small to conclude more precisely around this aspect. It is beyond any doubt, though, that - as default - to be forced to reflect on the function of one's action before carrying it out is likely to decrease spontaneity, phatic behavior and high interactivity. Above all, we believe that the optimistic results of our analysis have demonstrated that, once again, the problem is a complex one. And perhaps the usual controversial points to discuss in this respect will be "how" to do it and "to what extent". All these are questions to be further studied and pursued in the future.

References
Bates, A. W. (1999): "Managing Technological Change: Strategies for Academic Leaders" San Francisco. Jossey Bass.

Collis, B. (1996): Tele-learning in a Digital World. International Thomson Computer Press. 1996.

Collis, B. (1997): Experiences with Web-based environments for collaborative learning and the relationship of these experiences to HCI research. Published in the Proceedings of the Working Confernce of IFIP WG 3.3, Sozopol, Bulgaria.

Cornell, R. & Martin, B. L. (1997): The Role of Motivation in Web-Based Instruction. In Khan, B. H. (ed.): Web-Based Instruction. Educational Technology Publications Englewood Cliffs, NJ. 1997. pp. 93-100

Dillenbourg, P., Baker, M., Blaye, A., O’Malley, C. (1995) The Evolution of Research on Collaborative Learning. In Reimann, P., Spada, H. (eds.) Learning in human and machines. Towards an interdisciplinary learning science, London: Pergamon, pp. 189-211

Feenberg, A. (1989): The Written World. In Mason, R. & Kaye, A. R. (eds.): Mindweave. Communication, Computers, and Distance Education. Pergamon Press. Oxford. 1989. pp. 22-40.

Fjuk, A., Sorensen, E. K., Wasson, B. (1999): Incorporating Collaborative Learning, Networked Computers and Organisational Issues into Theoretical Frameworks. In the proceedings of The New Educational Frontier: Teaching and Learning in a Networked World, ICDE'99.

Gunawardena, C. N. (1995): Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications, 1(2/3), pp. 147-166.

Gutwin, C., Stark, G., Greenberg S. (1995): Support for Workspace Awareness in Educational Groupware. In the Proceedings of CSCL95, available from Lawrence Erlbaum Associates, Mahwah, NJ http://www-cscl95.indiana.edu/cscl95/ .

Harasim, L. M. (1989): Online Education. A new domain. In Mason, R. & Kaye, A. R, (eds.): Mindweave. Communication, Computers, and Distance Education. Pergamon Press. Oxford. pp. 50-62.

Harasim, L. (1999): The Virtual-U Field Trials: Lessons about Teaching and Learning Online.

Harasim, L. (1993): Collaborating in Cyberspace: Using computer conferences as a group learning environments. Interactive Learning Environments, 3(2), pp. 119-130.

Heidegger, M. (1986): Sein und Zeit. Max Niemeyer Verlag Tübingen.

Kaye, A. R. (1992): Learning Together Apart. In Kaye, A. R. (ed.): Collaborative Learning Through Computer Conferencing. NATO ASI Series, vol. 90. Springer-Verlag, pp. 1-24

Mason, R. D. (ed.) (1993): Computer Conferencing. The Last Word. Victoria. Beach Holme Publishers.

Moore, M. G. & Kearsley, G. (1996): Distance Education. A Systems View. Belmont, CA: Wadsworth.

Scardamalia, M. & Bereiter, C. (1996): Computer support for knowledge building communities. In Koschmann, T. (ed.): CSCL: Theory and Practice of an Emerging Paradigm. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 249-268

Sorensen, E. K. (1993): Dialogues in Networks. In Andersen et al. (eds.): The Computer as a Medium. Cambridge University Press. Cambridge. pp 389-421.

Sorensen, E. K. (1999): Intellectual Amplification through Reflection and Didactic Change in Distributed Collaborative Learning. In C. M. Hoadley and J. Roschelle (Eds.), Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, pp. 582-589. Palo Alto, CA: Stanford University [Available from Lawrence Erlbaum Associates, Mahwah, NJ]. http://kn.cilt.org/cscl99/A71/A71.HTM

Sorensen. E. K. (1997): Learning in Virtual Contexts. Navigation, Interaction, and Collaboration. Ph.d.-dissertation. Aalborg University. Danmark.

Sorensen, E. K. & Takle, E. S. (1999): Distributed Collaborative Learning across Disciplines and National Borders. In C. M. Hoadley and J. Roschelle (Eds.), Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference pp. 575-581. Palo Alto, CA: Stanford University [Available from Lawrence Erlbaum Associates, Mahwah, NJ]. http://kn.cilt.org/cscl99/A70/A70.HTM

Stahl, G. (1999): Reflections on WebGuide: Seven Issues for the Next Generation of Collaborative Knowledge-Building Environments. In C. M. Hoadley and J. Roschelle (Eds.), Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, pp. 600-610. Palo Alto, CA: Stanford University. [Available from Lawrence Erlbaum Associates, Mahwah, NJ]. http://kn.cilt.org/cscl99/A73/A73.HTM

Taber, M. R., E. S. Takle, and D. Fils (1997): Use of the internet for student self-managed learning. Preprints, Sixth Symposium on Education. American Meteorological Society, 2-7 February, Long Beach, CA.

Wenger, E. (2000): Communities of Practice. Learning, Meaning, and Identity. Cambridge University Press.

Acknowledgments
We acknowledge the contribution of Kathryn St. Croix, Iowa State University, for collaborating with us in the analysis of dialogues. Contributors to the design and implementation of the Global Change course include Doug Fils, Michael Taber, Jennifer Hodson, and David Flory. The course is implemented under the auspices of the International Institute of Theoretical and Applied Physics.