Paul Stacey


Measuring OER Outcomes

I’ve been immersed in analysing data from the last seven years of the BCcampus OER initiative.
I’m particularly interested in conveying the value for money or outcomes associated with our initiative.

So how does BCcampus measure its OER outcomes?

One way of expressing the outcome is to simply quantify the BCcampus OER deliverables.
This is the metric I’m most often asked by the Ministry. The answer goes like this:
Through the BCcampus OER initiative 131 projects have received grants leading to development of 317 courses, 10 workshops, 18 web sites/tools and 338 course components (learning objects, labs, textbooks, manuals, videos).

That’s one way of expressing outcomes.

However, in my view this basic quantification of deliverables does not get to the deeper value for money outcomes.
Over the last couple of months I’ve been working with a data specialist to construct a complete picture of the OER initiative and to analyse the results more deeply. Here then are some of what I think of as the deeper and more meaningful BCcampus OER outcomes:

1. Partnerships

The public post secondary system in British Columbia is made up of largely autonomous institutions. Part of the mandate of BCcampus is to foster collaborations and partnerships between institutions and others.

BCcampus OER Partnership Outcome:
105 of the 131 BCcampus OER initiatives, or 80%, involve collaborations and partnerships between multiple BC post-secondary institutions. In addition there have been 45 external partners involved in the 131 development projects.
External partners include:
– national and international universities
– professional associations
– K-12 school districts and school boards
– e-learning companies
– foundations
– First Nations tribal councils
– health authority’s
– literacy groups

I’ve been told over the years that the multi-institutional partnership requirement of the BCcampus OER initiative is one of the more challenging aspects for institutions to fulfill. Institutions form partnerships with each other based on academic program synergy and a mutual academic need. Partnering involves pooling expertise and developing an online resource that both institutions subsequently use. Each institution has what might be thought of as its own “trading partners” with whom they repeatedly form partnerships. Trading partners often share a similar stature in the system such as partnerships among remote rural colleges or partnerships among large research based universities. I’ve been told that once partnerships form the partnership often extends out into other activities beyond the BCcampus OER initiative.

2. Credentials

A goal of the BCcampus OER initiative is to increase credential opportunities available to students throughout the province by funding development of post-secondary online courses, programs, and resources. Credentials in BC’s post-secondary are categorized as follows:
– Apprentice-Entry Level
– Associate Degree
– Bachelor’s Degree
– Certificate
– Citation
– Diploma
– Doctoral Degree
– Grad Cert/Diploma
– No Credential Granted
– University Transfer
* Note: This credential categorization is taken from EducationPlanner.ca

Credentials are developed through the BCcampus OER initiative in four ways:
1. A single round of funding allows for development of all the courses required for a complete credential.
2. A complete credential is built out gradually through multiple rounds of funding.
3. The OER initiative provides funding needed for development of the last few courses required to make the complete credential online.
4. The OER initiative creates a number of online courses that can be used across multiple credentials or serve as the building blocks for creating credentials.

The BCcampus OER initiative has contributed to the development of 41 credentials:

Associate Degree
– Associate of Arts Degree & Associate of Arts Degree in Geography
– Associate of Arts Degree in First Nations Studies
– Web-based Associate of Science

Bachelor’s Degree
– BA Psychology
– Bachelor of General Studies (Police Studies)
– Bachelor of Tourism Management
– Bachelor’s Degree in Psychology
– Bachelor’s Degree with a Marketing Minor
– Classroom and Community Support Program
– Minor in Gerontology
– Northern Collaborative Baccalaureate Nursing Programme

Certificate
– Administration Assistant Certificate
– Certificate in First Nations Housing Managers Training
– Certificate in Gerontology
– Certificate in Tourism Event Management
– Community Development Certificate program
– Computer Technology Certificate
– Finance for First Nations Housing Managers
– First Nation Shellfish Aquaculture General Management Certificate
– Medical Office Assistant Certificate
– Practical Nursing Online Certificate
– Provincial Legal Administrative Assistant Online Certificate
– Renewable Energy Certificate Program

Diploma
– Aboriginal Business Administration Diploma
– Aboriginal Community Economic Development Diploma
– Access to Dental Hygeine Diploma
– Advanced Diploma in Fetal Alcohol Spectrum Disorder (FASD)
– Advanced Diploma in Human Resources
– Animal Health Technology Diploma
– Business Administration General Management Diploma
– Continuing Health Care Administration Diploma
– Diploma in Local Government Management
– Diploma in Public Sector Management
– Early Childhood Care and Education Diploma
– First Nations Public Administration Diploma

Graduate Certificate/Diploma
– Graduate Diploma in Public Health
– Graduate Certificate in Child and Youth Mental Health
– Post Baccalaureate Diploma in Gerontology
– Post Bacclaureate Diploma in Marketing
– Post Graduate Technical Diploma in Geographic Information Systems (GIS)

Master’s Degree
– Masters of Applied Arts

In addition to explicit development of the above credentials some BCcampus OER initiatives develop multi-purpose undergraduate online courses or smaller course components for unspecified credentials. These initiatives typically express their intent as developing core foundation level resources that can be used across multiple courses and credentials.

To deepen the analysis I’ve been drilling down from the credential level to fields of study. The BC Council on Admissions and Transfer’s Education Planner site categorizes BC’s higher education academic offerings into the following fields of study:

– Agriculture Natural Resources and Science
– Business Management
– Communications
– Computer and Information Services
– Construction and Precision Production
– Development Programs (e.g. ABE, ESL)
– Education and Library Studies
– Engineering/ Electrical and Electronics
– Health Related
– Legal and Social Services
– Liberal Arts and Humanities
– Mechanical and Related
– Recreation, Tourism, Hospitality Service
– Sciences
– Social Sciences
– Transportation (Air, Land, Marine)
– Visual, Performing and Fine Arts

I’ve analysed the BCcampus OER initiative data to determine what percent are in each of these fields of study.
It breaks out like this:

Agriculture Natural Resources and Science = 4.83%
Business Management = 10.07%
Communications = .13%
Computer and Information Services = 1.21%
Construction and Precision Production = 0%
Development Programs (e.g. ABE, ESL) = 6.17%
Education and Library Studies = 3.36%
Engineering/ Electrical and Electronics = 1.20%
Health Related = 12.35%
Legal and Social Services = 3.36%
Liberal Arts and Humanities = 5.23%
Mechanical and Related = 2.55%
Recreation. Tourism, Hospitality Service = 7.52%
Sciences = 16.38%
Social Sciences = 10.34%
Transportation (Air, Land, Marine) = 0%
Visual, Performing and Fine Arts = 4.16%

As you can see the field of study area that has received the most development is in science with 16.38%, the second highest is health with 12.35%, then social sciences with 10.34%, and business management with 10.07%. The remaining percentage of development has gone toward development of professional learning resources and a small amount to apprenticeship.

Going deeper still, each field of study breaks down into subject areas. Visit Education Planner’s Program Search to see list of over 200 subject areas available. I thought it would be interesting to consider the extent to which the BCcampus OER initiative has resulted in development of resources across the full spectrum of subject areas. The BCcampus OER initiative has developed resources in 63 of the 200 subject areas.

BCcampus OER initiative credential, field of study, and subject area outcomes are shaped each year by the call for proposals. Typically the call targets development in areas of high student demand and labour market need. In some years the Ministry makes explicit priorities. For example in the 2009 call for proposals the Ministry expressed the following as priorities – Early Childhood Education; Health-related Programs; Programs aimed at Aboriginal Learners as well as learners with disabilities, mature learners and recent immigrants; Technician and Technologist Programs; Tourism and Hospitality. However, post-secondary institutions can submit proposals for any area and to some extent the credential, field of study and subject area coverage represents priorities of the entire BC public post-secondary system.

3. Sharing and Reuse

Like other OER initiatives the “open” goal of the BCcampus OER initiative is to create a source of digital materials that are openly shared and available for reuse by others.

The BCcampus OER initiative gives developers the choice of Creative Commons and BC Commons licenses. Developers wanting to participate in the global OER movement can go with Creative Commons. Alternatively they can choose the BC Commons license which provides for open sharing locally at the provincial level among all 25 public post secondary institutions.

One outcome I’ve been interested in tracking is, when given an option between these two licenses what license choice do developers make. I think of this as a measure of “openness”. In the first two years of the BCcampus OER initiative over 90% of developers chose the BC Commons license. In the next two years 78% chose the BC Commons license. In the last three years 47% chose the BC Commons license. In 2003 when we started the BCcampus OER initiative developers let us know in no uncertain terms that they were uncomfortable with wide open sharing. This comes through loud and clear in their license choices. However, as OER become a more widely known global phenomenon and the risks many initially feared from sharing diminish developers are becoming increasingly willing, and in some cases advocates for, being globally open.

I often think of OER as existing on a continuum of openness. At the most closed end of the continuum is fully copyright protected. At the most open end of the contiuum is public domain. BC Commons licensed OER are more open than copyright protected resources but not as open as Creative Commons.

Another obvious outcome to measure is reuse. On the surface this measure seems obvious. How many of the resources developed are reused by others? However, the answer is less easy to arrive at. Part of the challenge is defining reuse.

What are the use cases for OER? For global OER the most common use case is translation and use in a developing country. For some OER, such as Connexions the use case is student self study or assembly of OER components into a print-on-demand textbook. For MIT’s OpenCourseWare a significant use case is marketing whereby the OER are seen as a calling card for attracting students to enroll and pursue a degree at MIT. Once enrolled another use case is when MIT students view the OpenCourseWare not for self study but as an academic planning aid helping them pick which courses they’ll sign up for next term.

For those of us who have been involved in e-learning for many years the reuse of digital learning resources has a long history. Prior to OER, learning objects, small 2 to 15 minute units of learning, were seen as a key form factor for reuse. The key affordance of learning objects is that instructors can custom assemble them into larger modules of learning that fit their understanding of a domain or way of teaching. Interestingly reuse of learning objects did not take off and reuse of OER in this fashion similarly remains low.

The most common instructor use case for OER that I’ve heard of (and been told by many faculty) is different than what might be expected. For faculty development of a course is a creative exercise that represents their unique expertise and understanding of a field of study. There is a prevailing notion that no one else’s course could possibly be as good as the one they develop themselves. If faculty look at an OER they initially do so not with an eye to determining which parts of it they might use themselves but rather with an eye to seeing how a professional peer represents their own knowledge of a domain and the pedagogy they use to deliver it. This is the most common use of an OER by an instructor – the OER serves as a comparative framework for their own course with a view to how it is sequenced, how comprehensive it is, and the type of learning activities the instructor uses to engage students. This is in itself a form of reuse and a significant OER value proposition.

Of course the value proposition can be further enhanced if an instructor downloads the OER and reuses it in whole or in part. The licenses used for all OER support customization of the existing resource so instructors are not obliged to use an OER as is. They are free to use just a piece or to modify it to fit their needs.

BCcampus makes the OER produced through it’s initiative available in a repository that supports search, preview, and download. One way we can track reuse is to monitor whats being searched, viewed and downloaded. The software application used for the repository has limited reporting capabilities though – downloads for example are not easily tracked. But let me report out on views. From January 2008 through November 2009 634 resources in the repository were viewed. The total number of views was 1,853. So clearly the OER are at least meeting the first use case I depicted above where it is at least viewed. We can also see which resources have been viewed the most, giving an indication of popularity or high value.

Large scale reuse of OER across the full gamut of use cases remains elusive for virtually all OER initiatives. I believe much more work is required to show how OER fit within the instructional design process and we need to ensure that the level of effort required to reuse OER is less than the level of effort required to simply develop a new resource yourself. Most of all the credential and academic integrity of the resource needs to be preserved so that reuse involves more than simply repurposed content.

This exploration around measuring OER outcomes is my own take on showing value for money.
I’d love to hear from other OER initiatives on the metrics they are using and reporting on to show outcomes.

There has been a lot of talk about the sustainability of the OER movement.
Based on the analysis above I’d say OER must generate a return on investment right away.
OER generated through the BCcampus initiative have an immediate academic practical use with real students leading to credentials.
This outcome is itself worth the investment.
With that outcome in hand additional value comes through partnerships and reuse.
Relying on reuse outcomes alone to justify value for money is, at this time, folly.



Foundation Funded OER vs. Tax Payer Funded OER – exploring the differences

Since 2003 I’ve been leading an Open Educational Resource (OER) initiative at BCcampus. This BCcampus Open Educational Resource initiative involves all 25 public post secondary institutions in the province of British Columbia, Canada.

Over the last seven years OER initiatives have proliferated. As the number of initiatives increase I’ve been following them with great interest and with an eye to compare and contrast them with each other and with ours at BCcampus.

I’ve noticed the BCcampus initiative is unique in a number of ways and thought I’d use this post to explore my thoughts about that uniqueness and raise some questions OER.

The BCcampus OER initiative differs from most others in that it is sectoral rather than institutional. In the BCcampus initiative OER are produced by all 25 public post secondary institutions in the province. As a sectoral initiative OER are primarily being developed via multi-institutional partnerships involving faculty and staff from more than one institution. Partner institutions each invest in the development of the OER and each use the developed resource. In addition many of the projects have formed partnerships with BC e-learning companies, not-for-profits, and professional associations to support the development effort. This maximizes the use of the resource and works toward the creation of an OER ecosystem within the province.

Other OER initiatives such as the MIT’s OpenCourseWare, Carnegie Mellon’s Open Learning Initiative, the UK Open University’s OpenLearn, and others are institutionally based with educational resources being produced from within the institution without partnerships across multiple institutions.

Another unique aspect of the BCcampus initiative is that it is funded using public tax payer money provided through the Ministry of Advanced Education. Investment is made annually via a competitive Request For Proposal (RFP) process. As of April 2010 BCcampus has done seven annual rounds of funding representing a total investment of $8.25 million dollars (CDN). Most other OER initiatives are funded through foundation grants including the Hewlett Foundation, Bill & Melinda Gates Foundation, and others.

With funding coming from the Ministry of Advanced Education the primary goal of the BCcampus OER initiative is to increase the credential opportunities available to students throughout the province by funding development of post-secondary online courses, programs, and resources. The BCcampus OER initiative targets development at credit based, fully online learning courses in areas of high student demand and labour market need. Grants are intended to support curriculum development building out complete online programs leading to a credential.

The BCcampus OER initiative is focused on solving a provincial higher education need around credential opportunities and academic completion. I’ve noticed that many foundation supported OER initiatives are not actually seeking to solve academic or credential needs at the institutional or regional level. They are simply creating a pool of educational resources without an applied academic purpose or credential context.

Like other OER initiatives the “open” goal of the BCcampus OER initiative is to create a source of digital materials that are available for immediate free use eliminating the weeks and months of time it can take to seek permission to use existing digital materials. Reuse of openly shared online learning resources leverages an initial investment, in this case of public taxpayers dollars, many times over. It also stimulates and supports a culture inherent to education – sharing knowledge and building new knowledge off the work of those who came before you.

However, the breadth of the BCcampus OER goals differ from foundation funded OER initiatives. Most foundation funded OER initiatives have a global orientation not a regional one. A goal of the Hewlett Foundation OER grant program is to “Equalize access to knowledge for teachers and students around the globe through Open Educational Resources.” For some there is a moral and ethical imperative around the open sharing of educational resources globally based on a recognition of the world wide shortage of education. Globally open OER increases access to education by supplying content to those in need especially developing countries. This global rallying spirit is captured nicely in the Capetown Open Education Declaration. Another way of conceptualizing this is as a form of education philanthropy.

With the BCcampus OER being funded by a provincial government the focus is more on serving the needs of students in BC. The global philanthropy aspect of OER has not been mandated. However, limiting the focus to the province rather than the globe may also limit the extent to which the open nature of these resources has an impact. An underlying principle of OER is that others are free to change and improve the resource but must share it back when they do so for the benefit of all including the original developer.

A critical question for all OER initiatives is, “To what extent are OER being improved and modified by others and shared back to the benefit of the original developer and everyone else.?” The startling answer at this point in time is little to not at all.

The BCcampus OER initiative is unique in the licensing approach used. Foundation supported and globally oriented OER initiatives all mandate use of Creative Commons licenses. The provincial BCcampus OER initiative differs in that it offers developers license options. To make the OER open and shareable BCcampus OER developers are given licensing options of Creative Commons or BC Commons. Developers wanting to participate in the global OER movement and contribute to education philanthropy can go with Creative Commons. Alternatively they can choose the BC Commons license which provides for open sharing at the provincial level among all 25 public post secondary institutions rather than globally with everyone as provided through Creative Commons.

An important principle in the BCcampus OER initiative is choice. Developers must openly share but they can choose to share regionally or globally. Participation in the global OER movement is recommended but not mandated. The BCcampus OER is one of the few who give developers license options and allow them to decide for themselves where they want to participate on the “open” continuum.

To manage both global and local sharing BCcampus has deployed a Shareable Online Learning Resources (SOL*R) repository which provides a means for searching, previewing, and downloading OER. The resources that get developed through the BCcampus OER initiative are online learning resources primarily developed for deployment through learning management systems (LMS) such as Moodle, Desire2Learn, and Blackboard/WebCT. The multi-platform online learning form factor of our OER is another unique aspect of the BCcampus OER initiative. It has also complicated our OER effort enormously as the production of interoperable online learning resources is fraught with technical challenges exacerbated by LMS vendors who want to lock clients in to using their platform.

One other unique aspect of the BCcampus OER initiative is that a portion of each round of funding goes toward developing professional learning OER resources for faculty and staff. This acknowledges the growing importance of complementing online learning development and delivery with educator professional development resources on how to effectively do so.

As you can tell from the above I’ve been thinking a lot about what we are doing here at BCcampus and how it compares to what others are doing. I know from the many presentations I’ve given on the BCcampus OER initiative that some see our OER initiative as reprehensible for not mandating global sharing of the OER. I certainly believe in sharing of OER but I also believe in choice and the right for education developers to choose how openly they want to participate.

In thinking about the future of OER I call for:
– more diversity of OER approaches
– more partnerships between OER initiatives
– greater emphasis on the academic utilization of OER
– a shift to use of OER for credentials

Comments Off on Foundation Funded OER vs. Tax Payer Funded OER – exploring the differences


Open Course Library Kickoff

I spent April 26-27, 2010 in Vancouver Washington attending the Open Course Library Kick-Off meeting of the Washington State Board for Community & Technical Colleges.

The Open Course Library initiative is one of four Washington higher education programs being funded through a $5.3 million Bill & Melinda Gates Foundation three year grant. The Open Course Library portion of the grant is $1.8 million.

The Open Course Library initiative has the following goals:
– design 81 high enrollment, high quality courses for face-to-face (FTF), hybrid, and online delivery
– lower textbook costs for students (< $30)
– provide new resources for faculty to use in their courses
– fully engage Washington's college system in the global Open Educational Resources (OER) movement
– improve completion rates through good design and affordability

The kick-off meeting is for developers from across the college system who have received grants to support development of the first 43 courses. Phase 2 for the remaining 38 courses is slated for 2011.

I attended this event as a guest at the kind invitation of Cable Green who is Project Director for the Open Course Library initiative.

I went with a number of goals:
1. Assess the degree of state level support for the initiative vs. foundation level support
2. Learn more about the open/low cost textbook strategy being used as part of Open Course Library
3. Compare and contrast the Open Course Library initiative with BCcampus' own open educational resource initiative
4. Explore the potential for some kind of collaboration or partnership between our initiatives

The Open Course Library initiative is fascinating in many ways.

At the state level a Strategic Technology Plan for Washington State Community and Technical Colleges notes that: “Using open educational resources – and contributing to them – requires significant change in the culture of higher education. It requires thinking about content as a common resource that raises all boats when shared. It requires replacing our “not invented here” attitude with a “proudly borrowed from there” orientation. And it requires a new willingness to share and distribute the best of our own course content and software, and to participate in creating and maintaining open textbooks.”

The Plan goes on to list a number of guiding principles which includes one that states: “We will cultivate the culture and practice of using and contributing to open educational resources.”

Kudos to the state for producing a Strategic Technology Plan and supporting OER!

I particularly like the way Open Course Library is targeting development of high enrollment courses. Of the 43 courses being developed in the first phase the top ten in terms of annual enrollments are:
1. English Composition I with enrollments of 42,301
2. Intermediate Algebra with enrollments of 25,747
3. Intermediate Algebra with enrollments of 25,255
4. General Psychology with enrollments of 24,611
5. Pre-college English with enrollments of 17,658
6. English Composition II with enrollments of 16,165
7. Introduction to Chemistry with enrollments of 10,382
8. Precalculus I with enrollments of 9,307
9. General Biology with enrollments of 8,830
10. Lifespan Psychology with enrollments of 7,853

By targeting high enrollment courses the Open Course Library initiative immediately generates benefits for over 188,109 students annually. Thats a lot of students!

Enhanced design and digital resources for these courses will improve the teaching and learning experience for students and faculty alike.

I also really like the cost savings/affordability goals of the Open Course Library initiative. As part of the development of these courses the accompanying textbook must be $30 or less. The text can be in digital or print form. Publishers have been given the list of courses being developed and asked to inform developers what they can make available for that price. In Washington annual tuition is around $3K/year with textbooks adding a further $1K. Many textbooks are in the $100 to $200 range and textbook prices have been increasing at approximately 6%/year, a rate higher than annual inflation. A strategy that reduces textbook price to less than $30, when multiplied by annual enrollments represents a significant cost savings to students.

The Open Course Library grants will produce a complete digital course including:
– syllabus with clear learning outcomes
– course curriculum
– instructional materials
– formative and summative assessments
– surveys
– grading rubric
– cover letter describing tips and tricks of how to teach the course
– cover letter for licensing
– etc.

All the files for the entire course are submitted at the end of development. They will then be ported into the Angel Learning Management System (LMS) and exported in IMS Common Course Cartridge format that allows use in any LMS. Twenty eight of the states thirty four colleges use Angel so this requirement is far easier to meet than ours in BC where institutions are using Moodle, Desire2Learn and Blackboard.

All courses will be licensed for open sharing using Creative Commons (CC_BY). Third party copyright material used in the course must be documented.

Courses will be shared within the state via the Washington Online Angel system and with the world via Connexions http://cnx.org.

It will be interesting to see how this strategy plays out. Sharing with the world through Connexions requires breaking a course down into modules. Anyone else wanting to use that content within their own LMS will not easily be able to do so as Connexions content requires use of the Connexions web site or export as a .pdf which is inherently non-modifiable.

One of the most impressive aspects of the Open Course Library initiative is the recognition that development of digital courses is a team effort. Each faculty course designer gets a grant of $15K. When the call for proposals went out other complementary grants were also made available. Complementing the faculty course designers who provide the subject matter expertise are grants awarded to:
– instructional designers
– librarians
– institutional researchers
– ADA/Disability universal designers, and
– global citizen/multicultural experts

These complementary team members are responsible for supporting the development of all 43 courses. Each instructional designer, librarian, institutional researcher, etc. also has received a $15K grant. A comprehensive team approach like this has tremendous potential for creating exemplary courses. As introductions and presentations were made by all groups over the two days enthusiasm and excitement built as it became apparent support needed to ensure success was in place.

To further enhance the liklihood of producing great courses the instructional design team will use the Quality Matters Program http://www.qualitymatters.org. This program, uses a peer review process and national standards of best practices to ensure a course is designed to promote student learning. I’m particularly impressed with the Quality Matters rubric standards. In addition faculty are required to get feedback on their course design plan with at least two other system faculty in their discipline who are not part of the grant project.

Finally, it is wonderful to see a program like this walk the talk. Communication and sharing about the program and ensuing development is being openly posted at:
http://opencourselibrary.ning.com
http://opencourselibrary.wikispaces.com

In looking around at all the OER initiatives taking place I see a need for more interaction and collaboration between initiatives. It seems everyone is willing to enter into development of OER but few are interested in actually reusing the OER of others or collaborating around building OER collectively to create credentials. Going forward this year I see this as a strategy for BCcampus.

In discussion with Cable Green I committed to assessing what OER resources the BCcampus initiative has generated that could contribute to the development efforts of the 43 high enrollment courses being developed through the Open Course Library. We also discussed the potential to create collaborations and interactions between developers working on OER resources in a common field of study across OER initiatives. I’d love to see a “swap meet” where all developers creating say, biology OER resources, meet and show each other what they’ve developed along with exploring the potential to collaborate together going forward.

This is such an exciting field and I thank all participants at the Open Course Library Kick Off for allowing me to be a guest. It’s fantastic to see the enthusiasm and drive from all involved.