As we shared yesterday, CLRN’s review process strives to be fair, unbiased, and non punitive. The CLRN Certification process is one way to shine a light on those courses that meet most of the course and online standards. With CLRN being a state-funded service, we do not charge for our review services. Still, course reviews are time and resource intensive, particularly when you’re striving to be thorough and consistent. While we hope all publishers will continue to improve their courses to be more interactive, engaging, and responsive, we’re unable to conduct full reviews of the same course when publishers slightly improve a course to better meet the standards. Because we want to continue to offer our services at no cost, we’ve modified our policies to provide publishers a narrow window when they’ve made specific improvements to meet the CLRN Certified criteria. Our new policy is listed below.
Secondary Review: New evidence for content or online course standards.
When CLRN’s review teams examine a course for its alignment to content standards and to iNACOL’s Standards for Quality Online Courses, they utilize publisher-provided correlation documents, also known as standards maps. These detail where reviewers should find evidence to confirm a standard’s presence. When a standard is not fully met, CLRN review teams provide a comment about components that are missing.
Once CLRN’s review is complete, we inform publishers that the course review will be published in seven calendar days. At this point, publishers may notify CLRN to publish the review immediately or they may allow the seven days to proceed with CLRN publishing the review afterwards.
However, if a publisher believes that there is new evidence, not previously submitted, that would clarify our review, they should notify CLRN during the seven-day window. CLRN will place a “hold” on the review and will schedule a secondary review for the specific standards for which the publisher provides new evidence. Once this secondary review is completed, CLRN will again inform publishers that the review will be published in seven days. Each course is allowed one secondary review.
Supplemental Update: New content and/or course standards.
Once a course review is published online, CLRN generally does not allow the resource to be re-reviewed for three years unless there is a major update to the course. However, in the case of courses that have not met the qualifications to be CLRN-Certified, CLRN may re-review courses provided the following:
Publishers have augmented the course to strengthen its alignment to the content standards and/or the course standards. Sufficient changes have been made to bring the course into compliance with the CLRN-Certified specifications.
Publishers provide specific evidence and locations about the standards that have been added or strengthened.
CLRN’s first priority during review sessions is to examine new course reviews. CLRN will conduct supplemental updates on a space available basis. Courses will remain in their published state during this time and publishers will be notified of the review results. No secondary review is available for supplemental updates. One supplemental update is permitted per course per year.
Major Course Update.
Courses that have experienced a major update contain substantially new content to meet the Common Core State Standards or California’s original content standards. The course’s structure, media integration, degree of interactivity and engagement, and learning management system have been significantly improved to better meet iNACOL’s course standards, particularly the 11 Instructional Design standards.
Major Course Updates are considered new courses and are reviewed in the order they are received.
With yesterday’s announcement of CLRN’s partnership with the University of California, online courses must now be CLRN-Certified before U.C. will review them for their A-G requirements. That we’ve set the bar at 80% of both the content standards and the online course standards may be seen as too generous. However, we believe it’s a fair starting point, given that not all standards are equal. The addition of our online course Power Standards requirement ensures that critical standards are met in order to earn CLRN-Certified status. We believe our partnership will help drive course quality and provide stronger courses to our customers.
So, how have CLRN’s currently reviewed courses fared?
Over the past year, CLRN has reviewed 55 online courses from seven publishers. 15 courses from six publishers, 27% of the total, have earned the CLRN-Certified badge. You can easily find certified courses through our search page . Simply select the “CLRN Certified” check-box and narrow your search by subject or grade level. In addition, we’ve added a CLRN-Certified badge to all certified course reviews.
11 additional courses from four publishers met the 80%/80% requirements for content and course standards, but failed to earn certification because of a single Power Standard: online course standard D10, which requires that course materials are accessible to all students. In each of the 11 courses, instructional lectures and/or narrated presentations did not include closed captions or transcripts, which we believe is an easy fix. If each of these courses are modified to meet D10, and we hope they are, 47% of our reviewed courses will be CLRN-Certified. For more information about why D10 is important, read “What Part of the Accessible memo didn’t you get?” from my “Advice for Course Creators and Buyers” post.
The most common problem with the majority of courses that are not certified is with their alignment with the content or Common Core State Standards. While there are exactly 52 online course standards, the number of content standards per course varies between 19 for AP Probability and Statistics to 106 for ninth grade English-language arts courses aligned to CA’s original standards. Regardless of the number of content standards, the 80% requirement applies. 20 courses (36%) failed to teach 80% of the content standards for their course. These range from a low of 34% to a high of 78%. The median (half the courses had more, half had less) percentage of content standards met was 57%.
As we mentioned in the beginning, we hope our partnership with the University of California helps drive course quality and improvement. While the CLRN’s reviews are effective for three years (and certification ends three years after a review is published), CLRN typically will not re-review a course unless there is a major update. However, because we don’t want the certification process to be punitive, we’ve created new policies to encourage course improvement. We’ll share these in tomorrow’s post.
It’s been quite a year for the California Learning Resource Network. After investing a year to create the standards and review process for online courses, CLRN began reviewing English-language arts and mathematics courses last summer, spending much of last year training our reviewers and norming the process to ensure consistency. Last spring, we added history-social science, science, and visual and performing arts courses and plans are under way to add world language reviews this winter.
During 2011/12, CLRN reviewed 55 online courses from seven publishers: Accelerate, Aventa, Education2020, K12, Inc, Pearson Digital Learning, Plato Learning, and Thesys International. Our 2012/13 review queue currently holds 88 course submissions, including many from Advanced Academics, Apex Learning, Cambium Education, National University High School, and Odysseyware. Still, many of our customers have asked us about course approval for the University of California’s A-G requirements, since A-G approval is required for courses students take for admission to the University of California. We now have an answer.
To achieve CLRN-Certified® status, online courses must address at least 80% of the course’s content standards (Common Core or California State Standards) and 80% of iNACOL’s Standards for Quality Online Programs. Fifteen online “Power Standards” must be among those verified by CLRN. These Power Standards include: Content: A3, A9 & A13; Instructional Design: B3, B4, B5 & B10; Student Assessment: C2, C3 & C4; Technology: D4, D10 & D11; and Course Evaluation and Support: E3 & E10. Course publishers may utilize the CLRN-Certified® term and logo in association with their certified products.
By request of the University of California, CLRN reviews will now include the “Not Met” content standards within posted reviews. Also displayed on CLRN’s browse page and within each course review, will be the percentage of content and online standards met. CLRN-Certified® courses will include both a reference to its CLRN-Certified® status and the new CLRN-Certified® badge.
With online and blended learning growing at double-digit rates and sliding past the tipping point, publishers and schools are both getting into the course creation business. It’s a little bit of the Wild West out there and we expect new publishers will appear on a regular basis. Remember when there were dozens of textbook companies? Eventually, the market consolidated to the three primary textbook publishers we see today, but until that happens, we’ll be tracking eLearning’s evolution. Still, like early digital cameras just before their tipping point, some eLearning courses are primitive, two-dimensional, and, well, boring.
There. I said it. Some emperors have pretty ugly clothes.
However, as this disruptive innovation continues to mature, we strongly believe that courses will increase in complexity and sophistication. Capitalism and competition will drive publishers to increasingly raise the bar, offering more engagement, media, and assessment choices.
For now, though, through our experiences with a wide variety of online and blended courses, I’d like to share some thoughts about what a great online course should (and should not) be, whether you’re creating your own or if you’re about to go shopping for one.
1. A Textbook is not a course
What if a teacher threw a book at you and said, “Read this and take the tests at the end of each chapter. When you’re finished, you’ve passed the course”? Would you choose that teacher or curriculum for your child? Would you recommend that course to others? What are the odds that an average or struggling student would complete the course, or better, be proficient in its skills? What do you think the AYP of this class would be compared with an engaging face-to-face course? A great course should include textbook content, but if teaching and learning doesn’t extend beyond the page, and the course is primarily limited to reading, you’re better off throwing a kid a book and locking her in a room with it.
So, if a textbook isn’t a course, why do so many courses look and feel line one?
2. Your course should be better than the worst teacher in my school.
The Ferris Bueller clip below is my generation’s idea of a boring teacher.
Aim higher than this economics teacher. Much higher. Please. Because in some cases, I’d rather send my child to this f2f class.
3. There can’t be quality outputs without quality inputs.
I understand the outcomes-based performance indicators movement promoted by iNACOL, Michael Horn, and others and I look forward to an agreed-upon set of evaluation criteria that can be applied to both virtual and brick-and-mortar schools. Still, until these metrics are created and standardized, we’ll all be subject to our current criteria: course completions, graduation rates, drop-out rates, and Annual Yearly Progress (AYP).
But for me, it’s Garbage In, Garbage Out. You can’t have quality outputs unless you begin with quality inputs. As Charles Babbage was asked, “Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?”
Students learn more from active discussions than from traditional lectures, and they need instructors who can engage them in the material, notes Diane Johnson, assistant director of faculty services at the Center for Online Learning at Florida’s St. Leo University. “Teacher quality is still a very important part of success in an online course, but so, too, is the course design,” Johnson says.
Teacher quality & course design. Quality In. Quality Out.
4. Just what part of the “Accessible” memo didn’t you get?
Your school built wheel chair ramps even though you may not have had any students with that need because some day a parent, student, or teacher WILL have that need. You also did it because the Federal government made you, but it was the right thing to do, yes? Of course. We want all students to have equal access to a quality education.
Should that be any different with online courses? eLearning’s equivalent of a wheel chair ramp is accessible media, particularly narrated presentations and video lectures.
Accessibility requirements for electronic media was confirmed by a US Department Of Justice letter to college presidents. In it, the DOJ reviewed its lawsuit against universities that had piloted the Kindle DX, which the DOJ won because the Kindle DX did not have a text to speech function. The letter states, “It is unacceptable for universities to use emerging technology without insisting that this technology be accessible to all students.”
Accessibility criteria for online courses was pioneered by the Texas Virtual School Network (TxVSN) which has created a detailed site, TxVSN Accessibility, with advice, criteria, and check lists. Here are their Accessibility Guidelines.
One of CLRN’s requirements when reviewing courses is that all narrated presentations and videos include transcripts or Closed Captions.
So, yes, when you’re constructing an online course, your customers, and the Office of Civil Rights, expects that course materials are accessible. There are free resources to assist you, including Universal Subtitles, which can be used to embed captions within YouTube videos.
Make use of them.
5. Consistency between courses is key
We’ve noticed that often two online courses taught in sequence, like English I and English II or Algebra I and Geometry, have very different levels of interactivity and engagement, as if they were created by two very different teams who weren’t talking to each other. While one course may have a variety of media, narrated presentations, and video lectures, the companion course would not. The basic pedagogy of each course would be the same, as if course developers were working from a template, but one course would be significantly weaker than the other.
My advice? Hire great educators to develop your courses and make sure a separate person is in charge of quality control. Include textbook snippets, but make sure lessons are embedded with short video lectures, discussions, collaboration opportunities, and media.
6. Formative Assessments Should Inform Instruction
Too often, as in most cases, they don’t.
If a student is struggling, a formative assessment is where we first discover his difficulty in grasping a concept or idea. If, after failing a formative assessment, your course just passes the student on to the next unit without 1) reteaching or 2) informing the teacher there may be a problem, you’re competing with my worst teacher for last place.
There’s a reason teachers love “clickers”. They enable teachers to instantly assess whether students comprehend concepts and they offer “just-in-time” teachable moments if students don’t.
Formative assessments, generally short multiple-choice quizzes, don’t have to be graded. They DO need to be an opportunity to determine whether a student is grasping a concept. When students have difficulty, the course should attempt to reteach using new examples or new methods until the student is ready to move on.
Otherwise, your course isn’t much better than a book.
7. Vary Your Assessments
You do know there are other assessment types besides multiple-choice, yes? Then why do so many courses use multiple chose tests exclusively? It’s fairly common for a course to have frequent formative assessments, consisting of a few multiple-choice questions that gauge a student’s progress. That’s great, and often appropriate. However, when multiple-choice tests are used exclusively in a course, you’ll never know whether a student has synthesized course concepts.
There’s a reason principals frown when teachers park themselves at the school Scantron machine scoring quizzes. Sometimes a multiple-choice test IS the best way to test knowledge. However, it’s never appropriate to use it exclusively. A great online course should have a variety of assessments, some which provide feedback about knowledge and comprehension, while others challenge students to apply, create, analyze and evaluate. The less a test is “Googleable”, the better chance you’ll have of actually evaluating student progress.
During the past 18 months, after I’ve delivered conference presentations about online learning, I’ve enjoyed the pleasure of answering questions from attendees as well as the opportunity to ask teachers and administrators about their online learning programs. In too many instances, I’m finding that districts haven’t thoroughly worked through a planning process that involved all stakeholders. Nor are districts carefully vetting online resources before purchase. Before we get to today’s sermon, I recommend you view a recent VW commercial entitled “Is it Safe?”
In this ad, we first see a young boy who is shopping for a bicycle asking the sales person,, “Is it fast?” The salesperson replies, “It’s got 10 speeds, my friend.”
The boy is a bit older and is shopping for a scooter. He asks the salesperson, “Is it fast?” and receives the reply, “It’s got a lightning bolt on it, doesn’t it?” As an older teen, he’s now shopping for a car and asks, “Is it fast?” and learns from the sales person, “Fast? I don’ t even know if it’s street legal.” In each case the consumer, in their search for a new product, trusts the salesperson as their sole source of information about the product. Is it fast? Is it safe? Sure. Ok, I’ll buy it.
This kid’s an idiot.
Sadly, too many school districts act just like him when they select online courses.
Yes, they do.
Do school and district administrators make their purchasing decisions based on a vendor ‘s presentation?
Absolutely.
For proof beyond my anecdotal experiences, I direct you to the Technology and Telecommunications Subcommittee (TTSC) survey of districts about how they selected credit recovery programs. The actual survey results may be found here:
Asked to “Describe the selection process that was used by your agency to choose your Online Credit Recovery Program”, the most popular responses were: 1) vendor demonstrations; 2) webcast; 3) regional collaborative; 4) using iNACOL standards; and 5) open bid to vendors. Most of these selection criteria are no different from asking, “Is it fast?” Basing a teaching and learning solution on either cost or a vendor demonstration is as irresponsible as the teenager who buys the salesperson story that a scooter is fast because it has a lightning bolt on it.
So, how should you buy a car? First, given the importance of purchasing a new car, no one should rely on a single information source or filter. Just listening to a friend’s recommendation, a magazine ad, an online review, or a sales person’s pitch is folly. Some of these should be among the initial filters that determine whether the car might meet your needs. Your friend’s recommendation provides you personal experience and the online review tells you the car’s strengths and shortcomings. Once a car passes your initial selection criteria, your next step is to take the car out for a test drive. Sitting behind the wheel, you’ll discover how the car handles and whether its features are accessible and easy to understand. Is the car comfortable, does your family like it, and does it meet your needs? Then, and only then, should you begin price negotiations.
The same should be said for selecting online courses.
Perhaps your initial attraction to an online course or course publisher will take place in a magazine ad, through a friend’s recommendation, or in a conference exhibition hall. We recommend that you that begin by examining how the course meets both the content and course standards. Is the math or English-language arts course aligned to the Common Core State Standards? Are courses from other subjects aligned to your state standards? Has an independent agency confirmed those alignments? Online courses are more than just content, though. They also represent a learning platform with learning activities, lectures, formative assessments, and student-student engagement. To judge course quality, has the course been compared to iNACOL’s criteria for quality online courses? Doing so will tell you if the course content is rigorous, that activities are engaging and involve high order thinking skills, that students receive timely and frequent feedback, and that all materials are accessible to all students.
California, Texas and Washington all have online course review projects that review for content and course standards. The California Learning Resource Network (CLRN) reviews courses for their alignment to the Common Core State Standards. For each standard, CLRN evaluates whether the course demonstrates, practices, and assesses each skill. CLRN also compares courses to iNACOL’s 52 course criteria in the areas of content, instructional design, student assessment, technology, and course evaluation and support.
However, content standards and online course standards validation is just one filter you should consider when you select a course. Just as you would use Consumer Reports to narrow down your selections for your next car and CLRN to narrow down your course options, your next step is to visit the dealer to take a test drive. You don’t want to watch someone drive the car. Nor do you want the salesperson to drive. The best test drive is one where you take the car through a number of scenarios: turns, traffic, freeway, and parking. The same can be said of an online course test drive.
Ask your provider for both teacher and student accounts for the course. Most have demonstration sites where multiple test drivers are checking out a course, away from actual student data. Then, spend some time getting to know the course as a student. Select several units, complete all activities, and take the formative assessments. Now, be a gifted student who answers all the questions correctly. Do you find the course highly engaging? Are you challenged beyond knowledge and comprehension within the questions and activities?
Next, be a struggling student. Have trouble completing work. Fail the formative assessments. How does the course react? Does it provide alternative paths to proficiency or reteach using different examples or modalities?
Then, be a teacher, and check out the learning management system’s features. Can you add content to the course? Is it easy to communicate with students, set up discussions, and find student grades? Are there a variety of assessments beyond multiple-choice tests and are those assessments matched to the content?
Yes, ask your friends what courses they use, check out CLRN’s content and course standards review, and listen to the publisher as they describe and demonstrate the course. However, you must conclude by thoroughly participating in the course as a student and as a teacher. Then, and only then, will you know if the course is suited for your students.
Learning in the 21st Century: 2011 Trends Update, a collaborative report by Blackboard and Project Tomorrow, shares results of the fall 2010 survey of nearly 300K students and 35K teachers. The report, released June 28th at the ISTE Conference, focuses on online learning’s growth, potential, and barriers.
Briefly, Project Tomorrow found that 30% of high school students have taken an online class, a three-fold increase from 2008, confirming data from other sources that online learning is steadily approaching a tipping point. Forty percent of middle and high school students believe that online courses are an important part of their school experience.
As with prior surveys, which indicated that mostly non-consumers were engaged in online learning, two of the top four reasons for taking online courses, according to administrators, are academic remediation and scheduling alternatives. However, the top two reasons this year, at 37% and 32%, are “keeping students engaged in school” and “increasing graduation rates,” which hints that something called “product substitution”, online course adoption by regular consumers, is beginning to occur. In 2008, 23% of administrators felt that online courses were helpful to effectively engage students in school. Now, 38% believe so.
However, student responses still align with the non-consumer class and are consistent with other survey data. The top four reasons students cited for taking online courses were that courses better fit their schedules, that they could earn college credit, that the course was not offered at their school, and that they could more easily review course materials. Strangely, middle school students seem more connected to online learning’s potential to create a “more engaging, personalized, and collaborative learning environment” than their high school counterparts.
Barriers
So, what’s stopping the online learning revolution from growing faster or being integrated in more schools? According to nearly a third of all administrators, and confirmed in other surveys and blogs, questions about quality teacher-student interactions, academically rigorous content, and online course evaluation are a concern.
What do administrators want to know about online courses? First, it’s about results (or as Michael Horn advocates, outcomes). 54% would like to know whether taking an online course resulted in greater student achievement. This is an important question, one requiring a longer, and more focused blog post. However, whether it’s an end-of-course test or a yearly state assessment, administrators want to know whether they’re being sold a fantasy or an engaging course that results in students having mastered the content.
36% of administrators believe it’s important to see completion rates for each course. 77% need to know that a course is aligned to content standards, while only 17% expect that courses are aligned to iNACOL’s standards for online courses. Ease of use concerns 63% of administrators while embedded assessments attracted only 40% attention. 42% prefer that online course content is developed by teachers or curriculum specialists, but only 20% are expecting that a course was recommended by their state department of education or a professional organization.
So, the good news is that administrators are turning their attention to the selection process for purchasing online course content. Rigor, content alignment, and achievement results top their lists, which is good. However, we believe that as more administrators struggle with course selection, CLRN’s online course reviews will provide them objective information about both standards alignment as well as details about how each course compares to iNACOL’s new online course standards that focus on content, instructional design, student assessment, technology, and course evaluation and support. CLRN‘s reviews will also contain both student and educator feedback surveys which will provide our customers more detailed information about how both groups experienced each course and whether the course met their expectations. Our work has only just begun.
A few weeks ago, Utah legislators introduced Senate Bill 65 (SB 65) which incorporates many of the recommendations made by Digital Learning Now’s report, “10 Elements of High Quality Digital Learning.”
SB 65 creates a Statewide Online Education Program, which would review online courses and provide them to high school students through a newly established Electronic High School. More importantly, authority for the program would come from Utah’s State Board of Education (SBE), which would help establish some of the rules to approve online providers.
The Utah SBE would be required to use iNACOL’s National Standards of Quality for Online Courses as well as to review courses for their alignment with the core curriculum standards. The SBE would also prescribe the qualifications for online instructors. The SBE would not be able to limit class size or set a minimum duration for a course.
ADA would float with students. If a brick-and-mortar student took one online course from another provider like the Electronic High School, the online school would be entitled to a portion of the student’s ADA from the student’s LEA.
