Presentation Outcomes

• Participants will understand the difference between Proficiency-Based Graduation Portfolios (PBGR) and the development of a Proficiency-Based Assessment System (PBAS) that utilizes a “Learning Matrix”
• Participants will give critical feedback about the Proficiency-Based Assessment System (PBAS) known as the “Learning Matrix,” which is being proposed
• Participants will be inspired to join the “Learning Matrix” Movement to encourage the development of a Proficiency-Based Assessment System (PBAS) statewide

Presentation Methods

• Interactive multimedia presentation, including:
• instant polling
• back channelling
• discussion forum
• Click here to go to our event’s GoSoapBox page
• access code: 571-002-069 (if needed)

Presentation Description

PBR’s – Something old is new again, a cautionary tale

At the same time one PBR (Pabst Blue Ribbon) is making a comeback in college towns across the nation, another PBR (proficiency-based requirements) is slowly making a comeback in our nation’s high schools.  Assessment using PBR’s evaluates students on what they can DO, rather than relying on test scores alone.  This philosophy of assessment has been around for more than a century, but only recently have schools obtained the technology that can make this dream a reality.  Come learn more about one school’s journey toward a proficiency-based graduation portfolio system, ponder the possibilities, and explore some of the new technologies that are finally helping schools make this type of assessment a reality.

Presentation Agenda

• Introduce the GoSoapBox tool and have participants login (5 Minutes)

• Click here to go to our event’s GoSoapBox page
• access code: 571-002-069 (if needed)

• Intro to PBR’s – Discuss how Proficiency-Based Requirements (PBR) can improve student learning and why it is a better method for assessing graduation readiness than Carnegie Units (10 Minutes)
• History & Obstacles – Discuss the history of PBR-based assessments and explore the major obstacles that have prevented PBR-systems from being successfully implemented. (10 Minutes)
• Overcoming the First Obstacle – How are schools in Vermont (and beyond) developing their own unique Proficiency-Based Requirements?  How has this been accomplished at my school (Harwood Union)? (10 Minutes)
• Overcoming the Second Obstacle, Why this time is different? – Discuss how new technologies are opening up radically new possibilities and what you can do to help. (15 Minutes)
• Questions and Answers with Special Guest via Skype! (10 Minutes)
• Drink the Kool-Aid – Join the Matrix!

Presentation Outline

Presentation Notes

Intro to Presentation

• To learn more about the interactive collaboration tool I used during this presentation, go to https://gosoapbox.com/
• To learn more about the presentation tool I used during this presentation, go to www.prezi.com
• To learn more about the screencasting tool I used to record my presentation, go to www.screencast-o-matic.com/

Intro to PBR’s: Why a PBR System is better!

What the heck is a PBGR and PBR?

• PBGR stands for Proficiency-Based Graduation Requirements.  PBR stands for Proficiency-Based Requirements.  These terms are often interchanged with the term Competency-Based, and sometimes Performance-Based.

• PBGR’s are a set of standards that students need to show their mastery, competency, or proficiency in.
• A diploma based on PBGR’s can be used as an alternative or as a complement to a diploma based on seat-time and/or Carnegie units.

Why is Proficiency-Based Grading better than what we currently use (e.g. seat-time and Carnegie Units)

• PBGR’s are a more accurate measure of a student’s abilities than a traditional Carnegie-based transcript.
• Under the Carnegie system, it is possible for a student who is ‘good at school’ to get straight A’s and still perform very poorly on state tests and other measures of proficiency.
• Under the PBGR system all students have to show, through their actual work, what they are able to do and what they have actually learned.  It is far more difficult to ‘game’ the system.
• A PBGR system helps students to better understand their strengths and areas for improvement.
• A PBGR system allows students to advocate for greater differentiation and flexible learning pathways.
• A PBGR system allows students to progress at their own pace.  It keeps the high-flyers engaged and it ensures that struggling students are not “left behind” or “pushed along.”
• Encourages the archiving of student work and promotes student reflection of growth over time and exhibitions of learning.
• Under a PBGR system, the learning goals are set ahead of time and it encourages teachers to behave more like learning coaches rather than judge and jury.
• Under a PBGR system, assignments and assessment are more valid and reliable and student expectations are more consistent.  Student work is calibrated through benchmarking.
• A PBGR system encourages teachers to collaborate and work together in an attempt to improve student learning.
• A PBGR system promotes fidelity to the curriculum.
• A PBGR system promotes the disaggregation of content from skills and encourages teachers to more effectively teach and assess skills.

Related Resources

• A great video on grading and assessment from Grant Wiggins and Tom Guskey http://youtu.be/Le6eevthvgI

• As stated in an article from The New England Secondary School Consortium (NESSC), “Simply put, a proficiency-based diploma is a graduation decision based on students demonstrating what they have learned. In practice, it means that every student must show—by writing a paper, delivering a presentation, or completing a challenging project, for example—that they have acquired a minimum level of proficiency and competence when it comes to mastering the essential knowledge and skills they need to succeed in college, work, and life.” Read the article from NESSC here (From the South Burlington’s Big Picture School’s Blog)

• Another great resource that defines Portfolio-Based Assessment
• One last resource on the Advantages and Disadvantages of Various Assessments

The Two Main Obstacles: Why isn’t this system in use already?

Well, it is being used, sort of…

• Pathwise Teacher Mentorship

• Teacher Licensure and M.Ed Programs
• Professional Teacher Portfolios (IPDP’s)

But what about real-life examples of graduate portfolios in schools?

• Google “High School Graduation Portfolios” and you will find one of the largest advocates of Portfolio-Based Assessment, Dr. Helen Barrett (website)
• Dr. Barrett gives plenty of evidence on the effectiveness of e-portfolios, but when you click on the examples she identifies you find all of them lead to dead links. (*one has a few examples from 2005 and earlier)

Examples of Online High School Portfolios from Helen Barrett’s Website on e-portfolios

Why is this?  Why have so many attempts to create portfolio-based assessment failed?

Let’s Listen to the Students

• “Many faculty, educators, and students at Beacon School and other schools throughout the nation find the system to be “very time consuming” and “Officials object to using portfolios for assessment because they are too subjective”. Depending on a given teacher’s standards, a student could pass the assignment with one teacher, and fail it with another.” http://itsoureducation.blogspot.com/
• “Especially in Rhode Island, the problems in education have been miscalculated. Proficiency-based Graduation Requirements (PBGR) allocate augmented burdens of stress to high school students already pressed with an assortment of academic demands.” Excerpt from student letter to Senator Jack Reed
• “For a student, we have to enter different assignments in our digital portfolio and by the end of our senior year we have to show we are proficient in every Proficiency Based Graduation Requirements (PBGRs). If we miss even one PBGR we held back and told that we have to graduate later rather than our assigned year and make up that PBGR in our second senior year or again we dont graduate. We also have to take classes that we dont want to take just to get these PBGRs, like cooking classes or parenting classes when we should have a choice as far as electives goes. It makes no sense whatsoever.
  For the teachers its even worse. Not only do they have to grade papers on hard copies but they also have to grade digital portfolio work. It tends to backlog the teachers a little too much. They also have to make sure we do all of our stuff in there, basically they have to baby us; its a necessity they have to do. Teachers are here to help us graduate, and babying is how they have to do it with digital portfolios when they shouldnt have to at all. Plus, they have to make sure they enter all grades and all PBGRs necessary, when all they should have to do it grade papers.” From Online Petition against MT. Hope’s PBGR Portfolio

Excerpt from the RAND Corporationreport entitled “Can Portfolios Assess Student Performance and Influence Instruction? The 1991-92 Vermont Experience.”

• “The most serious problem was continuing confusion on the part of many teachers about the purposes of the… portfolios and the proper practices to use to implement the assessment system.”
• “The rater reliability of portfolio scores in both mathematics and writing was very low.”
• “Most teachers felt they were unprepared to use the portfolios on at least some occasions.”
• “…evaluation of validity was hampered by the lack of a sufficiently clear definition of the attributes the portfolios are intended to measure.”
• “Teachers also raised concerns about the lack of information from the state and the rapid speed of the reform.”
• “…the portfolio assessment generated some negative attitudes on the part of teachers and principals. Both groups perceived the time and resources demands to be burdensome.”
• “…the partial success attained to date has come at a high price in time, stress, and money.”

So what are the largest challenges?

• Extremely time consuming and difficult to manage
• Can be overly subjective and too difficult to track and assess data. This is why standardized tests have been used.  Not because everybody loved them, but because they were the only form of data that could be managed on a large scale.

• “Gathering all of the necessary data and work samples can make portfolios bulky and difficult to manage.” (Venn, 2000, p. 538)
• “Scheduling individual portfolio conferences is difficulty and the length of each conference may interfere with other instructional activities.” (Venn, 2000, p. 538)

Two Main Obstacles

• “Developing a systematic and deliberate management system is difficult, but this step is necessary in order to make portfolios more than a random collection of student work.” (Venn, 2000, p. 538)
• “Scoring portfolios involves the extensive use of subjective evaluation procedures such as rating scales and professional judgment, and this limits reliability.” (Venn, 2000, p. 538)

An excerpt from an excellent resource on the history of Portfolio-Based Assessment.

“The notion of authentic assessment — judging a child’s work first hand rather than summing it up with a letter or a number — goes back to the beginnings of the progressive education movement a century ago. Even then portfolios were considered time-consuming, but many teachers and students liked them and they became a key part of the alternative public schools that were born in the 1960s and 1970s. At Central Park East Secondary School in Manhattan, Deborah Meier and other progressive educators began to judge low-income, inner city students based on collections of their best work and oral examinations, and found that if they did well on those alternative assessments, they got into college and did well there. “

Read more about the history and discover how a research study in Vermont actually led to a major decline in the use of portfolio as grading tool…
http://www.washingtonpost.com/ac2/wp-dyn/A30980-2004Jul6?language=printer

Overcoming the First Obstacle: How schools in Vermont are working to create PBR’s that are valid and reliable.

• Vergennes Union
• Big Picture Schools, South Burlington
• Harwood Union Middle/High School

• TAKE THE POLL “How many of your schools are working toward common assessments or proficiency-based graduation requirements”

• ENGAGE IN DISCUSSION “As your school has worked to develop common assessments or PBGR’s, what things have worked and what things have not worked?”

Recommendations for Successful Implementation:

• Develop specific, measurable common assessment criteria for observing and appraising student work
• When possible, use criterion-referenced rating approach instead of simple checklists
• Develop rubrics for greater consistency between raters
• Have more than one rater for each portfolio, establish inter-rater reliability through piloting designed to fine-tune rating criteria
• Provide training for raters and check inter-rater reliability
• Recognize that portfolios in which samples are selected by the student probably represent their best work rather than typical work
• Cross-validate portfolios with more controlled student assessments for increased validity

NOTE: While we will never be able to create a completely objective performance-based assessment, they will always be better than the alternative – a standardized test.

A great defense of portfolio-based graduation requirements in a Response to a student from Helen Barrett

Overcoming the Second Obstacle – Finally!
“Developing a systematic and deliberate management system is difficult, but this step is necessary in order to make portfolios more than a random collection of student work.” (Venn, 2000, p. 538)

WELCOME TO THE LEARNING MATRIX!

• The “learning matrix” is NOT a digital portfolio (or e-portfolio)
• A digital portfolio is NOT required for successful implementation of a Proficiency-Based Assessment System (PBAS)
• A digital portfolio should still be used, but not as an assessment or data management tool.  Rather it should be used to create an opportunity for reflection and celebration of learning.
• The “learning matrix” is a Proficiency-Based Assessment SYSTEM that is part and parcel of the everyday classroom experience.

• The “learning matrix” is a grid that keeps track of each student’s’ progress on the school’s Proficiency Standards.

• After schools develop valid and reliable tasks and tools (assignments and common assessment rubrics) then they are populated in this “system”
• When teachers create “portfolio-worthy” assignments, they pull the Proficiency-Based Rubric Criteria from the schools system – quickly and easily!

• After the teacher grades the student submission, the student’s proficiency is automatically tracked.
• The school determines how many proficient submissions are required for each graduation proficiency and when the student achieves that number of submissions they are notified as well as their validation team.

• The validation teams will be made up of teachers, administrators, support staff, and perhaps community partners.  They will be responsible for looking at each students work and determining if the student’s submissions are indeed proficient.
• If the student work is considered to be proficient, they are notified that they have achieved proficiency for that particular requirement and a digital badge or certificate will be offered.  (image here)
• If work is not considered to be proficient, the student will be given feedback on how to improve their work.  Students will revise based on feedback and apply for validation once again.
• Teachers who are frequently not in alignment with others will be asked to participate in calibration training.
• Once the student has received proficiency badges for all of the schools graduation requirements, they will receive a certificate that they have met the “Tier 1” of the school’s graduation requirements.
• This certificate will allow students the freedom to take more electives, or engage in learning outside of the typical school building.
• The “Tier 2” requirements, along with modified seat-time requirements, means that the student will have to continue learning.  But this “Tier 2” learning will be much more highly personalized and it will be something the student actually chooses to learn!

Additional Thoughts and Caveats

• This is a golden opportunity to radically rethink what a diploma should mean.
• We do NOT need every child to be highly proficient in every learning goal.  For example, we do NOT need EVERY child to be proficient in advanced mathematics.
• What our society needs, is for SOME of our students to be extremely proficient in advanced math, and EVERY student to be proficient in basic everyday math.
• This applies to all of our subjects.  We do not need EVERY student to be very knowledgable about U.S. History.  However, we do need SOME students to be extremely knowledgable about U.S. History, and EVERY student to be proficient in civics and what it means to be an active and engaged citizen.
• What this means, is that we need to be very careful about what we expect from EVERY student at the “Tier 1” level.  We need these expectations to be very realistic and we need to realize that these basic proficiency requirements are NOT representative of everything students will learn throughout their career.
• In fact, we might be wise to create proficiency-based requirements for each subject-area class, which will be used to assess whether or not the student passes the class.
• The “Tier 2” requirements should be designed to allow students to pursue individualized learning through felxible pathways.
• As such, this system will allow us to achieve two noble but often contradictory goals: 1) ensure that every student is proficient on common expectations, and 2) allow students to follow their passions through personalized learning opportunities.
• “Diploma Worth Having” by Grant Wiggins

The BIG PROBLEM with the “The MATRIX” is that it does not yet exist (to my knowledge)!

Why not?

• It is difficult to tell, but it is probably an issue of supply and demand.
• Since successful roll-out depends on the ability of a school to create valid and reliable proficiency-based graduation requirements.
• Since very few schools have ever done this, there has been very little demand for a system to manage this information.
• Also, the technology that has existed up this point has made the sleek, well-functioning, and dynamic tool we need difficult to develop.

What can we do?

• Join “The Matrix” movement and encourage LMS developers to create this system

“Hi Matt,

Related to my posting on the forum, we’d love to do a state-wide licensing deal with Vermont so that every school in Vermont had access to the premium version of EDU 2.0 in the cloud. As part of the deal, we would commit to any reasonable set of add-on features that would ensure that Vermont was on the leading edge of e-learning. If there’s anything you can do to facilitate such a deal, please let me know.

Cheers
Graham ”

• Raise awareness about the problems with the ‘traditional” approaches to digital portfolios.
• Get involved on your curriculum committees and send your curriculum coordinators here…

Example Advocacy

Re: Curricula/Proficiencies upgrade
Posted by Matthew Henchen 451 days ago  thread
Sounds great Graham.
We are still waiting on the ability to create proficiency-based rubrics that can be added to any assignment in a stackable way.  Basically, we need the ability to create a rubric for each proficiency and then the ability to choose one or more of these rubrics for a given assignment.  For instance, I could create an Essay assignment on the Civil War and quickly pull the school’s proficiency rubric for Civil War Content Knowledge and Essay Writing.  I could then modify the resulting rubric to meet the needs of the specific assignment.
This would REVOLUTIONIZE edu20 because it would allow teachers to keep track of a student’s progress on any of the school’s proficiencies.  Or better yet, on each of the criteria within the proficiency.  For instance, instead of simply knowing that Jimmy scores a 3 out fo 4 for essay writing, I could see that Jimmy scores a 2 out of 4 for Grammar, and a 4 out of 4 for Organization, etc. etc.  This would allow students and teachers to more effectively identify strengths and areas for improvement.
And if you wanted to make the site even MORE revolutionary – you could create an assignment type that allows students to create their own assignments with access to school proficiencies.
These features, along with individualized assignments and integrated SIS features would make edu20 go viral in the public schools like nothing we have ever seen before, I guarantee it!!!!”

Re: Our focus for this week
osted by Matthew Henchen 255 days ago  thread

“What a timely post.  I’ve been working this Summer with a number of schools in Vermont who are thinking about adopting edu20 schoolwide.
There are two major reasons these schools are adopting your amazing LMS:
1) As a platform for digital portfolios
2) As a digital space needed to manage new one-to-one laptop/iPad deployments
While edu20 has proven to be a fully-featured and comprehensive tool for schools rolling out one-to-one programs, other schools looking to use edu as their portfolio platform are a bit concerned.
Here’s why…
These schools are looking to completely restructure education as we know it.  We are talking about taking education into a totally new and exciting realm where students are assessed on what they can DO, rather than the number of classes they are able to pass.  In order to do this, we need a system that is capable of connecting student work to the school’s pre-determined proficiencies.  We also need a tool that can be organized quickly and easily.
With that in mind, these are the things that will allow edu20 to become a tool for REVOLUTION in education!!!!!

1. the ability for schools to create rubrics for pre-defined proficiencies.  We can already do this on edu20, but we need to be able to organize the proficiency tags at the criteria level, since some criteria will be shared in more than one proficiency rubric (for example, the criteria ‘Grammar, Usage, and Spelling’ will probably be used in many of the school’s writing rubrics)
2. the ability for teachers to pull specific CRITERIA from ONE or MORE of the school’s Proficiency-Based Rubrics (this could be done by allowing the nesting of rubrics, where the teacher can simply delete the rubric criteria they are not assessing)
3. the ability for schools to track student progress at the CRITERIA level.  This feature will provide schools with granularity of data that will make it possible to truly assess student progress.

For example, lets say a student gets a B on a Persuasive Essay on the Civil War.  Under the current system, the student would earn a B on the Essay writing proficiency and a B on the Civil War content proficiency.  That’s fine, but it doesn’t really provide us with the data we need to help the student progress because we don’t know exactly what the student did or did not do to earn that B.  Perhaps, they are brilliant writers but really didn’t know their history very well.  This student should earn an A for Essay writing and a C for history.  Or let’s say they generally write good essays, but they consistently struggle with organization and flow issues.  Unless we are able to see the data at the CRITERIA LEVEL, we will be unable to pinpoint those areas that need further growth.
The last thing we need is a bit more straightforward.  We need a way for students to better organize their portfolios.  A drag and drop interface would be wonderful for this.
There are hundreds of excited educators in Vermont dying for these features and I am just wondering if these features are still on your to-do list when you say things such as: ‘proficiency tagging of assignment submissions’
Thank you so much for everything you’ve already provided us. “

The Final Proclamation!

I am critical of the Digital Portfolio movement for the same reasons that I am critical of the U.S. when I teach History – because I care so deeply about both.  The success of a Proficiency-Based Assessment System, just like the success of our country, will depend on how well we learn from the past.

• Adding Proficiency to Edu20 – http://www.youtube.com/watch?v=E7ZdE4dzwHQ
• Graham Glass on future of education – http://www.youtube.com/watch?v=q2HbmDwvoFc

Join the Matrix!

Recommended Roll-Out

While I strongly believe that the “Learning Matrix” and proficiency-based assessment will lead to transformative change, I am not advocating for radical or swift revolution.  If the study of history has taught me anything, it is that true progress evolves slowly over time.  Radical change is often counterproductive and ends up creating a new series of issues, which are often worse than the initial problem that was being addressed.  Understanding the benefits of the “Learning Matrix” will take time for many educators and we must be patient, but consistent.  We must achieve the revolution we desire, not through top-down mandates, but through a grassroots democratic movement.  We can do this because we believe so strongly in the power of the idea.  Viva la revolution  the frustratingly slow, bumpy road to transformation!

YEAR ONE

• Start with the structures that already exist at your school.  If your school is comprised of departments then have each department identify its “essential proficiencies”.
• Perhaps start just with skills – these are generally more interdisciplinary and easier to develop high-quality rubrics.
• Have the department gather examples of common assessment rubrics that already exist on the web.
• Have each department create (or adopt) good rubrics for 2-3 essential skills.  Make sure that this work is done through a consensus model, as much as possible.
• Have each department share their essential skills with one another at a full faculty meeting.  Group similar proficiencies together and then ask for comments about what other skills should be listed as a graduation requirement.
• Identify a small number of proficiencies and develop the school-wide rubric in interdisciplinary groups.
• Populate the Proficiency-Based Assessment System (perhaps edu20) with your proficiencies and rubrics.

YEAR 2

• Ask that each teacher creates the opportunity for their students to create one or two artifact that aligns to one of the schools identified graduation proficiencies.
• Create “benchmarking groups” who will calibrate student work and give feedback to the faculty.
• Identify which proficiencies students had the opportunity to work on and identify redundancies and holes.

YEAR 3

• Ask teachers to map out one or two more additional proficiency opportunities and make sure that all students have the opportunity to achieve all of the schools identified proficiencies.
• Conduct another round of benchmarking and calibrating.

YEAR 4

• Re-visit your schools PBGR’s and add, delete, or modify
• Revise rubrics and share great lessons that have been developed
• Map and teach, benchmark and calibrate.
• Make the PBGR’s official!

YEAR 5

• Teach, revise, calibrate etc. etc. etc

Additional Related Resources:

Daniel Pink
Graham Glass
Chris Lehrman
Sir Ken Robinson

Kindred spirits.  That’s the first thought that comes to mind whenever I read anything from educator Grant Wiggins.  In a recent article entitled, “A Diploma Worth Having,” Wiggins lays out many of the common sense changes that will truly transform education as we know it.  This is a must-read for any educator interested how we can effectively improve our schools.

“If we train kids to be workers, that’s what we’ll get, but if we teach kids to be citizens… then we will also get the husbands, and wives, and fathers, and scholars, and activists, and community members, and parents, and friends that we need.”