Improve the improvement process

The recent Tasmanian School Report Cards attempted to accurately communicate school improvement but failed on at least three points

• The community read the reports as being measures of performance (see the previous posting), and thus
• Excellent performance was hidden by results of "Trend Downward' when a performance indicator had declined slightly (a fraction of a percent)
• And the reports focused on 'measurable' items without fully explaining
• That the improvements were based on measures of different cohorts, eg, this year's Kinder group is different from last year's, however
  
• That the different cohorts were presumably assumed to be equivalent (which is highly unlikely in small cohorts such a staff (Staff Attendance) and Kinder (School Readiness)
• The statistical limitations arising from small samples sizes in many schools, particularly in small schools
  
• Why the measures were individually and collectively valid for inclusion in the school report card (staff attendance?)
  
• Why the things measured were sufficiently significant to be included (staff attendance?)
• The quality (how current, comprehensive, and complete) of the data and the limitations on the data available
• Staff attendance was measured as a percentage of total staff attendance in only two successive years.
• Presumably a single staff member with a emerging chronic health problem could 'cause' a significant decrease in staff attendance.
• The highly specific (narrow) nature of the data used in some measures.
• Readiness for school was only measured for late Kinder students whereas readiness for school is an ongoing and daily for issue in relation to some students
• The interaction of most of the indicators
• attendance, retention, literacy, numeracy, student satisfaction, parent satisfaction and readiness for school all interact with each reinforcing the positive or negative effects that emerge for individual students.
• What other measures were not included (and perhaps why)
• The Report Cards did not include any information on the schools' (improving?) provision for students with special needs, disabilities, disorders... in the cohorts being reported.
• Similarly they did not include the school's provision for students with behaviours of concern and for families in distress (thus requiring support) were not reported yet these are some of the major constraints on schools and on student and staff success and well-being.
• The need to deal with problematic student behaviour is such that it determines aspects of the actual organisation of many schools. It may also consume a large proportion of the resources available ... resources that could be used to provide higher quality education.
• Certainly problematic student behaviour is far more significant than staff attendance in every school with which I am familiar. And some limited data is available in this area. Why as it not included?
• The report cards did not contain any contextual information related to, say, the percentages of students with additional needs (behavioural or special needs)
  

Since the School Report Cards will become the direct focus of initiatives to achieve real improvements, it is important that the data reported is comprehensive, valid and useful in relation to the overall success of the school, its students, staff and community. With questions over the current form (language, content...) of the Report Cards and the possibility that key data has been omitted, the next step must be to learn from this experience and act quickly on what that learning reveals.

There is clearly a need to improve the improvement process. By doing so, the system will model the very actions and strategies that it is hoping to promote in its schools. If it fails to do so, it runs the risk of alienating the very people upon whom it is dependent for achieving the improvements it desires.

Useful starting points for improving the improvement process might include

• Collating, summarising and reporting the same data at various departmental levels: cluster; learning service, whole of system
• Inviting schools to respond by reporting how, and to what extent, they make sense of their own report cards. As one recent corespondent wrote:
  
  "We spent some time y'day on our school report, personally I don't think they're going to be a big deal. It's too hard to draw worthwhile conclusion about your own school from them, let alone any real comparisons with other schools."

The performance- improvement trap

The Tasmanian School Report Cards strategy may have fallen into the performance - improvement trap in several ways.

The message communicated is always the message received. The Department/Government has attempted to report on school improvement but the School Report cards have been widely received as reporting on school performance. This makes sense, since student 'report cards have always reported student performance (albeit with some comments on improvement or otherwise).

The relationship between performance and improvement is an interesting one. While performance and improvement are directly related there are some subtleties requiring attention:

• measures of performance are used to to calculate improvement (or otherwise)
  
• measures of improvement do not indicate actual performance
• isolated measures of performance do not indicate improvement
• similar levels of performance may be part of very different degrees of improvement
• similar degrees of improvement may be part of very different levels of improvement
  
• the ease of improvement is (generally) inversely proportional to the level of performance
  
• too few measures of performance may not provide valid indications of improvement
  
• improvement is usually less likely (and more difficult) with better performances
• a perfect performance will result in either no improvement or deterioration
• poor performances are often very easy to improve
• to understand performance and improvement one first needs to understand variation
  • there is always some variation in any system
  • some variation is a result of the system
  • some variation comes from outside the system
  • variation in performance may not be an indication of improvement or deterioration at all... just variation
• one also needs to understand both change and improvement

Reporting performance is personal for those involved. In the book "The Greening of America" the author suggested that to 'assess another person is an act of violence'. Misreporting or misrepresenting performance is even more an 'act of violence'. The damage may be done to individuals, their confidence in themselves and each other and to their relationships.

The current School Report Cards strategy has already done considerable 'violence' in some schools. In its present form the costs involved are likely to greatly exceed the value added.

Repairing the damage done will not be simple. First impressions tend to last and some of the information contained in the Report Cards is clearly invalid or simply not relevant for particular students and their families. For example, I understand that a Kinder student who cannot stand on one leg for 10 seconds is deemed not ready for school. Does this mean that a student with cerebral palsy will never be ready for school? A school with significant improvement in an area may be rated Excellent while performing much less well than another school whose performance in the same area has declined slightly hence being rated as Trend.Down. The impression given by the Report Card contradict its intent. The gold medal goes to the athlete who wins the race not the one who achieves their personal best.

But simply correcting or discrediting the School Reports is a low level response strategy. It will be important to raise the level of the conversation around this matter. Quality data that is comprehensive, current, accurate and valid will be required but it will not be sufficient. And it is not yet clear that the data in the recent School Improvement Reports meets these criteria.

Fundamentally, it is about constructive, collaborative change management in order to achieve ongoing and sustainable improvement. Now there's a challenge!!

School performance - everyone's responsibility (no bystanders)

The recent Tasmanian School Improvement Report Cards are definitely about school performance. They were published online by the School Performance Services unit of the Department of Education.

And they were effectively league tables. The report cards were published in local newspapers as tables that facilitated comparisons between schools. This immediately confirmed the fears of many people involved with schools and contradicted the Minister's claims that the publishing the report cards would not result in league tables.

School improvement means improved school performance. But 'school performance' is actually the performance of all those involved: staff, students, their families, the community, the related professions, academia, the Department and the Government. It takes everyone working (and learning) together to make a school great.

Great schools help those involved to meet many of their needs, especially needs related to learning and being members of a community. In this sense, schools are best understood as purposeful communities in their own right. They are not 'numeracy and literacy factories' even though literacy and numeracy are very important.

In purposeful communities,

• members interact on the basis of shared purposes (derived from shared values)
• the community includes all those involved
• roles provide some useful structure
• everyday working relationships are the foundation for achievement
• members contribute according to their respective capacities
• responsibility and authority are dynamic
• the community is connected beyond its immediate locality
• …

I find the ideas implied by the School Improvement Reports somewhat confusing. In fact, I have been trying to analyse how 'schools' are being understood in this context. Who or what is a school? Is it a 'factory'? An institution? Is it the school staff? Is it the Department in a particular locality?

Measures such as staff, student, and parent satisfaction and staff attendance included in the School Improvement Report may imply that ‘the school’ is the Principal. But this is inconsistent with other measures such as student attendance, early school readiness and reporting to parents. School readiness and attendance are primarily outcomes of the family and reporting is highly prescribed by governments. I wonder if this inconsistency could be contributing to the current low levels of interest in principal positions?

The major educational initiatives in recent years have been focused on the structure of the Department, the curriculum, and assessment and reporting. It has been a long time (last century, in fact) since in-depth consideration was given to the nature of schools. This is not surprising given the current dominance of psychological thinking in both education and management. The nature and performance of schools also need to be considered in sociological terms - something sadly lacking in the current context.

What next? Clarifying the above issues will be a major challenge and a genuine opportunity for real gains. Failing to accept the challenge is likely to result in even greater polarisation of positions.

It would be easy to overlook the implications of using data extensively. Using data can be a double edged sword. Data does not have any meaning in its own right. The task is to construct useful knowledge from valid data and this means:

• checking concepts and assumptions (see above)
• understanding the current cultural and historical context in relation to the data
• using these to making sense of the data in order to construct the knowledge required for
• developing responses (actions and arrangements) that are likely to achieve sustainable improvements

And there are no guarantees. The task of improving school performance is not an engineering task – cause and effect are not often consistent over time and place. Similarly solutions may not be directly connected to the causes of problems. Contrary to everyday thinking, what works well in one school may not be all that useful in another. School Report Cards may be dramatically different but it is not always clear which school deserves the greater recognition for its actual achievements ('performance').

School improvement can only be achieved one school at a time and it takes everyone involved, working together to see that it happens. No-one can be a bystander.

Butterflies and storms

One of the key ideas of chaos theory is that small changes (a butterfly flapping its wings in the Amazon) can have a big impact (a storm in New York).

In a school with which I am associated, there has been a total turnover of staff in the office area (edge of chaos?) and several small but significant changes have occurred quite rapidly. The office now opens at 8.30,

Previously some office/admin people had been arriving earlier and the office had been informally open prior to 8.30 and they dealt with early bird parents (and other visitors) and students, and teaching staff who need to be in their classrooms by 8.30.... On any day, there were only a few of these early-bird 'customers' and for some of them the new arrangement can be 'catastrophic'.

For the rest of the school, it appears to be business as usual, except for the disruptive consequences of what might flow from the new arrangements.

Thus this small change (the butterfly) results in

(a) No consequences for most members of the school community

(b) Minor inconvenience for some, but...

(c) Very difficult situations for some individuals and some activities, and

(d) Occasionally, large scale disruption in terms of being catastrophic for all concerned and organisation as a whole.

On a small scale, the organisation is often rescued from the above by individuals and small groups (communities of practice) who absorb, contain and/or develop workarounds to deal with the disruptions and overcome the dysfunctions.

Not opening the school office until 8.30 is the butterfly that causes some storms (problems) for some people in other places and at other times . It is true that many of the problems arising are simply minor disruptions and/or some dysfunctions in other parts of the school throughout the school day. Many are hardly noticed. And there are also resultant storms in other places well beyond the school. Examples include: parents are late for work; misunderstanding occur; important matters not addressed because of the lack of an opportunity to communicate;...etc.

Whose performance?

Increasingly, education systems worldwide are reporting public school performance based on measures of student performance. In some countries this practice includes serious consequences for under-performance, e.g, closures, mergers, staff sackings...*

.

Most of the discussion on 'school performance' misses the main point. It is the government's responsibility to provide good quality public education for all its communities. Schools simply act on behalf of the government. Individual schools are directly guided and constrained by the government's policies and practices. Individual 'school performance' is actually government performance in that locality. In releasing the data, the government is actually reporting on the effectiveness of its own policies and practices. Viewed in this was the data is always interesting.

 

(*) One can only presume that these consequences are intended to imply the existence of strong committed leadership by those with responsible. But that begs the question: If those leading the system are so committed and strong, how is it possible for schools in the system to under-achieve?

More Emperor's New Clothes

Australian state and federal governments are keen to publish performance comparisons between like schools. But one has to wonder...

"Why are parents so concerned about comparing their own school with like schools?"

Of course they are not!! When was the last time anyone heard a parent wanting to compare like schools? .
The approach is widely understood as an attempt by governments to put all schools under pressure to improve by publicly identifying those that are 'under performing'. Publishing the results is based on a rationale of 'transparency'. But the rationale of transparency is patently false in a number ways :

• The approach actually hides the limited ability on the part of the education systems to promote and manage school change and improvement in more effective and constructive ways
• By focusing on 'school performance' the approach also hides the damaging effect of governments and education systems as they continually tamper with schools through demonstrable policy change rather than by supporting sustainable school improvement
• The approach also hides the differences between unlike schools, many of which are critical. And many of the root causes of these differences are beyond the capacity of the school to redress (in the short term): material resources, facilities, social capital and levels of crime, domestic violence, mental illness....

Most parents are not interested in comparisons of like schools at all. They do want to know how good their school is because they simply want a good education for their children. For example, when parents move their children from one school to another they don't search for like schools... they simply look for a good or better school in a manageable location.

To publish actual differences between all schools and the related contextual factors is almost certainly too problematic for any government and/or school system. The distribution of (dis)advantage and the difficulties of addressing the inherent injustices represent an (as yet) unmanageable challenge and a political powder keg.

That is, publishing league tables is a simplistic, low level approach to school improvement. Structuring the tables around 'like schools' is a deceptive and issue-avoiding approach that is likely to be ineffective and counter-productive. And this is a pity when there are a number of complexity-based approaches that offer genuine possibilities for rapid and sustainable gains through making sense of what is currently happening and identifying realistic opportunities for improvement.

Risking experts

Based on a current Tasmania initiative to improve student literacy rates it seems that experts generally propose a four part strategy::

1. Assess the performance of the subjects (in this case, students)
2. Respond by developing and applying resources (usually based on programs with which the expert has been deemed successful)
3. Provide training for staff (in order to ensure compliance)
4. Prove that the strategy works by running pilot studies (the easy part)
   

But as Myron Tribus says, "There is a simple answer to every question, and it is usually wrong". For example, such proposals generally involve serious 'misinformation', especially for the clients. Common misinformation includes

• The expert's recommendations are innovative and somehow 'visionary' (rare in education)
  
• The recommended practices are not currently in use (they usually are to a greater ot lesser extent)
  
• The strategy will solve the problem (education is a complex endeavour and does not have simple solutions)
  
• If it fails the blame should go to those who failed to implement the strategy properly (however, managers are responsible for the effectiveness of the system).

When staff accept this misinformation as valid, there are likely to be serious OH&S issues around stress and and mental health.

Perhaps as a result of the 'feel-good' vibes of the strategy, those charged with stewardship of the system generally fail to give careful consideration to the possible dangers involved, including

• grossly misrepresenting the current practices
• repeating previous practices in the hope of achieving different outcomes
• alienating those responsible for achieving the intended outcomes
• damaging the credibility of those responsible for achieving the intended outcomes
• making it harder for parties (in this case, schools, families and communities) to work together
• grossly over estimating the transferability of 'best practices'

Most pilot studies are usually quite successful, not so much because they demonstrate the veracity of a strategy but because of the Hawthorne effect.

The intentions may be right, the 'experts' may make their very best contributions. However, such strategies are fundamentally flawed if they are not based on a sound understanding of the current reality, the history of the local situation, and initiatives that nurture the emergence of more effective and sustainable local actions and arrangements.