This site was developed to support the Government 2.0 Taskforce, which operated from June to December 2009. The Government responded to the Government 2.0 Taskforce's report on 3 May 2010. As such, comments are now closed but you are encouraged to continue the conversation at agimo.govspace.gov.au.

Project 13: Government 2.0 Governance and Institutions: Embedding the 2.0 Agenda in the Australian Public Service

Heuris Partners examined the need for organisational and governance reforms to embed the Government 2.0 agenda within the Australian Public Service and concluded that cultural rather than technological change would be the critical success factor and that this should also influence the choice of a lead agency to mange this change. It also noted that Gov 2.0 required an APS shift towards and outward focused and tailored approach to meeting the needs of the Australian people. The report concludes that while elements of the Gov 2.0 relating to information and technology can be properly allocated to special purpose agencies, effective and timely delivery of the cultural and organisational change agenda needs the power, reach and coordinating capabilities of PM&C as the lead agency.

Author Comments

I have been out of senior positions in the Australian Public Service for well over ten years but have remained close to a number of major public policy debates. The implications for Government of Web 2.0, and the emerging concept of Gov 2.0 were, however, pretty much in my peripheral vision (at best) before being asked to look at the governance implications for the Taskforce. I came to the underlying topic with a somewhat sceptical eye.

But an examination of recent overseas developments (including the UK where I spent the first half of my professional career) and invaluable input from Peter Shergold convinced me that:

  • The concepts and approaches comprising Gov 2.0 are radical and transformational in terms of much of the current modes of thinking and operation within government.
  • The technical issues associated with the shift to a Gov 2.0 model for Australia are important, as is much wider availability to publicly financed/generated information, but not sufficient for the transition;
  • The major changes required are essentially cultural and organisational.

The APS has many important strengths on which to build towards Gov 2.0. In essence, much of what is implied by the concept and the reality of delivering Gov 2.0 involves tipping the current focus of public service policy and program delivery work on its head, from the essentially upward looking hierarchical structures towards an outward focused and much more tailored approach to meeting the needs of Australian people.

At the same time, current agency boundaries and methods for allocating people and money need to be reshaped to address the critical social, environmental and economic challenges of the 21st century, none of which reflect or respect bureaucratic architecture. (I was intrigued to find that it is in the military that offer some of the most interesting examples of melding information sharing and delegated authority in dealing with high consequence situations.)

Elements of these themes are reflected in recent broader thinking about the reform of the Australian Public Service which is being driven out of the Department of Prime Minister and Cabinet (PM&C) and the Public Service Commission. Thus, while elements of the Gov 2.0 relating to information and technology can be properly allocated to special purpose agencies, effective and timely delivery of the cultural and organisational change agenda needs the power, reach and coordinating capabilities of PM&C.

Project Documents

3 Responses
  1. 2010 April 13
    Madeleine Kingston permalink

    Two recent related postings by me on Club Troppo were in response to Nicholas Gruen’s blog on that site. One in particular was worth repeating here perhaps.

    see:

    “Esprit de l’escalier: how blogs can help government agencies and public servants do their jobs better”:

    I refer to Nicholas’ response to Andrew Podgers posting of 15 March I would like to particularly pick up on the former’s comments about evaluation of self-performance.

    A good while ago I collated a number of evaluation principles gleaned from postings and writings of excerpts on the topic, some forming part of the online internationally-based American Evaluation Association (AEA) Discussion Group known as EVALUTALK.

    With full citation and attribution to the authors I incorporated many suggestions from that splendid resource, to which I subscribe, avidly reading the postings made. Other sources including from Michael Patton’s work. The suggestions were incorporated in a number of my submissions to the public arenas including MCE published on http://www.ret.gov.au under my name; the Victorian Essential Services Commission (Part 2A to their 2008 Review of regulatory Instruments) and to the Productivity Commission.

    The context was in terms of formal evaluative assessment undertaken by professionals, but nevertheless the list below does raise some general evaluative questions that it would be prudent for any organization to ask of themselves or subject themselves to for external evaluation.

    I hope Gov2 will not mind my including them here as well, since I am not sure whether Club Troppo postings are linked to Gov2.

    ***

    SOME BURNING EVALUATION PRINCIPLES CONVERTING THEORY INTO PRACTICE:

    How many of these principles were adopted in the various evaluative processes undertaken by those guiding or undertaking major or minor policy reform in various State, Commonwealth or advisory arenas?
    They may assist with general evaluative and record-keeping best practice principles for all policy, regulatory and other entities working in the public policy arena.

    Recommendations: General evaluative principles

    1. What was the evaluand {Funnell and Lenne 1989} at several levels, mega, macro and micro, since different stakeholders will have different concerns at each of these levels {Owen 1999:27}.

    2. In choosing design and methods, were any cautions used against replacing indifference about effectiveness with a dogmatic and narrow view of evidence {Ovretveit, 1998:}.

    3. What external threats were identified and considered before the data gathering exercise was undertaken?

    4. What comparisons were used?

    5. What were the boundaries and objectives?

    6. Was an evaluability assessment undertaken to more precisely determine the objectives of the intervention, the different possible ways in which the item could be evaluated and the cost and benefits of different evaluation designs (Wholly JK (1977) Evaluability assessment in L Rutman (ed.) Evaluation Research Methods: A Basic Guide, Beverly Hills, CA: Sage) and Wholly JK (1983), Evaluation and Effective Public Management, Boston: Little, Brown c/f Ovretveit Evaluating Health Interventions. Open University Press. McGraw-Hill (reprinted 2005), Ch 2 p 41)

    7. What were the implied or explicit criteria used to judge the value of the intervention?
    8. Which evaluation design was employed was employed, since a decision on this issue would impact on the data-gathering measures?

    9. Was the evaluative design in this case case-control, formative, summative, a combination of process (formative) and summative; cost-utility or audit? Will assessment of the data gathered be contracted out to an informed researcher or research team with recent professional development updates and grasp of the extraordinary complexities in the evaluative process?

    (Patton, M. Q. (2002) Qualitative Research & Evaluation Method Sage Publications).

    In addition there some excellent resources more current focused on the not-for-profit sector which I can gather together at some stage and reproduced on this site or Gov2 or both if appropriate.

    I had also gone to some lengths to hunt down and collate for an enquirer on EVALUTALK some pertinent citations on institutional assessment. Sadly the material has become embedded in an old version of my client server and I have been unable to retrieve that material.

    If I re-gain access they may also be quite useful generally in terms of the Project 13 APS Agenda.

    This is such a topical matter that I thought I could do with highlighting again.

    I should say up front in my usual frank way that I would merely be highlighting the expert views of others and not putting the theory or practical suggestions as my own.

    My motto is that if one comes by useful resources that can be put to good use there is no point hiding them under a bushel.

    I applaud Nicholas’ view as expressed in the Club Troppo blog of 5 March that I have just re-read that perhaps Gov2 discussions could be undertaken more broadly since there are so many useful considerations that may not fall neatly into its current parameters.

    Regards

    Madeleine (Kingston)

  2. 2010 April 13
    Madeleine Kingston permalink

    Sorry I short-changed everyone. My original list, gleaned by me from several sources was much longer than I thought, so here’s the rest of it and should be read in conjunction with the suggestions above.

    The material had formed part of several submissions made to public consultative arena including the AEMC, ESC (Vic) and MCE arenas published on the RET website http://www.ret.gov.au

    Note I have retained the numbering sequence from the above post which ended with Evaluative question 9 (Patton)

    10. How was the needs assessment conceptualized?

    11. Was the program design clarifiable?

    12. How was the formative evaluation undertaken?

    13. What are or were the Program Implementation process evaluation parameters?

    14. What measures will be in place for evaluating the “settled program” (or policy change proposed)?

    15. How were short term impacts by conceptualized and identified for the proposed changes?

    16. What definitive outcomes are sought and how will these outcomes be determined by follow-up?

    17. Was/will there be time to activate the evaluation’s theory of action by conceptualizing the causal linkages? Whilst not ideal, if no theory of action was formulated, perhaps it is not too late to partially form a theory of action plan.

    {Patton, M. E. “The Program’s Theory of Action” in Utilization Focussed Evaluation, Sage, Thousand Oaks, 1997, pp 215-238}

    18. Was there be room or time in the data-gathering exercise to probe deeper into the answers provided by the people whose lives will be affected by any decision the Government may make to deregulate within the energy industry?

    19. The skilled questioner knows how to enter another’s experience?

    {From Halcom’s Epistemological Parables c/f ibid Qualitative Research and Evaluation Methods, Ch 7 Qualitative Interviewing}

    20. As Eyler (1979) said What are figures worth if they do no good to men’s bodies or souls?

    {c/f Ovretveit (1997) “Evaluating Health Interventions”. Open University Press. McGraw-Hill (reprinted 2005), Ch 1}

    21. What was be done do assess the intended impacts of the studies undertaken.

    22. Before the data-gathering exercise was undertaken, and considering the time constraints were these factors considered: feasibility, predictive value; simulations; front-end; evaluability assessment?

    23. What processes will be undertaken to ensure added-value components to the evaluation?

    24. How will the agencies/entities utilize case study example in augmenting the existing relatively generic study undertaken addressing standard demographics over a large sample without sub-segmentation of more vulnerable groups (such as residential tenants or regional consumers) with more in-depth evaluation?

    25. How carefully will the agencies/entities in their parallel Review/Inquiry review in tandem program documentation, especially where there is overlap; or examine complaints and incident databases; form a linkage unit for common issues.

    26. To what extent have the following evaluative process been undertaken by both bodies, and all Commonwealth and State bodies including the MCE and COAG Teams, policy advisers and policy-makers regulators {See Centre for Health Program Evaluation, Melbourne University

    27. Does all of the government, quasi-government, regulators and others a plan by which program analysis can be undertaken formally, and by which success criteria can be measured as the desired features of the outcomes represented in the outcomes hierarchy, defining more precisely the nature of the outcomes sought and the link between the stated outcome and the performance measures for that outcome in terms of both quantity and quality?”

    {See Funnell S, Program Logic (1997): “An Adaptable Tool for Designing and Evaluating Programs” in Evaluation News and Comment, V6(1), pp 5-17}

    a) How will the success of the policy changes ultimately effected be monitored and reevaluated and how often. Specifically, will there be a second phase of evaluation as one of accountability to managers, administrators, politicians and the people of Australia?

    b) What will be the rule change policy that will be transparent and accountable not only internally but to the general public as stakeholders?

    c) Generic protections such as those afforded by current and pending trade practices and fair trading provisions are currently insufficient and not quite as accessible as is often purported.

    d) Within an industry that represents an essential service and where large numbers of vulnerable and disadvantaged consumers (not just on financial grounds) are underrepresented how will the Government ensure that the rights of specific stakeholder groups are not further compromised?

    e) How accessible will Rule Changing be?
    (my question to the AEMC – which remains topical given a new proposed Rule Change on embedded generation arrangements and outsourcing that also has relevance to a current AER Determination impacting on outsourcing and cost determination – discussed elsewhere)

    f) How will the success of the policy changes ultimately effected by monitored and re-evaluated and how often. Specifically, will there be a second phase of evaluation as one of accountability to managers, administrators, politicians and the people of Australia?

    g) In choosing design and methods, what will be done about replacing indifference about effectiveness with a dogmatic and narrow view of evidence

    {Ovretveit, 1998:}.

    What will be the rule change policy that will be transparent and accountable not only internally but to the general public as stakeholders?

    a) How accessible will Rule Changing be?

    b) Perhaps the agencies and entities would consider seeking specialist evaluation input with further evaluation of data when making major regulatory reform decisions

    c) Does Government have a plan by which program analysis can be undertaken formally, and by which success criteria can be measured as the desired features of the outcomes represented in the outcomes hierarchy, defining more precisely the nature of the outcomes sought and the link between the stated outcome and the performance measures for that outcome in terms of both quantity and quality?”
    {Funnell, S, (1997) “Program Logic: An Adaptable Tool for Designing and Evaluating Programs” in Evaluation news and Comment, V6(1), pp 5-17}

    Evaluation is a sophisticated and scientific professional challenge. It is not just a trade, though compromises often make it so. Professional evaluators are humble people. They make no pretenses. Regardless of reputation or status, they are never too humble to ask for collaborative input and peer opinion and suggestion. Evaluation is a continuing process and does not start and end with data gathering. They recognize the challenges of best practice data gathering and evaluation and do not pretend to have all the answers.
    For instance, check out the University of Alabama’s EVALUTALK facility, American Evaluation Assocviation Discussion Group

    This group is the cutting edge of evaluative practice. The rest of the world respects the results this group achieves.

    One such evaluator could be Bob Williams a highly respected NZ evaluator with an international reputation and particular expertise in public policy evaluation. He is a frequent visitor to Australia, and is a fairly well known figure in Australasian evaluation, through evaluations, his work within the Australasian Evaluation Society (AES) (which merged with Evaluation News and Comment under Bob Williams’ supervision) and his contributors to the two Internet discussions groups Evalutalk and Govteval. He has vas experience of Governmental evaluations.

    On the online Evaluator’s Forum, EVALUTALK, Bob Williams responded that evaluators should not been seen as mere technicians doing what they are asked to do, but should be seen as craftspeople with a pride in their work and the outcomes of their findings long after the consultative process is over.

    Williams’ specialty is evaluation, strategy development, facilitating large-group processes and systemic organizational change projects. He has his own website under his name.

    Reviews books for Journal Management Learning, writes for Australasian Evaluation Society’s Journal. He wrote the entries on “systems” “systems thinking” “quality” and planning Encyclopaedia of Evaluation {Sage 2008) and co-written with Patricia Rogers in “Handbook of Evaluation” {Sage 2006}.

    There is a great deal of valuable consultative evaluation advice out there for the asking. Lay policymakers are not normally trained in this area.

    Bob Williams, has commented as follows on EVALUTALK:

    “The Ministry of Education here in New Zealand has been doing something very interesting for the past four or five years. The policymakers along with teachers university researchers and others have been developing a series of “best evidence syntheses”.

    The concept of “best evidence” is fairly comprehensive with a set of agreed criteria for what constitutes “best” and “evidence”. As each synthesis is developed it is opened up for discussion with practitioners and academics – and placed on the Ministry of Education’s website. I was involved in some of the early discussions (as a facilitator rather than evaluator) and was impressed by both the method and the content of the syntheses.

    What I found most impressive was that the policymakers were brave include evidence that challenged some of the assumptions that have dominated education policymaking in the past few decades (e. g. the extent to which socio-economic status effects student performance).”

    “The 2006 edition of the World Education Yearbook describes the BES Programme as the most comprehensive approach to evidence” and goes on to say: “What is distinctive about the New Zealand approach is its willingness to consider all forms of research evidence regardless of methodological paradigms and ideological rectitude and its concern in finding…effective appropriate e and locally powerful examples of ‘what works.”

    Bob Williams suggests that before data gathering is undertaken the underlying assumptions must be made, followed by identification of the environment and environmental factors that will affect the way in which the intervention and its underlying assumptions will interact and thus behave.

    A recent dialogue between evaluators on that Discussion List produced a useful list of criteria that would cover the processes that should ideally be undertaken.

    Though the inputs came from a number of Discussion List members, I cite below how Bob Williams a respected New Zealand evaluator with an international reputation summarized as follows inputs from various evaluators participating on the Discussion List.

    http://www.eval.org

    Position the evaluation – that is, locate the evaluation effectively in its context, in the broader systems.

    Bob Williams, Discussion List Member Evalutalk

    Wow that was a long post – just as well I split it up.

    Hope this is taken ion the spirit intended.

    More will follow when I get a chance assuming it is helpful material for this Project.

    Regards

    Madeleine (Kingston)

  3. 2010 April 14
    Madeleine Kingston permalink

    Hi Nicholas (Nicholas Gruen Chair Gov2 and others)

    The article on The Australian of 13 April 2010 concerning the state of heath policy prompted me to complete this hatrick on evaluation, though other references are also pertinent when I get to it. Hope this is not too long to post but it really belongs with the other two postings to complete the picture. Hope it is of some use in a very practical sense.

    I need to acknowledge that the evaluative theory models belong to others as meticulously cited. My role was to put the material together in some sort of sequence with the citations. Each author cited is responsible for the ideas.

    My last blog tome against Project 13 finished off with advice from advice from Bob Williams a New Zealand evaluator.

    I continue the thread with further discussion of that advice, quoting directly from William’s advice on positioning, interrupting with one or two observational interjections from me, and then back to the discussion between Bob Williams and Stanley Capella.

    From Bob Williams, NS Evaluator:

    “Position the evaluation – that is, locate the evaluation effectively in its context, in the broader systems.

    1. Clarify the purpose and possibilities, etc (design phase – why do it)

    2. Plan the evaluation (design phase) (what do we want to know)

    3. Data Gathering (how will we find out what we want to know)

    4. Making meaning from the data (e.g. analysis; synthesis; interpretation (how can we get people to be interested in the evaluation processes/results
    5. Using the results (shaping practice) (what would we like to see happen as a result of the evaluation and what methods promote that?)”

    MK Comment:

    This is impossible to achieve without a comprehensive informed SWOT analysis that goes well beyond background reading of other components of the internal energy market –a highly specialized exercise, especially in an immature market. Prior to undertaking the survey mentioned to ascertain market awareness, what steps were taken to mount a strengths and weakness analysis (SWOT).

    If undertaken, where can the results be located? This type of exercise is normally undertaken prior to the gathering of data so that the survey data is meaningful, is robust to address a range of relevant factors; and not simply narrowly focused on data-gathering that may yield compromised results if the goals and parameters that could have been initially identified in a SWOT analysis were not clearly identified and addressed in the study design.
    Stanley Capella on the University of Alabama Online Evaluation Discussion Group EVALUTALK has questioned whether evaluators should push for program decisions based on evaluation, or whether this an advocate’s role.
    Bob Williams responded that evaluators should not been seen as mere technicians doing what they are asked to do, but should be seen as craftspeople with a pride in their work and the outcomes of their findings.

    As suggested by Ovretveit (1997) in Evaluating Health Interventions. Open University Press. McGraw-Hill (reprinted 2005), Ch 6

    “Design is always balancing trade-off.” “Inexperienced evaluators are sometimes too quick to decide design before working through purposes, questions and perspectives.”

    “Ideas which are fundamental to many types of evaluation are the operational measure of outcome, the hypothesis about what produces the outcome, an open mind about all the (factors) that might affect the outcome and the idea of control of the intervention and variable factors other than the intervention.”

    “Randomized experimental designs are possible for only a portion of the sittings in which social scientists make measurements and seek interpretable comparisons. There is not a staggering number of opportunities for its use.

    {Webb et all 1966 c/f Ovretveit Evaluating Health Interventions, “Evaluation Purpose Theory and Perspectives” Ch 2, p31}.

    “Politicians often do not examine in detail the cost and consequences of proposed new policies, or of current policies.” Ibid, Ovretveit Ch 2, p 27
    In discussing better informed political decisions Ovretreit noted, for example, the lack of prospective evaluation or of even small scale testing of internal market reforms in Sweden, Finland and the UK. Whilst he did not infer that all new policies should be evaluated or that the results of an evaluation should be the only basis on which politicians decide whether to start, expand or discontinue health policies, just that politicians could sometimes save public money or put it to better use if they made more use of evaluation and of the “evaluation attitude.” ibid, Ch 2, p27

    In Ch 3 (p73 Ovretreit embraces six evaluation design types:

    Descriptive (type 1);

    Audit (type 2)

    Outcome (type 3); comparative (type 4);

    Randomized controlled experimental (type 5) and

    Intervention to a service (type 6)

    Each of these six broad designs can and have been successfully used in a variety of interventions targeted at examining policies and organizational interventions, depending on which of the four evaluation perspectives have been selected: quasi-experimental; economic; developmental or managerial.
    In recent years there has been increasing pressure on all scientists to communicate their work more widely and in more accessible ways. For evaluators, communication is not just a question of improving the public image of evaluation, but an integral part of their role and one of the phases of an evaluation. It is one of the things they are paid to do. Here we consider evaluators’ responsibility for communicating their findings and the different ways in which they can do so.

    Daniel L Shufflebaum’s Program Evaluations Metaevaluation Checklist is worth looking at.
    {Shufflebeam, D. L. (1999) “Program Evaluations Metaevaluation Checklist”, based on The Program Evaluation Standards (University of Michigan)}
    Michael Scriven’s Key Evaluation Checklist is a useful resource. Scriven’s Checklist poses some challenging questions that are touched on here in good spirit {see Key

    a) Can you use control or comparison groups to determine causation of supposed effects/outcomes?
    b) If there is to be a control group, can you randomly allocate subjects to it? How will you control differential attrition, cross-group contamination, and other threats to internal validity.

    c) If you can’t control these, what’s the decision-rule for aborting the study? Can you single or double-blind the study.

    d) If a sample is to be used, how will it be selected; and if stratified, how stratified?

    e) If none of these apply, how will you determine causation (the effects of the evaluand)

    f) If judges are to be involved, what reliability and bias controls will you need (for credibility as well as validity)?

    g) How will you search for side effects and side impacts, an essential element in almost all evaluations?
    h) Identify, as soon as possible, other investigative procedures for which you’ll need expertise, time, and staff in this evaluation, plus reporting techniques and their justification

    i) Is a literature review warranted to brush up on these techniques?

    j) Texts such as Schiffman and Kanuk’s Consumer Behaviour may provide some useful insights during the evaluative process.

    {Schiffman, Leon G and Kanuk, Leslie Lazar Consumer Behaviour. (1994) Prentice-Hall International Editions}

    As previously mentioned, The University of Alabama’s EVALUTALK site has a host of useful insights about evaluation design. As discussed by Fred Nichols o Distance Consulting, Recent discussions are focused on Roger Kaufman’s mega-planning model, based on his notion of needs assessment.

    “Logic models can be described as frameworks for thinking about (including evaluating a program in terms of its impact Stakeholders processes inputs etc. Typically these run from inputs through activities/processes to outputs/products outcomes/results and impact including beneficiaries.”
    {Fred Nichols, Senior Consultant, Distance Consulting on EVALUTALK, American Evaluation Association Discussion List [EVALTALK@BAMA.UA.EDU];

    In response to Fred Nichols comments, Sharon Stone on the same EVALUTALK, comments on the assumptions that include program theory and external conditions (meaning factors not included that could affect positively or negatively the hypothesized chain of outputs, outcomes.

    Stone poses two questions:

    “Are these just “logical chains” – or are these cause the effect”

    Either way – are things really that simple – or do we need to pay more attention to those ‘external’ factors” – and how they are identified as external Patton (1980)248 has estimated over a hundred approaches to evaluation. He describes four major framework perspectives – the experimental, the economic, the developmental and the managerial.
    {Sharon Stone, Evaluator, on EVALUTALK, University of Alabama September 2007}

    See Patton (1980) “Qualitative Evaluation Methods”, London Sage, c/f Evaluation Purpose and Theory in

    Evaluating Health Interventions

    Patton claims:

    “One reason why evaluation can be confusing is that there are so many types of evaluation. Case- control, formative, summative, process, impact, outcome, cost utility, audit evaluations.”

    {See Patton, M. Q. (1997) Utilisation Focused Evaluation. The new Century text 3rd edn.}

    Funnel (1996) has some views on Australian practices in performance measurement. Her 1996 article in the Evaluation Journal of Australasia provides broad-brush review of the state of evaluation for management in the public service.

    Funnell provides explanations of jargon such as benchmarking, TQM, quality assurance and she also explores issues relating to the current political climate of progressive cutbacks and how these have affected the use of process evaluation. The form of process evaluation she is examining is seen as ‘managerial accountability p452).

    As well Funnell explores the impact of cutbacks on the conduct of evaluations, the levels of evaluation expertise available and on evaluation independence and rigor. Her arguments on the impact of market-based policies imply there could be both benefits and dangers.
    {Funnell S (1996): “Reflections on Australian practices in performance measurement.” 1980-1995. Evaluation Journal of Australasia 8(1), 36-48}

    Funnell S 1996. Reflections on Australian practices in performance measurement 1980–1995. Evaluation Journal of Australasia 8:36-48.

    {See also Eggleton IRC 1990 (revised 1994). Using performance indicators to manage performance in the public sector. Decision Consulting & New Zealand Society of Accountants: 1-124. c/f Australian Institute of Health and Welfare (AIHW) 2000. Integrating indicators: theory and practice in the disability services field. AIHW cat. no. DIS 17. Canberra: AIHW. (Disability Series)}; Particularly Appendix: Participation data elements from the draft National Community Services Data Dictionary.

    Hawe Degeling and Hall (1990) have some ideas of survey methods and questionnaire design.

    {Hawe, P., Degeling D., & Hall, J (1990) Evaluating Health Promotion, Ch 7 Survey Methods and Questionnaire Design, Sydney, McLennan & Petty}

    These authors describe random, systematic, convenience and snowballing sampling and look at questionnaire layout and presentation; the need for piloting and some simpler basic description analysis of quantitative and qualitative data. Fore more sophisticated analysis such as may be warranted before any decision is made by the Government to deregulate in the energy industry may warrant the employment of a highly trained researcher, recently trained.

    These authors examine a) the types of items; (b) questionnaire layout and presentation; (c) the need for piloting (this is often overlooked by evaluators undertaking small-scale evaluations; d) maximizing response rates.

    Note their comments on the analysis of quantitative and qualitative data. These comments describe simple, basic descriptive analysis. For more sophisticated analysis evaluators should employ a trained researcher.

    Funnel (1997) has discussed program logic as a tool for designing and evaluating programs. This is simply a theory about the causal linkages amongst the various components of a program, its resources and activities, its outputs, its short-term impacts and long-term outcomes. It is a testable theory, and must be made explicit as a first step to testing its validity.
    The process by which this is achieved is program analysis. This is a job for an expert in evaluation where major government policy is being reexamined.
    {Funnel S (1997) “Program Logic: An adaptable tool for designing and Evaluating Programs” in Evaluation News and Comment v.6(1) 1997 pp 5-17. Sue Funnell is Director of Performance Improvement Pty Ltd and chair of the AES Awards Committee.}

    As Funnel points out, the many models of program theory
    …. “date back to the 1970s and include amongst others Bennett’s hierarchy of evidence for program evaluation within the context of agricultural extension programs and evaluability assessment techniques developed by Wholey and others.”

    A typical program logic matrix may include a grid that includes ultimate and intermediate outcomes, and immediate impacts, with success criteria being measurable and specific in accordance with the SMART principles.

    {Ibid Funnel Program Logic, p5}

    One theme in the responses (TO EVALUTALK) as summarized by Johnny Morrell), is that

    “…..logic models can be seen as constructions that can be used to test key elements of a program’s functioning.

    Related to 1.1 is the notion that logic models can be seen in terms of path models in analytical terms.
    To me, this gets at the notion that while there is a useful distinction between “design” and “logic model”, the distinction is a bit fuzzy. Presumably, if one had enough data, on enough elements of a logic model, one could consider the logic model as a path model that could be tested.

    From a practical point of view, I still see logic models as guides for interpretation, and design as the logic in which we embed data to know if an observed difference is really a difference. But the distinction is not clean.

    Related to 1.1 is the notion that logic models can be seen in terms of path models in analytical terms. To me, this gets at the notion that while there is a useful distinction between “design” and “logic model”, the distinction is a bit fuzzy.

    Presumably, if one had enough data, on enough elements of a logic model, one could consider the logic model as a path model that could be tested.

    {American Evaluation Association Discussion List [EVALTALK@BAMA.UA.EDU] as summarized by Johnny Morrell, PhD, Senior Policy Analyst, Member American Evaluation Association EVALUTALK Discussion Group}

    From a practical point of view, I still see logic models as guides for interpretation, and design as the logic in which we embed data to know if an observed difference is really a difference.

    But the distinction is not any given logic model is never anything more than a work in progress that has to be updated on a regular basis. With this approach, logic models (and the evaluation plans they drive), can be updated as the consequences of program action evolve.
    {Johnny Morrell on EVALUTALK, American Evaluation Association}.

    The major point in this category is that “design” means a lot more than a logic for looking at data. According to this view, “design” includes procedures for gathering data, schedules for doing evaluation tasks, and so on Johnny Morrell calls this:

    “an evaluation plan and reserve the term ‘design’ for the logical structure of knowing if observations have meaning.”

    {Ibid Johnny Morrell}

    There is a consensus amongst EVALUTALK members that:
    This task is typically undertaken by independent evaluators and can be a stand-alone evaluation if the only questions addressed focus on operational implementation, service delivery and other matters. This form of evaluation is often carried out in conjunction with an impact evaluation to determine what services the program provides to complement findings about what impact those services have.

    One example of a combined process and summation evaluation is shown in the study reported by Waller, A. E et al (1993)

    {Waller, A. E, Clarke, J. A., Langley, J. D. (1993). An Evaluation of a Program to Reduce Home Hot Water Temperatures. Australian Journal of Public Health (17(2), 116-23.}

    In that study, the summative component was inbuilt into the original program design. The findings were inclusive and relatively useless primarily because of flaws in conceptual assumptions made. However there were lessons to be learned in designing other similar studies, so the pilot study was not entirely wasted.
    Rossi examines outputs and outcomes as distinct components of an evaluative program, with the former referring to products or services delivered to program participants (which can be substituted for end-consumers) and with outcomes relating to the results of those program activities (or policy changes).

    Program monitoring can be integrated into a program’s routine information collection and reporting, when it is referred to as MIS, or management information system. In such a system data relating to program process and service utilization is obtained, compiled and periodically summarized for review.

    The University of Alabama’s EVALUTALK site has a host of useful insights about evaluation design. As discussed by Fred Nichols of Distance Consulting, Recent discussions are focused on Roger Kaufman’s mega-planning model, based on his notion of needs assessment.
    Patton (1980) has estimated over a hundred approaches to evaluation. He describes four major framework perspectives – the experimental, the economic, the developmental and the managerial.

    {Patton (1980) Evaluation Purpose and Theory}

    Patton claims:

    “One reason why evaluation can be confusing is that there are so many types of evaluation. Case control, formative, summative, process, impact, outcome, cost utility, audit evaluations.”

    {See Patton, M. Q. (1997) “Utilisation Focused Evaluation.” The New Century Text 3rd edn.}
    “the use of logic models (may be seen as) a consensus building tool. The notion is that logic models come from collaborative cross- functional input from various evaluator and stakeholder groups. Thus, the act of building a logic model works toward common vision and agreed upon expectations.”

    Swedish evaluator John Ovretreit (1987, reprinted 2005)257 has written a classic text on evaluative intervention. Though focused on health interventions, the principles are as relevant to other areas.
    Ovretreit (1997) Evaluating Health Interventions. Open University Press. McGraw-Hill (reprinted 2005)
    Rossi’s’ evaluation theory is about whether the intentions of the program were effected by delivery to the targeted recipients.

    {Rossi, P., Freeman and Lipsey, M. (1995) “Monitoring Program Process and performance: Evaluation: A Systematic Approach” (6th edition) Sage, pp 191-232}
    Funnel (1996) has some views on Australian practices in performance measurement. His 1996 article in the Evaluation Journal of Australasia provides broad-brush review of the state of evaluation for management in the public service.

    Funnell S (1996): “Reflections on Australian practices in performance measurement”, 1980-1995. Evaluation Journal of Australasia 8(1), 36-48

    Funnell provides explanations of jargon such as benchmarking, TQM, quality assurance and she also explores issues relating to the current political climate of progressive cutbacks and how these have affected the use of process evaluation. The form of process evaluation she is examining is seen as ‘managerial accountability p452)’.

    Swedish evaluator John Ovretreit (1987, reprinted 2005)263 has written a classic text on evaluative intervention. Though focused on health interventions, the principles are as relevant to other areas.
    Of quality assurance Davey and Dissinger said
    “Quality assurance (QA) and evaluation are complementary functions which collect data for the purpose of decision- making. At the process level, quality assurances provides both a system of management and also a framework for consistent ser4vice delivery with supporting administrative procedure. When implemented appropriately QA methods provide rapid feedback on services and client satisfaction, and a means to continuously upgrade organizational performance.

    Despite client feedback being part of QA, it lacks the depth provided by evaluation in determining individual client outcomes from a person centered plan for service delivery.”

    {Davey, R. V. and Dissinger, M (1999) “Quality Assurance and Evaluation: essential complementary roles in the performance monitoring of human service organisations.” Paper presented at Australasian Evaluation Society Conference, Melbourne 1999, p 534-550}

    In April 2008 Bill Fear as a regular online contribution to EVALUTALK, the American Evaluation Association Discussion Group April EVALUTALK, the American Evaluation Association Discussion Group on the topic of self-efficacy. His insights are topical so I quote them below:

    Why do policy makers make such bad policy most of the time? Why is good policy so badly implemented most of the time? Why don’t policy makers listen to honest evaluations and act on the findings? And so on.

    Could we actually bring about meaningful changes by giving people the tools to think things through and act accordingly? Does empowerment actually mean anything? (Well, yes, but it seems to lack substance as a term in its own right.)

    Does anybody ask these questions? Or is everybody just concerned with the latest methodology which will always be historic not least because it can only be applied to the past (there is an argument there).

    I digress. The point is, to my mind at least, the importance of self-efficacy in the field of evaluation has been overlooked at our expense.”

    {Note: Bill Fear, BA (Education) MSc (Social Science Research Methods), PhD (Cognitive Psychology). Member UK Evaluation Society. He sits on the UKES council, and the American Evaluation Association.

    He has excellent research and evaluation experience, as well as solid grounding in PRINCE project management. He has attended top level training programs in the US with both Michael Scriven and Michael Patton. Recent experience include working for the Office for National Statistics where he led a large index rebasing project, and helped set up the development of both a banking and insurance index for the corporate sector. He is currently running the Investing in Change project (a Wales Funders Forum project). This project is using an evaluation framework to explore funding of the voluntary sector from a funders perspective.

    A recent achievement in this includes building a partnership with the Directory of Social Change to deliver a Funding Guide for Wales. He presents workshops on the emerging findings of this project to a wide range of policy makers. He is frequently asked to comment on evaluation methodology and proposals.}
    As discussed in my 2007 submission to the AEMC’s Retail Competition Review, The Companion Wallis Consulting Retailer and Consumer Surveys identified fairly well matched perceptions according to the summary comparative findings. Awareness levels amongst consumers besides knowing of the ability to choose, as clearly extremely low.

    MK Comment:
    On the subject of energy is a low engagement commodity/service, active marketing is necessary with product differentiation and attractive offers including a range of convenience options or discount packages.

    MK Comment:

    Evaluation and analysis factors impacting on market failure. Interpretations that switching conduct is predictive of real outcomes in an unstable market are yet to be substantiated.

    Much discussion on the Productivity Commission site and in responses to AEMC and other consultative processes has focused on behavioural economics and the value of superficial evaluation of switching conduct. I will not repeat those arguments here, save to say that the data relied upon does not appear to robustly embrace these principles.

    Again I explain that my role was to source collate and present the above material. All the reasoning goes to the credit of the professional evaluators who took part in the EVALUTALK discussions highlight, or were authors of books and papers that I have personally read and felt would be useful to highlight these as matters of topical interest, more so as health policy and governance is high on the agenda for reform.

    Prepared and collated by Madeleine Kingston

    Cheers

    Madeleine

Comments are closed.