You are here: OGC Public Wiki>Ideas4OGC Web>TopicsTable>TopicSelfAssessment (revision 2)EditAttach

Topic: Self Assessment

The OGC should underake periodic self assessment reviews of its work towards its mission, integrating a mix of textual analysis by OGC Officers and numeric metrics, to produce reports describing the health of the organization and the effectiveness of its work in serving its goals.

Most effective organizations undertake, as part of their work cycle, periodic reviews of themselves. This often leads to a report detailing the ongoing successes and difficulties of the organization, including summaries of the financial health of the organization, of the composition of the organization, and the effectiveness of its efforts. Such self-evaluations and annual reports are part of the yearly cycle of all publically traded companies, and most serious non-profits. The goal is as much to identify weaknesses and past failures as to identify successes since the former can be addressed and the latter expanded.

The OGC, periodically, should produce an honest, self-critical report following a review of its work towards its updated mission. Part of the work cycle of any modern organization usually involves periodic self reviews and the creation of an annual report; the OGC should be no different. If the work to define the mission of the OGC has been well done, this would involve comparing the ongoing work to the mission statement. Automated metrics could be used to identify problem areas.

Perhaps this self-evaluation work already has been happening as part of the planning committee work, in which case this apparent oversight is actually a problem of communication. However, the issues with the lack of a coherent, updated mission suggest that pieces are missing for an effective self-evaluation.

Periodic Review

A Periodic review of the work of the OGC would attempt to evaluate the functioning of the organizations to identify problem areas and opportunities.

Several aspects of the organization could be considered:
  • health of participation including membership levels, diversity and turnover,
  • work production, including meetings organized, communication levels, standards revised, retired or adopted,
  • staff work, including issues and improvements to help them complete their work,

Review Metrics

Several aspects of self-evaluation can be partially captured with measurements, possibly automatically generated. The number of participants in OGC Meetings (and year to year changes in that number) could reveal issues with the misison to provide a forum for discussion, as might the level of use of OGC web tools and mailing lists. Changes in the length of time it takes to write a new specification could reveal increased workload or difficulty or process obstacles.

The automation of these metrics involves the tradeoff between the time it takes to automate the system and the time it would take to develop the metric manually.

Relations to other topics

  • Related to TopicUpdateVision since the review can only be done by evaluating the effectiveness of the work done against the explicit mission of the organization.
  • Related to TopicOAB which proposes that the OAB might itself undertake a periodic reviews of its own work.
  • Related to TopicImproveCommunication since the annual report would give the OGC Board of Directors and the OGC Offiers a chance to present the health of the organization, its finances, and its activities.

Possible Actions

Organize periodic reviews

The OGC Board of Directors and OGC Officers could establish a process through which the OGC and all its sub-committees would periodically evaluate their own work.

Benefits This would enable the OGC to tackle issues like those raised in this Ideas for OGC process before they became too serious. It would improve communication between different parts of the OGC and let membership know what the different committess are doing.

Drawbacks This involves an additional workload. The first few such reviews would be inefficient and therefore lengthy.

Priority

Resources This should be part of the ongoing work of the OGC and would involve mostly staff time and possibly work by one or more members of the OGC Board of Directors. It might also require feedback from the membership.

Complexity This would not be easy and would require inventing a process for the review of the activity of the whole organization.

Develop metrics

Several aspects of the work of the OGC are ammenable to measurement. As such measurements accumulate trends or changes can be identified. Examples of such metrics could be:
  • forum for discussion
    • Participants: events, activity (in person, wiki, teleconf)
    • Local fora: level of activity
    • TC: # participants, # new participants, #presentations, votes, ...
    • DWG: # presentations, level of activity, new arrivals, departures
    • Outreach: number of comments in RFC, unsolicited, other
  • common language
    • State of abstract spec: Last revision, citation, use in stds,
  • standards
    • numbers of new, revision, numbers without revision
    • duration of SWGs, revisions
    • uptake, use, test rates of Discovery, View, Access, Processing services
    • time to implement (survey), time to debug after implementation
  • testing
    • number of test suites: new, delay since las revision, unused
    • rates used for testing, new instances declared to OGCi
    • raters of certification, level of deployments of certified software
  • community
    • feedback on use of OGC standards
    • sampling of sentiment
  • institution
    • Budget: income versus expenditure
    • Membership: level, activity, incoming, leaving, type
    • Staff: satisfaction (survey?)

Benefits These would provide a way to monitor the work of the OGC which, after the first few years, can show changes from the 'normal' which might flag areas to assess.

Drawbacks It might take several years before some metrics are of much use since the first years would serve primarily to record the variability of a metric. Unless the collection of these metrics were automated, they might take time every year to generate.

Priority The priority is low since this mostly serves to improve the yearly review and the annual report.

Resources This would involve OGC Staff time to develop the metrics and possibly the automation scripts.

Complexity The proposed metrics are generally fairly simple.

Monitor published standards

The OGC could put in place a process to assess if any standards are languishing.

The OGC should periodically review all its standards to assess the health of each standard in terms of revision activity, addressing change requests, usage and other metrics. Where appropriate, the OGC should figure out what help it could bring to standards that are languishing.

ISO seems to have a rule that standards must be renewed periodically. The OGC has no such rule. There are many standards that seem to have fallen into the cracks. Where such standards are central to others, there perhaps ought to be an effort to sustain them.

Benefits

Drawbacks

Priority

Resources

Complexity

Publish an annual report

The OGC Staff could develop a process through which the periodic reviews and the performance metrics are integrated into an Annual Report which might be a text document or might be a section of the web site.

Benefits

Drawbacks

Priority

Resources

Complexity

Discussion


 

Consensus

The Leadership group has not yet discussed this topic.
Edit | Attach | Print version | History: r4 < r3 < r2 < r1 | Backlinks | View wiki text | Edit WikiText | More topic actions...
Topic revision: r2 - 08 Aug 2013, AdrianCuster
 

This site is powered by FoswikiThe information you supply is used for OGC purposes only. We will never pass your contact details to any third party without your prior consent.
If you enter content here you are agreeing to the OGC privacy policy.

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding OGC Public Wiki? Send feedback