What do you think? Should government departments embrace some of the principles of the open-source world in order to liberate public sector information?

God no, unless we can implement restrictions that allow verification. Otherwise, go for it!

So a truly resilient and dynamic data.gov.au would hold -
- policies (like the above) with a bias towards ‘always open meta data’ and ‘usually open unless there are good reasons raw data’
- directory of pointers
- showcase of best of breed voted by inside and outside the government.

But it would not hold data or to the foreseeable future even meta data but as standards evolve and open perhaps it could hold meta data too.

I don’t have a problem with data.gov.au being a storage manager too.
I don’t think it’s reasonable to for every data producer, or publisher to make their data available forever, and to take on those costs.

Suppose a small workgroup if formed to do some studies, and produce a quantity of data, should we expect them to host the data, and care for the infrastructure? Better to have a process when they can ’seal’ the data and submit it to an archive, along with whatever metadata will aid discovery.

Perhaps we need to ask, how long will published data need to be available?
Then we can think about how that might be achieved (assuming the metadata and discovery are thrashed out satisfactorily).

Regards

Matt

I agree we need both the cathedral and the bazaar to get some dynamic tension. Generally Stephen, Mike, Andrae, Daniel, Mia, Tony, Hanare, Bec, Kerry, Ben are all on the right track in my mind.

We also need to take some principle based ‘10 steps/rules’ to simplify and ensure balance and break the current ‘do nothing’ nexus [department heads note expect a mandate memo from your boss soon about releasing meta data and most data] -

1. clearly state what is available ? … meta data (structure data about data to be released immediately on request) .. OR plain data (release subject to controls see below) … OR Hybrid
2. treat ’sets’ as ‘endless streams’ not as static point-in-time files like open source code (the stream of web site stats from a site does not have a version but the meta does)
3. treat sets as having endless variations on their meta, because we are always seeking further improvement, the joy of the journey so keep a meta version number and when it started/finished
4. state a clear owner for each set/stream and a clear point/s for community support [department heads note]
5. possibly distinguish between sets and streams (sets being point in time, streams being ongoing and more alive)
6. certainly hold and debate and refine all data in a decentralised way
7. index and create a central pointer directory with minimal barriers to listing or access or replication/federation of the directory and ideally the data
8. avoid picking a mandated technology, let the (federated) market decide, an initial short list may simplify and speed startup but no-exclusivity, no caps on new channels or access technologies
9. release everything unless clear legal/privacy/security reasons can be identified within (say) 14 days of a request (bear cost of materials?) and encourage minimal duplication through good access but don’t mandate no-duplication
10. prohibit charging for raw data only for value adds

So a truly resilient and dynamic data.gov.au would hold -
- policies (like the above) with a bias towards ‘always open meta data’ and ‘usually open unless there are good reasons raw data’
- directory of pointers
- showcase of best of breed voted by inside and outside the government.

But it would not hold data or to the foreseeable future even meta data but as standards evolve and open perhaps it could hold meta data too.

Thoughts?

Cheers, Pete.

Accessible data is good – accessible data with an atom or rsync feed of updates is better.

Federated search is good – but only if you have a standard query interface[1]
but not if you need to perform joins or correlated queries.

When data is open anyway, where is the security risk? The actual risk is not security, but currency – hence my mention of atom/rsync above.

So the recommendation will need to be more nuanced. If we are willing to talk actual technologies[2], closer to:

1) Structured data should be made available in an open format with a well defined mapping into RDF (using standard vocabularies and RDFS/SKOS/OWL)[3]
2) Metadata should be made available in RDF compliant with AGRkMS, DCMI; with any extensions described in RDFS; supporting thesauri in SKOS; and ontology in OWL.
3) In addition, both structured-data and metadata should be exposed as a SPARQL endpoint.
4) Unstructured data should exposed using a full-text search api. Due to its field testing, the google search api is preferred (or even better a mapping of the GS-API into SPARQL).

It’s not enough to just have access to the data – we need to know what it means (hence points 1&2). It also isn’t enough to just slap on a web-form frontend to a text-search box (as is currently done) to count as “access provided”, you need real query facilities, with standard interfaces. Otherwise the transation costs kill you.

[1] SPARQL is a good option for structured queries; Defining a mapping from the google search api to a virtual RDF graph would also allow you to use it for feature-oriented search.
[2] If we want to remain tech-neutral, then we need to expand each of these technologies into a description of the capability they provide, and specify that – but that’s a lot more work, and not something I’m going to attempt in a short post.
[3] D2RQ and similar technologies are available to map SQL databases into RDF with minimal difficulty; although for really large datasets there are going to be unaddressed scalability issues.

any repository of permanent data in data.gov.au defeats the purpose of gov2.0 – the power of the internet is that we link to sources of information – not make copies. Data should be left where it was created and access provided. This would be a good “principle” for the Task Force to suggest as it would stop the proliferation of data with the attendant security risks. Keep the source data in its place of origin and do not make copies for access.

any repository of permanent data in data.gov.au defeats the purpose of gov2.0 – the power of the internet is that we link to sources of information – not make copies. Data should be left where it was created and access provided. This would be a good “principle” for the Task Force to suggest as it would stop the proliferation of data with the attendant security risks. Keep the source data in one place.

This centralization is pretty much like a standard that everybody conforms to so that things are interoperable. (Think of all the rules coming out of the IETF, W3C, and the Linux Standards Base – central control of standards with distributed production using those standards)

But the choice of keywords and alternate terms is important. Discrepancies between jurisdictions (each state and federal gov has their own thesaurus) make consolidation difficult.

If the keywords were properly filled in, it would be trivial to ask “give me all the agriculture or water policies or media releases associated with the Grampians” or “give me all the current inquiries across all agencies and jurisdictions that have anything to do with pricing controls”.

The same thing goes for laws and general efficiency. How long did it take McClelland to go through all laws in Oz to discover all the things to do with secrecy? (Proper discoverability using keywords and something like Google advanced operators would also make it easy on lawyers to check up on topics across different councils, states, etc – and thus easier on the pockets of their clients).

(Now, if only we could get RFC822 X-headers with AGLS/AGIFT contents put into, and displayed, by common webmail servers and email clients, then the majority of documents in government could be discovered easily by people in the same or different agencies).

It’s also worth noting that the AGLS defines the authoritative source URL for the document (and there is no reason why XML datasets couldn’t include AGLS keywords at different levels of the schema, just like any other document).

Thus, there would be core raw-data producers, readily identifiable whoever the intermediary or value-adder is, using centrally defined thesauri, and preferably a core set of centrally-defined schemata.

So… maybe not “Cathedral”, but a decent sized central chapel that welcomes everybody to help define evolving “dogma”, then the “primary producers” creating the base data, then the “industry” (including cottages) that add value and sell it in the bazaar.

At the most basic level, can you even reference “Australia” as a URI in such a way that a machine can know the difference between the Australia in the physical/geographical sense and Australia in the abstract/political/legal sense?

The traditional government method of getting a “product” like an ontology would be to pay a consultant to come up with a specification, put it out a tender, pay a contractor to develop it and get a finished product. The “product” then remains largely static, assuming it even works, and any maintenance requires paying the contractor to make changes.

Suppose instead an ontology was developed like any other open source or open standard project. If it is truly open then it could be developed by volunteer experts without having to go through the mess of tenders and contracts. Open standards projects like W3C generally have much better quality control than anything proprietary and developers will put a surprising amount of their own time into making it work the way it is intended.

My suggestion: pick a non-trivial ontology that needs to be developed and let it run as an open source project.

Opengov means open ‘linked data standards’ – Tim-Berners Lee [including video from TED]: http://opengov.ideascale.com/akira/dtd/5489-4049

Open – they need to be available to the general public, without legal encumbrance. For example, “Keyword AAA” is a proprietary specification owned and licensed for fee by NSW State Records (http://www.records.nsw.gov.au/recordkeeping/keyword-products/keyword-aaa). Unfortunately it is also included by reference by the NAA and every (AFAIK) state records agency.

Accessible – they need to be available at a stable URL (or PURL) without the need to access them via intermediary links (ie. click-license, or disclaimer) preventing direct access by computers without human intervention.

Locatable – the stable URL needs to be included in the various descriptions of the dependent standards, so programs can automatically pull the definition file.

Machine Interpretable – the definition needs to be available in a standard machine interpretable format. The preferred format would be an RDF serialisation of a SKOS thesaurus. RDFa is an ideal option that can permit the file to be served simultaneously as a human-interpretable XHTML file and a machine-interpretable RDF document.