Open Data Projects that Don’t Make Their Data Open

Sounds bizarre, but it seems to be the norm.

The very people who are at the forefront of consuming Open Data, funded by my tax money, almost always fail to make the fruits of their labours available as Open Data in their turn.

Let’s see: You get some public funds to build a demonstrator that shows how great Open Data is.

So you go away and harvest all sorts of data from a variety of sources (both easy and hard), and then add lots of value by linking it together and then synthesising new values and providing new insights.

Then hopefully in some time that corresponds to the proposal timescales, you launch your wizzo site, with fancy visualisations and everything. You get the loud praise and tweets from the great and good saying how this demonstrates how great Open Data is.


I come along with a little email asking if I can please have the data (so that I can use it for and other stuff I have that is set up to use the Open Data food chain).

I then get the standard response: “Ah, yes, we intend to make the data public, but <fill in a reason here />, and give us just a little time.”

Following that, I usually wait a few weeks or months, and then ask again. I might get another response along the same lines. I will probably ask another 2 or 3 times over the course of the next year or so, sometimes getting a response, but then decide that the data is probably too old now to be of interest, and I am wasting my time in any case because it ain’t going to happen. (It is unlikely that the original project actually included a real plan for keeping the application up to date by refreshing the data, or that the technology is actually able to do that.)

The point that anyone working in Open Data knows is that if you don’t build in the mechanisms and technology from the start to make data open, then grafting it on at the end is hard, expensive, unreliable, and various other bad words. So why don’t people put it in the proposal, and why don’t the funders require it to be part of the projects?

And the worst thing is that sometimes you hear the people who have done these projects (that are essentially data sinks) actually complaining that they couldn’t get data they wanted for their project from some other source!

Now ain’t that hypocritical?!

I would actually prefer it, by the way, if the response to my request for the data was more along the lines of, well we built a demonstrator, and we aren’t going to make the data open. There are actually quite a few sensible reasons for doing that.

Right, I’m off to think about which of my datasets is Open… 🙂

Flattr this!

Misquoting Harold Wilson’s “White Heat” speech

Usually Harold Wilson’s famous 1963 speech is quoted as something like Britain needed to be forged in the white heat of technology, or technological revolution.

Both of these are wrong.

I think Wikiquote has it right:
“In all our plans for the future, we are re-defining and we are re-stating our Socialism in terms of the scientific revolution. But that revolution cannot become a reality unless we are prepared to make far-reaching changes in economic and social attitudes which permeate our whole system of society. The Britain that is going to be forged in the white heat of this revolution will be no place for restrictive practices or for outdated methods on either side of industry.”

Why rewrite science to technology? Wilson knew about technology, but he chose science. And his audience would have understood the word at that time.

A good and honest short quote for this might be “forged in the white heat of this [scientific] revolution”.

It is a powerful view, given by Wilson, that it is the scientific revolution and the science that drives the world, and enables the technology, and it is the science that should be supported, and certainly not just the technology.

We can only assume that people who misquote are being deliberately mendacious or simply ignorant.

And it is a shame in this world of technology and Web Science that we lose the import of the middle sentence: that we need to be prepared to make far-reaching changes in economic and social attitudes.

Flattr this!

LinkedIn results on Google

Where to start with LinkedIn?

Well this is specific – the Google results.

I want to find someone’s personal or work page, and I put in their name, and get the first half a dozen results of different LinkedIn pages for people with that name (or another!). And none of them give much information if you click on them – they all ask you to sign up to LinkedIn in the hope of actually getting the details you want.

I now have a shortcut:, which helps a bit – maybe I need to add to it…

Flattr this!

And another thing – online bills crappy file names

So when I finally download the file, I get something like statement_1.pdf.

I mean, what is the use of that?
Am I meant to put it in the same folder as the other statements from the same company? Because if I want to maintain my account history, I need to keep the statements, because they throw them away, as we know.

This is the worst of them all, because I really actually have to edit the filename if I am going to put it in a folder.

But even if they are helpful enough to put a date in the PDF filename, does it help me sort? No, of course not. It’s something like statement_12 Jun 14 .pdf or statement_2014-01-02.pdf, which, apart front the amazing space before the dot and extension (!) in one of them, is no help at all for sorting. Why can’t they use ISO dates, like any sensible organisation?

And notice that they haven’t put any indication of what the organisation is in the file name.

In fact, why can’t they let me set a preference of exactly how I would like the filename to look? It wouldn’t be hard, would it?!!!

Flattr this!

And why only a few months of bills?

While we are at it, why only “15 months” or “6 months” of bills?

Come on, is disc (and backup) so expensive they can’t keep the tiny bit of ASCII text or numbers in a DB for a few years? The CSV is only 5KB, and the PDF if I ask for that is 177KB. It probably cost them more to get the code written and run to delete the old data than it would cost them to keep it.

Flattr this!

Online billing emails without the PDF

So I have to log in to the site to see their pathetic little apology for a bit of paper bill.

That’s all I wanted – the bill; so why couldn’t they send it to me, some utilities do.

But no, I have to “Log in and view you bill” (sic)*, which takes me to the Landing page. Then I get the honour of clicking on their “Log In” button, which takes me to the Log in page. If I can actually now manage to go through the authentication, I might, if I am lucky, get a button that lets me finally view the actual bill, but more likely it will be a summary page that has another button with “Download Your Bill”.

At this stage I am now at the same point as if they had simply sent me the PDF in the first place.

OK, I know that one of the reasons they do this is so that they can try and sell me stuff as I go from page to page, but some of them don’t even do that!

“Quick and easy” or whatever they say when they want you to sign up? I don’t think so. It is much easier to pick up the letter they send and open the envelope and look at the bill.

* “Log in and view you bill” is what I see in my mailer for this particular company. I don’t let emails download images by default (not least because of tracking), and so I see the Alt text, where as the button is right.

Flattr this!

The Sun does go around the Earth!

This sort of thing that really pisses me off:
It is actually based on a question used to assess science literacy according to OECD and similar organisations’ criteria about the place of the earth in the universe.

But that assessment is usually done with the question:
“Does the Earth go around the Sun or does the Sun go around the Earth?”
And the second of those gets marked incorrect!

There are lots of “respected” publications that quote the “poor” answer as evidence of ignorance. Ignorant the people might be, but this is not evidence. How can you take any author seriously if they deny the patent evidence and current orthodoxy and deride others because they don’t agree?

But most importantly, how can we expect kids to learn science and scientific method (as opposed to just answering questions about the “facts”), if we ask them to deny their observations and all experiments that they can perform, just because teacher tells them it isn’t true?!!!!!

In fact the Sun going around the earth wins 2-1. It goes around the Earth every 24 hours, and it goes around the Earth every year.

Now I come to think of it, I had a very pleasant time recently with my granddaughter at sunset in Israel explaining how the sun goes down and then comes back up again, having gone round the other side of the earth. It never crossed my mind to try to explain it from the sun’s or even the earth’s frame of reference – why do that? I used her frame of reference. Of course, it is hopefully part of an ongoing conversation that will of course include the rotation of the earth, as well as a heliocentric view of the solar system, and a something-centric view of the galaxy…, but I can’t bring myself to lie to her and suggest a non-relativistic mechanics.

Flattr this!

Windows 8 – It had to come

Don’t get me wrong – I actually like Windows. There are quite a few things it does better than OSX. And I have managed machines with almost every incarnation of the OS, by the way.

And while we are at it, there are a lot of things that Androids do better than IOS.

But starting to use Windows 8 really pisses me off.

Not so long ago I bought a new Mac; as best I recall, I fired it up, it understood (sorry about the anthropomorpism) I probably had an old one I wanted to move from, offered to help me do that, and a bit later I was up and running.
Similarly, every time I have upgraded the OSX system, I could work out how to do the basic things, even if I didn’t always like the changes (see blogs passim).

But Windows update, every time, takes the feckin’ biscuit.

I switched on the new laptop. Basically, I couldn’t actually get it to do anything without googling.

Let’s try email, I thought. There’s a button there, that’s exciting! Clicking on it actually took me into a Black Hole. It now wanted a MS email ID (which the owner doesn’t have); there was in fact no way out of this – WTF? Cancel and OK both did the same thing of suggesting I try and put the MS email ID in again.

Back to google – apparently if you push the mouse to various corners, things happen – thanks for being so obvious. Well excuse me, but there really should be visual cues somewhere that things can happen. So finally I get back to the button screen.

Let’s try some of the other exciting buttons. Every single mother’s son of them does the same thing as the Mail button. The whole thing is set up so that MS can get all he user’s login data and see the whole scope of the user’s online identity, through the MS ID.

In this respect it is just like Android – everything is linked to your Google email ID. And if you don’t have one, or want to let them link up all your online IDs, then it doesn’t work very well, and in fact keeps telling you how it could be working better.

And in this respect, both of them are unlike Apple. The “rapacious, closed” Apple devices just work fine without an Apple ID, and everything that can, does work and integrate well, and apps are fine that you don’t have an Apple ID.

Oh, I forgot, before all this it offered me the chance to install a bunch of browsers – the user likes to use Firefux, so I clicked on that. It installed it, but I couldn’t for the life of me find out how to get back to install other browsers, or even run the Firefux I just installed.

So I am still looking at this brick that I can’t actually get to do anything.

After a while, I realise that the Desktop button actually gets me a view that looks bit like the old windows thing. But no, I still can’t find Firefux or Control Panels, or anything useful.

More googling.

I finally find out the magic incantation. Apparently, what you do is go to the screens with all the buttons and *start typing*. Well stab me sideways with a laptop. That was completely obvious – I should use the *keyboard* to interact with all these buttons.

I interact with my computers (other than Unix command line) using a mouse for anything except text. That is what the mouse is for – my view of having to type to launch an app on a modern system is just – I am actually at a loss for words to describe just how brain-dead that is.

Actually, this is all a bit like a Unix command line. It gives you no help as to what to type, but if you do know, then you can drive it quite efficiently.

So now, remember that I will want to migrate from the old laptop? Fortunately I am sensible enough to realise that there really must be some way of doing that. But it did take quite a lot of googling (the system’s Help was no help) to find it. It was about as easy to find it as to find the right flags to tar to perform a similar function.

Since no-one will have read this far, I can report that in the end I did manage to get the apps the user needs installed.

Oh sorry, no, it turned out the the Skype button led me to one of those Black Holes, with no way to actually login using anything other than (you guessed it) the MS email ID. More googling found me the Skype for Desktop app that I had to download and install, which worked. Of course, I did try to put in the email associated with the Skype account, but the Skype app kept simply saying that the login had failed, which made me assume that the password was wrong – any half-way moronic programmer could have checked and actually reported that the email ID it was being given was not a MS email ID.

So, as I was saying, it seems pretty good now, with the data nicely migrated, and the apps the user wants installed on the button screen, as short cuts on the desktop, and pinned to the task bar, no password on login, etc.. (On the other hand I have resisted the temptation to hack the Registry to get rid of the splash screen with all that stuff on which is only suitable for tablet machines).

And of course it is a nice machine, costs a third of what a Mac would have cost, and Windows 8 works very nicely.

Flattr this!