Tuesday, 11 August 2015

Strapped

Hull is a low-lying city, as are some of the towns and villages that surround it. I have blogged before about the serious flooding in the area in 2007, just before I started mapping in OSM. What a lot of people won't know, even locals, is that flooding is a regular occurrence in some parts of the area. This flooding is on a much smaller scale than happened in 2007, but it still causes misery for anyone whose house is flooded. The Cottingham Flood Action Group have tried to understand what causes this regular flooding and then campaign to fix the causes. I have helped a bit by providing maps and overlaying data from various sources. Some of these sources are not open so I don't want to publish them here.

One source of contention is riparian ownership. This can apply when a water course runs through your land and you may then be required to maintain it. For example a ditch carrying surface water away may need to be cleaned out so water can actually flow in the ditch and not back up causing a flood elsewhere.

The East Riding of Yorkshire council is strapped for cash. They are pushing people to maintain ditches and in some cases ditches in culverts to prevent flooding, claiming these people are riparian owners of the ditches. To be a riparian owner the watercourse must be wholly with the property or form the boundary of the property. If the watercourse forms a boundary, then the property owner owns the watercourse to the mid line.

I have matched the Land Registry Inspire dataset to my surveys of a few of these and it clearly shows that the ditch is many metres outside of the privately owned land and well within the area of the public highway. This means that no matter how strapped the council is for cash, they must maintain the ditch or culvert. Of course, my survey probably isn't enough to convince the council, but it certainly should prompt people to get more evidence.

Don't believe your council. If they claim you are responsible for something unusual, make them prove it and make sure you get enough facts to stand up to them. The council may be right, but they be just trying to get you to pay for something they are responsible for.

Thursday, 25 June 2015

Lidar

Today I received two DVDs with Lidar data on them. They cover Hull and some of the surrounding area data showing centimetre height records at 50cm spacing. It shows DTM, so no buildings or trees, though DSM is available too.

I've just finished making a replacement window for the garage - the old one was completely rotten - so I fitted it this afternoon while the weather was fine. Now that's done I can turn to this detailed Lidar data.

Saturday, 6 June 2015

When does a postcode start?

I have mapped a fairly large area, Hull and most of East Yorkshire. When there was nothing on the map I just added what I found but now there is a fairly complete road network. Keeping that up to date can be hard work - how do you know what has been added since the last time I looked? OS Locator is really valuable for this as new roads appear on there as OS maps them. It would be better to know developments are starting and get in early and I think there may be a way: postcodes.

New postcodes are allocated all the time. Around a thousand are created every month in GB. The Office of National Statistics publish a postcode list based on Codepoint Open data but with some extra stuff in there. One thing is the month the postcode was set up. Looking at the recent ones may let us find places that need surveying.

I've created a map to visualise these: http://pcage.raggedred.net

I hope this proves useful.

Wednesday, 3 June 2015

New postcode layer

I've just finished extracting the data from a full download of OS Open Names and setting up a layer to see the new postcodes. The postcode data is only a centroid for each postcode area.

I maintain a layer of postcodes from the Office of National Statistics, one from Codepoint Open and now Open Names. The Codepoint Open and ONS postcodes are in the same locations, though Codepoint Open data is a bit more up-to-date right now. The OS Open Names postcode locations are now sub-metre, so I expected that the centroid might be in a slightly different location, but some are substantially different as you can see here.
 The Codepoint Open are in red, the OS Open Names are in magenta.

I don't store any of these tiles, I render them on the fly. Real tiles are often stored in a hierarchy of folders in the form of
tilename-base/zoom/xposition/yposition.png
Since I don't store any of these, when someone requests a tile they would normally get a "Not found (404)" error. I trap these errors and use the folder hierarchy to extract the postcode centroids from a database for the zoom, x and y area of the tile.  This is used to render a tile with a transparent background and centroid markers on it. I do this because I didn't want to use the considerable disk space the real tiles would take up, especially as I create to to zoom level 21 to make editing easier with the layer switched on.

I also extracted the road name and place name data, reprojected all of the location data from OS grid references to lon and lat and stored them for later use. If anyone wants to see this I'll happily pass it on, just ask.

If you want to see the new postcodes you can see the new layer on oscompare. Make sure you open the layer selection (blue+white +) and select what you want to see. You need to zoom in to see the postcodes.

Tuesday, 2 June 2015

Twenty by twenty

It seems that there are not missing sections in the OS Open Names downloads from OS OpenData. The sections are 20x20 km squares.

Monday, 1 June 2015

OS Open Names

As I noted in the previous post Ordnance Survey have said that OS Locator open data is being withdrawn and being replaced by OS Open Names. The file is a strange mixture of three types of data jumbled up. So I needed to separate them into something useful. The three types are postcode centroids, place names and road names. All of the locations are in the projection that Ordnance Survey use, OSGB36 or EPSG:27700. The great thing about the values used in this projection is that they are measured in metres from a fixed point. The previous OS open data, such as OS Locator or CodePoint Open (postcode centroids) use this projection too of course which fixes a location to a square metre. OS Open Names however has a decimal component so the location is now specified to the nearest millimetre - more accurate than OSM works to. The east-west specification in OS parlance is know as eastings and the north-south specification is known as northings. Eastings and northings can be converted to longitude and latitude for use in OSM using, for example, GDAL libraries.

The postcode centroids are very simple records. It needs the postcode and the easting and northing for the location of the centroid. Comparing a few OS Open Names centroids with Codepoint Open records the centroids are in slightly different locations.

OS Locator lists named roads with its name, a centroid and a bounding box for the road. There is also a hierarchy of place, borough, county or unitary authority.  All of this is in OS Open Names and the increased millimetre accuracy is there too.

I've not used the gazetteer of place names, but the name and location data are available as you would expect.

All of the data types have some URIs in the files too for many of the fields. Many that I have tried to open are dead links, but some show the hierarchy of data. I'm not sure why this is useful.

I think I'm going to write a routine to unzip and process all of the OS Open Names data. I'll load the data into a database, reproject it for OSM and make the processed data available if anyone wants it.

First, however, OS need to supply the missing sections of their open data.

OS Locator is to be withdrawn

I just received an email from Ordnance Survey, telling me that OS Locator, part of the OS Open Data, is to be withdrawn in a year's time. They say that OS Open Names has been published and that replaces OS Locator. They hint that it may replace CodePoint Open too, though there's no notice of withdrawal for that yet. I thought I should look at OS Open Names to see what it offers.

The first thing I saw was that the data is broken into the OS large-scale grid squares, which break the country into a 13x7 grid with 55 squares with actual data in them. Each one of these needs to be downloaded separately to cover GB. That is a pain to start with. I requested SE and TA which are both needed to cover the village I live in. After downloading them the .zip file contains each area broken down into 100 separate sections as I would expect, but I quickly realised that there were only 25 in SE and 10 in TA. Much of TA covers the North Sea, so there should be fewer sections, but clearly many were still missing. I downloaded another area and again the same 75 sections were missing. I emailed customer service at OS to point this out, with no answer yet.

I pressed on to look at the data in the sections that were there. The data is in a CSV format (there was another choice). There is a summary of column headings, but the data is a strange muddle of three types of data. Much of the data is a url to an OS website with the data summarised on a page for each column of each line of the text. It appears that there is a record type for place names, a record type for postcode centroids and a record type for named roads all freely mixed up throughout the file. There doesn't seem to be a field to simply identify what the record type is for each line. The first field is an id field. For place names it seems to start with osgb and has a number after that, for postcodes a postcode type id, with spaces removed, is in the id field and for road names an id that looks like a GUID is in the field. It doesn't seem to be a pattern for why the data is in the order that it is.

I'm going to throw some code together to disentangle these record types and see what useful data is then available. Hopefully we will not have lost anything useful in this process, though it does look as though processing this open data is going to be a bit harder than it used to be.

Anyone would think OS don't want to release open data.

Edit: There are two fields which distinguishes these different record types.