- The Oslo Records - The Oslo Records

Organisation: Fædrelandsvennen - (Norway)

Publication Date: 03/28/2017

Size of team/newsroom:large

Description provides a tool that helps you make use of your right to access public documents. The tool is for free, and it is quick to use and easy to understand. In stead of manual browsing through tons of paper, you can now find the documents in a searchable database, and file your FOIA-request in seconds. We systematically harvest metadata on public documents, and structure these data in a database which we combine with FOIA-system that is perhaps the fastest and most user-friendly on the planet. Since a FOIA-request in Norway is for free and you do not have to provide ID when filing one, we could narrow the whole process down to this: Search our database, fill your basket, FOIA-with-one-click. We started in it a very local service, since we are a small regional newspaper. It grew. Since we started in 2015, we have combined 98 public archives in Norway in the tool, and we keep growing. In October 2016, we launched out biggest project so far: The Oslo Records. The Association of Editors in Oslo asked us to help with a tool for finding document references and filing FOIA-requests in the capital of Norway. An honor for a small district newspaper, that we are. The problem is that Oslo has no common archive plan, and their 45 different archives are allowed to choose their own software. To do this, we had to change most of the technology we run at and rethink our user-interface. This was far more complex then what we had done before. The Oslo records alone combines 45 different archives and more than 4 million of our document references. Some archives did not want to share data at all. We had to use the law as a tool, and file complaints in order to get the data we wanted to work with. The result is an application that makes it easier to work with written sources in investigative journalism. But it is open also to other groups to use, free of charge. Journalists are our main audience, but we do have users from NGOs and industry as well, and even some politicians. We have made the archives available for all the 8 public universities, all the 27 police districts, most of the divisions of the Norwegian armed forces, to name a few. We have started with a big project to do the same with the public health corporations as well. (The site also contains a blog about how I use Right To Information as a journalistic tool. The blog is only in Norwegian for now, but we written the search tool itself with both a Norwegian and an English user interface.) (The featured image shows me, journalist Tarjei Leer-Salvesen, with paper equivalent to one year of the public records, the metadata on public documents, in our hometown Kristiansand, Norway. Realizing that a news journalists have no time to read that pile of paper was the moment when was born. Material like this must be read by computers) I have been in charge of the project. Other team members include three programmers: Kenneth Lykkås, Atle Brandt and Frode Nordbø.

What makes this project innovative? What was its impact?

More than 100 countries have different versions of Freedom of Information Act/Right to Public Information-legislation. But all of them struggle with the implementation of the laws. Finding your way in the information jungle, is difficult. Filing a FOIA-request is complicated. Few people actually do it. The Oslo Records by has made Oslo municipality more transparent. The mayor of Oslo thanked us for the project: More importantly, our journalist colleagues in Oslo have gotten the tool they need to do a better job with written sources. After a few months, I hear some of my colleagues there have filed a great number of FOIA-requests through the app. I know it is now also being used by a lot of users also outside the journalism-profession. Out website is now two years old, and have contributed to more transparency and better tools for journalists in a number of other instances as well. We made searchable records for the parliament in Norway (Stortinget): And we have done the same for all the public universities, all the police districts and for the armed forces as well. Our site is updated with new document entries every week. By the end of March 2017, we count more than 7,8 million document entries in our database. And this is when we keep the Church out of the count, since the Church still has it's own search-platform in out system due to special challenges with their software. (Users will find our Church-search less userfriendly, and we are working on getting them onto the platform we introduced with The Oslo Records soon)

Technologies used for this project:

We start a new source by filing a FOIA request for years of historic public records. Then, we subscribe to some datasources by email and treat those with an Outlook-macro (Visual Basic) as they reach us. It varies how the different archives make their records available, and what software they have used to generate them. But most data is harvested on a daily basis through scraping web pages where this is possible. Both tasks are automated and will copy documents with the data to a Dropbox App folder, while adding a unique lettercode to the filename to reflect the source of the document. From Dropbox, the files are uploaded to DocumentCloud for OCR treatment. Projects in DocumentCloud are reflected as archives in our database, and each archive can have several sources. From here, we import all data from the documents into out Postgres SQL database. A unique parser assigned to each archive source will create a structured representation of the document that are then inserted into the database. The site itself is in Wordpress. Summary: DocumentCloud, Node.js-app, Postgres SQL, Visual Basic, Javascript, Wordpress.
Follow this project

Comments (0)

You have to be connected to contribute

You have to be connected to follow

Leave this project and no longer be informed about this project

By joining this project, you will be informed by email when an update or a new contribution is posted on the website.

Thank you for your active participation !

The GEN Community Team