Drone Warfare data explorer

Drone Warfare data explorer

Organisation: The Bureau of Investigative Journalism (United Kingdom)

Publication Date: 04/07/2017

Size of team/newsroom:small


On March 17, the Bureau’s drone warfare project launched an open-source tool to transform how our extensive data on US air and drone strikes in Yemen, Pakistan, Somalia and Afghanistan was presented and how it could be accessed and used. The tool presents the latest figures on strikes and casualty numbers as well as allowing users to create bespoke, sharable reports. The Bureau of Investigative Journalism has been developing comprehensive datasets on US strikes and other actions in Pakistan, Yemen and Somalia since late 2010 and Afghanistan since 2015, collecting data on the number and location of attacks and details of those killed. We have reviewed tens of thousands of news reports, NGO reports, tweets and Facebook posts, we have trawled Wikileaks, we have carried out investigations in the field and we have finagled secret documents out of government sources in order to come up with best possible estimates on casualty numbers. We present our findings for each country in timelines comprised of narrative accounts of each reported action, a casualty count of the number of individuals reported killed or wounded in that action, and all sources used to gather the information. Secondly, we produce spreadsheets that are available online and can be downloaded. The creation of these datasets was an attempt to encourage public debate on US counter-terrorism operations and push for government transparency and accountability. We have become well-known worldwide as a primary reliable resource for data on the covert drone war. But as more and more data had been collated over the years, its storage and presentation had become seriously unwieldy, inconsistent and a challenge to navigate. Addressing this was a top priority of a major website redesign we launched last February. As well as making our narrative timelines consistent and easy to find, we introduced an interactive tool on the drone warfare project page. The tool’s interface presents key figures from across all four countries, automatically updating when additional information is added into the datasets. Its interactive feature allows users to explore and visualise the part/s of the data they are interested in. For instance, a user could create a report showing how many strikes hit Pakistan between 2010 and 2014, and how many civilians were killed; or a report showing how many people were killed in strikes in Yemen from January 2016 to now. The tool imports the relevant data from the datasets and creates a series of visualisations from it. Where the location of a strike is available, it is geocoded and presented on a map. Readers can then republish this information freely. The code used to make the tool is also available for anyone to use on the Bureau’s Github repository.

What makes this project innovative? What was its impact?

Our original goal in starting our data collection was to open the covert operations to public scrutiny. For years the US refused to even acknowledge that lethal drone attacks were being carried out. Today, there is still very little official recognition of individual strikes and the US rarely provides overall estimates of drone casualties. We have sought fill this gap and improve transparency and accountability in the process. Journalists, activists, members of civil society, and officials from governments and international organisations use our data in their reports and inquiries. The tool makes it exponentially easier for them to do this, making reams of data simple to interrogate and analyse. The tool can demonstrate patterns in strikes and civilian casualty figures with the click of a mouse, showing the user for example, that the peak of the CIA drone campaign in Pakistan was in 2010, that strikes in Yemen have drastically increased this year, or compare how many civilians died in two different countries over a particular time period. Prior to this, the user would have to access our spreadsheets, which contain long lists of individual strikes and confusing cell descriptions. The tool’s interface also clearly signposts where all the spreadsheets and narrative timelines can be found; on the old website, finding them often involved complicated and confusing journeys. The wider drone data collection has had additional major impact. A quarter of all traffic to the Bureau’s website goes to drone-related pages, the majority going to pages related to our data. And last summer, we made major progress in our mission to improve White House transparency. During a conversation with senior officials and two other organisations which had been collecting drone data, it was made clear to us that our efforts had directly influenced a government decision to release figures on the number of counter-terrorism strikes and casualties for the first time.

Technologies used for this project:

The Bureau maintains a set of four Google spreadsheets which contain the latest historical data on drone strikes in Afghanistan, Somalia, Yemen and Pakistan. The challenge was to access and collate all the data so it can be searched and visualised from the source data spreadsheets, whilst being simple to use and understand for a visitor to the Drone Wars site. We also knew that the data would need refreshing and should be available using a simple search in a way in which other programs can query and access the collated data, i.e. an API. We therefore built a simple Ruby on Rails application, hosted on Heroku, which could access the spreadsheets using the Google Drive ruby gem and Google API. This parsed the four spreadsheets (which have slightly different formats) and stored the data in a common format using a Heroku Postgres database. After the data is imported, the geocoding then takes place and each location is automatically geo-located where possible. This geo-location is also stored in the database. An admin web page was also created as part of the application to allow a manual refresh of the data from the spreadsheets and with a headline summary of the imported data. Once the data was imported, it can then be exposed via a simple JSON API, allowing queries on country, dates and type of data required. Currently the API is customised for the visualisations, collating the data to make the actual visualisation more straightforward. The final part of the application is the explorer, so that a visitor to the site can explore the database and see the visualisations of the data, as charts and maps. The charts use the Chart.js JavaScript library, whereas the maps use Leaflet.js and the OpenStreetMap map data.


Follow this project

Comments (0)

You have to be connected to contribute

You have to be connected to follow

Leave this project and no longer be informed about this project

By joining this project, you will be informed by email when an update or a new contribution is posted on the website.

Thank you for your active participation !

The GEN Community Team