Posts

Showing posts with the label Parse

2022: Week 12 - Gender Pay Gap

Image
 Challenge by: Jenny Martin I spent my International Women's Day being really fascinated by the Gender Pay Gap Bot on Twitter. I decided to look at the publicly available data that it was based on and found it a little confusing. It made me realise that the thing I enjoyed most about the Pay Gap Bot, was that it made it the insight from the data clear in a succinct manner. So let's do that for the historical data too!  Inputs We're using the data currently available on the Gender Pay Gap Service from 2017 to 2022:  There are 5 input files. Requirements Input the data Combine the files Keep only relevant fields Extract the Report years from the file paths Create a Year field based on the the first year in the Report name Some companies have changed names over the years. For each EmployerId, find the most recent report they submitted and apply this EmployerName across all reports they've submitted Create a Pay Gap field to explain the pay gap in plain English You may en...
Read more

2022: Week 10 - The Bechdel Test

Image
Challenge by: Jenny Martin In celebration of International Women's Day (even though it was yesterday), I thought it would be fun to take a look at which movies pass the Bechdel Test. If you're unfamiliar, there are 3 criteria to passing the Bechdel Test: The film has to have at least two [named] women in it, who talk to each other, about something besides a man The results may surprise you! Inputs We are taking our data from this website for this challenge. It has been webscraped to a certain point and we will continue parsing out the data. If you'd like to start the webscraping from the beginning as part of the challenge in your tool of choice then please feel free!  As part of cleaning up this data, we will have to deal with various html codes for different symbols.  Requirements Input the data Parse out the data in the Download Data field so that we have one field containing the Movie title and one field containing information about whether of not the movie passes the B...
Read more

2020: Week 46

Image
 Challenge by Tom Prowse. At Prep Air, we have decided to do some research into the risks of running an airline. We want to complete some analysis on some historic aviation incident reports so we can try to identify potential areas where we can make our airline safer. We have taken a selection of reports from the AeroInside  website, who document various incident reports from around the world. Each report contains information about the incident, but is a free text field so doesn't really have a structure. In this challenge, we want to parse out the key information from the string, and then see how many incidents occur that are related to our key categories.  Inputs Incident List Category List Requirements Input the Data Parse out the following information from the incident string:  Aircraft - eg, American B738 Location - Amsterdam Date - Apr 21st 2016 Incident Description - details about the incident Convert date field from string to a date Combine similar incident t...
Read more

2020: Week 40

Image
Challenge by: Jenny Martin I often see dashboards and wonder about the data prep behind them. Sometimes the most beautiful of dashboards can be hiding the most horrendous of data preparation. Let's take this Viz of the Day from dataschooler Matthew Armstrong . The visualisation itself is fairly simple, but how did the data start off?  Explore Matthew's viz here Inputs There are three inputs this week: The poems, scarped from everypoet.com The Scrabble scores for each letter (Optional) Scaffolding list Requirements Input the data Lines of the poem will not contain any HTML, css or js e.g. <head>, e9=new Object() etc. Filter out any rows which are not lines of the poem Wordsworth is very original, so there shouldn't be any duplicate lines in our data set. Filter out any repeated rows The first line of each poem is also the title of the poem. Ensure this is the case and number the lines of each poem Split the data out so there is a line for each word and assign a word ...
Read more