About this Database

Home   |   OTD data   |

This is a work-in-progress on a database collection of the wonderful On-This-Day science tweets series by Robert McNees.

A natural question might be: Why a database app, why not just do Twitter queries for this data? Twitter queries are of course a good direct option, but many people would like to see the data collected in chronological format, as is presented here. That's not really possible with Twitter queries. You can restrict the data to date ranges, but still the results will not be sorted in date order. So the idea with this database is to simply provide another option for viewing the data.

There is yet another option: a web page that displays this data in chronological order (mm-dd format) here. That can be useful too, but has limited searching capabilities compared to a database app.

Datasette is the open-source tool used to build the database and web app and publish the data. It was created by Simon Willison (github repo, @simonw on twitter). Many thanks to Simon for making this excellent tool available - it's a great match for this project and also especially for projects utilizing SQLite database files.

Usage Tips

The data was gathered from Twitter's search feature using parameters for mcnees OTD content by date range, e.g.: twitter.com/search?q=from%3Amcnees%20since%3A2020-12-01%20until%3A2020-12-31%20OTD&src=typed_query

I've noticed that sometimes a link returned by a search is a dead link, even though twitter returned it from their own query. When I see instances of that, I search for an active link to include in this app. However, there's so much data, I'm sure there will be some dead links in the above. When that happens, you could still find an active link by using the twitter search with keywords, e.g., twitter.com/search?q=from:mcnees%20OTD%20uhlenbeck

Q & A

Q. What is the 'rowid' shown in the column filter drop-down input above the Apply button? How might that be useful?

A. The rowid column is the primary key index for the OTD table. It has the same numeric value as the Link column in the list and thus is not shown on the page as it would be redundant. But filtering by rowid is potentially useful. Let's say you want to filter the data to show just a few specific data rows that you couldn't otherwise get via a different filter request. That can be done with rowid filter criteria: column => 'rowid', filter operator => 'in', text box => 10, 15, 35 (a comma-separated list of rowids). That will return just the rows with those rowid values.

Version dated 2021-02-01