| 0 |
Open Library provides dumps of all its data, generated every month. All of the dumps are formatted as tab separated files with the following columns: |
0 |
Open Library provides dumps of all its data, generated every month. Most of the data dumps are formatted as tab separated files with the following columns: |
| 1 |
|
1 |
|
| 2 |
* `type` - type of record (/type/edition, /type/work etc.) |
2 |
* `type` - type of record (/type/edition, /type/work etc.) |
| 3 |
* `key` - unique key of the record. (/books/OL1M etc.) |
3 |
* `key` - unique key of the record (/books/OL1M etc.) |
| 4 |
* `revision` - revision number of the record |
4 |
* `revision` - revision number of the record |
| 5 |
* `last_modified` - last modified timestamp |
5 |
* `last_modified` - last modified timestamp |
| 6 |
* `JSON` - the complete record in JSON format |
6 |
* `JSON` - the complete record in JSON format |
| 7 |
|
7 |
|
| 8 |
Dumps: |
8 |
## Dumps |
| 9 |
|
9 |
|
| 10 |
* [editions dump (~ 8.8G)](https://openlibrary.org/data/ol_dump_editions_latest.txt.gz) |
10 |
* [editions dump (~ 9.2G)](https://openlibrary.org/data/ol_dump_editions_latest.txt.gz) |
| 11 |
* [works dump (~ 2.8G)](https://openlibrary.org/data/ol_dump_works_latest.txt.gz) |
11 |
* [works dump (~ 2.9G)](https://openlibrary.org/data/ol_dump_works_latest.txt.gz) |
| 12 |
* [authors dump (~ 0.5G)](https://openlibrary.org/data/ol_dump_authors_latest.txt.gz) |
12 |
* [authors dump (~ 0.5G)](https://openlibrary.org/data/ol_dump_authors_latest.txt.gz) |
| 13 |
* [all types dump (~ 12.4G)](https://openlibrary.org/data/ol_dump_latest.txt.gz): includes editions, works, authors, redirects, etc. |
13 |
* [all types dump (~ 12.4G)](https://openlibrary.org/data/ol_dump_latest.txt.gz): includes editions, works, authors, redirects, etc. |
| 14 |
* [complete dump (~ 29.6G)](https://openlibrary.org/data/ol_cdump_latest.txt.gz): also includes past revisions of all the records in Open Library |
14 |
* [complete dump (~ 29.6G)](https://openlibrary.org/data/ol_cdump_latest.txt.gz): also includes past revisions of all the records in Open Library |
| 15 |
|
|
|
| 16 |
* [ratings dump (~ 5M)](https://openlibrary.org/data/ol_dump_ratings_latest.txt.gz): with columns: "Work Key, Edition Key (optional), Rating, Date" |
15 |
* [ratings dump (~ 5M)](https://openlibrary.org/data/ol_dump_ratings_latest.txt.gz): with columns: "Work Key, Edition Key (optional), Rating, Date" |
| 17 |
* [reading log dump (~ 30M)](https://openlibrary.org/data/ol_dump_reading-log_latest.txt.gz): with columns "Work Key, Edition Key (optional), Shelf, Date" |
16 |
* [reading log dump (~ 65M)](https://openlibrary.org/data/ol_dump_reading-log_latest.txt.gz): with columns "Work Key, Edition Key (optional), Shelf, Date" |
|
|
17 |
* [redirects dump (~ 50M)](https://openlibrary.org/data/ol_dump_redirects_latest.txt.gz) |
|
|
18 |
* [deletes dump (~ 75M)](https://openlibrary.org/data/ol_dump_deletes_latest.txt.gz) |
|
|
19 |
* [lists dump (~ 30M)](https://openlibrary.org/data/ol_dump_lists_latest.txt.gz) |
|
|
20 |
* [other dump (~ 10M)](https://openlibrary.org/data/ol_dump_other_latest.txt.gz) |
|
|
21 |
* [covers metadata dump (~ 70M)](https://openlibrary.org/data/ol_dump_covers_metadata_latest.txt.gz): with columns "id, width, height, created" |
|
|
22 |
* [wikidata dump (~ 700M)](https://openlibrary.org/data/ol_dump_wikidata_latest.txt.gz) (**NEW**): Wikidata records relevant to Open Library -- currently only authors. Columns "Wikidata ID, JSON" |
| 18 |
|
23 |
|
| 19 |
For past dumps, see: https://archive.org/details/ol_exports?sort=-publicdate |
24 |
For past dumps, see: https://archive.org/details/ol_exports?sort=-publicdate |
|
|
25 |
|
|
|
26 |
Downloading the dumps take too long? Checkout the link above and download via torrent for higher speeds! |
| 20 |
|
20 |
|
| ... |
|
... |
|
| 30 |
|
30 |
|
| 31 |
Please see this great guide by contributor on the `LibrariesHacked` github about how to load Open Library's data dumps into postgres to make it more useful and queriable: |
38 |
This guide by contributor on the `LibrariesHacked` GitHub about how to load Open Library's data dumps into PostgreSQL to make it more easily queriable: |
| 32 |
https://github.com/LibrariesHacked/openlibrary-search |
39 |
https://github.com/LibrariesHacked/openlibrary-search |
|
|
40 |
|
|
|
41 |
### DuckDB |
|
|
42 |
|
|
|
43 |
[DuckDB](https://duckdb.org/) is another easy tool to query the dump without much work. |
|
|
44 |
|
|
|
45 |
For example: |
|
|
46 |
If you wanted to get all the Wikidata IDs currently in the authors table: |
|
|
47 |
``` |
|
|
48 |
SELECT json_extract(column4, '$.remote_ids.wikidata') as wikidata_id |
|
|
49 |
FROM read_csv('ol_dump_authors_2024-07-31.txt.gz') |
|
|
50 |
WHERE wikidata_id IS NOT NULL |
|
|
51 |
LIMIT 100; |
|
|
52 |
``` |
| 33 |
|
53 |
|
| 34 |
## GraphQL |
54 |
## GraphQL |
| 35 |
|
55 |
|
| 36 |
DiFronzo on github has produced a graphql proxy to search books using work, edition and ISBN with Open Library API. Deployed with Deno and GraphQL: |
56 |
DiFronzo on GitHub has produced a GraphQL proxy to search books using work, edition and ISBN with the Open Library API. Deployed with Deno and GraphQL: |
| 37 |
|
37 |
|
| ... |
|
... |
|
| 42 |
|
42 |
|
| 43 |
We do not yet have rolling monthly dumps of our book covers, despite a shared desire for its existence. Some historical cover dumps may be explored here: |
63 |
We do not yet have rolling monthly dumps of our book covers, despite a shared desire for their existence. Some historical cover dumps may be explored here: |
| 44 |
https://archive.org/details/ol_data?query=identifier%3A*covers* |
64 |
https://archive.org/details/ol_data?tab=collection&query=identifier%3A*covers*&sort=-addeddate |
|
|
65 |
|
|
|
66 |
Most covers are archived in the following items. Note `covers_0006` and `covers_0007` are presently unavailable. |
|
|
67 |
|
|
|
68 |
- https://archive.org/details/covers_0000 |
|
|
69 |
- https://archive.org/details/covers_0001 |
|
|
70 |
- https://archive.org/details/covers_0002 |
|
|
71 |
- https://archive.org/details/covers_0003 |
|
|
72 |
- https://archive.org/details/covers_0004 |
|
|
73 |
- https://archive.org/details/covers_0005 |
|
|
74 |
- https://archive.org/details/covers_0008 |
|
|
75 |
- https://archive.org/details/covers_0009 |
|
|
76 |
- https://archive.org/details/covers_0010 |
|
|
77 |
- https://archive.org/details/covers_0011 |
|
|
78 |
- https://archive.org/details/covers_0012 |
|
|
79 |
- https://archive.org/details/covers_0013 |
|
|
80 |
- https://archive.org/details/covers_0014 |
|
