| 0 |
Open Library provides dumps of all its data, generated every month. All of the dumps are formatted as tab separated files with the following columns: |
0 |
Open Library provides dumps of all its data, generated every month. Most of the data dumps are formatted as tab separated files with the following columns: |
| 1 |
|
1 |
|
| 2 |
* `type` - type of record (/type/edition, /type/work etc.) |
2 |
* `type` - type of record (/type/edition, /type/work etc.) |
| 3 |
* `key` - unique key of the record. (/books/OL1M etc.) |
3 |
* `key` - unique key of the record (/books/OL1M etc.) |
| 4 |
* `revision` - revision number of the record |
4 |
* `revision` - revision number of the record |
| ... |
|
... |
|
| 16 |
* [reading log dump (~ 65M)](https://openlibrary.org/data/ol_dump_reading-log_latest.txt.gz): with columns "Work Key, Edition Key (optional), Shelf, Date" |
16 |
* [reading log dump (~ 65M)](https://openlibrary.org/data/ol_dump_reading-log_latest.txt.gz): with columns "Work Key, Edition Key (optional), Shelf, Date" |
| 17 |
* redirects dump ~ 50M |
17 |
* [redirects dump (~ 50M)](https://openlibrary.org/data/ol_dump_redirects_latest.txt.gz) |
| 18 |
* deletes dump ~ ?M |
18 |
* [deletes dump (~ 75M)](https://openlibrary.org/data/ol_dump_deletes_latest.txt.gz) |
| 19 |
* other dump ~ ?M |
19 |
* [lists dump (~ 30M)](https://openlibrary.org/data/ol_dump_lists_latest.txt.gz) |
|
|
20 |
* [other dump (~ 10M)](https://openlibrary.org/data/ol_dump_other_latest.txt.gz) |
|
|
21 |
* [covers metadata dump (~ 70M)](https://openlibrary.org/data/ol_dump_covers_metadata_latest.txt.gz): with columns "id, width, height, created" |
|
|
22 |
* [wikidata dump (~ 700M)](https://openlibrary.org/data/ol_dump_wikidata_latest.txt.gz) (**NEW**): Wikidata records relevant to Open Library -- currently only authors. Columns "Wikidata ID, JSON" |
| 20 |
|
20 |
|
| ... |
|
... |
|
| 34 |
|
34 |
|
| 35 |
Please see this great guide by contributor on the `LibrariesHacked` github about how to load Open Library's data dumps into postgres to make it more useful and queriable: |
38 |
This guide by contributor on the `LibrariesHacked` GitHub about how to load Open Library's data dumps into PostgreSQL to make it more easily queriable: |
| 36 |
https://github.com/LibrariesHacked/openlibrary-search |
39 |
https://github.com/LibrariesHacked/openlibrary-search |
|
|
40 |
|
|
|
41 |
### DuckDB |
|
|
42 |
|
|
|
43 |
[DuckDB](https://duckdb.org/) is another easy tool to query the dump without much work. |
|
|
44 |
|
|
|
45 |
For example: |
|
|
46 |
If you wanted to get all the Wikidata IDs currently in the authors table: |
|
|
47 |
``` |
|
|
48 |
SELECT json_extract(column4, '$.remote_ids.wikidata') as wikidata_id |
|
|
49 |
FROM read_csv('ol_dump_authors_2024-07-31.txt.gz') |
|
|
50 |
WHERE wikidata_id IS NOT NULL |
|
|
51 |
LIMIT 100; |
|
|
52 |
``` |
| 37 |
|
53 |
|
| 38 |
## GraphQL |
54 |
## GraphQL |
| 39 |
|
55 |
|
| 40 |
DiFronzo on github has produced a graphql proxy to search books using work, edition and ISBN with Open Library API. Deployed with Deno and GraphQL: |
56 |
DiFronzo on GitHub has produced a GraphQL proxy to search books using work, edition and ISBN with the Open Library API. Deployed with Deno and GraphQL: |
| 41 |
|
41 |
|
| ... |
|
... |
|
| 46 |
|
46 |
|
| 47 |
We do not yet have rolling monthly dumps of our book covers, despite a shared desire for its existence. Some historical cover dumps may be explored here: |
63 |
We do not yet have rolling monthly dumps of our book covers, despite a shared desire for their existence. Some historical cover dumps may be explored here: |
| 48 |
https://archive.org/details/ol_data?query=identifier%3A*covers* |
64 |
https://archive.org/details/ol_data?tab=collection&query=identifier%3A*covers*&sort=-addeddate |
|
|
65 |
|
|
|
66 |
Most covers are archived in the following items. Note `covers_0006` and `covers_0007` are presently unavailable. |
|
|
67 |
|
|
|
68 |
- https://archive.org/details/covers_0000 |
|
|
69 |
- https://archive.org/details/covers_0001 |
|
|
70 |
- https://archive.org/details/covers_0002 |
|
|
71 |
- https://archive.org/details/covers_0003 |
|
|
72 |
- https://archive.org/details/covers_0004 |
|
|
73 |
- https://archive.org/details/covers_0005 |
|
|
74 |
- https://archive.org/details/covers_0008 |
|
|
75 |
- https://archive.org/details/covers_0009 |
|
|
76 |
- https://archive.org/details/covers_0010 |
|
|
77 |
- https://archive.org/details/covers_0011 |
|
|
78 |
- https://archive.org/details/covers_0012 |
|
|
79 |
- https://archive.org/details/covers_0013 |
|
|
80 |
- https://archive.org/details/covers_0014 |
|
