imap: add option to cache headers

Add option to cache headers for imap accounts. Cache db is located at
$XDG_CACHE_DIR/aerc/{account name}. The cache is cleaned of stale
entries when aerc is first opened.

Two new account level configuration options are introduced:
* cache-headers (Default: false)
* cache-max-age (Default: 30 days (720 hours))

The change in worker/imap/open.go is to set the selected directory. This
is required to access the UIDVALIDITY field, which is used in
combination with the message ID to form the key for use in the cache db.
The key structure is: "header.{UIDVALIDITY}.{UID}"

Where reasonable, cache does not stop aerc from running. In general, if
there is an error in the cache, aerc should continue working as usual.
Errors are either displayed to the user or logged.

All messages are stored without flags, and when retrieved have the flags
set to SEEN. This is to prevent UI flashes. A new method to
FetchMessageFlags is introduced to update flags of cached headers. This
is done asynchronously, and the user will see their messages appear and
then any flags updated. The message will initially show as SEEN, but
will update to unread. I considered updating the cache with the
last-known flag state, however it seems prudent to spare the R/W cycle
and assume that - eventually - all messages will end up read, and if it
isn't the update will occur rather quickly.

Note that leveldb puts a lock on the database, preventing multiple
instances of aerc from accessing the cache at the same time.

Much of this work is based on previous efforts by Vladimír Magyar.

Implements: https://todo.sr.ht/~rjarry/aerc/2
Thanks: Vladimír Magyar <vladimir@mgyar.me>
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Tested-by: inwit <inwit@sindominio.net>
Reviewed-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
This commit is contained in:
Tim Culverhouse 2022-06-15 07:23:51 -05:00 committed by Robin Jarry
parent e1d8bc4d17
commit 7aa71d334b
9 changed files with 279 additions and 15 deletions

View File

@ -111,6 +111,19 @@ available:
Default: no folders Default: no folders
*cache-headers*
If set to true, headers will be cached. The cached headers will be stored
in $XDG_CACHE_HOME/aerc, which defaults to ~/.cache/aerc.
Default: false
*cache-max-age*
Defines the maximum age of cached files. Note: the longest unit of time
cache-max-age can be specified in is hours. Set to 0 to disable cleaning
the cache
Default: 720h (30 days)
# SEE ALSO # SEE ALSO
*aerc*(1) *aerc-config*(5) *aerc*(1) *aerc-config*(5)

7
go.mod
View File

@ -5,7 +5,7 @@ go 1.13
require ( require (
git.sr.ht/~sircmpwn/getopt v1.0.0 git.sr.ht/~sircmpwn/getopt v1.0.0
github.com/ProtonMail/go-crypto v0.0.0-20211221144345-a4f6767435ab github.com/ProtonMail/go-crypto v0.0.0-20211221144345-a4f6767435ab
github.com/arran4/golang-ical v0.0.0-20220517104411-fd89fefb0182 // indirect github.com/arran4/golang-ical v0.0.0-20220517104411-fd89fefb0182
github.com/creack/pty v1.1.17 github.com/creack/pty v1.1.17
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964 github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
github.com/ddevault/go-libvterm v0.0.0-20190526194226-b7d861da3810 github.com/ddevault/go-libvterm v0.0.0-20190526194226-b7d861da3810
@ -18,7 +18,7 @@ require (
github.com/emersion/go-sasl v0.0.0-20211008083017-0b9dcfb154ac github.com/emersion/go-sasl v0.0.0-20211008083017-0b9dcfb154ac
github.com/emersion/go-smtp v0.15.0 github.com/emersion/go-smtp v0.15.0
github.com/fsnotify/fsnotify v1.5.1 github.com/fsnotify/fsnotify v1.5.1
github.com/gatherstars-com/jwz v1.3.0 // indirect github.com/gatherstars-com/jwz v1.3.0
github.com/gdamore/tcell/v2 v2.4.0 github.com/gdamore/tcell/v2 v2.4.0
github.com/go-ini/ini v1.63.2 github.com/go-ini/ini v1.63.2
github.com/golang/protobuf v1.5.2 // indirect github.com/golang/protobuf v1.5.2 // indirect
@ -35,7 +35,8 @@ require (
github.com/pkg/errors v0.9.1 github.com/pkg/errors v0.9.1
github.com/riywo/loginshell v0.0.0-20200815045211-7d26008be1ab github.com/riywo/loginshell v0.0.0-20200815045211-7d26008be1ab
github.com/stretchr/testify v1.7.1 github.com/stretchr/testify v1.7.1
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 // indirect github.com/syndtr/goleveldb v1.0.0
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778
github.com/zenhack/go.notmuch v0.0.0-20211022191430-4d57e8ad2a8b github.com/zenhack/go.notmuch v0.0.0-20211022191430-4d57e8ad2a8b
golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3 // indirect golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3 // indirect
golang.org/x/net v0.0.0-20211029224645-99673261e6eb // indirect golang.org/x/net v0.0.0-20211029224645-99673261e6eb // indirect

14
go.sum
View File

@ -91,6 +91,7 @@ github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymF
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI= github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=
github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU= github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU=
github.com/gatherstars-com/jwz v1.3.0 h1:+lVRjWDsLupLL3tJneimJ7VRBCZ6x59R2OW9zB8Wvb4= github.com/gatherstars-com/jwz v1.3.0 h1:+lVRjWDsLupLL3tJneimJ7VRBCZ6x59R2OW9zB8Wvb4=
@ -134,6 +135,8 @@ github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db h1:woRePGFeVFfLKN/pOkfl+p/TAqKOfFu+7KPlMVpok/w=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
@ -161,6 +164,7 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU= github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
@ -200,6 +204,9 @@ github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY= github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
@ -220,6 +227,8 @@ github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81P
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1 h1:5TQK59W5E3v0r2duFAb7P95B6hEeOyEnHRa8MjYSMTY= github.com/stretchr/testify v1.7.1 h1:5TQK59W5E3v0r2duFAb7P95B6hEeOyEnHRa8MjYSMTY=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 h1:QldyIu/L63oPpyvQmHgvgickp1Yw510KJOqX7H24mg8= github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 h1:QldyIu/L63oPpyvQmHgvgickp1Yw510KJOqX7H24mg8=
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778/go.mod h1:2MuV+tbUrU1zIOPMxZ5EncGwgmMJsa+9ucAQZXxsObs= github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778/go.mod h1:2MuV+tbUrU1zIOPMxZ5EncGwgmMJsa+9ucAQZXxsObs=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@ -263,6 +272,7 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@ -306,6 +316,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -484,6 +495,9 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

174
worker/imap/cache.go Normal file
View File

@ -0,0 +1,174 @@
package imap
import (
"bufio"
"bytes"
"encoding/gob"
"fmt"
"os"
"path"
"time"
"git.sr.ht/~rjarry/aerc/models"
"git.sr.ht/~rjarry/aerc/worker/types"
"github.com/emersion/go-message"
"github.com/emersion/go-message/mail"
"github.com/emersion/go-message/textproto"
"github.com/mitchellh/go-homedir"
"github.com/syndtr/goleveldb/leveldb"
)
type CachedHeader struct {
BodyStructure models.BodyStructure
Envelope models.Envelope
InternalDate time.Time
Uid uint32
Header []byte
Created time.Time
}
// initCacheDb opens (or creates) the database for the cache. One database is
// created per account
func (w *IMAPWorker) initCacheDb(acct string) {
cd, err := cacheDir()
if err != nil {
w.cache = nil
w.worker.Logger.Panicf("cache: unable to find cache directory: %v", err)
return
}
p := path.Join(cd, acct)
db, err := leveldb.OpenFile(p, nil)
if err != nil {
w.cache = nil
w.worker.Logger.Printf("cache: error opening cache db: %v", err)
return
}
w.cache = db
w.worker.Logger.Printf("cache: cache db opened: %s", p)
if w.config.cacheMaxAge.Hours() > 0 {
go w.cleanCache()
}
}
func (w *IMAPWorker) cacheHeader(mi *models.MessageInfo) {
uv := fmt.Sprintf("%d", w.selected.UidValidity)
uid := fmt.Sprintf("%d", mi.Uid)
w.worker.Logger.Printf("cache: caching header for message %s.%s", uv, uid)
hdr := bytes.NewBuffer(nil)
err := textproto.WriteHeader(hdr, mi.RFC822Headers.Header.Header)
if err != nil {
w.worker.Logger.Printf("cache: error writing header %s.%s: %v", uv, uid, err)
return
}
h := &CachedHeader{
BodyStructure: *mi.BodyStructure,
Envelope: *mi.Envelope,
InternalDate: mi.InternalDate,
Uid: mi.Uid,
Header: hdr.Bytes(),
Created: time.Now(),
}
data := bytes.NewBuffer(nil)
enc := gob.NewEncoder(data)
err = enc.Encode(h)
if err != nil {
w.worker.Logger.Printf("cache: error encoding message %s.%s: %v", uv, uid, err)
return
}
err = w.cache.Put([]byte("header."+uv+"."+uid), data.Bytes(), nil)
if err != nil {
w.worker.Logger.Printf("cache: error writing header to database for message %s.%s: %v", uv, uid, err)
return
}
}
func (w *IMAPWorker) getCachedHeaders(msg *types.FetchMessageHeaders) []uint32 {
w.worker.Logger.Println("Retrieving headers from cache")
var need, found []uint32
uv := fmt.Sprintf("%d", w.selected.UidValidity)
for _, uid := range msg.Uids {
u := fmt.Sprintf("%d", uid)
data, err := w.cache.Get([]byte("header."+uv+"."+u), nil)
if err != nil {
need = append(need, uid)
continue
}
ch := &CachedHeader{}
dec := gob.NewDecoder(bytes.NewReader(data))
err = dec.Decode(ch)
if err != nil {
w.worker.Logger.Printf("cache: error decoding cached header %s.%s: %v", uv, u, err)
need = append(need, uid)
continue
}
hr := bytes.NewReader(ch.Header)
textprotoHeader, err := textproto.ReadHeader(bufio.NewReader(hr))
if err != nil {
w.worker.Logger.Printf("cache: error reading cached header %s.%s: %v", uv, u, err)
need = append(need, uid)
continue
}
hdr := &mail.Header{Header: message.Header{Header: textprotoHeader}}
mi := &models.MessageInfo{
BodyStructure: &ch.BodyStructure,
Envelope: &ch.Envelope,
Flags: []models.Flag{models.SeenFlag}, // Always return a SEEN flag
Uid: ch.Uid,
RFC822Headers: hdr,
}
found = append(found, uid)
w.worker.Logger.Printf("cache: located cached header %s.%s", uv, u)
w.worker.PostMessage(&types.MessageInfo{
Message: types.RespondTo(msg),
Info: mi,
}, nil)
}
if len(found) > 0 {
w.worker.PostAction(&types.FetchMessageFlags{
Uids: found,
}, nil)
}
return need
}
func cacheDir() (string, error) {
dir, err := os.UserCacheDir()
if err != nil {
dir, err = homedir.Expand("~/.cache")
if err != nil {
return "", err
}
}
return path.Join(dir, "aerc"), nil
}
// cleanCache removes stale entries from the selected mailbox cachedb
func (w *IMAPWorker) cleanCache() {
start := time.Now()
var scanned, removed int
iter := w.cache.NewIterator(nil, nil)
for iter.Next() {
data := iter.Value()
ch := &CachedHeader{}
dec := gob.NewDecoder(bytes.NewReader(data))
err := dec.Decode(ch)
if err != nil {
w.worker.Logger.Printf("cache: error cleaning database %d: %v", w.selected.UidValidity, err)
continue
}
exp := ch.Created.Add(w.config.cacheMaxAge)
if exp.Before(time.Now()) {
err = w.cache.Delete(iter.Key(), nil)
if err != nil {
w.worker.Logger.Printf("cache: error cleaning database %d: %v", w.selected.UidValidity, err)
continue
}
removed = removed + 1
}
scanned = scanned + 1
}
iter.Release()
elapsed := time.Since(start)
w.worker.Logger.Printf("cache: cleaned cache, removed %d of %d entries in %f seconds", removed, scanned, elapsed.Seconds())
}

View File

@ -56,6 +56,9 @@ func (w *IMAPWorker) handleConfigure(msg *types.Configure) error {
w.config.reconnect_maxwait = 30 * time.Second w.config.reconnect_maxwait = 30 * time.Second
w.config.cacheEnabled = false
w.config.cacheMaxAge = 30 * 24 * time.Hour // 30 days
for key, value := range msg.Config.Params { for key, value := range msg.Config.Params {
switch key { switch key {
case "idle-timeout": case "idle-timeout":
@ -114,9 +117,26 @@ func (w *IMAPWorker) handleConfigure(msg *types.Configure) error {
value, err) value, err)
} }
w.config.keepalive_interval = int(val.Seconds()) w.config.keepalive_interval = int(val.Seconds())
case "cache-headers":
cache, err := strconv.ParseBool(value)
if err != nil {
// Return an error here because the user tried to set header
// caching, and we want them to know they didn't set it right -
// one way or the other
return fmt.Errorf("invalid cache-headers value %v: %v", value, err)
}
w.config.cacheEnabled = cache
case "cache-max-age":
val, err := time.ParseDuration(value)
if err != nil || val < 0 {
return fmt.Errorf("invalid cache-max-age value %v: %v", value, err)
}
w.config.cacheMaxAge = val
} }
} }
if w.config.cacheEnabled {
w.initCacheDb(msg.Config.Name)
}
w.idler = newIdler(w.config, w.worker) w.idler = newIdler(w.config, w.worker)
w.observer = newObserver(w.config, w.worker) w.observer = newObserver(w.config, w.worker)

View File

@ -17,7 +17,15 @@ import (
func (imapw *IMAPWorker) handleFetchMessageHeaders( func (imapw *IMAPWorker) handleFetchMessageHeaders(
msg *types.FetchMessageHeaders) { msg *types.FetchMessageHeaders) {
toFetch := msg.Uids
if imapw.config.cacheEnabled && imapw.cache != nil {
toFetch = imapw.getCachedHeaders(msg)
}
if len(toFetch) == 0 {
imapw.worker.PostMessage(&types.Done{Message: types.RespondTo(msg)},
nil)
return
}
imapw.worker.Logger.Printf("Fetching message headers") imapw.worker.Logger.Printf("Fetching message headers")
section := &imap.BodySectionName{ section := &imap.BodySectionName{
BodyPartName: imap.BodyPartName{ BodyPartName: imap.BodyPartName{
@ -34,7 +42,7 @@ func (imapw *IMAPWorker) handleFetchMessageHeaders(
imap.FetchUid, imap.FetchUid,
section.FetchItem(), section.FetchItem(),
} }
imapw.handleFetchMessages(msg, msg.Uids, items, imapw.handleFetchMessages(msg, toFetch, items,
func(_msg *imap.Message) error { func(_msg *imap.Message) error {
reader := _msg.GetBody(section) reader := _msg.GetBody(section)
textprotoHeader, err := textproto.ReadHeader(bufio.NewReader(reader)) textprotoHeader, err := textproto.ReadHeader(bufio.NewReader(reader))
@ -48,17 +56,21 @@ func (imapw *IMAPWorker) handleFetchMessageHeaders(
return nil return nil
} }
header := &mail.Header{Header: message.Header{Header: textprotoHeader}} header := &mail.Header{Header: message.Header{Header: textprotoHeader}}
info := &models.MessageInfo{
BodyStructure: translateBodyStructure(_msg.BodyStructure),
Envelope: translateEnvelope(_msg.Envelope),
Flags: translateImapFlags(_msg.Flags),
InternalDate: _msg.InternalDate,
RFC822Headers: header,
Uid: _msg.Uid,
}
imapw.worker.PostMessage(&types.MessageInfo{ imapw.worker.PostMessage(&types.MessageInfo{
Message: types.RespondTo(msg), Message: types.RespondTo(msg),
Info: &models.MessageInfo{ Info: info,
BodyStructure: translateBodyStructure(_msg.BodyStructure),
Envelope: translateEnvelope(_msg.Envelope),
Flags: translateImapFlags(_msg.Flags),
InternalDate: _msg.InternalDate,
RFC822Headers: header,
Uid: _msg.Uid,
},
}, nil) }, nil)
if imapw.config.cacheEnabled && imapw.cache != nil {
imapw.cacheHeader(info)
}
return nil return nil
}) })
} }
@ -169,6 +181,24 @@ func (imapw *IMAPWorker) handleFetchFullMessages(
}) })
} }
func (imapw *IMAPWorker) handleFetchMessageFlags(msg *types.FetchMessageFlags) {
items := []imap.FetchItem{
imap.FetchFlags,
imap.FetchUid,
}
imapw.handleFetchMessages(msg, msg.Uids, items,
func(_msg *imap.Message) error {
imapw.worker.PostMessage(&types.MessageInfo{
Message: types.RespondTo(msg),
Info: &models.MessageInfo{
Flags: translateImapFlags(_msg.Flags),
Uid: _msg.Uid,
},
}, nil)
return nil
})
}
func (imapw *IMAPWorker) handleFetchMessages( func (imapw *IMAPWorker) handleFetchMessages(
msg types.WorkerMessage, uids []uint32, items []imap.FetchItem, msg types.WorkerMessage, uids []uint32, items []imap.FetchItem,
procFunc func(*imap.Message) error) { procFunc func(*imap.Message) error) {

View File

@ -12,13 +12,14 @@ import (
func (imapw *IMAPWorker) handleOpenDirectory(msg *types.OpenDirectory) { func (imapw *IMAPWorker) handleOpenDirectory(msg *types.OpenDirectory) {
imapw.worker.Logger.Printf("Opening %s", msg.Directory) imapw.worker.Logger.Printf("Opening %s", msg.Directory)
_, err := imapw.client.Select(msg.Directory, false) sel, err := imapw.client.Select(msg.Directory, false)
if err != nil { if err != nil {
imapw.worker.PostMessage(&types.Error{ imapw.worker.PostMessage(&types.Error{
Message: types.RespondTo(msg), Message: types.RespondTo(msg),
Error: err, Error: err,
}, nil) }, nil)
} else { } else {
imapw.selected = sel
imapw.worker.PostMessage(&types.Done{Message: types.RespondTo(msg)}, nil) imapw.worker.PostMessage(&types.Done{Message: types.RespondTo(msg)}, nil)
} }
} }

View File

@ -9,6 +9,7 @@ import (
sortthread "github.com/emersion/go-imap-sortthread" sortthread "github.com/emersion/go-imap-sortthread"
"github.com/emersion/go-imap/client" "github.com/emersion/go-imap/client"
"github.com/pkg/errors" "github.com/pkg/errors"
"github.com/syndtr/goleveldb/leveldb"
"git.sr.ht/~rjarry/aerc/lib" "git.sr.ht/~rjarry/aerc/lib"
"git.sr.ht/~rjarry/aerc/models" "git.sr.ht/~rjarry/aerc/models"
@ -49,6 +50,8 @@ type imapConfig struct {
keepalive_period time.Duration keepalive_period time.Duration
keepalive_probes int keepalive_probes int
keepalive_interval int keepalive_interval int
cacheEnabled bool
cacheMaxAge time.Duration
} }
type IMAPWorker struct { type IMAPWorker struct {
@ -63,6 +66,7 @@ type IMAPWorker struct {
idler *idler idler *idler
observer *observer observer *observer
cache *leveldb.DB
} }
func NewIMAPWorker(worker *types.Worker) (types.Backend, error) { func NewIMAPWorker(worker *types.Worker) (types.Backend, error) {
@ -178,6 +182,8 @@ func (w *IMAPWorker) handleMessage(msg types.WorkerMessage) error {
w.handleFetchMessageBodyPart(msg) w.handleFetchMessageBodyPart(msg)
case *types.FetchFullMessages: case *types.FetchFullMessages:
w.handleFetchFullMessages(msg) w.handleFetchFullMessages(msg)
case *types.FetchMessageFlags:
w.handleFetchMessageFlags(msg)
case *types.DeleteMessages: case *types.DeleteMessages:
w.handleDeleteMessages(msg) w.handleDeleteMessages(msg)
case *types.FlagMessages: case *types.FlagMessages:

View File

@ -133,6 +133,11 @@ type FetchMessageBodyPart struct {
Part []int Part []int
} }
type FetchMessageFlags struct {
Message
Uids []uint32
}
type DeleteMessages struct { type DeleteMessages struct {
Message Message
Uids []uint32 Uids []uint32