Fixes updating the flags in the imap backend. Before, the silent flag
was set incorrectly by 75fc42e ("imap: send message info updates for
bulk flag ops") which caused some imap servers to not send the updated
flags. By disabling the silent flag, the flag update will return a
corrsponding value that we can send back to the message store to update
the flags correctly.
Fixes: 75fc42e ("imap: send message info updates for bulk flag ops")
Reported-by: Jens Grassel <jens@wegtam.com>
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Tested-by: Jens Grassel <jens@wegtam.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Add option to open a message in the message viewer without setting the
seen flag. Enables the message viewer to be used as a preview pane
without changing the message flags unintentionally. Before, the message
viewer would set the seen flag by default. The IMAP backend will now
always fetch the message body with the peek option enabled (same as we
fetch the headers).
An "auto-mark-read" option is added to the ui config which is set to
true by default. If set the false, the seen flag is not set by the
message viewer.
Co-authored-by: "James Cook" <falsifian@falsifian.org>
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Add XOAUTH2 authentication support for IMAP and SMTP. Although XOAUTH2
is now deprecated in favor of OAuthBearer, it is the only way to connect
to Office365 since Basic Auth is now completely removed.
Since XOAUTH2 is very similar to OAuthBearer and uses the same
configuration parameters, this is basically a copy-paste of the existing
OAuthBearer code.
However, XOAUTH2 support was removed from go-sasl library, so this
change reimports the code that was removed from go-sasl and offers it
a new home in lib/xoauth2.go. Hopefully it shouldn't be too hard to
maintain, being less than 50 SLOC.
Link: https://github.com/emersion/go-sasl/commit/7bfe0ed36a21
Implements: https://todo.sr.ht/~rjarry/aerc/78
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Tested-by: Inwit <inwit@sindominio.net>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Pass message containing remaining directories to check. Account widget
will recursively call CheckMail with the remaining directories until
a Done message is returned.
Only needed for IMAP worker as other workers run check-mail-cmd in
a separate goroutine.
Suggested-By: Tim Culverhouse <tim@timculverhouse.com>
Signed-off-by: kt programs <ktprograms@gmail.com>
Tested-by: Tim Culverhouse <tim@timculverhouse.com>
Send message info updates back to to ui instead of posting a fetch
header action to the worker when performing a bulk flag operation. This
prevents the worker channels from filling up which can result in a
deadlock.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Tested-by: Tim Culverhouse <tim@timculverhouse.com>
Protect access to fields idleing and waiting via a mutex.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When an error is encountered fetching a header, the backends respond
with a type.Error worker message. On receipt of this message, the UI
deletes all pending headers. The headers are all requested again as they
remain on the screen, resulting in an infinite request loop - and an
infinite logging loop. The user only ever sees the spinner unless they
check the logs.
A previous commit intended to fix this, however it introduced a
regression where any message that was part of the fetch request would
also be marked as erroneous. This commit is reverted with commit
2aad2fea7d36 ("msgstore: revert 9fdc7acf5b48").
Send an erroneous message info message from the backend when an error is
encountered for a specific UID.
Fixes: 01f80721e2 ("msgstore: post MessageInfo on erroneous fetch")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Do not throw an error when the charset is unknown; the message entity
can still be read, but log the error instead.
Reported-by: falsifian
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When cached headers are fetched, an action is posted back to the Worker
to immediately fetch the flags for the message from the server (we can't
know the flags state, therefore it's not cached). When scrolling, a lag
occurs when loading cached headers because the n+1 message has to wait
for the flag request to return before the cached headers are retrieved.
Collect the message UIDs in the UI that need flags, and fetch them based
off a debounce timer in a single request. Post the action from the UI to
eliminate an (ugly) go routine in the worker.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
The IMAP worker has a few methods that post a new Action to itself. This
can create a deadlock when the worker.Actions channel is full: The
worker can't accept a new Action because it's trying to post an action.
This is most noticeable when cached headers are enabled and the message
list is scrolled fast.
Use a goroutine to post actions to the worker when posting from within
the worker.
Fixes: https://todo.sr.ht/~rjarry/aerc/45
Fixes: 7aa71d334b ("imap: add option to cache headers")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Implement MoveMessages in the imap backend. go-imap includes the MOVE
Imap extension by default, and if a server does not support it the
command fallsback to a copy-and-delete operation. Servers with the MOVE
extension will see a slight performance increase when moving messages
due to fewer round trips. The IMAP implementation uses a MessagesMoved
worker message to avoid polling the destination mailbox.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Commit fdfec2c07a seqmap: refactor seqmap to use slice instead of map
introduced a regression to imap sorting (if supported). The slice passed
to seqmap was being sorted in place by UID, thus breaking any sort order
sent by the server.
Create a copy of the slice to retain the sort order reported to the UI
while also keeping a correct map for seqnum -> UID in the worker.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Error wrapping as introduced in Go 1.13 adds some additional logic to
use for comparing errors and adding information to it.
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
Apply GoDoc comment policy (comments for humans should have a space
after the //; machine-readable comments shouldn't)
Use strings.ReplaceAll instead of strings.Replace when appropriate
Remove if/else chains by replacing them with switches
Use short assignment/increment notation
Replace single case switches with if statements
Combine else and if when appropriate
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
Replaces infinite for loops containing a select on a channel with a
single case with a range over the channel.
Removes redundant assignments to blank identifiers.
Remove unnecessary guard clause around delete().
Remove `if condition { return true } return false` with return condition
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
Empty branches are effectively dead code and should therefore be
removed.
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
The imap worker's seqmap is represented as a map of sequence number to
UID. This presents a problem when expunging group of messages from the
mailbox: each individual expunge decrements the sequence numbers by 1
(for every sequence number greater than the expunged). This requires a
looping around the map to update the keys. The use of a map also
requires that both the sequence number and the UID of a message be known
in order to insert it into the map. This is only discovered by fetching
individual message body parts (flags, headers, etc), leaving the seqmap
to be empty until we have fetched information about each message. In
certain instances (if a mailbox has recently been loaded), all
information is loaded in memory and no new information is fetched -
leaving the seqmap empty and the UI out of sync with the worker.
Refactor the seqmap as a slice, so that any expunge automatically
decrements the rest of the sequences.
Use the results of FetchDirectoryContents or FetchDirectoryThreaded to
initialize the seqmap with all discovered UIDs. Sort the UIDs in
ascending order: IMAP specification requires that sequence numbers start
at 1 increase in order of ascending UID.
Add individual messages to the map if they come via a MessageUpdate and
have a sequence number larger than our slice.
Update seqmap tests with new logic.
Reference: https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.2
Fixes: https://todo.sr.ht/~rjarry/aerc/69
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When a test fails with a uint comparison, assert displays the hex code
instead of an int, making it harder to debug. Use ints in sequmap test
asserts instead of uints for better readability when tests fail
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Reorder seqmap asserts to properly show display expected and actual when
performing go test -v ./...
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Remove unused code in the handleDeleteMessages routine. During
debugging, it was found that the channel for expunge updates was not
working and that all expunge details were coming through as
ExpungeUpdates. The reporting channel is unneeded.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Use the capabilities returned by the backend to check whether sort is
implemented when the user tries to use the sort command. Print a warning
to the log when a sort request is silently dropped by the backend.
Suggested-by: |cos|
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
This patch updates the seqNums after an Expunge operation. When an
expunge operation occurs, the seqNum of the deleted message is reported.
The Imap spec [0] states that an immediate decrement of all seqnums greater
than the deleted occurs, even before the next reporting of an expunge
update.
[0]: https://datatracker.ietf.org/doc/html/rfc3501#section-7.4.1
Fixes: https://todo.sr.ht/~rjarry/aerc/61
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Signed-off-by: Robin Jarry <robin@jarry.cc>
Do not pass logger objects around anymore. Shuffle some messages to make
them consistent with the new logging API. Avoid using %v when a more
specific verb exists for the argument types.
The loggers are completely disabled (i.e. Sprintf is not even called)
by default. They are only enabled when redirecting stdout to a file.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
This causes all raw email bodies to be dumped along with actual
debugging messages. I don't believe we neither need nor want such
a thing.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
When opening unread emails from certain people (I won't name any names,
sorry), an annoying error message is displayed on the status line:
could not get section &imap.BodySectionName{BodyPartName:
imap.BodyPartName{Specifier:"", Path:[]int(nil), Fields:[]string(nil),
NotFields:false}, Peek:false, Partial:[]int(nil), value}
This does not occur for already read messages. This issue is similar to
the one that was fixed in commit 8ed95b0d2a ("imap: avoid crash when
replying to unread message").
This happens because the flags are updated in the callback that receives
the message itself. It causes the flag update to arrive in the same
channel/request. Ignore the messages that have an empty body (i.e. only
containing flag updates). This is inherently racy but there seems no way
to get rid of these extra messages.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Acked-by: Koni Marti <koni.marti@gmail.com>
This patch enables the filtering of a threaded view which uses
server-built threads. Filtering is done server-side, in order to
preserve the use of server-built threads.
In adding this feature, the filtering of notmuch folders was brought up
to feature parity with the other workers. The filters function the same
(ie: they can be stacked). The notmuch filters, however, still use
notmuch syntax for the filtering.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
This patch provides a method to report backend capabilities to the UI.
The intial capabilities included in the report are Sort and Thread.
Having these available to the UI enables the client to better handle
server side threading.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
There are concurrent threads that are accessing and modifying
IMAPWorker.seqMap (the mapping of sequence numbers to message UIDs).
This can lead to crashes when trying to add and remove a message ID.
panic: runtime error: index out of range [391] with length 390
goroutine 1834 [running]:
git.sr.ht/~rjarry/aerc/logging.PanicHandler()
logging/panic-logger.go:47 +0x6de
panic({0xa41760, 0xc0019b3290})
/usr/lib/golang/src/runtime/panic.go:838 +0x207
git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1()
worker/imap/fetch.go:214 +0x185
created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages
worker/imap/fetch.go:209 +0x12b
Use a map which makes more sense than a simple array for random access
operations. Also, it allows better typing for the key values. Protect
the map with a mutex. Add internal API to access the map. Add basic unit
tests to ensure that concurrent access works.
Fixes: https://todo.sr.ht/~rjarry/aerc/49
Signed-off-by: Robin Jarry <robin@jarry.cc>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
Add option to cache headers for imap accounts. Cache db is located at
$XDG_CACHE_DIR/aerc/{account name}. The cache is cleaned of stale
entries when aerc is first opened.
Two new account level configuration options are introduced:
* cache-headers (Default: false)
* cache-max-age (Default: 30 days (720 hours))
The change in worker/imap/open.go is to set the selected directory. This
is required to access the UIDVALIDITY field, which is used in
combination with the message ID to form the key for use in the cache db.
The key structure is: "header.{UIDVALIDITY}.{UID}"
Where reasonable, cache does not stop aerc from running. In general, if
there is an error in the cache, aerc should continue working as usual.
Errors are either displayed to the user or logged.
All messages are stored without flags, and when retrieved have the flags
set to SEEN. This is to prevent UI flashes. A new method to
FetchMessageFlags is introduced to update flags of cached headers. This
is done asynchronously, and the user will see their messages appear and
then any flags updated. The message will initially show as SEEN, but
will update to unread. I considered updating the cache with the
last-known flag state, however it seems prudent to spare the R/W cycle
and assume that - eventually - all messages will end up read, and if it
isn't the update will occur rather quickly.
Note that leveldb puts a lock on the database, preventing multiple
instances of aerc from accessing the cache at the same time.
Much of this work is based on previous efforts by Vladimír Magyar.
Implements: https://todo.sr.ht/~rjarry/aerc/2
Thanks: Vladimír Magyar <vladimir@mgyar.me>
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Tested-by: inwit <inwit@sindominio.net>
Reviewed-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When moving/copying/deleting/archiving a message in imap, the RUE counts
displayed in the dirlist would not update properly. Maildir has (had) an
implementation that recounts the entire directory and updates the
DirectoryInfo after one of these actions.
This patch implements a more efficient method of updating, and also
enables it to apply to IMAP without any additional requests. Upon
completion of the action, the counts are manually updated with the count
of messages in the action and recent and/or unseen states of those
messages. This is more efficient for maildir, because we aren't counting
everything in the store. For IMAP, we get the updates for free because
we are only performing the update after confirmation from the server
that the action has happened.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Check for new mail (recent, unseen, exists counts) with an external
command, or for imap with the STATUS command, at start or on
reconnection and every X time duration
IMAP:
The selected folder is skipped, per specification. Additional config
options are included for including/excluding folders explicitly.
Maildir/Notmuch:
An external command will be run in the background to check for new mail.
An optional timeout can be used with maildir/notmuch. Default is 10s
New account options:
check-mail
check-mail-cmd (maildir/notmuch only)
check-mail-timeout (maildir/notmuch only), default 10s
check-mail-include (IMAP only)
check-mail-exclude (IMAP only)
If unset, or set less than or equal to 0, check-mail will be ignored
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Tested-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
Extract the tcp connection details and timeout the tcp connect functions
(net.ResolveTCPAddr and net.DialTCP). If timed out, ensure that the
connection is properly closed.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Add a debouncer to the idle mode. Avoid unnecessary idling when another
job arrives within a certain time frame.
For example, the ui sends three messages to the worker at the same time
when we open a message (FlagMessage, FetchMessageBodyPart, and the
FetchMessageHeaders). The debouncer prevents the unnecessary entering
and leaving of the idle mode between those messages.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Untangle the observer functionality from the message handling routine.
Observe the imap client's logout channel and trigger a connection error
when necessary to start the reconnect cycle.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Untangle the idle functionality from the message handling routine. Wait
for the idle mode to properly exit every time to ensure a consistent
imap state. Timeout when hanging in idle mode and inform the ui.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Extract the imap config and move the configure part out of the message
handler.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Check slice bounds before using it for the message and expunge updates.
Log the error but ignore the affected updates.
Link: https://lists.sr.ht/~rjarry/aerc-devel/%3CCJEHBFFUI11T.1AYGOMVGZ87ZS%40rek2system%3E
Reported-by: ReK2 <rek2@hispagatos.org>
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When running `:reply -q` on an unread message, aerc crashes after
opening the editor:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5d1019]
goroutine 63 [running]:
bufio.(*Reader).fill(0xc000086ef8)
/usr/lib/golang/src/bufio/bufio.go:106 +0xd9
bufio.(*Reader).Peek(0xc00020bef8, 0x1)
/usr/lib/golang/src/bufio/bufio.go:144 +0x5d
github.com/emersion/go-message/textproto.ReadHeader(0xc00004a700?)
emersion/go-message@v0.15.0/textproto/header.go:525 +0x5f
git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessageBodyPart.func1(0xc00056e280)
worker/imap/fetch.go:99 +0x1ab
git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1()
worker/imap/fetch.go:178 +0xd7
created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages
worker/imap/fetch.go:172 +0x12b
This happens because the flags are updated in the callback that receives
the message itself. It causes the flag update to arrive in the same
channel/request. Ignore the messages that have an empty body (i.e. only
containing flag updates). This is inherently racy but there seems no way
to get rid of these extra messages.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Tested-by: Connor Kuehl <cipkuehl@gmail.com>
Since panics still regularly "destroy" the terminal, it is hard to get a
stack trace for panics you do not anticipate. This commit adds a panic
handler that automatically creates a logfile inside the current working
directory.
It has to be added to every goroutine that is started and will repair
the terminal on a panic.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
This commit fixes all occurrences of the abovementioned lint-error in
the codebase.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
Start the reconnect cycle when the initial connect fails. Make the
connection observer send a connection error when the imap client is nil.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
waits an increasing amount of time before attempting a reconnect.
Wait is capped at 16s. Prevents many reconnect attemps in a short time period.
Fixes commit 05ad96a30c ("imap: improve reconnect stability") that
improved the reliability of the reconnect mechanism but did not
implement controls to prevent the triggering of too many reconnects
within a short period of time.
Fixes: 05ad96a30c ("imap: improve reconnect stability")
Signed-off-by: Koni Marti <koni.marti@gmail.com>
improves the robustness of the imap reconnect feature which was
introduced in commit beae17a6da ("imap: auto-reconnects on connection
error").
If a connection error is emitted, the message list is cleared and a
corresponding error message is shown in the ui. Status bar is updated as
well. Upon reconnect, the directories and the message list will be
re-fetched (same behavior as the connect command).
Reconnect can be enabled and disabled with the connect and the
disconnect commands.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Some messages contain invalid headers, textproto.ReadHeader fails with
an error:
malformed MIME header key: From nobody Fri Jan 14 19
And all other messages of the selected folder are ignored.
Return an explicit error message to the UI and continue processing other
messages. This is consistent with what the maildir worker does.
Signed-off-by: Robin Jarry <robin@jarry.cc>
if the worker emits a connection error, the ui will automatically send back a
reconnect command. The worker then establishes a new connection. Auto-reconnect
is disabled when the user sends the disconnect command.
Fixes: https://todo.sr.ht/~rjarry/aerc/1
Signed-off-by: Koni Marti <koni.marti@gmail.com>
implements a new connection error message. This allows the worker to emit a
connection-related error message to the ui when the imap client closes the
loggedOut channel.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Koni Marti
Julian Pidancet
kt programs
Tim Culverhouse
Moritz Poldrack
Robin Jarry