By default "notmuch new" will index files in place and won't move new
files from the new/ directory to cur/ because it assumes that is the
job of the email client.
During normal operation, once moved by the client, the "notmuch new"
command will "fix" the database by detecting file renames.
This workflow is a problem because we need to move the new/ files to
cur/, otherwise the maildir lib will not work properly, but at the same
time we cannot afford the notmuch database to be out of sync with the
location of message files on disk, because we rely on it for listing
folders, displaying emails, ect...
This change uses a trick that request notmuch to synchronize message
tags to maildir flags, that will effectively rename new files and cause
them to be moved into the cur/ directory.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Add a method to retrieve the message key associated with a message file
path, if indexed in the database.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Print "failed to remove tag" instead of "failed to add tag" when
removing tag failed.
Fixes: 5ca6022d00 ("lint: ensure errors are at least logged (errcheck)")
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Send a DirectoryInfo message when check-mail completes to re-fetch
directory content in case it changed.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Refactor the code emitting DirectoryInfo messages. Reduce function
call indirections by retiring gatherDirectoryInfo(), buildDirInfo() and
emitDirectoryInfo(), and replacing them with getDirectoryInfo()
(aligning the code with what is done in the maildir worker by the same
occasion).
Also merge queryFromName(), which no longer needs to be called from
different places, in handleOpenDirectory().
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Add the "maildir-store" account configuration option to select the
maildir store to associate with the notmuch database.
This also allows the previous changes to be backward compatible since
not specifying this option will make the backend behave the same as if
there were no changes.
Fixes: https://todo.sr.ht/~rjarry/aerc/73
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
By associating the notmuch database with a maildir store, we can add the
Copy/Move/Delete operations on messages to the notmuch backend.
This change assumes that the notmuch database location is also the root
of the maildir store.
In a previous change, we added the ability to dynamically add and remove
message files to the notmuch DB. This change uses this facility to
synchronize the database with the filesystem operations on maildir
files.
While it's still possible to use the query-map file to create virtual
folders from notmuch search queries, the sidebar is now loaded with the
folders found in the maildir store.
With notmuch, two identical but distinct message files can be indexed in
the database with the same key. This change takes extra care of only
deleting or removing message files from the maildir corresponding to the
folder that is currently selected (if any).
Implements: https://todo.sr.ht/~rjarry/aerc/88
Fixes: https://todo.sr.ht/~rjarry/aerc/73
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Replace ListFolder with a new method that returns a map indexed by
folder names instead of a list of folder names. A map is simpler to use
and more efficient in case we only want to check the presence of a
specific folder in the Maildir store.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
This change moves code that could be common to both notmuch and maildir
workers in worker/lib.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
The new IndexFile and DeleteMessage allow dynamically inserting and
removing files to/from a notmuch database.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Multiple files on the filesystem can be referenced under a single key in
the notmuch database. MsgFilenames() returns a list of filenames
associated with an index key.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Previous change "maildir: hide invalid folders" prevents filepath.Walk
from descending into directories that don't contain new/, tmp/ or cur/
because the WalkDirFunc returns filepath.SkipDir.
We have to assume that these directories can be the parent of valid
maildir folders even if they're not themselves valid maildir folders, so
it is a mistake not to recurse into these subfolders. Fix it by
returning nil instead of filepath.SkipDir in the WalkDirFunc.
Fixes: 150aa0f498 ("maildir: hide invalid folders")
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
This field is always nil. Either we should initialize it to a proper
value or remove it. Since it is unused, remove it.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
A little coding hygiene cannot hurt. Add a simple awk script to check
all source files for bad white space habits:
- trailing white space
- trailing new lines at the end of files
- missing new line at the end of files
- spaces followed by tabs
The script outputs color when the terminal supports it. It exits with
a non-zero code when there was at least one white space issue found.
Call the script in the lint step.
Example output of the awk script:
config/default_styleset:1:# <-- trailing whitespace
config/default_styleset:3:# <-- trailing whitespace
doc/aerc.1.scd:78: Executes an arbitrary command in the background. Aerc will set the <-- trailing whitespace
doc/aerc.1.scd:234: <-- trailing whitespace
doc/aerc.1.scd:237: <-- trailing whitespace
worker/types/thread_test.go:74: // return ErrSkipThread<-- space(s) followed by tab(s)
worker/lib/testdata/message/invalid/hexa: trailing new line(s)
Fix issues reported by the script.
NB: The ENDFILE match is a GNU extension. It will be ignored on BSD-awk
and trailing new lines will not be detected. The lint make target is
only invoked on alpine linux which has GNU awk anyway.
NB: Empty cells in scdoc tables require trailing white space... Avoid
this by setting content in these cells. I don't really see a use for
empty cells.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Tested-by: Moritz Poldrack <moritz@poldrack.dev>
Non-selected folders will now have their total/unread/new counts updated
in the background when a check-mail happens.
Signed-off-by: Ben Cohen <ben@bencohen.net>
Acked-by: Robin Jarry <robin@jarry.cc>
Combine tcell events with WorkerMessages to better synchronize state
with IO and UI. Remove Tick loop for rendering. Use events to trigger
renders.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Fixes updating the flags in the imap backend. Before, the silent flag
was set incorrectly by 75fc42e ("imap: send message info updates for
bulk flag ops") which caused some imap servers to not send the updated
flags. By disabling the silent flag, the flag update will return a
corrsponding value that we can send back to the message store to update
the flags correctly.
Fixes: 75fc42e ("imap: send message info updates for bulk flag ops")
Reported-by: Jens Grassel <jens@wegtam.com>
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Tested-by: Jens Grassel <jens@wegtam.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Add option to open a message in the message viewer without setting the
seen flag. Enables the message viewer to be used as a preview pane
without changing the message flags unintentionally. Before, the message
viewer would set the seen flag by default. The IMAP backend will now
always fetch the message body with the peek option enabled (same as we
fetch the headers).
An "auto-mark-read" option is added to the ui config which is set to
true by default. If set the false, the seen flag is not set by the
message viewer.
Co-authored-by: "James Cook" <falsifian@falsifian.org>
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
The worker uses a buffered channel to queue tasks. Buffered channels
are effective at FIFO, but are prone to blocking. The design of aerc is
such that the UI must always accept a response from the backends, and
the backends must always accept a request from the UI. By using buffered
channels for both of these communication channels, a deadlock will
occur.
Break the chain by using a doubly linked list (container/list from the
standard library) to queue tasks for the worker. Essentially, this is an
infinitely buffered channel - but more memory efficient as it can change
size dynamically.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Signed-off-by: Robin Jarry <robin@jarry.cc>
The maildir worker currently populates the list of mail folders by
listing all the filesystem subdirectories in the maildir directory.
Although there's no official specification for maildir subfolders,
they should all have cur/ new/ and tmp/ subdirectories to be valid.
This patch prevents directories that don't have those subdirectories
present on the filesystem from appearing in the account folder list.
This is useful for example to prevent ".notmuch" and ".notmuch/xapian"
from showing up in the folder list if using notmuch to index emails
while using aerc's maildir backend.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Subsitute the format specifier %w for %v in the logging facility. The
logging functions use a fmt.Sprintf call behind the scene which does not
recognize %w. %w should be used in fmt.Errorf when you want to wrap
errors. Hence, the log entries that use %w are improperly formatted like
this:
ERROR 2022/10/02 09:13:57.724529 worker.go:439: could not get message
info %!w(*fmt.wrapError=&{could not get structure: [snip] })
^
Links: https://go.dev/blog/go1.13-errors
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
Add XOAUTH2 authentication support for IMAP and SMTP. Although XOAUTH2
is now deprecated in favor of OAuthBearer, it is the only way to connect
to Office365 since Basic Auth is now completely removed.
Since XOAUTH2 is very similar to OAuthBearer and uses the same
configuration parameters, this is basically a copy-paste of the existing
OAuthBearer code.
However, XOAUTH2 support was removed from go-sasl library, so this
change reimports the code that was removed from go-sasl and offers it
a new home in lib/xoauth2.go. Hopefully it shouldn't be too hard to
maintain, being less than 50 SLOC.
Link: https://github.com/emersion/go-sasl/commit/7bfe0ed36a21
Implements: https://todo.sr.ht/~rjarry/aerc/78
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Tested-by: Inwit <inwit@sindominio.net>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
Commit 716ade8968 ("worker: lock access to callback maps") introduced
locks to the worker callback maps. The locks also locked the processing
of the callback, which had the unintended side effect of deadlocking the
worker if any callbacks attempted to post a new action or message.
Refactor the locks to only lock the worker while accessing the maps.
Fixes: 716ade8968 ("worker: lock access to callback maps")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Pass message containing remaining directories to check. Account widget
will recursively call CheckMail with the remaining directories until
a Done message is returned.
Only needed for IMAP worker as other workers run check-mail-cmd in
a separate goroutine.
Suggested-By: Tim Culverhouse <tim@timculverhouse.com>
Signed-off-by: kt programs <ktprograms@gmail.com>
Tested-by: Tim Culverhouse <tim@timculverhouse.com>
Moves logic for creating dynamic folders from the dirlist widget to the
backend. Since dynamic folders are notmuch-specific, the notmuch backend
should be responsible for correctly setting up those folders. It does
that by sending two DirectoryInfos: the first to create the message
store, the second to fetch the directory content.
This approach also fixes a deadlock introduced by 716ade8968
("worker: lock access to callback maps").
Reported-by: Bence Ferdinandy <bence@ferdinandy.com>
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Tested-by: Tim Culverhouse <tim@timculverhouse.com>
When parsing address header fields, the field charset is automatically
decoded to UTF-8. If the charset is unknown, use the raw field value.
Fixes: https://todo.sr.ht/~rjarry/aerc/91
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Tested-by: Tim Culverhouse <tim@timculverhouse.com>
Send message info updates back to to ui instead of posting a fetch
header action to the worker when performing a bulk flag operation. This
prevents the worker channels from filling up which can result in a
deadlock.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Tested-by: Tim Culverhouse <tim@timculverhouse.com>
Protect access to fields idleing and waiting via a mutex.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Worker callbacks are inherently set and called from different
goroutines. Protect access to all callback maps with a mutex.
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When an error is encountered fetching a header, the backends respond
with a type.Error worker message. On receipt of this message, the UI
deletes all pending headers. The headers are all requested again as they
remain on the screen, resulting in an infinite request loop - and an
infinite logging loop. The user only ever sees the spinner unless they
check the logs.
A previous commit intended to fix this, however it introduced a
regression where any message that was part of the fetch request would
also be marked as erroneous. This commit is reverted with commit
2aad2fea7d36 ("msgstore: revert 9fdc7acf5b48").
Send an erroneous message info message from the backend when an error is
encountered for a specific UID.
Fixes: 01f80721e2 ("msgstore: post MessageInfo on erroneous fetch")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Do not throw an error when the charset is unknown; the message entity
can still be read, but log the error instead.
Reported-by: falsifian
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When cached headers are fetched, an action is posted back to the Worker
to immediately fetch the flags for the message from the server (we can't
know the flags state, therefore it's not cached). When scrolling, a lag
occurs when loading cached headers because the n+1 message has to wait
for the flag request to return before the cached headers are retrieved.
Collect the message UIDs in the UI that need flags, and fetch them based
off a debounce timer in a single request. Post the action from the UI to
eliminate an (ugly) go routine in the worker.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
This reverts commit 7473571159.
The commit has introduced a regression that lead to the pager not being
filled with content, thereby making reading mails impossible.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
Send to worker.Messages in goroutine to prevent deadlocks: the UI can
fill the worker.Actions channel. The worker can generate more than one
Message per action, and if it generates enough to fill the
worker.Messages channel from a single message while the worker.Actions
channel is full, a deadlock occurs.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
The IMAP worker has a few methods that post a new Action to itself. This
can create a deadlock when the worker.Actions channel is full: The
worker can't accept a new Action because it's trying to post an action.
This is most noticeable when cached headers are enabled and the message
list is scrolled fast.
Use a goroutine to post actions to the worker when posting from within
the worker.
Fixes: https://todo.sr.ht/~rjarry/aerc/45
Fixes: 7aa71d334b ("imap: add option to cache headers")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Fix reggression introduced by 70bfcfef42 ("lint: work nicely with
wrapped errors (errorlint)").
Discovered this because it broke my arec-notmuch config where I have
`exclude-tags=deleted`. Queries with `tag:deleted` would now fail with
error message saying "Argument was ignored".
Fixes: 70bfcfef42 ("lint: work nicely with wrapped errors (errorlint)")
Signed-off-by: Jose Lombera <jose@lombera.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
The maildir worker watches the file system for events (new mail, moved
mail, copied mail, etc). On an event, the worker sends a
DirectoryContents message and a DirectoryInfo message.
Do not send DirectoryContents on FS Event in the maildir worker. The
DirectoryInfo message already triggers the UI to request the new
DirectoryContents, and does so in a more predictable way (IE any
currently applied filter is applied).
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
The built-in maildir.Dir.Move method performs an OS level file rename,
which allows for preserving file creation time. Commit c98f704874
("move: enable MoveMessages from msgstore") enabled the use of the Move
method for maildir. One particular maildir synchronizer (isync/mbsync)
encodes the UID within the filename of the email and cannot recover if
the UID is preserved during a move. mbsync encodes filenames like so:
/path/to/email/{maildir-key},U={uid}:2,S
OfflineIMAP encodes the UID within the filename, but also encodes a hash
of the originating folder so that it can recover from a move without a
rename of the underlying file. OfflineIMAP encodes like so:
/path/to/email{maildir-key},U={uid},FMD5={folder-hash}:2,S
Remove encoded UIDs of the form `,U={uid}` from filenames to prevent
sync issues.
Fixes: https://todo.sr.ht/~rjarry/aerc/75
Fixes: c98f704874 ("move: enable MoveMessages from msgstore")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
When there is no Date header in a message, aerc falls back to the
Received header and tries to extract an rfc1123z date from it
(introduced in commit d1600e46). The current regex for extracting the
date incorrectly allows for trailing whitespace, causing time.Parse() to
fail inside of parseReceivedHeader(). As a result, the message's date is
shown as "???????????????????" in the message list and as
"0000-12-31 07:03 PM" in the message view (the latter is likely related
to the zero value of time.Time).
Steps to reproduce:
1) Send yourself a message with no Date header, e.g. with msmtp:
printf 'Subject: foo bar\n\nbody text\n' | msmtp --set-date-header=off me@example.com
2) Note the message's displayed date in aerc's message list and message
view.
Signed-off-by: Thomas Faughnan <tom@tjf.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
Implement MoveMessages in the maildir worker. go-maildir supports Move
operations by default, and is functionally equivalent to a OS-level
rename. Creation date of the file is preserved by using Move, which is
used by at least one maildir-to-IMAP synchronizer (isync/mbsync). The
previous move method of copy-and-delete would reset the creation date of
the message, and potentially cause sorting issues in other email
clients.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Implement MoveMessages in the imap backend. go-imap includes the MOVE
Imap extension by default, and if a server does not support it the
command fallsback to a copy-and-delete operation. Servers with the MOVE
extension will see a slight performance increase when moving messages
due to fewer round trips. The IMAP implementation uses a MessagesMoved
worker message to avoid polling the destination mailbox.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Implement MoveMessages handling in the mbox backend. The mbox backend
exists entirely in memory, so the handling is equivalent to a
copy-and-delete.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Introduce a MoveMessages worker message to use for improved message
moving in subsequent patches.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Commit fdfec2c07a seqmap: refactor seqmap to use slice instead of map
introduced a regression to imap sorting (if supported). The slice passed
to seqmap was being sorted in place by UID, thus breaking any sort order
sent by the server.
Create a copy of the slice to retain the sort order reported to the UI
while also keeping a correct map for seqnum -> UID in the worker.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Use FetchDirectoryContents for filtering instead of the SearchDirectory
message. This was an omission from rebasing the mbox worker and from not
realizing that c2f4404f ("threading: enable filtering of server-side
threads") changed the way we filter in the message store for the
server-side threading implementation. This patch enables filtering for
the mbox worker.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Mark or unmark the shown message threads. Threads must be available in the
message store. Use the -T option for the mark or unmark commands. Can be
used in combination with the toggle flag (-t).
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Since the minimum required version of Go has been bumped to 1.16, the
deprecation of io/ioutil can now be acted upon. This Commit removes the
remaining dependencies on ioutil and replaces them with their io or os
counterparts.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
Fix the following test failures:
FAIL: TestMessageInfoHandledError (0.00s)
parse_test.go:53: could not parse envelope: date parsing failed:
unrecognized date format:
FAIL: TestReader (0.07s)
gpg_test.go:27: using GNUPGHOME = /tmp/TestReader2384941142/001
reader_test.go:108: Test case: Invalid Signature
reader_test.go:112: gpg.Read() = gpgmail: failed to read PGP
message: gpg: failed to run verification: exit status 1
Fixes: 5ca6022d00 ("lint: ensure errors are at least logged (errcheck)")
Fixes: 70bfcfef42 ("lint: work nicely with wrapped errors (errorlint)")
Signed-off-by: Robin Jarry <robin@jarry.cc>
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Error wrapping as introduced in Go 1.13 adds some additional logic to
use for comparing errors and adding information to it.
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
Apply GoDoc comment policy (comments for humans should have a space
after the //; machine-readable comments shouldn't)
Use strings.ReplaceAll instead of strings.Replace when appropriate
Remove if/else chains by replacing them with switches
Use short assignment/increment notation
Replace single case switches with if statements
Combine else and if when appropriate
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
Julian Pidancet
Robin Jarry
Ben Cohen
Tim Culverhouse
Koni Marti
kt programs
Moritz Poldrack
Jose Lombera
Thomas Faughnan