Some messages contain invalid headers, textproto.ReadHeader fails with
an error:
malformed MIME header key: From nobody Fri Jan 14 19
And all other messages of the selected folder are ignored.
Return an explicit error message to the UI and continue processing other
messages. This is consistent with what the maildir worker does.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Since commit 01c96e78df ("Update DirectoryInfo handling for maildir"),
flags are checked for every message of a folder when entering it.
Iterating over all messages of a folder takes a long time for large
collections of emails.
Only count the number of messages and state that the directory info
counts are not accurate. Defer the parsing of message flags in
a goroutine to have a more responsive UI.
Fixes: https://todo.sr.ht/~rjarry/aerc/16
Suggested-by: Koni Marti <koni.marti@gmail.com>
Signed-off-by: Robin Jarry <robin@jarry.cc>
This fixes piped full message (:pipe -m) being empty.
Fixes: 904ffacb0e ("maildir,notmuch: avoid leaking open files")
Signed-off-by: Nguyễn Gia Phong <mcsinyx@disroot.org>
Previously, Message.NewReader returned the wrapped buffered reader
without a reference to the opened file, so the files descriptors
were left unclosed after reading. Now, the file reader is returned
directly and closed on the call site. Buffering is not needed here
because it is an implementation detail of go-message.
Fixes: https://todo.sr.ht/~rjarry/aerc/9
if the worker emits a connection error, the ui will automatically send back a
reconnect command. The worker then establishes a new connection. Auto-reconnect
is disabled when the user sends the disconnect command.
Fixes: https://todo.sr.ht/~rjarry/aerc/1
Signed-off-by: Koni Marti <koni.marti@gmail.com>
implements a new connection error message. This allows the worker to emit a
connection-related error message to the ui when the imap client closes the
loggedOut channel.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
fixes the panic when the user sends multiple connect commands and is
already connected. The panic is caused by closing an already closed
channel. This happens when the idle re-init code is not executed, e.g.
when there's a return statement in the switch block. A defer func()
before the switch block will prevent this. The existing behavior of only
creating a new idleStop channel when properly connected is preseverd.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
This reverts commit c605ada3dd.
This breaks reading message bodies. I am not sure why, I'll take some
time to fix it later.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Attempt to reconnect to the server when there is an unexpected
disconnection or network error.
Use the Client.LoggedOut() channel which is closed when the connection
is closed.
This patch is rather flaky and is certainly bugged. However, it is
a start.
Signed-off-by: Robin Jarry <robin@jarry.cc>
There was a change in how build tags are formatted. Use this as new
reference.
Link: https://go.dev/doc/go1.17#gofmt
Signed-off-by: Robin Jarry <robin@jarry.cc>
Fix the following build error on mac os:
worker/imap/worker.go:368:29: undefined: syscall.TCP_KEEPCNT
worker/imap/worker.go:376:29: undefined: syscall.TCP_KEEPINTVL
These symbols are not defined on darwin.
Fixes: 5dfeff75f3 ("imap: add tcp connection options")
Signed-off-by: Robin Jarry <robin@jarry.cc>
In preparation for tcp keepalive options, we need access to the
net.TCPConn object associated with an IMAP connection. The only way to
do this is to create the connection ourselves.
No functional change.
Signed-off-by: Robin Jarry <robin@jarry.cc>
This will prepare for extra tcp connection options support and for
automatic reconnect. No functional change.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Messages flags can also be changed without changing directories.
Changing flags in maildirs means renaming the message files. Also take
renames into account.
Link: https://cr.yp.to/proto/maildir.html
Fixes: f4d3c8fc77 ("maildir: watch for external changes")
Signed-off-by: Robin Jarry <robin@jarry.cc>
When a maildir is synchronized by an external process while aerc is
running (e.g. mbsync), some emails may be moved out of "new" to "cur" or
completely deleted.
These deletions are ignored and aerc may assume these messages are still
here, leading to errors.
Take file deletions into account. Also, add "cur" to the watched
folders since these can be modified as well.
Signed-off-by: Robin Jarry <robin@jarry.cc>
When using the notmuch backend, it often makes more sense to sort
folders (actual virtual folders, or queries) by the order specified in
the query-map file, rather than alphabetically. This patch introduces a
configuration option (disabled by default) that allows this.
Additionally, due to the notmuch backend previously using maps (which
are order-undefined) to store the list of queries, default query
selection on aerc startup fluctuated. This patch fixes that by using
slices to store query order.
Display threads in the message list. For now, only supported by the
notmuch backend and on IMAP when the server supports the THREAD
extension.
Setting threading-enable=true is global and will cause the message list
to be empty with maildir:// accounts.
Co-authored-by: Kevin Kuehler <keur@xcf.berkeley.edu>
Co-authored-by: Reto Brunner <reto@labrat.space>
Signed-off-by: Robin Jarry <robin@jarry.cc>
In the maildir worker we manually need to track the Recent flag in order for the
notification command etc to work.
Push that responsibility to the container, we must make sure to manually add the
flag though if one grabs the message info.
We frequently had issues with notmuch segfaulting and my guess is that this
was due to the garbage collection magic used in the module.
This changes to a fork that ripped the functionality out.
We made a new type out of go-message/mail.Address without any real reason.
This suddenly made it necessary to convert from one to the other without actually
having any benefit whatsoever.
This commit gets rid of the additional type
This change handles message parse errors by printing the error when the
user tries to view the message. Specifically only handling unknown
charset errors in this patch, but there are many types of invalid
messages that can be handled in this way.
aerc currently leaves certain messages in the msglist in the pending
(spinner) state, and I'm unable to view or modify the message. aerc also
only prints parse errors with message when they are initially loaded.
This UX is a little better, because you can still see the header info
about the message, and if you try to view it, you will see the specific
error.
This allows us to hook into the std libs implementation of parsing related stuff.
For this, we need to get rid of the distinction between a mailbox and a host
to just a single "address" field.
However this is already the common case. All but one users immediately
concatenated the mbox/domain to a single address.
So this in effects makes it simpler for most cases and we simply do the
transformation in the special case.
The `:rmdir` command removes the current directory (`-f` is required if
the directory is not empty).
This is not supported on the notmuch backend.
An issue with the maildir backend is that some sync programs (e.g.
offlineimap) may recover the directory after it is deleted. They need
to specifically be configured to accept deletions, or special commands
need to be executed (e.g. `offlineimap --delete-folder`) to properly
delete folders.
A danger of using this on the IMAP backend is that it is possible for a
new message to be added to the directory and for aerc to not show it
immediately (due to a slow connection) - using `:rmdir` at this moment
(with `-f` if the directory already contains messages) would delete the
directory and the new message that just arrived (and all other
contents). This is documented in aerc(1) so that users are aware of
possible risks.
If a message date would fail to parse, the worker would never receive
the MessageInfo it asked for, and so it wouldn't display the message.
The problem is the spec for date formats is too lax, so trying to ensure
we can parse every weird date format out there is not a strategy we want
to pursue. On the other hand, preventing the user from reading and
working with a message due to the error format is also not a solution.
The maildir and notmuch workers will now fallback to the internal date, which
is based on the received header if we can't parse the format of the Date header.
The UI will also fallback to the received header whenever the date header can't
be parsed.
This patch is based on the work done by Lyudmil Angelov <lyudmilangelov@gmail.com>
But tries to handle a parsing error a bit more gracefully instead of just returning
the zero date.
If accounts.conf contains an invalid maildir url, return a nice
error instead of panicking.
Log a couple of different error cases to provide extra
information about the error to the user.
Aerc usually used the path []int{1} if it didn't know what the proper path is.
However this only works for multipart messages and breaks if it isn't one.
This patch removes all the hard coding and extracts the necessary helpers to lib.
Provide search and filter with the option to specify more flag based
conditions.
Use '-x <flag>' to search for messages with a flag (seen, answered,
flagged) and '-X <flag>' to search for messages without a flag.
When message dates failed to parse, the error displayed would try to
include the time object it failed to obtain, which would display as
something like 0001-01-01 00:00:00 UTC, which isn't of much help.
Instead, display the text we were trying to parse into a date, which
makes the problem easier to debug.
More mail flags can now be set, unset, and toggled, not just the
read/seen flag.
This functionality is implemented with a new `:flag` and `:unflag`
command, which are extensions to the matching `:read` and `:unread`
commands, adding support for different flags. In fact, the
`read`/`unread` commands are now recognized aliases to `flag`/`unflag`.
The new commands are also well documented in aerc(1).
The change mostly extends the previous read/unread setting functionality
by adding a selection for the flag to change.
- Add maildir flags to complement a messages imap flags
- Set the "seen" flag on sent messages when using the maildir backend
- Cleanup AppendMessage interface to use models.Flag for both IMAP and
maildir
There was an unused error value as well as unnecessary usage of the sort
interface. There should now be less copying so a bit better performance
in some cases.
This ensures that the directory info is up to date on events in the
maildir worker. This also sets up the initial dirinfo for other
directories and updates them when using built-in commands.
FS events are still only watched for the selected directory. This should
be changed in a future patch to watch other directories too in order to
cover UI updates for folders when an event occurs in a non-selected
folder.
Apparently sending an event for every incoming messageInfo slows down
the application significantly.
Therefore this slows down the emmision rate, on the cost of being out of date
in some cases.
Actions such as read / unread or the addition of new messages do change
the read/unread/recent count. Hence we request an update from the workers.
Workers going over the network should probably cache the information and invalidate
it only if necessary
The idle restart code is at the end of handleMessage in the worker.
However if an unsupported msg comes in, we returned early, skipping the re-init.
That lead to a crash due to double closing idleStop in the next iteration.
Opening a notmuch DB gives you a snapshot of the stage at that specific time.
Prior to this, we only reopened the DB upon writing.
However, if say a mail sync program like offlineimap is fetching new mail,
we would never pick it up.
This commit caches a db for a while, so that we don't generate too much overhead
and does a reconnect cycle after that.
I hardcoded a value as I don't think that having an option would be beneficial.
Any write operation (meaning reading mail) anyhow flushes the DB by necessity.
(we need to close to commit tag changes, which changing the read state is)
Previously the workers returned a mixture of decoded / encoded parts.
This lead to a whole bunch of issues.
This commit changes the msgviewer and the commands to assume parts to already
be decoded
Me again,
this time fixing encoding of subjects and attachments. It was problem in
IMAP backend. While other backends user MessageInfo() function which
generates MessageInfo decoded via go-message methodes, IMAP worker is
creating MessageInfo directly, so all non-utf8 subjects and filenames
were in raw form.
This patch fixes it. Not sure if we should care about errors (if
DecodeHeader fails it returns raw string back).
>From what I see, this should solve all encoding problem (tested only
IMAP). So, now I can focus on features. ;-)
Have a great weekend!
Leszek
When deleting a message, sometimes FetchDirectoryContents will fire.
FetchDirectoryContents will return a smaller set of UIDs since messages
have been deleted. This operation races with fetching from the seqMap in
client.ExpungeUpdate. Only recreate the seqMap if it can grow so that
messages will continue to be expunged.
Signed-off-by: Kevin Kuehler <keur@xcf.berkeley.edu>
Right now notmuch panics if something goes wrong in the configure event.
This patch checks for that and returns an error instead, so that we can at least
get the UI up and running (and all the other accounts)
The experience will be completely degraded until another configure event occurs.
Configure an oauthbearer source without a token_endpoint
parameter would panic due to nil pointer dereference
Example
source=imaps+oauthbearer://frode.aa%40gmail.com@imap.gmail.com:993
source-cred-cmd=pass oatuh2 frode.aa@gmail.com
token_endpoint is not required as it will use the provided
password as access_token when it is not set
This changes the search flags for maildir and imap backends.
They now no longer use -t for searching all text. This seems to make
more sense as being the targeted recipient. I have similarly added Cc
for -c. The text search now resides under -a for all text.
There is a command and config option. The criteria are a list of the
sort criterion and each can be individually reversed.
This only includes support for sorting in the maildir backend currently.
The other backends are not supported in this patch.
This populates the directory info model properly when requested,
allowing the fields to be relied upon elsewhere.
This also sends the dirinfo when new messages come in.
Basic searching is supported with the following:
- read messages
- unread messages
- from addresses
- text in body
- text in subject
- text in all
The implementation loops through all messages in the selected directory.
It tries to be smart by detecting which parts of each message the search
query needs to use and only loads these from the filesystem.
For some reason the current code frequently segfaults due to an
invalid C memory address. This commit mediates that by never keeping an object
alive longer than absolutely necessary.
Notmuch only allows a single write connection, all other clients trying to
modify the db block. Hence we should only open one when we actually need it.
Apparently we also need to refresh the RO DB connection upon modification,
else we get stale message tag results
This commit introduces the notmuch backend.
The backend is conditionally compiled in if the "notmuch" tag is provided.
Most of the message types are implemented, with the notable exceptions
of DeleteMessages as well as any copy / move / append type.
Reason being, that those aren't normally applicable in a notmuch based workflow.
Changes v2 --> v3, based on review comments
* Use account config for configuration
Things like FetchEntityPartReader etc can be reused by most workers
working with raw email files from disk (or any reader for that matter).
This patch extract that common functionality in a separate package.
This fixes ~sircmpwn/aerc2#245. This sets up the imap client to send
error messages to the logger of the worker. Errors now end up in the
bottom status line.
https://todo.sr.ht/~sircmpwn/aerc2/245
When a directory is opened, start watching its "new" subdirectory for
incoming messages using the fsnotify library. When creation events are
detected, run the Unseen routine to move the message from new to cur and
add new UIDs to the store, updating the UI's list of directory contents
as we go.
Email headers can be encoded with different charsets, which is signalled
using a special character sequence. The go-message package provides two
different methods for accessing header values, Get(key) (actually
inherited from the embedded textproto.Header) which returns the raw
header value and Text(key), which returns the header's value decoded as
UTF-8.
Before, in the maildir backend, we were using the Get method which
sometimes resulted in encoded headers being displayed in the UI. This
patch replaces the incorrect usage of Get() with Text().
Add the initial implementation of a backend for Maildir accounts. Much
of the functionality required is implemented in the go-message and
go-maildir libraries, so we use them as much as possible.
The maildir worker hooks into a new maildir:// URL scheme in the
accounts.conf file which points to a container of several maildir
directories. From there, the OpenDirectory, FetchDirectoryContents, etc
messages work on subdirectories. This is implemented as a Container
struct which handles mapping between the symbolic email folder names and
UIDs to the concrete directories and file names.
imaps+oauthbearer://user:token@host?token_endpoint=...
- the config Source password is used as access token if
no token_endpoint parameter is set
- the config Source password is used as refresh token if
token_endpoint parameter is set, and used to exchange
with an access token
The implementation has only been tested with Gmail.
source = imaps+oauthbearer://{username}:{refersh_token}@imap.gmail.com:993? \
client_id=XX&\
client_secret=XX&\
token_endpoint=https%3A%2F%2Faccounts.google.com%2Fo%2Foauth2%2Ftoken
client credentials created with
https://console.developers.google.com/apis/credentials
refresh token created with
https://github.com/google/gmail-oauth2-tools/blob/master/python/oauth2.py
rel: https://todo.sr.ht/~sircmpwn/aerc2/42
Before, we were using several IMAP-specific concepts to represent
information being displayed in the UI. Factor these structures out of
the IMAP package to make it easier for other backends to provide the
required information.
A sequence-set is an IMAP-specific implementation detail. Throughout the
UI, aerc simply operates using lists of opaque identifiers. In order to
loosen the coupling between the UI and IMAP in particular, replace most
usages of imap.SeqSet with []uint32, leaving the translation to a SeqSet
to the IMAP backend as needed.
Before, the information needed to display different parts of the UI was
tightly coupled to the specific messages being sent back and forth to
the backend worker. Separating out a models package allows us to be more
specific about exactly what a backend is able to and required to
provide for the UI.
Adds an archive command that moves the current message into the folder
specified in the account config entry.
Supports three layouts at this point:
- flat: puts all messages next to each other
- year: creates a folder per year
- month: same as above, plus folders per month
This also adds a "-p" argument to "cp" and "mv" that works like
"--parents" on mkdir(1). We use this to auto-create the directories
for the archive layout.
If a MailboxInfo has the attribute \Noselect,
it is not possible to use this name as a selectable mailbox.
Therefore it should not be passed to the directory handlers.
The issue pops up if one has a hierarchy like this:
INBOX
INBOX/lists/stuff
INBOX/lists/otherStuff
Even though lists is not a valid inbox (doesn't contain mail, only other maildirs)
it will show up in the directory listing, when we iterate over the MailboxInfo.
It does have the corresponding attribute set though and we can simply filter it out.
Worker.callbacks contains two types of callbacks: some are action callbacks,
some are message callbacks. Each of those is access from one side of the
communication channel (UI goroutine vs. worker goroutine).
Instead of using a channel, we can use two different maps for each kind. This
simplifies the code and also ensures we don't call an action callback instead
of a message callback (or the other way around).
Message IDs are allocated for both messages from UI to workers and the other
way around. Hence, the global nextId variable is accessed from multiple
goroutines.
Instead, use atomic to access the global counter.
Robin Jarry
Nguyễn Gia Phong
Koni Marti
Kalyan Sriram
y0ast
Reto Brunner
Jeff Martin
ARaspiK
Tero Koskinen
Tobias Wölfel
Andrew Jeffery
Lyudmil Angelov
Srivathsan Murali
elumbella
Grégoire Delattre
Ben Fiedler
Amir Yalon
Drew DeVault
Simon Ser
Galen Abell
Timmy Douglas
Leszek Cimała
Kevin Kuehler
Matt Snider
Frode Aannevik
Wagner Riffel
Ben Burwell
Jelle Besseling
Nicolai Dagestad
Daniel Bridges
Robert Günzler
Yash Srivastav