Solr re-index?

dnwk

Is there a way to ask Solr reindex everything? I am running my mailcow on a KVM and have to restore server from previous snapshot. In SOGO, emails won’t show up but number of mail is showing up. Using IMAP clients, I could view newer incoming mail. In SOGo, nothing is showing up.

pkernstock

That should help: https://mailcow.github.io/mailcow-dockerized-docs/u_e-dovecot-fts/

dnwk

Got this error message out of Solr docker. Not sure what it means

= null, domain = solr.core.dovecot-fts, service url = null, agent id = null] for registry solr.core.dovecot-fts / com.codahale.metrics.MetricRegistry@7a190256
2020-11-16 23:33:35.627 ERROR (coreContainerWorkExecutor-2-thread-1) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup
org.apache.solr.common.SolrException: Unable to create core [dovecot-fts]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1208) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:699) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ~[metrics-core-3.2.6.jar:3.2.6]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [solr-solrj-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:34]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1048) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1187) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
… 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2137) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2257) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1106) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:993) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1187) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
… 7 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1714906210 vs expected header=1071082519 (resource=BufferedChecksumIndexInput(MMapIndexInput(path=“/opt/solr/server/solr/dovecot-fts/data/index/7k0.si”)))
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:95) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:361) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:845) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:124) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:97) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2098) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2257) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1106) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:993) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1187) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
… 7 more
Suppressed: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=795438691 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path=“/opt/solr/server/solr/dovecot-fts/data/index/7k0.si”)))
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:266) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:361) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:845) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:124) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:97) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2098) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2257) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1106) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:993) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1187) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:699) ~[solr-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:32]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ~[metrics-core-3.2.6.jar:3.2.6]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [solr-solrj-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:37:34]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]

dnwk

And this error message when I run actual reindex

doveadm(@.com): Error: fts_solr: Indexing failed: 503 Service Unavailable
doveadm(@.com): Error: Mailbox Gmail: Mail search failed: Internal error occurred. Refer to server log for more information. [2020-11-16 15:59:45]
doveadm(@.com): Error: Mailbox Gmail: Transaction commit failed: FTS transaction commit failed: backend deinit
doveadm(@.com): Error: Mailbox INBOX: UID=2193: read(/var/vmail/.com//Maildir/cur/1604233466.M601152P28762.mail..com,S=4681,W=4790:2,S) failed: Cached message size larger than expected (4681 > 3192, box=INBOX, UID=2193) (read reason=mail stream)
doveadm(@.com): Error: Corrupted record in index cache file /var/vmail/.com//Maildir/dovecot.index.cache: UID 2193: Broken physical size in mailbox INBOX: read(/var/vmail/.com//Maildir/cur/1604233466.M601152P28762.mail..com,S=4681,W=4790:2,S) failed: Cached message size larger than expected (4681 > 3192, box=INBOX, UID=2193)
doveadm(@.com): Error: Mailbox INBOX: UID=2193: read(/var/vmail/.com//Maildir/cur/1604233466.M601152P28762.mail..com,S=4681,W=4790:2,S) failed: Cached message size larger than expected (4681 > 3192, box=INBOX, UID=2193) (read reason=)
doveadm(@.com): Error: Mailbox INBOX: UID=2193: read(/var/vmail/.com//Maildir/cur/1604233466.M601152P28762.mail.ccfolder.com,S=4681,W=4790:2,S) failed: Cached message size larger than expected (4681 > 3192, box=INBOX, UID=2193) (read reason=fts indexing)
doveadm(@.com): Error: Failed to read mailbox INBOX mail UID=2193 stream: Mailbox INBOX: UID=2193: read(/var/vmail/.com/general/Maildir/cur/1604233466.M601152P28762.mail..com,S=4681,W=4790:2,S) failed: Cached message size larger than expected (4681 > 3192, box=INBOX, UID=2193) (read reason=fts indexing)
doveadm(@.com): Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer to server log for more information. [2020-11-16 16:04:42]
doveadm(@*.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: transaction context

diekuh

Did you forget to migrate the crypt-vol-1 stuff?

Looks like you have a mix of mail encrypted with a new key and some with an old key.

You could try to delete the whole vmail volume and restore vmail and crypt while Dovecot and SOGo are not running.

dnwk

I was restoring from KVM full disk snapshot. So when snapshot was taken, the machine was still running.

diekuh

Delete the Solr volume then and reindex after it was recreated. Delete it while mailcow is down.

dnwk

diekuh This is exactly what I did. After the reindex, IMAP seems to work but not SOGo. Because there seems to have a couple most recent email went corrupt in dovecat. When loading the most recently email list, SOGo has trouble loading the newest 20 emails and somehow refuse to work. If I keep scrolling down in SOGo, older messages eventually show up. Same for IMAP. Some clients are capable of handling error and could load messages but some IMAP clients refuse to work when error hits. Not sure what to do. I connect my Outlook to the email box and export all the good emails. Delete the account and recreate it. Import back all the mails via Outlook. Not an ideal way of restoring it. I guess I need to take a look at proper server backup.

umirzak

dnwk how u resolved the issue, i have similar

esackbauer

umirzak Did you read the second posting with the Link to the documentation?

umirzak

esackbauer i have enabled and changed mailconf, Solr is disabled now and I am unable to send emails to outside of organization.

esackbauer

https://docs.mailcow.email/troubleshooting/debug/

umirzak

thanks resolved