open https://gitlab.synchro.net/main/sbbs/-/issues/786

Running chksmb on dove-syncops I get the following message

```
Checking dovenet/dove-syncops Headers

93% #14922 (3DA220) SYS64738 Header field contains control characters
100%
```

After running fixsmb

```
31% #11465 (152120) Charles Blackburn
(152420) smb_getmsghdr returned -103:
smb_getmsghdr corrupt message header ID (52 54 2F 46) at offset 1385504
68% #13511 (2D2620) Charles Blackburn
(2D2920) smb_getmsghdr returned -103:
smb_getmsghdr corrupt message header ID (54 48 45 46) at offset 2959648
70% #13615 (2E5020) Ebojager
(2E5320) smb_getmsghdr returned -103:
smb_getmsghdr corrupt message header ID (2F 4F 55 54) at offset 3035936
79% #14159 (34AB20) kk4qbn
(34AE20) smb_getmsghdr returned -103:
smb_getmsghdr corrupt message header ID (00 02 00 04) at offset 3452448
89% #14716 (3B2E20) Amessyroom
(3B3120) smb_getmsghdr returned -103:
smb_getmsghdr corrupt message header ID (00 00 00 00) at offset 3879200
91% #14854 (3CD420) Amessyroom
(3CD720) smb_getmsghdr returned -103:
smb_getmsghdr corrupt message header ID (04 00 00 00) at offset 3987232
100%
Sorting index...
Re-writing index...
```

If I rerun chksmb then I still get the same issue. Can we have fixsmb strip out invalid control characters from the header, or better yet, don't let them get there in the first place when importing the message?
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5590

I doubt the control characters were in the header field(s) as a part of import. But if they were, that'd be a bug.

Instead, more likely, the control characters are a result of some other corruption to the message base that occurred *after* the message was imported. Running chksmb with the '-e' option and reporting the results would help me to know.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5591

As per the request "Can we have fixsmb strip out invalid control characters from the header" - it could, but that's likely only a symptom of the problem. Looking at the header fields of the message(s) in question would be a good idea to determine if the header is even recoverable. Simply stripping control character isn't likely what we actually want to do in this case.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5592

As requested

```
Checking dove-syncops Headers

93% #14922 (3DA220) SYS64738 Header field contains control characters

message base dove-syncops
To Digital Man
Subject Re: Mail Server Issue
Sender SYS64738
SenderNetAddr REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
@TZ: C168

IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG

SenderNetType Internet
Message-ID <66A9C774.764.998internetdiscussi@bbs.tangonine.net> X-FTN-MSGID 55877.sync_sys@1:103/705 2b102c17
X-FTN-PID Synchronet 3.20a-Linux master/d3c114993 Feb 1 2024 GCC 12.2.0 X-FTN-CHRS CP437 2
OtherHeader WhenImported: 20240731041114-0700 c1e0
OtherHeader WhenExported: 20240731041558-0700 c1e0
OtherHeader ExportedFrom: VERT sync_sys 55877
OtherHeader @REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
OtherHeader @TZ: C168
OtherHeader > IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I
OtherHeader > SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG OtherHeader WhenImported: 20240731141116+0900 121c
OtherHeader WhenExported: 20240731201115+0900 121c
OtherHeader ExportedFrom: TANGO99 998internetdiscussi 764
OtherHeader @REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
OtherHeader @TZ: C168
OtherHeader > IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I
OtherHeader > SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG SenderNetAddr REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
@TZ: C168

IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG

SenderNetType QWKnet
when_written 66A983A8 0000 Tue Jul 30 19:22:00 2024 UTC
when_imported 66AA1CFB C168 Wed Jul 31 06:16:11 2024 CDT
type 0000h
version 0300h
attr 0000h ()
auxattr 00000000h ()
netattr 00000000h ()
header_offset 3DA220h
header_fields 25
header_length 1200 (calc: 1200)
number 14922
thread_id 14922
data_offset 48B600h
data_field[0] TEXT_BODY, offset 0, length 67

100%

Checking dove-syncops Index

100%

Checking dove-syncops Hashes

100%

Status Total (=): 5165
Total Indexes (=): 5165
Active Indexes (=): 5165
Active Headers (=): 5165
Active Header Blocks ( ): 15514 3,971,584 bytes used Active Data Blocks ( ): 18174 4,652,544 bytes used Header Records ( ): 5636
Deleted Indexes ( ): 0
Deleted Headers ( ): 471
Deleted Header Blocks ( ): 1446 370,176 bytes used Deleted Data Blocks ( ): 1755 449,280 bytes used Oldest Message (import) ( ): 965 days (0 max)
Largest Message (data) ( ): 72125 bytes (#13716)
Control Characters in Header Fields (!): 1

dove-syncops Message Base has Errors!

Total Deleted Messages : 471 819,456 bytes used
```
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5593

That message is well corrupt

{width=918 height=709}
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5594

I recommend deleting the message and then let me know if this happens again (is reproducible).
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5599

Now it seems even more screwed up.

```
92% #14922 (3DA220) SYS64738 Header field contains control characters
```

So if I use smbutil to try and find 14922

```
4756/#14921 Errol Casey phigan Cannot see all message boards
4757/#14923 Denn Errol Casey Cannot see all message boards
4758/#14924 Errol Casey Denn Cannot see all message boards
4759/#14925 SYS64738
```

It's not there.

So let's run fixsmb

```
smb_getmsghdr corrupt message header ID (04 00 00 00) at offset 3987232
92% #14922 (3DA220) SYS64738 Not indexing deleted message
```

Says it's deleted. So let's make sure

```
bbs@bbs:/sbbs/data/subs/dovenet$ smbutil mp1000 dove-syncops.shd

SMBUTIL v3.20-Linux master/d232ecdc6 SMBLIB 3.00 - Synchronet Message Base Utility

Opening dove-syncops
Maintaining dove-syncops
Maintaining dove-syncops hash file
Loading index...

Done.

Scanning for pre-flagged messages...
100% (0 pre-flagged for deletion)
Scanning for read messages to be killed...
100% (0 flagged for deletion due to read status)
dove-syncops locked successfully
Packing dove-syncops
Analyzing dove-syncops
820992 less than 1024000 compressible bytes.

dove-syncops unlocked successfully
```

Nothing flagged for deletions so let's double check with another chksmb

```
Checking dove-syncops Headers

92% #14922 (3DA220) SYS64738 Header field contains control characters
```

Still there. I don't seem to be able to get rid of it.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5600

Use 'smbutil p' (pack) the message base to remove deleted messages.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5601

Might need to use 'smbutil -a p' to force the pack.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5602

Yes, the -a p did it. Thank you.
I'll maybe run an smbchk every hour and have it flag me if there's corruption. Try and catch it quick.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5603

I'll make a change to chksmb to *not* check header fields of deleted messages for ctrl chars
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5604

Cool, while you're diddling with smb stuff, could you have smbutil l show the type of message if it's not a netmail, for example and UPVOTE or DOWNVOTE instead of who it's too. The blanks confused me for a bit. Also a date would be good, either date written or date imported.

I can open a separate issue for that.

{width=960 height=197}

{width=804 height=252}
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5608

I think we can close this issue now.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5785

chksmb -e is no longer displaying extended information.


```
$ chksmb dove-techtalk -e -p -b

CHKSMB v3.20-Linux master/d84b8e3d6 SMBLIB 3.00 - Check Synchronet Message/File Base

Checking dove-techtalk Headers

35% #86 (02F120) TLDR Header field contains control characters
100%

Checking dove-techtalk Index

100%

Checking dove-techtalk Hashes

100%

Status Total (=): 230
Total Indexes (=): 230
Active Indexes (=): 230
Active Headers (=): 230
Active Header Blocks ( ): 2104 538,624 bytes used Active Data Blocks ( ): 66765 17,091,840 bytes used Header Records ( ): 230
Deleted Indexes ( ): 0
Deleted Headers ( ): 0
Deleted Header Blocks ( ): 0 0 bytes used Deleted Data Blocks ( ): 0 0 bytes used Oldest Message (import) ( ): 249 days (0 max)
Largest Message (data) ( ): 6308005 bytes (#88)
Control Characters in Header Fields (!): 1

dove-techtalk Message Base has Errors!

'fixsmb' can be used to repair many message/file base problems.
```
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5786

@rswindell prod?
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5798

I'm not sure what you're asking here. The '-e' option never had any impact on the output of the detected header fields with control characters. "no longer" does not apply here. Are you asking for chksmb to report more information for those particular detected msg header problems when using the '-e' option?
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/786#note_5800

Okay, I think I see now what you're reporting: any/all problematic message headers are indeed dumped when using the '-e' option (even those with unexpected ctrl chars in them), but you have to specify the '-e' option *before* the message base(s) on the chksmb command-line.
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)