I've now configured a dupe area, but the point shouldn't bee seeing dupes as the message bas was started fresh with no messages in it.

Ah, I see. Yeah dupe detection needs some work. at present it is supposed to check a history file with a crc of the msgid. That doesn't work so well when messages don't have msgids. (That's not the problem here, I don't know why it would say it found a dupe when it hasn't)

if anymore pop up and go into your dupe base let me know and i'll get you to send me the base and the dupe history file and see if i can figure it out.

Could you also please add pressing enter to quote along with the space key?

Yeah, that shouldn't be too hard.

I've just done a git clone, and compile.. If this one goes out it's working...

:)

Andrew

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Hi Apam,

I've now configured a dupe area, but the point shouldn't bee
seeing dupes as the message base was started fresh with no
messages in it.

Ah, I see. Yeah dupe detection needs some work. at present it is
supposed to check a history file with a crc of the msgid. That doesn't
work so well when messages don't have msgids. (That's not the problem
here, I don't know why it would say it found a dupe when it hasn't)

I just did a test.. Deleted the message base files, and the dupehist.dat
file. Then re tossed the last three days worth of packets. Postie
reported lots of dupes.

Just wondering if this is a linux m68k thing or a general postie bug?

I also noticed that when listing messages with the index reader, there
were replies in the list before the message being replied to.



\/orlon
aka
Stephen


--- Talisman v0.40-dev (Linux/m68k)
* Origin: Vorlon Empire: Sector 550 (21:1/195.1)

I also noticed that when listing messages with the index reader, there were replies in the list before the message being replied to.

Yeah, thats due to the order that they're tossed, they're not sorted by date

Andrew

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Just wondering if this is a linux m68k thing or a general postie bug?

I think it might be a general postie bug. I've made a new branch of postie with new dupe checking, branch is called: "newdupechecking"

this one will record 3 CRCs per message, 1, msg id, 2, ctrl content and 3, msgbody.

if all three match it will be considered a dupe. (note it doesn't check seenby's and path, as they might change and still be a dupe)

if you can try this with your m68k installation that would be good. I don't get enough messages to make it break.

Also, now messages without msgids will be dupe checked, previously they werent.

Andrew

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

I think it might be a general postie bug. I've made a new branch of
postie with new dupe checking, branch is called: "newdupechecking"

I just merged this into main. I tested it on m68k and all the messages i was able to pull from 21:1/999 worked. If it's still not working can you zip up your packets and send them to me.. if you don't have my email address, can you upload it to the sysop at positronicbrain.net:11892 (telnet)

Andrew

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

apam wrote to vorlon <=-

Also, now messages without msgids will be dupe checked, previously they werent.

I get quite a few dupes in COOKING from one person... not sure how or why.
If you compile a windows version and need me to test it feel free to send
it over.

Shawn

... Mary had a little RAM -- only about a MEG or so.

--- Talisman v0.40-dev (Windows/x86)
* Origin: Tiny's BBS II - tinysbbs.com:4323/ssh:4322 (21:1/222)

HI Apam,

I just merged this into main. I tested it on m68k and all the messages
i was able to pull from 21:1/999 worked. If it's still not working can
you zip up your packets and send them to me.. if you don't have my
email address, can you upload it to the sysop at
positronicbrain.net:11892 (telnet)

I've just done a git clone, will test and report back latter tonight.



\/orlon
aka
Stephen


--- Talisman v0.41-dev (Linux/m68k)
* Origin: Vorlon Empire: Sector 550 (21:1/195.1)

I've just done a git clone, will test and report back latter tonight.

Thanks, I appreciate it. I'm going to do a release of 0.41 as there are a few other (unrelated) bugs squashed, and I don't think the dupe checking will be any worse if not better.

I was wondering, maybe the messages are actually having the same message-id? And so getting treated as dupes by postie, but not by whatever is infront of postie.

I noticed when trawling through my dupes base on happyland that there were a lot of (mostly generated) messages with duplicate message ids. Stuff like BBS advertisements and stuff.

My theory was that many msgids are generated by unix time, some are poorly implemented and may shoot of several different messages in a second, which some other more robust dupe checkers would notice because they work off the whole message not just message id, but postie did not.

Anyway, hopefully this new method will work better - and if it doesn't we can have a closer look at the messages and work it out.

Andrew

--- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Re: Re: dupe checking
By: apam to vorlon on Sat Jun 25 2022 05:50 pm

My theory was that many msgids are generated by unix time, some are poorly implemented and may shoot of several different messages in a second, which some other more robust dupe checkers would notice because they work off the whole message not just message id, but postie did not.

I have a rant about that very subject here: https://wiki.synchro.net/faq:misc#ftn_msgid

The FTN message-ID was just badly conceived and documented. Especially considering there was already a better concept well-documented in IETF's RFC-822.
--
digital man (rob)

This Is Spinal Tap quote #27:
As long as there's, y'know, sex and drugs, I can do without the rock and roll. Norco, CA WX: 76.3øF, 54.0% humidity, 12 mph S wind, 0.00 inches rain/24hrs
--- SBBSecho 3.15-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (21:1/183)