• 1.4b notes.ini and config document.

    By Dan Cihon 2 decades ago

    I have started using 1.4b and it seems that something has changed. Allot of spam is getting through and getting into the good mail database. Also it looks like the settings in the config document doesn't match the notes.ini settings. How does the config document work? I will attach what the notes.ini shows and the config document has.

    Notes.ini :

    KS_BAYESIAN_FILTER=1

    KS_SINGLESERVER=1

    KS_BAYESIAN_MARK=1

    KS_BAYESIAN_PREP_2=0

    KS_BAYESIAN_PREP=0

    KS_BL_DUMPLISTS=1

    KS_BL_PERIOD=180

    KS_STATS=1

    KS_DEBUG=1

    KS_BAYESIAN_BOUNDARY=90

    KS_BAYESIAN_RATIO=2

    KS_MIN_FROM_LENGTH=4

    KS_MARK=1

    ;KS_COPIED_DB=copied.nsf

    KS_RELOAD=1

    KS_DEFAULTACTION=3

    KS_BL_PROB=1

    KS_FILTER_FROM_INT=1

    KS_MAX_FROM_INTS=7

    ————————————————————————————–

    Config Document:



    Instance ID

    This is a string of six characters or less, should be the same for every server in your organisation (KS_IID). Dccath

    Default action

    This is the default action to be taken when one of the hard coded rules is matched (KS_DEFAULTACTION). 0 - Accept

    Default probability increase

    The default probability increase, only used if increase probability is selected for the default action (KS_DEFAULT_PROB_INC). 0%

    Mark messages with a reason field?

    Add KS_REASON item to an email if a rule is matched (KS_MARK). Yes

    Reload configuration every hour?

    (KS_RELOAD). Yes

    Show statistics?

    Log statistics under smtp.kSpam.* (KS_STATS). Yes

    Minimum "From:" header length?

    Minimum length of the From: header (KS_MIN_FROM_LENGTH). 1

    Maximum numbers in sender's username?

    Maximum number of integers in the sender's username (KS_MAX_FROM_INTS). 7

    First character in "From:" header must not be a number?

    (KS_FILTER_FROM_INT). Yes

    Other forms to scan?

    Forms other than Memo and Reply delimited by commas (KS_INTERESTING_FORMS).

    Add recipients list to copied and denied messages?

    Add KS_RECIPIENTS readers field to denied messages, username in email address must me included in the recipients username field in their person document. ( KS_RECIPIENTS). Yes

    Copied mail database

    Database to copy copied messages to, default is mailspam.nsf (KS_COPIED_DB).

    Turn on debugging?

    Create log in ks_debug.txt (KS_DEBUG). No

    ———————————————————————————————–

    Bayesian filter enabled

    Enable the Bayesian filter (KS_BAYESIAN_FILTER). Yes

    Token reload period

    Period of time between recalculating probabilities (KS_BL_PERIOD). 180

    Probability boundary

    Boundary probability at which email is considered spam (KS_BAYESIAN_BOUNDARY). 90%

    Mark messages with token list and probability?

    Add KS_BL_PROB and KS_BL_TOKENS to incoming emails (KS_BAYESIAN_MARK). Yes

    Good message ratio

    Ratio of good emails passing through the server before an email is copied to the good mail database (KS_BAYESIAN_RATIO). 10

    Bayesian action

    Action to take when an emails are probably spam (KS_BAYESIAN_ACTION). 3 - Copy & Deny

    Mark with

    Text to mark emails with is the default Bayesian action is mark. (KS_BAYESIAN_ACTION_MARK_WITH). [Possible Spam]

    Tokens to ignore

    Tokens to ignore when calculating probabilities (KS_BL_IGNORE).

    Dump token lists to file

    Write token lists to files goodlist.txt and spamlist.txt (KS_BL_DUMPLISTS). Yes

    Preparation setting 1

    All emails that pass all rules without being matched are placed in the mailgood.nsf database (KS_BAYESIAN_PREP). No

    Preparation setting 2

    All emails with a probability greater than 90% are copied to mailspam.nsf, all emails with a probability of less than 10% are copied to mailgood.nsf (KS_BAYESIAN_PREP_2). Yes

    Turn on debugging?

    Create bload.txt log file (KS_BL_DEBUG). No

    ————————————————————————————————————————

    I just noticed that the PREP_2 entry is set to Yes but look at the notes.ini it has 0



    I went back and looked at the config document. Now I am getting more confused. In the config document it says this for the PREP_2 variable:

    Copy any incoming message with a probability above 90% to mailspam.nsf and any with a probability of less that 10% to mailgood.nsf. Use after KS_BAYESIAN_PREP.



    So does this mean that if I don't set this to yes mail won't get copied?



    I will change prep_2 to no but still let me know which it is using notes.ini or the config document.

    ————————————————————————————————————————————

    I noticed one other thing. My messages only have the KS_IID varible in them now. Why did I lose everythig else?



    Thanks

    This is a great tool. I appreciate the work you have done.

    • The config doc...

      By Tom Lyne 2 decades ago

      … overrides the settings in the notes.ini file. You should turn off the Bayesian Prep_2 setting.



      -tom

      • config doc

        By Dan Cihon 2 decades ago

        Tom,

        I decided to start over fresh.<br/>

        Here is what I have done.

        Wiped out both mailspam and mailgood databases.

        Applied the new templates.

        Created a config file and didn't change any of the default settings.

        Right now there are not .txt files in the domino directory. I thought they get created when bload starts.

        Here are my settings and config file information.

        I will wait to hear from you on what to change setting wise.

        What data you want me to collect.

        How the settings should be adjusted as we go.

        From looking at all of the forum responses it looks like there are many different ways to adjust settings.

        My main goal is obviously to eliminate the spam.

        I would also like to do this with as little management of it as possible. ( Like having to move messages all over the place.)

        Things were working pretty well until I installed 1.4b. But I expected to have some glitches.

        Ok here is the info.



        Bayesian Tab



        Bayesian filter enabled

        Enable the Bayesian filter (KS_BAYESIAN_FILTER). No

        Token reload period

        Period of time between recalculating probabilities (KS_BL_PERIOD). 360

        Probability boundary

        Boundary probability at which email is considered spam (KS_BAYESIAN_BOUNDARY). 90%

        Mark messages with token list and probability?

        Add KS_BL_PROB and KS_BL_TOKENS to incoming emails (KS_BAYESIAN_MARK). Yes

        Good message ratio

        Ratio of good emails passing through the server before an email is copied to the good mail database (KS_BAYESIAN_RATIO). 10

        Bayesian action

        Action to take when an emails are probably spam (KS_BAYESIAN_ACTION). 3 - Copy & Deny

        Mark with

        Text to mark emails with is the default Bayesian action is mark. (KS_BAYESIAN_ACTION_MARK_WITH).

        Tokens to ignore

        Tokens to ignore when calculating probabilities (KS_BL_IGNORE).

        Dump token lists to file

        Write token lists to files goodlist.txt and spamlist.txt (KS_BL_DUMPLISTS). No

        Preparation setting 1

        All emails that pass all rules without being matched are placed in the mailgood.nsf database (KS_BAYESIAN_PREP). No

        Preparation setting 2

        All emails with a probability greater than 90% are copied to mailspam.nsf, all emails with a probability of less than 10% are copied to mailgood.nsf (KS_BAYESIAN_PREP_2). No

        Turn on debugging?

        Create bload.txt log file (KS_BL_DEBUG). No



        General Tab



        Instance ID

        This is a string of six characters or less, should be the same for every server in your organisation (KS_IID). Dccath

        Default action

        This is the default action to be taken when one of the hard coded rules is matched (KS_DEFAULTACTION). 0 - Accept

        Default probability increase

        The default probability increase, only used if increase probability is selected for the default action (KS_DEFAULT_PROB_INC). 0%

        Mark messages with a reason field?

        Add KS_REASON item to an email if a rule is matched (KS_MARK). Yes

        Reload configuration every hour?

        (KS_RELOAD). Yes

        Show statistics?

        Log statistics under smtp.kSpam. (KS_STATS). Yes

        Minimum "From:" header length?

        Minimum length of the From: header (KS_MIN_FROM_LENGTH). 0

        Maximum numbers in sender's username?

        Maximum number of integers in the sender's username (KS_MAX_FROM_INTS). 0

        First character in "From:" header must not be a number?

        (KS_FILTER_FROM_INT). No

        Other forms to scan?

        Forms other than Memo and Reply delimited by commas (KS_INTERESTING_FORMS).

        Add recipients list to copied and denied messages?

        Add KS_RECIPIENTS readers field to denied messages, username in email address must me included in the recipients username field in their person document. ( KS_RECIPIENTS). No

        Copied mail database

        Database to copy copied messages to, default is mailspam.nsf (KS_COPIED_DB).

        Turn on debugging?

        Create log in ks_debug.txt (KS_DEBUG). No



        show stat smtp.kspam.

        SMTP.kSpam.Bayesian.Action = Copy & Deny

        SMTP.kSpam.Bayesian.Boundary = 90

        SMTP.kSpam.Bayesian.Filter = Off

        SMTP.kSpam.Bayesian.Mark = On

        SMTP.kSpam.Bayesian.Prep = Off

        SMTP.kSpam.Bayesian.Prep_2 = Off

        SMTP.kSpam.Bayesian.Ratio = 10

        SMTP.kSpam.Conf.AddRecipientsItem = Off

        SMTP.kSpam.Conf.Debug = Off

        SMTP.kSpam.Conf.DefaultAction = Accept

        SMTP.kSpam.Conf.DefaultProbIncrease = 0

        SMTP.kSpam.Conf.FilterFromInt = Off

        SMTP.kSpam.Conf.IID = Dccath

        SMTP.kSpam.Conf.Mark = On

        SMTP.kSpam.Conf.MaxNumbersInFrom = 0

        SMTP.kSpam.Conf.MinFromLength = 0

        SMTP.kSpam.Conf.Reload = On

        SMTP.kSpam.Version = 1.40 beta 0

        18 statistics found



        One other thing I left my config rules still in effect. So some mail my be getting deleted from those rules.

        After I submit this I am going to turn debug on on both tabs.



        Update on stats as of 9:39 am 12/27/2004

        SMTP.kSpam.Bayesian.Action = Copy & Deny

        SMTP.kSpam.Bayesian.Boundary = 90

        SMTP.kSpam.Bayesian.Filter = On

        SMTP.kSpam.Bayesian.Mark = On

        SMTP.kSpam.Bayesian.Prep = Off

        SMTP.kSpam.Bayesian.Prep_2 = Off

        SMTP.kSpam.Bayesian.Ratio = 10

        SMTP.kSpam.Conf.AddRecipientsItem = Off

        SMTP.kSpam.Conf.Debug = On

        SMTP.kSpam.Conf.DefaultAction = Accept

        SMTP.kSpam.Conf.DefaultProbIncrease = 0

        SMTP.kSpam.Conf.FilterFromInt = Off

        SMTP.kSpam.Conf.IID = Dccath

        SMTP.kSpam.Conf.Mark = On

        SMTP.kSpam.Conf.MaxNumbersInFrom = 0

        SMTP.kSpam.Conf.MinFromLength = 0

        SMTP.kSpam.Conf.Reload = On

        SMTP.kSpam.Denied = 3

        SMTP.kSpam.LastMinorError = 12/27/2004 09:02:47 AM: ERROR 5.2.1: kSpam.bload calculating probabilties or not loaded.

        SMTP.kSpam.Rule.90 = 1

        SMTP.kSpam.Rule.91 = 1

        SMTP.kSpam.Rule.92 = 1

        SMTP.kSpam.Version = 1.40 beta 0

        23 statistics found



        What does Error 5.2.1 indicate?

        • By Tom Lyne 2 decades ago

        • Re:

          By Tom Lyne 2 decades ago

          If you have wiped all the emails from mailspam.nsf and mailgood.nsf then bload won't have anything to work with so the Bayesian filter won't work at all. There needs emails in both mailspam and mailgood for it to work at all.



          -tom

          • Config doc

            By Dan Cihon 2 decades ago

            OK I will put some emails in. I believe you said about a 2 to 1 ratio of spam to good and about 200 emails in spam and 100 in good. Should that be enough. Is that why I am getting the 5.2.1 error and no .txt files except the log.txt file.

            Thanks

            Dan

            • Re:

              By Tom Lyne 2 decades ago

              Yes, that's why there are no .txt files and the 5.2.1 error.



              -tom

              • config doc

                By Dan Cihon 2 decades ago

                Tom,

                I'm not sure but I don't think it created the .txt files until I put the DUMPLIST variable in the .ini file. I changed it to yes in the config file, restarted notes but it didn't create the files. Also some of the switches in the config file will let you check yes and no. Which one counts if you check both?<br/>

                Here are my latest stats:

                show stat smtp.kspam.*

                SMTP.kSpam.Bayesian.Action = Copy & Deny

                SMTP.kSpam.Bayesian.Boundary = 90

                SMTP.kSpam.Bayesian.Filter = On

                SMTP.kSpam.Bayesian.Mark = On

                SMTP.kSpam.Bayesian.Prep = Off

                SMTP.kSpam.Bayesian.Prep_2 = Off

                SMTP.kSpam.Bayesian.Ratio = 10

                SMTP.kSpam.Conf.AddRecipientsItem = Off

                SMTP.kSpam.Conf.Debug = On

                SMTP.kSpam.Conf.DefaultAction = Accept

                SMTP.kSpam.Conf.DefaultProbIncrease = 0

                SMTP.kSpam.Conf.FilterFromInt = Off

                SMTP.kSpam.Conf.IID = Dccath

                SMTP.kSpam.Conf.Mark = On

                SMTP.kSpam.Conf.MaxNumbersInFrom = 0

                SMTP.kSpam.Conf.MinFromLength = 0

                SMTP.kSpam.Conf.Reload = On

                SMTP.kSpam.Version = 1.40 beta 0

                18 statistics found



                Is this how it should be set now?

                Should I just let it run for awhile now?

                Thanks

                Dan

                • Re:

                  By Tom Lyne 2 decades ago

                  Are there any errors in the KS_LOG.txt file?



                  Are you getting any denied (or denied and copied) emails because of the Bayesian filter?



                  -tom

                  • config doc

                    By Dan Cihon 2 decades ago

                    Here is the last entries in the log file. I am getting errors as shown. Right now allot of spam is getting through. Some spam is going to the mailgood file and little if any is going to mailspam.



                    Number of rules: 94

                    Reloading configuration…

                    freeing rule list

                    freed rule list

                    loading rule list…

                    Number of rules: 94

                    12/28/2004 10:05:48 PM - Form: Memo

                    *SUBJECT:Sit back, relax and get paid for your opinion*<br/>
*FROM:&quot;ITD Survey Co.&quot; &lt;r-12-8054570-390@ianagram.com&gt;*<br/>
*MAIL_FROM:sender-12-8054570-390@w4.floralegality.com*<br/>
*BODY LENGTH: 1541* <br/>
*Scanning message*<br/>

                    ERROR 5.2.1: kSpam.bload calculating probabilties or not loaded.

                    ERROR 5.1.1: Bayesian data not loaded. P:0.0000



                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    Number of rules: 94

                    • Re:

                      By Tom Lyne 2 decades ago

                      After letting it run for a while do you still get the 5.2.1 errors?



                      -tom

                      • config doc

                        By Dan Cihon 2 decades ago

                        ERROR 2.1.14: Empty buffer returned by NIFReadEntries.

                        01/01/2005 10:35:43 AM - Form: Memo

                        *SUBJECT:activity logs*<br/>
*FROM:&quot;Traci Cihon&quot; &lt;tcihon@columbus.rr.com&gt;*<br/>
*MAIL_FROM:tcihon@columbus.rr.com*<br/>
*BODY LENGTH: 1723* <br/>
*ATTACHMENT: dailyactivitylog12-28-04.doc*<br/>
*ATTACHMENT: weeklyactivitylog1-2-05.doc*<br/>
*ATTACHMENT: smokingchart.xls*<br/>
*Scanning message*<br/>

                        ERROR 5.2.1: kSpam.bload calculating probabilties or not loaded.

                        ERROR 5.1.1: Bayesian data not loaded. P:0.0000



                        Tom,

                          Any Ideas?<br/>



                        It looks like the 5.2.1 error happens when a new mail comes in.

                      • Looks like new mail is causing the error?

                        By Dan Cihon 2 decades ago

                        ERROR 2.1.14: Empty buffer returned by NIFReadEntries.

                        01/01/2005 10:35:43 AM - Form: Memo

                        SUBJECT:activity logs

                        FROM:"Traci Cihon"

                        MAIL_FROM:tcihon@columbus.rr.com

                        BODY LENGTH: 1723

                        ATTACHMENT: dailyactivitylog12-28-04.doc

                        ATTACHMENT: weeklyactivitylog1-2-05.doc

                        ATTACHMENT: smokingchart.xls

                        Scanning message

                        ERROR 5.2.1: kSpam.bload calculating probabilties or not loaded.

                        ERROR 5.1.1: Bayesian data not loaded. P:0.0000



                        Tom,

                        Any Ideas?



                        It looks like the 5.2.1 error happens when a new mail comes in.

                        • Re: config doc

                          By Tom Lyne 2 decades ago

                          The 2.1.14 error means kSpam couldn't find the config doc. The 5.2.1 error means that kSpam couldn't connect to the shared memory filled by nbload.exe. Are you using the latest version of nbload.exe, and are you giving nbload.exe enough time to finish before viewing the ks_log.txt file?



                          -tom

                          • config doc

                            By Dan Cihon 2 decades ago

                            Tom,

                            I'm bad. In trying to figure out what was working before, I deleted the config.doc and was only using the .ini file for the settings.

                            I probably am cheking too quickly also.

                            Thanks



                            Is there going to be another beta release coming soon?

                            Thanks

                            Dan

                            • Re:

                              By Tom Lyne 2 decades ago

                              I haven't got any plans for one anytime soon.



                              -tom