I-D ACTION:draft-klensin-email-envelope-00.txt

[John C Klensin]

The draft posted/described in the attached is a bizarre idea,
partially to see if it is possible to consider a radical
solution to an increasingly troublesome problem, and partially
to see if the supportive comments about "Email NG" in
Minneapolis were really serious.

I am not at all convinced that it is a _good_ idea, only that,
if we are talking about radical changes to the mail
infrastructure to support various extended services, this is the
sort of "clean up the warts that get in the way" option we might
want to consider.

And even if it were a good idea, some of the details are
probably not right -- if this looks like it received a day's
thought, you would probably be guessing much too high.

Discussion should probably go to the SMTP list; IMAA is copied
only because this could interact a bit with some of the "UTF-8
header" discussions.

john

[Hector Santos]

Hello John,

In general, I have much to say about any SMTP proposal that suggests SMTP
change at the software level. To me, a change in software opens the door for
other possible concepts that may address a particular needs in the email
industry. IMTV, acceptance and deployment aspects of the proposal are
important and major considerations of any SMTP change proposal. IMTV,
that's a matter of a having a strong functional specification dictation.

With that said, I am putting my "Software Engineering Hat" on as if I was
going to implement your ENVL proposal today to see what would be the
technical implementations issues. Based on this technical point of view,
here are my specific comments about your draft:

What does this ENVL proposal attempt to address? (What problem(s) does it
solve?)

Excuse me if I missed it in the draft, but I see only three (3) concepts
that addresses a "problem is solved:"

1) Proposal provides a cleaner SMTP envelope or proposal to create a cleaner
RFC 822 trace header?

2) Cleaner SMTP envelope minimizes mail relay issues.

3) Syntax Error checking.

Is this correct?

What is it about the headers today that makes it problematic?

You touch base with section 3, item 6 indicating "Syntax Error." Syntax
Error needs to be clarified, i.e, can (or may) this include an
implementation that attempts to perform some sort of sender/machine
validation?

What is considered the ENVL header?

The term "Trace Fields" are used. This needs to be clarified for the
software engineer who may not be in the same league as protocol police or
IETF gurus. It seems to indicate only the Received: and Return-Path: lines
are considered and are the ENVL transaction.

Lets ask this another way:

Does this proposal basically attempts to separate the transmission of a RFC
822 formatted message normally done at the DATA state into two separate
state transactions? For example:

C: ENVL
S: 250 Start Mail Header input; end with <CRLF>.<CRLF>
C: [sends header followed by "." line]
S: 250 Mail Header OK. Continue with Data Command

C: DATA
S: 354 Start Mail Body input; end with <CRLF>.<CRLF>
C: [send message body only (no header) followed by "." line]
S: 250 Message accepted for delivery.

or are we just adding a new state that allows a client to provide a server
with access to "DATA" header (all or specific) information using the ENVL
before the actual DATA transaction takes place?

C: ENVL
S: 250 Start Mail (Trace??) Header input; end with <CRLF>.<CRLF>
C: [sends mail header (all or just some specific headers??) followed by
"." line]
S: 250 Mail Header OK. Continue with Data Command

C: DATA
S: 354 Start Mail Header/Body input; end with <CRLF>.<CRLF>
C: [sends mail header again and body followed by "." line]
S: 250 Message accepted for delivery.

In short, an example for the "laymen" would be nice :-)

Anyway, I personally believe your ENVL is a good start. It has promise.
Off hand, without grasping yet the true intent this proposal (what does it
solve), what defines the ENVL "headers", etc, I believe the answer to this
will have a major factor on the acceptance and deployment aspect of the
proposal. Besides other impact considerations which I don't wish to muddy
the water just yet without getting the answer to the above, if this ENVL or
another other SMTP change proposal proves to be useful, I'll be among the
first to implement such a new SMTP feature like this in our SMTP product I
will be interested on providing an immediate "proof of concept" wide test
bed of customers.
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com






----- Original Message -----
From: "John C Klensin" <john-***@jck.com>
To: <ietf-***@imc.org>; <ietf-***@imc.org>
Sent: Friday, January 23, 2004 4:53 PM
Subject: FWD: I-D ACTION:draft-klensin-email-envelope-00.txt

Post by John C Klensin
The draft posted/described in the attached is a bizarre idea,
partially to see if it is possible to consider a radical
solution to an increasingly troublesome problem, and partially
to see if the supportive comments about "Email NG" in
Minneapolis were really serious.
I am not at all convinced that it is a _good_ idea, only that,
if we are talking about radical changes to the mail
infrastructure to support various extended services, this is the
sort of "clean up the warts that get in the way" option we might
want to consider.
And even if it were a good idea, some of the details are
probably not right -- if this looks like it received a day's
thought, you would probably be guessing much too high.
Discussion should probably go to the SMTP list; IMAA is copied
only because this could interact a bit with some of the "UTF-8
header" discussions.
john

[w***@elan.net]

The proposal does not really address "cleaner" or "more detailed" message
routing data or define standards for it. What it does is to separate the
actual message from the routing/envelope/trace data being added by mail
servers while the message is in transit. Additional proposals would have
to build on this one to define more trace data or more information about
message routing, etc.

One possible benefit is that mail filtering systems would be able to deny
message or bounce it if data is not acceptable to it (not properly signed,
bad origin, etc). This has certain processing/bandwidth benefits as
message would not have had to be processed or tranmitted in full (and then
bounced back to "envelope-from" address which may not even exist) and
could be bounced immediatly oe possibly certain authentication data could
be requested and immediatly check on.

In general to other remarks I have to agree, it is not perfectly clear if
its better to add number of additional extensions that may solve (close
holes) in current email transport system or if its better to just design
new mail transport protocol all together. However it maybe of benefit to
everybody if we work on both approaches at the same time for now and
decide what is better in the future (several prcidents to that - CRISP has
worked on both FIRS/LDAP and IRIS/XML variants; LEMONADE is working on
extensions for IMAP to do both do remote mail editing & submission and
different approach to reference parts of imap message in message being
submitted direct from end-user computer)

Post by Hector Santos
Hello John,
In general, I have much to say about any SMTP proposal that suggests SMTP
change at the software level. To me, a change in software opens the door for
other possible concepts that may address a particular needs in the email
industry. IMTV, acceptance and deployment aspects of the proposal are
important and major considerations of any SMTP change proposal. IMTV,
that's a matter of a having a strong functional specification dictation.
With that said, I am putting my "Software Engineering Hat" on as if I was
going to implement your ENVL proposal today to see what would be the
technical implementations issues. Based on this technical point of view,
What does this ENVL proposal attempt to address? (What problem(s) does it
solve?)
Excuse me if I missed it in the draft, but I see only three (3) concepts
that addresses a "problem is solved:"
1) Proposal provides a cleaner SMTP envelope or proposal to create a cleaner
RFC 822 trace header?
2) Cleaner SMTP envelope minimizes mail relay issues.
3) Syntax Error checking.
Is this correct?
What is it about the headers today that makes it problematic?
You touch base with section 3, item 6 indicating "Syntax Error." Syntax
Error needs to be clarified, i.e, can (or may) this include an
implementation that attempts to perform some sort of sender/machine
validation?
What is considered the ENVL header?
The term "Trace Fields" are used. This needs to be clarified for the
software engineer who may not be in the same league as protocol police or
IETF gurus. It seems to indicate only the Received: and Return-Path: lines
are considered and are the ENVL transaction.
Does this proposal basically attempts to separate the transmission of a RFC
822 formatted message normally done at the DATA state into two separate
C: ENVL
S: 250 Start Mail Header input; end with <CRLF>.<CRLF>
C: [sends header followed by "." line]
S: 250 Mail Header OK. Continue with Data Command
C: DATA
S: 354 Start Mail Body input; end with <CRLF>.<CRLF>
C: [send message body only (no header) followed by "." line]
S: 250 Message accepted for delivery.
or are we just adding a new state that allows a client to provide a server
with access to "DATA" header (all or specific) information using the ENVL
before the actual DATA transaction takes place?
C: ENVL
S: 250 Start Mail (Trace??) Header input; end with <CRLF>.<CRLF>
C: [sends mail header (all or just some specific headers??) followed by
"." line]
S: 250 Mail Header OK. Continue with Data Command
C: DATA
S: 354 Start Mail Header/Body input; end with <CRLF>.<CRLF>
C: [sends mail header again and body followed by "." line]
S: 250 Message accepted for delivery.
In short, an example for the "laymen" would be nice :-)
Anyway, I personally believe your ENVL is a good start. It has promise.
Off hand, without grasping yet the true intent this proposal (what does it
solve), what defines the ENVL "headers", etc, I believe the answer to this
will have a major factor on the acceptance and deployment aspect of the
proposal. Besides other impact considerations which I don't wish to muddy
the water just yet without getting the answer to the above, if this ENVL or
another other SMTP change proposal proves to be useful, I'll be among the
first to implement such a new SMTP feature like this in our SMTP product I
will be interested on providing an immediate "proof of concept" wide test
bed of customers.

[Keith Moore]

I'm all for separating the envelope from the message content, but I'd
be interested in investigating an even more radical change - (probably)
scrapping SMTP entirely or (less likely) branching out of the SMTP
state machine very early. Or in other words, I could see making the
new protocol similar enough to SMTP that a single server could
implement both, but that might just invite confusion.

[w***@elan.net]

First of all end-users would not care much what protocol it is when
"you've got mail" pops up.

And for designers, programmers, and quicker implementation its quite a bit
better if new protocol and old one share certain components. For example
MIME encoding/decoding, etc. It would faciliate process if protocol can
be implemented on same mail server software that would be able to decide
based on certain dns or other parameters what protocol to use when sending
email to the other end.

And look at the example of IPv4 vs IPv6 - IPv6. While there are substantial
improvements in features available with IPv6, as far as implementation and
design and application programmings, its not that different.

Post by Keith Moore
I'm all for separating the envelope from the message content, but I'd
be interested in investigating an even more radical change - (probably)
scrapping SMTP entirely or (less likely) branching out of the SMTP
state machine very early. Or in other words, I could see making the
new protocol similar enough to SMTP that a single server could
implement both, but that might just invite confusion.

[Keith Moore]

Post by w***@elan.net
First of all end-users would not care much what protocol it is when
"you've got mail" pops up.

No, but they might care about having email work more reliably.

Post by w***@elan.net
And for designers, programmers, and quicker implementation its quite a bit
better if new protocol and old one share certain components. For example
MIME encoding/decoding, etc. It would faciliate process if protocol can
be implemented on same mail server software that would be able to decide
based on certain dns or other parameters what protocol to use when sending
email to the other end.

Yes that's why I would consider running both protocols on the same port.

On the other hand if the new protocol were designed well then there
would be less need to share components. For instance if all mail were
binary transparent and delivered directly from the sender's submission
server to the recipient's message store (which is close to what you
need for reliable error reporting) there would be not be as much need
for such an MTA to encode or decode MIME.

Post by w***@elan.net
And look at the example of IPv4 vs IPv6 - IPv6. While there are substantial
improvements in features available with IPv6, as far as implementation and
design and application programmings, its not that different.

Not at first glance. But for a variety of reasons (e.g. address
scoping and new kinds of hosts) I suspect we'll end up programming
differently in IPv6 than in IPv4.

[J-F C. (Jefsey) Morfin]

I an idea we work on is to keep SMTP as it is, and use it for mail
signaling (weemail). The interest is that nothing is to be changed except
to modify slightly the user agent and develop a new agent - a store and
retrieve mail server which may also be used as a gateway between the
current and the new system. The "you've got mail" is then not refering to a
piece of junk with emebeded virus you have been forfced to accept into your
computer and spend bandwidth for, but to a file of any kind you can chose
to disregard and leave on the sender's server or read totally or in part, etc.

This does not kill spam as there are ways to fake a server. But it makes
wild spam far more complex and drastically reduces the bandwidth usage.

The possibilties it easily allows, like dynamic distribution lists, also
permits to prioritize the reader interest and to leave the spam at the
bottom of the basket. Mail URLs also allows to more easily hunt for known
spam or to disregard known commercial sources - or already read posts.

It is there quite easy to use 0-Z numbering to name LHS and to have a
resolution system into any vernacular (ML, menus, emoticon, etc.).
jfc

Post by w***@elan.net
First of all end-users would not care much what protocol it is when
"you've got mail" pops up.
And for designers, programmers, and quicker implementation its quite a bit
better if new protocol and old one share certain components. For example
MIME encoding/decoding, etc. It would faciliate process if protocol can
be implemented on same mail server software that would be able to decide
based on certain dns or other parameters what protocol to use when sending
email to the other end.
And look at the example of IPv4 vs IPv6 - IPv6. While there are substantial
improvements in features available with IPv6, as far as implementation and
design and application programmings, its not that different.

Post by Keith Moore
I'm all for separating the envelope from the message content, but I'd
be interested in investigating an even more radical change - (probably)
scrapping SMTP entirely or (less likely) branching out of the SMTP
state machine very early. Or in other words, I could see making the
new protocol similar enough to SMTP that a single server could
implement both, but that might just invite confusion.

[Hector Santos]

Don't wish to insult anyone with the obvious but here some points we should
keep in mind when considering revamping SMTP.

1) "worthy new solutions" are those that best address real problems. While
there will also be a market where needless change is performed ("latest toy
syndrome"), in general, costly revamps become more feasible for
consideration when they provide major benefits.

2) Consideration is enhanced when new functional specs include well outlined
migration, acceptance and deployment plans, ideally including or referencing
source code examples and/or API, SDK tools to help the migration process.

3) Responsive vendors will help the process by making the "conversion"
process easier via their software, and lastly

4) Where there is a market for change, a tertiary market of tools (proxies,
middle ware, APIs) will be made available to address legacy software and/or
provide easier development and migration.

In short, "lets not fear change"

Also, consider Yahoo's plan for implementing their YDK (Yahoo Domain Keys)
proposal. According to recent cyber news, Yahoo has said they are going to
modify open source mail servers (I believe they mentioned sendmail and
qmail) to implement their YDK and according them, plan to release it when
it all send and done. I am seriously interested what others think about
this.

Thanks

[Martin Duerst]

Post by Hector Santos
Also, consider Yahoo's plan for implementing their YDK (Yahoo Domain Keys)
proposal. According to recent cyber news, Yahoo has said they are going to
modify open source mail servers (I believe they mentioned sendmail and
qmail) to implement their YDK and according them, plan to release it when
it all send and done. I am seriously interested what others think about
this.

I have just looked at an article about YDK for a couple minutes, and
don't claim to understand it fully. There seem to be some similarities
to SPF and other proposals. We (W3C) have deployed SPF records just
recently. The main benefit we are expecting is that we can avoid
ourselves and our mailing lists being spammed by impersonators faking
our own email addresses. That's not all of spam, but it's a very
nasty and troubling bit of it.

Regards, Martin.

[Hector Santos]

Martin,

We have added DMP to our anti-spam package and it works great in this area
(checking your local domain and machine spoofs). WCSAP performs 4 checks:
Internal White/Black List, RBL, DMP, and CBV and it is called at the RCPT
state (only if the recipient is acceptable, if refused, wcsap is not
called).

What I did not get 100% clear about SPF, is whether it works from a central
authority (database) concept. Its documentation seems to indicate both (a
central and your own). The author should clarify this more. I am somewhat
resistance to having a "central" database system, however I am not naive to
know that this is probably ultimately the direction the email world will
head to. So what we did was add a general DNS lookup rules/parser into our
system so that it can ready for anything in the future.

In any case, in my research and testing of the various DNS lookup proposals,
the only "current" benefit I see from all the anti-spam DNS based lookup
proposals is checking spoofs against your own domains. If you begin to
check other domains, the DNS overhead skyrockets. It was pointed out to me
by the DMP author that this is a function of your DNS setup. I am not too
sure about this. But I don't pretend to be an DNS expect. All I know is
that looking up an unknown domain (which is what a majority of the spammers
are) yields a long initial lookup delay.

Thanks
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com





----- Original Message -----
From: "Martin Duerst" <***@w3.org>
To: "Hector Santos" <***@winserver.com>; <ietf-***@imc.org>;
<ietf-***@imc.org>
Sent: Monday, January 26, 2004 10:52 AM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

Post by Martin Duerst

Post by Hector Santos
Also, consider Yahoo's plan for implementing their YDK (Yahoo Domain Keys)
proposal. According to recent cyber news, Yahoo has said they are going to
modify open source mail servers (I believe they mentioned sendmail and
qmail) to implement their YDK and according them, plan to release it when
it all send and done. I am seriously interested what others think about
this.

I have just looked at an article about YDK for a couple minutes, and
don't claim to understand it fully. There seem to be some similarities
to SPF and other proposals. We (W3C) have deployed SPF records just
recently. The main benefit we are expecting is that we can avoid
ourselves and our mailing lists being spammed by impersonators faking
our own email addresses. That's not all of spam, but it's a very
nasty and troubling bit of it.
Regards, Martin.

[Martin Duerst]

Hello Hector,

Post by Hector Santos
What I did not get 100% clear about SPF, is whether it works from a central
authority (database) concept. Its documentation seems to indicate both (a
central and your own). The author should clarify this more.

Please tell him directly. I don't have any direct contacts.

Post by Hector Santos
In any case, in my research and testing of the various DNS lookup proposals,
the only "current" benefit I see from all the anti-spam DNS based lookup
proposals is checking spoofs against your own domains.

That's the initial benefit, as I mentioned.

Post by Hector Santos
If you begin to
check other domains, the DNS overhead skyrockets. It was pointed out to me
by the DMP author that this is a function of your DNS setup. I am not too
sure about this. But I don't pretend to be an DNS expect. All I know is
that looking up an unknown domain (which is what a majority of the spammers
are) yields a long initial lookup delay.

I'm no DNS expert either. But giving the spammers some delay may be a
nice side effect :-).


Regards, Martin.

[Paul Hoffman / IMC]

This discussion does not belong on the IMAA mailing list. Please trim
your Cc lists. Thanks!

--Paul Hoffman, Director
--Internet Mail Consortium

[Nathaniel Borenstein]

Post by w***@elan.net
And for designers, programmers, and quicker implementation its quite a bit
better if new protocol and old one share certain components. For example
MIME encoding/decoding, etc.

Well, let's not rule out improvements to MIME in the process. The name
MIME includes a lot of things, from the two-level content-type
architecture to the quoted-printable and base64 transfer encodings.
I'd certainly like to see us keep the former, but the latter can only
be seen as warts for backward compatibility with SMTP/822. I wouldn't
make it one of the major goals of email NG, but I certainly wouldn't
mind seeing the new protocol eliminate the need for
content-transfer-encodings.

Another comment: I strongly urge moving this discussion to the new
mail-ng mailing list once it's up and running. The imaa and smtp
mailing lists will make a lot more short-term progress if the
longer-term discussions have their own venue. -- Nathaniel

[Hector Santos]

I agree with you. I believe SMTP can be used to offer a "dual" state
transaction model. Yes, design and implementation confusion can be an
issue. However, I believe this only means it would become one of many
aspects for the new design to address and minimize.

I would like to suggest an approach borrowing old concepts such as Fidonet
when it was faced with new industry needs and state machine protocols.

The original and standard protocol was called FTSC1. This was mostly based
on a kludged XMODEM handshake. A second one called WAZOO offered an easier
session handshake and introduced ZMODEM for data transfer. A 3rd one called
EMSI addressed the session handshake issues. EMSI introduced text based tags
or attributes that defined and establish the compatibility level between the
client and server. It also introduced new tags to define new file transfer
protocols.

If EMSI or WAZOO was not established, the fallback was FTSC1. As a side
note, for historical perspective, the demise of Fidonet was not helped by
the fact that the Fidonet or FTSC (Fidonet Technical Standards Committee)
protocol police insisted on backward compatibility. WAZOO/EMSI flexibility
allowed for growth including new "internet readiness." As new developers
come aboard to support the new direction (internet), most used the easier to
implement WAZOO/EMSI protocols and avoided FTSC1 mostly due to the fact it
was a complex outdated modified XMODEM7 binary protocol handle shake which
is probably akin to the idea of SMTP having no states other than DATA! It
didn't fit well in packet switching networks. Security was weak and it
created conflicts between the development community and the FTSC. Fidonet
operators or members found to be using non-FTSC1 compliant mailers were
excommunicated from Fidonet. To get an Fidonet address (akin to getting a
MX record maybe), your HOST (akin to maybe an ISP) would check for FTSC1
compliance. Of course, IETF does not have strong "network" membership
requirements such as this, but I would hate to see the demise of the IETF
because it could not or was slow to address the current industry issues
and/or needs. It is all deja vu when you hear news that YAHOO is taking the
lead in changing the SMTP specification.

In any case, WAZOO/EMSI was established with the SERVER answering with
special greeting allowing for EMSI or WAZOO detection. The compatible
client detected either one and choose its method. If the server did not see
a response within X seconds, it finally issued a FTSC1 detection packet.

With SMTP, I believe it would be a lot easier because SMTP is already
Text/Tag based.

The initial greeting 220 line can be used to define the SMTP attributes. In
a way, we sort of having something like this now. The BCP for the greeting
line is to use a "ESMTP" tag to indicate an Extended SMTP ready server.
However, I am not sure if anyone uses this since '2821' requires you to use
EHLO first anyway.

But we can use the greeting to define the server attributes which compliant
clients can read and use.

In Fidonet, to support WAZOO, WAZOO tag was used in the server greeting.
For EMSI, it used a EMSI tag. In addition, if EMSI was detected a handshake
was done to exchange and establish the session attributes.

With SMTP, it is easier:

examples:

220 host, SMTP, SMTP3 [product info] Server Reader
220 host, SMTP3 [product info] Server Reader

SMTP indicates old SMTP state support. SMTP3 indicates the new state
support. Non-compliant clients would not know the difference. Only
clients looking for SMTP3 will know the level of support this server offers.
Whether backward support is required, I leave it to you guys to debate.
Please just remember what I noted above. Confusion and conflicts is reduced
when developers (old or new) do what is only demanded by the market needs.
If old support is mandatory and is conflictive with security issues, then it
needs to be discussed whether old support is going to complicate migration
and acceptability. The concept of allowing it to be option for system
administrators to decide should be discussed.

Once the SMTP3 session is established, the new state machine prevails!
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com




----- Original Message -----
From: "Keith Moore" <***@cs.utk.edu>
Sent: Sunday, January 25, 2004 7:27 PM
Subject: Re: I-D ACTION:draft-klensin-email-envelope-00.txt

Post by Keith Moore
I'm all for separating the envelope from the message content, but I'd
be interested in investigating an even more radical change - (probably)
scrapping SMTP entirely or (less likely) branching out of the SMTP
state machine very early. Or in other words, I could see making the
new protocol similar enough to SMTP that a single server could
implement both, but that might just invite confusion.