Mail-based Comment System

Motivation

To get feedback, Comment System are essential to web pages.

Most of modern web pages use 3rd-party services like Disqus. They are criticized by ads or trivial steps for sign-up/validation. For my Point of View, even using Github issue is not a good idea. I mean… You did make contents by yourself with markdown or some similar things, but why let others hold comments for you?

CGI is another solution if you can access a machine with public IP. To show comments in your pages, you just need to transform HTTP Request into HTML. For example, this one-line comment button would be easily implemented! But consider spam or offensive content, a way to identify visitor still matters.

For each comment, all you need to know are inside Mail header: sender title, mail address, datetime and SMTP server. By SPF/DKIM/DMARC, receiver can nearly 100% confirm Mail is from a specific sender.

Mailing List

Of course there is already a Comment System based on Mail: Mailing List.

Actually I am not a mailing-list user (though I did subscribe some lists of FOSS projects). But thanks to public web page for archive¹, I can realize how powerful it is.

User just need to finish subscription, then everyone can start a new thread by mail, and get replied by another mail.

But not everyone can build such this Comment System, you need to:

1. Be system admin to build related services
2. Tune HTTP web server to show archive
3. Afford traffic of SMTP for each new message in thread

This might be a little overkill for basic Comment System on simple web pages.

MDA and .forward

If you can login as a normal user in Unix System (like tilde.club), ~/.forward is what you need.

From decades ago, Unix use program called MDA to deliver mail/message² to local user (Most of time it is Sendmail). User can write their own ~/.forward file to tell MDA where this mail should goes. For each entry in this file, it could be:

1. Another local user, for example: alice, bob
2. User in another host, for example: [email protected]
3. Absolute pathname of a file, the mail would be appended to it. For example:
   /home/alice/mails
   Of course you can use /dev/null to drop any mail!
4. A command with leading char |, it would be executed with mail as STDIN. For example:
   "|/home/alice/mail-handler.sh arg1 arg2"³

With this approach, you can easily create a Common System based on mail:

1. Put a mailto link in your web page
2. Create a command to store incoming mail as comment in HTML format
3. Write an entry for this command in .forward
4. In your web page, display comments made by step.2

Let’s do it!

comment.sh

Let’s build a bash script called comment.sh for all of these:

1. Check mail is for comment

I use pattern Comment on page: <PATH_OF_WEB_PAGE> in mail subject, to determine a mail should be stored as a comment. If it is not, script just return 0 safely:

# RFC 2822 use Carriage Return, remove it for safety
MAIL="$(tr -d '\r')"

# join multi-line header field value into one line
header="$(<<<"$MAIL" sed '/^$/ q; :a; N; s/\n\s\+//; ta')"

# just mail body
body="$(<<<"$MAIL" sed -n '/^$/,$ p' | sed '1d')"

# determine mail is for comment by pattern
pattern='^Subject: .*[cC]omment on page:? +(https?://[^/]+)?(/?[^ ]+).*$'
<<<"$header" grep -E "$pattern" >/dev/null || exit 0

2. Determine output and text process tool

For your HTTP routing, set output_dir to where HTML for comments locates.

Also, I use markdown_bin to specify command to process markdown text in mail body

output_dir=...
markdown_bin=...

3. Save each header field as variable

• For easier usage, save each field key as upper-case, and replace - with _
• Save datetime as rfc-3339 for more readable

# enable execute last command in pipe under current shell
shopt -s lastpipe; set +m;

# save each field of header into variables
echo "$header" | \
while read field value; do
  echo "$field" "$value" >>/tmp/header
  declare field=$(<<<$field tr [:lower:] [:upper:] | tr '-' '_' | tr -d ':')
  declare $field="${value}"
done

# formated datetime
DATE=${DATE:+$(date --rfc-3339 seconds --date "$DATE")}

4. Get path of output file

Based on mail subject, determine path of output. For example:
if mail subject contains Comment on page: /posts/comment-by-mail.html, then set output to $output_dir/posts/comment-by-mail.comment.html

path=...
output=$output_dir/${path}.comment.html
umask 022; mkdir -p $(dirname $output)
exec 1>$output

5. Get comment from mail body

By RFC 1341, header with Content-Type: multipart/*; is separated by value of boundary. Some mail client automatically sends mail with both text part and html part. Here we extract text part if necessary:

printTextPart() {
  boundary=$1
  beginPat="\\|^--${boundary}\$|"
  endPat="\\|^--${boundary}(--)?\$|"
  sed -En "${beginPat},${endPat} { \@^Content-Type: text/plain@,$ {1,/^$/d; ${beginPat}d; p} }"
}

# check mail includes multiple part
boundary="$(<<<"$CONTENT_TYPE" sed -En 's/^.*boundary="?([^"]+)"?.*$/\1/p')"
if [ -n "${boundary}" ]; then
  # print mail part in MIME: text/plain
  body="$(<<<"$body" printTextPart ${boundary})"
fi

6. Write comment to output file

Time to write mail body to output file. For Unix System like tilde.club, set it readable for everyone:

umask 133

1. This mail is the first comment on page
2. Output file is illegal HTML

if [ ! -f $output ] || ! xmllint --html --nofixup-base-uris $output &>/dev/null; then
  <<-LAYOUT cat >$output
    <style>
       ul {
         ...
       }
      .comment-body {
        ...
      }
      .replies {
        ...
      }
    </style>
    <ul>
    </ul>
    LAYOUT
fi

# Append STDIN after line number with variable, or after first <ul> tag
<<-COMMENT sed -i "${line:-/<ul>/}r /dev/stdin" $output
    <li>

    <time datetime="${DATE}">${DATE}</time>
    <a href="mailto:${RECIPIENT}?subject=Comment on page: ${path}&in-reply-to=${MESSAGE_ID}">[reply]</a>

    <div class='comment-body'>
    $(<<<"${body}" ${markdown_bin})
    </div>

    <details class="replies" open="true">
    <summary>replies</summary>
    <ul>
    <!-- ${MESSAGE_ID} -->
    </ul>
    </details>

    </li>
COMMENT

1. Variable $line to insert this comment into output file
2. Variables comes from mail header, like $DATE or $RECIPIENT
3. An anchor element to reply to this comment (with Message-ID in mail header)
4. Command $markdown_bin to render HTML from mail body
5. Use <details> to create a foldable part for replies
6. In block of replies, add value of Message-ID as marker. It could be used to locate where to put other comments as replies

If visitor click the anchor element to reply, then mail client would prepare a new mail with header contains In-Reply-To with $Message-ID

With step.5, we can get $line number for new mail (which reply to this mail) by marker:

# get $line by field "In-Reply-To"
if [ -n "${IN_REPLY_TO}" ]; then
  line=$(grep -n "^<!-- ${IN_REPLY_TO} -->$" $output | cut -d':' -f1)
fi

7. ~/.forward

To co-work with MDA, just put this script into ~/.forward:

# Inside ~/.forward

alice
|"/<PATH>/<TO>/comment.sh --output /srv/http"

Set web page

To include HTML file generated by comment.sh, we can use <object> or <iframe>:

<object type="text/html" data="/mypage.comment.html"></object>

<object ... onload="observeResize(this)"></object>
<script>
  function observeResize(commentBlock) {
    const doc = commentBlock.contentDocument.documentElement
    new ResizeObserver(() => {
      commentBlock.style.height = doc.clientHeight + 'px';
    }).observe(doc);
  }
</script>

For tilde.club

This script could be used by non-admin Unix user, like member of tilde.club.

For tilde.club user, you can easily add it into .forward by one-line command:

echo '"|/home/pham/helper/bin/mail/comment.sh --output_dir /home/<USER>/public_html"' >>~/.forward

...
<div style="border-radius: 6px; background: lightyellow">
  <a style="display: inline-block; margin: 0.5em 0.5em 0 0; float: right" href="mailto:<USER>@tilde.club?subject=Comment on page: /index.html">[Comment on this page]</a>
  <object type="text/html" data="/index.comment.html" onload="observeResize(this)" style="width: 100%;"></object>
  <script>
    function observeResize(commentBlock) {
      const doc = commentBlock.contentDocument.documentElement
      new ResizeObserver(() => {
        commentBlock.style.height = doc.clientHeight + 'px';
      }).observe(doc);
    }
  </script>
</div>
...

Improvement

Still, this script only implements very basic features. It could be better with:

1. Build new HTML file from IMAP or Maildir, because they are stored as mail! (In current stage, you will lose all comments if output file is removed)
2. A proper way to show name of commenter. For privacy, this script doesn’t expose mail address of sender.
3. Works as CGI to get comment from a web form, I don’t think every user can/want/expect launch mail client from browser. Send a confirmation mail for their comments might be a valid solution.
4. Prevent texts in <pre> element fails on checking pattern

I am using this script on my own site or tilde.club page. Will make changes by my need. You can always get latest version by the following URL:

• https://tilde.club/~pham/comment.sh
• https://git.topo.tw/helper/file/bin/mail/comment.sh.html

Or by git archive from my git server:

git archive --remote git://git.topo.tw/helper HEAD bin/mail/comment.sh | \
tar --strip=2 -xf -