JEP 467: Markdown Documentation Feedback

Abstract

Allow JavaDoc documentation feedback to be written in Markdown reasonably
than solely in a combination of HTML and JavaDoc @-tags.

Objectives

• Make API documentation feedback simpler to put in writing and simpler to learn in supply
  type by introducing the power to make use of Markdown syntax in documentation
  feedback, alongside HTML parts and JavaDoc tags.

• Don’t adversely have an effect on the interpretation of present documentation feedback.

• Prolong the Compiler Tree API to allow different instruments that analyze
  documentation feedback to deal with Markdown content material in these feedback.

Non-Objectives

• It’s not a aim to allow automated conversion of present documentation
  feedback to Markdown syntax.

Motivation

Documentation feedback are stylized feedback showing in supply code, close to to
the declarations that they serve to doc. Documentation feedback
in Java supply code use a mix of HTML and customized JavaDoc tags
to mark up the textual content.

The selection of HTML for a markup language was affordable in 1995. HTML is
highly effective, standardized, and was highly regarded on the time. However whereas it’s no
much less well-liked right this moment as a markup language consumed by internet browsers, within the years
since 1995 HTML has turn into a lot much less well-liked as markup that’s manually produced
by people as a result of it’s tedious to put in writing and arduous to learn. Lately it’s extra
generally generated from another markup language that’s extra appropriate for
people. As a result of HTML is tedious to put in writing, nicely-formatted documentation
feedback are additionally tedious to put in writing, and much more tedious since many new
builders usually are not fluent in HTML on account of its decline as a human-produced
format.

Inline JavaDoc tags, resembling {@hyperlink} and {@code}, are additionally cumbersome and
are even much less acquainted to builders, typically requiring the creator to seek the advice of the
documentation for his or her utilization. A current evaluation of the documentation feedback
within the JDK supply code confirmed that over 95% of the makes use of of inline tags have been for
code fragments and hyperlinks to elsewhere within the documentation, suggesting that
less complicated types of these constructs can be welcome.

Markdown is a well-liked markup language for easy paperwork that’s
simple to learn, simple to put in writing, and simply reworked into HTML. Documentation
feedback are sometimes not sophisticated structured paperwork, and for the
constructs that sometimes seem in documentation feedback, resembling paragraphs,
lists, styled textual content, and hyperlinks, Markdown offers less complicated types than HTML. For
these constructs that Markdown doesn’t instantly assist, Markdown permits the
use of HTML as properly.

Introducing the power to make use of Markdown in documentation feedback would deliver
collectively one of the best of each worlds. It will permit concise syntax for essentially the most
frequent constructs and cut back the necessity for HTML markup and JavaDoc tags, whereas
retaining the power to make use of specialised tags for options not out there in
Markdown. It will make it simpler to put in writing and simpler to learn documentation
feedback in supply code, whereas retaining the power to generate the identical type
of generated API documentation as earlier than.

Description

For example of using Markdown in a documentation remark, take into account the
remark for java.lang.Object.hashCode:

/**
 * Returns a hash code worth for the article. This methodology is
 * supported for the good thing about hash tables resembling these supplied by
 * {@hyperlink java.util.HashMap}.
 * <p>
 * The final contract of {@code hashCode} is:
 * <ul>
 * <li>At any time when it's invoked on the identical object greater than as soon as throughout
 *     an execution of a Java utility, the {@code hashCode} methodology
 *     should constantly return the identical integer, supplied no data
 *     utilized in {@code equals} comparisons on the article is modified.
 *     This integer needn't stay constant from one execution of an
 *     utility to a different execution of the identical utility.
 * <li>If two objects are equal in accordance with the {@hyperlink
 *     #equals(Object) equals} methodology, then calling the {@code
 *     hashCode} methodology on every of the 2 objects should produce the
 *     similar integer end result.
 * <li>It's <em>not</em> required that if two objects are unequal
 *     in accordance with the {@hyperlink #equals(Object) equals} methodology, then
 *     calling the {@code hashCode} methodology on every of the 2 objects
 *     should produce distinct integer outcomes.  Nevertheless, the programmer
 *     must be conscious that producing distinct integer outcomes for
 *     unequal objects might enhance the efficiency of hash tables.
 * </ul>
 *
 * @implSpec
 * So far as within reason sensible, the {@code hashCode} methodology outlined
 * by class {@code Object} returns distinct integers for distinct objects.
 *
 * @return  a hash code worth for this object.
 * @see     java.lang.Object#equals(java.lang.Object)
 * @see     java.lang.System#identityHashCode
 */

The identical remark will be written by expressing its construction and styling in
Markdown, with no use of HTML and just some JavaDoc inline tags:

/// Returns a hash code worth for the article. This methodology is
/// supported for the good thing about hash tables resembling these supplied by
/// [java.util.HashMap].
///
/// The final contract of `hashCode` is:
///
///   - At any time when it's invoked on the identical object greater than as soon as throughout
///     an execution of a Java utility, the `hashCode` methodology
///     should constantly return the identical integer, supplied no data
///     utilized in `equals` comparisons on the article is modified.
///     This integer needn't stay constant from one execution of an
///     utility to a different execution of the identical utility.
///   - If two objects are equal in accordance with the
///     [equals][#equals(Object)] methodology, then calling the
///     `hashCode` methodology on every of the 2 objects should produce the
///     similar integer end result.
///   - It's _not_ required that if two objects are unequal
///     in accordance with the [equals][#equals(Object)] methodology, then
///     calling the `hashCode` methodology on every of the 2 objects
///     should produce distinct integer outcomes.  Nevertheless, the programmer
///     must be conscious that producing distinct integer outcomes for
///     unequal objects might enhance the efficiency of hash tables.
///
/// @implSpec
/// So far as within reason sensible, the `hashCode` methodology outlined
/// by class `Object` returns distinct integers for distinct objects.
///
/// @return  a hash code worth for this object.
/// @see     java.lang.Object#equals(java.lang.Object)
/// @see     java.lang.System#identityHashCode

(For the aim of this instance, beauty modifications resembling reflowing the textual content
are intentionally prevented, to assist in before-and-after comparability.)

Key variations to look at:

• Using Markdown is indicated by a brand new type of documentation
  remark wherein every line begins with /// as a substitute of the present
  /** ... */ syntax.

• The HTML <p> ingredient is just not required; a clean line signifies
  a paragraph break.

• The HTML <ul> and <li> parts are changed by Markdown bullet-list
  markers, utilizing - to point the start of every merchandise within the checklist.

• The HTML <em> ingredient is changed by utilizing underscores (_) to point
  the font change.

• Situations of the {@code ...} tag are changed by backticks
  (`...`) to point the monospace font.

• Situations of {@hyperlink ...} to hyperlink to different program parts are changed by
  prolonged types of Markdown reference links.

• Situations of block tags, resembling @implSpec, @return, and @see, are
  usually unaffected besides that the content material of those tags is now additionally in
  Markdown, for instance right here within the backticks of the @implSpec tag.

Here’s a screenshot highlighting the variations between the 2 variations, facet
by facet:

Syntax

Markdown documentation feedback are written within the CommonMark variant
of Markdown. There are enhancements to hyperlinks, to permit handy linking to
different program parts. Easy GFM pipe tables are supported,
as are all JavaDoc tags.

Hyperlinks

You possibly can create a hyperlink to a component declared elsewhere in your API by utilizing an
prolonged type of Markdown reference link, wherein the label
for the reference is derived from an ordinary JavaDoc reference
to the ingredient itself.

To create a easy hyperlink whose textual content is derived from the identification of the ingredient,
merely enclose a reference to the ingredient in sq. brackets. For instance, to
hyperlink to java.util.Listing, you’ll be able to write [java.util.List], or simply [List] if
there may be an import assertion for java.util.Listing within the code. The textual content of
the hyperlink might be displayed within the monospace font. The hyperlink is equal to
utilizing the usual JavaDoc {@hyperlink ...} tag.

You possibly can hyperlink to any sort of program ingredient:

/// - a module [java.base/]
/// - a bundle [java.util]
/// - a category [String]
/// - a subject [String#CASE_INSENSITIVE_ORDER]
/// - a technique [String#chars()]

To create a hyperlink with different textual content, use the shape [text][element]. For
instance, to create a hyperlink to java.util.Listing with the textual content an inventory, you’ll be able to
write [a list][List]. The hyperlink might be displayed within the present font,
though you should use formatting markup inside the textual content. The hyperlink is equal
to utilizing the usual JavaDoc {@linkplain ...} tag.

For instance:

/// - [the `java.base` module][java.base/]
/// - [the `java.util` package][java.util]
/// - [a class][String]
/// - [a field][String#CASE_INSENSITIVE_ORDER]
/// - [a method][String#chars()]

In reference hyperlinks, you could escape any use of sq. brackets. This would possibly happen
in a reference to a technique with an array parameter; for instance, a hyperlink to
String.copyValueOf(char[]) is written as [String#copyValueOf(char[])].

You should utilize all different types of Markdown hyperlinks, together with hyperlinks to URLs, however
hyperlinks to different program parts are more likely to be the most typical.

Tables

Easy tables are supported, utilizing the syntax of GitHub Flavored Markdown.
For instance:

/// | Latin | Greek |
/// |-------|-------|
/// | a     | alpha |
/// | b     | beta  |
/// | c     | gamma |

Captions and different options that could be required for accessibility usually are not supported.
In such conditions, using HTML tables continues to be really useful.

JavaDoc tags

JavaDoc tags, each inline tags resembling {@inheritDoc} and block
tags resembling @param and @return, could also be utilized in Markdown
documentation feedback:

/// {@inheritDoc}
/// As well as, this strategies calls [#wait()].
///
/// @param i the index
public void m(int i) ...

JavaDoc tags will not be used inside literal textual content, resembling code spans
(`...`) or code blocks, that’s, blocks of textual content which might be
both indented or enclosed inside fences resembling ``` or ~~~. In different phrases, the character sequences @... and {@...} have
no particular that means inside code spans and code blocks:

/// The next code span comprises literal textual content, and never a JavaDoc tag:
/// `{@inheritDoc}`
///
/// Within the following indented code block, `@Override` is an annotation,
/// and never a JavaDoc tag:
///
///     @Override
///     public void m() ...
///
/// Likewise, within the following fenced code block, `@Override` is an annotation,
/// and never a JavaDoc tag:
///
/// ```
/// @Override
/// public void m() ...
/// ```

For these tags that will include textual content with markup, in a Markdown documentation
remark that markup can be in Markdown:

/// @param the checklist, or `null` if no checklist is offered

The {@inheritDoc} tag incorporates documentation for a technique
from a number of supertypes. The format of the remark containing the tag does
not must be the identical because the format of the remark containing the
documentation to be inherited:

interface Base {
    /** A way. */
    void m()
}

class Derived implements Base {
    /// {@inheritDoc}
    public void m() { }
}

Person-defined JavaDoc tags could also be utilized in Markdown documentation feedback. For
instance, within the JDK documentation we outline and use {@jls ...} as a brief type
for hyperlinks to the Java Language Specification, and block tags resembling @implSpec
and @implNote to introduce sections of explicit data:

/// For extra data on feedback, see {@jls 3.7 Feedback}.
///
/// @implSpec
/// This implementation does nothing.
public void doSomething() { }

Standalone Markdown recordsdata

Markdown recordsdata in doc-files subdirectories are processed appropriately, in a
related method to HTML recordsdata in such directories. JavaDoc tags in such recordsdata are
processed. The web page title is inferred from the primary heading. YAML metadata,
resembling that supported by the Pandoc Markdown processor, is just not supported.

The file containing the content material for the generated top-level overview page might
even be a Markdown file.

Syntax highlighting and embedded languages

The opening fence in a fenced code block could also be adopted by an info
string. The primary phrase of the data string is used to derive the CSS
class identify within the corresponding generated HTML, and may be utilized by
JavaScript libraries to allow syntax highlighting (resembling with
Prism) and rendering diagrams (resembling with
Mermaid).

For instance, along with the suitable libraries, this might show a
fragment of CSS code with syntax highlighting:

/// ```css
/// p { coloration: crimson }
/// ```

You possibly can add JavaScript libraries to your documentation by utilizing the javadoc
--add-script possibility.

Syntactical particulars

As a result of horizontal whitespace initially and finish of every line of Markdown
textual content could also be vital, the content material of a Markdown documentation remark is
decided as follows:

• Any main whitespace and the three preliminary / characters are faraway from every line.

• The traces are shifted left, by eradicating main whitespace characters, till the non-blank
  line with the least main whitespace has no remaining main whitespace.

• Extra main whitespace and any trailing whitespace in every line is preserved,
  as a result of it might be vital. For instance, whitespace initially of a line might
  point out an indented code block or the continuation of an inventory merchandise,
  and whitespace on the finish of a line might point out a hard line break.

(The coverage to take away main incidental whitespace is just like that for
String.stripIndent(), besides that there isn’t a want for any particular remedy
for a trailing clean line.)

There aren’t any restrictions on the characters that will seem after the /// on every line of
the remark. Particularly, the remark might include code samples which can include feedback
of their very own:

/// Right here is an instance:
///
/// ```
/// /** Good day World! */
/// public class HelloWorld {
///     public static void foremost(String... args) {
///         System.out.println("Good day World!"); // the normal instance
///     }
/// }
/// ```

In addition to serving to visually distinguish the brand new sort of documentation
remark, using end-of-line (//) feedback eliminates the restrictions on
the content material of the remark which might be inherent with using conventional (/* ... */) feedback. Particularly, it’s not doable to make use of the character
sequence */ inside a conventional remark (JLS §3.7) though it might
be fascinating to take action when writing instance code containing conventional feedback,
strings containing glob expressions, and strings containing common
expressions.

For a clean line to be included within the remark, it should start with any elective
whitespace after which ///:

/// That is an instance ...
///
/// ... of a 3-line remark containing a clean line.

A totally clean line will trigger any previous and following remark to be
handled as separate feedback. In that case, all however the final remark might be
discarded, and solely the final remark might be thought-about as a documentation
remark for any declaration that will observe:

/// This remark might be handled as a "dangling remark" and might be ignored.

/// That is the remark for the next declaration.
public void m() { }

The identical is true for some other remark not starting with /// that will seem
between two /// feedback.

API and implementation

Parsed documentation feedback are represented by parts of the
com.sun.source.doctree bundle within the Compiler Tree
API.

We introduce a brand new sort of tree node, RawTextTree, which comprises
uninterpreted textual content, along with a brand new tree-node form, DocTree.Type.MARKDOWN,
which signifies Markdown content material in a RawTextTree. We add corresponding new
visitRawText strategies to DocTreeVisitor and its subtypes, DocTreeScanner
and DocTreePathScanner.

RawTextTree nodes with a sort of MARKDOWN symbolize Markdown content material,
together with HTML constructs however excluding any JavaDoc tags resembling {@inheritDoc}
and @param.

Markdown textual content is processed in two phases:

1. Parsing — Markdown feedback are parsed right into a sequence of RawTextTree
   nodes, every with a sort of DocTree.Type.MARKDOWN and containing Markdown
   content material, interspersed with normal DocTree nodes for inline and block
   tags. The inline and block tags are parsed in the identical means as for conventional
   documentation feedback, besides that tag content material can be parsed as Markdown.
   The sequence of nodes is saved in a DocCommentTree node, within the regular
   method.

   In contrast to a conventional documentation remark, HTML constructs usually are not parsed
   into corresponding DocTree nodes, as a result of an excessive amount of of the encircling
   context must be taken under consideration.

   The Markdown content material within the DocCommentTree ensuing from the preliminary parse
   is then examined for any reference links with no related
   link reference definition, and for which the link
   label syntactically matches a reference to a program ingredient.
   Any such hyperlink is changed by an equal node representing both {@hyperlink ...} or {@linkplain ...}.

2. Rendering — The DocCommentTree is rendered by the javadoc instrument into
   HTML that’s appropriate for inclusion within the web page being generated.

   Any sequence of RawTextTree nodes and different nodes is transformed right into a
   single string containing the textual content of the RawTextTree nodes with the Unicode
   OBJECT REPLACEMENT CHARACTER (U+FFFC) standing in for
   non-Markdown content material. The ensuing string is rendered by the Markdown
   processor after which the U+FFFC characters are changed within the ensuing output
   by the rendered types of the non-Markdown content material nodes.

   Whereas a lot of the rendering is simple, particular consideration is given to
   Markdown headings:

   • The heading stage is adjusted in accordance with the enclosing context. This
     applies whether or not the heading was initially written within the documentation
     remark as an ATX-style heading (utilizing a prefix of #
     characters to point the extent) or as a Setext-style
     heading (utilizing underlining with = or - to point
     the extent).

     For instance, a stage 1 heading within the documentation remark for a module,
     bundle, or class is rendered as a stage 2 heading within the generated web page,
     whereas a stage 1 heading within the documentation remark for a subject,
     constructor, or methodology is rendered as a stage 4 heading within the generated
     web page.

     This adjustment applies solely to Markdown headings, to not any direct use of
     HTML headings.

   • An id identifier attribute is included within the rendered HTML in order that the
     heading can simply be referenced from elsewhere. The identifier is
     generated from the content material of the heading, in the identical method as different
     identifiers generated by javadoc. (You possibly can simply receive a hyperlink to the
     heading by clicking on the popup hyperlink icon when viewing the heading in a
     browser.)

   • The textual content of the heading is added to the principle search index for the
     generated documentation.

The implementation leverages an inside copy of the well-known
commonmark-java library. By design, using the library is just not revealed in
any public supported JDK API.

Future Work

It will be doable to detect some stylized makes use of of headings adopted by
acceptable content material and convert them into equal JavaDoc tags.

For instance, a heading of Parameters adopted by an inventory of parameter names and
their descriptions might be transformed into equal @param tags:

The same coverage might be adopted for the checklist of exceptions that could be thrown by a technique:

• Remark

  # Throws
  
  * NullPointerException      if the primary parameter is `null`
  * NullPointerException      if the second parameter is `null`
  * IllegalArgumentException  if an argument is just not accepted

• Translation

  @throws NullPointerException      if the primary parameter is `null`
  @throws NullPointerException      if the second parameter is `null`
  @throws IllegalArgumentException  if an argument is just not accepted

There ought to solely ever be a single description of the return worth for a technique, so there
isn’t any want to make use of an inventory on this case:

The proposed types do seem like regular Markdown, however additionally they take up extra
vertical house. Builders might desire to stick with the extra concise types, utilizing
old-style JavaDoc tags.

It could be troublesome to increase this technique to all block tags, together with
user-specified tags, however within the JDK code base simply 5 tags (@param,
@return, @see, @throws, and @since) account for over 90% of all makes use of of
block tags.

Options

Pluggable implementation

As an alternative of leveraging a particular Markdown parser implementation, we might
as a substitute assist using different user-specified Markdown processors, offering
totally different flavors of Markdown. Nevertheless, such an strategy might result in
inconsistencies when producing documentation spanning totally different libraries for
little perceived acquire.

Translating extra Markdown to HTML

We might translate further Markdown constructs into equal DocTree
nodes, representing plain textual content, HTML, and JavaDoc tags. Whereas such an strategy
would have the benefit that API shoppers might not must be conscious that the
unique supply for the remark was in Markdown, there are additionally numerous
disadvantages:

• The extra eliminated the illustration is from the unique syntax tree, the
  tougher it’s to provide correct and related diagnostics, ought to any be
  vital. For instance, messages a couple of artificial <desk> ingredient might
  be complicated if there isn’t a such merchandise explicitly within the unique remark.

• When synthesizing DocTree nodes for HTML parts derived from
  Markdown constructs, it’s troublesome to provide correct place data
  that relates the node again to its place within the unique remark,
  because the node has no illustration within the unique remark.
  At finest, you may give a close-by place. This downside has an analog
  within the Java compiler, javac, when assigning positions for artificial
  parts such because the default no-args constructor, or for bridge strategies.

• A normal resolution is troublesome as a result of it could require data of any
  and the entire JavaDoc tags that could be concerned, as a result of many tags
  allow wealthy content material, resembling Markdown or HTML, as half however not all of their
  content material.

  For instance, the @param tag is adopted by a parameter identify earlier than the
  description, and the identify could also be enclosed in <...> if the identify is that of a
  sort parameter. It will be incorrect to interpret that identify as a fraction of
  HTML. Likewise, the @serialField tag is adopted by a reputation and a kind earlier than
  the outline. Whereas these are normal tags recognized to the usual doclet,
  the doclet additionally permits using user-defined tags.

Inline tags

Whereas the makes use of of most block tags might be changed by stylized use of headings
and ensuing content material, there isn’t a such equal for a lot of the much less frequent inline
tags. Of those, {@inheritDoc} is the most typical, and there’s no apparent
analog in Markdown. Relatively than invent another syntax for the sake of it,
it appears higher to proceed with the present inline tag syntax.

We might parse Markdown embedded in conventional /**...*/ feedback as a substitute of,
or along with, introducing /// feedback.

That is untenable, nevertheless, as a result of Markdown and HTML are totally different languages
with totally different syntax guidelines. In HTML, whitespace is barely vital as literal
textual content in a <pre> ingredient. In Markdown, in contrast, vertical whitespace might
point out a paragraph break, main horizontal whitespace might point out an
indented code block or a nested checklist, and trailing whitespace might point out a
arduous line break, equal to <br> in HTML. Moreover, there are quite a few
examples within the JDK code of sq. brackets in narrative textual content, which might danger
being interpreted as hyperlinks to program parts; for instance, The knowledge is returned as a two-dimensional array (array[x][y]).

We might construct a configurable system that accepts some /** ... */
documentation feedback in Markdown and others in HTML. It’s not clear,
nevertheless, that such a mechanism would have any vital benefit over the
extra overt use of /// feedback for feedback in Markdown and the continued use
of /** ... */ for feedback in HTML.

Dangers and Assumptions

• The implementation employs a third-party library, commonmark-java, to
  rework Markdown to HTML. If that library turns into unmaintained then we
  should keep a fork of the library to be used within the JDK, or else discover
  an equal different.

• There’s a danger of extra errors in generated API documentation, due to
  the diminished capacity to examine for dangerous code, and since authors generally
  overlook to examine the generated type of their documentation.

  For instance, in a conventional remark a paragraph containing an unterminated
  code tag resembling {@code abc will trigger a diagnostic message to be displayed when
  JavaDoc is invoked, and might be displayed within the generated documentation as
  ▶ invalid @code
  In Markdown, the equal unclosed code span `abc is specified to
  be handled as literal textual content, and might be displayed as such, with no corresponding
  diagnostic messages.