Rewriting the Ruby parser | Rails at Scale

At Shopify, we now have spent the final 12 months writing a brand new Ruby parser, which we’ve known as YARP (But One other Ruby Parser). As of the date of this put up, YARP can parse a semantically equal syntax tree to Ruby 3.3 on each Ruby file in Shopify’s foremost codebase, GitHub’s foremost codebase, CRuby, and the 100 hottest gems downloaded from rubygems.org. We not too long ago received approval to merge this work into CRuby, and are very excited to share our work with the group. This put up will take you thru the motivations behind this work, the best way it was developed, and the trail ahead.

In case you’re unfamiliar with the idea of parsers or how they apply to Ruby, there’s a background part obtainable on the backside of this put up that ought to get you in control.

Motivations

The present CRuby parser has a few long-standing points that we wished to handle. Broadly these fall into 4 classes: maintainability, error tolerance, portability, and efficiency. We’ll go into every of those in flip under.

Maintainability

Maintainability is nearly totally subjective, or no less than could be very tough to measure. The general idea will be damaged down into many various sides, together with however not restricted to how simple it’s to: learn and perceive the code, contribute to the code, change the code, doc the code, and check the code.

The present CRuby parser has no actual documentation that we might discover. There have been exterior initiatives which have tried to doc its design, notably the Ruby Hacking Guide from 2002 and Ruby Under A Microscope from 2013. Aside from these decades-old efforts, the very best probability you’ve got is studying the 14 thousand-line parse.y file and making an attempt to grasp it. This can be a daunting process to say the least, and one which we don’t suppose anybody ought to must do.

On account of its complexity, the parser can also be tough to alter. Think about bug #19392 from two months in the past, when it was found that def check = places("foo") and places("bar") doesn’t work in any respect such as you would count on. Due to the best way the parser is structured, it’s not possible to fix this bug with out breaking different code. This can be a frequent theme in generated parsers, the place seemingly easy adjustments can have far-reaching penalties.

Trying on the contribution checklist, it’s unsurprising to seek out that the prevailing parser can solely be maintained by a few folks. Within the 25 years that the parser has existed, solely 65 folks have contributed to it, and solely 13 of these have contributed greater than 10 commits. Within the final 12 months, solely 9 folks have contributed to the parser, of which solely 2 have contributed greater than 10 commits.

Maintainability is on the coronary heart of open-source, and sadly the scenario we discover ourselves in is devoid of a maintainable parser.

Error tolerance

Error tolerance is the power of a parser to proceed parsing a program even when it encounters syntax errors. In different phrases, an error-tolerant parser can nonetheless generate a syntax tree even within the presence of syntax errors.

Error tolerance is essential for a lot of causes. Editors, language servers, and kind checkers like Sorbet or steep, depend on parsers to offer correct metadata — varieties, arguments, scope, and so on. — concerning the code being edited or analyzed. With out an error-tolerant parser, that metadata can get dumped on the first syntax error. This pushes the error-tolerance drawback right down to the customers of the parser who must attempt to reconcile their lack of metadata to get again to a steady state, which will be tough and error-prone.

As of Ruby 3.2, the CRuby parser has some minor error tolerance, however nothing that you’d name a systemic method. This implies even probably the most trivial syntax errors end result within the parser failing to generate a syntax tree. The downstream results of this are that when you’ve got a number of syntax errors in your file (normally due to copy-pasting) you find yourself having to repair them one by one, which is a really sluggish course of. These slower cycles will be irritating, and stand in distinction to the beliefs of “developer happiness” that Ruby is thought for.

For example, contemplate in case your editor might solely show one error at a time. Every time one is mounted, the following would seem. This may be time-consuming and irritating for builders. Think about the next snippet:

class Foo
  def initialize(a: 1, b = 2)
    true &&
  finish

There are 3 syntax errors within the supply above (the order of parameters, the lacking expression on the correct aspect of the &&, and the lacking finish key phrase). Operating this by way of ruby -c at the moment (which checks for syntax errors) you get:

check.rb: check.rb:2: syntax error, surprising native variable or technique (SyntaxError)
  def initialize(a: 1, b = 2)
                       ^
check.rb:4: syntax error, surprising `finish'
  finish
  ^~~

This mentions the primary problem, the second problem is confused for one thing else, and the third is lacking totally.

Error tolerance is due to this fact one thing we wished to bake into YARP from the start, accounting for it at each degree of the design.

Portability

Portability refers back to the means to make use of the parser outdoors of the CRuby codebase. At present, the parser is tightly tied to CRuby internals, requiring information constructions and capabilities solely obtainable within the CRuby codebase. This makes it unattainable to make use of in different tooling.

Accordingly, the group fractured and developed a number of options, every with their very own points. Over time there have been many different parsers written, virtually all by taking the grammar file and producing a brand new form of parser. In our analysis, we discovered parsers written in 9 totally different languages. A few of these made their manner into tutorial papers, in any other case into manufacturing programs. As of writing, we all know of 12 which can be being actively maintained (6 runtimes, 6 instruments):

Every of those parsers in addition to the reference implementation have their very own points. Which means that every of the instruments constructed on these parsers due to this fact inherit those self same points. The fracture due to this fact spreads into tooling. For instance, some instruments are primarily based on Ripper, together with Syntax Tree, rubyfmt, rufo, syntax_suggest, and ruby-lsp. Much more are primarily based on the parser gem, together with rubocop, standard, unparser, ruby-next, solargraph, and steep. Even extra are primarily based on the ruby_parser gem, similar to debride, flay, flog, and fasterer.

Clearly that is removed from optimum. Each time new syntax is launched into Ruby, the entire parsers must replace. This implies alternatives to introduce bugs, which all get flushed right down to their corresponding instruments. For example, Ruby 2.7 was launched 4 years in the past, and it got here together with sample matching syntax. Of the ten non-CRuby parsers, solely 5 of them assist all of sample matching to at the present time, and solely 2 of them with none caveats.

To maintain updated with the CRuby parser, each certainly one of these parsers should fastidiously look ahead to any adjustments to parse.y and try to duplicate them in their very own language/runtime. This can be a huge quantity of labor for a major quantity of people that might as an alternative all be serving to preserve and enhance a single parser as an alternative.

Portability additionally has to do with the usability of your syntax tree. Even for those who can extract the syntax tree from the parser, in case your syntax tree is simply too tightly tied to your runtime, it’s not transportable. We’ll revisit this matter later once we focus on the design of YARP’s tree.

Efficiency

Over time, processors and C compilers have gotten a lot better utilizing a few methods. These embrace pipelining, inlining capabilities, and department prediction. Sadly, the parsers generated by most parser turbines make it tough for any of those methods to use. Most generated parsers function with a mix of leap tables and gotos, rendering a few of the extra superior optimization methods impotent. Due to this, generated parsers have a most efficiency cliff that’s extraordinarily tough to beat with out vital effort.

Improvement

With these issues and motivations in thoughts, final Could we sat down and began designing options. It turned clear fairly rapidly that whereas a full-scale rewrite was a frightening process, it could be crucial to handle the entire points we had recognized. So, we sat right down to design what would change into But One other Ruby Parser.

Design

Initially we created a design doc for the undertaking, which you’ll be able to nonetheless find. We shared this doc internally earlier than additionally going to debate with Matz and the CRuby group, in addition to JRuby, TruffleRuby, and maintainers of as many tooling gems as we might discover (notably together with parser and irb).

Among the extra essential design selections that got here out of those discussions are included under. As soon as Matz and the CRuby group had been proud of the design, agreed on the method, and decided that they might merge YARP in when it was prepared, the work started in earnest.

Language

The parser could be written in C. Whereas there was some full of life debate concerning the implementation language, we ended up deciding on C. Different choices that had been thought of included C++ and Rust with numerous interop choices (even WASM cross-compilation). There ended up being two compelling causes that settled the choice. The primary is technical: the parser ought to be capable of goal any platform that has a C compiler. The second is human: the Ruby parser goes to be maintained by the CRuby group which is a gaggle of C builders. Since certainly one of our foremost acknowledged targets is maintainability and these are the those that will likely be sustaining it, it made sense to make use of the language they had been most comfy with.

Construction

The parser could be a hand-written recursive descent parser. This follows the development of most main programming languages. Of the top 10 languages utilized by builders, 7/10 of them are hand-written recursive descent. Many instruments have undergone the identical swap from Bison to hand-written, for instance gcc and golang. You too can discover the explanation why C# determined to go together with this method.

The three exceptions of the languages that don’t use hand-written recursive descent are Python, PHP, and Ruby. PHP and Ruby at present use Bison, whereas Python additionally not too long ago switched to a different taste of recursive descent known as PEG parsing. For extra on that, see PEP-617. That article is especially attention-grabbing in that it outlines a few of the ambiguities within the grammar that it’s a must to work round in the identical manner we needed to traditionally work round them in Ruby. For example they cite that within the under snippet:

with (
    open("a_really_long_foo") as foo,
    open("a_really_long_baz") as baz,
    open("a_really_long_bar") as bar
):

it’s really unattainable to precise this grammar for context managers utilizing LL(1) parsing (the fashion of parser they had been producing) as a result of the open parenthesis character is ambiguous on this context. To get round it they made their grammar extra ambiguous after which enforced that the precise grammar was enforced of their tree builder.

It’s not totally stunning that extra established languages would transfer away from Bison. Bison is a device meant to generate parsers for context-free grammars. These are courses of languages the place every rule within the grammar will be lowered to a deterministic set of tokens. Ruby’s grammar — as we noticed with Python’s — requires fairly a little bit of context to parse appropriately, making it fall into the set of grammars labeled context-sensitive. To get Bison to generate a parser that can be utilized by CRuby, quite a lot of the context, logic, and state has been pushed into the lexer. This implies you can’t precisely lex Ruby code with out preserving the entire set of parsing state round.

Laurence Tratt, a professor at King’s School London has achieved intensive analysis into this space. His work was really cited 3 times at Ruby Kaigi this 12 months, in The future vision of Ruby Parser, Parsing RBS, and our personal discuss on YARP. Within the first discuss through which his work was cited, within the second paragraph he writes:

It’s doable to hand-craft error restoration algorithms for a particular language. These typically enable higher restoration from errors, however are difficult to create.

Then, in a blog post particularly about LR versus recursive descent parsing, he states:

Current languages have usually advanced in a fashion that makes it tough, or unattainable, to specify an LR grammar. There’s no level in making an attempt to combat this: simply use recursive descent parsing.

and

In case you want the absolute best efficiency or error restoration, recursive descent parsing is the only option.

The fact is, Ruby’s grammar can’t be precisely parsed with an LR parser (the form of parser that Bison generates) with out vital state being saved within the lexer. A lot of the programming group has come to the identical conclusion about their very own parsers and have due to this fact moved towards hand-written recursive descent parsers. It’s time for Ruby to do the identical.

The final purpose to change to hand-written recursive descent really comes from Matz himself. In version 0.95 of Ruby — launched in 1995 — a small ToDo file was included within the repository. One in all only a few objects in that file was:

Hand written parser(recursive first rate)

API/AST

Initially, we had supposed on preserving the identical syntax tree as CRuby, to trigger the least quantity of disruption. Nevertheless, after dialogue with the assorted groups of each runtimes and instruments, it was determined to design our personal tree from the bottom up. This tree could be designed to be simple to work with for each runtimes and instruments. It might even be designed to be simple to keep up and lengthen going ahead.

The present tree in CRuby generally comprises info that’s irrelevant to customers and generally is lacking crucial info. For example, the idea of a vcall is a parser concern: it’s an identifier that may very well be a neighborhood variable or a technique name. Nevertheless, that is resolved at parse time. It’s nonetheless uncovered within the Ripper API although, resulting in confusion as to its that means. Contrastingly, the tree is nearly totally lacking column info, which is crucial for utilization in linters and editors.

Together with the tree redesign, we labored carefully with the JRuby and TruffleRuby groups to develop a serialization API that may enable for these runtimes to make a single FFI name and get again a serialized syntax tree. As soon as they’ve the serialized syntax tree, by way of our structured documentation they will generate Java courses to deserialize it into objects that they will use to construct their very own bushes and intermediate representations.

The tree redesign has ended up being one of the essential components of the undertaking. It has delivered one thing that Ruby has by no means had earlier than: a standardized syntax tree. With a typical in place, the group can begin to construct a collective information and language round how we focus on Ruby construction, and we will begin to construct tooling that can be utilized throughout all Ruby implementations. Going ahead this will imply extra cross-collaboration between instruments (like Rubocop and Syntax Tree), maintainers, and contributors.

Constructing

With the design in place, we went about implementing it. Throughout implementation, it rapidly turned clear that the most important hurdle was going to be a sufficiently intensive check suite. Since we had our personal tree, it meant we couldn’t check in opposition to any current check suites. Fortuitously, we carried out parity with the lexer output, so we might check to make sure the tokens that our parser produced matched the prevailing lexer. Utilizing this method, we incrementally made progress towards 100% parity in lexer output in opposition to the Shopify monolith. As soon as we hit that, we labored on ruby/ruby, rails/rails, and numerous different massive codebases. Lastly, we pulled down the highest 100 most downloaded gems from rubygems.org.

Alongside the best way, we encountered all types of challenges, notably associated to the ambiguities within the grammar. In case you’re , they’re a enjoyable detour by way of a few of the eccentricities of Ruby, detailed on the backside of this put up within the challenges part.

Maintainability

From the beginning we wished to be centered on the issues we initially famous. The make this parser as maintainable as doable, each node within the tree is documented with examples and explicitly examined. You could find that documentation here. You too can discover documentation for as a lot of the design as we might match into markdown in here. Lastly, there may be copious inline comments to make it as maintainable as doable.

Fortuitously since open-sourcing the repository initially of this 12 months, we’ve had 31 contributors add code to the parser. We’ve been working to enhance our contributing tips and steerage to make it even simpler to contribute going ahead.

Error tolerance

YARP consists of a lot of error tolerance options out of the field, and we’re planning on including many extra within the months/years to return.

At any time when supply code is being edited, it virtually all the time comprises syntax errors till the builders will get to the top of the expression. As such, it’s frequent for the underlying syntax tree to be lacking tokens and nodes that it could in any other case have in a legitimate program. The primary error tolerance characteristic that we constructed, due to this fact, is the power to insert lacking tokens. For instance, if the parser encounters a lacking finish key phrase the place one was anticipated, it is going to robotically insert the lacking token and proceed parsing this system.

YARP may also insert lacking nodes within the syntax tree. For instance, if the parser encounters an expression like 1 + and not using a right-hand aspect, it is going to insert a lacking node for the right-hand aspect and proceed parsing this system.

Moreover, when YARP encounters a token in a context that it merely can not perceive, it skips previous that token and makes an attempt to proceed parsing. That is helpful when one thing will get copy-pasted and there may be additional surrounding content material that by accident sneaks in.

Lastly, YARP features a approach we’re calling context-based restoration, which permits it to recuperate from syntax errors by analyzing the context through which the error occurred. That is much like a technique employed by Microsoft once they wrote their very own PHP parser. For instance, if the parser encounters:

foo.bar(baz, qux1 + qux2 + qux3 +)

it is going to insert a lacking node into the + name on qux3, then bubble the entire manner as much as parsing the arguments as a result of it is aware of that the ) character closes the argument checklist. At this level it is going to proceed parsing as if there have been nothing improper with the arguments.

Placing this all collectively, if we take our snippet from above once more, you possibly can see the crimson underlines that YARP will add by way of its language server to point the situation of each error within the file:

Going ahead, there are lots of extra methods we’d prefer to discover associated to error tolerance, however we’re proud of the state of the parser as it’s at the moment. In case you’d prefer to see it in motion, YARP ships with a language server and VSCode plugin that you should utilize to strive it out. You’ll discover within the doc describing the way it works, that a number of syntax errors will be displayed within the editor directly, due to the prevailing error tolerance options.

Portability

YARP has no dependencies on exterior packages, capabilities, or constructions. In different phrases it’s totally self-contained. It may be constructed by itself and utilized in any tooling that wants it. In languages with good FFI or bindgen assist, this will imply immediately accessing the parse perform and its returned constructions immediately. Going ahead, this implies you might construct Ruby tooling in languages like Rust or Zig with minimal effort.

For languages with out this assist or for whom calling C capabilities will be costly, we offer a separate serialization API. This API first parses the syntax tree into its inside construction, then serializes it to a binary format that may be learn by the calling language/device. This API was designed particularly with JRuby and TruffleRuby in thoughts, and members of these groups have been actively serving to in its improvement.

At this level JRuby has a practical prototype and TruffleRuby has merged YARP in and is actively engaged on making YARP its foremost parser. One attention-grabbing discovering from this course of was that YARP deserialization is round 10 occasions sooner than parsing. Going ahead, it’s doable that TruffleRuby might to ship serialized variations of the usual library for sooner boot speeds.

With each the C and serialization APIs in place, we will now construct standardized tooling that can be utilized throughout all Ruby implementations and as a group begin to develop a typical language round how we focus on Ruby syntax bushes. Going ahead this might probably imply the entire instruments talked about above may very well be operating on the identical underlying parser.

Whereas we’re very pleased concerning the technical win that this represents, we’re much more excited concerning the group win. With the entire glorious builders who’ve needed to spend their time sustaining separate parsers now freed up, they will now make investments that point in what makes their instruments particular. In the event that they encounter errors with the parser, this implies extra eyes on the code, extra folks to assist repair bugs, and extra folks to assist add new options.

See Also

SizeOf.Life — Dimension Of Life

Efficiency

As soon as the parser was in a position to produce semantically equal syntax bushes, we started taking a look at efficiency. We don’t have nice comparability numbers but as a result of as mentioned our tree is totally different and does extra issues typically (for instance we offer unescaped variations of strings on our string nodes to make life simpler on the customers of YARP).

What we will share up to now is that YARP is ready to parse round 50,000 of Shopify’s Ruby information in about 4.49 seconds, with a peak reminiscence footprint of 10.94 Mb. Evidently, we’re thrilled with these outcomes up to now.

Going ahead efficiency will likely be prime of thoughts, and we now have many optimizations we’ve been experimenting with. These embrace lowering reminiscence utilization by way of specialised tree nodes, improved locality by way of area allocation, and sooner identifier decision with extra performant hash lookups.

Integration

As soon as we received to a state the place we might parse easy expressions, we wished to validate our method and design by integrating with different runtimes and instruments.

JRuby and TruffleRuby groups started experimenting with the serialization API, and we labored with them to verify it was adequate for his or her wants. With some attention-grabbing tweaks (serializing variable width integers, offering a continuing pool, and different optimizations) we discovered a format that suited their wants. Each runtimes now have invested vital power in integrating YARP into their runtimes, and Oracle has somebody working full time on making YARP TruffleRuby’s foremost parser.

We additionally labored with different instruments to validate that our tree contained sufficient metadata for static evaluation and compilation. Syntax Tree is a syntax tree device suite that may also be used as a formatter, and it has an experimental department operating with YARP as its parser as an alternative of Ripper. Early outcomes present that by changing Ripper with YARP, in some instances efficiency elevated by almost two fold. We additionally constructed a VSCode plugin that yow will discover contained in the repository to make sure that our error areas and messages had been right, and work continues on that at the moment.

Just lately, we started experimenting with producing the identical syntax tree because the parser and ruby_parser gems to be able to seemlessly enable customers of those libraries to learn from the brand new parser. Early outcomes are very promising and present each a discount in reminiscence and a rise in pace.

Lastly, within the final week we now have begun work on mirroring YARP into the CRuby repository, constructing it inside CRuby, and operating it throughout the similar check suite and steady integration. That is the ultimate step earlier than merging YARP into CRuby, and we’re very excited to see it come to fruition. This work will likely be achieved within the subsequent couple of labor days.

Path ahead

This brings us to at the moment and the trail ahead. Work continues on integrating YARP into the entire numerous Ruby runtimes, and we’re excited to strive it out on extra initiatives going ahead (for instance mruby and Sorbet). We’ll proceed to work on pace, reminiscence consumption, and accuracy. Matz and the CRuby group have agreed to ship YARP as a library with Ruby 3.3 (to be launched this December), so within the subsequent model of Ruby it is possible for you to to require "yarp" and mess around with your personal syntax bushes. A few issues that may occur within the meantime earlier than that thrilling launch:

• We are going to seemingly launch the undertaking as a gem, in order that third-parties can start working with it and integrating it into their very own initiatives.
• We’ll proceed to work with the JRuby and TruffleRuby groups to make sure that the construction of the syntax tree and the serialization API are adequate for his or her wants. Hopefully quickly we’ll get a launch of those language runtimes that features YARP as their foremost parser.
• Syntax Tree goes to undertake YARP as its foremost parser, which in flip implies that ruby-lsp will reap all the advantages.
• We’ll proceed to enhance our compatibility with Ripper in order that libraries that depend on that (admittedly unstable) API can use our compatibility layer as a way of migrating.

Much more work is deliberate for the parser itself as soon as it’s merged into CRuby. This consists of, however is actually not restricted to:

• Ahead scanning error tolerance – in locations the place the parser encounters syntax errors that may very well be interpreted in a number of methods, one method is to parse with all doable interpretations ahead by some variety of tokens after which to simply accept the trail that yields the least variety of subsequent syntax errors
• Area allocation – at present nodes are allotted with particular person malloc calls, which will be costly and result in fragmentation/an absence of reminiscence locality
• Reminiscence utilization – typically we now have saved the tree comparatively small in reminiscence, however there may be all the time room to take out any redundant info or typically scale back the scale of the tree in reminiscence
• Efficiency – clearly it is a huge matter, however now that we now have reached parity with CRuby, we will begin to take a look at methods to enhance efficiency

Wrapping up

Total, we’re very enthusiastic about this work and the way forward for Ruby tooling that it implies. We will’t wait to see what you construct with it! When you have any questions this didn’t reply or are excited about contributing, please attain out to us on GitHub or Twitter!

For these of you that will need much more background or particulars, we’ve included some additional info under.

Background

A parser is the a part of a programming language that reads supply code and converts it right into a format that may be understood by the runtime. On the excessive degree, this entails making a tree construction that represents the stream of this system. If you’re taking a look at supply code you possibly can usually see this tree construction within the indentation of the code. For instance, within the following code snippet:

a def could be the highest degree node, containing numerous attributes like foo as a reputation. That node would have a statements as a baby, which is a listing of statements inside its physique. The primary assertion could be a name node with bar as the strategy title.

The parser’s duty is to create these nodes and construct the tree construction earlier than handing it off to different components of the programming language for execution. Within the case of CRuby, the parser is accountable for producing the syntax tree that’s then handed off to the YARV (But One other Ruby Digital Machine) digital machine for compilation. As soon as compiled, the generated bytecode is what’s used for execution.

Step one to producing the tree is to interrupt the supply code into particular person tokens, a course of aptly known as tokenization. Within the case of Ruby, this implies discovering issues like operators (~, +, **, ..., and so on.), key phrases (do, for, BEGIN, __FILE__, and so on.), numbers (1, 0b01, 5.5e-5, and so on.), and extra. These tokens are evaluated lazily since they want massive quantities of context to find out what they’re (an identifier like foo generally is a naked technique name, a neighborhood variable, or generally even a logo). You’ll be able to consider this as a stream of tokens that the parser can pull from because it wants them.

The second step to producing the tree is to investigate the tokens by making use of a grammar. A grammar is a algorithm that outline how the tokens will be mixed to type a legitimate program. For instance, the grammar may say {that a} program generally is a checklist of statements, and a press release generally is a technique definition, a technique name, or a continuing definition. The grammar may also specify the order through which the tokens will be mixed. For instance, a technique definition generally is a def key phrase, adopted by an identifier, adopted by a listing of arguments, adopted by a physique, adopted by an finish key phrase. That is known as a manufacturing rule.

As soon as the grammar has been utilized and the entire ambiguities resolved, the tree is lastly constructed. This tree is then handed off to the digital machine for compilation and execution.

The parser that CRuby has used is generated by a device known as Bison, a parser generator that generates LR (left-to-right, rightmost derivation) parsers. Bison accepts a grammar file (within the CRuby codebase that is parse.y) and generates a parser in C (parse.c). Importantly, Bison requires the token stream we talked about earlier. There are instruments to generate these token streams, however CRuby has used a hand-written lexer. This lexer is accountable for tokenizing the supply code after which offering the tokens to Bison because it wants them (by way of a perform known as yylex).

Challenges

Operators/key phrases

The * operator can generally imply multiply and may generally imply splat, and generally it comes right down to the variety of areas between the operator and the operand. Equally ... can generally imply vary and generally imply ahead arguments. The do key phrase can be utilized in a lot of totally different contexts, together with blocks (foo do finish), lambdas (-> do finish), and loops (whereas foo do finish). Figuring out which operator or key phrase to pick out depends upon a lot of various factors, none of that are documented.

Terminators

In Ruby, expressions will be separated by newlines, feedback, or semicolons in virtually all contexts, however not all. Plenty of state is tracked to find out if a newline must be ignored or not. For instance, in

the newlines are ignored and the 1 is related to the bar: label, however in

the newline after bar: just isn’t ignored and the 1 is the one assertion within the foo technique.

Native variables

As a result of you possibly can have technique calls with out parentheses, it may be tough to find out if a given identifier is a neighborhood variable or a technique name. For instance,

will be interpreted as a technique name to a with a daily expression argument, or as a neighborhood variable a divided by b. It depends upon if a is a neighborhood variable or not (For extra eccentricities like this, see a fascinating tric entry from 2022). Due to this ambiguity, a Ruby parser must carry out native variable decision as it’s parsing.

Common expressions

You’d think about that common expressions could be simple to parse, as a result of you possibly can merely skip to the terminator. Nevertheless, the terminator will be certainly one of many characters. For instance, you possibly can write %r{foo}. On this case it’s not laborious as a result of yow will discover the following }, however sadly common expressions (like the opposite % literals) really steadiness their terminators. Which means that %r{foo {}} is a legitimate common expression as a result of the parser retains monitor of the variety of { and } characters it has seen.

Common expressions are additionally sophisticated by the truth that they will introduce native variables into the present scope. For instance, /(?<foo>.*)/ =~ bar introduces a foo native variable into the present scope that comprises a string matching the named seize group. This, mixed with the native variable complexity above, meant that we moreover needed to ship a daily expression parser to be able to correctly parse Ruby. (CRuby embeds the Onigmo parser which it fortunately delegates this work to, however once more we didn’t wish to ship with any exterior dependencies).

Encoding

CRuby by default assumes your supply file is encoded utilizing UTF-8 encoding, however you possibly can change that by including a magic remark to the highest of the file. The parser is accountable for understanding these magic feedback after which switching to utilizing the brand new encoding for all subsequent identifiers. That is essential for figuring out, for instance, if one thing is a continuing or a neighborhood which is encoding-dependent.

CRuby really ships with 90 encodings (as of three.3) which can be each not dummy encodings and are “ASCII suitable” which implies they can be utilized as an choice for encoding supply information. YARP ships with the preferred 23 of these encodings, with plans to assist extra as wanted.