Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Documenting Regular Expressions in Groovy

Here are some suggestions for using regular expressions in Groovy. This page mainly focuses on documenting regular expressions in Groovy, however, those suggestions are applicable to any programming language that supports regular expressions such as Perl and Java.

Here are some useful reference links that you may want to open up along side this page:

Documenting the RegEx

It's important to document any regular expression, or "regex" for short, that is more than a trivial match. Documenting regexes is the key to making them understandable so they can be debugged and modified either by someone else or by you after you've had time to forget the details.

Overview

  • Include a sample of text to match
    • give a plain English description of your goal
    • omit excess lines of sample if long
  • Use extended patterns with comments
    • mark capturing groups by number
    • include "landmark" keys in the pattern
  • Include debugging feedback
    • use debugging lines

Include a sample of text to match

Having a sample of the input that the regular expression is being applied to look at right on screen is a great help in deciphering what the pattern is trying to match. This is best done in a block comment before the regular expression pattern is defined.

  • give a plain English description of your goal
    • note "landmark" keys in the pattern that you rely on to reliably parse the data
    • list any sub-parts (captured groups) of the pattern you wish to use after the match
  • omit excess lines of sample if long

For example, on a system that has remotely mounted disk space with names like "/nfs/data" or "/nfs/DATA" we wish to gather the space free in kilobytes and the name on which the space is mounted. The output from the "df -k" (disk free space in kilobytes, on linux/mac/unix systems) could be parsed by this pattern:

The "(?i)" is a match flag that means the pattern is case insensitive.

To summarize the parts (regular expression constructs) here:

  • (\d+) - One or more digits, captured for later use, "+" means 1 or more repetitions, see the Pattern API
  • \s+ - One or more whitespace characters
  • \d+% - One or more digits followed by "%", the percentage of disk used
  • (\/nfs\/data.*) - look for a partition name that starts out with "/nfs/data"
    • \/ - A literal "/", escaped by "\" since a slash by itself starts or ends the pattern
    • .* - Matches 0 or more characters, "." is a wildcard, "*" means 0 or more repetitions

Following the suggestions above, a header comment is added:

Use extended patterns with comments

The extended match mode is enabled by a pattern match flag which allows white space and comments to be embedded into the pattern. You can then describe, piece by piece, the parts of the regular expression without dumping those details into the already large header comment suggested above. Pattern match flags are discussed in more detail later in this document.

In Groovy, this match flag is "(?x)" and can be combined with other flags you wish to turn on such as "(?ix)" for both extended and case-insensitive modes. This is done in conjunction with Groovy "here" documents (triple quoting), which is handled somewhat differently than the "slashy" quoting used for regular expression patterns. The three examples below are equivalent, but I've highlighted in red what is removed from the first, and colored green the new text in the second and third examples. // 1. slashy regex
pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data.*)/

// 2. string converted to a regex
regex = "(?i)(\\d+)\\s+\\d+%
s+(/nfs/data.*)"
pattern = ~regex

// 3. here document string converted to a regex
regex = '''(?ix)(\\d+)\\s+\\d+%
s+(/nfs/data.*)'''
pattern = ~regex

Essentially:

  • Forward slashes don't need to be escaped by back slashes so "\/" becomes "/"
  • Double the remaining back slashes. Back slashes need to be escaped by back slashes when quoting strings (either normal or here documents)
  • If you want to match whitespace, then you must use "
    s"
  • You can match "#" with "
    #" so that it's not interpreted as the beginning of a comment

What does the third example buy you? Now newlines and comments can be included. The third example (here document) above can also be written: // 3. here document string converted to a regex
regex = '''(?ix) # comments are now allowed!
(
d+) # disk space
s+
d+% # one or more numbers followed by "%"
s+
(/nfs/data.*) # partition name'''
pattern = ~regex
This allows you to

  • mark capturing groups by number
    • I mark these with a numbered comment like "# 1: The disk space we want"
  • explain "landmark" keys in the pattern
    • For example "
      d% # a number followed by %". Not every line needs a comment, but don't leave out any important key matches.

Expanding on the example above:

If that's not any easier to understand than what we started out with,

I'll just assume you're the sort of person who never reads code comments.

Include debugging feedback

  • use debugging lines
    • they are easy to turn on/off with a flag variable
    • they verify the regular expression is working

While developing regular expressions, you will probably want to be able to easily test the result. An easy option is to add debugging lines The debugging lines can be controlled by a boolean flag to turn them on or off. For little development programs and snippets in the groovyConsole, this is easier than setting up logging. The example above can be expanded with a 'debugging' flag and debugging lines like this:

With 'debugging = true', this prints some information to show the regular expression is working on the test data:

matcher pattern:
/---------------------------------\\\\\\\\\
(?ix) # enable case-insensitive matches, extended patterns
(\d+) # 1: The disk space we want
\s+ # some whitespace
\d+% # a number followed by %
\s+ # some more whitespace
(/nfs/data.*) # 2: partition name
---------------------------------/
match count=2
text matched in matcher0: '3885824 63% /nfs/data_d2/dog_data'
free space in (group 1): '3885824'
partition name (group 2): '/nfs/data_d2/dog_data'
text matched in matcher1: '259683200 18% /nfs/DATA-1/cat_data'
free space in (group 1): '259683200'
partition name (group 2): '/nfs/DATA-1/cat_data'
KB available=263569024

You can see by the output above that most of the input is ignored because it doesn't meet the described pattern. For those entries that are split across two lines, it turns out that all the information we want is in the second line, which still meets the pattern criteria, and the first line is ignored for not matching.

And if you set 'debugging = false', only the result is printed:

KB available=263569024

Pattern Match Flags

The Java regular expression support includes many options modeled after Perl, which is one of the strongest regular expression parsing languages. Since Groovy gets its regular expression capability from Java (which copied Perl), what works in Java applies equally well to Groovy. Looking at the Java java.util.regex.Pattern API we see that there is support for pattern match flags under the section called "Special constructs (non-capturing)." Specifically the line indicating

(?idmsux-idmsux) Nothing, but turns match flags on - off

These capture nothing, but activate the match flags "idmsux". These correspond mostly to similarly named flags in Perl:

Match Flags

Flag

Java/Groovy

Perl

Description

i

CASE_INSENSITIVE

ignore case

Do case insensitive pattern matching

d

UNIX_LINES

not in Perl

Enables Unix lines mode, only '\n' line terminator affects ., ^ and $

m

MULTILINE

multiline

Enables multiline mode. In multiline mode the expressions ^ and $ match just after or just before, respectively, a line terminator or the end of the input sequence. By default these expressions only match at the beginning and the end of the entire input sequence.

s

DOTALL

single line

In Perl this is called Single-line mode, treating the input as as single line even if it includes line terminators. Normally the "." wildcard doesn't match line terminators, but in Dotall mode it matches all characters.

u

UNICODE_CASE

not in Perl

Enables Unicode-aware case folding.

x

COMMENTS

eXtended

Extended mode allows whitespace, including newlines, and comments beginning with "#" and ending with a newline. Since whitespace in the pattern is ignored, use '\s' to indicate whitespace you wish to match.

g

automatic in Java/Groovy

global

Global matching keeps track of a current position in the input, so you can step through each place the pattern matches the input. Groovy sets up a result array that is an array of arrays of strings. The first dimension represents the number of matches, the second dimension contains the substrings that represent actual text matched and any captured groups.

https://www.ngdc.noaa.gov/wiki/index.php?title=Regular_Expressions_in_Groovy&action=edit&section=8Example usage of Match Flags

This is how the Match Flags may be used in Groovy. Below is a pattern that will grab the disk space value from the output of the unix "df" (disk freespace) command, but only on lines containing "/nfs/data", the "(?i)" indicates a case insensitive match: pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data\/*)/
Note that the "(?i)" match flag comes at the beginning and is wrapped in parentheses. The "?" indicates this is a non-capturing group. Normally anything in a regular expression you wish to capture is wrapped in parentheses. Assuming the match over all succeeded, this "captured" group is available for later use as a substring that matched the portion of the regular expression in the parentheses.

For comparison, here is the equivalent pattern in Perl: $dfkOutput =~ /(\d+)\s+\d+%\s+(\/nfs\/data\/*)/i
Notice that the "i" indicating a case-insensitive match is appended at the end of the pattern.

The eXtended Pattern Match Flag (x)

The key flag to making regular expressions more readable is the "x" flag, which meant "eXtended" in Perl, and in Java/Groovy refers to "Comments," in a less mnemonic way.

Although Java and Groovy use the java.util.regex.PatternAPI, which was modeled after Perl regular expressions, the extended mode wasn't of much use in Java. You could write

But it would be a little less typing to simulate this with ordinary comments:

However, in Groovy a regex combined with a here document, this becomes much cleaner (and more Perl-like):

For comparison, here's the Perl equivalent:

In Perl you can use "m_<delim>_" to start a match instead of "/", so I could pick "!" as the delimiter and avoid having to escape the "/" characters in the partition name. Also Perl regular expressions use a single backslash, as in Groovy single line slashy regex patterns. Unfortunately, for multi-line Groovy patterns defined as a here document string, you need to double the backslashes.

Other Notes

Test code fragments in the groovyConsole

Groovy takes a couple of seconds to compile and run, and during iterative testing of a regular expression, this time becomes noticeable. Paste your code fragment into groovyConsole, set up some test input and a print statement to show the result. You can then run rapidly there without the compile lag. The print statement can become your debugging line when copying code back to your main project.

Conquer Complex Patterns by Dividing them into Simpler Categories

Beware of trying to deal with complicated patterns with a "one-size-fits-all" monster regular expression. Often it's easier to use an initial pattern to decide the category of input, and then select (if-then-else or switch-case) an appropriate simpler regex that deals with just that sub-category of input. You may need to include messages to warn against unexpected input that doesn't match any of the known categories.

  • No labels