"Lexical analysis in java"

– Sophisticated lexical analysis techniques – better that what you can hope to achieve manually • Examples: lex and flex for C, JLex and Jflex for Java • Can be used to generate – Standalone scanners (i.e., have a “main”) – Scanners integrated with automatically-generated parsers (using yacc, bison, CUP, etc.) 15.

Lexical Analysis-1 BGRyder Spring 99 4 Course Overview • Learn by doing - project orientation; supplemented by theoretical underpinings - weekly problem assignments • Some familiarity with object-oriented programming essential (C++, Java, ST80)

Lexical Analysis •Sentences consist of string of tokens (a syntactic category) For example, number, identifier, keyword, string •Sequences of characters in a token is a lexeme for example, 100.01, counter, const, “How are you?” •Rule of description is a pattern

Lexical Analysis-3 BGRyder Spring 99 6 Using JLex • First, run x.lex file through Jlex to produce x.lex.java, a scanner for the tokens described in x.lex • Second, compile x.lex.java to byte code • Third, write a data file with examples of the tokens in it • Fourth, run Parse.Main main method that

–Copied directly to Java file JLex directives –Define macros, state names Lexical analysis rules –Optional state, regular expression, action –How to break input to tokens –Action when token matched %% %% Possible source of javac errors down the road DIGIT= [0-9] LETTER= [a-zA-Z]

NumReader.java • Look at NumReader.java example – Implements a token recognizer using a switch statement. 33 The Story So Far • We can write tokens types as regular ... • The lexical analysis generator then creates a NFA (or DFA) for each token type and combines them into one big NFA. 50 From REs to a Tokenizer

Lexical Analysis Handout written by Maggie Johnson and Julie Zelenski. The Basics Lexical analysis or scanning is the process where the stream of characters making up the source program is read from left-to-right and grouped into tokens. Tokens are sequences of characters with a collective meaning. There are usually only a small number of tokens

Laboratory 2 – Lexical Analysis The function of a lexical analyzer (lexer) is to take an input stream of characters and break it into tokens. JFlex is a Java-based lexical analyzer generator based on Lex (C-based). Its output is a Java source program that contains the necessary functions to perform a particular lexical analysis.

UNC Chapel Hill Brandenburg — Spring 2010 04: Lexical Analysis COMP 524: Programming Language Concepts Tokens Not every character has an individual meaning. In Java, a ʻ+ʼ can have two interpretations: ‣A single ʻ+ʼ means addition. ‣A ʻ+ʼ ʻ+ʼ followed by another means increment. A sequence of characters that has an atomic meaning is called a token.

Lexical Analysis and Parsing Lecture 2 CS 212 – Fall 2007 Recall! Compiling Java ! Compiling Bali Java Program Java Compiler Java Byte Code (JBC) JVM Interpreter Bali Program ... Java has a class (introduced in Java 5) java.util.Scanner" Can recognize identifiers, numbers, quoted strings, and various comment styles

The Java library has a Character class which provides some static methods that act on the char data type. They can make your job easier. Its part of java.lang, so remember to import java.lang.*; Some useful methods for lexical analysis of strings are: boolean isDigit(char c) boolean isLetter(char c) boolean isLetterOrDigit(char c)

Writing lexical analysers, it is possible to associate identifiers with regular expressions, and use these identifiers later, by enclosing them in {}. For example the following definitions describe many of the tokens that occur in Java. (refer JAVA) package grammar; import java.io.*; import java_cup.runtime.*; %% %public %type Symbol %char %{

Lexical Analysis 22-2 Lecture Overview Lexical analysis = breaking programs into tokens is the first stage of a compiler. The structure of tokens can be specified by regular expressions. The ML-Lex tool can automatically derive a lexical analyzer from a description of tokens specified by regular expressions.

01-27: Lexical Analyzer Generator JavaCC is a Lexical Analyzer Generator and a Parser Generator Input: Set of regular expressions (each of which describes a type of token in the language) Output: A lexical analyzer, which reads an input file and separates it into tokens

Regular Expressions in Lexical Specification • Last Lecture: a specification for the predicate s L(R) • But a yes/no answer is not enough! • Instead: partition the input into tokens • We adapt regular expressions to this goal c 1 c 2 c 3 c 4 c 5 c 6 c 7 … Set of strings

Will cover one component of the compiler: lexical analysis, parsing, semantic analysis, and code generation. Each project will ultimately result in a working compiler phase which can interface with other phases. You must implement the project in Java. For this project, you are to write a lexical analyzer, also called a scanner, using a lexical ...

Lexical Specification → Regex in five steps 3. Let input be x 1…x n For 1 ≤ i ≤ n check x 1…x i ∈ L(R) 4. If success, then we know that x 1…x i ∈ L(R j) for some j 5. Remove x 1…x i from input and go to (3) Lexer → Regex NFA DFA Tables

Lexical Analysis • A lexical analyzer collects input characters into groups (lexemes) and assigns an internal code (a token) to each group. • Lexemes are recognized by matching the input against patterns. • Tokens are usually coded as integer values, but for the sake of readability, they are often referenced through named constants.