Adapter for Plex-like sub-scanners: module venture.parser.venture_script.subscanner

Adapter for using Plex-like scanner libraries for writing VentureScript sublanguages.

The standard VentureScript sublanguage interface (see venture.ripl.ripl.Ripl.register_language()) expects the VentureScript parser to control the input stream, and expects the sublanguage parser to manifest as a procedure that accepts characters and reports whether the utterance is complete and what it parsed to.

Many parser and scanner libraries, however, expect to themselves control the input stream, i.e. provide a method like read(stream) -> result. Notably, this includes Plex, the library in which the VentureScript scanner itself is written.

This module provides an adapter which uses multithreading (!) to convert between the two interfaces, so a Plex-like scanner can be used to implement a VentureScript sublanguage.

class venture.parser.venture_script.subscanner.Reader(char_queue, token_queue)

Bases: object

An input stream for consumption by a Plex-like scanner for a VentureScript sublanguage.

The consuming scanner is expected to call read to fetch input. The requested block size is ignored and input is given one character at a time.

Relevant instances will be constructed by Scanner; do not make instance of Reader directly.

read(_n)

class venture.parser.venture_script.subscanner.Scanner(scanner)

Bases: object

Adapter from a Plex-like scanner to a callable scanner as needed in a VentureScript sublanguage.

Usage:

class MyScanner(Plex.Scanner):
  def __init__(self, stream):
    ...
ripl.register_language("some name", lambda : Scanner(MyScanner))

The scanner object provided to __init__ must be a constructor for a Plex-like scanner. Specifically, scanner will be called with one argument representing the input stream (an instance of Reader). It must return an object that has a read() method, which, when called, consumes some quantity of the input stream represented by the Reader and returns a sequence consisting of the parsed result in the first location and an arbitrary collection of additional values. (Plex scanners return a 2-tuple by default.)

The input scanner must consume only as much input as needed to determine the boundary of a complete utterance in its language, as additional characters cannot be put back into the input stream for consumption by the surrounding VentureScript parser. The sublanguage must consume the closing curly brace in the @{<name> <utterance>} syntax, but no additional characters.