yapij-js

Table of Contents generated with DocToc

• GitLab CI
• Building locally
• GitLab User or Group Pages
• Did you fork this project?
• Troubleshooting

Use

Basic Use

// const { Server, config } = require("yapij-js");
var { Server } = require("./src/index.js");

// Initialize server
var s = new Server();

// Start the server
s.start().then(
  () => {
    console.log("Started");

    // Executes in python shell
    s.exec('print("Hello, World! (from python)")');
  },
  e => {
    throw e;
  }
);

var hdl = s.exec('"Execute returns a handler."');

// Stop the server
s.stop().then(() => {
  console.log("Shut down!");
});

Further Use

Interrupting Scripts

You can interrupt a long-running script by calling the interrupt attribute.

const { Server, config } = require("./src/index.js"); //require("yapij");
var s = new Server();

// Start the server (async)
s.start().then(
  () => {
    // A script with a longer run that is interrupted
    s.exec(`
import time

a = 0
for ii in range(5):
  print(ii)
  a += 1;
  time.sleep(2)
  `);

    // Call interrupt to interrupt it
    return setTimeout(() => {
      s.interrupt();
      s.exec("print('a = {}'.format(a))");
      s.exec('"done!"');
      return setTimeout(() => {
        return s.stop().then(() => console.log("Stopping server."));
      }, 1000);
    }, 2500);
  },
  e => {
    console.error(`Caught an error:\n`);
    throw e;
  }
);

Note. There appears to be something funny going on with the node.js shell console. If you run the whole script above as one it will have mostly expected behavior. However, if you try to execute the sleeping script then subsequently call the interrupt command the output will be stopped but the execution will not.

Magics

Magics are specialized built-in python commands. They take their inspiration from iPython Magics but are not the same thing as ipython magics (in either structure or commands).

Here is the full set of commands:

%pwd         # Get the current working directory
%cd [path]   # Set the working directory
%help quit   # Currently only works on magics.
%interrupt   # Interrupt interpreter. Can also be called by s.interrupt()
#%quit       # Not implemented yet
#%restart    # Not implemented yet

%save               # Save current file
%load file1.gtmm    # Load a file
%rename file2.gtmm  # Renames the file
%copy file3.gtmm    # Copy the current file into a new file
%delete file2.gtmm  # Delete the file. If no argument,
%new file_new.gtmm  # Create a new workspace
%reset              # Reset the current workspace, including the file

%who env # Get info about the current file.

%run script.py # Run a python script in the working environment

!ls   # Prefacing a command with an `!` will cause it to be executed in a shell.

?print    # Help information about objects in workspace.

To execute these magics just pass them to Server.exec:

const { Server, config } = require("yapij");
var s = new Server();

// Start the server (async)
s.start();

s.exec("%new file1.gtmm");
s.exec("import time; time.sleep(500)");
s.exec("%interrupt"); // Does the same thing as s.interrupt() from above.

These commands are executed on a separate thread from that of the main python interpreter. Therefore, they can be executed even if some code is being executed.

Server.start and Server.stop

These two methods must be called at the beginning and end of any session. In particular:

• Server.start:
  1. Finds 3 open ports that can be bound to.
  2. Starts the python process (at Server.kernel)
  3. Starts the pub/sub routines that are used in the main routines (at Server.workers.session)
  4. Starts the ping-pong heartbeat monitor (at Server.workers.heartbeat)
  5. Starts the pub/sub "bridge" (at Server.workers.bridge).
• Server.stop:
  1. Ends the python process and all ZMQ communications.
  2. Resets the ports to null

Both of these methods return a Promise.

For start, the promise is not resolved until an initial heartbeat is registered with the python process.

If the Server is already started a promise is returned right away with value true. If a Server is already stopped then a promise is resolved with value 2 (rather than 1).

Server.restart

This method calls Server.stop then Server.start and returns a promise.

Installation

TBD

File Handling and Workspaces

A major goal of this package is to make it easy to....

Saving

The saved workspace exists in a thread that is separate from the thread that carries out operations.

While Code is Executing

If a workspace is saved while code is being executed, the workspace from prior to the execution will be saved. For example:

Suppose that we started out by executing: a = 5.

Once this script is complete, the workspace will be updated. Then we carried out the script:

import time
a = 9
time.sleep(500)

Finally, we save while the above script is running (i.e. within 500 s. of the initial call): %save.

The value a=5 will be saved to file because the a=9 command is part of an currently-running event.

If we wait 500 seconds and save again then a=9 will be in the workspace (becuase time.sleep is complete) and that will be saved to file.

Exceptions

If an exception is raised by the python interpreter then current state of the interpreter workspace is still transferred to the overall workspace.

For example, the following script will raise an exception:

a = 5
raise ValueError
a = 9

The first line a=5 will be interpreted then the exception is raised. The third line is never reached.

If we were to save now we would have a=5 in the workspace.

Interrupt

Options

On Server Construction

pythonArgs

pythonArgs is an object passed to Server that specifies different options to be passed to the python side of the script.

Here are the options and their defaults:

config

A second argument to be passed to Server is config, which deals with the actual python process itself:

var config = {
  python: {
    ...
  },
  spawn: {
    ...
  }
};

The elements of python are:

The elements of spawn pertain to options for child_process.spawn. Here, I only list the arguments that are specified by default. Any other options found in the child_process.spawn docs may additionally be specified.

Callback Events

There are several events for which one can register callbacks. Set these callbacks by calling the "on" events for the relevant workers and kernel. The snippet below shows all of the available events with their defaults as the arguments:

// Initialize server
var s = new Server();

// Kernel
s.kernel.on("close", code => console.log(`[YAPIJ:CLOSE] ${code}`)); // s
s.kernel.on("stdout", msg => console.log(`[YAPIJ:STDOUT] ${msg}`));
s.kernel.on("stderr", msg => console.error(`[YAPIJ:STDERR]\n${msg}`));

// Session
s.workers.session.on("stream", msg => console.log(`[YAPIJ] ${msg}`));
s.workers.session.on("warn", msg => console.warn(`[YAPIJ] ${msg}`));
s.workers.session.on("error", msg => console.error(`[YAPIJ] ${msg}`));

// Heartbeat
s.workers.heartbeat.on("fail", msg => console.error(`[YAPIJ:FAIL] ${msg}`)); // s

Note. Server.stop is called on heartbeat failure and kernel close (i.e. those with s) already. Therefore, it is not necessary to add this step to any new callback.

You may also place monitoring events on any of the ZMQ sockets. See the ZMQ Network Diagram below for information concerning the location of the sockets. See these docs for an example of monitoring ZMQ sockets in node.

Interprocess Communication

ZMQ Network Diagram

(Add image of network here)

Heartbeats

Project Information

To Do

• [ ] Better tests
• [x] Better handling of case where PID is already shut down.
  • Instead of raising an error if the PID is not found, passed by.
  • Also make a call to PY_QUIT_[key] that tells the system to shut down.
  • Check this by calling s.kernel.proc.kill then calling stop.
• [ ] Custom magics.
• [ ] Check Linux/Mac compatibility.
  • Main sources now should have to do with child_process.
• [ ] Figure out how want to track io (e.g. which output comes from which?)
• [x] Work out whether interrupt is doing everything that needs to be done.
• [ ] Ensure builds zeromq well with electron. See ZMQ docs
• [x] Add %cd and %pwd magics.
• [ ] ~Add untrusted execution mode.~ (Never mind. Putting this aside for now.)
• [ ] Add ability to set initial working directories.
• [ ] Figure out easiest way to install with python package.
• [ ] Chase down why it appears that heartbeat still seems to be running even when clearInterval was called?
  • Tried additionally setting the interval objects to be null in object.
  • Check that this works as expected

Contact

Michael Wooley michael.wooley@us.gt.com michaelwooley.github.io

License

UNLICENSED

(Sorry, not my choice.)