Package org.apache.sysds.api.mlcontext

Class MLContext

java.lang.Object
org.apache.sysds.api.mlcontext.MLContext
All Implemented Interfaces:
ConfigurableAPI
public class MLContext extends Object implements ConfigurableAPI
The MLContext API offers programmatic access to SystemDS on Spark from languages such as Scala, Java, and Python.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    MLContext.ExecutionType
    The different types of execution environments supported by SystemDS.
    static enum 
    MLContext.ExplainLevel
    The different explain levels supported by SystemDS.
    class 
    MLContext.InternalProxy
    Used internally by MLContextProxy.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static boolean
    welcomePrint
    Welcome message

  • Constructor Summary

    Constructors
    Constructor
    Description
    MLContext(org.apache.spark.api.java.JavaSparkContext javaSparkContext)
    Create an MLContext based on a JavaSparkContext for interaction with SystemDS on Spark.
    MLContext(org.apache.spark.SparkContext sparkContext)
    Create an MLContext based on a SparkContext for interaction with SystemDS on Spark.
    MLContext(org.apache.spark.sql.SparkSession spark)
    Create an MLContext based on a SparkSession for interaction with SystemDS on Spark.

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    buildTime()
    Obtain the SystemDS jar file build time.
    void
    close()
    Closes the mlcontext, which includes the cleanup of static and local state as well as scratch space and buffer pool cleanup.
    MLResults
    execute(Script script)
    Execute a DML or PYDML Script.
    MLResults
    execute(Script script, ScriptExecutor scriptExecutor)
    Execute a DML or PYDML Script object using a ScriptExecutor.
    static MLContext
    getActiveMLContext()
    Retrieve the currently active MLContext.
    MLContext.ExecutionType
    getExecutionType()
    Obtain the current execution environment.
    MLContext.InternalProxy
    getInternalProxy()
    Used internally by MLContextProxy.
    org.apache.spark.sql.SparkSession
    getSparkSession()
    Obtain the SparkSession associated with this MLContext.
    int
    getStatisticsMaxHeavyHitters()
    Obtain the maximum number of heavy hitters that are printed out as part of the statistics.
    ProjectInfo
    info()
    Obtain information about the project such as version and build time from the manifest in the SystemDS jar file.
    boolean
    isExplain()
    Whether or not an explanation of the DML/PYDML program should be output to standard output.
    boolean
    isForceGPU()
    Whether or not the "force" GPU mode is enabled.
    boolean
    isGPU()
    Whether or not the GPU mode is enabled.
    boolean
    isInitBeforeExecution()
    Whether or not the default ScriptExecutor should be initialized before execution.
    boolean
    isMaintainSymbolTable()
    Obtain whether or not all values should be maintained in the symbol table after execution.
    boolean
    isStatistics()
    Whether or not statistics of the DML/PYDML program execution should be output to standard output.
    void
    resetConfig()
    Reset configuration settings to default settings.
    void
    setConfig(String configFilePath)
    Set SystemDS configuration based on a configuration file.
    void
    setConfigProperty(String propertyName, String propertyValue)
    Set configuration property, such as setConfigProperty("sysds.localtmpdir", "/tmp/systemds").
    void
    setExecutionScript(Script executionScript)
    Sets the script that is being executed
    void
    setExecutionType(MLContext.ExecutionType executionType)
    Set the execution environment.
    void
    setExplain(boolean explain)
    Whether or not an explanation of the DML/PYDML program should be output to standard output.
    void
    setExplainLevel(String explainLevel)
    Set the level of program explanation that should be displayed if explain is set to true.
    void
    setExplainLevel(MLContext.ExplainLevel explainLevel)
    Set the level of program explanation that should be displayed if explain is set to true.
    void
    setForceGPU(boolean enable)
    Whether or not to explicitly "force" the usage of GPU.
    void
    setGPU(boolean enable)
    Whether or not to use (an available) GPU on the driver node.
    void
    setInitBeforeExecution(boolean initBeforeExecution)
    Whether or not the default ScriptExecutor should be initialized before execution.
    void
    setLineage(boolean lineage)
    Set whether or not lineage should be traced
    void
    setLineage(LineageCacheConfig.ReuseCacheType reuse)
    Set type of lineage-based reuse caching and enable lineage tracing
    void
    setMaintainSymbolTable(boolean maintainSymbolTable)
    Set whether or not all values should be maintained in the symbol table after execution.
    void
    setStatistics(boolean statistics)
    Whether or not statistics of the DML/PYDML program execution should be output to standard output.
    void
    setStatisticsMaxHeavyHitters(int maxHeavyHitters)
    Sets the maximum number of heavy hitters that are printed out as part of the statistics.
    String
    version()
    Obtain the SystemDS version number.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • welcomePrint

      public static boolean welcomePrint
      Welcome message

  • Constructor Details

    • MLContext

      public MLContext(org.apache.spark.sql.SparkSession spark)
      Create an MLContext based on a SparkSession for interaction with SystemDS on Spark.
      Parameters:
      spark - SparkSession

    • MLContext

      public MLContext(org.apache.spark.SparkContext sparkContext)
      Create an MLContext based on a SparkContext for interaction with SystemDS on Spark.
      Parameters:
      sparkContext - SparkContext

    • MLContext

      public MLContext(org.apache.spark.api.java.JavaSparkContext javaSparkContext)
      Create an MLContext based on a JavaSparkContext for interaction with SystemDS on Spark.
      Parameters:
      javaSparkContext - JavaSparkContext

  • Method Details

    • getActiveMLContext

      public static MLContext getActiveMLContext()
      Retrieve the currently active MLContext. This is used internally by SystemDS via MLContextProxy.
      Returns:
      the active MLContext

    • resetConfig

      public void resetConfig()
      Description copied from interface: ConfigurableAPI
      Reset configuration settings to default settings.
      Specified by:
      resetConfig in interface ConfigurableAPI

    • setConfigProperty

      public void setConfigProperty(String propertyName, String propertyValue)
      Description copied from interface: ConfigurableAPI
      Set configuration property, such as setConfigProperty("sysds.localtmpdir", "/tmp/systemds").
      Specified by:
      setConfigProperty in interface ConfigurableAPI
      Parameters:
      propertyName - property name
      propertyValue - property value

    • execute

      public MLResults execute(Script script)
      Execute a DML or PYDML Script.
      Parameters:
      script - The DML or PYDML Script object to execute.
      Returns:
      the results as a MLResults object

    • execute

      public MLResults execute(Script script, ScriptExecutor scriptExecutor)
      Execute a DML or PYDML Script object using a ScriptExecutor. The ScriptExecutor class can be extended to allow the modification of the default execution pathway.
      Parameters:
      script - the DML or PYDML Script object
      scriptExecutor - the ScriptExecutor that defines the script execution pathway
      Returns:
      the results as a MLResults object

    • setExecutionScript

      public void setExecutionScript(Script executionScript)
      Sets the script that is being executed
      Parameters:
      executionScript - script that is being executed

    • setConfig

      public void setConfig(String configFilePath)
      Set SystemDS configuration based on a configuration file.
      Parameters:
      configFilePath - path to the configuration file

    • getSparkSession

      public org.apache.spark.sql.SparkSession getSparkSession()
      Obtain the SparkSession associated with this MLContext.
      Returns:
      the SparkSession associated with this MLContext.

    • isExplain

      public boolean isExplain()
      Whether or not an explanation of the DML/PYDML program should be output to standard output.
      Returns:
      true if explanation should be output, false otherwise

    • setExplain

      public void setExplain(boolean explain)
      Whether or not an explanation of the DML/PYDML program should be output to standard output.
      Parameters:
      explain - true if explanation should be output, false otherwise

    • setLineage

      public void setLineage(boolean lineage)
      Set whether or not lineage should be traced
      Parameters:
      lineage - true if lineage should be traced, false otherwise

    • setLineage

      public void setLineage(LineageCacheConfig.ReuseCacheType reuse)
      Set type of lineage-based reuse caching and enable lineage tracing
      Parameters:
      reuse - reuse cache type to use

    • isMaintainSymbolTable

      public boolean isMaintainSymbolTable()
      Obtain whether or not all values should be maintained in the symbol table after execution.
      Returns:
      true if all values should be maintained in the symbol table, false otherwise

    • setMaintainSymbolTable

      public void setMaintainSymbolTable(boolean maintainSymbolTable)
      Set whether or not all values should be maintained in the symbol table after execution.
      Parameters:
      maintainSymbolTable - true if all values should be maintained in the symbol table, false otherwise

    • setExplainLevel

      public void setExplainLevel(MLContext.ExplainLevel explainLevel)
      Set the level of program explanation that should be displayed if explain is set to true.
      Parameters:
      explainLevel - the level of program explanation

    • setExplainLevel

      public void setExplainLevel(String explainLevel)
      Set the level of program explanation that should be displayed if explain is set to true.
      Parameters:
      explainLevel - string denoting program explanation

    • setGPU

      public void setGPU(boolean enable)
      Whether or not to use (an available) GPU on the driver node. If a GPU is not available, and the GPU mode is set, SystemDS will crash when the program is run.
      Parameters:
      enable - true if needs to be enabled, false otherwise

    • setForceGPU

      public void setForceGPU(boolean enable)
      Whether or not to explicitly "force" the usage of GPU. If a GPU is not available, and the GPU mode is set or if available memory on GPU is less, SystemDS will crash when the program is run.
      Parameters:
      enable - true if needs to be enabled, false otherwise

    • isGPU

      public boolean isGPU()
      Whether or not the GPU mode is enabled.
      Returns:
      true if enabled, false otherwise

    • isForceGPU

      public boolean isForceGPU()
      Whether or not the "force" GPU mode is enabled.
      Returns:
      true if enabled, false otherwise

    • getInternalProxy

      public MLContext.InternalProxy getInternalProxy()
      Used internally by MLContextProxy.
      Returns:
      InternalProxy object used by MLContextProxy

    • isStatistics

      public boolean isStatistics()
      Whether or not statistics of the DML/PYDML program execution should be output to standard output.
      Returns:
      true if statistics should be output, false otherwise

    • setStatistics

      public void setStatistics(boolean statistics)
      Whether or not statistics of the DML/PYDML program execution should be output to standard output.
      Parameters:
      statistics - true if statistics should be output, false otherwise

    • setStatisticsMaxHeavyHitters

      public void setStatisticsMaxHeavyHitters(int maxHeavyHitters)
      Sets the maximum number of heavy hitters that are printed out as part of the statistics.
      Parameters:
      maxHeavyHitters - maximum number of heavy hitters to print

    • close

      public void close()
      Closes the mlcontext, which includes the cleanup of static and local state as well as scratch space and buffer pool cleanup. Note that the spark context is not explicitly closed to allow external reuse.

    • info

      public ProjectInfo info()
      Obtain information about the project such as version and build time from the manifest in the SystemDS jar file.
      Returns:
      information about the project

    • version

      public String version()
      Obtain the SystemDS version number.
      Returns:
      the SystemDS version number

    • buildTime

      public String buildTime()
      Obtain the SystemDS jar file build time.
      Returns:
      the SystemDS jar file build time

    • getStatisticsMaxHeavyHitters

      public int getStatisticsMaxHeavyHitters()
      Obtain the maximum number of heavy hitters that are printed out as part of the statistics.
      Returns:
      maximum number of heavy hitters to print

    • isInitBeforeExecution

      public boolean isInitBeforeExecution()
      Whether or not the default ScriptExecutor should be initialized before execution.
      Returns:
      true if ScriptExecutor should be initialized before execution, false otherwise

    • setInitBeforeExecution

      public void setInitBeforeExecution(boolean initBeforeExecution)
      Whether or not the default ScriptExecutor should be initialized before execution.
      Parameters:
      initBeforeExecution - true if ScriptExecutor should be initialized before execution, false otherwise

    • getExecutionType

      public MLContext.ExecutionType getExecutionType()
      Obtain the current execution environment.
      Returns:
      the execution environment

    • setExecutionType

      public void setExecutionType(MLContext.ExecutionType executionType)
      Set the execution environment.
      Parameters:
      executionType - the execution environment