Package org.apache.sysds.api.mlcontext

Class Matrix

java.lang.Object
org.apache.sysds.api.mlcontext.Matrix
public class Matrix extends Object
Matrix encapsulates a SystemDS matrix. It allows for easy conversion to various other formats, such as RDDs, JavaRDDs, DataFrames, and double[][]s. After script execution, it offers a convenient format for obtaining SystemDS matrix data in Scala tuples.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Matrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks, MatrixMetadata matrixMetadata)
    Create a Matrix, specifying the SystemDS binary-block matrix and its metadata.
    Matrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
    Convert a Spark DataFrame to a SystemDS binary-block representation.
    Matrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, long numRows, long numCols)
    Convert a Spark DataFrame to a SystemDS binary-block representation, specifying the number of rows and columns.
    Matrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)
    Convert a Spark DataFrame to a SystemDS binary-block representation.
    Matrix(MatrixObject matrixObject, SparkExecutionContext sparkExecutionContext)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    MatrixMetadata
    getMatrixMetadata()
    Obtain the matrix metadata
    boolean
    hasBinaryBlocks()
    Whether or not this matrix contains data as binary blocks
    boolean
    hasMatrixObject()
    Whether or not this matrix contains data as a MatrixObject
    double[][]
    to2DDoubleArray()
    Obtain the matrix as a two-dimensional double array
    org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>
    toBinaryBlocks()
    Obtain the matrix as a JavaPairRDD<MatrixIndexes, MatrixBlock>
    org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
    toDF()
    Obtain the matrix as a DataFrame of doubles with an ID column
    org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
    toDFDoubleNoIDColumn()
    Obtain the matrix as a DataFrame of doubles with no ID column
    org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
    toDFDoubleWithIDColumn()
    Obtain the matrix as a DataFrame of doubles with an ID column
    org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
    toDFVectorNoIDColumn()
    Obtain the matrix as a DataFrame of vectors with no ID column
    org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>
    toDFVectorWithIDColumn()
    Obtain the matrix as a DataFrame of vectors with an ID column
    org.apache.spark.api.java.JavaRDD<String>
    toJavaRDDStringCSV()
    Obtain the matrix as a JavaRDD<String> in CSV format
    org.apache.spark.api.java.JavaRDD<String>
    toJavaRDDStringIJV()
    Obtain the matrix as a JavaRDD<String> in IJV format
    MatrixBlock
    toMatrixBlock()
    Obtain the matrix as a MatrixBlock
    MatrixObject
    toMatrixObject()
    Obtain the matrix as a SystemDS MatrixObject.
    org.apache.spark.rdd.RDD<String>
    toRDDStringCSV()
    Obtain the matrix as a RDD<String> in CSV format
    org.apache.spark.rdd.RDD<String>
    toRDDStringIJV()
    Obtain the matrix as a RDD<String> in IJV format
    String
    toString()
    If MatrixObject is available, output MatrixObject.toString().

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • Matrix

      public Matrix(MatrixObject matrixObject, SparkExecutionContext sparkExecutionContext)

    • Matrix

      public Matrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)
      Convert a Spark DataFrame to a SystemDS binary-block representation.
      Parameters:
      dataFrame - the Spark DataFrame
      matrixMetadata - matrix metadata, such as number of rows and columns

    • Matrix

      public Matrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, long numRows, long numCols)
      Convert a Spark DataFrame to a SystemDS binary-block representation, specifying the number of rows and columns.
      Parameters:
      dataFrame - the Spark DataFrame
      numRows - the number of rows
      numCols - the number of columns

    • Matrix

      public Matrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks, MatrixMetadata matrixMetadata)
      Create a Matrix, specifying the SystemDS binary-block matrix and its metadata.
      Parameters:
      binaryBlocks - the JavaPairRDD<MatrixIndexes, MatrixBlock> matrix
      matrixMetadata - matrix metadata, such as number of rows and columns

    • Matrix

      public Matrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
      Convert a Spark DataFrame to a SystemDS binary-block representation.
      Parameters:
      dataFrame - the Spark DataFrame

  • Method Details

    • toMatrixObject

      public MatrixObject toMatrixObject()
      Obtain the matrix as a SystemDS MatrixObject.
      Returns:
      the matrix as a SystemDS MatrixObject

    • to2DDoubleArray

      public double[][] to2DDoubleArray()
      Obtain the matrix as a two-dimensional double array
      Returns:
      the matrix as a two-dimensional double array

    • toJavaRDDStringIJV

      public org.apache.spark.api.java.JavaRDD<String> toJavaRDDStringIJV()
      Obtain the matrix as a JavaRDD<String> in IJV format
      Returns:
      the matrix as a JavaRDD<String> in IJV format

    • toJavaRDDStringCSV

      public org.apache.spark.api.java.JavaRDD<String> toJavaRDDStringCSV()
      Obtain the matrix as a JavaRDD<String> in CSV format
      Returns:
      the matrix as a JavaRDD<String> in CSV format

    • toRDDStringCSV

      public org.apache.spark.rdd.RDD<String> toRDDStringCSV()
      Obtain the matrix as a RDD<String> in CSV format
      Returns:
      the matrix as a RDD<String> in CSV format

    • toRDDStringIJV

      public org.apache.spark.rdd.RDD<String> toRDDStringIJV()
      Obtain the matrix as a RDD<String> in IJV format
      Returns:
      the matrix as a RDD<String> in IJV format

    • toDF

      public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> toDF()
      Obtain the matrix as a DataFrame of doubles with an ID column
      Returns:
      the matrix as a DataFrame of doubles with an ID column

    • toDFDoubleWithIDColumn

      public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> toDFDoubleWithIDColumn()
      Obtain the matrix as a DataFrame of doubles with an ID column
      Returns:
      the matrix as a DataFrame of doubles with an ID column

    • toDFDoubleNoIDColumn

      public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> toDFDoubleNoIDColumn()
      Obtain the matrix as a DataFrame of doubles with no ID column
      Returns:
      the matrix as a DataFrame of doubles with no ID column

    • toDFVectorWithIDColumn

      public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> toDFVectorWithIDColumn()
      Obtain the matrix as a DataFrame of vectors with an ID column
      Returns:
      the matrix as a DataFrame of vectors with an ID column

    • toDFVectorNoIDColumn

      public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> toDFVectorNoIDColumn()
      Obtain the matrix as a DataFrame of vectors with no ID column
      Returns:
      the matrix as a DataFrame of vectors with no ID column

    • toBinaryBlocks

      public org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> toBinaryBlocks()
      Obtain the matrix as a JavaPairRDD<MatrixIndexes, MatrixBlock>
      Returns:
      the matrix as a JavaPairRDD<MatrixIndexes, MatrixBlock>

    • toMatrixBlock

      public MatrixBlock toMatrixBlock()
      Obtain the matrix as a MatrixBlock
      Returns:
      the matrix as a MatrixBlock

    • getMatrixMetadata

      public MatrixMetadata getMatrixMetadata()
      Obtain the matrix metadata
      Returns:
      the matrix metadata

    • toString

      public String toString()
      If MatrixObject is available, output MatrixObject.toString(). If MatrixObject is not available but MatrixMetadata is available, output MatrixMetadata.toString(). Otherwise output Object.toString().
      Overrides:
      toString in class Object

    • hasBinaryBlocks

      public boolean hasBinaryBlocks()
      Whether or not this matrix contains data as binary blocks
      Returns:
      true if data as binary blocks are present, false otherwise.

    • hasMatrixObject

      public boolean hasMatrixObject()
      Whether or not this matrix contains data as a MatrixObject
      Returns:
      true if data as binary blocks are present, false otherwise.