Package org.apache.sysds.resource.enumeration

Class Enumerator

java.lang.Object
org.apache.sysds.resource.enumeration.Enumerator
Direct Known Subclasses:
GridBasedEnumerator, InterestBasedEnumerator, PruneBasedEnumerator
public abstract class Enumerator extends Object

  • Field Details

    • DEFAULT_MIN_EXECUTORS

      public static final int DEFAULT_MIN_EXECUTORS
      See Also:

    • DEFAULT_MAX_EXECUTORS

      public static final int DEFAULT_MAX_EXECUTORS
      A reasonable upper bound for the possible number of executors is required to set limits for the search space and to avoid evaluating cluster configurations that most probably would have too high distribution overhead
      See Also:

    • COST_DELTA_FRACTION

      public static final double COST_DELTA_FRACTION
      Time/Monetary delta for considering optimal solutions as fraction
      See Also:

  • Constructor Details

  • Method Details

    • setCostsWeightFactor

      public static void setCostsWeightFactor(double newFactor)

    • setMinTime

      public static void setMinTime(double maxTime)

    • setMinPrice

      public static void setMinPrice(double maxPrice)

    • setCpuQuota

      public static void setCpuQuota(int newQuotaValue)

    • preprocessing

      public abstract void preprocessing()
      Called once to enumerate the search space for VM instances for driver or executor nodes. These instances are being represented as

    • processing

      public void processing()
      Called once after preprocessing to fill the pool with optimal solutions by parsing the enumerated search space. Within its execution the number of potential executor nodes is being estimated (enumerated) dynamically for each parsed executor instance.

    • postprocessing

      public EnumerationUtils.SolutionPoint postprocessing()
      Retrieving the estimated optimal configurations after processing.
      Returns:
      optimal cluster configuration and corresponding costs

    • evaluateSingleNodeExecution

      public abstract boolean evaluateSingleNodeExecution(long driverMemory, int cores)

    • estimateRangeExecutors

      public abstract ArrayList<Integer> estimateRangeExecutors(int driverCores, long executorMemory, int executorCores)
      Estimates the minimum and maximum number of executors based on given VM instance characteristics, the enumeration strategy and the user-defined configurations
      Parameters:
      driverCores - CPU cores for the currently evaluated driver node
      executorMemory - memory of currently evaluated executor node
      executorCores - CPU cores of currently evaluated executor node
      Returns:
      - [min, max]

    • updateOptimalSolution

      public void updateOptimalSolution(double newTimeEstimate, double newMonetaryEstimate, EnumerationUtils.ConfigurationPoint newPoint)
      Invokes the estimation of the time and monetary cost based on the compiled program and the given cluster configurations. Following the optimization strategy, the given current optimal solution and the new cost estimation, it decides if the given cluster configuration can be potential optimal solution having lower cost or such a cost that is negligibly higher than the current lowest one.
      Parameters:
      newTimeEstimate - estimated time cost for the given configurations
      newMonetaryEstimate - estimated monetary cost for the given configurations
      newPoint - new cluster configuration for estimation

    • getInstances

      public HashMap<String,CloudInstance> getInstances()
      Meant to be used for testing purposes
      Returns:
      the available instances for enumeration

    • getDriverSpace

      public EnumerationUtils.InstanceSearchSpace getDriverSpace()
      Meant to be used for testing purposes
      Returns:
      the object representing the driver search space

    • setDriverSpace

      public void setDriverSpace(EnumerationUtils.InstanceSearchSpace inputSpace)
      Meant to be used for testing purposes
      Parameters:
      inputSpace - the object representing the driver search space

    • getExecutorSpace

      public EnumerationUtils.InstanceSearchSpace getExecutorSpace()
      Meant to be used for testing purposes
      Returns:
      the object representing the executor search space

    • setExecutorSpace

      public void setExecutorSpace(EnumerationUtils.InstanceSearchSpace inputSpace)
      Meant to be used for testing purposes
      Parameters:
      inputSpace - the object representing the executor search space

    • getEnumStrategy

      public Enumerator.EnumerationStrategy getEnumStrategy()
      Meant to be used for testing purposes
      Returns:
      applied enumeration strategy

    • getOptStrategy

      public Enumerator.OptimizationStrategy getOptStrategy()
      Meant to be used for testing purposes
      Returns:
      applied optimization strategy

    • getCostsWeightFactor

      public double getCostsWeightFactor()
      Meant to be used for testing purposes
      Returns:
      configured weight factor optimization function 'costs'

    • getMaxTime

      public double getMaxTime()
      Meant to be used for testing purposes
      Returns:
      configured max time for consideration (seconds)

    • getMaxPrice

      public double getMaxPrice()
      Meant to be used for testing purposes
      Returns:
      configured max price for consideration (dollars)

    • getOptimalSolution

      public EnumerationUtils.SolutionPoint getOptimalSolution()
      Meant to be used for testing purposes
      Returns:
      current optimal solution