DMLite: How to use it as a backend in Python LOGO

Introduction

These examples are the same as those explained using the C wrapper and the C++ API, but using the Python wrapper.

Initialize the library and getting a context

We will start with a simple appplication skeleton which does nothing.

#!/usr/bin/env python2
import pydmlite

Now, we want to start using DMLite, so first of all we need to instantiate a PluginManager, which controls the loading and instantiation process of a plugin stack. Then, we will request the manager to use the configuration file specified as the first parameter of the application.

if __name__ == "__main__":
  manager = pydmlite.PluginManager()

  if len(sys.argv) < 2:
    print "Need a configuration file"
    sys.exit(1)

  manager.loadConfiguration(sys.argv[1])

It is that easy.

Stat of a file

Initialize a context

To actually start to perform actions on the namespace, we need to instantiate a StackInstance before, and to set a security context, either directly, or passing some credentials first.

#!/usr/bin/env python2
import pydmlite
import sys

if __name__ == "__main__":
  manager = pydmlite.PluginManager()

  if len(sys.argv) < 3:
    print "Usage:", sys.argv[0], "<configuration file> <path>"
    sys.exit(1)

  manager.loadConfiguration(sys.argv[1])
  stack = pydmlite.StackInstance(manager)
  
  creds = pydmlite.SecurityCredentials()
  creds.clientName   = "/C=CH/O=CERN/OU=GD/CN=Test user 1"
  creds.remoteAdress = "127.0.0.1"
  creds.fqans.append("dteam")

  stack.setSecurityCredentials(creds)

  # Action

  # Done
  sys.exit(0)

Please, note that the client name you are setting must be defined in the file /etc/lcgdm-mapfile, so a default group can be picked. You can always initialize the field fqans to contain at least one group (i.e. dteam), as it is done here.

Perform the action

Let's add the code that is going to actually do something.

  # Action
  catalog = stack.getCatalog()
  xstat = catalog.extendedStat(sys.argv[2], True)
  print "File name: ", xstat.name
  print "File owner:", xstat.stat.st_uid
  print "File group:", xstat.stat.st_gid
  print "File mode: %o" % xstat.stat.st_mode

You should get this output now:

# ./dmlite.py /etc/dmlite.conf /
File name:  /
File owner: 0
File group: 0
File mode: 40755
# ./dmlite.py /etc/dmlite.conf /xx
Traceback (most recent call last):
  File "./dmlite.py", line 25, in 
    xstat = catalog.extendedStat(sys.argv[2], True)
RuntimeError: [#00.000002] Entry 'x' not found under ''

List a directory

For this example, we will reuse the base code as shown just before.

  # Action
  catalog = stack.getCatalog()
  dir = catalog.openDir(sys.argv[2])
  while True:
      try:
        f = catalog.readDirx(dir)
        print "\t%s" % f.name
      except:
        break

The output this time should be:

$ ./dmlite.py /etc/dmlite.conf /
	dpm

Get a physical location where to read

This use case is a little bit trickier, since replicas do not necessarily point to a physical file somewhere. It may be the case, but you should not rely on it, since some translation may be needed (i.e. for Hadoop or S3 pools).

  poolManager = stack.getPoolManager()
  location = poolManager.whereToRead(sys.argv[2])

  for l in location:
    print "Chunk: %s:%s (%d-%d)" % (l.host, l.path, l.offset, l.offset + l.size)
    for k in l.getKeys():
      print "\t%s: %s" % (k, l.getString(k))

The output will depend on what you have on your DPM server, but the output should look something like this:

# ./dmlite.py /etc/dmlite.conf /dpm/cern.ch/home/dteam/image.jpg
Chunk: arioch.cern.ch:/storage/dteam/2012-04-16/image.jpg.1183.0 (0-179181)
	token: v1T+LT2gy1RsWZm0IDdGCV5hdRw=@1348146036@0

Now, this gives the physical location of the replica, but it is not tied to any protocol in particular. The front-end is the responsible of determining which protocol should be used to access the actual data.

For writing, whereToWrite works in a similar manner, but a doneWriting must be done when finished.