BlackRay 0.10.0 Manual

To install BlackRay from a binary tarball, download the archive matching your system environment from the BlackRay website. The naming scheme for the binary tarball is blackray_VERSION_SYSTEM.tar.gz, where VERSION is the BlackRay version number and SYSTEM describes the operating system and architecture the package was built for. The binary archive contains the third-party libraries as well as the BlackRay binaries, so there is no need to download any additional packages. When the download is finished, install the packages by entering the following commands (Make sure you have administrator privileges):

shell> cd /opt
shell> tar zxf blackray_VERSION_SYSTEM.tar.gz

To install BlackRay from rpm packages, download the packages matching your environment from the BlackRay website. The naming scheme for the packages is blackray_VERSION_SYSTEM.rpm and blackray-thirdparty_VERSION_SYSTEM.rpm, where VERSION is the BlackRay version number and SYSTEM describes the operating system and architecture the package was built for. When the download is finished, install the packages by entering the following commands:

shell> rpm -i blackray-thirdparty_VERSION_SYSTEM.rpm
shell> rpm -i blackray_VERSION_SYSTEM.rpm

To install the BlackRay Debian packages, download the packages matching your environment from the BlackRay website. The naming scheme for the packages is blackray_VERSION_SYSTEM.deb and blackray-thirdparty_VERSION_SYSTEM.deb, where VERSION represents the BlackRay version number and SYSTEM describes the operating system and architecture the package was built for. When the download is finished, you can install the packages by entering the following commands:

shell> dpkg -i blackray-thirdparty_VERSION_SYSTEM.deb
shell> dpkg -i blackray_VERSION_SYSTEM.deb

To install the BlackRay Solaris packages, download the packages matching your environment from the BlackRay website. The naming scheme for the packages is blackray_VERSION_SYSTEM.pkg and blackray-thirdparty_VERSION_SYSTEM.pkg, where VERSION represents the BlackRay version number and SYSTEM describes the operating system and architecture the package was built for. When the download is finished, you can install the packages by entering the following commands:

shell> pkgadd -d blackray-thirdparty_VERSION_SYSTEM.pkg
shell> pkgadd -d blackray_VERSION_SYSTEM.pkg

The following prerequisites are needed for building BlackRay from source:

BlackRay source distributions are provided as gzip compressed tar archives. The naming scheme for the distribution is blackray-VERSION.tar.gz, where VERSION is the BlackRay version number (e.g. 0.10.0).

BlackRay requires several third-party libraries. The third-party sources are bundled into a separate package. The naming scheme for the third-party distribution is blackray-thirdparty-VERSION.tar.gz, where VERSION is the BlackRay version number.

For obtaining the latest source distributions, visit the BlackRay website at http://www.blackray.org. After you have downloaded the sources, unpack them by executing the following commands:

shell> gunzip blackray-thirdparty-VERSION.tar.gz
shell> tar xf blackray-thirdparty-VERSION.tar
shell> gunzip blackray-VERSION.tar.gz
shell> tar xf blackray-VERSION.tar

By executing these steps, two directories blackray-thirdparty-VERSION and blackray-VERSION will be extracted, containing the sources of the third-party libraries and the BlackRay sources.

The basic installation procedure for the BlackRay third-party libraries can be summarized by the following steps, which we will describe in more detail afterwards. Please change to the directory containing the third-party sources before executing those steps.

shell> cmake -DCMAKE_INSTALL_PREFIX=/opt/blackray/thirdparty .
shell> make
shell> make install

Before you start building and installing BlackRay from Source, make sure that the third-party libraries are installed on your system. If you want to install them from source, please refer to the section called “Installing the BlackRay Third-Party Libraries”.

The installation procedure for BlackRay can be summarized in the following steps, which will be described in more detail later. Please change to the directory containing the blackray sources before executing those steps.

shell> cmake -DCMAKE_INSTALL_PREFIX=/opt/blackray .
shell> make
shell> make test 
shell> make install

This is an optional step. You can skip it if you do not want to run the regression tests.

BlackRay 0.10.0 introduces a postgres listener for connecting to and querying a running instance.

The BlackRay distribution does not provide any custom drivers. Instead you can use the standard postgres JDBC or ODBC drivers for connecting to blackray. We believe this approach reduces development overhead and enables us to focus on development of the engine itself. The postgresql drivers can be obtained from http://jdbc.postgresql.org/ and http://pgfoundry.org/projects/psqlodbc/.

Currently only a subset of SQL as well as a limited subset of the BlackRay API is supported. For a list of supported commands please refer to Appendix E, SQL Grammar Reference.

shell> psql -h localhost -p 5432 -U blackray
Password for user blackray: 
Welcome to psql 8.3.7 (server 0.10.0), the PostgreSQL interactive \
terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

WARNING:  You are connected to a server with major version 0.10,
but your psql client is major version 8.3.  Some backslash commands,
such as \d, might not work properly.

blackray=> show schemas;
   name    | size 
-----------+------
 <Default> | 0
(1 row)

blackray=>

Querying using the JDBC and ODBC drivers

For programmatic querying, the postgreSQL JDBC and ODBC drivers can be used. The following example illustrates the usage of the JDBC driver for connecting to BlackRay from a Java application.

import java.sql.*;

public class JDBCTest {
  public static void main(String[] argv) {
  
  try {
    Class.forName("org.postgresql.Driver"); 
  } catch (ClassNotFoundException cnfe) {
    System.out.println("Could not register driver");
    cnfe.printStackTrace();
    System.exit(1);
  }
  
  System.out.println("Driver registered, trying to connect");
  
  String user = "blackray"; 
  String pass = "blackray";
  String host = "localhost";
  String db   = "sample";

  try {
    Connection con = DriverManager.getConnection(
        "jdbc:postgresql://" + host + "/" + db, user, pass); 

    System.out.println("Connected, executing query");
    Statement stmt = con.createStatement();
    ResultSet rs = stmt.executeQuery("SHOW SCHEMAS"); 
    while( rs.next() ) { 
      System.out.println(rs.getString(1));
    }   

    rs.close();
    stmt.close();
    con.close();
  } catch (SQLException se) {
    System.out.println("Could not connect:");
    se.printStackTrace();
  }

  }
  
}

	Registers the postgres JDBC driver.
	Defines username, password, hostname, and database name for the connection. The database name is currently ignored by BlackRay, but should be set to the name of the instance.
	Connects to the BlackRay instance.
	Executes a query.

Please make sure that the postgreSQL JDBC driver is in your classpath when trying to run the example:

shell> javac -cp postgresql-8.4-701.jdbc4.jar JDBCTest.java
shell> java -cp postgresql-8.4-701.jdbc4.jar:. JDBCTest
Driver registered, trying to connect
Connected, executing query
<Default>
shell> 

BlackRay also provides a custom API for connecting to a running instance. The API is based on zeroc ICE and can be used for querying data from an instance as well as loading data into a running instance.

While the postgreSQL listener provides a more generic approach, only the custom API currently provides the full featureset. However, future versions of BlackRay will provide all of BlackRay's features for the postgres protocol as well.

The full API is currently available for Java. Rudimentary support for the python language has been added in BlackRay 0.10.0.

The following example demonstrates usage of the API for running simple queries.

import org.blackray.connection.*;
import org.blackray.select.*;
import Ice.InitializationData;

public class SelectWhereSample {

  public static void main(String[] args) {
    InitializationData init = new InitializationData();
    init.properties = Ice.Util.createProperties();

    ConnectionChannel chan = new ConnectionChannel(init,
        "ice://localhost:8889", "Default"); 
    Connection con = chan.getConnection();

    System.out.println("SELECT member_name FROM samples.band_members");
    System.out.println("  WHERE member_name='John';");

    Select select = new Select("samples.band_members"); 

    select.addSelectFieldValue("member_name"); 

    SelectWhere selectWhere = Select.createSelectWhereEqual("member_name",
        "John"); 
    select.setSelectWhere(selectWhere);

    try {
      CommandResult cr = con.execute(select); 
      ResultSet rs = (ResultSet) cr.results.get("resultset"); 

      while (rs.next()) {
        System.out.println(rs.getString(0));
      }   

    } catch (Throwable t) {
      t.printStackTrace();
      System.exit(1);
    } finally {
      try {
        con.close();
      } catch (Throwable t1) {
        t1.printStackTrace();
        System.exit(1);
      }   
      chan.close();
    }   
  }
}

	Creates the connection to the instance Default running on localhost port 8889.
	Creates a new select object for selecting from the table samples.band_members.
	Adds the column member_name to the result.
	Adds a where condition.
	Executes the select.
	Retrieves the resultset.

Please make sure to add BlackRay API driver as well as the ICE API to the classpath when trying to run the example:

shell> javac -cp blackray.jar:Ice.jar SelectWhereSample.java
shell> java -cp blackray.jar:Ice.jar:. SelectWhereSample
Connection to default://localhost:8889 established
Working connection found
SELECT member_name FROM samples.band_members
  WHERE member_name='John';
John Lennon
shell> 

The BlackRay distribution contains several additional examples for demonstrating the usage of the API. They are located in /opt/blackray/share/samples/. For information on how to use them please have a look at the README file in each sample directory.

For a running instance, data can be loaded by the use of the blackray_loader utility. The loader essentially parses the xml load-script, creates the schemas and tables and finally loads the csv data onto the running instance. Optionally, functions can also be added to the load-script by using the function tag in the loader-script.

To load data successfully, blackray_loader accepts 3 mandatory arguments:

For Example:

shell> blackray_loader -c load.xml -d /tmp -e "tcp -p 8890"

One of the most interesting features of blackray is its support of Function Based indices.

Currently 5 types of functions are supported by the loader utility:

Functions can either be applied:

Implementing the both of the above are easy and the former is done by using the function tag of a column in the loader-script. For Example:

<load-script>
...
<load-data table="foo" schema="bar" file="test_data.csv">
<column name="col1" csv-column="0">
<function type="trim"/>
</column>
</load-data>
...
</load-script>

To implement function-based indices, the tags "insert-function" and "select-function" are used under the root node "index". Refer to Appendix C, Load Script Reference for more details and examples.

Interestingly, Functions can also be nested or stacked, providing a rich featureset all together. For Example (Stacked functions):

<load-script>
...
<function type="lower"/>
<function type="trim"/>
...
</load-script>

Caution: When the functions are stacked, the implementation is always "in order". Sometimes, this can totally shadow out other functions. For Example:

<function type="lower" />
<function type="upper" />

As the loader-script is essentially a xml file, multiple Schemas and Table can be defined in the same loader-script file too. The following sample shows how to define multiple tables:

<load-script>

<table-def name="bar1" schema="foo">
<column-def name="col1" type="string" searchable="yes" compress="yes"
            wildcards="yes" />
</table-def>

<table-def name="bar2" schema="foo">
<column-def name="col1" type="string" searchable="yes" compress="yes"
            wildcards="yes" />
</table-def>


<load-data table="bar1" schema="foo" file="test_data.csv">
<column name="col1" csv-column="0">
<function type="trim"/>
<function type="upper" />
</column>
</load-data>

<load-data table="bar2" schema="foo" file="test_data.csv">
<column name="col1" csv-column="0">
<function type="lower"/>
</column>
</load-data>

</load-script>