BlackRay 0.9.0 Manual

BlackRay has several specific expressions, some of them are explained in the following to get a better understanding.

In order to use BlackRay, you need to have a working installation. If BlackRay is not yet installed on your system, please refer to Chapter 2, Installation for further instructions.

After successfully installing BlackRay, the best way to get started is to run the sample programs. They are located in the directory samples of your BlackRay distribution.

Each sample is located within a separate sub-directory and should contain a file named README with instructions on how to run the sample.

In order to run a BlackRay instance, you have to configure your installation first. The BlackRay configuration is located in a centralized configuration file called /opt/blackray/config/blackray.conf. Let's start by creating a simple configuration file for a single blackray instance:

ManagementServer.Endpoints=default -p 8888 
ManagementServer.Instances=Samples 

ManagementServer.Instance.Samples.Name=Samples
ManagementServer.Instance.Samples.Directory=/opt/blackray/data 
ManagementServer.Instance.Samples.Endpoints.Search=default -p 8889 
ManagementServer.Instance.Samples.Endpoints.Infra=default -p 8890 
ManagementServer.Instance.Samples.Autostart=true 
ManagementServer.Instance.Samples.ReadOnly=false 
ManagementServer.Instance.Samples.Log=true 
ManagementServer.Instance.Samples.Log.Directory=/opt/blackray/log 
ManagementServer.Instance.Samples.LazyWrite=true 

	This tells the management server to listen on tcp port 8888. For further information on the syntax of this parameter please take a look at the ManagementServer.Endpoints (string, mandatory) configuration parameter documentation.
	The management server is responsible for managing the instance Samples. Although the management server is able to manage multiple instance, we start off with a single instance for the sake of simplicity.
	Sets the directory where the instance Samples reads and writes its snapshots. Make sure that the directory exists and that the instance server has read and write permissions in this directory.
	Sets the endpoints for querying (port 8889) and loading (port 8890) the instance. The syntax is the same as for the management server.
	Tells the management server to automatically start the instance.
	Allows manipulation of the instance.
	Enables logging. The log files are written to the directory /opt/blackray/log.
	Configures the instance to not write snapshots to disk after modification. Snapshots have to be taken manually.

After the configuration file has been created, it is time to start the BlackRay Data Engine. To do so, you can use the command blackray_start:

shell> /opt/blackray/bin/blackray_start
    __    __           __
   / /_  / /___ ______/ /___________ ___  __
  / __ \/ / __ `/ ___/ //_/ ___/ __ `/ / / /
 / /_/ / / /_/ / /__/ ,> / /  / /_/ / /_/ /
/_.___/_/\__,_/\___/_/|_/_/   \__,_/\__, /
                                   /____/
  Blackray Management Server Administrator V0.9.0

     [Starting] Management Server
      [Started] Management Server
shell>

For stopping the BlackRay Data Engine, there exists a similar command named blackray_stop.

After startup, check if everything is working as it should by connecting to the instance on the command line interface (blackray_cli):

shell> /opt/blackray/bin/blackray_cli
    __    __           __
   / /_  / /___ ______/ /___________ ___  __
  / __ \/ / __ `/ ___/ //_/ ___/ __ `/ / / /
 / /_/ / / /_/ / /__/ ,< / /  / /_/ / /_/ /
/_.___/_/\__,_/\___/_/|_/_/   \__,_/\__, /
                                   /____/
  Blackray Command Line Interface V0.9.0

blackray_cli> connect to 'default -p 8889'; 
blackray_cli> show schemas; 

|-----------|------|
| name      | size |
|-----------|------|
| <Default> | 0    |
|-----------|------|

1 row selected.

blackray_cli> disconnect; 
blackray_cli> quit; 
shell> 

	Connect to the search endpoint of your sample instance running at port 8889. Note that you can also specify the endpoint by starting the command line interface with blackray_cli -e "default -p 8889".
	Issue a command to show the schemas of the instance. Currently, only the default schema is defined because we have just started the instance without loading any data.
	Disconnect and exit the BlackRay CLI.

Now that the BlackRay server is up and running we should go ahead and load some data into it. This is done by using the blackray_loader command.

First we have to create a loader definition file. Let's name it load.xml:

<load-script>

 <table-def name="my_table" schema="my_schema"> 
  <column-def name="my_column" type="string" tokenizing=" "
                 searchable="yes" compress="no" wildcards="yes" /> 
 </table-def>

 <load-data table="my_table" schema="my_schema" file="data.csv"> 
  <column name="my_column" csv-column="0"> 
   <function type="trim" /> 
  </column>
 </load-data>

</load-script>

	Defines a table named my_table in the schema named my_schema.
	The schema holds a single column named my_column. The column is of type string and a search index is created for this column, as the attribute searchable is set to yes. The content of this column is tokenized with an empty space as delimiter (parameter tokenized), so you can do a full text search in this column. Also, searching with leading wildcards is enabled, as the attribute wildcards is set to yes. Note that trailing wildcards are always enabled by default if the column is searchable.
	Instructs the loader to load data into our table my_table from the file data.csv.
	The data for the column my_column is loaded from the column at position 0 in the csv file.
	Additionally, the column is trimmed before it is loaded into the database. If the sequence of columns in my_table is the same as in the csv file, and if you do not want to specify some additional load functions, you can omit the definition of the columns for the loader.

Now you need to fill your csv file data.csv with some content:

"Lorem ipsum dolor sit amet",0
" consectetuer sadipscing elitr  ",1
"sed diam nonumy eirmod tempor",2

To load the data into your running instance, use the command blackray_loader (let's assume your csv data file is contained in the directory /tmp):

shell> /opt/blackray/bin/blackray_loader -c load.xml -d /tmp \
       -e "default -p 8890"
    __    __           __
   / /_  / /___ ______/ /___________ ___  __
  / __ \/ / __ `/ ___/ //_/ ___/ __ `/ / / /
 / /_/ / / /_/ / /__/ ,< / /  / /_/ / /_/ /
/_.___/_/\__,_/\___/_/|_/_/   \__,_/\__, /
                                   /____/
  Blackray Loader V0.9.0

Connecting to endpoint: default -p 8890
Creating schema my_schema
Creating table my_schema.my_table
Loading data into table my_schema.my_table from file data.csv
shell>

Now that you have created your first BlackRay table and loaded some data into it, you can start playing around on the command line interface:

shell> /opt/blackray/bin/blackray_cli -e "default -p 8889" -q
blackray_cli> select * from my_schema.my_table;

|-------------------------------|
| my_column                     |
|-------------------------------|
| Lorem ipsum dolor sit amet    |
| consectetuer sadipscing elitr |
| sed diam nonumy eirmod tempor |
|-------------------------------|

3 rows selected.

blackray_cli> select * from my_schema.my_table where my_column = 'dolor';

|----------------------------|
| my_column                  |
|----------------------------|
| Lorem ipsum dolor sit amet |
|----------------------------|

1 row selected.

blackray_cli> select * from my_schema.my_table where my_column like '%m';

|-------------------------------|
| my_column                     |
|-------------------------------|
| Lorem ipsum dolor sit amet    |
| sed diam nonumy eirmod tempor |
|-------------------------------|

2 rows selected.

blackray_cli> 

Once your data has been loaded into blackray, you can make the data permanent by creating a snapshot. This way, your data will automatically be reloaded when your instance is restarted. To do so, use the command blackray_inst_take_snapshot to take a snapshot of your Samples instance:

shell> /opt/blackray/bin/blackray_inst_take_snapshot Samples
    __    __           __
   / /_  / /___ ______/ /___________ ___  __
  / __ \/ / __ `/ ___/ //_/ ___/ __ `/ / / /
 / /_/ / / /_/ / /__/ ,< / /  / /_/ / /_/ /
/_.___/_/\__,_/\___/_/|_/_/   \__,_/\__, /
                                   /____/
  Blackray Management Server Administrator V0.9.0

Taking snapshot on Instance Server: Samples.
Taking snapshot on Instance Server: Samples finished after 0s.
shell> 

Binary distributions are the easiest way to install BlackRay on your system. Visit the BlackRay website at http://www.blackray.org and download the distribution for your operating system. If there is no binary distribution available for your environment, you can try to install BlackRay from the source distribution.

	Note
	Installation of software packages usually requires administrator privileges. The following instructions assume that you are performing the required steps as root. When unsure, please contact your system administrator.

shell> cd /opt
shell> tar zxf blackray_VERSION_SYSTEM.tar.gz

shell> rpm -i blackray-thirdparty_VERSION_SYSTEM.rpm
shell> rpm -i blackray_VERSION_SYSTEM.rpm

shell> dpkg -i blackray-thirdparty_VERSION_SYSTEM.deb
shell> dpkg -i blackray_VERSION_SYSTEM.deb

shell> pkgadd -d blackray-thirdparty_VERSION_SYSTEM.pkg
shell> pkgadd -d blackray_VERSION_SYSTEM.pkg

This chapter describes how to build and install the BlackRay Data Engine from the source code distribution.

shell> gunzip blackray-thirdparty-VERSION.tar.gz
shell> tar xf blackray-thirdparty-VERSION.tar
shell> gunzip blackray-VERSION.tar.gz
shell> tar xf blackray-VERSION.tar

shell> cmake .
shell> make
shell> make install

Installing BlackRay

Before you start building and installing BlackRay from Source, make sure that the third-party libraries are installed on your system. If you want to install them from source, please refer to the section called “Installing the BlackRay Third-Party Libraries”.

The installation procedure for BlackRay can be summarized in the following steps, which will be described in more detail later:

shell> cmake .
shell> make
shell> make test 
shell> make install

This is an optional step. You can skip it if you do not want to run the regression tests.

1. Configure

   Before you can build the BlackRay sources, you have to configure the source tree for your build environment. This is achieved by executing cmake in the root directory of the source tree. Execute cmake . to configure the source tree with the default settings for your build environment. This will generate the Makefiles needed for building BlackRay on your system.

   The compiler used for building the source can be controlled by setting the environment variables CC and CXX. cmake will use the specified compilers to configure your build environment.

   	Note
   	To compile BlackRay on Solaris, you should use the Sun Studio compilers. If it is found on your system, cmake will choose gcc by default. In order to use the Sun compilers on Solaris, you have to invoke cmake by running CC=cc CXX=CC cmake . on the command line.

2. Build

   After you have configured the source tree, you can start building the sources by running GNU make from the toplevel directory of the source tree. To start the build, type make on the command line.

   The build will take several minutes depending on your hardware. So it's time to grab yourself a cup of coffee, or just sit back and relax.

3. Test

   After the binaries have been compiled, you can run the regression tests if you like to. To do so, type make test on the command line.

4. Install

   When BlackRay has been successfully built, you can install it by entering make install. BlackRay will be installed in the directory /opt/blackray. This normally needs to be done with superuser permissions. If you are installing the files as a normal user, make sure you have the appropriate permissions to create the installation directories.

The following parameters control the operation of the management server.

The following parameters control the operation of an instance server. They have to be configured for each instance listed in the ManagementServer.Instances parameter. Replace the variable NAME with the name of each instance from the list.