by Paul Chapman

Scale Out with NuoDB Community Edition

For those of you who aren’t familiar with it, NuoDB Community Edition (CE) is our free developer version of the NuoDB product. It’s for those looking to do research, deploy a database with a small project, or simply get started developing with our technology.

One of the big advantages of using NuoDB is its ability to scale out. However, earlier versions of NuoDB Community Edition didn’t allow you to do this. Deployment was limited to one Transaction Engine (TE) and one Storage Manager (SM) process on a single host. 

For those of you who’d like  a refresher, NuoDB has a peer-to-peer, two-layer architecture, which includes an in-memory layer of TEs and a storage layer of SMs. The in-memory layer allows the application to naturally build up its own caches of frequently accessed data and the storage layer provides ACID-guarantees and data persistence. Each layer has the ability to elastically scale out (and back) and provide active-active redundancy by simply adding and removing TEs and SMs.

We’ve given NuoDB Community Edition a boost by increasing the number of TEs and hosts you can deploy on. You can now scale out your in-memory transaction layer up to three TEs and deploy NuoDB Community Edition as a distributed database across multiple hosts. This allows you to investigate improved operational workload throughput, fault tolerance, and continuous availability.

We’re excited for you to get your hands on NuoDB CE and try it out! Here’s a straightforward tutorial that will get you up and running with NuoDB Community Edition, including how to:

  1. Install NuoDB Community Edition
  2. Create a NuoDB database
  3. Populate your database with data
  4. Explore your database using SQL commands and System schema tables
  5. Connect a simple Java client and discover how NuoDB elastically scales out and back in

Let's get started!



Our example shows how to deploy NuoDB in a Linux or Windows environment using NuoDB documentation as a supplemental guide. You can download NuoDB Community Edition here.

To help with the installation, there’s a cheat sheet of useful commands at the end of this article (we recommend that you open this now in a separate tab).

Running this on MS Windows or Ubuntu Desktop is the easiest as they are not server systems and less configuration is required.


Before installing NuoDB there are some prerequisites:

  1. Ensure you are running on a 64 bit system
    NuoDB CE is not supported on any 32 bit systems (the Windows 32-bit client-only installation is not suitable for what we are about to do). 
  2. Confirm that you are running Java JRE 1.8
    This example requires you to be running Java JRE 1.8, either from Oracle’s website (but be aware of the new licensing restrictions) or from the OpenJDK project (see cheat sheet for more information).
  3. Python V2.7
    Python V2.7 (not V3) is required. This is usually pre-installed on Linux machines, but definitely needs installing on Windows (see cheat sheet).
    On Windows, reboot after installing Python.

    4.1 Disable Transparent Huge Pages (THP)
    Transparent Huge Pages (THP) is a Linux feature that reduces overhead when using large amounts of memory. However, THP causes a memory management problem for several database technologies, including NuoDB, and therefore must be disabled.

    To determine if transparent huge pages is enabled, run the following command:
    cat /sys/kernel/mm/transparent_hugepage/enabled
    If the result is [always] madvise never then you will need to disable THP. You can disable it for now, until the system is restarted with the following commands:
    $ echo madvise | sudo tee -a
    $ echo madvise | sudo tee -a
    You can also permanently disable THP by editing the kernel bootloader and system startup scripts.

4.2 Install libcurses

You may need to install libcurses to use nuosql:

     Ubuntu/Debian: sudo apt-get install libncurses5 libncursesw5
     RHEL/Centos: sudo yum install ncurses-compat-libs.x86_64


If you have previously installed NuoDB on your machine, uninstall it now.

Download and install NuoDB Community Edition

Visit the Download page on to download NuoDB Community Edition. Download and run the installer of your choice. There are four installers available:

  1. Linux installer for Ubuntu/Debian systems
  2. Linux installer RPM for RHEL/CentOS systems
  3. A compressed Linux tar
  4. An MS Windows executable (not Windows 8 x32 client edition) — accept all defaults when running installer

NOTE: During installation you may be prompted to enter a domain password, however this is an old feature and NuoDB can use TLS instead.

  • Windows installer: Enter “bird” (which is what we use in our online examples), just to allow the installer to run (we won’t be using it anyway).
  • Linux installer: After installation you may be asked to modify to set the password manually — there is no need to do this.

After running the installers:

  • NuoDB for Linux will be installed into your /opt/nuodb folder. Some additional files will be in /var/opt/nuodb and /etc/nuodb
    • NOTE: If you prefer, you can download the Linux compressed tar (.tar.gz) and the installation will be self-contained wherever you choose to unpack it.

  • NuoDB for Windows installs by default into C:\Program Files\nuodb

We will refer to this installation directory as NUODB_HOME.

Note: To install on a VM on a public cloud, copy the download link, login to your VM and use wget to fetch it directly onto the VM, (you may have to install wget first).

System Overview

A NuoDB Domain is a collection of machines (or cloud instances) that are managed by a Domain administrator and have been provisioned to work together to support one or more NuoDB databases. Each database runs on one or more hosts in the Domain.

A NuoDB “database” is therefore a collection of processes working together: one or more TEs, one or more SMs and some Administration Processes (APs) managing them — see Figure 1. Remember that the Community Edition only supports up to three TEs and one SM.

Example NuoDB Database Architecture

Figure 1: Example NuoDB database architecture

NOTE: A given TE or SM only ever works to support a single database, but it is possible to have multiple TEs or SMs on the same host, each handling different databases.

System Setup

NuoDB cannot be run immediately after installation; some configuration is required first. Using the editor of your choice, open nuoadmin.conf. Run as Administrator (Windows) or use sudo on Linux.

Default locations are different depending on the installation:

● Linux:   /etc/nuodb/nuoadmin.conf
● Windows: C:\Program Files\nuodb\etc\nuoadmin.conf
● tar:     $NUODB_HOME/etc/nuoadmin.conf

The default setup assumes secure connections between NuoDB’s processes using TLS and certificates. Clearly this is a good thing, but requires keys to be generated and a keystore configured. You could set up a self-signed certificate, but to just get up and running quickly, it is easiest to simply disable this feature.

At the bottom of nuoadmin.conf, look for "ssl": "true" and set it to false. Note that this is a JSON format file.

Disable TLS

Figure 2: Disable TLS

Note that this file also defines many other properties such as the server’s id (server0) and the ports that NuoDB processes listen on. There should be no need to change any other properties.

There are two other configuration files in the same directory but there is no need to modify either file:

  1. — allows you to configure both HTTP and HTTPS, payload logging and thread usage for the APs
  2. nuoadmin.logback.xml — fine grained logging control for the administration processes (which are written in Java)

Set Up Your Environment

To run NuoDB commands, make sure the NuoDB bin directory is in your path.

  • Linux Installer: put these, or equivalent commands, in your shell’s initialization file (for example .bashrc if you use BASH)
    • export NUODB_HOME=/opt/nuodb
    • export PATH=$PATH:$NUODB_HOME/bin
  • Linux Tar: as above setting NUODB_HOME appropriately
  • Windows: set NUODB_HOME and modify the PATH using the System Environment dialog


With NuoDB, you can scale out by dynamically provisioning additional in-memory capacity or availability. This is accomplished by running services across virtual, physical, or cloud hosts that enable resources to be pooled and managed collectively. You can configure NuoDB to meet various levels of throughput, response times, and durability requirements.

In our example, we are running everything on a single host for simplicity, so we only need to start a single admin process. However, if we were to run NuoDB on more than one virtual or physical host, then we’d typically run one admin process per host and set up a peer network so they can communicate with each other. Learn more about peering hosts by reading the NuoDB documentation.

NuoDB V3.2+

NuoDB has recently moved to a new administration layer (replacing the old nuoagent broker processes). This is nuoadmin and is designed to work in multiple environments such as bare metal, in-house, on public clouds, or when using a containerized system such as Kubernetes. We also think it is much easier to use.

However, it is not the default in NuoDB V3 releases and not yet the default in NuoDB 4 on Windows. To activate nuoadmin, run the script nuoadmin-enable. Assuming a default installation:

● Linux:   $NUODB_HOME/etc/nuoadmin-enable
● Windows: C:\Program Files\nuodb\etc\nuoadmin-enable.bat
● tar:     $NUODB_HOME/etc/nuoadmin-enable

This script both starts the nuoadmin service and also disables the old nuoagent processes (the domain password you were prompted about earlier is only used by nuoagent, which is why we won’t be needing it).

The nuoadmin service should now be running, so skip to Validate Your Installation below.

Start NuoDB Services

Otherwise, start the services directly:

  • Linux:   sudo service nuoadmin start
    • If you have the service command available
  • tar:     sudo $NUODB_HOME/etc/nuoadmin start
    • This command works for any Linux installation
  • Windows: Open the Services panel, look for nuoadmin. You can control it just like any other service. If you prefer you can use net start "nuodb admin"

Validate Your Installation

At this point, you’re ready to start using NuoDB! Let’s check the status of our new domain. Run nuocmd show domain. You should just see the admin process running — note that its default name is server0.

Initial Show Domain

 Figure 3: Initial show domain

Also, you can review log files for errors, located in:

  • Linux: /var/log/nuodb/*.log
  • tar: $NUODB_HOME/var/log
  • Windows: %ALLUSERSPROFILE%\nuodb\LogFiles

If things don’t seem to be working, check to see if the nuoadmin process is actually running (see cheat sheet for details).




nuocmd is the command line tool you can use to monitor, analyze, and manage your domain and databases. We’ll be using this tool to create a new database and then scale that database by adding and removing TEs.

This tool has many, many options but for now we need to start with these two steps:

  • Create a database archive (where the data is stored)
  • Start the database processes (SMs and TEs)

There is reference documentation here.


Allocate Storage

For our purposes we can assign the database to any area of disk NuoDB has access to.  For example, you might choose:

  • Linux:   /tmp/databases — note this will not survive a reboot
  • Linux:   ~/databases (but see note in red below)
  • Windows: C:\databases

Create this directory if it does not already exist.

  • On Linux, ensure NuoDB has access: sudo chown nuodb:nuodb
  • On Linux, ensure the parent directory is accessible
    • For example, on RHEL/CentOS chmod a+x ~ is essential

nuocmd create archive --db-name < database name >
                    --server-id < server-id >
                    --archive-path < archive-dir >/< database name >

Let’s create a database called testdb: Recall this machine (NuoDB refer to hosts as servers or nodes) is server0:

● Linux:   nuocmd create archive --db-name testdb --server-id server0 --archive-path ~/databases/testdb
● Windows: nuocmd create archive --db-name testdb --server-id server0 --archive-path c:\databases\testdb

Change the database location to suit your system.

  • NOTE: In a real deployment you typically mount a fast disk dedicated to NuoDB.

A NuoDB database requires at least two processes to be running — one TE and one SM. For simplicity we will run both on the same host. The command we need is:

nuocmd create database --db-name < database name >
                       --dba-user < user-name >
                       --dba-password < password >
                       --te-server-ids < te-server-id >

NOTE: We are defining the username and password required to access the new database.

In our case, we simply run:

nuocmd create database --db-name testdb --dba-user dba 
--dba-password dba --te-server-ids server0

If you look in your databases directory you should now see that the testdb subdirectory is no longer empty. Run show domain again and note that the database is now a “running archive” — it has an SM and a TE managing it.

Show Domain With Running Archive

Figure 4: Show domain with running archive

Congratulations! You’ve created your first NuoDB database. Now it’s time to add in some data using the NuoDB SQL tool. 



NuoDB ships with a sample database, which contains data related to North American ice hockey. It’s located in NUODB_HOME/samples/quickstart/sql/ and we’re going to use these SQL files to populate our new database with hockey data.

NuoDB comes with a command-line SQL tool called nuosql. It is located in NUODB_HOME/bin, so it should be in your path. It can be used interactively or to load a file of SQL statements and then exit. This section will use nuosql.

If you prefer to use a GUI tool, we recommend using DbVisualizer, which comes with a NuoDB integration built in (V10+ is recommended). Other GUI tools work with NuoDB, but you’ll need to configure a suitable driver.


Make  NUODB_HOME/samples/quickstart/sql your current directory.

Now create the schema and load the Hockey table using create-db.sql.

nuosql testdb@localhost --user dba --password dba --file create-db.sql
    • On RHEL you may be missing libnsl — run yum install libnsl.x86_64
    • nuosql will echo everything to the screen as it runs. This is perfectly normal.
Create Schema Using nuosql

Figure 5: Create schema using nuosql

Load the additional hockey data

Enter the following commands to load up the remaining hockey data. This will take a short while and echo a lot of output to the screen.

nuosql testdb@localhost --user dba --password dba --file Players.sql
nuosql testdb@localhost --user dba --password dba --file Scoring.sql
nuosql testdb@localhost --user dba --password dba --file Teams.sql



Beyond providing ACID compliance and elastic scale-out, NuoDB is also easy for developers familiar with SQL databases to work with. NuoDB supports a rich and complete set of SQL functionality, which includes the full ANSI SQL standard as well as additional procedural extensions. It also provides a typical SQL database permission model, giving you control over who or which roles are allowed to perform actions on a schema, table, view, or procedure.

Let’s try a few basic SQL commands to get a sense of what’s in our database.


NuoDB includes NuoDB SQL, a command line tool to explore a database, execute SQL statements, run scripts, access database metadata information, and perform DBA functions. Read more about it by consulting the NuoDB documentation.

Enter the NuoDB SQL tool by specifying the database you are accessing, your username, and password:

nuosql testdb --user dba --password dba

Once you’re in NuoDB SQL, you will see the SQL> prompt. To leave enter “quit” at any time.

You can get help by simply using the help command.

Get nuosql Help

Figure 6: Get nuosql help


By definition, NuoDB always includes two schemas — user and system. The system schema is a read-only schema whereas the user schema can hold tables and data. You can also create additional schema as needed.

Use the show schemas command to take a look at what schemas our database includes: 

Show Schemas

Figure 7: Show schemas

Where is our hockey data? If you don’t create and specify a new schema for your data, NuoDB will automatically enter data into the user schema by default.

Let’s check by entering the user schema and showing the tables in that schema.

Show Tables

Figure 8: Show tables

We can also look at the data in a specific table:

Show Contents Of Hockey Table

Figure 9: Show contents of Hockey table

Let’s try something a bit more complex. This query lists all the goalies who played their first NHL game with the league in 2011:

SELECT p.firstname, p.lastname, p.firstnhl, p.lastnhl,
       s.teamid, s.stint, gamesplayed
    FROM players p LEFT OUTER JOIN scoring s ON p.playerid = s.playerid AND p.firstnhl = s.year AND s.position = 'G'
    WHERE p.firstnhl = 2011 AND s.gamesplayed IS NOT NULL
Result Of Goalies Query

Figure 10: Result of goalies query


We’ve been using the NuoDB SQL client to connect to our database. Let’s learn how to get more information about these transactions and how they are occurring. To do this, we can use SQL commands to query the tables found in the System schema

The next few commands produce long lines of output that are hard to read. So first, tell nuosql to output data vertically (this command is unique to nuosql, if you are using a different SQL tool skip this step):

SQL > set output vertical;

Now the queries...

This asks the table for the first three names and positions, and adds the id of the TE that we ran the query on using the built-in getnodeid() function.

SQL > select name, position, getnodeid() as te from limit 3;
Show TE Node

Figure 11: Show TE node

Here’s a command that presents information about all current connections to the database.

SQL > select * from system.connections;
Show System Connections

Figure 12: Show system connections

You can also show information about each of the TE and SM processes in the database. Each row in the SYSTEM.NODES table represents one process.

Most NuoDB users do not have DBA permissions and cannot typically use nuocmd, so the following is a nice alternative to nuocmd show domain for seeing how many SMs and TEs are running.

SQL >  select * from system.nodes;
Show Database Nodes Processes

Figure 13: Show database nodes (processes)

Notice there is one Storage node (the SE) and one Transaction node (the TE).

We now have three NuoDB processes running:

  • One AP (the admin process nuoadmin) listening on ports 48004, 48005, and 8888
  • One SM listening on port 48006 — node id = 1
  • One TE listening on port 48007 — node-id = 2

Remember that the TE has node-id 2.

Quit nuosql for now.



Now that we have a populated NuoDB database, we want to see how an application interacts with the database, and how that changes when we add TEs. First, we need to build and run the application.

Download GettingStarted Project

We’ll be using this simple Java application called, available in the nuodb-samples Github repository.

The program runs a SQL query in a loop, in multiple threads. Each thread will make its own connection to NuoDB and run its query 100 times. Application stops after 1s by default, but you can use -time to specify a different time (in seconds).

Create a new folder somewhere convenient for the repository, then download and navigate to where the file is located.

git clone

  • If you do not have git installed, use your favorite browser to open Click on the “Clone or Download” button and then select “Download ZIP.” Unpack the zip file somewhere convenient.

Make sure nuodb-samples/GettingStarted is your current directory. 

Compile and Run the Application

The application uses a maven build, but maven is packaged with the code for you and hidden by convenient scripts:

  • Linux: Update the permissions and run the scripts to build and run the program. 
    • chmod u+x *.sh
    • ./
    • ./
  • Windows: Run the scripts to build and run the program.
    • build.bat
    • run.bat

NOTE: Edit or run.bat if you didn't use the suggested db name, user, and/or password.

When you run the program, it will report out which TE within NuoDB each thread (task) has connected to in order to execute the SQL command. You can see that when NuoDB consists of one SM and one TE, it will always connect to the single TE (called TE 2 because, if you recall, 2 is its node-id).

Getting Started Showing Which TE It Is Using

Figure 14: Getting Started showing which TE it is using

Rerun nuocmd show domain. We have one TE and one SM running on our host (server0). Note the output includes both their node-ids and their start-ids:

Domain Node ids

Figure 15: Domain node-ids

The node-ids are useful when reporting (such as in the application above).

The start-ids are useful when using nuocmd (see below).

Let's start two more TEs. Run the following command twice:

nuocmd start process --db-name testdb --server-id server0 --engine-type TE

Note we are specifying server0, so they will both start on this machine.

Now run nuocmd show domain again. Notice there are three TEs running and each has a different node-id: 2, 3 and 4. Once an engine (TE or SM) has been created, we can refer to it uniquely by its start-id. Start ids are never reused, so the next time we run a new engine it will have start-id 4.

Domain With Three TEs

Figure 16: Domain with three TEs

Run the program again. This time, we see that NuoDB distributes SQL command execution evenly among all TEs (nodes 2, 3, 4), spreading the load using a “round robin” approach.

Getting Started Showing Multiple TEs

Figure 17: Getting Started showing multiple TEs

With this scaled out deployment, the processing load is spread across all available Transaction Engines.

Kill a TE — Application Continues Without Interruption

With NuoDB, you can scale out and scale back your processes with no interruption to the application. Let’s try it now. This example is more complicated and uses two terminal/command windows in parallel so read the instructions for this section first before executing it, so you know what you have to do.

  1. You will need to run up a second terminal/command window. Do so now, then return to the original window.
  2. Run the application directly from the command line. Set the time option to run the application continuously for 30 seconds.
    • Linux:   ./ -time 30
    • Windows: run -time 30
  3. While the program is running, in your second terminal/command window shut down a TE process, essentially scaling-in our NuoDB deployment. In this example, we shut down TE #3 by specifying its start-id (2):

    nuocmd shutdown process --start-id 2

    By default, NuoDB will perform a graceful shutdown of the process, allowing it to finish any queries it is running.
    Shutdown Process
  4. Once the process is shut down, you can go back to the first terminal/command window running the program and check to see how it handled the loss of TE 3.

    NuoDB will generate a transient exception when the client thread (task) tries to communicate with its assigned TE — the one that’s now shut down. However, once the client process retries the request, NuoDB simply connects the client thread to one of the other, still running, TEs. Notice also that the client threads that were not connected to the TE that we shut down continued processing with no loss of service.

    You can see that once the affected client threads have reconnected to a running TE, all transactions continue to be completed without interruption.
    Service Continues No Interruption
    With traditional monolithic databases (such as Oracle, SQLServer, or MySQL), clients typically have to treat connection errors as fatal. However, with NuoDB, many of these errors are transient instead and the transaction can be retried. As long as there is a TE with availability in the NuoDB database, then a retry using a new connection will subsequently be successful.


In this article we have installed NuoDB, allocated a database archive, and run the processes that manage it. We have exercised our database using both raw SQL and a Java application.

Most importantly, we also saw how client load is shared when scaling out the number of TEs and how the client application was able to recover and continue with no loss of data when one of the TEs was deliberately shutdown. This responsive scalability shows how NuoDB provides continuous availability even in cases of hardware, network, and other failures. 

For simplicity we’ve deployed all processes on a single machine. However, if we had a more demanding application than our simple program, running all processes on a single machine would limit performance in terms of workload throughput, processing power, and storage capacity.

When NuoDB is deployed across multiple hosts (one for each engine), application performance can improve dramatically.

However, the Community Edition has a restriction. If we repeated the last test, but shut down the SM instead, the application would fail because there is no second SM to take over. NuoDB Enterprise Edition removes this restriction allowing as many TEs and SMs as you may need.

If you'd like to talk to our team about an enterprise license, please let us know — we'd be happy to schedule a time to discuss it

Follow us on Twitter at @NuoDB and sign up for the NuoDB Blog RSS feed!



Here are the additional useful commands mentioned earlier.

Windows Users

Installing OpenJDK Java:

  • Go to
  • Enter what you need — 1) Java 8 and 2) Windows
  • Make sure to pick the JDK (not the JRE) and the 64-bit version (typically it is the only MSI option)
  • Once downloaded, run the installer in the usual way
  • Make sure Java is in your path (run java -version in a command window)
  • Make sure JAVA_HOME is set in your environment (see here for how to set Windows environment variables)

Installing Python:

  • Go to
  • Download the 64-bit MSI
  • Run installer in the usual way
  • Make sure python is on your path — run python --version in a command window
  • Add the requests library. Assuming you installed Python into the default location, run: c:\Python27\Scripts\pip install requests

To check if nuoadmin is running, look in the services panel or run

  • net start | findstr Nuo

The services nuoagent and nuorestsvc should not be running.

If nuoadmin is running but nuocmd can’t connect to it (you will get an unable to connect to port 8888 error), try opening a CMD window as Administrator, then stop and start the service manually using:

  • net stop
  • net start "NuoDB Admin"

Linux Users

What is your operating system version?

  • Centos/RHEL: hostnamectl or cat /etc/*elease
  • Ubuntu/Debian: lsb_release -a

Working with the BASH shell:

Getting wget:

  • sudo apt-get install wget
  • sudo yum install wget

Installing OpenJDK Java:

  • Centos:
    • sudo yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
  • Ubuntu/Debian
    • sudo apt update
    • sudo apt install openjdk-8-jdk-headless
  • Centos/Ubuntu: If more than one Java is installed
    • Run sudo alternatives --config java
    • When prompted, enter number to pick the right version

Python is usually pre installed in Linux - make sure it is V2.7 not V3.  If necessary, you should be able to get it using yum or apt-get.  If not, you can either download the source from the Python website or you will need to do an Internet search for python on your specific platform to find a pre-built installer.

Installing NuoDB (where XXX is the version number):

  • Centos/RHEL: sudo rpm -i nuodb-ce-XXX.x86_64.rpm
  • Ubuntu/Debian: sudo dpkg -i nuodb-ce-XXX.x86_64.deb

Running nuoadmin

  • sudo vi /etc/nuodb/nuoadmin.conf - change ssl to false
  • sudo service nuoadmin start

Check using ps:

>$ ps -ef | grep nuo
nuodb     3976     1 30 13:16 ?        00:00:11 /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/nuodb/crash -Xmx500m -jar /opt/nuodb/jar/nuoadmin.jar
ec2-user  4064  3609  0 13:17 pts/0    00:00:00 grep --color=auto nuo

This post was originally published January 31, 2017, but has been revised and updated.