Scale Out with NuoDB Community Edition
For those of you who aren’t familiar with it, NuoDB Community Edition (CE) is our free developer version of the NuoDB product. It’s for those looking to do research, deploy a database with a small project, or simply get started developing with our technology.
One of the big advantages of using NuoDB is its ability to scale out. However, earlier versions of NuoDB Community Edition didn’t allow you to do this. Deployment was limited to one Transaction Engine (TE) and one Storage Manager (SM) process on a single host.
For those of you who’d like a refresher, NuoDB has a peer-to-peer, two-layer architecture, which includes an in-memory layer of TEs and a storage layer of SMs. The in-memory layer allows the application to naturally build up its own caches of frequently accessed data and the storage layer provides ACID-guarantees and data persistence. Each layer has the ability to elastically scale out (and back) and provide active-active redundancy by simply adding and removing TEs and SMs.
We’ve given NuoDB Community Edition a boost by increasing the number of TEs and hosts you can deploy on. You can now scale out your in-memory transaction layer up to three TEs and deploy NuoDB Community Edition as a distributed database across multiple hosts. This allows you to investigate improved operational workload throughput, fault tolerance, and continuous availability.
We’re excited for you to get your hands on NuoDB CE and try it out! Here’s a straightforward tutorial that will get you up and running with NuoDB Community Edition, including how to:
- Install NuoDB Community Edition
- Create a NuoDB database
- Populate your database with data
- Explore your database using SQL commands and System schema tables
- Connect a simple Java client and discover how NuoDB elastically scales out and back in
Let's get started!
STEP 1: INSTALL NUODB COMMUNITY EDITION
To help with the installation, there’s a cheat sheet of useful commands at the end of this article (we recommend that you open this now in a separate tab).
Running this on MS Windows or Ubuntu Desktop is the easiest as they are not server systems and less configuration is required.
SET UP YOUR ENVIRONMENT
Before installing NuoDB there are some prerequisites:
- Ensure you are running on a 64 bit system
NuoDB CE is not supported on any 32 bit systems (the Windows 32-bit client-only installation is not suitable for what we are about to do).
- Confirm that you are running Java JRE 1.8
This example requires you to be running Java JRE 1.8, either from Oracle’s website (but be aware of the new licensing restrictions) or from the OpenJDK project (see cheat sheet for more information).
- Python V2.7
Python V2.7 (not V3) is required. This is usually pre-installed on Linux machines, but definitely needs installing on Windows (see cheat sheet).
On Windows, reboot after installing Python.
- LINUX ONLY: Disable Transparent Huge Pages (THP)
Transparent Huge Pages (THP) is a Linux feature that reduces overhead when using large amounts of memory. However, THP causes a memory management problem for several database technologies, including NuoDB, and therefore must be disabled.
To determine if transparent huge pages is enabled, run the following command:
[always] madvise neverthen you will need to disable THP. You can disable it for now, until the system is restarted with the following commands:
$ echo madvise | sudo tee -a /sys/kernel/mm/transparent_hugepage/enabled $ echo madvise | sudo tee -a /sys/kernel/mm/transparent_hugepage/defrag
- RHEL/Centos Linux ONLY: Install libcurses
sudo yum install ncurses-compat-libs.x86_64
INSTALL AND SET UP NUODB COMMUNITY EDITION
If you have previously installed NuoDB on your machine, uninstall it now.
Download and install NuoDB Community Edition
Visit the Download page on NuoDB.com to download NuoDB Community Edition. Download and run the installer of your choice. There are four installers available:
- Linux installer for Ubuntu/Debian systems
- Linux installer RPM for RHEL/CentOS systems
- A compressed Linux tar
- An MS Windows executable (not Windows 8 x32 client edition) — accept all defaults when running installer
NOTE: During installation you may be prompted to enter a domain password, however this is an old feature and NuoDB can use TLS instead.
- Windows installer: Enter “bird” (which is what we use in our online examples), just to allow the installer to run (we won’t be using it anyway).
- Linux installer: After installation you may be asked to modify
default.propertiesto set the password manually — there is no need to do this.
After running the installers:
- NuoDB for Linux will be installed into your
/opt/nuodbfolder. Some additional files will be in
NOTE: If you prefer, you can download the Linux compressed tar (
.tar.gz) and the installation will be self-contained wherever you choose to unpack it.
NuoDB for Windows installs by default into
We will refer to this installation directory as
Note: To install on a VM on a public cloud, copy the download link, login to your VM and use
wget to fetch it directly onto the VM, (you may have to install
A NuoDB Domain is a collection of machines (or cloud instances) that are managed by a Domain administrator and have been provisioned to work together to support one or more NuoDB databases. Each database runs on one or more hosts in the Domain.
A NuoDB “database” is therefore a collection of processes working together: one or more TEs, one or more SMs and some Administration Processes (APs) managing them — see Figure 1. Remember that the Community Edition only supports up to three TEs and one SM.
Figure 1: Example NuoDB database architecture
NOTE: A given TE or SM only ever works to support a single database, but it is possible to have multiple TEs or SMs on the same host, each handling different databases.
NuoDB cannot be run immediately after installation; some configuration is required first. Using the editor of your choice, open
nuoadmin.conf. Run as Administrator (Windows) or use
sudo on Linux.
Default locations are different depending on the installation:
● Linux: /etc/nuodb/nuoadmin.conf ● Windows: C:\Program Files\nuodb\etc\nuoadmin.conf ● tar: $NUODB_HOME/etc/nuoadmin.conf
The default setup assumes secure connections between NuoDB’s processes using TLS and certificates. Clearly this is a good thing, but requires keys to be generated and a keystore configured. You could set up a self-signed certificate, but to just get up and running quickly, it is easiest to simply disable this feature.
At the bottom of nuoadmin.conf, look for "
true" and set it to
false. Note that this is a JSON format file.
Figure 2: Disable TLS
Note that this file also defines many other properties such as the server’s id (server0) and the ports that NuoDB processes listen on. There should be no need to change any other properties.
There are two other configuration files in the same directory but there is no need to modify either file:
nuoadmin.rest.yml— allows you to configure both HTTP and HTTPS, payload logging and thread usage for the APs
nuoadmin.logback.xml— fine grained logging control for the administration processes (which are written in Java)
Set Up Your Environment
To run NuoDB commands, make sure the NuoDB bin directory is in your path.
- Linux Installer: put these, or equivalent commands, in your shell’s initialization file (for example
.bashrcif you use BASH)
- Linux Tar: as above setting
- Windows: set
NUODB_HOMEand modify the
PATHusing the System Environment dialog
START NUODB SERVICES
With NuoDB, you can scale out by dynamically provisioning additional in-memory capacity or availability. This is accomplished by running services across virtual, physical, or cloud hosts that enable resources to be pooled and managed collectively. You can configure NuoDB to meet various levels of throughput, response times, and durability requirements.
In our example, we are running everything on a single host for simplicity, so we only need to start a single admin process. However, if we were to run NuoDB on more than one virtual or physical host, then we’d typically run one admin process per host and set up a peer network so they can communicate with each other. Learn more about peering hosts by reading the NuoDB documentation.
NuoDB has recently moved to a new administration layer (replacing the old
nuoagent broker processes). This is nuoadmin and is designed to work in multiple environments such as bare metal, in-house, on public clouds, or when using a containerized system such as Kubernetes. We also think it is much easier to use.
However, it is not the default in NuoDB V3 releases and not yet the default in NuoDB 4 on Windows. To activate nuoadmin, run the script
nuoadmin-enable. Assuming a default installation:
● Linux: $NUODB_HOME/etc/nuoadmin-enable ● Windows: C:\Program Files\nuodb\etc\nuoadmin-enable.bat ● tar: $NUODB_HOME/etc/nuoadmin-enable
This script both starts the nuoadmin service and also disables the old nuoagent processes (the domain password you were prompted about earlier is only used by nuoagent, which is why we won’t be needing it).
The nuoadmin service should now be running, so skip to Validate Your Installation below.
Start NuoDB Services
Otherwise, start the services directly:
sudo service nuoadmin start
- If you have the service command available
sudo $NUODB_HOME/etc/nuoadmin start
- This command works for any Linux installation
- Windows: Open the Services panel, look for nuoadmin. You can control it just like any other service. If you prefer you can use
net start "nuodb admin"
Validate Your Installation
At this point, you’re ready to start using NuoDB! Let’s check the status of our new domain. Run
nuocmd show domain. You should just see the admin process running — note that its default name is server0.
Figure 3: Initial show domain
Also, you can review log files for errors, located in:
If things don’t seem to be working, check to see if the nuoadmin process is actually running (see cheat sheet for details).
STEP 2: CREATE A NUODB DATABASE
THE NUODB COMMAND LINE TOOL
nuocmd is the command line tool you can use to monitor, analyze, and manage your domain and databases. We’ll be using this tool to create a new database and then scale that database by adding and removing TEs.
This tool has many, many options but for now we need to start with these two steps:
- Create a database archive (where the data is stored)
- Start the database processes (SMs and TEs)
There is reference documentation here.
CREATE A SIMPLE DATABASE
For our purposes we can assign the database to any area of disk NuoDB has access to. For example, you might choose:
/tmp/databases —note this will not survive a reboot
~/databases(but see note in red below)
Create this directory if it does not already exist.
- On Linux, ensure NuoDB has access:
sudo chown nuodb:nuodb
- On Linux, ensure the parent directory is accessible
- For example, on RHEL/CentOS
chmod a+x ~is essential
- For example, on RHEL/CentOS
nuocmd create archive --db-name < database name > --server-id < server-id > --archive-path < archive-dir >/< database name >
Let’s create a database called testdb: Recall this machine (NuoDB refer to hosts as servers or nodes) is server0:
● Linux: nuocmd create archive --db-name testdb --server-id server0 --archive-path ~/databases/testdb ● Windows: nuocmd create archive --db-name testdb --server-id server0 --archive-path c:\databases\testdb
Change the database location to suit your system.
- NOTE: In a real deployment you typically mount a fast disk dedicated to NuoDB.
A NuoDB database requires at least two processes to be running — one TE and one SM. For simplicity we will run both on the same host. The command we need is:
nuocmd create database --db-name < database name > --dba-user < user-name > --dba-password < password > --te-server-ids < te-server-id >
NOTE: We are defining the username and password required to access the new database.
In our case, we simply run:
nuocmd create database --db-name testdb --dba-user dba --dba-password dba --te-server-ids server0
If you look in your databases directory you should now see that the
testdb subdirectory is no longer empty. Run show domain again and note that the database is now a “running archive” — it has an SM and a TE managing it.
Congratulations! You’ve created your first NuoDB database. Now it’s time to add in some data using the NuoDB SQL tool.
STEP 3: POPULATE YOUR DATABASE WITH HOCKEY DATA
NuoDB ships with a sample database, which contains data related to North American ice hockey. It’s located in
NUODB_HOME/samples/quickstart/sql/ and we’re going to use these SQL files to populate our new database with hockey data.
NuoDB comes with a command-line SQL tool called
nuosql. It is located in
NUODB_HOME/bin, so it should be in your path. It can be used interactively or to load a file of SQL statements and then exit. This section will use
If you prefer to use a GUI tool, we recommend using DbVisualizer, which comes with a NuoDB integration built in (V10+ is recommended). Other GUI tools work with NuoDB, but you’ll need to configure a suitable driver.
CREATE THE SCHEMA AND LOAD HOCKEY DATA
NUODB_HOME/samples/quickstart/sql your current directory.
Now create the schema and load the Hockey table using
nuosql testdb@localhost --user dba --password dba --file create-db.sql
- On RHEL you may be missing
yum install libnsl.x86_64
nuosqlwill echo everything to the screen as it runs. This is perfectly normal.
- On RHEL you may be missing
Figure 5: Create schema using nuosql
Load the additional hockey data
Enter the following commands to load up the remaining hockey data. This will take a short while and echo a lot of output to the screen.
nuosql testdb@localhost --user dba --password dba --file Players.sql nuosql testdb@localhost --user dba --password dba --file Scoring.sql nuosql testdb@localhost --user dba --password dba --file Teams.sql
STEP 4: EXPLORE YOUR DATABASE USING SQL COMMANDS AND THE SYSTEM TABLES
Beyond providing ACID compliance and elastic scale-out, NuoDB is also easy for developers familiar with SQL databases to work with. NuoDB supports a rich and complete set of SQL functionality, which includes the full ANSI SQL standard as well as additional procedural extensions. It also provides a typical SQL database permission model, giving you control over who or which roles are allowed to perform actions on a schema, table, view, or procedure.
Let’s try a few basic SQL commands to get a sense of what’s in our database.
ENTER AND LEARN ABOUT THE NUODB SQL COMMAND LINE TOOL
NuoDB includes NuoDB SQL, a command line tool to explore a database, execute SQL statements, run scripts, access database metadata information, and perform DBA functions. Read more about it by consulting the NuoDB documentation.
Enter the NuoDB SQL tool by specifying the database you are accessing, your username, and password:
nuosql testdb --user dba --password dba
Once you’re in NuoDB SQL, you will see the
SQL> prompt. To leave enter “
quit” at any time.
You can get help by simply using the
Figure 6: Get nuosql help
EXPLORE YOUR DATABASE
By definition, NuoDB always includes two schemas —
system. The system schema is a read-only schema whereas the user schema can hold tables and data. You can also create additional schema as needed.
show schemas command to take a look at what schemas our database includes:
Figure 7: Show schemas
Where is our hockey data? If you don’t create and specify a new schema for your data, NuoDB will automatically enter data into the user schema by default.
Let’s check by entering the user schema and showing the tables in that schema.
Figure 8: Show tables
We can also look at the data in a specific table:
Figure 9: Show contents of Hockey table
Let’s try something a bit more complex. This query lists all the goalies who played their first NHL game with the league in 2011:
SELECT p.firstname, p.lastname, p.firstnhl, p.lastnhl, s.teamid, s.stint, gamesplayed FROM players p LEFT OUTER JOIN scoring s ON p.playerid = s.playerid AND p.firstnhl = s.year AND s.position = 'G' WHERE p.firstnhl = 2011 AND s.gamesplayed IS NOT NULL ORDER BY LASTNAME,FIRSTNAME,TEAMID;
Figure 10: Result of goalies query
DISCOVER DATABASE INFORMATION USING THE SYSTEM SCHEMA
We’ve been using the NuoDB SQL client to connect to our database. Let’s learn how to get more information about these transactions and how they are occurring. To do this, we can use SQL commands to query the tables found in the System schema.
The next few commands produce long lines of output that are hard to read. So first, tell
nuosql to output data vertically (this command is unique to
nuosql, if you are using a different SQL tool skip this step):
SQL > set output vertical;
Now the queries...
This asks the
user.hockey table for the first three names and positions, and adds the id of the TE that we ran the query on using the built-in
SQL > select name, position, getnodeid() as te from user.hockey limit 3;
Figure 11: Show TE node
Here’s a command that presents information about all current connections to the database.
SQL > select * from system.connections;
Figure 12: Show system connections
You can also show information about each of the TE and SM processes in the database. Each row in the
SYSTEM.NODES table represents one process.
Most NuoDB users do not have DBA permissions and cannot typically use
nuocmd, so the following is a nice alternative to
nuocmd show domain for seeing how many SMs and TEs are running.
SQL > select * from system.nodes;
Figure 13: Show database nodes (processes)
Notice there is one Storage node (the SE) and one Transaction node (the TE).
We now have three NuoDB processes running:
- One AP (the admin process nuoadmin) listening on ports 48004, 48005, and 8888
- One SM listening on port 48006 — node id = 1
- One TE listening on port 48007 — node-id = 2
nuosql for now.
STEP 5: CONNECT A SIMPLE JAVA CLIENT AND DISCOVER HOW NUODB ELASTICALLY SCALES OUT AND BACK IN
Now that we have a populated NuoDB database, we want to see how an application interacts with the database, and how that changes when we add TEs. First, we need to build and run the application.
Download GettingStarted Project
We’ll be using this simple Java application called
GettingStarted.java, available in the nuodb-samples Github repository.
The program runs a SQL query in a loop, in multiple threads. Each thread will make its own connection to NuoDB and run its query 100 times. Application stops after 1s by default, but you can use
-time to specify a different time (in seconds).
Create a new folder somewhere convenient for the repository, then download and navigate to where the
GettingStarted.java file is located.
- If you do not have
gitinstalled, use your favorite browser to open https://github.com/nuodb/nuodb-samples. Click on the “Clone or Download” button and then select “Download ZIP.” Unpack the zip file somewhere convenient.
nuodb-samples/GettingStarted is your current directory.
Compile and Run the Application
The application uses a maven build, but maven is packaged with the code for you and hidden by convenient scripts:
Linux:Update the permissions and run the scripts to build and run the program.
chmod u+x *.sh
Windows:Run the scripts to build and run the program.
run.bat if you didn't use the suggested db name, user, and/or password.
When you run the program, it will report out which TE within NuoDB each thread (task) has connected to in order to execute the SQL command. You can see that when NuoDB consists of one SM and one TE, it will always connect to the single TE (called TE 2 because, if you recall, 2 is its node-id).
Figure 14: Getting Started showing which TE it is using
nuocmd show domain. We have one TE and one SM running on our host (server0). Note the output includes both their node-ids and their start-ids:
Figure 15: Domain node-ids
The node-ids are useful when reporting (such as in the application above).
The start-ids are useful when using
nuocmd (see below).
Let's start two more TEs. Run the following command twice:
nuocmd start process --db-name testdb --server-id server0 --engine-type TE
Note we are specifying server0, so they will both start on this machine.
nuocmd show domain again. Notice there are three TEs running and each has a different node-id: 2, 3 and 4. Once an engine (TE or SM) has been created, we can refer to it uniquely by its start-id. Start ids are never reused, so the next time we run a new engine it will have start-id 4.
Figure 16: Domain with three TEs
GettingStarted.java program again. This time, we see that NuoDB distributes SQL command execution evenly among all TEs (nodes 2, 3, 4), spreading the load using a “round robin” approach.
Figure 17: Getting Started showing multiple TEs
With this scaled out deployment, the processing load is spread across all available Transaction Engines.
Kill a TE — Application Continues Without Interruption
With NuoDB, you can scale out and scale back your processes with no interruption to the application. Let’s try it now. This example is more complicated and uses two terminal/command windows in parallel so read the instructions for this section first before executing it, so you know what you have to do.
- You will need to run up a second terminal/command window. Do so now, then return to the original window.
- Run the application directly from the command line. Set the
timeoption to run the application continuously for 30 seconds.
./run.sh -time 30
run -time 30
- While the program is running, in your second terminal/command window shut down a TE process, essentially scaling-in our NuoDB deployment. In this example, we shut down TE #3 by specifying its start-id (2):
nuocmd shutdown process --start-id 2
By default, NuoDB will perform a graceful shutdown of the process, allowing it to finish any queries it is running.Image
- Once the process is shut down, you can go back to the first terminal/command window running the program and check to see how it handled the loss of TE 3.
NuoDB will generate a transient exception when the client thread (task) tries to communicate with its assigned TE — the one that’s now shut down. However, once the client process retries the request, NuoDB simply connects the client thread to one of the other, still running, TEs. Notice also that the client threads that were not connected to the TE that we shut down continued processing with no loss of service.
You can see that once the affected client threads have reconnected to a running TE, all transactions continue to be completed without interruption.
With traditional monolithic databases (such as Oracle, SQLServer, or MySQL), clients typically have to treat connection errors as fatal. However, with NuoDB, many of these errors are transient instead and the transaction can be retried. As long as there is a TE with availability in the NuoDB database, then a retry using a new connection will subsequently be successful.Image
In this article we have installed NuoDB, allocated a database archive, and run the processes that manage it. We have exercised our database using both raw SQL and a Java application.
Most importantly, we also saw how client load is shared when scaling out the number of TEs and how the client application was able to recover and continue with no loss of data when one of the TEs was deliberately shutdown. This responsive scalability shows how NuoDB provides continuous availability even in cases of hardware, network, and other failures.
For simplicity we’ve deployed all processes on a single machine. However, if we had a more demanding application than our simple
GettingStarted.java program, running all processes on a single machine would limit performance in terms of workload throughput, processing power, and storage capacity.
When NuoDB is deployed across multiple hosts (one for each engine), application performance can improve dramatically.
However, the Community Edition has a restriction. If we repeated the last test, but shut down the SM instead, the application would fail because there is no second SM to take over. NuoDB Enterprise Edition removes this restriction allowing as many TEs and SMs as you may need.
If you'd like to talk to our team about an enterprise license, please let us know — we'd be happy to schedule a time to discuss it.
Here are the additional useful commands mentioned earlier.
Installing OpenJDK Java:
- Go to https://www.azul.com/downloads/zulu
- Enter what you need — 1) Java 8 and 2) Windows
- Make sure to pick the JDK (not the JRE) and the 64-bit version (typically it is the only MSI option)
- Once downloaded, run the installer in the usual way
- Make sure Java is in your path (run
java -versionin a command window)
- Make sure
JAVA_HOMEis set in your environment (see here for how to set Windows environment variables)
- Go to https://www.python.org/downloads/release/python-2714
- Download the 64-bit MSI
- Run installer in the usual way
- Make sure python is on your path —
run python --versionin a command window
- Add the requests library. Assuming you installed Python into the default location, run:
c:\Python27\Scripts\pip install requests
To check if nuoadmin is running, look in the services panel or run
net start | findstr Nuo
nuorestsvc should not be running.
If nuoadmin is running but
nuocmd can’t connect to it (you will get an unable to connect to port 8888 error), try opening a
CMD window as Administrator, then stop and start the service manually using:
net start "NuoDB Admin"
What is your operating system version?
Working with the BASH shell:
sudo apt-get install wget
sudo yum install wget
Installing OpenJDK Java:
sudo yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
sudo apt update
sudo apt install openjdk-8-jdk-headless
- Centos/Ubuntu: If more than one Java is installed
sudo alternatives --config java
- When prompted, enter number to pick the right version
Python is usually pre installed in Linux - make sure it is V2.7 not V3. If necessary, you should be able to get it using
apt-get. If not, you can either download the source from the Python website or you will need to do an Internet search for python on your specific platform to find a pre-built installer.
Installing NuoDB (where
XXX is the version number):
sudo rpm -i nuodb-ce-XXX.x86_64.rpm
sudo dpkg -i nuodb-ce-XXX.x86_64.deb
sudo vi /etc/nuodb/nuoadmin.conf- change
sudo service nuoadmin start
>$ ps -ef | grep nuo nuodb 3976 1 30 13:16 ? 00:00:11 /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/nuodb/crash -Xmx500m -jar /opt/nuodb/jar/nuoadmin.jar ec2-user 4064 3609 0 13:17 pts/0 00:00:00 grep --color=auto nuo
This post was originally published January 31, 2017, but has been revised and updated.