Openstack installation : Image Services – Glance.

Content from “openstack.org”, listed here with minor changes – just noting down what I did – online notes.

.. will be updated shortly

Posted in Uncategorized | Comments Off on Openstack installation : Image Services – Glance.

Openstack installation : Keystone (Authentication Services)

Content from “openstack.org”, listed here with minor changes – just noting down what I did – online notes.

Create database for keystone services

CREATE DATABASE keystone;
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY '{password}';
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY '{password}';

Apache HTTP server is used to handle authentication requests, install it along with keystone

apt install --assume-yes keystone  apache2 libapache2-mod-wsgi

Update the configuration file : Edit /etc/keystone/keystone.conf file and complete the following actions . Note the section names where to make the configuration updates

[database]
# ...
connection = mysql+pymysql://keystone:{password}@controller/keystone


[token]
# ...
provider = fernet

Note : Comment out any other connection option in database section.

Populate the identity service database and initialize fernet key repositories

# su -s /bin/sh -c "keystone-manage db_sync" keystone
# keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
# keystone-manage credential_setup --keystone-user keystone --keystone-group keystone

Bootstrap the identity service (default domain gets created)

keystone-manage bootstrap --bootstrap-password midlodza --bootstrap-admin-url http://controller:5000/v3/ --bootstrap-internal-url http://controller:5000/v3/ --bootstrap-public-url http://controller:5000/v3/ --bootstrap-region-id RegionOne

Edit /etc/apache2/apache2.conf file and configure the ServerName option to reference the controller node – Add if not already found

ServerName controller

Restart apache and keystone

# service apache2 restart

When using the openstack client to perform operations invariably we need to pass the username, password, authentication URL, domain, etc., as command line parameters. Alternately we can have openstack client specific environment variables hold the value for the same – to avoid providing as command line parameters every time. For convenience create openstack client environment script ‘admin-openrc’ with the following contents

export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD={password}
export OS_AUTH_URL=http://controller:5000/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

Load / Set the client environment variables (not as root user)

$ . admin-openrc

Create a project ‘service’ that contains a unique user for each service that will be added to the environment

sandeep@controller:~$ openstack project create --domain default --description "Service Project" service
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Service Project |
| domain_id | default |
| enabled | True |
| id | 7a9d86ac1bde48eea52ebb562599c9d3 |
| is_domain | False |
| name | service |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+

Verify the functioning of keystone services

sandeep@controller:~$ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2019-01-14T13:36:17+0000 |
| id | gAAAAABcPIJBUrlkBTiqVwkyhmKirFSx5Wnod-4YFeMAAayv2tr_W0nNJgmy_ThI0zyFb0HJ7SweBewFYxlYinymw0DA8iIQIyGU3tqm-9JNj7ZZUS8t4Gr3ndOCzccRYi9NdLXZOhlq8Ye6L1uGqyA0bQjbGZSSSkE_iqunWyysWRjNDTgo9UQ |
| project_id | fa8d2cf9a9ca4ed79c3379de4f215a30 |
| user_id | 580026fd75d3441c9d10c247e1bdf814 |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Posted in Installation / How To, Notes | Tagged , , , , , , | Comments Off on Openstack installation : Keystone (Authentication Services)

Openstack installation : Minimal services and Controller – Pre-requisites

Content from “openstack.org”, listed here with minor changes – just noting down what I did – online notes.

The OpenStack system consists of several key services that are separately installed. These services work together depending on your cloud needs and include the Compute, Identity, Networking, Image, Block Storage, Object Storage, Telemetry, Orchestration, and Database services. You can install any of these projects separately and configure them stand-alone or as connected entities.

For the home lab, will require the following services

On the controller node :

Keystone : Identity Service
Glance : Image services
Nova : Compute services (All except nova-compute)
Neutron : Networking services
Cinder : Block storage (All except cinder-volumes)
Horizon : Dashboard / Management UI

On the compute node :

Nova    : Compute services (nova-compute only)
Neutron : Networking services (linux-bridge-agent only)
Cinder : Block storage (cinder-volumes only)

Note : In view many services, there will be number of passwords to be maintained to be configured for accessing the services. For ease of learning I had preferred to use a single password in all the places – This is fine for learning period. Nevertheless any reference to data surrounded by curly-braces need to be replaced with actual value.

All of the services require a database for managing service entities and end points, and all the services are managed typically by APIs exposed on the controller node and hence a SQL database needs to be installed in controller. For the home lab using MySQL. Install and configure MySQL on controller.

apt install --assume-yes mariadb-server python-pymysql

Create /etc/mysql/mariadb.conf.d/99-openstack.cnf with following contents :

[mysqld]
bind-address = {management-ip-address-of-controller-node}
default-storage-engine = innodb
innodb_buffer_pool_size = 1536000000
innodb_file_per_table = on
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8

Had preferred to configure 1.5G of buffer pool size based on the fact that my controller node has 8G only. No specific data points based on sizing 1.5G. Restart MySQL service on controller node and also secure the installation by setting a password for ‘root’ user on MySQL.

# service mysql restart
# mysql_secure_installation

Communication between services happen via message queues. Install Rabbit MQ on controller and create a user ‘openstack’ with password {message-queue-password} and set required (configuration, read, write) permissions – in our case all. Also tag the user account as administrator account.

# apt install --assume-yes rabbitmq-server
# rabbitmqctl add_user openstack {password-for-rabbit-mq}
# rabbitmqctl set_permissions openstack ".*" ".*" ".*"
# rabbitmqctl set_user_tags openstack administrator

Though the following is not mandatory, preferred to enable the Management Plugins (management via browser UI)

#rabbitmq-plugins enable rabbitmq_management

After restart of the node, had observed in logs that mq srever does not come up successfully, since the IP addresses were not assigned to the interface. Reviewing the systemd service (/lib/systemd/system/rabbitmq-server.service) script for MQ server, noticed the dependency “After=network.target”. Changing the same to “After=network-online.target” helped solve the issue. Also had observed that stopping of rabbitmq service will not complete. Searching for known issues, had come across a solution to change the value for ExecStop to “/usr/sbin/rabbitmqctl shutdown” which did work.

The authentication service uses memcached to cache tokens. So need to install memcached on controller node. (Security aspects wrt using memcached not considered at this stage of learning).

#apt --assume-yes install memcached python-memcache

Edit the binding interface in conf (/etc/memcached.conf) file

-l 10.0.0.15 #Ip Address of controller

Restart memcached after configuration change

#service memcached restart

Openstack services may use etcd – install the same

#apt --assume-yes install etcd

Update the etcd configuration file (/etc/default/etcd) (Note the usage of controller node management IP address)

ETCD_NAME="controller"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER="controller=http://10.0.0.15:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.15:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.15:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.0.15:2379"

Enable and start etcd

# systemctl enable etcd
# systemctl start etcd

Posted in Installation / How To, Notes | Tagged , , , , , , | Comments Off on Openstack installation : Minimal services and Controller – Pre-requisites

Openstack installation : Installing OS, configure pre-requisites and install openstack client

Content from “openstack.org”, listed here with minor changes – just noting down what I did – online notes.

In the course of learning installation of Openstack found that Canonical team maintain separate repositories for ‘Rocky’ release of Openstack. so decided to install Openstack on Ubuntu 18.04 server. One other benefit : no need to worry about installing dirvers.

Did the following post installation of OS, in controller and compute node.

Never end up with issues association failure at application level, due to insufficient file descriptors. edit /etc/security/limits.conf and add the following at end.

*  hard nofile 262140
* soft nofile 262140

Add ‘Rocky’ cloud-archive repository, update repository information and upgrade all that can be

apt install --assume-yes software-properties-common
add-apt-repository cloud-archive:rocky
apt update && apt dist-upgrade

Enable root login – This is something I do in all my systems @ home lab – Not mandatory. edit /etc/ssh/sshd_config, uncomment the following configuration and configure as follows

PermitRootLogin yes

Need to ensure that controller and compute nodes are reachable using hostnames. Edit /etc/hosts and add necessary entries

10.0.0.15       controller
10.0.0.41 iserver
10.0.0.50 aserver

Configure local time

unlink /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Kolkata /etc/localtime

Enable bridge net filtering – will be required prior to having neutron (networking services) configured. Edit /etc/modules-load.d/bridge.conf and add the following line so that the module gets loaded on node startup.

br_netfilter

Create /etc/sysctl.d/bridge.conf and add the following lines

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge-nf-call-arptables = 1

Disable app-armor and remove

systemctl stop apparmor
systemctl disable apparmor
apt purge --assume-yes apparmor

snapd not required. Also we will manage network configurations manually, do not need any network manager – uninstall netplan and snapd and install ifupdown.

apt purge --assume-yes snapd ubuntu-core-launcher squashfs-tools
apt-get --assume-yes purge nplan netplan.io
rm -rf /etc/netplan
rm -rf /usr/share/netplan/netplan/cli/commands/
apt install --assume-yes ifupdown

Configure one interface for overlay network (typically management) and one for provider network via which the guest VMs will get access to external networks. (Should be possible to use more than one interface – yet to learn how-to). Edit /etc/network/interfaces and add the following. Note : interface names could vary in different hosts.

source /etc/network/interfaces.d/*

#The loopback network interface
auto lo
iface lo inet loopback

#Interface for overlay/management
allow-hotplug eno1
auto eno1
iface eno1 inet static
address 10.0.0.31
netmask 255.255.255.0
broadcast 10.0.0.255
gateway 10.0.0.1
dns-nameservers 8.8.8.8 8.8.4.4

#Provider interface
auto eno2
iface eno2 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down

Had some challenges while installing neutron services, had to disable IPv6 for the provider network interface after which it got solved. (Yet to validate if disabling is required), add the following entry in /etc/sysctl.conf.

#eno2 is the interface name
net.ipv6.conf.eno2.disable_ipv6 = 1

Unmask and enable networking service, disable and mask systemd-networkd

systemctl unmask networking

systemctl enable networking

systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online

Edit /etc/systemd/resolved.conf, uncomment DNS= entry and add the DNS server IP

DNS=8.8.4.4

On every ssh login to node, a small delay was observed, figured out that disabling motd-news would help – this could be Ubuntu specific. Edit /etc/default/motd-news and set ENABLED=0

Reboot the system so the networking changes take effect.

Install NTP services on all nodes

apt install --assume-yes chrony

Let one node (controller) synchronize with servers on internet and other nodes synchronize with controller. Edit /etc/chrony/chrony.conf and comment out all pool entries. On controller node update the configuration as below

#pool ntp.ubuntu.com        iburst maxsources 4
#pool 0.ubuntu.pool.ntp.org iburst maxsources 1
#pool 1.ubuntu.pool.ntp.org iburst maxsources 1
#pool 2.ubuntu.pool.ntp.org iburst maxsources 2

server 0.asia.pool.ntp.org iburst
allow 10.0.0.0/24

On compute nodes update as below

#pool ntp.ubuntu.com        iburst maxsources 4
#pool 0.ubuntu.pool.ntp.org iburst maxsources 1
#pool 1.ubuntu.pool.ntp.org iburst maxsources 1
#pool 2.ubuntu.pool.ntp.org iburst maxsources 2

server controller iburst

Restart the NTP service after configuration changes

service restart chrony

Verify the services on controller by checking the output of ‘chronyc sources’ and ‘chronyc clients’ and on compute nodes the output of ‘chronyc sources’.

Install openstack client

apt install --assume-yes python-openstackclient


Posted in Installation / How To, Notes | Tagged , , , , , , | Comments Off on Openstack installation : Installing OS, configure pre-requisites and install openstack client

Openstack Installation : My Requirements

Note : Most of the contents here are from “openstack.org” site.  I have listed in sequence what I did to have open stack up and running in my home lab.  It is just my preference to have the notes online.

Preamble

Until now, had been using KVM for my home lab needs. Working VMs hosted on Openstack environment had observed quite a few issues when testing the application, more specifically clustering. Not observing the same when testing on VMs hosted on KVM, wanted to know more about issues related to using Openstack for hosting computes. Of course liked the ‘networking’ (neutron) services features. So decided to migrate from KVM to Openstack.

Being a Debian fan, had tried to have it installed on Debian stretch. However when I came to know that ‘cloud-archive’ repositories were maintained by Canonical, quickly shifted over to Ubuntu 18.04. The follow sequence of posts are steps I had followed to have a basic functioning Openstack environment at home.

Simple lab setup

Based on my observations working on Openstack environment, had decided to have a dedicated controller node and two compute nodes. Of course the choice was not to install it on VMs, but on physical servers. One other reason to go for dedicated controller, was to make use of a single board computer I had in my inventory. Pretty interesting piece of hardware.

Intel N4200 based SBC, 8 GB DDR4, 128 GB EMMC, 250G SSD, 2 x Gigabit Ethernet Port

For compute node, had decided to used Dell Power Edge R430, 12 Core 24 Threads, 64G DDR4, 600G SAS, 1.8T HDD.

Posted in Installation / How To, Notes | Tagged , , , , , , , | Comments Off on Openstack Installation : My Requirements

Debian – Qualcom 6174 Wifi not working

On my new Acer Nitro laptop, after installing Debian 9.4, found that Wifi was not working.

Looking at dmesg output looked like the Qualcom Athreos 10K was not found.

Installed firmware-linux package, but still did not work.

So visited the site as below

 

Clicked on the commit message on master

 

Clicked the download link fir linux-firmware-master.tar.gz

Expanded the file and copied the ath10k folder to /lib/firware/

Restarted the system and Wifi started working.

Posted in Uncategorized | Tagged , , , , , , | Comments Off on Debian – Qualcom 6174 Wifi not working

Spring 5 REST / GraphDB : Installing ArangoDB

Pr-requisite: http://www.sandeeprao.net/index.php/2018/05/12/spring-5-rest-graph-db-part-2-basic-spring-config-with-simple-controller/

Had used Neo4j in one of thea past projects.

However wanted to check out on ArangoDB, primary factor being Apache 2 License, Kubernetes, active community, supportive engineering team.  Following the development for couple of months now.  Time to get my hands dirty – Here it goes…

Install ArangoDB (Source : https://www.arangodb.com/download-major/debian/ )

root@server:~# curl -OL https://download.arangodb.com/arangodb33/Debian_9.0/Release.key
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 3924 100 3924 0 0 3425 0 0:00:01 0:00:01 --:--:-- 3427
root@server:~# apt-key add - < Release.key
OK
root@server:~# echo 'deb https://download.arangodb.com/arangodb33/Debian_9.0/ /' | sudo tee /etc/apt/sources.list.d/arangodb.list
deb https://download.arangodb.com/arangodb33/Debian_9.0/ /
root@server:~# sudo apt-get install apt-transport-https
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following NEW packages will be installed:
 apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 171 kB of archives.
After this operation, 243 kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian stretch/main amd64 apt-transport-https amd64 1.4.8 [171 kB]
Fetched 171 kB in 1s (123 kB/s) 
Selecting previously unselected package apt-transport-https.
(Reading database ... 108748 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_1.4.8_amd64.deb ...
Unpacking apt-transport-https (1.4.8) ...
Setting up apt-transport-https (1.4.8) ...

root@server:~# apt update
Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Ign:2 http://ftp.us.debian.org/debian stretch InRelease 
Ign:3 https://download.arangodb.com/arangodb33/Debian_9.0 InRelease
Get:4 http://ftp.us.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:5 http://ftp.us.debian.org/debian stretch Release [118 kB]
Get:6 https://download.arangodb.com/arangodb33/Debian_9.0 Release [1,206 B]
Get:8 https://download.arangodb.com/arangodb33/Debian_9.0 Release.gpg [819 B]
Get:9 https://download.arangodb.com/arangodb33/Debian_9.0 Packages [6,067 B]
Fetched 311 kB in 2s (105 kB/s) 
Reading package lists... Done
Building dependency tree 
Reading state information... Done
All packages are up to date.

root@server:~# apt install arangodb3=3.3.8
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following NEW packages will be installed:
 arangodb3
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 35.5 MB of archives.
After this operation, 294 MB of additional disk space will be used.
Get:1 https://download.arangodb.com/arangodb33/Debian_9.0 arangodb3 3.3.8 [35.5 MB]
Fetched 35.5 MB in 36s (961 kB/s) 
Preconfiguring packages ...
Selecting previously unselected package arangodb3.
(Reading database ... 108754 files and directories currently installed.)
Preparing to unpack .../arangodb3_3.3.8_amd64.deb ...
Unpacking arangodb3 (3.3.8) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up arangodb3 (3.3.8) ...
2018-05-13T15:41:41Z [64277] WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 2048000
2018-05-13T15:41:41Z [64277] WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=2048000"'
2018-05-13T15:41:41Z [64277] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2018-05-13T15:41:42Z [64277] INFO {startup} Server will now shutdown due to upgrade, database init or admin restoration.
Database files are up-to-date.
Created symlink /etc/systemd/system/multi-user.target.wants/arangodb3.service → /lib/systemd/system/arangodb3.service.
root@server:~#

Verify the services are started

root@server:~# systemctl status arangodb3
 arangodb3.service - ArangoDB database server
 Loaded: loaded (/lib/systemd/system/arangodb3.service; enabled; vendor preset: enabled)
 Active: active (running) since Sun 2018-05-13 21:11:48 IST; 2min 27s ago
 Process: 64477 ExecStartPre=/bin/chmod 700 /var/lib/arangodb3-apps (code=exited, status=0/SUCCESS)
 Process: 64473 ExecStartPre=/bin/chown -R arangodb:arangodb /var/lib/arangodb3-apps (code=exited, status=0/SUCCESS)
 Process: 64469 ExecStartPre=/bin/chmod 700 /var/lib/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64466 ExecStartPre=/bin/chown -R arangodb:arangodb /var/lib/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64462 ExecStartPre=/bin/chmod 700 /var/log/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64459 ExecStartPre=/bin/chown -R arangodb:arangodb /var/log/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64456 ExecStartPre=/usr/bin/install -g arangodb -o arangodb -d /var/run/arangodb3 (code=exited, status=0/SUCCESS)
 Process: 64452 ExecStartPre=/usr/bin/install -g arangodb -o arangodb -d /var/tmp/arangodb3 (code=exited, status=0/SUCCESS)
 Main PID: 64481 (arangod)
 Tasks: 70 (limit: 131072)
 CGroup: /system.slice/arangodb3.service
 └─64481 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb3/arangod.pid --temp.path /var/tmp/arangodb3 --log.foreground-tty true

May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {authentication} Jwt secret not specified, generating...
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO detected operating system: Linux version 4.16.8 (root@server) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Fri May 11 21:04:11 IST 2018
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 2048000
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=2048000"'
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO using storage engine mmfiles
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {cluster} Starting up with role SINGLE
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {syscall} file-descriptors (nofiles) hard limit is 131072, soft limit is 131072
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
 May 13 21:11:48 server arangod[64481]: 2018-05-13T15:41:48Z [64481] INFO using endpoint 'http+tcp://127.0.0.1:8529' for non-encrypted requests
 May 13 21:11:50 server arangod[64481]: 2018-05-13T15:41:50Z [64481] INFO ArangoDB (version 3.3.8 [linux]) is ready for business. Have fun!

Verify the web interface access

After logging in with the password configured while installing

 

Just a check for the empty graphs view

Proceeding adding code to create a vertex in the graph (in our case a Tenant Entry).

Update pom.xml – Add maven dependency for Arangodb Java driver.  Also we would be using Spring-data – add the necessary dependancies

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersiaon>4.0.0</modelVersion>
  <groupId>GraphDBPoC</groupId>
  <artifactId>GraphDBPoC</artifactId>
  <version>1.0</version>
  <packaging>war</packaging>
  <properties>
    <failOnMissingWebXml>false</failOnMissingWebXml>
    <java.version>1.8</java.version>
    <spring.version>5.0.6.RELEASE</spring.version>
    <servlets.version>3.1.0</servlets.version>
    <jackson.version>2.9.5</jackson.version>
    <arango.driver.version>4.2.2</arango.driver.version>
    <spring.data.version>2.0.0<spring.data.version>
  </properties>
  <dependencies>
    ...
    ...
    ...
    <dependency>
      <groupId>com.arangodb</groupId>
      <artifactId>arangodb-java-driver</artifactId>
      <version>${arango.driver.version}</version>
    </dependency>
    <dependency>
        <groupId>com.arangodb</groupId>
        <artifactId>arangodb-spring-data</artifactId>
        <version>${spring.data.version}</version>
    </dependency>
  </dependencies>
  <build>
    ...
    ...
  </build>
</project>

Will be continuing on this shortly…..

Posted in Uncategorized | Tagged , , , , | Comments Off on Spring 5 REST / GraphDB : Installing ArangoDB

Spring 5 REST / Graph DB : Part 2 – Simple RestController

Pr-requisite : http://www.sandeeprao.net/index.php/2018/04/22/spring-5-rest-arango-graphdb-basic-poc-from-scratch/

It is possible to have the web app deployed in a container without xml configurations (web.xml) and the traditional dispatcher servlet configurations.  Extending AbstractAnnotationConfigDispatcherServletInitializer and providing the implementations for the following should suffice.

getRootConfigClasses() — for “root” application context (non-web infrastructure) configuration.

getServletConfigClasses() — for DispatcherServlet application context (Spring MVC infrastructure) configuration

Had decided to have a package separately for configs with the implementations.

ApplicationConfig.java – Instruct the servlet container to load REST resources from a specific package.

package in.dibs.config;

import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;

@Configuration
@EnableWebMvc
@ComponentScan(basePackages = "in.dibs")
public class ApplicationConfig {
}

ApplicationInitializer.java – initializer class which is loaded at the startup of the application, it defines the configuration class of the application along with the context url.

package in.dibs.config;

import org.springframework.web.servlet.support.AbstractAnnotationConfigDispatcherServletInitializer;

public class ApplicationInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {
  @Override
  protected Class<?>[] getRootConfigClasses() {
    return new Clasas[] { ApplicationConfig.class };
  }

  @Override
  protected Class<?>[] getServletConfigClasses() {
    // No views/JSP for now
    return null;
  }

  @Override
  protected String[] getServletMappings() {
    return new String[] { "/api/*" };
  }
}

Let us have all the message constants at one place so that it becomes easier to handle internationalization – For now just a simple constants package.

APIConstants.java – A simple place holder for constants

package in.dibs.constants;

import java.util.HashMap;
import java.util.Map;

public class APIConstants {
  private static Map<Integer, String> messages = new HashMap<Integer, String>();
  // 0 to 999 Common
  public static final int FAILED = 0;
  public static final int SUCCESS = 1;
  public static final int DATA_NOT_FOUND = 2;
  public static final int DATA_READ_FAILED = 3;
  public static final int INVALID_INPUT = 4;

  // 1000 to 1999 Tenant
  public static final int TENANT_CREATED = 1000;

  // TODO : Move to external resource file
  public static final String MSG_NOT_FOUND = "Matching data for the provided ID not found.";
  public static final String MSG_READ_FAILED = "Error while reading from data store.";
  public static final String MSG_INVALID_INPUT = "One or more information required as input not provided";

  public static final String MSG_TENANT_CREATED = "Tenant : %s, persisted in data store";
  
  static {
    messages.put(DATA_NOT_FOUND, MSG_NOT_FOUND);
    messages.put(DATA_READ_FAILED, MSG_READ_FAILED);
    messages.put(INVALID_INPUT, MSG_INVALID_INPUT);
  }

  public static String getMessage(int code) {
    if (messages.containsKey(code)) {
      return messages.get(code);
    } else {
      return "Unknown status";
    }
  }
}

Placing all entities in the project in appropriate package

Tenant.java – A simple entity to start with

package in.dibs.entity;

public class Tenant {
  private String name;
  private String id;

  public String getName() {
    return name;
  }

  public void setName(String name) {
    this.name = name;
  }

  public String getId() {
    return id;
  }

  public void setId(String id) {
    this.id = id;
  }

  @Override
  public String toString() {
    return "Tenant [name=" + name + ", id=" + id + "]";
  }
}

Placing all the REST API implementations in separate package


APIResponse.java – It would be better to have all API return response in standard format.  Just defining a basic standard

 

package in.dibs.api;

import in.dibs.constants.APIConstants;

public class APIResponse<T> {
  private int status;
  private String message;
  private String systemMessage;
  private T responseObject;

 /**
  * Success scenario with no response object expected.
  * 
  * @param code
  * API operation result code.
  */
  public APIResponse(int code) {
    this.status = code;
    this.message = APIConstants.getMessage(code);
    this.systemMessage = "NA";
    this.responseObject = null;
  }

 /**
  * Success scenario with response object expected. Message to be set by caller.
  * 
  * @param code
  * API Operation result code.
  * @param responseObject
  */
  public APIResponse(int code, T responseObject) {
   this.status = code;
   this.systemMessage = "NA";
   this.responseObject = responseObject;
  }

 /**
  * RT exception occurred.
  * 
  * @param systemMessage
  * Exception from underlying system / framework / library
  */
  public APIResponse(String systemMessage) {
    this.status = APIConstants.FAILED;
    this.message = APIConstants.getMessage(this.status);
    this.systemMessage = systemMessage;
    this.responseObject = null;
  }

 /**
  * Application exception occurred.
  * 
  * @param code
  * API Operation result code.
  * @param systemMessage
  * Associated RT Exception message if any.
  * @param responseObject
  * Response object to be passed on.
  */
  public APIResponse(int code, String systemMessage, T responseObject) {
    this.status = code;
    this.message = APIConstants.getMessage(code);
    this.systemMessage = systemMessage;
    this.responseObject = responseObject;
  }

  public int getStatus() {
    return status;
  }

  public void setStatus(int status) {
    this.status = status;
  }

  public String getMessage() {
    return message;
  }

  public void setMessage(String message) {
    this.message = message;
  }

  public Object getResponseObject() {
    return responseObject;
  }

  public void setResponseObject(T responseObject) {
    this.responseObject = responseObject;
  }

  public String getSystemMessage() {
    return systemMessage;
  }

  public void setSystemMessage(String systemMessage) {
    this.systemMessage = systemMessage;
  }

  @Override
  public String toString() {
    return "APIResponse [status=" + status + ", message=" + message + ", systemMessage=" + systemMessage
    + ", responseObject=" + responseObject.toString() + "]";
  }
}

TenantAPI.java – A Simple REST API

package in.dibs.api;

import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

import in.dibs.constants.APIConstants;
import in.dibs.entity.Tenant;

@RestController
@RequestMapping("/tenant")
public class TenantAPI {
  @RequestMapping(method = RequestMethod.POST)
  public APIResponse<Tenant> createTenant(@RequestBody Tenant tenant) {
    // For now just a success response
    APIResponse<Tenant> response = new APIResponse(APIConstants.TENANT_CREATED, tenant);
    response.setMessage(String.format(APIConstants.MSG_TENANT_CREATED, tenant.getName()));
    return response;
  }
}

Now build the war file

Now the war file is built.  Let us deploy it as Tomcat webapp and start Tomcat.

Testing the API functionality from Postman

At this point we have the base code to start developing REST APIs.  However our objective is the have REST APIs exposed for CRUD operations to persist/fetch from GraphDB.  So let us proceed with having GraphDB persistence layer added in the next part.

 

 

 

 

 

Posted in Uncategorized | Tagged , , , | Comments Off on Spring 5 REST / Graph DB : Part 2 – Simple RestController

Spring 5 REST / GraphDB : Part 1 – Environment Setup

Quite some time since I had worked on web application development.

Have been planning on developing a framework for analytics, where the data sources and relation between sources result in the need of graph databases.

So thought of kick starting a PoC with Spring 5, Tomcat 9, ArangoDB 3.x (>3.3).  Here it goes from scratch.  The intention is to record all kick start procedure so that any new comer to such needs finds it helpful.

Created a VM (Debian Stretch)
Install OpenJDK 1.8
Install Maven 3.3.x

Install Eclipse IDE

Create a new Dynamic Web Project

Convert to Maven Project


Edit build path and remove JRE 1.7 System Library if found

Add JRE 1.8

Edit and update the generated pom.xml – Set properties for versions of Java / Spring / Servlets, Add spring dependencies and update build for deploying to tomcat.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>GraphDBPoC</groupId>
 <artifactId>GraphDBPoC</artifactId>
 <version>1.0</version>
 <packaging>war</packaging>
 <properties>
   <failOnMissingWebXml>false</failOnMissingWebXml>
   <java.version>1.8</java.version>
   <spring.version>5.0.6.RELEASE</spring.version>
   <servlets.version>3.1.0</servlets.version>
 </properties>
 <dependencies>
   <!-- Servlet Dependency -->
   <dependency>
     <groupId>javax.servlet</groupId>
     <artifactId>javax.servlet-api</artifactId>
     <version>${servlets.version}</version>
     <scope>provided</scope>
   </dependency>
   <!-- Spring Dependency -->
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-core</artifactId>
     <version>${spring.version}</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-context</artifactId>
     <version>${spring.version}</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-web</artifactId>
     <version>${spring.version}</version>
   </dependency>
   <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-webmvc</artifactId>
     <version>${spring.version}</version>
     </dependency>
   </dependencies>
   <dependency>
     <groupId>com.fasterxml.jackson.core</groupId>
     <artifactId>jackson-databind</artifactId>
     <version>${jackson.version}</version>
   </dependency>
   <build>
     <sourceDirectory>src/main/java</sourceDirectory>
     <resources>
       <resource>
         <directory>src/main/resources</directory>
       </resource>
     </resources>
     <plugins>
       <plugin>
         <artifactId>maven-compiler-plugin</artifactId>
         <version>3.7.0</version>
         <configuration>
           <source>${java.version}</source>
           <target>${java.version}</target>
         </configuration>
       </plugin>
       <plugin>
         <artifactId>maven-war-plugin</artifactId>
         <version>3.0.0</version>
         <configuration>
           <warSourceDirectory>WebContent</warSourceDirectory>
         </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.tomcat.maven</groupId>
        <artifactId>tomcat7-maven-plugin</artifactId>
        <version>2.2</version>
        <configuration>
          <path>/</path>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Update Maven Project

Install Apache Tomcat

#cd /usr/local/
#wget http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
#tar -xvzf apache-tomcat-9.0.8.tar.gz 
#ln -fs /usr/local/apache-tomcat-9.0.8 tomcat

Optionally create setenv.sh

#cd /usr/local/tomcat
#nano bin/setenv.sh

Contents of file

export CATALINA_OPTS="$CATALINA_OPTS -Xms8192m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx8192m"
export CATALINA_OPTS="$CATALINA_OPTS -Xss1920k"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"

Configure Run Time Environment in Eclipse (Window->Preferences->Server->Runtime Environments)



Configure Server





We are done with setting up the development environment.




 

 

 

 

 

Posted in Uncategorized | Tagged , , , | Comments Off on Spring 5 REST / GraphDB : Part 1 – Environment Setup

Open Day Light – Toaster Tutorial App

Few of my contacts were finding it difficult to get going on with the Open Day Light tutorial app.

Reported that they were not able to cross the first step of creating a simple ‘Example’ project using Maven and an archetype called ‘opendaylight-startup-archetype’.

Tried the following and it worked.

Installed JDK and updated JAVA_HOME environment path variable.
Installed Maven and update MAVEN_HOME and M2_HOME environment variable. Updated PATH environment variable to include Maven bin path.

In home folder created .m2 folder and created settings.xml with the following contents

<?xml version="1.0" encoding="UTF-8"?>
<!-- vi: set et smarttab sw=2 tabstop=2: -->
<!--
 Copyright (c) 2014, 2015 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the
 terms of the Eclipse Public License v1.0 which accompanies this distribution,
 and is available at http://www.eclipse.org/legal/epl-v10.html
-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">

<profiles>
 <profile>
 <id>opendaylight-release</id>
 <repositories>
 <repository>
 <id>opendaylight-mirror</id>
 <name>opendaylight-mirror</name>
 <url>https://nexus.opendaylight.org/content/repositories/public/</url>
 <releases>
 <enabled>true</enabled>
 <updatePolicy>never</updatePolicy>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 </repositories>
 <pluginRepositories>
 <pluginRepository>
 <id>opendaylight-mirror</id>
 <name>opendaylight-mirror</name>
 <url>https://nexus.opendaylight.org/content/repositories/public/</url>
 <releases>
 <enabled>true</enabled>
 <updatePolicy>never</updatePolicy>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </pluginRepository>
 </pluginRepositories>
 </profile>

<profile>
 <id>opendaylight-release</id>
 <repositories>
 <repository>
 <id>opendaylight-release</id>
 <name>opendaylight-release</name>
 <url>https://nexus.opendaylight.org/content/repositories/opendaylight.release/</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 </repositories>
 <pluginRepositories>
 <pluginRepository>
 <id>opendaylight-release</id>
 <name>opendaylight-release</name>
 <url>https://nexus.opendaylight.org/content/repositories/opendaylight.release/</url>
 <releases>
 <enabled>true</enabled>
 </releases>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </pluginRepository>
 </pluginRepositories>
 </profile>
 </profiles>

<activeProfiles>
 <activeProfile>opendaylight-release</activeProfile>
 </activeProfiles>
</settings>

Created sample project using Maven as follows

mvn archetype:generate -DarchetypeGroupId=org.opendaylight.controller -DarchetypeArtifactId=opendaylight-startup-archetype -DarchetypeCatalog=remote -DarchetypeVersion=1.2.3-Boron-SR3

For selecting the archetypeVersion – access the following link in browser and picked one which had the startup-archetype as artificat Id.

https://nexus.opendaylight.org/content/repositories/opendaylight.release/archetype-catalog.xml

Posted in Uncategorized | Tagged , , , , | Comments Off on Open Day Light – Toaster Tutorial App