Warmfusion.co.uk

NFS vs iSCSI - My failed experiment

2025-07-19T00:00:00+01:00

NFS vs iSCSI in a Homelab: A Performance and Setup Experiment

This post documents my experience comparing NFS and iSCSI storage protocols in my homelab setup.

During testing, I’d realised that my NAS was only ever able to provide 20MB/s throughput over the NFS, which significantly bottlenecked performance for my media management and data processing experiments.

This doc describes my (failed) attemps to find which protocol would offer better performance and manageability for media storage.

TLDR: NFS ~25MB/s, isci ~8MB/s with CPU spikes on NAS. NFS sucks, but its ok.

🧪 Test Environment

Client:

Debian Linux (bare-metal) HP-Gen8 MicroServer (16G RAM, Intel Xeon E3-1230 V2)
Connected directly to NAS with a dedicated 1gbit/s path
- Jumbo frames enabled (MTU 9000) and verified

NAS:

Asustor AS6706T
- 6x 3TB ST33000650NS CMR SeaGate enterprise NAS drives
- 2x 1TB CT1000P3PSSD8 Crucial M.2 SSD cache drives
Supports NFS and iSCSI
LUN-backed storage volume (~4TB)

📁 NFS Setup and Performance

Systemd Mount Unit

[Unit]
Description=NFS Mount to Media
After=network.target

[Mount]
What=192.168.2.15:/volume1/Media
Where=/mnt/nfs/media/volume1/Media
Type=nfs
Options=_netdev,auto

[Install]
WantedBy=multi-user.target

Viewing the output of mount shows that my Debian OS defaulted the NFS mount with the following options;

vers=4.2,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev

FIO Test Command

The Flexible I/O (FIO) tester is a versatile tool for measuring storage performance. It allows simulating various I/O patterns and workloads to evaluate how storage systems perform under different conditions.

In this test, I used FIO to generate a mixed read/write workload with 128KB block sizes, which is somewhat representative of media file access patterns.

The test runs for 60 seconds using asynchronous I/O (libaio) with a queue depth of 32 to measure sustainable throughput.

fio --name=nfs-test --directory=/mnt/nfs/media/volume1/Media \
    --size=1G --readwrite=rw --bs=128k --ioengine=libaio --iodepth=32 --runtime=60 --time_based

The results however, we’re not promising, and aligned with my observed speeds…

Run status group 0 (all jobs):
   READ: bw=24.7MiB/s (25.9MB/s), 24.7MiB/s-24.7MiB/s (25.9MB/s-25.9MB/s), io=1480MiB (1552MB), run=60019-60019msec
   WRITE: bw=24.7MiB/s (25.9MB/s), 24.7MiB/s-24.7MiB/s (25.9MB/s-25.9MB/s), io=1482MiB (1554MB), run=60019-60019msec

So 25MB/s for both Read and Write… rubbish.

Netcat Bandwidth Test

As those numbers were suspiciously similar, I wondered if there was a network bottleneck causing the issues, so while I have limited SSH access to the Asustor (running BusyBox), I used the handy netcat nc tool, along with pipeviewer pv to create a improvised network speed test…

The process is simple enough, you create a recieving socket that writes to /dev/null with pv -l 12345 > /dev/null and on the sending side you simply write a stack of data with nc <host> 12345. Pop a pv in the middle and get yourself a progress bar to show bandwidth usage.

# Client that recieves with throughput view
nc 192.168.2.15 12345 | pv > /dev/null

# Server that sends with throughput view
pv nfs-test.0.0 | nc -l -p 12345

These tests showed a typical result: ~117MB/s, which is pretty much spot-on 1gbit/s.

🔄 iSCSI Setup and Performance

So… the rabbit hole appears before me… lets see if block devices over iscsi will be performant.

Using the asustor, i setup a new LUN and Mount with a 4TB thinly provisioned disk, setup some CHAPS authentication and went to work connecting it to my server…

Setup

# Install the iscsi stuff
sudo apt install open-iscsi

# And see what you can find announced from the NAS
iscsiadm -m discovery -t sendtargets -p 192.168.2.15

Once you’ve confirmed that the discovery works, you’ll get a lun target such as iqn.2011-08.com.asustor:as6706t-425067.media

So onwards to setting up auth…

# Setup auth
sudo iscsiadm -m node -T iqn.2011-08.com.asustor:as6706t-425067.media -p 192.168.2.15 --op=update -n node.session.auth.authmethod -v CHAP
sudo iscsiadm -m node -T iqn.2011-08.com.asustor:as6706t-425067.media -p 192.168.2.15 --op=update -n node.session.auth.username -v <username>
sudo iscsiadm -m node -T iqn.2011-08.com.asustor:as6706t-425067.media -p 192.168.2.15 --op=update -n node.session.auth.password -v <password>
# Save auth between boots
sudo iscsiadm -m node -T iqn.2011-08.com.asustor:as6706t-425067.media --op update -n node.startup -v automatic


# now login to the LUN and you should find yourself with a nice new drie
sudo iscsiadm -m node -T iqn.2011-08.com.asustor:as6706t-425067.media -p 192.168.2.15 --login

Once the login completes, you’ll find that lsblk shows a new drive (in my case /dev/sdf) which you can go ahead and partition, format, mount etc as if it was a normal block device.

mkfs.ext4 /dev/sdf1
mkdir -p /mnt/media
# Quick mount to test - Note using the UUID ensures that if the drives reorder on boot, you don't mount the wrong things
mount UUID=$(blkid -s UUID -o value "/dev/sdf1") /mnt/media

Systemd Mount Unit

Making the mount paths permanant…

[Unit]
Description=Mount iSCSI Media Volume
Requires=network-online.target
After=network-online.target

[Mount]
What=UUID=446c88c1-a51b-4ec4-9ada-9c18856d4b04
Where=/mnt/media
Type=ext4
Options=defaults,_netdev
TimeoutSec=30

[Install]
WantedBy=multi-user.target

FIO Test Command

fio --name=iscsi-test --directory=/mnt/media \
    --size=1G --readwrite=rw --bs=128k --ioengine=libaio --iodepth=32 --runtime=60 --time_based

So I ran the FIO test again… and…

READ:  ~8.2MB/s
WRITE: ~8.2MB/s

iSCSI performance was significantly lower, with notable CPU spikes observed on the NAS during tests.

What..?

📉 Observations

iSCSI imposed heavy CPU load on the Asustor NAS, degrading throughput; Unclear why but some googling suggests thin-provisioning might be to blame
- media1_* kernel threads pointed to LUN-related processing.
NFS used less CPU and offered better throughput overall.

✅ Final Verdict

NFS “wins” for now… but i’m still pretty disapointed in the overall performance observed.

More research is needed, as I bet theres something DUMB that i need to fix, but despite my efforts so far, i’m still stuck with a strangely slow NAS.

Appendix

🪯 iSCSI Teardown Steps

Unmount and Stop Mount

umount /mnt/media
systemctl stop mnt-media.mount
systemctl disable mnt-media.mount

Logout and Delete Node

iscsiadm -m node -T iqn.2011-08.com.asustor:as6706t-425067.media -p 192.168.2.15 --logout
iscsiadm -m node -o delete -T iqn.2011-08.com.asustor:as6706t-425067.media -p 192.168.2.15

Disable iSCSI Service (Optional)

systemctl disable iscsid
systemctl stop iscsid

NAS GUI Cleanup

Go to: Storage Manager > iSCSI
Unmount and Delete the LUN and targets

NFS vs iSCSI - My failed experiment was originally published by Toby Jackson at Warmfusion.co.uk on July 19, 2025.

IPv6 on my network

2015-09-22T00:00:00+01:00

Whats IPv6?

IPv6 is a ‘new’ method of identifying devices uniquely on the internet which amongst other useful features allows for a considerably larger number of machines to be connected at once. You can find plenty of information about IPv6 on the internet, so rather than trying to repeat other peoples work I’ll just explain my motivations for my experimentation and assume you have some moderate level of technical ability.

Learning experience
- It’s inevitable that IPv6 will become used more and more frequently in my professional work and if I want to remain relevant it makes sense to experiment with this technology sooner rather than later
Internet of Things (IoT)
- As I create or purchase more and more network enabled devices I’m using IP addresses at home but I can’t make them publically accessible - IPv6 lets me trivially access these devices publically
Docker
- Like (1) combined with (2), it lets me explore allocating IPv6 addresses to docker containers and setup more complex network stacks without having to use subnet tricks

Equipment

I’m using an Asus RT-N16 flashed with the DD-WRT firmware connected to a Virgin Media SuperHub set in Modem mode. This means all real network routing work is handled by the Asus device, and I can set it up to manage an IPv6 network transparently across my whole LAN (including WiFi) for clients that can support it.

The Process

Step 1 - Obtain an IPv6 address

I’m using the (Highly recommended) Hurricane Electric TunnelBroker service which provides a free IPv6 /64 range. It can be a little bit technical to setup, but if you’re reading this far I’m going to assume some level of technical ability.

If you select a broker location near home you should get excellent performance; but if you’re feeling adventurous it may be possible to select an international broker endpoint and use your IPv6 traffic as a rudementary proxy

Step 2 - Enable IPv6

Navigate to Administration >> Management and enable IPv6 and Radvd.

In the Radvd configuration section, add the following config block and ensure you update the <REMOTEIP> with your IPv6 prefix that was just created. For example, 2001:db8:0::/64

interface br0 {
  MinRtrAdvInterval 3;
  MaxRtrAdvInterval 10;
  AdvLinkMTU 1480;
  AdvSendAdvert on;
  prefix <REMOTEIP>::/64 {
    AdvOnLink on;
    AdvAutonomous on;
    AdvValidLifetime 86400;
    AdvPreferredLifetime 86400;
    # Base6to4Interface vlan2;
  };
};

Step 3 - Install ip6tables

This step is incomplete - I’ve not worked out how to set this up, and that basically means all my devices are currently exposed to the world-wild-web.

If you want to protect all your IPv6 devices, which will be fully exposed to the internet when they get an IPv6 address, you will either need to run firewalls (or equivalent) on each of the clients, or install ip6tables on your router itself (Or get a dedicated network firewall).

Once again the DD-WRT community come to the rescue with instructions on how to install ip6tables on dd-wrt where the basic idea is:

Download or compile the modules
SCP the .ko files to /jffs/lib/modules/2.6.24.111
- This assumes you’ve setup the Secure Shell access to your router
… TBC

Step 3.1 - Userland things

The ASUS RT-N16 uses a Broadcom chip, so you could download pre-compiled modules from here

Step 4 - Script the IPv6 configuration

Luckily, the DD-WRT community were able to help set this up, with a very simple tutorial

Using the tutorial above, and some tinkering, I was able to get my IPv6 configuration working using the following simple set of commands:

SERVER_IP4=<TunnelBroker Server IPv4 Address>
CLIENT_IP4=<Your Public IPv4>

REMOTE_IP6=<TunnelBroker Server IPv6 Address> # Eg 2001:db8:0:1/64
CLIENT_IP6=<TunnelBroker Client IPv6 Address> # Eg 2001:db8:0:2/64


#Ensure IPv6 modules are loaded
insmod ipv6
# Create a IPv6 over IPv4 tunnel
ip tunnel add he-ipv6 mode sit remote ${SERVER_IP4} local ${CLIENT_IP4} ttl 255
ip link set he-ipv6 up

# Set our Remote IPv6 address on the tunnel device
ip addr add ${REMOTE_IP6} dev he-ipv6

# Transmit all IPv6 traffic through this tunnel
ip route add ::/0 dev he-ipv6

# Setup IPv6 on the Routers internal Bridge device
ip -6 addr add ${CLIENT_IP6} dev br0


# Truth be told - I have no idea why this is needed :-(
ROUTED_ADDRESS=`sed -n -e 's,^ *prefix *\([^ ]*\) *{,\1,p' /tmp/radvd.conf`
BR0_MAC=$(ifconfig br0 |sed -n -e 's,.*HWaddr \(..\):\(..\):\(..\):\(..\):\(..\):\(..\).*,\1\2:\3\4:\5\6,p')
ip -6 addr add $(echo "$ROUTED_ADDRESS"|sed "s,::/..,::$BR0_MAC/64,") dev br0
ip -6 route add 2000::/3 dev he-ipv6

# Let the router boot up, otherwise radvd doesn't work
sleep 5

# Start the Router Advertisement Daemon so machines on the LAN get some IPv6 loveliness
radvd -C /tmp/radvd.conf

Step 5 - Persistence

Goto Administration >> Commands and copy/paste the script above, changing the configuration variables as required, and save as startup.

Reboot the router, and everything should just work.

IPv6 on my network was originally published by Toby Jackson at Warmfusion.co.uk on September 22, 2015.

Musings On Neo4j Graph Database

2014-10-19T00:00:00+01:00

Musings On Neo4j

I’ve just finished watching a GeekOut video entitled Using ASCII art to analyse source code which introduced Neo4j’s Cypher syntax to me with some specific examples of its implementation.

This post is a musing on how to apply Neo4J in the payments world, and the kinds of analysis that would become possible using such a tool. I’ve never used Neo4J, and this is my first experiement in using it. Don’t assume I know what I’m talking about.

The Scenario

There are two main players in the scenario I’m considering, where the payment service provider is simply the mechanism to deliver these functions. The PSP itself may have some needs, but they’re not being considered here.

As a customer, I want to transfer money between myself and merchants so that merchants will provide me the services I require

As a merchant, I want to tools that provide information about my customers and transactions, So that I can make buisness decisions with the best information possible

The Graph

So given the scenario above, and some simple assumptions, I would have a few nodes;

Customer
PaymentMethod
PaymentEvent
Merchant
MerchantBankDetails

I’m not too fussed right now about solving the specific needs of the Users defined above, but I am interested in understanding Neo4J a little more, with some more specific experiments.

Lets Experiment

So without further ado, lets get cracking.

Install

Rather than setting up a local machine, I’m going to use Heroku with a Neo4J addon to make my life simpler.

From the Documentation you just need to;

Create an application: heroku apps:create your-app-name
Add the Database Add-on: heroku addons:add graphenedb
Open the Add-On Administration page: heroku addons:open graphenedb

If that works, you’ll find yourself at the GrapheneDB administration screen on a new heroku application. Magic.

Database

First steps, lets see how to create a new data record:

CREATE (c:customer { name:'Toby' })-[:PAYEE]->(p:payment { id:1234 }),
(m:merchant { name:'Bobs Cars' })-[:PAYMENT_FOR]->(p),
(pm:paymentMethod { type:'VISA', cardHash:'1234abcd'})-[:PAYMENT_METHOD]->(p),
(bd:bankDetails { bank:'BankOfToby', account_number:98765, accepts:['VISA'] })-[:BANK_DETAIL]->(m)

If you get that little record loaded into the administration console, you should be able to do a trivial query and get back a graph that looks something like this (I’ve renamed elements etc here)

MATCH n RETURN n LIMIT 25

Add a link

So we probably want to associate customers with merchants directly, rather than inference through transactions, so we can update the database by performing a MATCH operations where we locate customers who’ve paid for a transaction, and the merchants who’ve had payments made to them, and then joining them update with a ‘customer_of’ relationship.

MATCH (c)-[:PAYEE]->(t)<-[:PAYMENT_FOR]-(m) 
CREATE (c)-[:CUSTOMER_OF]->(m)

Bigger data

Having only a single record isn’t very interesting, so lets up the ante a little with some generated data.

I have created a set of demo data using generatedata.com which contains a bunch of ‘payments’ which you can use. To import into Neo4j you can use the following query:

LOAD CSV WITH HEADERS FROM "https://gist.github.com/warmfusion/90e91fe4e7a142bb2dbb/raw/c3925dff448aa77bb6ac87076e0d9efe5e1ce8c5/demo-dataset" AS csvLine
MERGE (c:customer { name: csvLine.name})
MERGE (pm:paymentMethod {cardNumber: csvLine.cardNumber })
MERGE (m:merchant {name: csvLine.merchant_name})
CREATE (p:payment {status: csvLine.status, amount: toFloat(csvLine.amount) })
CREATE (c)-[:PAYEE]->(p),
(p)-[:PAYMENT_FOR]->(m),
(pm)-[:PAYMENT_METHOD]->(p)

The use of MERGE above means that if Neo4j has already loaded a record that matches the defintion, it updates it, otherwise it’s created. This is particularlly useful for merchant records as we’re using 4 companies over and over and it wouldn’t make sense to create new unique nodes for each.

Note that Payment is a CREATE operation - This is because payments are never used more than once, and realistically the CSV should have some unique ID, but we’ll just create new records instead.

Declined Transactions for Tobys bikes

I’d like to know the transactions which have been declined for Tobys Bikes for a report. This can be done quite simply by creating a Match where we look for merchant by name, and follow any relationship to a payment which has been declined.

MATCH (m:merchant { name:'Tobys Bikes'})--(c:payment { status:'Declined'})
RETURN m,c

Find relationships between customers

Similar to the 6-degrees of Kevin Bacon, its actually very easy to find a relationship between two customers.

The sample dataset includes a few unusual associations, where the same credit card number is used by different customers, or more usually a customer makes payments to a few different merchants.

Because of this, the following query shows there is a relationship between Wynter Crosby and Gary Perry:

MATCH (wynter:customer { name:"Wynter Crosby" }),(gary:customer { name:"Gary Perry" }),
  p = shortestPath((wynter)-[*..15]-(gary))
RETURN p

But what does this really show? Well, in this case it’s telling us that;

Wynter made a payment using a credit card which was used again to pay Daves Trucks.
Gary meanwhile simply made a payment to Daves Trucks as well…

So basically: Someone used Wynters credit card to buy something from the same place Gary did too…

Tip: Madeline Moss is the ‘missing link’ between these customers - She’s used Wynters card :-)

Finding Declines

Wondering how many transactions were declined for each merchant, I found the aggregate functions. This query returns a list of merchants and a sum of the unique payments which were Declined:

MATCH (m:merchant)--(p:payment {status:'Declined'})
RETURN DISTINCT m, count(DISTINCT p), sum(p.amount)

Conclusion

I think Neo4J is remarkably powerful, but I found myself slipping up on the definition of nodes and their relationships; How does a Payment relate to a merchant or a customer, is a customer related to a merchant directly, or just through their payments etc. Having too many relationships would likely impact on performance and scale.

I may experiment futher with considerably larger datasets, but will need to run locally to avoid the 1k node limit on heroku. It seems that mapping between a RDMS with foreign keys would be trivial to implement into a graph database as the join tables or key lookups are just replaced with directional relationships as required. Simple.

Tricks Learnt

Whilst writing this blog post, I found a few things tripped me up as I went. Nothing breaking with Neo4J, but just my approach itself.

Truncate Data

When you’ve got messed up data, you can start fresh by deleting all the records with this;

MATCH (n)
OPTIONAL MATCH (n)-[r]-()
DELETE n,r

Heroku Limit 1k Nodes

If you load too many elements, eg the csv a few times, you may find you go over the 1k node limit on the free graphene installation

You’ll need to totally remove and readd the addon as once it goes read-only you can’t even delete nodes… :-(

Directionality

I noticed that I’d gotten the Bank and Merchant relationship wrong earlier.. We actually want to point merchant to a bank detail. So to swap this relationship around, you find all the current nodes with the existing relationship, make a new one that points the other way, and remove the original relationships:

MATCH (bank)-[r:BANK_DETAIL]->(merchant)
CREATE (merchant)-[r2:BANK_DETAIL]->(bank)
// Changes scope back to r 
WITH r
DELETE r

Musings On Neo4j Graph Database was originally published by Toby Jackson at Warmfusion.co.uk on October 19, 2014.

Bringing the dashboards

2014-04-01T00:00:00+01:00

One of my passions is to support the people around me to achieve the best they can. Be that by simply providing a listening post to help answer questions in the form of Rubber Ducking, supporting tools that help developers build and validate code, through to providing “inspiration” for new ways of working.

I’ve always been very interested in presenting information to users in innovative new ways, or perhaps finding innovative new data from the noise. An example of this was undertaken during a hackathon at PayPoint.net using a Raspberry Pi and an instance of Dashing to generate a wallboard for the team as a whole.

Retrospective

We ran the hackathon in early January as a relaxing way to get back into things after a long break over the new year, but after four months I thought it’d be interesting to write up a few of the lessons I learnt during this time.

Gather Yourself

I went into the hackathon with what I considered to be a pretty clear idea of what I was going to do. I wanted to create, using my RPi and Dashing a portal that can sit in the shared area of the office and present “at a glance” pieces of information that don’t require alot of understanding or interpretation to understand, that’d then spur additional questions and start a conversation.

The problem was, I didn’t take stock of what I was able to do with Dashing “out of the box”, and so spent considerable time making fairly elaborate new widgets and views only to decide they were cluttered, badly written or simply “didn’t work”, only to end up using some of the default widgets and simply populating the data from my background jobs.

Moral: Ask what your tools can do for you, not what you can do for your tools.

Take screenshots

I found that one of the most fascinating aspects of my experimenting was reviewing the screenshots I took during the event. Every so often, typically around the same time I made a significant git commit, i’d take a screenshot to commemorate my progress through the 24 hours. This was fabulous documentary evidence when it came to presenting my activities to the rest of the team as it allowed me to present my thinking, the evolution of the delivery and gave some meaning to the eventual delivery. I think it also helped demonstrate the work involved in creating elegant, minimalistic designs for customers.

Moral: Look to where you have been to see where you may go next

Release Soon, Release Often

Within the first half hour I was running the bare dashboard on a second monitor beside my desk, using a built in browser for the Pidori OS that I’d installed. This was a fantastic tool for myself in a number of ways.

Firstly it made people aware of what I was doing, the monitor was clearly visible from anywhere in the office and people were often coming over to see the progress I was making, and importantly making suggestions to how to shape the UX, additional data to consume, and starting to think about new ways to shape their own work, both in the hackathon but also in day to day development, that’d help bring this information out in the future.

The second brilliant thing about running it loud and proud was the quick feedback I obtained when I started pushing the little Raspberry too hard. Running a full GUI and browser, with quite alot of javascript and CSS takes a considerable amount of CPU, with pages often taking 15 seconds to render. This formed a significant part of my minimalisation of the UX and also highlighted quirks in the resolutions being used between my desktop and the monitor attached to the Pi.

Moral: If you show it, they will come. And sometimes bring the sprinkles.

Where are we now?

So four months on, what have we done?

Frankly, nothing. Its incredibly disappointing as the “front-end” components were basically delivered, but one of the hardest things I’ve found is getting data out of our systems in meaningful ways, and ways that don’t consume my whole working day to obtain. The monitor has been moved, the Pi has returned to my laboratory (i.e. my loft) and noone seems to have really noticed/cared.

Where next then?

Rather than end on a downer, I’m going to start kicking up a fuss once again. Martin Owen recently brought a Pi back from the API Conference held in Amsterdam, which has rekindled my interest in such a device. This coupled with some desires for some general ongoing statistics means I’m going to once again grabble with obtaining data from our systems, and representing back to the teams. So to start things off, heres my short list of stuff to start monitoring, recording, and presenting:

Jenkins builds per hour
List of CR’s due for completion in the next week
Issues open per sprint team for current sprint
- I think this is considerably easier with the 6.x JIRA installation we’re now using - (I hope)

And because I’m still an experimenter at heart, and it may be another post:

Cups of tea brewed today
- Set of scales - WH-B05
- Some MSP430 chips
- A couple of Nordic RF transceivers
- Some imagination

Wish me luck…

Bringing the dashboards was originally published by Toby Jackson at Warmfusion.co.uk on April 01, 2014.

Performance Testing in CI

2014-03-24T00:00:00+00:00

Systems like Jenkins, Travis-CI, Bamboo and others make it so easy to automatically build, and functionally test your code that I’m amazed at how little we leverage them for our non-functional testing.

Non-functional testing refers to anything in your application that isn’t necessarily a “yes/no” behaviour; topics such as performance, usability, and security are considered non-functional as your product may well work but may take a week to respond, or make your users weep with frustration at poorly considered workflows.

A topic I’m interested in solving is around bringing what is typically a manual, after the fact action; “Great work on delivering that new feature, now lets see how it works in production” into our day to day build cycles.

I want to understand the ongoing behavioural changes in products by using a combination of automated performance and security testing coupled with periodic human reviews so that when someone changes code under the hood we know the impact it has made on the system.

The problems that need to be solved, or at least embraced are:

Reporting
- How do we observe behaviour over time
Monitoring
- How do we know what caused the change in observed behaviour
Consistency
- Unlike functional testing, non-functional tests are more concerned with the environment upon which you are testing; how we do know the system is in a good steady state before running?
Ease of use
- Theres no point having a framework for these tests if its quickly made obsolete by developers failing to update tests as the product changes

First Steps

As a competent developer I want to understand the performance of my application over time So that I can make appropriate decisions about changing the product

Using a JMeter project performing simple API actions - as its something people tend to know and understand
Setup the performance-plugin in Jenkins
Bring the two together
Setup job for nightly execution
???
Profit

Performance Testing in CI was originally published by Toby Jackson at Warmfusion.co.uk on March 24, 2014.

Setting up Shinken on OMD

2014-03-19T00:00:00+00:00

Whats Shinken

The home page for Shinken has the breakdown on what Shinken is, but as a summary, its a drop in replacement for the Nagios core engine re-written to be 100% compatible with your existing checks, monitors plugins etc using Python as its base. It provides a new WebUI for displaying service issues and supports greater scalability and service monitoring.

I’m very much a beginner, having only thus far installed and setup monitoring on a few hosts, but I hope to be able to integrate Shinken with Graphite, as well as some of my custom information radiators setup in the office.

How do I get it?

I’m using OMD here, which means its pretty simple to configure Shinken as an alternative check core to Nagios.

Known Issues

A bug in OMD that stops you from adding new hosts and applying changes from the check_mk UI. It appears that OMD thinks nagios.cfg exists even when using Shinken and so when applying those changes the system falls over. The workaround is to provide a nagios.cfg from the shinken configuration, as well as configure the check_submission system (I dont understand that bit though).

ln -s /omd/sites/<yoursite>/tmp/shinken/shinken-apache.cfg /omd/sites/<yoursite>/tmp/nagios/nagios.cfg
echo "check_submission = 'pipe'" >> /omd/sites/<yoursite>/etc/check_mk/main.mk

Swap Nagios out

Lets make a new site called “shishi” and swap out Nagios for Shinken

omd create shishi
omd config shishi set CORE shinken

Add Contact Group

To allow users to login to the Shinken Web UI, you need to configure a contact configuration for Nagios (odd but true) Enter the following block of configuration into /omd/sites//etc/nagios/conf.d/contact.cfg and restart your omd site

define contact {
  contact_name                  omdadmin
  alias                         omdadmin admin contact
  host_notification_commands    check-mk-dummy
  service_notification_commands   check-mk-dummy
  host_notification_options     n
  service_notification_options  n
  host_notification_period      24X7
  service_notification_period   24X7
}

define contactgroup {
  contactgroup_name             omdadmin
  alias                         omdadmin admin contact group
  members                       omdadmin
}

Enable MongoDB for Dashboard persistence

Next enable mongodb so Shinken can store its dashboard configuration. The port may need to be changed

omd config shishi set MONGODB on
omd config shishi set MONGODB-TCP_PORT 27018

Add some configuration to mongodb to shinkens module_webui.cfg - Lines 13 , 22-27

define module{
    module_name      WebUI
    module_type      webui
    host             0.0.0.0       ; mean all interfaces
    port             7767
    # CHANGE THIS VALUE!!!!!!!
    auth_secret      CHANGE_ME
    # Advanced options. Do not touch it if you don't
    # know what you are doing
    #http_backend    wsgiref
    # ; can be also : cherrypy, paste, tornado, twisted
    # ; or gevent
    modules          Apache_passwd,Mongodb
    # Modules for the WebUI.
}
define module{
    module_name      Apache_passwd
    module_type      passwd_webui
    # WARNING : put the full PATH for this value!
    passwd           /omd/sites/shishi/etc/htpasswd
}
define module{
  module_name Mongodb
  module_type mongodb
  uri mongodb://localhost:27018
  database shinken
}

Next Steps

Bring The Graphite

use_with_graphite

Reap the rewards

Learn more about what makes Shinken different to Nagios by experience
See if I can get better integration of Monitored events against continuous datasets
- For example, I want to show when a Jenkins build occured on a build node as a vertical line on my CPU monitors for that node.

References

Alot of googling around was needed to bring these pieces together, here are the main sources of my “knowledge”

Quick start installation/configuration of Shinken - how to install bleeding edge shinken in a minute with omd
Logging into Shinken (Contact.cfg) - Adding contacts to Shinken
MongoDB Configuration example taken from - MongoDB in Shinken

Setting up Shinken on OMD was originally published by Toby Jackson at Warmfusion.co.uk on March 19, 2014.

Warmfusion Reborn

2014-03-17T20:48:00+00:00

After a long stay of absence, I’ve decided to try and revive my blogging by using something a bit more portable, and importantly a bit easier to manage.

Using jekyll and quite possibly a bit of bootstrap once I work my way around the framework, I’ll get this blog collecting a few of my project ideas and other bits and bobs

Warmfusion Reborn was originally published by Toby Jackson at Warmfusion.co.uk on March 17, 2014.

Building The Internet

2013-11-03T00:00:00+00:00

Built yourself The Internet

A really simple weekend project using some serious overengineering to create a black box with a blinky light.

List of components;

2x 1.1K resistors
MSP430g2231 Microcontroller
1 Red LED (5mm)
AA Battery holder
Stripboard
Small wires(for jumpers)
Black Hobby Box

I’d have liked a nice LED mount, but Maplin didn’t have any available, so that might be a later improvement.

The Idea - Make a blinky box that matches the “feel” of The Internet from the IT Crowd

I wanted to create a properly wireless device, with as long life as possible. Originally planned to have a few solar cells for indefinite runtime, but I couldn’t find any in my “Spare parts” drawer. Instead, after failing to work out how proper electrical engineers do blinky things, settled on using a microcontroller.

Aside from being horribly overengineered, I can excuse myself by suggesting that it may be extended at later dates with some features like an RGB LED, LDRs etc. For now… shrugs

Again, I needed it to be long lasting; I don’t want to waste batteries on such a thing, so I spent some time looking at running the MSP430 in low power mode, along with some interrupt coding.

The code is available from my GitHub repository here: (warmfusion/TheInternet)[https://github.com/warmfusion/TheInternet]

The basics are;

Set the P1 ports to OUTPUT as this reduces current consumption on the MSP430 as it doesn’t try and read any input
Turn on the input VLO oscillator ready to start counting
Ensure interrupts are going, and go into Low Power Mode 3 (LPM3)
After a delay, the interrupt is called, toggling the LED
As an addition, I change the cycle counter on each toggle so that the LED blink has a longer pause than the on time.

Building The Internet was originally published by Toby Jackson at Warmfusion.co.uk on November 03, 2013.