I’ve been doing some noodling around with Confluent’s Kafka Connect recently, as part of gaining a wider understanding into Kafka. If you’re not familiar with Kafka Connect this page gives a good idea of the thinking behind it.

One issue that I hit defeated my Google-fu so I’m recording it here to hopefully help out fellow n00bs.

The pipeline that I’d set up looked like this:

• Eneco’s Twitter Source streaming tweets to a Kafka topic
• Confluent’s HDFS Sink to stream tweets to HDFS and define Hive table automagically over them

It worked great, but only if I didn’t enable the Hive integration part. For me the integration with Hive to automatically define schemas was one of the key interests for this platform, so I wanted to see if I could get it to work. The error I got was

With my new server I bought a UPS, partly just as a Good Thing, but also because I suspect a powercut fried the motherboard on a previous machine that I had, and this baby is too precious to lose ;)

The idea is that the UPS will smooth out the power supply to my server, protecting it from surges or temporarily blips in power loss. If there’s a proper power cut, the UPS is connected to my server and can initiate a graceful shutdown instead of system crash. It seems unintuitive in this day and age of laptops and iPads that you just close or switch off to “suspend” them that killing the power to a server can damage it, but when you think about it just a moment more, it’s hardly surprising.

Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error:

java.io.IOException: No input paths specified in job

scala> sqlContext.read.json("/u02/custom/twitter/twitter.json")
java.io.IOException: No input paths specified in job
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202)

Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://:

scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json")
res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string...
scala>

An alternative to file:// is hdfs://, assuming you have some data residing there too:

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32!

On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error

ssh_exchange_identification: read: Connection reset by peer

On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

(Ref)

The bit that caught me out was this kept failing with

Error: Password not present	

and a Python stack trace that ended with

subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1

The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!)

Launch shell:

export HUE_SECRET_KEY=foobar
/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell

Reset password for hue, activate account and make it superuser

from django.contrib.auth.models import User
user = User.objects.get(username='hue')
user.is_active=True
user.save()
user.is_superuser=True
user.save()
user.set_password('hue')
user.save()

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example:

[oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
apache drill 1.6.0
"the only truly happy people are children, the creative minority and drill users"
0: jdbc:drill:zk=local> SELECT version FROM sys.version;
No current connection
0: jdbc:drill:zk=local>

Whether SELECT version FROM sys.version; or any other command - same result - No current connection. Trying to run Drill in distributed mode also failed, with a class error

I wasted literally two hours on this one, so putting down a note to hopefully help future Googlers.

Symptom

Here’s all the various errors that I got in the hive-server2.log during my attempts to get a CREATE EXTERNABLE TABLE to work against a MongoDB table in Hive:

Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.io.BSONWritable
Caused by: java.lang.ClassNotFoundException: com.mongodb.util.JSON
Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson
Caused by: java.lang.ClassNotFoundException: org.bson.io.OutputBuffer

Whilst Hive would throw errors along the lines of:

Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included)

Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included)

Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed:

org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364"
java.lang.NumberFormatException: For input string: "18446744073709550364"

Looking down the stack trace, this came from org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin.readProcMemInfoFile, which the source code shows is reading /proc/meminfo. Looking at this file on each node showed:

I’ve got an EdgeRouter LITE (ERL) which I used as my home router until a powercut fried it a while ago (looks like I’m not the only one to have this issue). The symptoms were it powering on but not giving any DHCP addresses, or after a factory reset responding on the default IP of 192.168.1.1. It was a real shame, because it had been a great bit of kit up until then. I am a complete hack when it comes to networking, and it struck the balance right between letting me do what I needed to do, without overwhelming me with complexity. I’d replaced it with a SonicWall TZ105 but having utterly failed to get the latter to permit OpenVPN traffic (so I can access my home server when on the road), which I had done with no problem on the ERL I thought I’d try and resurrect the ERL using the instructions here.

(Previously, previously, previously)

Since Proxmox 4 has a recent Linux kernel and mainline one at that, it means that Docker can be run on it. I’ve yet to really dig into Docker and work out when it makes sense in place of Linux Containers (LXC), so this is going to be a learning experience for me.

To install Docker, add Backports repo to apt:

root@proxmox01:~# cat /etc/apt/sources.list.d/backports.list
deb http://ftp.debian.org/debian jessie-backports main

And then install:

(Previously, previously)

I’ve got a bunch of existing VirtualBox and VMWare VMs that I want to run on Proxmox. Eventually I’ll migrate them to containers, but for the time being run them as “fat” VMs using Proxmox’s KVM virtualisation. After copying the OVA files that I had to the server, I uncompressed them:

root@proxmox01:/data04/vms/bdl44-biwa# cd ../bdl44
root@proxmox01:/data04/vms/bdl44# ll
total 27249328
-rw------- 1 root root 27903306752 Jun  1 10:14 BigDataLite440.ova
root@proxmox01:/data04/vms/bdl44# tar -xf BigDataLite440.ova
root@proxmox01:/data04/vms/bdl44# ll
total 54498668
-rw------- 1 root root  7300486656 Feb 18 21:25 BigDataLite440-disk1.vmdk
-rw------- 1 root root  1261044224 Feb 18 21:26 BigDataLite440-disk2.vmdk
-rw------- 1 root root 19295202816 Feb 18 21:48 BigDataLite440-disk3.vmdk
-rw------- 1 root root    46550528 Feb 18 21:48 BigDataLite440-disk4.vmdk
-rw------- 1 root root 27903306752 Jun  1 10:14 BigDataLite440.ova
-rw------- 1 root root       19619 Feb 18 21:15 BigDataLite440.ovf

and then converted each disk image to qcow2 format: (You can read more about how and why here and here).

(Previously)

With my server in place, I ran a memtest on it … which with 128G took a while ;)

And then installed Proxmox 4, using a bootable USB that I’d created on my Mac from the ISO downloaded from Proxmox’s website. To create the bootable USB, create the img file:

hdiutil convert -format UDRW -o target.img source.iso

and then burn it to USB:

sudo dd if=target.img of=/dev/rdiskN bs=1m

Replace N with the correct device based on diskutil list output. Don’t get it wrong, else you’ll properly knacker your machine :D

After a long and painful delivery, I’m delighted to announce the arrival of a new addition to my household … :

This custom-build from Scan 3XS is sat in my study quietly humming away. I’m going to use it for hosting VMs for R&D on OBIEE, Big Data Lite, Elastic, InfluxDB, Kafka, etc. I’ll blog various installations that I’ve done on it as a reference for myself, and anyone else interested. Which I guess means, myself ;)

Oracle’s excellent Big Data Lite VM has been updated, to version 4.5.

Download it here

Changes:

• CDH 5.5 -> 5.7
• Big Data Spatial and Graph 1.1 -> 1.2
• Big Data Discovery 1.1 -> 1.2
• Oracle Big Data Connectors 4.4 -> 4.5
• Oracle NoSQL 3.5 -> 4.0
• GoldenGate 12.2.0.1 -> 12.2.0.1.1

I’ve been spending some interesting hours digging into OBIEE 12c recently, with some interesting blog posts to show for it. Some of it is just curiosities discovered along the way, but the real meaty stuff is the in the RESTful APIs - lots of potential here for cool integrations I think…

• Lifting the Lid on OBIEE 12c Web Services - Part 1
• Lifting the Lid on OBIEE 12c Web Services - Part 2
• Extended Subject Areas (XSA) and the Data Set Service
• Changes in BI Server Cache Behaviour in OBIEE 12c : OBIS_REFRESH_CACHE
• Dynamic Naming of OBIEE 12c Service Instance Exports
• OBIEE 12c - “Add Data Source” in Answers

(Photo credit: https://unsplash.com/@jluebke)

Presentation Services can provide some very detailed logs, useful for troubleshooting, performance tracing, and general poking around. See here for details.

There’s no bi-init.sh in 12c, so need to set up the LD_LIBRARY_PATH ourselves:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/oracle/biee/bi/bifoundation/web/bin/:/app/oracle/biee/bi/lib/:/app/oracle/biee/lib/:/app/oracle/biee/bi/bifoundation/odbc/lib/

Run sawserver with flag to list all log sources

/app/oracle/biee/bi/bifoundation/web/bin/sawserver -logsources > saw_logsources_12.2.1.txt

Full list: https://gist.github.com/rmoff/e3be9009da6130839c71181cb58509a0

In OBIEE 12c data-model-cmd is a wrapper for some java code which ultimately calls an internal RESTful web service in OBIEE 12c, bi-lcm. We saw in the previous post how these internal web services can be opened up slightly, and we’re going to do the same again here. Which means, time for the same caveat:

None of these Web Services are documented, and they should therefore be assumed to be completely unsupported by Oracle. This article is purely for geek interest. Using undocumented APIs leaves you at risk of the API changing at any time.

exportServiceInstance will export the RPD, Presentation Catalog, and Security model (application roles & policies etc – but not WLS LDAP) into a single .bar file, from which they can be imported to another environment, or restored to the same one at a later date (e.g. for backup/restore).

To run exportServiceInstance you need to launch WLST first. The following demonstrates how to call it, and embeds the current timestamp & machine details in the backup (useful info, and also makes the backup name unique each time).

So this had me scratching my head for a good hour today. Comparing SampleApp v511 against a vanilla OBIEE 12c install I’d done, one had “Add Data Source” as an option in Answers, the other didn’t. The strange thing was that the option wasn’t there in SampleApp – and usually that has all the bells and whistles enabled.

After checking and re-checking the Manage Privileges option, and even the Application Policy grants, and the manual, I hit MoS - and turned up Doc ID 2093886.1.

I had the pleasure of not one but two fry-ups in York, UK last weekend.

The first was courtesy of Bill’s Restaurant

Overall, pretty good, and I’ve had much worse. All the ingredients seemed decent. The black pudding was overcooked and almost biscuit-like, but that’s my only serious grumble. The bacon was cooked well. That black pudding, beans and the mashed/friend potato thing were each extra charges annoyed me. Particularly with a hangover, I just want to be able to order a full english, without playing Mastermind to work out what’s in or not.