Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error: java.io.IOException: No input paths specified in job scala> sqlContext.read.json("/u02/custom/twitter/twitter.json") java.io.IOException: No input paths specified in job at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202) Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://: scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json") res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string.

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32! On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error ssh_exchange_identification: read: Connection reset by peer On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

(Ref) The bit that caught me out was this kept failing with Error: Password not present and a Python stack trace that ended with subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1 The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!) Launch shell: export HUE_SECRET_KEY=foobar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell Reset password for hue, activate account and make it superuser from django.contrib.auth.models import User user = User.objects.get(username='hue') user.

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example: [oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 com.fasterxml.jackson.databind.JavaType.isReferenceType()Z apache drill 1.6.0 "the only truly happy people are children, the creative minority and drill users" 0: jdbc:drill:zk=local> SELECT version FROM sys.version; No current connection 0: jdbc:drill:zk=local> Whether SELECT version FROM sys.

I wasted literally two hours on this one, so putting down a note to hopefully help future Googlers. Symptom Here’s all the various errors that I got in the hive-server2.log during my attempts to get a CREATE EXTERNABLE TABLE to work against a MongoDB table in Hive: Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.io.BSONWritable Caused by: java.lang.ClassNotFoundException: com.mongodb.util.JSON Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson Caused by: java.lang.ClassNotFoundException: org.bson.io.OutputBuffer Whilst Hive would throw errors along the lines of:

Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included) Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included) Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed: org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364" java.lang.NumberFormatException: For input string: "18446744073709550364" Looking down the stack trace, this came from org.

I’ve got an EdgeRouter LITE (ERL) which I used as my home router until a powercut fried it a while ago (looks like I’m not the only one to have this issue). The symptoms were it powering on but not giving any DHCP addresses, or after a factory reset responding on the default IP of 192.168.1.1. It was a real shame, because it had been a great bit of kit up until then.

(Previously, previously, previously) Since Proxmox 4 has a recent Linux kernel and mainline one at that, it means that Docker can be run on it. I’ve yet to really dig into Docker and work out when it makes sense in place of Linux Containers (LXC), so this is going to be a learning experience for me. To install Docker, add Backports repo to apt: root@proxmox01:~# cat /etc/apt/sources.list.d/backports.list deb http://ftp.debian.org/debian jessie-backports main And then install:

(Previously, previously) I’ve got a bunch of existing VirtualBox and VMWare VMs that I want to run on Proxmox. Eventually I’ll migrate them to containers, but for the time being run them as “fat” VMs using Proxmox’s KVM virtualisation. After copying the OVA files that I had to the server, I uncompressed them: root@proxmox01:/data04/vms/bdl44-biwa# cd ../bdl44 root@proxmox01:/data04/vms/bdl44# ll total 27249328 -rw------- 1 root root 27903306752 Jun 1 10:14 BigDataLite440.ova root@proxmox01:/data04/vms/bdl44# tar -xf BigDataLite440.

(Previously) With my server in place, I ran a memtest on it … which with 128G took a while ;) And then installed Proxmox 4, using a bootable USB that I’d created on my Mac from the ISO downloaded from Proxmox’s website. To create the bootable USB, create the img file: hdiutil convert -format UDRW -o target.img source.iso and then burn it to USB: sudo dd if=target.img of=/dev/rdiskN bs=1m Replace N with the correct device based on diskutil list output.