Presented without comment, warranty, or context - other than these might help a wandering code hacker. When using SigV4, you must specify a ‘host’ parameter boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter. To fix, switch conn_s3 = boto.connect_s3() for conn_s3 = boto.connect_s3(host='s3.amazonaws.com') You can see a list of endpoints here. boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request Make sure you’re specifying the correct hostname (see above) for the bucket’s region.

Similar to the previous issue, the sample config in the docs causes another snafu: OGG-15051 Java or JNI exception: oracle.goldengate.util.GGException: Class not found: "kafkahandler". kafkahandler Class not found: "kafkahandler". kafkahandler This time it’s in the kafka.props file: gg.handler.kafkahandler.Type = kafka Should be gg.handler.kafkahandler.type = kafka No capital T in Type! (Image credit: https://unsplash.com/@vanschneider)

In the documentation for the current release of Oracle GoldenGate for Big Data (12.2.0.1.1.011) there’s a helpful sample configuration, which isn’t so helpful … [...] gg.handler.kafkahandler.ProducerRecordClass = com.company.kafka.CustomProducerRecord [...] This value for gg.handler.kafkahandler.ProducerRecordClass will cause a failure when you start the replicat: [...] Class not found: "com.company.kafka.CustomProducerRecord" [...] If you comment this configuration item out, it’ll use the default (oracle.goldengate.handler.kafka.DefaultProducerRecord) and work swimingly! (Image credit: https://unsplash.com/@vanschneider)

There are various reasons for this error, but the one I hit was that the table name is case sensitive, and returned from Oracle by the JDBC driver in uppercase. If you specify the tablename in your connecter config in lowercase, it won’t be matched, and this error is thrown. You can validate this by setting debug logging (edit etc/kafka/connect-log4j.properties to set log4j.rootLogger=DEBUG, stdout), and observe: (I’ve truncated some of the output for legibility)

I’ve been doing some noodling around with Confluent’s Kafka Connect recently, as part of gaining a wider understanding into Kafka. If you’re not familiar with Kafka Connect this page gives a good idea of the thinking behind it. One issue that I hit defeated my Google-fu so I’m recording it here to hopefully help out fellow n00bs. The pipeline that I’d set up looked like this: Eneco’s Twitter Source streaming tweets to a Kafka topic Confluent’s HDFS Sink to stream tweets to HDFS and define Hive table automagically over them It worked great, but only if I didn’t enable the Hive integration part.

With my new server I bought a UPS, partly just as a Good Thing, but also because I suspect a powercut fried the motherboard on a previous machine that I had, and this baby is too precious to lose ;) The idea is that the UPS will smooth out the power supply to my server, protecting it from surges or temporarily blips in power loss. If there’s a proper power cut, the UPS is connected to my server and can initiate a graceful shutdown instead of system crash.

Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error: java.io.IOException: No input paths specified in job scala> sqlContext.read.json("/u02/custom/twitter/twitter.json") java.io.IOException: No input paths specified in job at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202) Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://: scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json") res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string.

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32! On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error ssh_exchange_identification: read: Connection reset by peer On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

(Ref) The bit that caught me out was this kept failing with Error: Password not present and a Python stack trace that ended with subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1 The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!) Launch shell: export HUE_SECRET_KEY=foobar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell Reset password for hue, activate account and make it superuser from django.contrib.auth.models import User user = User.objects.get(username='hue') user.

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example: [oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 com.fasterxml.jackson.databind.JavaType.isReferenceType()Z apache drill 1.6.0 "the only truly happy people are children, the creative minority and drill users" 0: jdbc:drill:zk=local> SELECT version FROM sys.version; No current connection 0: jdbc:drill:zk=local> Whether SELECT version FROM sys.