Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included)

Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included)

Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed:

org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364"
java.lang.NumberFormatException: For input string: "18446744073709550364"

Looking down the stack trace, this came from org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin.readProcMemInfoFile, which the source code shows is reading /proc/meminfo. Looking at this file on each node showed:

[root@cdh57-01-node-02 hadoop-yarn]# cat /proc/meminfo
MemTotal:       24576000 kB
MemFree:        22123008 kB
MemAvailable:   22123008 kB
Buffers:               0 kB
Cached:          1194376 kB
SwapCached:            0 kB
Active:         73536116 kB
Inactive:       21903364 kB
Active(anon):   64138128 kB
Inactive(anon): 11784916 kB
Active(file):    9397988 kB
Inactive(file): 10118448 kB
Unevictable:    26832052 kB
Mlocked:        26832052 kB
SwapTotal:             0 kB
SwapFree:       18446744073709550384 kB
Dirty:              2008 kB
[...]

Erm … SwapFree is 16 million petabytes???

In my LXC configuration in Proxmox I’d set zero swap, thinking that disabling swap would be a good idea. Evidently not.

As soon as I updated the container Swap to 128Mb, the SwapFree looked better:

[root@cdh57-01-node-02 hadoop-yarn]# cat /proc/meminfo
[...]
SwapTotal:        131072 kB
SwapFree:         129840 kB
[...]

To apply this to all the six container nodes, I could have used the Proxmox web GUI, but took advantage of the CLI to save some time with a little bash iteration over the six container IDs (111 to 116) and the pct set command

for i in 11{1..6}; do pct set $i -swap 512;done

To check the value across each node at once, I used pdsh from my laptop to run the same command on each node directly:

rmoff@asgard:~> pdsh -l root -w cdh57-01-node-0[1-6] "grep Swap /proc/meminfo"|sort
cdh57-01-node-01: SwapCached:            0 kB
cdh57-01-node-01: SwapFree:         515496 kB
cdh57-01-node-01: SwapTotal:        524288 kB
cdh57-01-node-02: SwapCached:            0 kB
cdh57-01-node-02: SwapFree:         523056 kB
cdh57-01-node-02: SwapTotal:        524288 kB
cdh57-01-node-03: SwapCached:            0 kB
cdh57-01-node-03: SwapFree:         523476 kB
cdh57-01-node-03: SwapTotal:        524288 kB
cdh57-01-node-04: SwapCached:            0 kB
cdh57-01-node-04: SwapFree:         523760 kB
cdh57-01-node-04: SwapTotal:        524288 kB
cdh57-01-node-05: SwapCached:            0 kB
cdh57-01-node-05: SwapFree:         522272 kB
cdh57-01-node-05: SwapTotal:        524288 kB
cdh57-01-node-06: SwapCached:            0 kB
cdh57-01-node-06: SwapFree:         519456 kB
cdh57-01-node-06: SwapTotal:        524288 kB

From the Cloudera Manager Cluster Setup page I then clicked Retry and YARN came up successfully.